Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency
Authors: Owen Queen
Thomas Hartvigsen
Teddy Koker
Huan He
Theodoros Tsiligkaridis
Marinka Zitnik
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Explainability
Interpretability
Time Series
Explanations
Temporal patterns
Model Understanding
Latent space
Self-supervised learning
TL;DR: TimeX is an in-hoc explainer for time series models that generates interpretable attributions and also learns a latent space of explanations, along with landmarks, to summarize groups of informative temporal patterns.

Abstract: Interpreting time series models is uniquely challenging because it requires identifying both the location of time series signals that drive model predictions and their matching to an interpretable temporal pattern. While explainers from other modalities can be applied to time series, their inductive biases do not transfer well to the inherently challenging interpretation of time series. We present TimeX, a time series consistency model for training explainers. TimeX trains an interpretable surrogate to mimic the behavior of a pretrained time series model. It addresses the issue of model faithfulness by introducing model behavior consistency, a novel formulation that preserves relations in the latent space induced by the pretrained model with relations in the latent space induced by TimeX. TimeX provides discrete attribution maps and, unlike existing interpretability methods, it learns a latent space of explanations that can be used in various ways, such as to provide landmarks to visually aggregate similar explanations and easily recognize temporal patterns. We evaluate TimeX on eight synthetic and real-world datasets and compare its performance against state-of-the-art interpretability methods. We also conduct case studies using physiological time series. Quantitative evaluations demonstrate that TimeX achieves the highest or second-highest performance in every metric compared to baselines across all datasets. Through case studies, we show that the novel components of TimeX show potential for training faithful, interpretable models that capture the behavior of pretrained time series models.

Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency
Owen Queen
Thomas Hartvigsen University of Virginia, MIT
Harvard University
owen_ owen__uees harvard edu
hartwigsen@virginia.cc edu
Teddy Koker
Huan He
Theodoros Tsiligkaridis MIT Lincoln Laboratory tsiliell. mit. edu
MIT Lincoln Laboratory
Harvard University
thomas. kokerrll mwt.miieee.
Quan_he@hhnn hawvard edu
Marinka Zitnik Harvard University marinka@hms. harvard. edu
Abstract
Interpreting time series models is uniquely challenging because requires iden- tifying both the location of time series signals that drive model predictions and their matching to an interpretable temporal pattern. While explainers from other modalities can be applied to time series, their inductive biases do not transfer well to the inherently challenging interpretation
time series. We present TIMEX, a time series consistency model for training explainers. TIMEX trains inter- pretable surrogate to mimic the behavior of a pretrained time series model. It addresses the issue of model faithfulness by introducing model behavior consis- tency, a novel formulation that preserves relations in the latent space induced by the pretrained model with relations in the latent space induced by TIMEX. TIMEX provides discrete attribution maps and, unlike existing interpretability methods, it learss latent space of explanations that can be used in various ways, such as provide landmarks to visually aggregate similar explanations and easily recognize temporal patterns. We evaluate TIMEX eight synthetic and real-world datasets and compare its performance against state-of-the-ee interpretability methods. We also conduct studies using physiological time series. Quantitative evaluations demonstrate that TIMEX achieves the highest or second-highess performance in every metric compared to basclines across all datasets. Through case studies, we show that the novel components of TIMEX show potential for training faithful, interpretable models that capture the behavior of pretrained time series models.
Introduction
Prevailing time series models high-capacity pre-trained neural networks [ 2]. which are often seen as black boxes due to their internal complexity and lack of interpretability [3. However, practical requires techniques for auditing and interrogating these models rationalize their predictions. Interpreting time series models poses a distinct set of challenges due to the need to achieve two goals: pinpointing the specific location of time series signals that influence the model's predictions and aligning those signals with interpretable temporal patterns [4]. While explainers designed for other modalities can be adapted time series, their inherent biases can miss important structures in time series, and their reliance on isolated visual interpretability does not translate effectively to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the time series where data are less immediately interpretable. The dynamic nature and multi-scale dependencies within time series data require temporal interpretability techniques.
Research in model understanding and inter- pretability developed post-hoc explainers that treat pretrained models as black boxes and do not need access to internal model parameters, ac- tivations, and gradients. Recent research, how- ever, shows that such post-hoc methods suffer from a lack of faithfulness and stability, among other issues [5] 6 77. model can also be un- derstood by investigating what parts of the input it attends to through attention mapping K 10] and measuring the impact of modifying individ- ual computational steps within a model [ 12
Protrained model's latent space
Input time series
TimeX latent space
Another major line of inquiry investigates inter- nal mechanisms by asking what information the model contains [3 14 [5]. For example, it has been found that even when a language model is conditioned to output falsehoods, it may include a hidden state that represents the true answer in- ternally [16]. The gap between external failure modes and internal states can only be identified by probing model internals. Such representation probing has been used to characterize the behaviors of language models, but leveraging these strate- gies to understand time series models has yet to be attempted. These lines of inquiry drive the
Identify what signals the model uses and where they are v
Identify landmarks that explain model behavior v Figure 1: TIMEX learns latent space of expla- nations and landmarks to summarize groups of informative temporal patterns in time series.
development of in-hoc explainers [17 19 22 [2] that build inherent interpretability into the
model through architectural modifications [8] 19, 23 20 21 or regularization [17 22 However, no in-hoc explainers have been developed for time series data. While explainers designed for other modalities can be adapted time series, their inherent biases do not translate effectively to the uninterpretable nature of time series data. They can miss important structures in time series.
Explaining time series models is challenging for many reasons. First, unlike imaging text datasets, large time series data are not visually interpretable. Next, time series often exhibit dense informative features, unlike more explored modalities such as imaging, where informative features often sparse, In time series datasets, timestep-to-timestep transitions can be negligible, and temporal patterns only show up when looking al time segments and long-ferm frends. In contrast, in language datasets, word-to-word transitions are informative for language modeling and understanding. Further, time series interpretability involves understanding the dynamics of the model and identifying trends or patterns. Another critical issue with applying prior methods is that they treat all time steps as separate features, ignoring potential time dependencies and contextual information; we need explanations that are temporally connected and visually digestible. While understanding predictions of individual samples valuable, the ability to establish connections between explanations of various samples (for example, in an appropriate latent space) could help alleviate these challenges.
Present work. We present TIMEX, novel time series surrogate explainer that produces interpretable attribution masks as explanations over time series inputs (Figure l). ! A key contribution of TIMEX the introduction of model behavior consistency, novel formulation that ensures the preservation of relationships in the latent space induced by the pretrained model, as well as the latent space induced by TIMEX. ② In addition to achieving model behavior consistency, TIMEX offers interpretable attribution maps, which are valuable tools for interpreting the model's 's predictions, generated using discrete straight-through estimators (STEs), a type of gradient estimator that enables end-to-end training of TIMEX models. 3 Unlike existing interpretability methods, TIMEX goes further by learning a latent space of explanations. By incorporating model behavior consistency and leveraging a latent space of explanations, TIMEX provides discrete attribution maps and enables visual aggregation of similar explanations and the recognition of temporal patterns. 4 We test our approach on eight synthetic and real-world time series datasets, including datasets with carefully processed ground- truth explanations to quantitatively benchmark it and compare it to general explainers. state-of of-c--oftthe-at time series explainers, and in-hoc explainers. TIMEX is at htttpss: /tte:://ggtiie /gtggttt..b.... com com/nims- haresrhaararrrr TimeX

---

2
Related work
Model understanding and interpretability. As deep learning models grow in size and complexity, so does the need to help users understand the model's behavior. The vast majority of explainable AI research (XAI)
24 has focused natural language processing (NLP) 25 26 27 and computer
vision (CV) (28 29 30). Commonly used techniques, such as Integrated Gradients [3 and Shapley Additive Explanations (SHAP) 3 and their variants have originated from these domains and gained popularity, XAI has gained significant interest in NLP and CV due to the inherent interpretability of the data. However, this familiarity can introduce confirmation bias [3]. Recent research has
expanded to other data modalities, including graphs [33] 6 and time series 34 35 as outlined
below. The literature primarily focuses on post-hoc explainability, where explanations are provided for a trained and frozen model's behavior [36] 371. However, saliency maps, a popular approach
38 31 39), have pitfalls when generated post-hoc: they are surprisingly fragile (5). and lack sensitivity to their explained models [4)]. Surrogate- basee approaches have also been proposed
41 42 43), but these simplified surrogate models fall short compared to the original predictor
they aim to explain. Unlike post-hoc explainability. in-hoc methods aim for inherently interpretable models. This can be accomplished by modifying the model's ' architecture (20), training procedure
using bointly-trained explainers [4), adversarial training [55 46 47 48 regularization techniques
17 [2]] or refactorization of the latent space [49 50]. However, such models often struggle to
achieve state-of-the-ar predictive performance, and to date, these methods have seen limited use for time series.
Beyond instance-based explanations. Several methods have been proposed to provide users with information on model behavior beyond generating instance-based saliency maps explaining individual predictions. Prototype models strive to offer representative sample or region in the latent space 52). Such methods are inherently interpretable, as predictions are directly tied to patterns in the feature space. Further, explainability through human- interprraater exemplars has been gaining popularity. Concept-basee methods decompose model predictions into inntrrinttrprettbble concepts. Many works rely on annotated datasets with hand-picked concepts (e.g., "stripes" in image of zebra). Relying on access to a priori defined concepts, concept bottleneck models learn a layer that attributes each neuron one concept 23 This limitation has spurred research in concept discovery by composing existing concepts [3 54 or grounding detected objects to natural language (55]. However, the CV focus of these approaches limits their applicability other domains like time series.
Time series explainability. In contrast to other modalities, time series often have multiple variables, and their discriminative information is spread over many timesteps. Building on these challenges, recent works have begun exploring XAI for time series [56] 57
58
61 34] 62]. Many 64 to work with time series 158] WinIT learns saliency
59
60
methods modify saliency maps B35 63 58 or surrogate methods [9
data. Two representative methods are WinIT (65) and Dynamask
maps with temporal feature importances, while Dynamask regularizes saliency maps include temporal smoothing. However, these methods rely on perturbing timesteps [63. causing them to lack faithfulness. Common perturbation choices in CV, like masking with zeros, make less sense for time series [56]. Perturbed time series may be out-of-distribuu resulting in unfaithful explanations akin to adversarial perturbation [67]
for the model due to shifts in shape [66].
3 Problem formulation
Notation. Given is time series dataset D = (X) {(x,, vi)li = 1,...,N I, N) where x, are input samples and yi are labels associated to each sample. Each sample x; E RRxx is said to have T time steps and sensors. A feature is defined as a time-sensor pair, where the time L and sensor k for imput x; is x,[t,k], Without loss of generality. we consider univariate (d = 1) and multivariate (d > 1) settings. Each y E {1. 2, .... C} belongs to one of C' classes. A classifier model consists of an encoder G and predictor F. The encoder G produces
embedding of input x,, i.e. G(x,)= = 2, E Rdd,, while the predictor produces some prediction from the embedding in the form of logit, i.e., F(G(x))) = 0z E 1j where argmax, [r(5) E (1,..., C) is the predicted label.
The latent space induced by G is defined as Z, e.g., G X + Z. We will refer to F F(G()) as the reference model while G is the reference encoder and F is the reference predictor. An explanation is defined as a continuous map of the features that conveys the relative importance of each feature for the prediction. The explanation for sample x; is given as an attribution map Fx(x,) E R T'xd where for any times & t2 and sensors k1, b2. E(x, (t,, ka]) > Exx (t2, k2]) implics that [t[t, ki] is more
3

---

x;
Predictor Model
F
MMBC| Z, z
M (Z, A(Z L))
(
Lasent Space
Explanations
HE
GE
FE
E(x)
g
Figure 2: Overview of TIMEX approach.
important feature for the task than x; [/2, k2). Finally, we define an occlusion procedure whereby function 9 generates an mask Mx for sample x from the explanation Exx), e.g., (?(((x)) = Mx This mask is applied
derive masked version of the input x" through operation O e.g.. My + x = x". When describing TIMEX, we generally refer to @ as an element-wiss multiplication.
3.1 Self-supervised model behavior consistency
surrogate model for pretrained time series models. The surrogate model produces explanations by optimizing two main objectives: interpretability and faithfulness model behavior. First, TIMEX generates interpretable explanations
TIMEX creates an
an attribution map E(x:) that identifies succinct, connected regions of input important for the prediction. To ensure faithfulness to the reference model, we introduce novel objective for training TIMEX: model behavior consistency (MBC). With MBC, TIMEX model learns to mimic internal layers and predictions of the reference model, yielding a high-fidelity time series explainer. MBC is defined as:
Definition 3.1 (Model Behavior Consistency (MBC)). Explanation and explanation encoder GE are consistent with pretrained model G and predictor F on dataset 'D if the following is satisfied:
Consistent reference encoder: Relationship between 2; (xx,) and z G((x) in the space of reference encoder is preserved by the explainer, z G (x"") and z G (x("",,, where x." 22 (x(((x;)) @ and x5" 22 [x)x))) @ Xj, through distance functions the reference encoder's and explainer's latent spaces Dz and D2s, respectively, such that: Dz(zi, 2,) Dze (2f 2 for samples X; 'D2
Consistent reference predictor: Relationship between reference predictor jt = F(z,) and latent explanation predictor ū/"h F (z,") is preserved, ~ jf for every sample x; D.
Our central formulation is defined as realizing the MBC between a reference model and an inter- pretable TIMEX model:
Problem statement 3.1 (TIMEX). Given pretrained time series encoder G and predictor F that trained on a time series dataset DD, TIMEX provides explanations E(x,) for every sample x; E D in the form of interpretable attribution maps. These explanations satisfy model behavior consistency through the latent space of explanations Z generated by the explanation encoder GE
TIMEX is designed to counter several challenges interpreting time series models. First, TIMEX avoids the pitfall known as the occlusion problem [68] Occlusion occurs when some features in an input x; are perturbed in an effort that the predictor forgets those features. Since it is well-known known that occlusion can produce out-of nut-of--istribun didstributiin samples 669, this can cause unpredictable shifts in
behavior of fixed, pretrained model 770 71 72 contrast, TIMEX avoids directly masking input samples to G. TIMEX trains an interpretable surrogate G to match the behavior of G. Second, MBC is designed to improve the faithfulness of TIMEX G. By learning to mimic multiple states of F(G(-)) using the MBC objective, TIMEX learns highly-faithie explanations, unlike many post hoc explainers that provide no explicit optimization of faithfulness. Finally, TIMEX's explanations are driven by learning latent explanation space, offering richer interpretability data.

---

TIMEX method
We now present TIMEX, an approach to train an interpretable surrogate model to provide explanations for pretrained time series model. TIMEX learns explanations through consistency learning objective where an explanation generator H and explanation encoder G are trained to match intermediate feature spaces and the predicted label space. We will break down TIMEX in the following sections by components: H the explanation generator, G the explanation encoder, and the training objective of G (HH()) followed by discussion of practical considerations of TIMEX. An overview of TIMEX is depicted in Figure
4.1 Explanation generation
Generating an explanation involves producing an explanation map E(x) where if [x k1)) > E(x(t2, kg]), then feature x[t1, k1] is considered as more important for the prediction than x(t2, ka]- Explanation generation is performed through explanation generator H X + [0, 1]7xd where (x(x) = p. We learn p based on a procedure proposed by [50], but we adapt their procedure for time series. Intuitively, parameterizes a Bernoulli at each time-sensor pair, and the mask Mx is sampled from this Bernoulli distribution during training, i.e., Mx ~ Pp (MM = II., Bern(Pr.x). This parameterization is directly interpretable as attribution scores: low Pr, means that time-sensor pair (t,k) has low probability of being masked-in. Thus, is also the explanation for x;, i.e. F(x) = P.
The generation of p is regularized through divergence with Bernoulli distributions Bern(r), where r is a user-chosen hyperparameter. As (50), denote the desired distribution of p as @(Mx) I[((,)) Bern(r). Then the objective becomes:
Pr.k Σ Pr.s log
1 Pr.
(p) E[DRi.
r + (1-Pr,,) log
1
The sampling of Mx Pp (Mx is performed via the Gumbel-Soffe Softmax trick 73 74], which is
differentiable approximation of categorical sampling. Importantly. Mx is stochastically generated, which, as discussed in [50) (75), regularizes the model to learn robust explanations.
To generate interpretable attribution masks, TIMEX optimizes for the connectedness of predicted distributions:
d T-1 Σ
Leen (p)
(Pi,k P141,8)
(2)
T x
The generator of explanations H learns directly on input time series samples X' to return p. We build transformer encoder-decoder structure for H², using autoregressive transformer decoder and a sigmoid activation to output probabilities for each time-sensor pair.
4.2 Explanation encoding
now describe how to embed explanations with the explanation encoder GE, Intuitively, G learns on the masked distribution of X, which can be denoted as X Motivated by the occlusion problem, we avoid directly applying the masks onto the pretrained, frozen G, as x and are fundamentally different distributions. Therefore, copy the weights of G into G and fine-tune G X
Discretizing attribution masks. When passing inputs G. is important for the end-to-end optimization to completely ignore regions identified unimportant by H Therefore, we use straight-through estimator (STE) [73 to obtain a discrete mask Mxx (0, Tx Introduced by (76). STEs utilize surrogate function to approximate the gradient of a non-differentiable operation used the forward pass, such as binary thresholding.
Applying masks to time series samples. We use two types of masking procedures: attention masking and direct-value masking. First, we employ differentiable attention masking through multiplicative operation proposed by Nguyen et al. [77]. When attention masking does not apply, we use a direct-value masking procedure based on architecture choice or multivariate inputs. We approximate bascline distribution: Bx II No,xNN(ua (uue, o% where μek and of are the mean and
5

---

variance over time-senson pairs. Masking is then performed through a multiplicative replacement as: x = (Mx @ x; + (1 Mx) @ where b Ex-
are discrete instead of continuous. with continuous masks since applying such masks is differentiable with dlement-wiss multipiication. However, continuous masking has a distinctly different interpretation: it uses continuous deformation of the input towards a baseline value. While such an approach is reasonable for data modalities with discrete structures, such as sequences of tokens (as in (78 49 or nodes in graphs (50), such deformation may result in change of the shape of time series data, which is known to be important for prediction [66]. As toy example, consider an input time series x; where the predictive pattern is driven by feature xi[t1, . k1] is larger than all other features. Suppose Mx is continuous. In that case, it is possible that for a less important feature x; [|2, k2]. MM [M ki] < Mx [22, k2] while ( (Mx[t1, ki] O x; [1,, k1]) > (Mx(12, ko] @ xi[12, ka]), thereby preserving the predictive pattern. At the same time, the mask indicates that x, x,(42; kig] is more important than x, (4/4,,A)] A1]. If surrogate model is trained on M,x C X,,
Justification for discrete masking. It is essential that masks My
Previous works have considered masking techniques 77
49 50
Mx may violate the ordinality expected by attribution map as defined in Section 3 Discrete masking alleviates this issue by forcing Mx to be binary, removing the possibility of confounds created by continuous masking. Therefore, discrete masking is necessary when learning interpretable masks on continuous time series.
4.3 Model behavior consistency
The challenge lies in training (F( (.)) faithfully represent F(G()). We approach this by considering the latent spaces of G and G If G considers x, and X, to be similar in Z, we expect that a faithful & would encode x and X5 similarly. However, directly aligning G' and G is not optimal due potential differences in the geometry of the explanation embedding space compared to the full input latent space. To address this, we introduce model behavior consistency (MBC). This novel self-supervised objective trains the explainer model to mimic the behavior of the original model without strict alignment between the spaces. Denote the latent space induced by G and G as Z and z respectively. The MBC objective is thus defined as:
LMBC LMBc(2,
E Σ 2..2 x EE
(Dz(z,2z)) DZE(Z, 2
N
where Dz and D22 are distance functions on the reference model's latent space and the explanation encoder's latent space, respectively, and N is the size of the minibatch, thereby making N equal to the number of pairs on which LBC is optimized. This objective encourages distances to be similar across both spaces, encouraging E to retain similar local topology to Z without performing direct alignment. This is closely related to cycle-consiytency loss, specifically cross-modal cycle- consistency loss [79]. We use cosine similarity for Dz and Dzs throughout experiments in this study, but any distance can be defined on each respective space.
In addition to MBC, we use a label consistency (LC) objective to optimize TIMEX. We train a predictor F on E to output logits consistent with those output by F. We use a Jensen-Shannon Divergence (Dis) between the logits of both predictors:
L..c(2,7%)
Σ Σ (Pis(F(z)))(((2)) - (F MIFE
(((((z)))) (zZ)))²
(4)
Z, EF EEEZ
Our total loss function on F can then be defined as a combination of losses: Lz = LMBC ALcLue Consistency learning justification. MBC offers three critical benefits for explainability. ① MBC enables consistency optimization across two latent spaces Z and _ without requiring that both X and x be encoded by the same model, allowing the learning of E on a separate model E EG F(G())) This avoids the out-of-dine distribuutioon problems induced by directly masking inputs to G. MBC comprehensively represents model behavior for explainer optimization. This in contrast to perturbation explanations [38 80, 58) which seek a label -abee-preserving perturbation P on F(G()) where ((.)))))))) xs F(G()))). ) By using G(x,) and F(G())) to capture the behavior of the reference model, MBC's objective is richer than simple label-preserving objective. 3 While
MBC stronger than label matching alone, it more flexible than direct alignment. An alignment
objective, which enforces z; 2, inhibits 5 from learning important features of explanations not
E

---

represented in Z. The nuance and novelty of MBC are in learning a latent space that is faithful to model behavior while being flexible enough to encode rich relational structure about explanations that can be exploited to learn additional features such as landmark explanations. Further discussion of the utility of MBC is in Appendix
4.4 Learning explanation landmarks and training TIMEX models
Leveraging the latent space, TIMEX generates landmark explanations z IRd Such landmarks are desirable as they allow users to compare similar explanation patterns across samples used by the predictor. Landmarks are learned by landmark consistency loss, and their optimization is detached as not to harm explanation quality. Denote the landmark
from the gradients of the explanations matrix as L E RT xd, where n corresponds to the number of landmarks (a user-chosen value) and d_ is the dimensionality of = For each sample explanation embedding z," we use Gumbel-Sone -Softmxx STE (GS) to stochastically match z to the nearest landmark in the embedding space. Denote the vector of similarities to each z! as s szz
R. L). Then the assimilation A is described as:
(5) where sg denotes the stop-grad function. The objective for learning landmarks is then LBCC (2, A(ZE;L)) L)), optimizing the consistency between the assimilated prototypes and the refer- ence model's latent space. Landmarks are initialized as a random sample of explanation embeddings from - and are updated via gradient descent. After learning landmarks, we can measure the quality of each landmark by the number of 2 embeddings closest to it in latent space. We filter out any landmarks not sufficiently close any samples (described in Appendix [B)
Alz F. L) = GS(soffmax(s(ss(22)) L))),
TIMEX training. The overall loss function for TIMEX has four components: L LMBC + ALCCLC- + Ae + Acon where ALC,ÀE: Acon E E are weights for the label consistency loss, total explanation loss, and connective explanation loss, respectively. TIMEX can be optimized end-to-end, requiring little hyperparameter choices from the user. The user must also choose the parameter for the explanation regularization. Explanation performance stable across choices of r (as found r = 0.5 to remain consistent throughout experiments. A lower value may be provided if the underlying predictive signal is sparse; this hyperparameter is analyzed in Appendix C.3 In total, TIMEX optimizes H., G and F.F
in (50)),
Experimental setup
Datasets. We design four synthetic datasets with known ground-truth explanations: FreqShapes, SeqComb-UV, Comb--VV, and VawVar. Datasets are designed to capture diverse temporal dynamics in both univariate and multivariate settings. We employ four datasets from real-world time series classification tasks: ECG [8] ECG arrhythmia detection; PAM (82] human activity
recognition: Epilepsy 833 EEG seizure detection; and Boiler (84) mechanical fault detection. We define ground-truth explanations for ECG as QRS intervals based on known regions of ECG signals where arrhythmias can be detected. The R, P. and T wave intervals are extracted following 85). Dataset details are given in Appendix [.Taandd C.4
Baselines. We evaluate the method against five explainability basclines. As general explainer. we (IG) 31: for recent time sprccisseecific explainers, we use Dynamask (58). and WinIT [86] for an explainer that uses contrastive learning, we use CoRTX 887: and for an in-hoc explainer which has been demonstrated for time series, we use SGT + Grad
use integrated gradients
Evaluation. We consider two approaches. Ground-trufh explanations: Generated explanations are compared to ground-truit explanations, i.e., known predictive signals in each input time series sample when interpreting a strong predictor, following established setups [6]. We use the area under precision (AUP) and area under recall (AUR) curves evaluate the quality of explanations (58). We also use the explanation AUPRC, which combines the results of AUP and AUR. For all metrics, higher values are better. Definitions of metrics are in Appendix CC4 Feature importance under occlusion: We occlude the bottom p-percentile of features as identified by the explainer and measure the change in prediction AUROC (Sec. [.2). The most essential features a strong explainer identifies should retain prediction performance under occlusion when p is high. To control for potential misinterpretations, we include a random explainer reference. Our experiments use transformers [8X] with time-based positional encoding, Hyperparameters and compute details are given in Appendix
7

---

FreqShapes
SeqComb- UV
Method
AUPRC
AUR
AUPRC
AUP
IG
0.7516=000003
0.6912
0.5975=40.0020
0.5760-+0.002
08157:20.002
0. 0.2868-=000022
Dynamask
0.2001=000013
0.2952=000057
0.5037=00.015
0:4421=000016
0.8782+000009
0.0029=0000007
WinfT
0,5071=:0.002
0.5546:10.0026
0.4557=10.0016
0.0067
0.7072-000007
0.2253:10.0016
CURTX
0.6978110.0156
0.4938.10.0004
0..061-10.0012
0.5643.10.0024
0.824110.0022
0.17491000007
SGT Grad
0.5312==0.0019
0.4138=000011
0.3931=0.0015
0.5731#000021
0.7828=000013
0.2136::000008
TIMEX
0.8324=0.0034
0.7219=000031
0.7124=0.0017 0.6081=0.0022
0.0411=000006
0.3300=000014
Table 1: Attribution explanation performance on univariate synthetic datasets.
SegComb-MV AUP
Low Var AUP
Method
AUPRC
AUR
AUPRC
AUR
IG
0.3298=10.0015
0.7483:10.0027
0.2581=::00000
0.8691:0000055
0.4027:000029
1.1155-000016
Dynamask
0.3136=0.0019
0.5481=0.0053
0.1053=000025
0.1391=0.0012
0.1640=0.003
0.2106=0.0018
WinlT
0.2809=000018
0.7594=:00002
0.2077=#0.0021
0.1667=:00005
0.1140=000022
0.3842==000017
CoRTX
0.3029110.022
0.5625-10.0006
0.0011
0.4983.10.0014
0.3281 10.0027
0..011
SGT Grad
0.4893-10.0005
0.4970:10.0005
0..019
0.3449110.0010
0.2133:10.0029
0.3528:10.0015
TIMEX
0.6078=0.0021
0.8326=0.0008
0.3872=0.0015
0.8673=000033
0.5451=000028
0.900400.0024
Table 2: Attribution explanation performance on multivariate synthetic datasets.
Results
R1: Comparison existing methods on synthetic and real-world datasets.
Synthetic datasets. We compare TIMEX to existing explainers based on how well they can identify essential signals in time series datasets. Tables M show results for univariate and multivariate datasets, respectively. Across univariate and multivariate settings, TIMEX is the best explainer on 10/12 (3 metrics in datasets) with an average improvement in the explanation AUPRC (10.01 %), AUP (6.01%) and AUR (3.35%) over the strongest baselines. Specifically, TIMEX improves ground- truth explanation in terms of AUP by 3.07% on FreqShapes, 6.3% on SeqComb-US UV,, 8.43% on SeqComb-MV, and 6.24% on Low Var over the strongest baseline each dataset. In all of these settings, AUR less critical than AUP since the predictive signals have redundant information. TIMEX achieves high AUR because it is optimized to output smooth masks over time, tending to include larger portions of entire subsequence patterns than sparse portions, which is relevant for human interpretation. To visualize this property, show TIMEX's explanations in Appendix C.5]
Real-world datasets: arrhythmia detection. We demonstrate TIMEX on ECG arrhythmia detection. TIMEX's attribution maps show state-of-the-art performance for finding relevant QRS intervals driving the arrhythmia diagnosis and outperform the strongest baseline by 5.39% (AUPRC) and 9.83% (AUR) (Table 3 Integrated gradients achieve slightly higher AUP, whereas state-of-the-ar art time series explainers perform poorly. Notably, TIMEX's 's explanations are significantly better in AUR, identifying larger segments of the QRS interval rather than individual timesteps.
Ablation study on ECG data. We conduct ablations on the ECG data using TIMEX (Table 3 First, we show that the STE improves performance opposed to soft attention masking, resulting in an AUPRC performance gain of 9.44%; this validates our claims about the pitfalls of soft masking for time series. Note that this drop in performance becomes more significant when including direct-value masking, as shown in Appendix C.6. Second, we use SimCLR loss to align Z E to Z as opposed to MBC; SimCLR loss can achieve comparable results in AUPRC and AUR, but the AUP is 13.6% lower than the base TIMEX. Third, we experiment with the usefulness of MBC and LC objectives. MBC alone produces poor explanations with AUPRC at 65.8% lower score than the base model. LC alone does better than MBC alone, but its AUPRC is still 21.5% lower than the base model. MBC and LC, in conjunction, produce high-quality explanations, showing the value in including more intermediate states for optimizing (*(HHE Extensive ablations are provided in Appendix C.6
R2: Occlusion experiments on real-world datasets.
We evaluate TIMEX explanations by occluding important features from the reference model and observing changes in classification 63 58 87]. Given generated explanation E(x;), the bottom p-percentile of features are occluded: we expect that if the explainer identifies important features for the model's prediction, then the classification performance to drop significantly when replacing these important features (determined by the explainer) with baseline values. We compare the performance under occlusion to random explanations to counter misinterpretation (Sec. 3.I. We adopt the masking
8

---

ECG
TIMEX
FCG
Method
AUPRC
AUP
AUR
Ablations
AUPRC
AUP
AUR
IG
0.182210.0014
0.5949=10.0023
0.1204:10.0012
Full
0.47211.0.018
0.5663-10.0025
0.4457:10.0018
Dynamask
0.3280:10.001
0.5249110.0030
0.1082:1:000000
STE
0.4014:10.0019
0.5570:10.0032
0.1364:10.0007
WinIT
0.3049.10.00)
00.001
0.3474:10.0011
SimCCL
0.4767110.0021
0.489510.0024
0.4779:1000013
CoRTX
0,3735:50.008
0.496:=0.0021
0.3031+:0.0000
Only LC
0,3704:50.0088
0.3296:2.0019
0.5084-20.0088
SGT Grad
0.3144::00010
0.4241=0.0024
0.2639:+0.0013
Only MBC
0.1615=0.0006
0.1348=0.0006
0.5504::0.011
TIMEX
0.472111:00018
0.00055
0.44571:00018
||
Table 3: (Left) Benchmarking TIMEX the ECG dataset. (Right) Results of ablation analysis.
816
0.
Preditton AUPOO
0.50
2.00
Predetinn
0.75
0.05
0.70
TrreX
2465
Danarask
3.0
2:90
(a) Rpilepsy
PAM
(e) Boiler
Figure 3: Occlusion experiments on real-world datasets. Higher values indicate better performance.
procedure described in Sec. masking otherwise.
4.2 performing attention masking where applicable and direct-value
compares TIMEX to Dynamask, a strong time-series explainer. On all datasets, TIMEX's explanations are either or above the performance Dynamask, and both methods perform above the random baseline. On the Boiler dataset, demonstrate average of 27.8% better classification AUROC across each threshold than Dynamask, with up to 37.4% better AUROC at the 0.75 threshold. This gap in performance between TIMEX and Dynamask is likely because the underlying predictor for Boiler is weaker than that of Epilepsy or PAM, achieving 0.834 AUROC compared to 0.979 for PAM and 0.939 for Epilepsy. We hypothesize that TIMEX outperforms Dynamask because it only considers changes in predicted labels under perturbation while TIMEX optimizes for consistency across both labels and embedding spaces in the surrogate and reference models. TIMEX performs well across both univariate (Epilepsy) and multivariate (PAM and Boiler) datasets.
Figure SesureBcomee
R3: Landmark explanation analysis on ECG.
To demonstrate TIMEX's landmarks, we show how landmarks serve as summaries of diverse patterns in an ECG dataset. V
visualizes the learned landmarks in the latent space of explanations. We choose four representative landmarks based on the previously described landmark ranking strategy (Sec. (.4). Every landmark occupies different regions of the latent space, capturing diverse types of explanations generated by the model. We show the three nearest explanations for the top two landmarks regarding the nearest neighbor in the latent space. Explanations O. 2. and 3 are all similar to each other while distinctly different from ④ 2 and @, both in terms of attribution and temporal structure. This visualization shows how landmarks can partition the latent space of explanations into interpretable temporal patterns. We demonstrate the high quality of learned landmark explanations through a quantitative experiment in Appendix C.10)
Additional experiments demonstrating flexibility of TIMEX.
In the Appendix, we present several additional experiments to demonstrate the flexibility and superi- ority of TIMEX. In Appendix [C8) we replicate experiments on two other time series architectures. LSTM and CNN, and show that TIMEX retains -tate-of-the-art performance. In Appendix C..9 we demonstrate the performance of TIMEX multiple task settings, including forecasting and peegullary-samplle time series classification. TIMEX can be adapted to these settings with minimal effort and retains excellent explanation performance.
7 Conclusion
develop TIMEX, an interpretable surrogate model for interpreting time series models. By introducing the novel concept of model behavior consistency (i.e., preserving relations in the latent space induced by the pretrained model when compared to relations the latent space induced by the explainer), ensure that TIMEX mimics the behavior of pretrained time series model, aligning

---

Lalent Space of Explanations
Figure 4: Landmark analysis of TIMEX the ECG dataset. Shown is UMAP plot of the latent explanation space along with learned landmark explanations. For two selected landmarks (in red), we show three explanation instances most similar each landmark.
influential time series signals with interpretable temporal patterns. The generation of attribution maps and utilizing latent space of explanations distinguish TIMEX from existing methods, Results on synthetic and real-world datasets and case studies involving physiological time series demonstrate the superior performance of TIMEX compared to state-of-the off-the-att art interpretability methods. TIMEX's innovative components offer promising potential for training interpretable models that capture the behavior of pretrained time series models.
Limitations. While TIMEX is not limited a specific task explainer, our experiments focus on time series classification. TIMEX can explain other downstream tasks, assuming we can access the latent pretrained space, meaning could be used to examine general pretrained models for time series. Appendix C.9 gives experiments on various setups. However, the lack of such pretrained time series models and datasets with reliable ground-truth explanations restricted our testing this area. One limitation of our approach is its parameter efficiency due to the separate optimization of the explanation- -uned model. However, we conduct runtime efficiency test in Appendix C.7 that shows TIMEX has comparable runtimes to baseline explainers. Larger models may require adopting parameter-efficient tuning strategies.
Societal impacts. Time series data pervades critical domains including finance, healthcare, energy, and transportation. Enhancing the interpretability of neural networks within these areas has potential to significantly strengthen peocisionmmakinne processes and foster greater trust. While explain- ability plays a crucial role in uncovering systemic biases, thus paving the way for fairer and more inclusive systems, it is vital to approach these systems with caution. The risks of misinterpretations or an over-reliance on automated insights are real and substantial. This underscores the need for a robust framework that prioritizes human-centeree evaluations and fosters collaboration between humans and algorithms, complete with feedback loops to continually refine and improve the system. This approach will ensure that the technology serves to augment human capabilities, ultimately leading to more informed and equitable decisions across various sectors society.
Acknowledgements
We gratefully acknowledge the support of the Under Secretary of Defense for Research and Engineer- ing under Air Force Contract No. FA8702-15-D 15-D-0001 and awards from NIH under No. ROIHD RO1HD108794 108794, Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, Bayer Early Excellence Science, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Pfizer Research, Chan Zuckerberg Initiative, and the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Any opinions, findings, conclusions, or recommenda- tions expressed in this material those of the authors and do not necessarily reflect the views of the funders. The authors declare that there are no conflicts of interest.
References
[1] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via frequency consistency. Advances in Neural
10

---

Information Processing Systems, 2022.
[2] Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. TemeNet: pre-trained deep recurrent neural network for time series classification, arXiv: 700.0/8800.8.8888 2017.
[3] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
[4] Ferdinand Küsters, Peter Schichtel, Sheraz Ahmed, and Andreas Dengel. Conceptual explanations of neural network prediction for time series. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-6, 2020.
[5] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable Al: Interpreting, explaining and visualizing deep learning, pages 267-280, 2019.
[6] Chirag Agarwal, Owen Queen, Himabindu Lakkaraju, and Marinka Zitnik. Evaluating explain- ability for graph neural networks. Scientific Data, 10(1):144, 2023.
[7] Chirag Agarwal, Marinka Zitnik, and Himabindu Lakkaraju. Probing gnn explainers: A rigorous theoretical and empirical analysis of gnn explanation methods. In International Conference on Artificial Intelligence and Statistics, pages 8969-8996. PMLR, 2022.
[8] Jesse Vig. A multiscale visualization of attention the transformer model. arXiv: 1rXiv:19060057// 2019.
[9] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. ACL, 2020.
[10] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv:2209.1188 11895, 2022.
[11] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359 2022.
[12] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33: 12388-1401,, 2020.
[13] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguisties, 8:842-866 866, 2021.
[14] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv:2104, 08696, 2021.
[15] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-22) 2022.
[16] Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv:2212.03822 2022.
[17] Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[18] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé IIII and Aarti Singh, editors, Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5338-5348. PMLR, 13-18 Jul 2020.
[19] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699 2021.
[20] Bryan Lim, Sercan Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 17(4):17481777 2021.
[21] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. ArXiv, abs/1905. 2019,
11

---

[22] Ryan Henderson, Djork-Amé Clevert, and Floriane Montanari. Improving molecular graph neural network explainability with orthnnoraaliation and induced sparsity, In International Conference on Machine Learning, pages 2203-4213. PMLR, 2021.
[23] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In International Conference on Learning Representations, 2023.
[24] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning.
arXiv preprint arXiv: 702.08608, 2017.
[25] Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj
Sen. survey of the state of explainable ai for natural language processing. arXiv preprint arXiv:2010.00711, 2020.
[26] Andreas Madsen, Siva Reddy, and Sarath Chandar. Post-hoc interpretability for neural nip: A survey. ACM Computing Surveys, 55(8):1-42, 2022.
[27] Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Salvatore Rinzivillo. Benchmarking and survey of explanation methods for black box models. arXiv preprint arXiv:2102. 13076, 2021.
[28] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis, Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18, 2020.
[29] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In British Machine Vision Conference, 2018.
[30] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: A review of methods and applica- tions. Proceedings of the IEEE, 109(3):247-278, 2021.
[31] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017.
[32] Marzych Ghassemi, Luke Oakden-Ray Rayner, and Andrew L Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(1 (11))745--7500 2021.
[33] Zhitao Ying. Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.
[34] Udo Schlegel, Hiba Arnout, Mennatallah El-Assady, Daniela Oelke, and Daniel Keim. Towards a rigorous evaluation of xai methods on time series. In 2019 IEEE/CVE International Conference on Computer Vision Workshop (ICCVW), pages 4197-4201. IEEE, 2019.
[35] Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. Bench- marking deep learning interpretabilityy in time series predictions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6441-6.52. Curran Associates, Inc., 2020.
[36] Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on Al,
Ethics, and Society, pages 131-138, 2019.
[37] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. Proceedings of the AAAI conference on artificial intelligence, 32((), 2018.
[38] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Swirzerland, September 6- 12, 2014, Proceedings, Part I 13, pages 818-833 138-333. Springer, 2014.
[39] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618-626. 2017.
[40] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
12

---

[41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?"; Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, page 1135-1144, San Francisco California USA, Aug 2016. ACM.
[42] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence, 2(1):56-67, 2020.
[43] Haoran Zhang, Quaid Morris, Berk Ustun, and Marzyeh Ghassemi. Learning optimal predictive checklists. Advances in Neural Information Processing Systems, 34:1215-1229) 2021.
[44] Indro Spinelli, Simone Scardapane, and Aurelio Uncini. A meta-learning approach for training explainable graph neural networks. IEEF Transactions on Neural Networks and Learning Systems, page 1-9, 2022.
[45] Theodoros Tsiligkaridis and Jay Roberts. Understanding and increasing efficiency of frank- wolfe adversarial training. In CVPR, 2022.
[46] Jay Roberts and Theodoros Tsiligkaridis. Controllably Sparse Perturbations of Robust Classifiers for Explaining Predictions and Probing Learned Concepts. In Daniel Archambault, Ian Nabney, and Jaakko Peltonen, editors, Machine Learning Methods in Visualisation for Big Data. The Eurographics Association, 2021.
[47] Jay Roberts and Theodoros Tsiligkaridis. Ultrasound diagnosis of covid-19: Robustness and explainability. In Medical Imaging meets NeurIPS Workshop. 2020.
[48] Ian E. Nielsen, Dimah Dera, Ravi P. Ramachandran Ghulam Rasool, and Nidhal Carla Bouay. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine, 39(4), July 2022.
[49] Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with dif- ferentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy, July 2019. Association for Computational Linguistics.
[50] Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 15524-15543 PMLR, 17-23 Jul 2022.
[51] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[52] Srishti Gautam, Ahcene Boubekki, Stine Hansen, Suaiba Salahuddin, Robert Jenssen, Marina Höhne, and Michael Kampffmeyer. Protovae: A trustworthy self-explainable prototypical variational model. Advances in Neural Information Processing Systems, 35:17940 1777949--799,2 2022.
[53] Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv: 1707,0338) 2017.
[54] Tailin Wu, Megan Tjandrasuwita. Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosic, and Jure Leskovee. Zeroc:
syuro--ymbolic model for zero-shot concept recognition and acquisition at inference time. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[55] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. International Conference on Learning Representations, 2019.
[56] Prathyush S Parvatharaju, Ramesh Doddaiah, Thomas Hartvigsen, and Elke A Rundensteiner. Learning saliency maps to explain deep time series classifiers. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pages 1406-1415, 2021.
13

---

[57] Ramesh Doddaiah, Prathyush Parvatharaju, Elke Rundensteiner, and Thomas Hartvigsen. Class- specific explainability for deep time series classifiers. In Proceedings of the International Conference on Data Mining, 2022.
[58] Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2166-2111 2177. PMLR, 18-24 Jul 2021.
[59] João Bento, Pedro Saleiro, André F. Cruz, Mário A. T. Figueiredo, and Pedro Bizarro. Timeshap: Explaining recurrent models through sequence perturbations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, page 2573,, Aug 2021. arXiv:2012.00073 [es].
[60] Riccardo Guidotti and et al. Explaining any time series classifier. International Conference on Cognitive Machine Intelligence, 2020.
[61] Fand Mujkanovie. timexplain-a framework for explaining the predictions of time series classifiers. ArXiv, 2020.
[62] Thomas Rojat, Raphaël Puget, David Filliat, Javier Del Ser, Rodolphe Gelin, and Natalia Rodríguur Explainable artificial intelligence (xai) on timeseries data: A survey, arXiv preprint 10..xx:214400955 2021.
[63] Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David K Duvenaud, and Anna Goldenberg. What went wrong and when? instance-wise feature importance for time-series black-box models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 799-809. Curran Associates, Inc., 2020.
[64] Torty Sivill and Peter Flach. Limesegment: Meaningful, realistic time series explanations. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, page
3418-3433, PMLR, May 2022.
[65] Clayton Rooke, Jonathan Smith, Kin Kwan Leung, Maksims Volkovs, and Saba Zuberi. Temporal dependencies in feature importance for time series predictions. arXiv preprins arXiv:2107.14311 2021.
[66] Lexiang Ye and Eamonn Keogh. Time series shapelets: new primitive for data mining. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 947-956, 2009.
[67] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.60 2013.
[68] Q Vera Liao and Kush R. Varshney. Human-centered explainable ai (xai): From algorithms to user experiences, 2022.
[69] Mohammad Azizmalayeri, Arshia Soltani Moakar, Arman Zarei, Reihaneh Zohrabi, Moham- mad Taghi Manzuri, and Mohammad Hossein Rohban. Your ofits--dist detection method is not robust! In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[70] Peter Hase, Harry Xie, and Mohit Bansal. The search methods for feature importance explanations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 3650-3666. Curran Associates, Inc., 2021.
problem in explainability and
[71] Cheng- Yu Hsieh, Khih-Kuan Yeh, Xuanqing Liu, Pradeep Kumar Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsich. Evaluations and methods for explanation through robustness analysis. In International Conference Learning Representations, 2021.
[72] Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yueyuan Zheng. Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. Resisting out-of-distribution data problem in perturbation of xai. arXiv preprint arXiv:2107.1 14000, 2021.
[73] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In International Conference Learning Representations, 2017.
14

---

[74] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation discrete random variables. In International Conference on Learning Representations. 2017.
[75] Siqi Miao, Yunan Luo, Mia Liu, and Pan Li. Interpretable geometric deep learning via learnable randomness injection. In The Eleventh International Conference on Learning Representations, 2023.
[76] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
[77] Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. Differentiable window for dynamic local attention. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages Linguistics.
Online, July 2020. Association for Computational
[78] Tao Lei. Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas, November 2016. Association for Computational Linguistics.
[79] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic contrastive language-iman image pretraining. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[80] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference computer vision, pages 2950-2953 2958, 2019.
[81] George B. Moody and Roger Mark. The impact of the mit-bih arrhythmia database. IEEE Engineering in Medicine and Biology Magazine, 20:45-50, 2001.
[82] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.
[83] Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and dimee-dimensional structures in time series of brain electrical activity: Dependence recording region and brain state. Physical Review E, 64(6):06190) 1907, 2001.
[84] R. Shohet, M. Kandil, and J.J. McArthur. Simulated boiler data for fault detection and classifi- cation, 2019.
[85] Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. NeuroKit2: A python toolbox for neurophysiologica signal processing. Behavior Research Methods, 1669 feb 2021.
[86] Kin Kwan Leung, Clayton Rooke, Jonathan Smith, Saba Zuberi, and Maksims Volkovs. Tempo- ral dependencies feature importance for time series prediction. In The Eleventh International Conference on Learning Representations, 2023.
[87] Yu-Neng Chuang, Guanchu Wang, Fan Yang, Quan Zhou, Pushkar Tripathi, Xuanting Cai, and Xia Hu. Cortx: Contrastive framework for real-time explanation. ArXiv, abs/2303.02794, 2023.
[88] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, In NIPS, 2017.
15

---

Appendix A Further discussion of background
Straight-through estimators. Discrete operations, such as thresholding, are often avoided in neural network architectures due to difficulties differentiating discrete functions. To circumvent these issues, [76) introduces the straight-through estimator (STE), which uses a surrogate function during backpropagation to approximate the gradient for non-differentiable operation. STEs have seen usage in quantized neural networks [S1]. This method shows empirical performance even though there is little theoretical justification behind it [S2].
Self-supervised learning. Methods in sulf--sp learning (SSL) have become a common pretraining technique for settings in which large, unlabeled datasets are available [S3, S4, S5, S6]. Common approaches for self-supervised learning are contrastive learning, which seeks learn representations for samples under invariant data augmentations, and metric learning, which aims to generate a latent space in which a distance function captures some pre-defined relations on data [S7]. Consistency learning has emerged as another promising SSL approach; intuitively, this family of methods seeks to learn latent spaces in which similar pairs are expected to be embedded similarly, i... preserving some consistent properties. Consistency learning has seen in aligning videos [S8], enhancing latent geometry for multimodal contrastive learning [79], and pretraining time series models across time and frequency domains (.]
The use of consistency for explaining machine learning models. Consistency has been considered in previous XAI literature in two ways: I) consistency between explanations and 2) consistency as an explainability metric, as explained below.
Consistency between explanations: This notion has been introduced in previous works in ex- plainability literature. Pillai et al. [S9] train saliency explainer via contrastive learning that preserves consistency across the saliency maps for augmented versions of images. A few other works have explored maintaining consistency of explanations across various perturbations and augmentations, specifically computer vision [S10, S11]. In one of the only previous works to consider explanation consistency in time series, Watson et al. [S12] train explainer an ensemble of classifiers to optimize the consistency of explanations generated by an explainer applied to each classifier. TIMEX does not seek to optimize consistency between explanations but rather a consistency to the predictor model on which is explaining.
Dasgupta al. [S13] defines explanation consistency as similar explanations corresponding to similar predictions; this metric is then used as proxy faithfulness to evaluate the quality of explainers. However, Dasgupta al. use consistency to evaluate explainers, not to train and design new explainer method. TIMEX uses consistency as a learning objective rather than simply a metric.
Consistency as an explainability metric:
Our work differs from these previous formulations of explanation consistency. We seek to optimize the consistency not between explanations directly, as mentioned in previous works, but rather between the explainer and the model it is tasked with explaining. MBC attempts to ensure that the behavior of the explainable surrogate matches that of the original model. The definition of consistency in Dasgupta al. the closest to our definition of MBC; however, Dasgupta et al. seek not to optimize the consistency of explainers but rrather to evaluate the output of post-hoc explainers. TIMEX directly optimizes the consistency between the surrogate model and the original predictor through the MBC loss, novel formulation that seeks to increase the faithfulness of explanations generated by TIMEX.
Appendix B Further theoretical discussions
B.1 Differentiable attention masking
As is described in Section 4.1 we use differentiable attention masking (77). which is defined as such:
QKT O Mx)V,
a
(softmax
where Q, K, V represent query, key and values operators, de is used as a normalization factor, and Mxx is mask with self-attention values. This procedure is fully differentiable, and given that M is binarized via the STE, it sets all attention values zero that are be ignored based on output from E

---

B.2 Further discussion on the utility of model behavior consistency
The model behavior consistency (MBC) framework in TIMEX is a method to train interpretable surrogate model G In Section 4.3 we discuss the conceptual advances of this approach. Here, we will outline another advantage of the discussion on the broader uses of MBC in other domains and applications.
classification performance- and brief
Training, model with an interpretability bottleneck such as TIMEX is often challenging, as the inherent interpretability mechanism can hinder the performance and expressiveness the method; this is an advantage of post-hoc methods. MBC allows one to preserve the performance of the underlying predictor. TIMEX, surrogate method, allows one to keep the predictions from a pretrained time series encoder and develop explanations on top of it, which is practical for real-world use when drop in classification performance is highly undesirable.
MBC is not limited to time series classification tasks. We demonstrate the utility of MBC for time series due the particularly challenging nature of the data modality and the lack of available time series explainers. However, MBC gives general framework for learning interpretable surrogate models through learning the H and G modules. MBC also has the potential to be applied to tasks outside of classification; since MBC is defined on the embedding space, any model with such an embedding space could matched through a surrogate model as TIMEX. This opens the possibility of learning on general pretrained models or even more complex tasks such as forecasting (as shown in Appendix C.9). Finally, we see MBC as having potential beyond explainability as well; one could imagine MBC being way to distill knowledge into smaller models [S14, S15, S16]. We leave these discussions and experiments for future work.
B.3 Explanation landmark selection strategy
We proceed describe how landmarks are selected for final interpretation. As described in Section 4.4 landmarks are initialized with the embeddings given by G for random number of training samples. Practically, we stratify this selection across classes in the training set. Landmarks are then updated during the learning procedure. After learning landmarks, not every landmark will be helpful as an explanation; thus, we perform a filtration procedure. Intuitively, this filtration consists of detecting landmarks for which the landmark is the nearest landmark neighbor for many samples. This procedure described in Algorithm| I
Algorithm 1: Landmark filtration
Input: Landmark matrix L E Rd= training explanation embeddings {zf ..... 2 for 2 E IR: threshold number neighbors n. E N
N -t
for
+-1 to N do
Compute similarity to all landmarks S = sim(2 LL)
Jmax < argmax, S (Gets nearest landmark for sample explanation i) Append Jmmx to N₁
end
F < Frequency of occurrence of each unique element in N;
Liter + every landmark in L s.t. F; 2 n.
Return Laher
Appendix C
Additional experiments and experimental details
C.1 Description of datasets
We conduct experiments using both synthetic and real-world datasets. This section describes each synthetic and real-world dataset, including how ground-truth explanations are generated when applicable.

---

C.1.1 Synthetic datasets
We employ synthetic datasets with known ground-truth explanations to study the capability to identify the underlying predictive signal. We follow standard practices for designing synthetic datasets, including tasks that are predictive and not susceptible to shortcut learning [S17] induced by logical shorteuts. These principles are defined in [S18] concerning graphs, but we extend these to synthetic datasets for time series. Each time series is initialized with non-autoregressive moving average (NARMA) noise base, and then the described patterns are inserted. We will briefly describe the construction of each time series dataset in this section, and the codebase contains full details at https: / com/mims harvard/Time.. We designed four synthetic datasets to test different time series dynamics:
FreqShapes. Predictive signal is determined by the frequency of occurrence of an anomaly signal. To construct the dataset, take two upward and downward spike shapes and two frequencies, 10 and 17 time steps. There are four classes, each with a different combination of the attributes: class has downward spike occurring every 10-time steps, class 1 has upward spike occurring every 10-time steps, class has a downward spike occurring every 17-time steps, and class has an upward spike occurring every 17-time steps. Ground-truth explanations are the locations of the upward and downward spikes.
Predictive signal is defined by the presence of two shapes of subsequences: increasing (I) and decreasing (D) trends. First, two subsequence regions are chosen within the time series so neither subsequence overlaps; each subsequence is 10-20 time steps long. Then, pattern is inserted based on the class identity; the increasing or decreasing trend is created with a sinusoidal noise with randomly-chosen wavelength. Class 0 is null, following strategy in [S18] that recommends using null classes for simple logical identification tasks in synthetic datasets. Class I, I; class 2 is D, D; and class 3 is I, Thus, the model tasked with identifying both subsequences to classify each sample. Ground-truth explanations are the and D sequences determining class labels.
SeqComb-UV.
SeqComb-MV. This dataset is multivariate version of SeqComb-UV The construction and class structure are equivalent, but the and D subsequences are distributed across different sensors in the input. Upon constructing the samples, the subsequences are chosen to be random sensors throughout the input. Ground-truth explanations are given as the predictive subsequences on their respective sensors, i.e., the explainer is required to identify the time points at which the causal signal occurs and the sensors upon which they occur.
Low Predictive signal is defined by regions of low variance time that occur in multivariate time series sample. Similar to CemComb datasets, we choose a random subsequence the input and, in that subsequence. replace the NARMA background sequence with Gaussian noise a low variance. The subsequence further discriminated by the mean of the Gaussian noise and the sensor on which the low variance sequence occurs. For class 0, the subsequence is at mean -1.5 on sensor O; for class 1, the subsequence is at mean 1.5 on sensor O; for class 2, the subsequence is at mean -1.5 on sensor 1; for class 3, the subsequence is at mean 1.5 on sensor 1. This task distinctly different from other synthetic datasets, requiring recognition of subsequence that is not anomalous from the rest of the sequence. This presents more challenging explanation task; simple change-point detection algorithm could not determine the explanation for this dataset.
We create 5,000 training samples, 1,000 testing samples, and 100 validation samples for each dataset. summary of the dimensions of each dataset can be found Table
Table 4: Synthetic Dataset Deseription
Dataset
# of Samples
Length
Dimension
Classes
Fhep
6,100
50
SeqComb-UV
6,100
200
4
SeqComb-MV
6,100
200
Low VarDetec
6,100
200
3

---

C.1.2 Real-world datasets
We employ four datasets from real-world time series classification tasks: PAM
human activity recognition: ECG [8T] ECG arrhythmia detection; Epilepsy 83 EEG seizure detection; and
82
Boiler 844 automatic fault detection.
PAM 82 measures the daily living activities of subjects with three inertial measurement units. We excluded the ninth subject due to the short length of sensor readouts, We segment the continuous signals into samples with a time window of 600 and the overlapping rate of 50%. PAM initially has 18 activities of daily life. We exclude the ones associated with fewer than 500 samples, leaving us with eight activities. After modification, the PAM dataset contains 5,333 segments (samples) of
sensory signals. Each sample is measured by 17 sensors and contains 600 continuous observations with sampling frequency of 100 Hz. PAM is labeled into eight classes, where each class represents an activity of daily living. PAM does not include static attributes, and the samples are approximately balanced across all eight classes.
MIT-BIH (ECG) [81]. The MIT-BIH dataset has ECG recordings from 47 subjects recorded at the sampling rate 360Hz. The raw dataset was then window-sliced into 92511 samples of 360 timestamps each. Two cardiologists have labeled each beat independently. Of the available annotations, we choose to use three for classification: normal reading (N), left bundle branch block beat (L.), and right bundle branch block beat (R). We choose these because I. and R diagnoses are known to rely on the QRS interval [S19, S20], which will then become our ground-trute explanation (see Section C.4). The Arrhythmia classification problem involves classifying each fragment of ECG recordings into different beat categories.
Epilepsy 83 The dataset contains single-channel EEG measurements from 500 subjects. For every subject, the brain activity was recorded for 23.6 seconds. The dataset was then divided and shuffled (to mitigate sample-subject association) into 1 1,500 samples of second each, sampled at 178 Hz. The raw dataset features five classification labels corresponding to different states of subjects or measurement locations eyes open, eyes closed, EEG measured in the healthy brain region, EEG measured in the tumor region, and whether the subject has a seizure episode. To emphasize the distinction between positive and negative samples, merge the first four classes into one, and each time series sample has a binary label indicating whether an individual is experiencing a seizure. There are 1,,500 EEG samples in total.
Boiler 84). This dataset consists of simulations of hot water heating boilers undergoing different mechanical faults. Various mechanical sensors are recorded over time to derive time series dataset. The learning task is to detect the mechanical fault of the blowdown valve of each boiler. The dataset is particularly challenging because it includes a large cimension-to-lengee ratio, unlike the other datasets, which contain many more time steps than sensors (Table
Table 5: World Dataset Description
Dataset of Samples
Length
Dimension
Classes
Task
PAM
5,333
600
17
8
Action recognition
MIT-BIH
92,511
360
1
5
ECG classification
Epilepsy 11,500
178
1
2
EEG classification
Boiler 160,719
36
20
2
Mechanical fault detection
C.2 Descriptions of baseline methods
We now describe each bascline method in further detail.
IG (31. Integrated gradients is classical attribution method that utilizes the gradients of the model to form an explanation. The method compares the gradients to a baseline value and performs Riemannian integration to derive the explanation. Integrated gradients is popular data-type agnostic interpretability method [S21], but it has no inductive biases specific for time series. We use the Captum [S22] implementation of this method, including default hyperparameters such the bascline value.
Dynamask (58]. This explainer is built specifically for time series and uses a perturbation- procedure to generate explanations. The method performs iterative occlusion of various input portions, learning a mask that deforms the input time series towards -eetrlllyddeerrmined baseline value.

---

This method is different from TIMEX in a few key ways. First, it performs continuous masking; TIMEX performs discrete masking through STEs. Second, it measures perturbation impact on the original model F(G(-)): TIMEX trains a surrogate model G to learn the explanations and measure the impact of masking the input. Third, Dynamask learns the explanations iteratively for each sample; TIMEX trains the surrogate, which can then output explanations in one forward pass of H
WinIT 86). This explainer is a feature removal explainer, similar to Dynamask. WinIT measures the impact of removing features from a time series on the final prediction value. It eliminates the impact of specific time intervals and learns feature dependencies across time steps. WinlT uses a generative model to perform in-distribution replacement of masked-out features. WinlT improves on a previous time series explainer, FIT [3), which is a popular baseline in time series explainability literature but is excluded in our work because WinlT is more recent and improves on FIT both conceptually and empirically.
CoRTX [87]. Contrastive real-time explainer (CoRTX) is an explainer method that utilizes contrastive learning to approximate SHAP 3 values. This method developed for computer vision, but we implement a custom version that works with time series encoders and explanation generators. We include this method because uses self-supervised learning to learn explanations. TIMEX also uses self-supervised objective to learn explanations, but our method differs from CoRTX in several ways. First, CoRTX performs augmentation-basse contrastive learning while we use MBC, which avoids the definition of negatives or the careful choice of augmentations specific to the data modality. Second, CoRTX fundamentally attempts to approximate SHAP values via a small number of SHAP explanations. In contrast, TIMEX includes masking system that can produce masks without fine- tuning a model on a set of explanations derived from an external method. CorRTX parallels ours in using self- self-supe learning but is fundamentally different from TIMEX.
SGT + Grad [17]. Saliency-guided training (SGT), an in-hoc explainer, is based on modification to the training procedure. During training. features with low gradients are masked to steer the model to focus on more important regions for the prediction. The method is not an explainer alone but requires another post-hoc explainer to derive explanations. In our experiments, we consider saliency explanations, which the SGT authors recommend. The authors found that this method can improve performance time series data. For this reason, we include it one of our baselines to demonstrate the effectiveness of TIMEX against modern in-hoc explainers.
C.3 Hyperparameter selection
We list hyperparameters for each experiment performed in this work. For the ground-truth attribution experiments (Section results R1), the hyperparameters are listed in Table The hyperparameters used for the occlusion experiment (Section results R2) with real-world datasets are in Table 7 We also list the architecture hyperparameters for the predictors trained each dataset in Tables 8
few abbreviations are used for hyperparameters that are not mentioned in the main text. "Weight decay" refers to an L1 regularization on the model weights; the value for weight decay is equivalent to the weight on that term in the loss function compared to the rest of the loss terms (Section (4.4). "Scheduler?" refers to using learning rate scheduler that decreases the learning rate by a factor of 10 if a plateau occurs. We use a scheduler that delays decreasing learning rates until after 20 epochs; not every experiment utilizes the scheduler as it is based on which choice yields lower validation loss upon convergence. "Distance norm." refers to a normalization of the distances in LMBC; the loss is divided by the variance of the distances on the Z embedding space. is the temperature parameter used for the Gumbel-Softmal reparameterization (73), Section dy refers to the dimensionality of hidden layers in the transformer predictor. Finally, "Norm. embedding" refers to architecture choice that normalizes Z when training the predictor; this is used to prevent a poor latent space when a collapse is observed via poor latent space geometry.
few other notes on implementation and design of TIMEX: The architecture of H uses the same size of E and encoder for _ as for the predictor on each task. The number of transformer decoder layers is fixed at 2. Please reference the codebase for more details on these hyperparameters and htwpaattiiiiittttttttot implementations https: h/gp:::://ttttt ggtbbbtthhb....... cwm/mims hawaaaaaaaaaaarrreeaaea
Choosing the r parameter. One of the most significant parameters in training TIMEX is r, the parameter which controls sparsity of the learned masks. We conduct an experiment where we vary the parameters and measure explanation quality. We use the SeqComb-UV dataset and hold all hyprraarameters constant while varying the parameter. The result is visualized in Figure 5 Low
5

---

AUPRO
0.9
AUP
0.8
0.7
0.4
0.5
Figure Experiment on SeqComb-UV dataset varying the parameter.
Parameter
FreqShape
SegComb-UV
SoyComb-MV
LowVarDetoct
ECG
Learning rate
0.001
0.001
0.001
0.003
0.0005
Batch size
64
64
64
Weight decny
0.001
0.001
0.001
0,0001
0.0001
Schoduler?
Yes
Yos
No
No
No
Epochs
100
100
0.5
0.5
0.5
Distance
No
No
Yes
No
ALc
1.0
1.0
1.0
1.0
1.0
2.0
Avon
2,0
2,0
2.0
2.0
2.0
50
50
50
50
50
Table 6: Training parameters for TIMEX across all ground-truth attribution experiments.
values lead to a drop in explainer performance with respect AUPRC and AUP. Importantly, for values above 0.4, the explainer performance stable, suggesting that robust to choice of value. Thus, recommend choosing value near 0.5 for experiments, the pattern is hypothesized to be sparse, one can set the value lower, and vice versa the pattern is hypothesized to be dense.
did throughout experiments.
C.4 Evaluation details
Following [58), we use AUP and AUR to evaluate the goodness of identification of salient attributes as binary classification task, which is defined C.
Definition C.1 (AUP.AUR
Let Q be matrix {0. Txxx whose elements indicate the true saliency of the inputs contained in Xxxxx By definition, Qt,i if the feature Ft, is salient
Parameter
Epilepsy
PAM
Boiler
Leuringg rate
0.0001
0.002
0.0001
Batch size
32
32
32
Weight decay
0.001
Scheduler?
Yes
No
Yes
Epochs
100
0.5
0.1
0.5
Distance norn.
Aue
1,0
1.0
1.0
A
2.0
2.0
Aue
0.0
2.0
1.0
1.0
1.0
TL
Table 7: Training parameters for TIMEX across all real-world datasets used for the occlusion experiments.

---

Parameter
FregShupe
SeqComb-UV
SeyComb-AY MV
LowVerDetect
BCG
Num. layers dy
2
1
64
128
32
64
Drepout
0.25
0.25
0.25
0.1
Norm, embedddigg
No
No
No
Yes
Yes
Learning
0.001
0.001
5e.4
0.001
2e.3
Weight decay
0.1
0.01
0.001
0.01
0.001
Epochs
200
1000
120
500
Table 8: Training parameters for transtormer predictors across all ground-trutt -truth attribution experiment datasets.
Purm.
Epilepsy
PAM
Boiler
Nom, layers
da
16
72
32
Dropout
0.1
0.25
0.25
Norm, embedding
No
No.
Yes
Learinng
0.0001
0.001
Weight decay
0.001
0.01
0.001
Epochs
300
100
500
Table 9: Training parameters for TIMEX across real-world datasets used for the occlusion experiments.
and otherwise. Let M be mask in {0, 1 obtained with saliency method. Let (0, 1) be the detection threshold for Me, to indicate that the feature Xt,5 is salient. This allows to convert the mask into an estimator (T) via:
(
Mf, 2T
else.
By considering the sets of truly salient indexes and the set of indexes selected by the saliency method:
((t, e [1 7 [1 dx] 4t,
À(7) = ((4,)) E
[1 dx]
(.,(T) = 1}
the precision and recall curves that map each threshold to a precision and recall score:
P: (0, - [0, 17:17 1An A(T)
IA(T))
IAn A(T)
R (0, 1) - 10,1 17
A
The AUP and AUR scores are the area under these curves:
AUP = 1. P(T)dr
AUR [( R(T)dr.
Groud-truth explanations for ECG datasets. We extract ground-truth explanations via QRS detection strategy following [85 because an initial of beat labels was produced by simple slope- sensitive QRS detector and were then given to two cardiologists, who worked on them independently. The cardiologists added additional beat labels where the detector missed beats, deleted false detections as necessary, and changed the labels for all abnormal beats. We employ Neurokit to extract QRS complexes and also take care to ensure that the QRS the proper explanation for each class. We consider two types of arrhythmias: left bundle branch block beat and right bundle branch block beat, to categorize our "abnormal" class. We perform the ground- ttuth evaluation on only the abnormal

---

Method
IoU
AUPRC
0.3750-10.0022
0.5760-10.0022
Dynamask TIMEX
0.2958: 0014
0.41
0.5214=000019
0.7124::0.0017
Table 10: IoU metric compared to AUPRC on the SeqComb-UT UV dataset.
No STR
STE
Method
AUPRC
AUP
AUR
AUPRC
AUP
AUR
FregShopes
0.6695-10.000
0.6398-10.0088 0.8723=2000006
0.545411..006
0.8324
0.7219 0.8326=0.000 0.5663-10.0025
0.6381=10.0022
SogComb-MV
0,5694-20.0023
0.3229=000017
0.6078=0.0021
0.3872=0.0015
BCG
0.4014:10.0019
0.5570-10.0032
0.156410..007
0..001
0..457110.0018
Table II: Ablation I: Ablation using the STE vs. no STE. "No STE" is equivalent to continuous
masking, as discussed in Section 4.2
class, as the normal class signifies negative information, which may harder to pinpoint based on model logic.
Statistical analysis. We evaluate each experiment on a 5-fold cross- cross-validation of each dataset. We then report average performance and standard error across all folds of evaluation for each experiment, which results the error bars seen in all tables throughout this work.
Evaluation with IoU metric. To further demonstrate the superior performance of TIMEX and the soundness of our evaluation setup, we additionally use an Intersection over Union (IoU) metric to measure the quality of explanations produced by baseline explainers compared to that of TIMEX when compared to ground- truth explanations. We calculate the loU score on the SeqComb-UI dataset, as shown in Table T0] For comparison to our metrics, we include the AUPRC results for the same methods. The IoU metric is highly correlated with the AUPRC metric, with each metric resulting in the same ranking of methods and TIMEX achieving the highest metric.
C.5 Visualization of explanations
Figure 6 shows an example of TIMEX explainer versus IG and Dynamask. Shown is the SeqComb- UV dataset, which has increasing and decreasing subsequences that determine the class label. Each explainer identifies the regions driving the prediction. IG identifies very sparse portions of the predictive region, choosing only one point out of the sequences for the explanation; this is not reasonable when scaling to large and noisier datasets where the signal might not be as clear. Dynamask seems to miss some important subsequences, identifying one or two subsequences. identifies a larger portion of the important subsequences due to the connection loss in Equation 2 This property becomes crucial when scaling to time series datasets with more noise as it becomes more difficult to intuitively deduce the causal signal through visual inspection.
contrast, TIMEX
C.6 Further ablation experiments
We present a more in-depth study of ablations TIMEX on three datasets: FreqShapes (univariate), SeqComb-MV (multivariate)) and ECG (real-world). This an extension to the ablations on the ECG dataset in Section 6, R1 in Table 3
Ablation No STE. We now conduct an experiment examining the effectiveness of using the STE for training TIMEX. Table [ shows the results of this ablation experiment. Using the STE provides over 17% increase in AUPRC for attribution identification for every dataset. Furthermore, AUR better when using an STE for every dataset, but the AUP is better for SeqComb-MV without the STE than with the STE. Using the STE also shows benefits both the univariate (FreqShapes,, ECG) and multivariate (SeqComb-MV) settings. In conclusion, the STE provides a noticeable benefit over continuous masking approach, giving empirical evidence for the claims made in Section 4.2
Ablation 2: SimCLR vs. MBC. We now test classical SimCLR [S5] contrastive learning loss against our proposed model behavior consistency (MBC). The SimCLR objective is designed to decrease the distance between explanation embeddings and embeddings in the reference model's latent space. We do not perform data augmentations as in the original SimCLR work. The SimCLR loss that we use is given as:
8

---

Ground Truth
Are
www
Nu uw
-13
-18
160
150
390
196
100
2.8
Integrated Gradients
Anm
unraa mart
sm whw
M
1.a
-18
20-
Dynamask
10-
M
Whwa
36
Auy uhv m
sma Lumm
60-
39
15-
TimeX
M
30
05
Au
man mmm Mum
37
-04-
2000
159
2.00
350
189
250
Figure 6: Visualization of explanations SeqComb-UV dataset. Each column corresponds to unique sample. All are of Class 3, which consists of one increasing subsequence and one decreasing subsequence. The methods that generate each figure shown for each of the rows, while ground truth explanations are provided the top row.
LSiMCLR R (7, EB)
1
Σ
- llg exp(D(es 2f)) Eje, expl D (Z,, 2)
N
2.EZ,=EEZR
For each SimCLR trial, we fixed the number of sampled negatives at 32 and kept all other parameters equal. In addition, early stopping strategy was performed where the stopping value was based on cosine similarity between explanation embeddings and reference sample embeddings (higher similarity is better).
SimCLR loss provides valuable objective training TIMEX relative to baseline explainers, but MBC optimization produces more robust explanations. SimCLR delivers a slightly better AUPRC for FCG, but its AUPRC values are below that of MBC for FreqShapes and SeqComb-MV. SimCLR loss yields explanations with consistently lower AUP: AUP is closest for SeqComb-MV with only

---

SimCLR
MBC
Method
AUPRCC
AUPRC
FregShapes
0,7014:10000 0046
0 0.5991 5915
0.5915:200007
0.8324 0034
0.7219 0031
0.6381=000002 0.3672=00.015
ScgComb-MV
0.6645=00.000
08148==000009
0.3717=0.0017 0.4779:000003
0.6878=0.0021 0.4721 0.0008
0.8326= 0008
BOG
0.0002
0.009511000024
0.5663:10.00
0.4457=000018
Table 12: Ablation 2: Ablation considering SimCLR objective for training TIMEX versus an MBC objective as outlined in the main text.
Dalaset
Ablation
AUPRC
AUP
AUR
MBC only
0.2316=20.0000
0.1533==0.0015
0,4763==0.0022
FreqShapes
LC only
0,2629:#0.0022
0,1850=:0.000
0.5893=00000
MBC
0.8324:0.0034
0.7219=:00001
0.6381=0.0022
MBC only
0.0000++00008
0.0576:4:0
0,4996+000001
SegComb-MV
LC only
0.0788==0.0009
0.0570=+0.0006
0.5294=0.0034
MBC LC
0.6078:10.0021
0.8326:1
0.3872::000015
MBC
0.16151100006
0,1348
0.5004_0..001
oCyyl
0.37044:00018
0.3296::0.0019
0,508440.0000
MBC LC
0..0211.0.018
0.566310.008
0.44571.0.001
Table 13: Ablation 3: Effects of model behavior consistency (MBC) and label consistency (LC)
losses on explanation performance.
a 3.4% drop from MBC, but is at a 17.0% decline for FreqShapes and a 13.6% drop for ECG. is important note that in addition to increased performance, MBC loss is more computationally efficient than SimCLR loss, avoiding inference on negative samples.
Ablation 3: Effect of MBC and LC losses. We now examine the effectiveness of using both model behavior consistency (Eq. 3 (MBC) and label consistency (Eq. 4 (LC) losses. Table 13 shows that using LC and MBC in combination is always better than using either alone. In isolation, LC performs better than MBC, which is expected given its (obviously) higher correlation with the classification predictions than MBC, which relies powerful explainer that achieves over 27.5% higher AUPRC than MBC or LC alone. MBC and LC work together capture rich information about the model's behavior, allowing TIMEX to be state-of-the-an explainer.
earlier layer's 's embedding space. Using both losses results
To justify these results, we recall an argument presented in Section 4.3 where we justify MBC. We remark that perturbation-based methods have a similar idea to TIMEX: find some sparse perturbation to the input that can preserve the output state of the model. This is often done by observing the predicted label from the model under an applied mask or perturbation, c.g., in one of our basclines, Dynamask. A perturbation that preserves the output state is said to be "faithful" to the model because it is assumed that the model is invariant to the perturbation. a sense, MBC generalizes this idea to latent spaces, ensuring that invariances preserved in the latent space of the model well as the prediction space.
Beyond the introduction of MBC alone, another core contribution of our work focuses optimizing faithfulness to predictor models on multiple levels. We use multiple hidden or output states of the model, e.g.. a latent and logit space, on which the explainable surrogate should match the reference predictor. The hypothesis behind this innovation is that model behavior (the exact objective we are trying to explain) cannot be fully captured by one state, e.g., a predicted label, but rather by multiple states throughout the model. similar assumption is made in knowledge distillation, where methods often optimize student models to match the teacher in various network layers. Therefore, MBC and LA together enforce adherence to model behavior on two fronts: the latent space and prediction space, respectively. This explains the observed behavior: MBC and LA perform poorly alone, but together, these two losses provide a powerful objective to train the model.
C.7 Implementation and computing resources
Implementation. We implemented all methods this study using Python 3.8+ and PyTorch 2.0. In our experiments, we employed the Vanilla Transformer [88] as the classification model for all methods. We verified that the classification models achieved satisfactory performance on the testing set to ensure that the models are strong predictors, which previously pointed out by Faber et al.
10

---

[S18] as necessary for explainability evaluation. Complete classification results are in Table [5] We followed the hyperparameters recommended by the respective authors for all baseline methods. Computational resources. For computational resources, we use a GPU cluster with various GPUs, ranging from 32GB Tesla V100s GPU 4SGB RTX8000 GPU. TIMEX and all models were only trained on a single GPU at any given time. The average experiment runtime in this work was around minutes per fold, with ECG taking the longest at approximately 13 minutes per fold when training TIMEX to convergence.
Epilepsy
Training
Inference
Training
Inference
Dynamask
N/A
102
NIA
358
WinTT
1810
5730
93.3
84.7
TIMEX
1.04
1.07
Table 14:
Runtime results for Dynamask, WinlT, and TIMEX PAM and Epilepsy datasets. Time is shown in seconds. Note that Dynamask requires no training, thus a "N/A" designation.
Runtime experiment. We conducted runtime experiment to understand how TIMEX compares to baseline explainers. Table X shows the training and inference time in seconds of TIMEX versus two state-of-the-art time series-specific baselines, Dynamask and WinlT. We chose two real-world time series datasets, PAM and Epilepsy, which are of varying sizes. PAM contains 4266 training samples and 534 testing samples, each of 600 time steps in length. Epilepsy contains 8280 training samples and 2300 testing samples, each of 178 time steps in length. Table [4 shows the time in seconds needed to train each explainer and to perform inference on the testing set.
TIMEX is the most efficient model inference time for both datasets. This result is expected, as Dynamask and WinIT both require iterative procedures for each sample at inference time, while TIMEX requires only a forward pass at inference. Combining training and inference time, TIMEX is the second-fastes on both datasets. However, WinlT and Dynamask times vary greatly between each dataset, with Dynamask being the fastest on PAM and WinIT being the fastest Epilepsy. WinIT scales poorly to samples with many time steps, while Dynamask scales poorly to large testing sets. TIMEX strikes compromise between these extremes, scaling better than Dynamask to larger testing sets while scaling better than WinIT to longer time series.
C.8 Flexible use of TIMEX with different time series architectures
We now study the ability of TIMEX to work with different underlying time series architectures. This means that of the original architecture, G and G are now alternative architecture, while = remains as described in Section 4.1 Since experiments in the main text are based on transformer architectures, we now use convolutional neural network (CNN) and long-short term memory (LSTM) network as the underlying predictors with the following hyperparameters:
LSTM: 3 layer bidirectional LSTM MLP on mean of last hidden states
CNN: 3 layer CNN MLP meanpool
Tables 19 show the results of TIMEX against strong baselines with a CNN predictor. TIMEX retains the state-of-the-art prediction observed for the transformer-base architecture, achieving
Dataset
F1
AUPRC
AUROC
PreqShapes
0.9716:10.0034
0.9940110.0008
0.9990-10.0003
SeqComb-UTV UV
0.941511..052
0.9798.10.0028
1..0011
Som
0.0065
0.90712000005
0.990011.0001
Low
0.9748:10.0056
0.9967::0.0013
0.9988+0.0005
Boiler
0.8045:20.0089
0.8344::0.0071
1.005590000199
ECG
0.9154.1..0134
0.9341 19.0016
0.9587 10.01
Fpilepsy
0.9201 1:0.0099
0.924611000130 30
1000117
PAM
0.8845:00.051
0.9251 40.0029
0.9786+0.0009
Table 15: Classification (i.e., predictive) performance achieved by transformer time series models on datasets used in this study. These models considered time series predictors throughout experiments in this study.
11

---

PreqShupes AUIP
SeqCommbMMi MV
Method
AUPRC
AUP
IG
0.9282=0.0016
0.7775=0.0010
0.6926=0.0017
0.23694: 00200
0.5150:20.0048
0.3001
Dyuamask
0.2290=0.0012
0.3422=0.0057
0.5170=0.0013
0.2836+=0.000
0.6309=000044
0.0816=0.0015
WinIT
0.0016
0.5106=0.00226
0.3909=000001
0.3515
0.6047
0.3423+0.0021
Ours
0.007210.00002
0.001310.0001
0.1298 0.0001
0.1307
0..001
Table 16: Explainer with LSTM predictor on FreqShapes and SeqComb-MV synthetic datasets.
Method
AUPRC
AUP
AUR
0.5057=000018
0.6129=0.0026
0.4026=0.0015
Dynamask WinIT
0.3030:00.0012
0.0299-0.0030
0.1102
0.3628::0.0013
0.3805-0.0022
0.4055=000009
Ours
0.6057:00.018
0.6416=0.0024
0,4436=0.0017
Table 17: Explainer results with LSTM predictor on ECG dataset.
the best AUPRC on SeqComb-MV and ECG datasets. However, the performance for FreqShapes saturates at very high values for both TIMEX and IG, making the comparison more difficult for AUPRC. Tables 17 show the results of TIMEX against strong baselines with an LSTM predictor. TIMEX performs very well for both FreqShapes and ECG datasets, achieving the highest AUPRC, AUP, and AUR for both datasets. For SeqComb-MV, TIMEX did not converge. However, no explainer performed well for this task, achieving lower results than for the transformer and CNN predictors.
Visualization under different architectures. We add a visualization on the FreqShapes dataset different underlying architectures. Figure 7 shows this visualization.
with various explainers Explanations are similar across models. IG outputs appear similar, but TIMEX has a higher recall for essential patterns, which is reflected in quantitative results.
C.9 TIMEX diverse tasks and datasets
We experiment with TIMEX on various tasks and types of datasets to demonstrate the generality of the method.
Irregular time series. We experiment with TIMEX on irregular time series classification. We introduce irregularly sampled version of SeqComb-MV, randomly dropping average of 15% of time steps. This results in variable-length time series, addressing both W1 and W2. We then follow Zhang et al. [S23], using time series transformer with irregular attention mask. We train only with direct-value masking to avoid direct interference with this mechanism.
Table 20 shows the results of this experiment. We compare TIMEX to Integrated Gradients (IG) performance because, given the nuance of learning from irregularly-sampled datasets [S23], most
ProqShapes AUP
SogComb-MV AUP
Method
AUPRC
AUR
AUPRC
AUR
IG
0.0055=
0.875410.0000
0.7240:10.0015
0.5077
0.8858 10.0014
0.2294110.0013
Dynamask
0.2574-100000
0.442210.0032
0.5257-10.0015
0.4550-10.0016
10.0005
0.0001
WinlT
0.5321
0.6020=0.0025
0.3966=000017
0.5334+.00011
0.8324+0.0020
0.2259=40.0020
Ours
0.9041=0.0002
0.6915=0.0010
0.8522=000009
0,70162:00019
0.7670=0000022
0.0009
Table 18: Explainer results with CNN predictor on ShapSsapes and Cem synthetic datasets.
ECG
Method
AUPRC
AUP
0.5374-0.0012 0.4049-10.0010
0.3306=000010
Dynamask
0.4598=000000
0.7216=0.0027
0.1014=0.0008
WinIT
0.3963=000011
0.3292=000020
0.3518=0.0012
Ours
0.7044::000014
0.8706:0.0012
0.3972=0000010
Table 19: Explainer results with CNN predictor on ECG dataset.
12

---

Transformer
CNN
LSTM
Dynamask
Figure 7: Visualization of FreqShapes dataset of TIMEX explainer (top row), integrated gradients (IG), and Dynamask explainers on the FreqShapes dataset with transformer, CNN, and LSTM architectures.
AUPRC
AUP
AUR
IG
0,286=000044
0.617=20.009 0.606.10.010
0.372-:0.009
TIMEX
0.404.10.007
0.196.10.003
Table 20: Performance TTMEX and on modified version the SeqComb-MV dataset such that the samples are irregularly sampled.
baselines do not apply to prrggularly-samplee time series datasets without significant changes that out-of-scope for our work. TIMEX demonstrates superior performance this setting, outperforming IG by an over 1.5x improvement in AUPRC.
Input
-0.75
-1.10
-1.25
1.15
1.20
100
800
300
400
500
700
720
725
730
736
740
-1.10
-1.25
MuN M4
-1.50
1.16
-2.00
100
800
300
400
500
500
700
720
725
730
736
740
Time
Prediction
Figure 8: Forecasting experiment the ETThl dataset with TIMEX explanations.
Forecasting, We demonstrate TIMEX 's generalizability to diverse tasks by explaining forecasting model. We use the ETTh1 dataset [S24] and vanilla forecasting transformer. To modify TIMEX for forecasting, we first extract embeddings used for MBC by max pooling over hidden states of the decoder. Next, we used revised LC loss that used MSE as the distance function between predictions rather than JS divergence. We show visualizations of two samples in Figure Explanations are in left column, while forecasted time steps are in the right column. few patterns emerge:
TIMEX identifies late time steps as crucial for the forecast, an expected result for a dataset with small temporal variance.
13

---

2. (a): the forecast is an increasing sequence, and TIMEX identifies a region of an expanding sequence (time 450-485). This suggests the model uses this increasing sequence reference for forecasting.
3. (b): a sharp upward spike is forecasted at the beginning of the predicted window. Corre- spondingly, TIMEX identifies local minimum around time 260 and sharp upward spike around time 475-490.
This preliminary experiment demonstrates that TIMEX shows the potential to extract meaningful explanations from forecasting datasets.
C.10 Quantitative analysis of landmark explanations
1.2
Random Explanations
Top 10 Landmarks
Relative Counts
0.8
02
0.0
1.5
2.0
3.0
3.5
4.0
Average DTW
Figure 9: Distributions of DTW distance between structures of masked-in portions of explanations.
To examine landmark quality the ECG dataset, compare two groups: 1) the landmark group, with the five ncarest-neighbon around the top 10 filtered landmarks, and 2) the random group, with five random explanations. We then compare the structure of the most salient values in the samples. We mask in the top-10 time steps as identified by TIMEX, then compute DTW [S25] distance between samples in each group. We then plot the distribution of average within- group DTW distances in Figure A lower average DTW within their groups demonstrates the high quality of learned landmarks.
Clustering method
Landmark
0.0014
0.152=0.026
K-means
0.007
0.025=0.001
Random
0.109.0..00
0.027=00042
Table 21: Quantitative experiment comparing learned landmark explanations to K-means and Random clusters in the explanation latent space.
We have additional experiment comparing the clustering performance of our landmarks to those learned by k-means on the final latent space. On the ECG dataset, we match embeddings to their nearest landmarks, their nearest k-means cluster centroids, and randomly-selected points as a baseline. For each method, we use 50 clusters. We then evaluate each clustering method's Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI), which are standard clustering metrics, against the ground ground-truth labels and report the standard error (+/-) computed over 5-fold cross-validattin. Higher metrics for one set of centroids would indicate that proximity more meaningful for prediction. The results are shown in Table 21 Here, see higher NMI and ARI for landmarks, which means the landmarks are better heuristic for forming clusters than the K-means centroids for this task.
14

---

Appendix References
[SI] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv: 1903, 0903,0562,, 2019.
[S2] Pengyu Cheng, Chang Liu, Chunyuan Li, Dinghan Shen, Ricardo Henao, and Lawrence Carin. Straight-through estimator as projected wasserstein gradient flow. arXiv preprint arXiv:991000217,, 2019.
[S3] Veenu Rani, Syed Tufael Nabi, Munish Kumar, Ajay Mittal, and Krishan Kumar. Self- supervised learning: A succinct review. Archives of Computational Methods in Engineering, pages 1-15, 2023.
[S4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[S5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[S6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[S7] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5022-5030, 2019.
[S8] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1801-1810, 2019.
[S9] Vipin Pillai, Soroush Abbasi Koohpayegani, Ashley Ouligian, Dennis Fong, and Hamed Pirsiavash. Consistent explanations by contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10213-10222, 2022.
[S10] Tao Han, Wei-Wei Tu, and Yu-Feng Li. Explanation consistency training: Facilitating consistency-based semi-supervised learning with interpretability. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7639-7646, 2021.
[S11] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 729-739, 2019.
[S12] Matthew Watson, Bashar Awwad Shickh Hasan, and Noura Al Moubayed. Using model explanations to guide deep learning models towards consistent explanations for ehr data. Scientific Reports, 12(1):19899, 2022.
[S13] Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithffulesss of local explanations. In International Conference on Machine Learning, pages 4794 4815. PMLR, 2022.
[S14] Huanlai Xing, Zhiwen Xiao, Dawei Zhan, Shouxi Luo, Penglin Dai, and Ke Li. Selfmatch: Robust semisupervised time-series classification with self-distillation. International Journal of Intelligent Systems, 37(11):8583 8610, 2022.
[S15] Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Neural Information Processing Systems, 35:635-649, 2022.
[S16] Defang Chen, Jian-Ping Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 1 1993--19944 2022.
[S17] Robert Geirhos, Jöm--Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning deep neural networks. Nature Machine Intelligence, 2(11):655-63, Nov 2020.
15

---

[S18] Lukas Faber, Amin K. Moghaddam, and Roger Wattenhofer. When comparing to ground truth is wrong: On evaluating gnn explanation methods. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 332-341, 2021.
[S19] Borys Surawicz, Rory Childers, Barbara Deal, and Leonard S Gettes. Aha/accf/hrs rec- ommendations for the standardization and interpretation of the electmoardiogram: part iii: intraventricular conduction disturbances: scientific statement from the american heart asso- ciation electrocardiography and arrhythmias committee, council clinical cardiology; the american college of cardiology foundation; and the heart rbythm society: endorsed by the international society for computerized electrocardiology. Circulation, 119(10):e235 e2400 2009.
[S20] Mariana Floria, Alexandra Noela Parteni, Ioana Alexandra Neagu, Radu Andy Sascau, Cristian Statescu, and Daniela Maria Tanase. Incomplete right bundle branch block: Challenges in electrocardiogram diagnosis. Anatolian Journal of Cardiology, 25(6):380, 2021.
[S21] Chirag Agarwal, Eshika Saxena, Satyapriya Krishma, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitmik, and Himabindu Lakkaraju. Openxai: Towards transparent evaluation model explanations. arXiv preprint arXiv:2206. 1104, 2022.
[S22] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020.
[S23] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, ICLR, 2022.
[S24] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AÅAI conference artificial intelligence, volume 35, pages 11 11106-11115 2021.
[S25] Mcinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69-84, 2007.
16


Reviews:
[{"review_id": "tWT9kL7KLM", "reviewer": "Reviewer_Uw18", "summary": "The paper introduces a novel time series interpretability model called TIMEX. The challenge of interpreting time series models arises from the need to identify both the specific time series signals influencing predictions and their alignment with interpretable temporal patterns. TIMEX addresses the issue of model faithfulness by introducing model behavior consistency, ensuring relations in the latent space induced by the pretrained model are preserved when compared to relations in the TIMEX-induced latent space.\n\nTIMEX provides discrete attribution maps, giving interpretable explanations for the model's predictions. Unlike existing methods, TIMEX goes further by learning a latent space of explanations, enabling visual aggregation of similar explanations and recognition of temporal patterns.\n\nThe evaluation of TIMEX on 4 synthetic and 4 real-world datasets, which includes case studies involving physiological time series, demonstrates its superior performance compared to state-of-the-art interpretability methods. TIMEX's innovative components hold promise for training interpretable models that capture the behavior of pretrained time series models.", "questions": "1. In appendix, some of the explanations of the SeqComb-UV is visualized. The repeating patterns can all be seen to be happening around the same time points. Is TimeX capable of identifying patterns that are not necessarily aligned in absolute time between two patients? \n\n2. Paper would be improved if there were additional experiments on different choices of $r$ presented, as at its current state, a reader unfamiliar with [46] will have a hard time deciphering the effect of different values of $r$.\n\n3. Why is cosine similarity used in Equation 3? For SimCLR this would have made sense as the embedding of SimCLR is on the unit hyper-sphere and thus cosine similarity is an appropriate choice. But since the embeddings here are not constrained to be on the hyper-sphere I am uncertain why encoding through the notion of directionality would be helpful here.\n\n4. Wouldn’t it be more appropriate to normalize $L_{MBC}$?\n\n5. How does the differentiable attention masking presented in the appendix come into play for the model? Also this should be referring to section 4.2 instead of 4.1 as currently printed in the appendix.\n\n6. Following the methods section, I am still uncertain as to why TIMEX is considered to be an in-hoc model and not a surrogate method. The paper would become easier to follow, if the authors can draw a parallel here and clear this uncertainty in the introduction.\n\n7. The appendix currently provides an extensive list of experiments. One additional point to improve the already thorough analysis would be to visualize and see how much the explanations found differ for different underlying models tested (transformer, LSTM, CNN).\n\n8. In the appendix the running time of the model is presented, but how does it compare to other baselines in your experiments?\n\n\nMinor comments:\n1. In section 3.1, $F^E$ should be defined as otherwise $F(z_i^E)$ would imply that $z_i^E$ and $z_i$ belong to the same space.\n2. Line 213 has a typo: we [use] a direct-value masking procedure. (\"use\" is missing).\n3. Specifics of how the transformer encoder-decoder was used should be added to appendix for reproducibility of results.", "limitations": "The paper does not contain any discussion on the potential societal impact of their work. As interpretability opens a broad avenue for discussion about this topic, I believe this section is essential to be added to the paper.\n\nAdditionally, the current limitations section is quite coarse. The authors can improve the paper by better expanding on the limitations of TimeX at its current state and the interesting directions for the future works.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall the proposed method is novel, effective, and well-evaluated. The main strengths of the paper are as follows:\n\n1. The paper proposes a novel method for explaining time series models that is based on self-supervised learning. This approach has several advantages over traditional explanation methods, such as being able to learn explanations for models with complex temporal dynamics. The paper provides a clear and concise overview of the problem of explaining time series models and provides a detailed description of the TimeX method, including the rationale behind the design choices.\n\n2. The authors presents a comprehensive evaluation of TimeX on a variety of experiments with time series models and datasets. The results show that TimeX is able to provide interpretable explanations that are both faithful to the model's predictions and informative about the underlying temporal patterns.\n\n3. The paper also includes a number of case studies that demonstrate the use of TimeX to explain the predictions of time series models in real-world applications. These experiments help highlighting TimeX's capability to be used for gaining insights into the behavior of time series models and to identify potential problems with the models.\n\n4. The authors evaluate TimeX through an extensive set of ablations and on a variety of time series models. TimeX shows to be effective in explaining a variety of time series models, including LSTM, CNN, and vanilla-Transformer model. This suggests that TimeX is a general-purpose method for explaining time series models.", "weaknesses": "The are a few weaknesses that if addressed or identified as a limitation for future can improve the paper:\n\n1. As the current setup is presented, the model does not handle variable length time-series which is a common occurrence in medical time-series data. This could prove to be challenging to be extended from the current state of TimeX as aligning different time-series that have variable length would also become important.\n\n2. Another common challenge of dealing with time-series data that is not addressed in the paper, is handling irregular time-step intervals. A discussion of this important limitation is missing in the paper. Irregularly sampled time-series is predominantly present in healthcare data that this paper focuses on and is important weakness of the TimeX that is not mentioned in the paper.\n\n3. The method has not been evaluated on a wide range of different tasks. The authors of the paper evaluated TimeX on a variety of time series models and datasets, but they did not evaluate TimeX on a wide range of tasks. This means that it is not clear how well TimeX would perform on other tasks, such as anomaly detection or forecasting.\n\n4. TimeX currently does not handle temporal prediction (i.e. a prediction task where prediction is done at every time-step), and it is unclear how it can be expanded to provide explanations for models designed for such tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your extensive comments, critiques, and praises for our work. We hope our response and experiments further convince you of TimeX’s novelty and effectiveness, and we kindly ask you to consider raising your score. Please reach out if there are additional questions.\n### W1 and W2: Irregular time series\nWe introduce an irregularly sampled version of SeqComb-MV, randomly dropping an average of 15% of time steps. This results in variable-length time series, addressing both W1 and W2. We then follow Zhang et al. [1] using a time series transformer with an irregular attention mask. To avoid direct interference with this mechanism, we train TimeX only with direct-value masking.\n\n**Table R2** shows the results of this experiment. We compare TimeX to Integrated Gradients (IG) performance because given the nuance of learning from irregularly-sampled datasets [1], most baselines do not apply to irregularly-sampled time series datasets without significant changes that are out-of-scope for our work. TimeX demonstrates superior performance in this setting, outperforming IG by an over 1.5x improvement in AUPRC.\n### W3 and W4: TimeX on a wide range of tasks\nWe agree that TimeX can in principle be applied to a wide range of tasks. We focus on classification for close comparison to evaluations in prior works. Still, to address your concerns, we demonstrate TimeX’s generalizability to diverse tasks by explaining a forecasting model. We use the ETTh1 dataset [2] and a vanilla forecasting transformer. To modify TimeX for forecasting, we first extract embeddings used for MBC by max pooling over hidden states of the decoder. Next, we used a revised LA loss that used MSE as the distance function between predictions rather than JS divergence. \n\nWe show visualizations of two samples in **Figure R1**. Explanations are in the left column while forecasted time steps are in the right column. A few patterns emerge:\n1. TimeX identifies late time steps as important for the forecast, an expected result for a dataset with small temporal variance.\n2. **(a)**: the forecast is an increasing sequence, and TimeX identifies a region of an increasing sequence (time ~450-485). This suggests the model is using this increasing sequence as a reference for forecasting.\n3. **(b)**: a sharp upward spike is forecasted at the beginning of the predicted window. Correspondingly, TimeX identifies a local minimum around time 260 and a sharp upward spike around time 475-490.\n\nTimeX extracts meaningful explanations from forecasting datasets. Regarding anomaly detection and temporal prediction, TimeX could be transferred by similarly modifying the training procedure. Given the nuances of these individual tasks, we leave this for future work.\n### Q1: SeqComb-UV visualization\nThe randomly-chosen samples for SeqComb-UV visualization have ground-truth explanations that fall in similar time points, but this is not true in general across our synthetic datasets. Each synthetic dataset contains different temporal dynamics that vary across time, and because of TimeX’s consistently-high performance, we are confident in its capability to identify important patterns in a wide variety of time series settings.\n### Q2: Varying r parameter\nWe conduct an experiment where we vary the $r$ parameter and measure explanation quality. We use the SeqComb-UV dataset and hold all hyperparameters constant while varying the $r$ parameter. The result is visualized in **Figure R3**. Low $r$ values lead to a drop in explainer performance with respect to AUPRC and AUP. Importantly, for $r$ values above 0.4, the explainer performance is stable, suggesting that TimeX is robust to choice of $r$ value.\n### Q3: Cosine similarity\nIn principle, Equation 3 can accept any distance function, but we choose cosine similarity because it is used in popular approaches such as SimCLR and InfoNCE [3,4].\n### Q4: Normalizing $L_{MBC}$\nWe do indeed normalize $L_{MBC}$ during training. We have made an appropriate amendment to Section 4.3 to fix this error.\n### Q5: Differentiable attention masking\nWe use the differentiable masking approach presented in Nguyen et al. [5] to ensure that we can learn the masks in TimeX in an end-to-end differentiable manner.\n### Q6: In-hoc vs. surrogate method\nThank you for pointing out this error. We have adopted the term “surrogate” to refer to our method. Please reference Point 2 in the general response.\n### Q7: Explanations from different models\nWe have generated **Figure R2** to show the explanations across different models on the FreqShapes dataset. Explanations are similar across models. TimeX and IG outputs appear similar, but TimeX has higher recall for important patterns, which is reflected in quantitative results.\n### Q8: Runtime\nPlease see general response Point 3.\n### Minor Comments\nThank you for pointing out several typos and errors in our submission. We have made these edits within the text.\n### Limitations:\nIn response to your comment, we will provide a more detailed limitations section and a societal impacts section. Our societal impacts section will read: “Time series data spans domains like finance, healthcare, energy, and transportation. Enhanced interpretability can bolster decision-making and trust. While it aids in identifying systemic biases for fairer, inclusive systems, caution is paramount. Misinterpretations or over-reliance on automated insights can backfire, highlighting the need for human and algorithmic collaboration.”\n\n**References**: [1] Zhang et al., “Graph-guided network for irregularly sampled multivariate time series”, ICLR 2022. [2] Zhou et al, “Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting”, AAAI 2021\n[3] Chen et al., “A Simple Framework for Contrastive Learning of Visual Representations”, ICML 2020.\n[4] van den Oord et al., “Representation Learning with Contrastive Predictive Coding”, arXiv 2018.\n[5] Nguyen et al., “Differentiable window for dynamic local attention”, ACL 2020."}, {"review_id": "qBqTa0F91w", "reviewer": "Reviewer_Mf92", "summary": "TimeX proposes a explanation module that is trained along-side the model to provide more consistent explainiations. This is done by making the internal embeddings of explaination modules consistent with that of full model such as distance consistency and label consistency.", "questions": "See weaknesses", "limitations": "The paper presents a model behaviour consistency based method of generate explainations for time-series models. While the results are very significant, computational requirements relative to other post-hoc methods and the performance vs compute tradeoff is not clear.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The methodology is well motivated and presented.\n2. The consistency losses encourage expaliation module features to learn from latent space of the model.\n3. The synthetic experiments are well setup and model provides significant performance improvements in synthetic and real-world dataset.", "weaknesses": "1. The novelty of the method seems unclear. Model consistency has been previously explored in neural explainations and this work dovtails to time-series models.\n2.What is the additional computational complexity or resources required to train the explainer modules? How does it compare to other methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and criticism about our work. We greatly appreciate your claims that our work is “well motivated and presented” and that our explainer provides “significant performance improvements” over baseline explainers. We address concerns about novelty and computational efficiency in this response. If you feel our comments address your concerns, we kindly ask that you raise your score. Please reach out during the author-reviewer discussion if any questions are left unanswered.\n\n### W1: Novelty\nThank you for this comment. Our method is novel in its formulation, specifically one of our central contributions, model behavior consistency (MBC), is a novel formulation for training surrogate explainers for neural networks in time series. MBC states that the surrogate model should match the original time series model through mimicking its latent space and predictions. This is a novel extension of previous perturbation-based methods which seek to find some perturbation on the original input that preserves the prediction of the model. Instead of only preserving the predictions of the original model, TimeX also seeks to preserve the structure of the original model’s latent space, which increases faithfulness to original model behavior. See Section 4.3 for a more formal justification of MBC. While consistency has been explored in previous XAI works, we are the first to consider consistency between the predictor model and the explainable surrogate model. To draw this distinction, we contrast our method to two notions of consistency: 1) **consistency between explanations** and 2) **consistency as an explainability metric**.\n\n**Consistency between explanations**: This notion has been introduced in previous works in explainability literature. Pillai et al. [1] train a saliency explainer via contrastive learning that preserves consistency across the saliency maps for augmented versions of images. A few other works have explored maintaining consistency of explanations across various perturbations and augmentations, specifically in computer vision [2,3]. In one of the only previous works to consider explanation consistency in time series, Watson et al. [4] train an explainer on an ensemble of classifiers to optimize the consistency of explanations generated by an explainer applied to each individual classifier. TimeX does not seek to optimize simply consistency between explanations but rather consistency to the predictor model on which it is explaining.\n \n**Consistency as an explainability metric**: Dasgupta et al. [5] defines explanation consistency as similar explanations corresponding to similar predictions; this metric is then used as a proxy to faithfulness to evaluate the quality of explainers. However, Dasgupta et al. uses the notion of consistency to evaluate explainers, not to train and design a new explainer method. TimeX uses consistency as a learning objective rather than simply a metric.\n\nOur work differs from these previous formulations of explanation consistency. We seek to optimize the consistency not between explanations directly, as mentioned in previous works, but rather between the explainer and the model it is tasked with explaining. MBC attempts to ensure that the behavior of the explainable surrogate matches that of the original model. The definition of consistency in Dasgupta et al. is the closest to our definition of MBC; however, Dasgupta et al. does not seek to optimize the consistency of explainers but rather to evaluate the output of post-hoc explainers. TimeX directly optimizes the consistency between the surrogate model and the original predictor through the MBC loss, a novel formulation that seeks to increase the faithfulness of explanations generated by TimeX. Thus, we respectfully disagree with the sentence in your summary, which states that TimeX is trained to “provide more consistent explainiations [*sic*].” We have added these citations and arguments to the Appendix to make our distinction from previous work clearer. \n\n### W2: Computational resources\nPlease reference the general response Point 3 for discussion on this point. We sincerely thank you for requesting this analysis as this enriches the evaluation for TimeX.\n\n**References**:\n[1] Pillai et al., “Consistent Explanations by Contrastive Learning”, CVPR 2021.\n[2] Han et al., “Explanation Consistency Training: Facilitating Consistency-Based Semi-Supervised Learning with Interpretability”, AAAI 2021.\n[3] Guo et al., “Visual attention consistency under image transforms for multi-label image classification”, CVPR 2019.\n[4] Watson et al., “Using model explanations to guide deep learning models towards consistent explanations for EHR data”, Scientific Reports 2022.\n[5] Dasgupta et al., “Framework for Evaluating Faithfulness of Local Explanations”, ICML 2022."}, {"review_id": "7WqqWNPMDx", "reviewer": "Reviewer_7JHq", "summary": "This paper presents an in-hoc interpretability mechanism to explain time series prediction. In particular, the authors train an interpretable surrogate model by learning H^E and G^E in the embedding space.\n\nThe objective function optimizes model behavior consistency by considering the distance in the training embedding, Z, and explained embedding, Z^E.\n", "questions": "Table 12 results in the Supplementary materials require an explanation of the SeqComb-MV dataset. Specifically, the  MBC and LA loss in isolation has very poor AUP, ~0.0576. But the combined MBC + LA results in AUP 0.8326. The performance boost is really large.\n\nDo the authors have any explanation for this behavior?\n", "limitations": "Yes, the authors explained use cases where their model is not the best.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Discrete masking as opposed to continuous masking\n2. Landmark to partition latent space visualization and shape of the signal\n3. Can be generalized to other classification tasks and neural model architectures\n", "weaknesses": "As demonstrated by the ablation studies, the label alignment (LA) loss is always better than MBC in isolation.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your constructive feedback about our work. We appreciate that you recognized the core contributions of our work, as well as the novelty of multiple components of TimeX, such as discrete masking, landmark explanations, and model behavior consistency learning. We respond to your concerns about our work in the following comment. Please reach out if you have any additional questions or clarifications about our work.\n\n### W1 and Q1: LA vs. MBC Loss in Isolation\nWe thank you for noting this peculiar result in our work. The explanation for this behavior gets at a core of the motivation behind TimeX: faithful explainers should match multiple internal states of the model.\n\nFirst, we recall an argument presented in Section 4.3, where we justify MBC. We remark that perturbation-based methods have a similar idea to TimeX: find some sparse perturbation to the input that can preserve the output state of the model. This is often done through observing the predicted label from the model under an applied mask or perturbation, e.g., in one of our baselines, Dynamask [1]. A perturbation that preserves the output state is said to be “faithful” to the model because it is assumed that the perturbation causes. In a sense, MBC generalizes this idea to latent spaces, ensuring that invariances are preserved on the latent space of the model as well as the prediction space.\n\nBeyond the introduction of MBC alone, another core contribution of our work focuses on optimizing faithfulness to predictor models on multiple levels. We use multiple hidden or output states of the model, e.g., a latent space and a logit space, on which the explainable surrogate should match the reference predictor. The hypothesis behind this innovation is that model behavior (the exact objective we are trying to explain) cannot be fully captured by one state, e.g., a predicted label, but rather by multiple states throughout the model. A similar assumption is made in knowledge distillation, where methods often optimize student models to match the teacher on multiple layers of the network. **Therefore, MBC and LA together enforce adherence to model behavior on two fronts: the latent space and prediction space, respectively.** This explains the behavior that you mention in our ablation studies: MBC and LA perform poorly alone, but together, these two losses provide a powerful objective to train the model. \n\n**References**:\n[1] Crabbe and van der Schaar, “Explaining Time Series Predictions with Dynamic Masks,” ICML 2021."}, {"review_id": "VMjlKwRrQ3", "reviewer": "Reviewer_9pJd", "summary": "This paper proposed TimeX that creates an interpretable surrogate model for pretrained models. To ensure faithfulness to the reference model, this paper introduces a self-supervised objective, model behavior consistency, a novel formulation that ensures the preservation of relationships in the latent space induced by the pretrained model, as well as the latent space induced by TimeX. As a result, TimeX could find well-distributed landmarks and highlight a range of salient timestamps.\n", "questions": "1. Any metric for the quality of the explanation? For example, better explanation should show high IoU with ground truth. This kind of qualitative analysis would assure the superiority of TimeX.\n2. If Table 3 is at the top of the page, the visibility would be improved.\n\n", "limitations": "The limitation that authors provided is acceptable as there is no ground truth for other tasks except classification.\n", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. It is well-written and has strong motivation.\n2. This is the first paper to suggest in-hoc explanations for time series data.\n3. This article offers additional justifications for its design decisions, including discrete masking and consistency learning.\n4. The evaluation is very convincing, using diverse datasets and many compared methods.", "weaknesses": "1. No discussion about the quality of explanation. Please see questions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful feedback about our work. We appreciate the reviewer’s acknowledgement of the novelty of our work, as well as the utility and diversity of our extensive evaluations. We have responded to your concerns in the below comments, running 1 additional experiment. We encourage you to reach out if there are any further questions or clarifications needed.\n\n### W1 and Q1: Quality of explanations\nWe believe there is some confusion regarding our experiments. We use two primary setups to evaluate the quality of TimeX explanations - ground-truth comparison and ablation analysis. These are both common analyses in XAI literature, including time series XAI literature [1,2,3]. \n\nIn the ground-truth comparison, we compare the explanations of TimeX and baselines to known temporal signals in each dataset that are known to drive the model prediction. We use four synthetic datasets and one real-world dataset (ECG) for this evaluation setup. We then use three separate metrics that compare **ground-truth explanations to explanations from each XAI method**. The area under the precision-recall curve (AUPRC), area under precision curve (AUP), and area under recall curve (AUR) are all metrics to evaluate the correspondence between explanations from each method and ground-truth explanations. These evaluations are described in detail throughout Section 5, 6, and Appendix C. Intersection over Union (IoU) measures the same qualities of explanations—correspondence to ground-truth explanations—as AUPRC, AUP, and AUR. Therefore, our chosen metrics in this setup measure the same qualities of explanations as the IoU metric. \n\n| Method | IoU | AUPRC |\n|---|---|---|\n| Integrated Gradients | 0.3750 $\\pm$ 0.0022 | 0.5760 $\\pm$ 0.0022 |\n| Dynamask | 0.2958 $\\pm$ 0.0014 | 0.4421 $\\pm$ 0.0016 |\n| TimeX | 0.5214 $\\pm$ 0.0019 | 0.7124 $\\pm$ 0.0017 |\n\n\n\nTo illustrate this point, we include results in the table above of TimeX and two of the strongest baselines—Dynamask and Integrated Gradients—measuring the IoU score on the SeqCombSingle dataset. For comparison to our metrics, we include the AUPRC results for the same methods. The IoU metric has high correlation with the AUPRC metric, with each metric resulting in the same ranking of methods and TimeX achieving the highest metric.\n\nThe other setup we use to evaluate explanation quality is an occlusion analysis. In this case, we mask-out (or occlude) values predicted by the explainer to be un-important, and we measure the drop in classification performance for the predictor under these explanation-dependent perturbations. This allows us to evaluate on real-world datasets where no ground-truth explanation is available. We describe these experiments in Section 6, R2 and results are shown in in-text Figure 3. \n\nWe hope that this clears up some confusion about our evaluations. All of our experiments seek to evaluate the quality of explanations from each method. We have extensive evaluations in this regard, using 8 datasets with a combination of synthetic and real-world time series datasets. In addition, our Appendix includes additional analyses on a variety of time series classification architectures. We have included a brief experiment using IoU metrics to illustrate the point that our experiments capture similar qualities to that measured by the IoU score.\n\n### Q2: Table 3\nThank you for this suggestion concerning the readability of our manuscript. In response, we have moved Table 3 to the top of the page.\n\n**References**:\n[1]: Agarwal et al., “Evaluating explainability for graph neural networks”, *Scientific Data*, 2023.\n[2]: Crabbe and van der Schaar, “Explaining Time Series Predictions with Dynamic Masks,” ICML 2021.\n[3]: Tonekaboni et al., “What went wrong and when? Instance-wise Feature Importance for Time-series Models”, NeurIPS 2019."}, {"review_id": "EaFzTUNf6p", "reviewer": "Reviewer_wJLT", "summary": "This paper introduces a novel method to train an interpretable surrogate of pre-trained  time series classification models. TimeX produces a latent embedding of time series observation and outputs classification probabilities that are both consistent with a reference model. At the same time, it identifies time localized patterns in the time series that allowed it to make predictions. Last, it learns landmarks in the latent space that allow the user to easily compare the observed time series.\n\nFrom pre-trained encoder $G$ and decoder $F$, TimeX trains an explanation generator $H_E$, encoder $G_E$ and decoder $F_E$. $H_E$ produces a discrete masking function that allows to localize and consider only sparse yet connected regions of interest in the input time series. $G_E$ and  $F_E$ are jointly trained to respectively preserve the \"topology\" of the latent spaces and the output distributions.  \n\nPlease note that the submission is not in my area of expertise. I can assess the soundness of the claims and the presentation but it is hard to evaluate its broader contribution to the field.", "questions": "1) I did not understand how and exactly when the Landmarks are learned. Is it post-hoc ? Or is there some partial overlap with the training of the rest of the model which encourages the finding a parsimonious latent embedding ?\n\n2) It is not really discussed how the two latent spaces dimensions are related. Since $G^E$ is initialized with $G$, I am assuming that they are the same, but I imagine that one might use a lower dimensional embedding space of explanation. It would be relatively straightforward to initialize $G^E:= P G$ where $P$ is a linear projection on a lower dimensional space. Would it make the model even more interpretable (and avoid post-hoc treatment with UMAP as in Figure 2) or would it penalize the model too much ?", "limitations": "Authors adequately addressed the limitations of their model.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The notion of Model Behavior Consistency is, from my understanding, the main contribution of this paper. Although many methods aim at  embedding high dimensional observations in topology-preserving latent spaces, the idea of learning an interpretable embedding consistent with pre-trained models is novel to my knowledge. I find the approach particularly relevant as high-capacity models become more and more accessible.\n\nAlthough the technical description of the method is not very well written (see below), each modeling choice is convincing and well motivated (each term of the *ad-hoc* loss is easily interpretable and the need for discrete masking for time series is well demonstrated). \n\nThe experimental results demonstrate improved performance that I cannot really evaluate (see above).\n\n\n\n", "weaknesses": "1) Some statements seem inaccurate if not incorrect. (I will be happy to be proven wrong, as I am not an expert in Time-Series Classification)\n\n   - line.22 \"State-of-the-art time series models are high-capacity pre-trained neural networks\" -> This might be true for time-series classification, but seems wrong in other areas like forecasting, where uncertainty quantification is key. \n\n   - There is an apparent conflict between the need for \"pinpointing the specific location [...] that influence the model's prediction\" (line 24) and the fact that \"temporal patterns only show up when looking at time segments and long term trends\" (line 54). For example, if the feature of interest is the frequency content of a time series, it seems pointless to precisely localize it in time. From my understanding, TimeX reaches an in-between suitable to pinpoint patterns \"broadly-localized\" in time. The paper would gain in clarity by stating more clearly what the method is trying to achieve. \n\n   - line 47. \"the uninterpretable nature of time series data\". I would avoid such vague statements. For example, the QRS intervals subsequently analyzed are straightforwardly interpretable and corresponds to ventricles depolarization and heart contraction.\n\n2) The description of the method is not intelligible but could easily be improved. \n   - line 154. The functions $D_Z$ and $D_Z^E$ are used to motivate Model Behavior Consistency, but it is not explained that they convey a notion of distance in the latent space (this only shows up line 241)\n\n   - Although I understand  the relationship $D_Z(z_i, z_j) \\approx D_Z^E(z_i^E, z_j^E)$ (line 154) and its implication, it is very confusing to write $z_j^E = G^E(E(x_i))$ (line 153).  $z_j^E$ is a latent representation of the masked time series $x^m_j$. And $E$ is defined above as explanation that takes value in $[0, 1]$ (see line 189). Did I miss something ? \n\n   - The notation $H^E: \\mathcal{X} \\rightarrow p \\in [0, 1]^{T \\times d}$ does not make sense (line 184). Is $\\mathcal{X}$ the input space of the function $H^E$ or its input ? In the former case, one should write $H^E: \\mathcal{X} \\rightarrow [0, 1]^{T \\times d}$. In the latter case it implies that the mask applies uniformly to the whole dataset $\\mathcal{X} = (x_i)_i$ which contradicts my intuition from Figure 2.\n\n3) Figure 4 clearly illustrates what part of the ECG the model based its prediction on. Nevertheless, there is no further analysis of the discovered temporal patterns. Are they unexpected or are they stereotyped signals easily classifiable by a physician ? In the latter case, it would have been interesting to use an additional classification task and illustrate how TimeX can identify new temporal patterns in observations, and possibly be used for exploratory data analysis.\n\nMinor points: line 91. XAI: abrevation not defined ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable comments and critiques of our work. We have worked hard to improve the communication of our method, and we kindly ask you to raise your score. Please reach out to us with any questions.\n### W1a: State-of-the-art models\nThank you for pointing out this misleading statement. Many variants of transformers, often very large models, have become preeminent models for various time series tasks. See recent works [1,2] for time series classification, [3,4] for forecasting, and [5] for anomaly detection. Additionally, uncertainty quantification methods have been developed for large neural networks used in forecasting [6]. We have revised our statement: “Prevailing time series models are often high-capacity pre-trained neural networks.”\n### W1b: Modeling frequency content of time series\nThere is indeed value in identifying the specific location of a temporal pattern. For example, in physiological signal diagnosis via ECG or EEG, the exact time that a pattern occurred is of interest to practitioners [7]. TimeX is highly-effective in this setting, identifying the location of a temporal pattern while providing similar explanations through the learned landmarks. \n\nHowever, predictions in time series can be driven by a variety of patterns, such as from the frequency domain. In these settings, TimeX’s explanations would be suboptimal as it cannot provide frequency-level explanations. However, baselines time series explainers (Dynamask and WinIT) would also fail in this case. While many time series predictors model the frequency domain [3], to the knowledge of the authors, no time series explainers have been developed with this capability. This desideratum is left for future work\n### W1c: Uninterpretability of time series\nTime series data can be visually uninterpretable as in noisy samples and long sequences can obstruct visualization of meaningful signals. Thus, one saliency map is not enough for interpreting time series data, unlike computer vision and natural language processing where explainers rely on human perception by overlaying saliency maps on images or tokens. This motivates landmark explanations to provide a mechanism to relate instances on the dataset. We agree that QRS intervals are indeed interpretable by experts, but this is a special case. Decades of research has focused on understanding ECG due to its importance in medicine, thus these data are readily interpretable. \n\nWe have edited this claim: “While explainers designed for other modalities can be adapted to time series, their inherent biases can miss important structures in time series and their reliance on isolated visual interpretability do not translate effectively to the time series where data are less-immediately interpretable.”\n### W2a: D_Z and D_{Z^E}:\nWe have now moved our initial definition of $D_Z$ and $D_{Z^E}$ to Section 3 to make the problem formulation more readable.\n### W2b: Notation of G^E:\nWe agree it is an abuse of notation to use $E$ to represent both the explanation and the masked version of the input sample. We have edited the text to use $x_j^m$ to denote the input to $G^E$. \n### W2c: Notation of H^E:\nThank you for pointing this out, $H^E: \\mathcal{X} \\rightarrow [0,1]^{T \\times d}$ is the proper notation. We have made this edit in text.\n### W3: Discovered temporal patterns\nThe landmark explanations group together similar explanations that constitute temporal patterns across the dataset. It would be ideal to consult domain experts and conduct a user study to understand if the landmarks represent novel patterns. However, this cannot be conducted within the time allotted for rebuttals. Therefore, we provide a quantitative analysis in **Figure 4**.\n\nTo examine landmark quality on the ECG dataset, we compare two groups: 1) the landmark group, with the five nearest-neighbors around the top-10 filtered landmarks and 2) the random group, with five random explanations. We then compare the structure of the most salient values in the samples. We mask-in the top-10 time steps as identified by TimeX, and then compute DTW [8] distance between samples in each group. We then plot the distribution of average within-group DTW distances in **Figure R4**. The high quality of learned landmarks is demonstrated by a lower average DTW within their groups. We have added this analysis to the Appendix.\n### Q1: How landmarks are learned:\nLandmarks are learned during training. We do not backpropagate the gradients of the landmark explanation learning to the entirety of the TimeX model to prevent the landmark explanations from harming the performance of the explainer.\n### Q2: Latent spaces:\n$G$ and $G^E$ embed $Z$ and $Z^E$ respectively, but the inputs to both $G$ and $G^E$ are different—$\\mathcal{X}$ and $\\mathcal{X}^m$, respectively—as described in Section 4.2. This allows TimeX to optimize the mask $M_{\\mathcal{X}}$ on $G^E$ rather than $G$, where $\\mathcal{X}^m$ might be out-of-distribution from $\\mathcal{X}$. Therefore, the formulation of $G^E := PG$ is too strict and might prevent the model from learning complex temporal patterns. \n\n**References**: [1] Zerveas et.al., A transformer-based framework for multivariate time series representation learning. KDD 2021.\n[2] Chowdhury et.al., TARNet: Taskaware reconstruction for time-series transformer. KDD 2022.\n[3] Zhou et.al., FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. ICML 2022.\n[4] Wu et.al., Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. NeurIPS 2021.\n[5] Tuli et.al., TranAD: Deep transformer networks for anomaly detection in multivariate time series data. VLDB 2022.\n[6] Wu et.al., Quantifying Uncertainty in Deep Spatiotemporal Forecasting. KDD 2021.\n[7] Lundberg et al., Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. *Nat. Bio. Eng.* 2018.\n[8] Muller, Dynamic time warping. Information retrieval for music and motion 2007."}]

Author rebuttal:
Thank you to all the reviewers for thoughtful and insightful feedback! We are pleased that reviewers are excited about the novel contributions of our work. Reviewers remark that TimeX **“provide[s] interpretable explanations that are both faithful to the model’s predictions and informative about the underlying temporal patterns”** [Uw18]. Reviewers are also impressed with our strong evaluation results, noting that **“the evaluation is very convincing, using diverse datasets and many compared methods”** [9pJd] and that TimeX **“provides significant performance improvements in synthetic and real-world dataset[s]”** [Mf92]. Finally, reviewers find our work **“particularly relevant as high capacity models become more … accessible”** [wJLT], noting how TimeX **“shows to be effective in explaining a variety of time series models”** [Uw18] and **“can be generalized to other classification tasks and neural model architectures”** [7JHq]. We thank the reviewers for the strong praise of our work and contributions.

We now highlight a few important points raised by reviewers that warrant inclusion in the general response:
### Highlights of New Experiments
In response to reviewers comments, we have run 6 additional experiments and generated 4 additional plots. We have included these experimental details and results in individual rebuttals, with an experiment on computational efficiency detailed here in the overall response. We have also attached a single PDF page containing the four figures and two tables we generated in response to reviewer comments. We now briefly describe these experiments and the figures/tables referenced in each:
* **Runtime experiment** [Uw18,Mf92]: We show that TimeX is much faster at inference time than time series explainer baselines. Results are presented in **Response Table 1**. Please see Point 3 in the general response below.
* **Irregular time series** [Uw18]: We show that TimeX can achieve high explanation performance on an irregular time series synthetic dataset. The table is shown in **Response Table 2**.
* **Forecasting** [Uw18]: We demonstrate that TimeX can be used to explain forecasting tasks. A visualization is shown in **Response Figure 1**.
* **Visualizing explanations across model architectures** [Uw18]: We show a visualization of TimeX explanations across the transformer, CNN, and LSTM architecture on the FreqShapes dataset. This visualization is shown in **Response Figure 2**.
* **Varying $r$ parameter** [Uw18]: We find that TimeX is robust to choices of the $r$ parameter in the explanation loss. A visualization of these results is shown in **Response Figure 3**.
* **Applying IoU metric** [9pJd]: We add a new metric, Intersection-over-Union (IoU), that corroborates TimeX’s high explanation performance. The results are presented in the rebuttal to Reviewer 9pJd.
* **Landmark analysis** [wJLT]: We show that the landmarks learned by TimeX are high-quality and capture regions of the latent space with high similarity. Results are visualized in **Response Figure 4**. 

### Note about in-hoc vs. surrogate explainers
We mistakenly referred to TimeX as an “in-hoc predictor” in-text. We appreciate Reviewer Uw18 pointing out this error. While the result of the model is an end-to-end differentiable time series explainer, TimeX is fundamentally a surrogate explainer method. We have appropriately removed the claims of TimeX being an in-hoc method and have clarified that it is a surrogate method in-text.

### Computational runtime analysis
We thank Reviewer Uw18 and Reviewer Mf92 for requesting more analysis of the computational resources required for TimeX against baseline explainers. Table X shows the training and inference time in seconds of TimeX versus two state-of-the-art time series-specific baselines, Dynamask and WinIT. We choose two real-world time series datasets, PAM and Epilepsy, which are of varying sizes. PAM contains 4266 training samples and 534 testing samples, each of 600 time steps in length. Epilepsy contains 8280 training samples and 2300 testing samples, each of 178 time steps in length. **Table 1** shows the time in seconds needed to train each explainer and to perform inference on the testing set.

TimeX is by-far the most efficient model at inference time for both datasets. This result is expected, as Dynamask and WinIT both require iterative procedures for each sample at inference time, while TimeX requires only a forward pass of $H^E$ at inference. Combining training and inference time, TimeX is the second-fastest on both datasets. However, WinIT and Dynamask times vary greatly between each dataset, with Dynamask as the fastest on PAM while WinIT the fastest on Epilepsy. WinIT scales poorly to samples with many time steps while Dynamask scales poorly to large testing sets. TimeX strikes a compromise between these extremes, scaling better than Dynamask to large numbers of testing samples while scaling better than WinIT to long time series.

### General Notes to Reviewers
Every figure and table reference starting with “R#” refers to figures and tables in the PDF attached to the general response; to make this clear, we bold figure names such as **Figure R1**. We abbreviate weaknesses by “W#” and questions by “Q#”. Despite being barred from submitting a revised manuscript, we assure the reviewers that edits have been incorporated into our revisions. We also use several common acronyms throughout the responses:
* XAI: Explainable artificial intelligence
* MBC: Model behavior consistency (Section 4.3)
* LA: Label alignment
* AUPRC, AUP, AUR: Metrics mentioned in Appendix C.4
We thank all reviewers again for their thoughtful commentary. We worked hard to improve our paper, and we sincerely hope the reviewers find our responses informative and helpful. If you feel the responses have not addressed your concerns to motivate increasing your score, we would love to hear what points of concern remain and how we can improve our work. Thank you again!

Meta-review: The authors provided very detailed responses to reviews to address concerns including running 6 additional experiments, and the reviewers' ratings unanimously leaned toward acceptance. I thus recommend accepting this paper as well. Especially as this paper addresses a rather broadly applicable problem of interpreting time series models with very promising results, I am further nominating this paper for a spotlight.
Decision: Accept (spotlight)