SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Owen Queen
owen_queen@hms.harvard.edu
University of Virginia, MIT
hartvigsen@virginia.edu
Teddy Koker
thomas.koker@ll.mit.edu
Huan He
huan_he@hms.harvard.edu
Theodoros Tsiligkaridis
tsili@ll.mit.edu
Marinka Zitnik
marinka@hms.harvard.edu
Interpreting time series models is uniquely challenging because it requires identifying both the location of time series signals that drive model predictions and their matching to an interpretable temporal pattern. While explainers from other modalities can be applied to time series, their inductive biases do not transfer well to the inherently challenging interpretation of time series. We present TIMEX, a time series consistency model for training explainers. TIMEX trains an interpretable surrogate to mimic the behavior of a pretrained time series model. It addresses the issue of model faithfulness by introducing model behavior consistency, a novel formulation that preserves relations in the latent space induced by the pretrained model with relations in the latent space induced by TIMEX. TIMEX provides discrete attribution maps and, unlike existing interpretability methods, it learns a latent space of explanations that can be used in various ways, such as to provide landmarks to visually aggregate similar explanations and easily recognize temporal patterns. We evaluate TIMEX on eight synthetic and real-world datasets and compare its performance against state-of-the-art interpretability methods. We also conduct case studies using physiological time series. Quantitative evaluations demonstrate that TIMEX achieves the highest or second-highest performance in every metric compared to baselines across all datasets. Through case studies, we show that the novel components of TIMEX show potential for training faithful, interpretable models that capture the behavior of pretrained time series models.
Prevailing time series models are high-capacity pre-trained neural networks [1, 2], which are often seen as black boxes due to their internal complexity and lack of interpretability [3]. However, practical use requires techniques for auditing and interrogating these models to rationalize their predictions. Interpreting time series models poses a distinct set of challenges due to the need to achieve two goals: pinpointing the specific location of time series signals that influence the model's predictions and aligning those signals with interpretable temporal patterns [4]. While explainers designed for other modalities can be adapted to time series, their inherent biases can miss important structures in time series, and their reliance on isolated visual interpretability does not translate effectively to
the time series where data are less immediately interpretable. The dynamic nature and multi-scale dependencies within time series data require temporal interpretability techniques.
Research in model understanding and interpretability developed post-hoc explainers that treat pretrained models as black boxes and do not need access to internal model parameters, activations, and gradients. Recent research, however, shows that such post-hoc methods suffer from a lack of faithfulness and stability, among other issues [5, 6, 7]. A model can also be understood by investigating what parts of the input it attends to through attention mapping [8, 9, 10] and measuring the impact of modifying individual computational steps within a model [11, 12]. Another major line of inquiry investigates internal mechanisms by asking what information the model contains [13, 14, 15]. For example, it has been found that even when a language model is conditioned to output falsehoods, it may include a hidden state that represents the true answer internally [16]. The gap between external failure modes and internal states can only be identified by probing model internals. Such representation
Figure 1: TIMEX learns a latent space of explanations and landmarks to summarize groups of informative temporal patterns in time series.
probing has been used to characterize the behaviors of language models, but leveraging these strategies to understand time series models has yet to be attempted. These lines of inquiry drive the development of in-hoc explainers [17, 18, 19, 20, 21, 22] that build inherent interpretability into the model through architectural modifications [18, 19, 23, 20, 21] or regularization [17, 22]. However, no in-hoc explainers have been developed for time series data. While explainers designed for other modalities can be adapted to time series, their inherent biases do not translate effectively to the uninterpretable nature of time series data. They can miss important structures in time series.
Explaining time series models is challenging for many reasons. First, unlike imaging or text datasets, large time series data are not visually interpretable. Next, time series often exhibit dense informative features, unlike more explored modalities such as imaging, where informative features are often sparse. In time series datasets, timestep-to-timestep transitions can be negligible, and temporal patterns only show up when looking at time segments and long-term trends. In contrast, in language datasets, word-to-word transitions are informative for language modeling and understanding. Further, time series interpretability involves understanding the dynamics of the model and identifying trends or patterns. Another critical issue with applying prior methods is that they treat all time steps as separate features, ignoring potential time dependencies and contextual information; we need explanations that are temporally connected and visually digestible. While understanding predictions of individual samples is valuable, the ability to establish connections between explanations of various samples (for example, in an appropriate latent space) could help alleviate these challenges.
Present work. We present TIMEX, a novel time series surrogate explainer that produces interpretable attribution masks as explanations over time series inputs (Figure 1). ① A key contribution of TIMEX is the introduction of model behavior consistency, a novel formulation that ensures the preservation of relationships in the latent space induced by the pretrained model, as well as the latent space induced by TIMEX. ② In addition to achieving model behavior consistency, TIMEX offers interpretable attribution maps, which are valuable tools for interpreting the model's predictions, generated using discrete straight-through estimators (STEs), a type of gradient estimator that enables end-to-end training of TIMEX models. ③ Unlike existing interpretability methods, TIMEX goes further by learning a latent space of explanations. By incorporating model behavior consistency and leveraging a latent space of explanations, TIMEX provides discrete attribution maps and enables visual aggregation of similar explanations and the recognition of temporal patterns. ④ We test our approach on eight synthetic and real-world time series datasets, including datasets with carefully processed ground-truth explanations to quantitatively benchmark it and compare it to general explainers, state-of-the-art time series explainers, and in-hoc explainers. TIMEX is at https://github.com/mims-harvard/TimeX
Model understanding and interpretability. As deep learning models grow in size and complexity, so does the need to help users understand the model's behavior. The vast majority of explainable AI research (XAI) [24] has focused on natural language processing (NLP) [25, 26, 27] and computer vision (CV) [28, 29, 30]. Commonly used techniques, such as Integrated Gradients [31] and Shapley Additive Explanations (SHAP) [3], and their variants have originated from these domains and gained popularity. XAI has gained significant interest in NLP and CV due to the inherent interpretability of the data. However, this familiarity can introduce confirmation bias [32]. Recent research has expanded to other data modalities, including graphs [33, 6] and time series [34, 35], as outlined below. The literature primarily focuses on post-hoc explainability, where explanations are provided for a trained and frozen model's behavior [36, 37]. However, saliency maps, a popular approach [38, 31, 39], have pitfalls when generated post-hoc: they are surprisingly fragile [5], and lack sensitivity to their explained models [40]. Surrogate-based approaches have also been proposed [41, 42, 43], but these simplified surrogate models fall short compared to the original predictor they aim to explain. Unlike post-hoc explainability, in-hoc methods aim for inherently interpretable models. This can be accomplished by modifying the model's architecture [20], training procedure using jointly-trained explainers [44], adversarial training [45, 46, 47, 48], regularization techniques [17, 22], or refactorization of the latent space [49, 50]. However, such models often struggle to achieve state-of-the-art predictive performance, and to date, these methods have seen limited use for time series.
Beyond instance-based explanations. Several methods have been proposed to provide users with information on model behavior beyond generating instance-based saliency maps explaining individual predictions. Prototype models strive to offer a representative sample or region in the latent space [51, 52]. Such methods are inherently interpretable, as predictions are directly tied to patterns in the feature space. Further, explainability through human-interpretable exemplars has been gaining popularity. Concept-based methods decompose model predictions into human-interpretable concepts. Many works rely on annotated datasets with hand-picked concepts (e.g., "stripes" in an image of a zebra). Relying on access to a priori defined concepts, concept bottleneck models learn a layer that attributes each neuron to one concept [23]. This limitation has spurred research in concept discovery by composing existing concepts [53, 54] or grounding detected objects to natural language [55]. However, the CV focus of these approaches limits their applicability to other domains like time series.
Time series explainability. In contrast to other modalities, time series often have multiple variables, and their discriminative information is spread over many timesteps. Building on these challenges, recent works have begun exploring XAI for time series [56, 57, 58, 59, 60, 61, 34, 62]. Many methods modify saliency maps [35, 63, 58] or surrogate methods [59, 64] to work with time series data. Two representative methods are WinIT [65] and Dynamask [58]. WinIT learns saliency maps with temporal feature importances, while Dynamask regularizes saliency maps to include temporal smoothing. However, these methods rely on perturbing timesteps [63], causing them to lack faithfulness. Common perturbation choices in CV, like masking with zeros, make less sense for time series [56]. Perturbed time series may be out-of-distribution for the model due to shifts in shape [66], resulting in unfaithful explanations akin to adversarial perturbation [67].
3 Problem formulation
Notation. Given is a time series dataset  $\mathcal{D} = (\mathcal{X},\mathcal{Y}) = \{(\mathbf{x}_i,y_i)|i = 1,\dots,N\}$  where  $\mathbf{x}_i$  are input samples and  $y_{i}$  are labels associated to each sample. Each sample  $\mathbf{x}_i\in \mathbb{R}^{T\times d}$  is said to have  $T$  time steps and  $d$  sensors. A feature is defined as a time-sensor pair, where the time  $t$  and sensor  $k$  for input  $\mathbf{x}_i$  is  $\mathbf{x}_i[t,k]$ . Without loss of generality, we consider univariate  $(d = 1)$  and multivariate  $(d > 1)$  settings. Each  $y_{i}\in \{1,2,\dots,C\}$  belongs to one of  $C$  classes. A classifier model consists of an encoder  $G$  and predictor  $F$ . The encoder  $G$  produces an embedding of input  $\mathbf{x}_i$ , i.e.,  $G(\mathbf{x}_i) = \mathbf{z}_i\in \mathbb{R}^{d_z}$ , while the predictor produces some prediction from the embedding in the form of a logit, i.e.,  $F(G(\mathbf{x}_i)) = \hat{y}_i\in [0,1]^C$  where  $\mathrm{argmax}_j\hat{y}_i[j]\in \{1,\dots,C\}$  is the predicted label.
The latent space induced by  $G$  is defined as  $Z$ , e.g.,  $G: \mathcal{X} \to Z$ . We will refer to  $F(G(\cdot))$  as the reference model while  $G$  is the reference encoder and  $F$  is the reference predictor. An explanation is defined as a continuous map of the features that conveys the relative importance of each feature for the prediction. The explanation for sample  $\mathbf{x}_i$  is given as an attribution map  $E(\mathbf{x}_i) \in \mathbb{R}^{T \times d}$  where for any times  $t_1, t_2$  and sensors  $k_1, k_2$ ,  $E(\mathbf{x}_i[t_1, k_1]) > E(\mathbf{x}_i[t_2, k_2])$  implies that  $\mathbf{x}_i[t_1, k_1]$  is a more
Figure 2: Overview of TIMEX approach.
important feature for the task than  $\mathbf{x}_i[t_2,k_2]$ . Finally, we define an occlusion procedure whereby a function  $\Omega$  generates an mask  $M_{\mathbf{x}}$  for a sample  $\mathbf{x}$  from the explanation  $E(\mathbf{x})$ , e.g.,  $\Omega(E(\mathbf{x})) = M_{\mathbf{x}}$ . This mask is applied to  $x$  to derive a masked version of the input  $\mathbf{x}^m$  through an operation  $\odot$ , e.g.,  $M_{\mathbf{x}} \odot \mathbf{x} = \mathbf{x}^m$ . When describing TIMEX, we generally refer to  $\odot$  as an element-wise multiplication.
3.1 Self-supervised model behavior consistency
TIMEX creates an inherently-interpretable surrogate model for pretrained time series models. The surrogate model produces explanations by optimizing two main objectives: interpretability and faithfulness to model behavior. First, TIMEX generates interpretable explanations via an attribution map  $E(\mathbf{x}_i)$  that identifies succinct, connected regions of input important for the prediction. To ensure faithfulness to the reference model, we introduce a novel objective for training TIMEX: model behavior consistency (MBC). With MBC, a TIMEX model learns to mimic internal layers and predictions of the reference model, yielding a high-fidelity time series explainer. MBC is defined as:
Definition 3.1 (Model Behavior Consistency (MBC)). Explanation  $E$  and explanation encoder  $G^{E}$  are consistent with pretrained model  $G$  and predictor  $F$  on dataset  $\mathcal{D}$  if the following is satisfied:
- Consistent reference encoder: Relationship between  $\mathbf{z}_i = G(\mathbf{x}_i)$  and  $\mathbf{z}_j = G(\mathbf{x}_j)$  in the space of reference encoder is preserved by the explainer,  $\mathbf{z}_i^E = G^E (\mathbf{x}_i^m)$  and  $\mathbf{z}_j^E = G^E (\mathbf{x}_j^m)$ , where  $\mathbf{x}_i^m = \Omega (E(\mathbf{x}_i))\odot \mathbf{x}_i$  and  $\mathbf{x}_j^m = \Omega (E(\mathbf{x}_j))\odot \mathbf{x}_j$ , through distance functions on the reference encoder's and explainer's latent spaces  $D_Z$  and  $D_{ZE}$ , respectively, such that:  $D_Z(\mathbf{z}_i,\mathbf{z}_j)\simeq D_{ZE}(\mathbf{z}_i^E,\mathbf{z}_j^E)$  for samples  $\mathbf{x}_i,\mathbf{x}_j\in \mathcal{D}$ .
- Consistent reference predictor: Relationship between reference predictor  $\hat{y}_i = F(\mathbf{z}_i)$  and latent explanation predictor  $\hat{y}_i^E = F^E (\mathbf{z}_i^E)$  is preserved,  $\hat{y}_i\simeq \hat{y}_i^E$  for every sample  $\mathbf{x}_i\in \mathcal{D}$ .
Our central formulation is defined as realizing the MBC between a reference model and an interpretable TIMEX model:
Problem statement 3.1 (TIMEX). Given pretrained time series encoder  $G$  and predictor  $F$  that are trained on a time series dataset  $\mathcal{D}$ , TIMEX provides explanations  $E(\mathbf{x}_i)$  for every sample  $\mathbf{x}_i \in \mathcal{D}$  in the form of interpretable attribution maps. These explanations satisfy model behavior consistency through the latent space of explanations  $Z^E$  generated by the explanation encoder  $G^E$ .
TIMEX is designed to counter several challenges in interpreting time series models. First, TIMEX avoids the pitfall known as the occlusion problem [68]. Occlusion occurs when some features in an input  $\mathbf{x}_i$  are perturbed in an effort that the predictor forgets those features. Since it is well-known that occlusion can produce out-of-distribution samples [69], this can cause unpredictable shifts in the behavior of a fixed, pretrained model [70, 71, 72]. In contrast, TIMEX avoids directly masking input samples to  $G$ . TIMEX trains an interpretable surrogate  $G^E$  to match the behavior of  $G$ . Second, MBC is designed to improve the faithfulness of TIMEX to  $G$ . By learning to mimic multiple states of  $F(G(\cdot))$  using the MBC objective, TIMEX learns highly-faithful explanations, unlike many post hoc explainers that provide no explicit optimization of faithfulness. Finally, TIMEX's explanations are driven by learning a latent explanation space, offering richer interpretability data.
4 TIMEX method
We now present TIMEX, an approach to train an interpretable surrogate model to provide explanations for a pretrained time series model. TIMEX learns explanations through a consistency learning objective where an explanation generator  $H^{E}$  and explanation encoder  $G^{E}$  are trained to match intermediate feature spaces and the predicted label space. We will break down TIMEX in the following sections by components:  $H^{E}$ , the explanation generator,  $G^{E}$ , the explanation encoder, and the training objective of  $G^{E}(H^{E}(\cdot))$ , followed by a discussion of practical considerations of TIMEX. An overview of TIMEX is depicted in Figure 2.
4.1 Explanation generation
Generating an explanation involves producing an explanation map  $E(\mathbf{x})$  where if  $E(\mathbf{x}[t_1, k_1]) > E(\mathbf{x}[t_2, k_2])$ , then feature  $\mathbf{x}[t_1, k_1]$  is considered as more important for the prediction than  $\mathbf{x}[t_2, k_2]$ . Explanation generation is performed through an explanation generator  $H^E: \mathcal{X} \to [0, 1]^T \times d$ , where  $H^E(\mathbf{x}) = \mathbf{p}$ . We learn  $\mathbf{p}$  based on a procedure proposed by [50], but we adapt their procedure for time series. Intuitively,  $\mathbf{p}$  parameterizes a Bernoulli at each time-sensor pair, and the mask  $M_{\mathcal{X}}$  is sampled from this Bernoulli distribution during training, i.e.,  $M_{\mathcal{X}} \sim \mathbb{P}_{\mathbf{p}}(M_{\mathcal{X}}|\mathcal{X}) = \prod_{t,k} \mathrm{Bern}(\mathbf{p}_{t,k})$ . This parameterization is directly interpretable as attribution scores: a low  $\mathbf{p}_{t,k}$  means that time-sensor pair  $(t,k)$  has a low probability of being masked-in. Thus,  $\mathbf{p}$  is also the explanation for  $\mathbf{x}_i$ , i.e.,  $E(\mathbf{x}) = \mathbf{p}$ .
The generation of  $\mathbf{p}$  is regularized through a divergence with Bernoulli distributions  $\operatorname{Bern}(r)$ , where  $r$  is a user-chosen hyperparameter. As in [50], denote the desired distribution of  $\mathbf{p}$  as  $\mathbb{Q}(M_{\mathcal{X}}) = \prod_{(t,k)} \operatorname{Bern}(r)$ . Then the objective becomes:
$$
\mathcal {L} _ {m} (\mathbf {p}) = \mathbb {E} \left[ D _ {\mathrm {K L}} \left(\mathbb {P} _ {\mathbf {p}} \left(M _ {\mathcal {X}} \mid \mathcal {X}\right) \mid \mid \mathbb {Q} \left(M _ {\mathcal {X}}\right)\right) \right] = \sum_ {t, k} \mathbf {p} _ {t, k} \log \frac {\mathbf {p} _ {t , k}}{r} + (1 - \mathbf {p} _ {t, k}) \log \frac {1 - \mathbf {p} _ {t , k}}{1 - r} \tag {1}
$$
The sampling of  $M_{\mathcal{X}} \sim \mathbb{P}_{\mathbf{p}}(M_{\mathcal{X}}|\mathcal{X})$  is performed via the Gumbel-Softmax trick [73, 74], which is a differentiable approximation of categorical sampling. Importantly,  $M_{\mathcal{X}}$  is stochastically generated, which, as discussed in [50, 75], regularizes the model to learn robust explanations.
To generate interpretable attribution masks, TIMEX optimizes for the connectedness of predicted distributions:
$$
\mathcal {L} _ {\mathrm {c o n}} (\mathbf {p}) = \frac {1}{T \times d} \sum_ {k = 1} ^ {d} \sum_ {t = 1} ^ {T - 1} \sqrt {\left(\mathbf {p} _ {t , k} - \mathbf {p} _ {t + 1 , k}\right) ^ {2}}. \tag {2}
$$
The generator of explanations  $H^{E}$  learns directly on input time series samples  $\mathcal{X}$  to return  $\mathbf{p}$ . We build a transformer encoder-decoder structure for  $H^{E}$ , using an autoregressive transformer decoder and a sigmoid activation to output probabilities for each time-sensor pair.
4.2 Explanation encoding
We now describe how to embed explanations with the explanation encoder  $G^{E}$ . Intuitively,  $G^{E}$  learns on the masked distribution of  $\mathcal{X}$ , which can be denoted as  $\mathcal{X}^m$ . Motivated by the occlusion problem, we avoid directly applying the masks onto the pretrained, frozen  $G$ , as  $\mathcal{X}^m$  and  $\mathcal{X}$  are fundamentally different distributions. Therefore, we copy the weights of  $G$  into  $G^{E}$  and fine-tune  $G^{E}$  on  $\mathcal{X}^m$ .
Discretizing attribution masks. When passing inputs to  $G^{E}$ , it is important for the end-to-end optimization to completely ignore regions identified as unimportant by  $H^{E}$ . Therefore, we use a straight-through estimator (STE) [73] to obtain a discrete mask  $M_{\mathcal{X}} \in \{0,1\}^{T \times d}$ . Introduced by [76], STEs utilize a surrogate function to approximate the gradient of a non-differentiable operation used in the forward pass, such as binary thresholding.
Applying masks to time series samples. We use two types of masking procedures: attention masking and direct-value masking. First, we employ differentiable attention masking through a multiplicative operation proposed by Nguyen et al. [77]. When attention masking does not apply, we use a direct-value masking procedure based on architecture choice or multivariate inputs. We approximate a baseline distribution:  $\mathbb{B}_{\mathcal{X}} = \prod_{t,k}\mathcal{N}(\mu_{tk},\sigma_{tk}^2)$ , where  $\mu_{tk}$  and  $\sigma_{tk}^{2}$  are the mean and
variance over time-sensor pairs. Masking is then performed through a multiplicative replacement as:  $\mathbf{x}_i^m = (M_{\mathcal{X}} \odot \mathbf{x}_i) + (1 - M_{\mathcal{X}}) \odot b$ , where  $b \sim \mathbb{B}_{\mathcal{X}}$ .
Justification for discrete masking. It is essential that masks  $M_{\mathcal{X}}$  are discrete instead of continuous. Previous works have considered masking techniques [78, 49, 50] with continuous masks since applying such masks is differentiable with element-wise multiplication. However, continuous masking has a distinctly different interpretation: it uses a continuous deformation of the input towards a baseline value. While such an approach is reasonable for data modalities with discrete structures, such as sequences of tokens (as in [78, 49]) or nodes in graphs [50], such deformation may result in a change of the shape of time series data, which is known to be important for prediction [66]. As a toy example, consider an input time series  $\mathbf{x}_i$  where the predictive pattern is driven by feature  $\mathbf{x}_i[t_1,k_1]$  is larger than all other features. Suppose  $M_{\mathcal{X}}$  is continuous. In that case, it is possible that for a less important feature  $\mathbf{x}_i[t_2,k_2]$ ,  $M_{\mathcal{X}}[t_1,k_1] < M_{\mathcal{X}}[t_2,k_2]$  while  $(M_{\mathcal{X}}[t_1,k_1] \odot \mathbf{x}_i[t_1,k_1]) > (M_{\mathcal{X}}[t_2,k_2] \odot \mathbf{x}_i[t_2,k_2])$ , thereby preserving the predictive pattern. At the same time, the mask indicates that  $\mathbf{x}_i[t_2,k_2]$  is more important than  $\mathbf{x}_i[t_1,k_1]$ . If a surrogate model is trained on  $M_{\mathcal{X}} \odot \mathbf{x}_i$ ,  $M_{\mathcal{X}}$  may violate the ordinality expected by an attribution map as defined in Section 3. Discrete masking alleviates this issue by forcing  $M_{\mathcal{X}}$  to be binary, removing the possibility of confounds created by continuous masking. Therefore, discrete masking is necessary when learning interpretable masks on continuous time series.
4.3 Model behavior consistency