SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The challenge lies in training  $G^{E}(H^{E}(\cdot))$  to faithfully represent  $F(G(\cdot))$ . We approach this by considering the latent spaces of  $G$  and  $G^{E}$ . If  $G$  considers  $\mathbf{x}_i$  and  $\mathbf{x}_j$  to be similar in  $Z$ , we expect that a faithful  $G^{E}$  would encode  $\mathbf{x}_i^m$  and  $\mathbf{x}_j^m$  similarly. However, directly aligning  $G$  and  $G^{E}$  is not optimal due to potential differences in the geometry of the explanation embedding space compared to the full input latent space. To address this, we introduce model behavior consistency (MBC). This novel self-supervised objective trains the explainer model to mimic the behavior of the original model without strict alignment between the spaces. Denote the latent space induced by  $G$  and  $G^{E}$  as  $Z$  and  $Z^{E}$ , respectively. The MBC objective is thus defined as:
$$
\mathcal {L} _ {\mathrm {M B C}} (Z, Z ^ {E}) = \frac {1}{N ^ {2}} \sum_ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in Z} \sum_ {\mathbf {z} _ {i} ^ {E}, \mathbf {z} _ {j} ^ {E} \in Z ^ {E}} \left(D _ {Z} \left(\mathbf {z} _ {i}, \mathbf {z} _ {j}\right) - D _ {Z ^ {E}} \left(\mathbf {z} _ {i} ^ {E}, \mathbf {z} _ {j} ^ {E}\right)\right) ^ {2}, \tag {3}
$$
where  $D_Z$  and  $D_{Z^E}$  are distance functions on the reference model's latent space and the explanation encoder's latent space, respectively, and  $N$  is the size of the minibatch, thereby making  $N^2$  equal to the number of pairs on which  $\mathcal{L}_{\mathrm{MBC}}$  is optimized. This objective encourages distances to be similar across both spaces, encouraging  $Z^{E}$  to retain a similar local topology to  $Z$  without performing a direct alignment. This is closely related to cycle-consistency loss, specifically cross-modal cycle-consistency loss as [79]. We use cosine similarity for  $D_Z$  and  $D_{Z^E}$  throughout experiments in this study, but any distance can be defined on each respective space.
In addition to MBC, we use a label consistency (LC) objective to optimize TIMEX. We train a predictor  $F^{E}$  on  $Z^{E}$  to output logits consistent with those output by  $F$ . We use a Jensen-Shannon Divergence ( $D_{\mathrm{JS}}$ ) between the logits of both predictors:
$$
\mathcal {L} _ {\mathrm {L C}} (Z, Z ^ {E}) = \sum_ {\mathbf {z} _ {i}, \mathbf {z} _ {j} \in Z} \sum_ {\mathbf {z} _ {i} ^ {E}, \mathbf {z} _ {j} ^ {E} \in Z ^ {E}} \left(D _ {\mathrm {J S}} \left(F (\mathbf {z} _ {i}) | | F (\mathbf {z} _ {j})\right) - D _ {\mathrm {J S}} \left(F ^ {E} \left(\mathbf {z} _ {i} ^ {E}\right) | | F ^ {E} \left(\mathbf {z} _ {j} ^ {E}\right)\right)\right) ^ {2} \tag {4}
$$
Our total loss function on  $Z^{E}$  can then be defined as a combination of losses:  $\mathcal{L}_{Z^E} = \mathcal{L}_{\mathrm{MBC}} + \lambda_{\mathrm{LC}}\mathcal{L}_{\mathrm{LC}}$ .
Consistency learning justification. MBC offers three critical benefits for explainability. ① MBC enables consistency optimization across two latent spaces  $Z$  and  $Z^{E}$  without requiring that both  $\mathbf{x}_i$  and  $\mathbf{x}_i^m$  be encoded by the same model, allowing the learning of  $E$  on a separate model  $F^{E}(G^{E}(\cdot)) \neq F(G(\cdot))$ . This avoids the out-of-distribution problems induced by directly masking inputs to  $G$ . ② MBC comprehensively represents model behavior for explainer optimization. This is in contrast to perturbation explanations [38, 80, 58] which seek a label-preserving perturbation  $P$  on  $F(G(\cdot))$  where  $F(G(P(\mathbf{x}_i))) \approx F(G(\mathbf{x}_i))$ . By using  $G(\mathbf{x}_i)$  and  $F(G(\mathbf{x}_i))$  to capture the behavior of the reference model, MBC's objective is richer than a simple label-preserving objective. ③ While MBC is stronger than label matching alone, it is more flexible than direct alignment. An alignment objective, which enforces  $\mathbf{z}_i \approx \mathbf{z}_i^E$ , inhibits  $G^{E}$  from learning important features of explanations not
represented in  $Z$ . The nuance and novelty of MBC are in learning a latent space that is faithful to model behavior while being flexible enough to encode rich relational structure about explanations that can be exploited to learn additional features such as landmark explanations. Further discussion of the utility of MBC is in Appendix B.
4.4 Learning explanation landmarks and training TIMEX models
Leveraging the latent space, TIMEX generates landmark explanations  $\mathbf{z}^L\in \mathbb{R}^{d_z}$ . Such landmarks are desirable as they allow users to compare similar explanation patterns across samples used by the predictor. Landmarks are learned by a landmark consistency loss, and their optimization is detached from the gradients of the explanations so as not to harm explanation quality. Denote the landmark matrix as  $\mathbf{L}\in \mathbb{R}^{n_L\times d_z}$  where  $n_{L}$  corresponds to the number of landmarks (a user-chosen value) and  $d_{z}$  is the dimensionality of  $Z^{E}$ . For each sample explanation embedding  $\mathbf{z}_i^E$ , we use Gumbel-Softmax STE (GS) to stochastically match  $\mathbf{z}_i^E$  to the nearest landmark in the embedding space. Denote the vector of similarities to each  $\mathbf{z}_i^E$  as  $s(\mathbf{z}_i^E,\mathbf{L})$ . Then the assimilation  $A$  is described as:
$$
A \left(\mathbf {z} _ {i} ^ {E}; \mathbf {L}\right) = \operatorname {G S} \left(\operatorname {s o f t m a x} \left(s \left(\operatorname {s g} \left(\mathbf {z} _ {i} ^ {E}\right), \mathbf {L}\right)\right)\right) \mathbf {L}, \tag {5}
$$
where  $\mathrm{sg}$  denotes the stop-grad function. The objective for learning landmarks is then  $\mathcal{L}_{\mathrm{MBC}}(Z,A(Z^E;\mathbf{L}))$ , optimizing the consistency between the assimilated prototypes and the reference model's latent space. Landmarks are initialized as a random sample of explanation embeddings from  $Z^{E}$  and are updated via gradient descent. After learning landmarks, we can measure the quality of each landmark by the number of  $\mathbf{z}_i^E$  embeddings closest to it in latent space. We filter out any landmarks not sufficiently close to any samples (described in Appendix B).
TIMEX training. The overall loss function for TIMEX has four components:  $\mathcal{L} = \mathcal{L}_{\mathrm{MBC}} + \lambda_{\mathrm{LC}}\mathcal{L}_{\mathrm{LC}} + \lambda_E(\mathcal{L}_m + \lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{con}})$ , where  $\lambda_{\mathrm{LC}}, \lambda_E, \lambda_{\mathrm{con}} \in \mathbb{R}$  are weights for the label consistency loss, total explanation loss, and connective explanation loss, respectively. TIMEX can be optimized end-to-end, requiring little hyperparameter choices from the user. The user must also choose the  $r$  parameter for the explanation regularization. Explanation performance is stable across choices of  $r$  (as found in [50]), so we set  $r = 0.5$  to remain consistent throughout experiments. A lower  $r$  value may be provided if the underlying predictive signal is sparse; this hyperparameter is analyzed in Appendix C.3. In total, TIMEX optimizes  $H^E$ ,  $G^E$ , and  $F^E$ .
5 Experimental setup
Datasets. We design four synthetic datasets with known ground-truth explanations: FreqShapes, SeqComb-UV, SeqComb-MV, and LowVar. Datasets are designed to capture diverse temporal dynamics in both univariate and multivariate settings. We employ four datasets from real-world time series classification tasks: ECG [81] - ECG arrhythmia detection; PAM [82] - human activity recognition; Epilepsy [83] - EEG seizure detection; and Boiler [84] - mechanical fault detection. We define ground-truth explanations for ECG as QRS intervals based on known regions of ECG signals where arrhythmias can be detected. The R, P, and T wave intervals are extracted following [85]. Dataset details are given in Appendix C.1 and C.4.
Baselines. We evaluate the method against five explainability baselines. As a general explainer, we use integrated gradients (IG) [31]; for recent time series-specific explainers, we use Dynamask [58], and WinIT [86]; for an explainer that uses contrastive learning, we use CoRTX [87]; and for an in-hoc explainer which has been demonstrated for time series, we use SGT + Grad [17].
Evaluation. We consider two approaches. Ground-truth explanations: Generated explanations are compared to ground-truth explanations, i.e., known predictive signals in each input time series sample when interpreting a strong predictor, following established setups [6]. We use the area under precision (AUP) and area under recall (AUR) curves to evaluate the quality of explanations [58]. We also use the explanation AUPRC, which combines the results of AUP and AUR. For all metrics, higher values are better. Definitions of metrics are in Appendix C.4. Feature importance under occlusion: We occlude the bottom  $p$ -percentile of features as identified by the explainer and measure the change in prediction AUROC (Sec. 4.2). The most essential features a strong explainer identifies should retain prediction performance under occlusion when  $p$  is high. To control for potential misinterpretations, we include a random explainer reference. Our experiments use transformers [88] with time-based positional encoding. Hyperparameters and compute details are given in Appendix C.
Method	AUPRC	FreqShapes	AUR	AUPRC	SeqComb-UV
AUP	AUR	AUP	AUR
IG	0.7516±0.0032	0.6912±0.0028	0.5975±0.0020	0.5760±0.0022	0.8157±0.0023	0.2868±0.0023
Dynamask	0.2201±0.0013	0.2952±0.0037	0.5037±0.0015	0.4421±0.0016	0.8782±0.0039	0.1029±0.0007
WinIT	0.5071±0.0021	0.5546±0.0026	0.4557±0.0016	0.4568±0.0017	0.7872±0.0027	0.2253±0.0016
CoRTX	0.6978±0.0156	0.4938±0.0004	0.3261±0.0012	0.5643±0.0024	0.8241±0.0025	0.1749±0.0007
SGT + Grad	0.5312±0.0019	0.4138±0.0011	0.3931±0.0015	0.5731±0.0021	0.7828±0.0013	0.2136±0.0008
TIMEX	0.8324±0.0034	0.7219±0.0031	0.6381±0.0022	0.7124±0.0017	0.9411±0.0006	0.3380±0.0014
Table 1: Attribution explanation performance on univariate synthetic datasets.
Method	AUPRC	SeqComb-MV AUP	AUR	AUPRC	LowVar AUP	AUR
IG	0.3298±0.0015	0.7483±0.0027	0.2581±0.0028	0.8691±0.0035	0.4827±0.0029	0.8165±0.0016
Dynamask	0.3136±0.0019	0.5481±0.0053	0.1953±0.0025	0.1391±0.0012	0.1640±0.0028	0.2106±0.0018
WinIT	0.2809±0.0018	0.7594±0.0024	0.2077±0.0021	0.1667±0.0015	0.1140±0.0022	0.3842±0.0017
CoRTX	0.3629±0.0021	0.5625±0.0006	0.3457±0.0017	0.4983±0.0014	0.3281±0.0027	0.4711±0.0013
SGT + Grad	0.4893±0.0005	0.4970±0.0005	0.4289±0.0018	0.3449±0.0010	0.2133±0.0029	0.3528±0.0015
TIMEX	0.6878±0.0021	0.8326±0.0008	0.3872±0.0015	0.8673±0.0033	0.5451±0.0028	0.9004±0.0024
Table 2: Attribution explanation performance on multivariate synthetic datasets.
R1: Comparison to existing methods on synthetic and real-world datasets.
Synthetic datasets. We compare TIMEX to existing explainers based on how well they can identify essential signals in time series datasets. Tables 1-2 show results for univariate and multivariate datasets, respectively. Across univariate and multivariate settings, TIMEX is the best explainer on 10/12 (3 metrics in 4 datasets) with an average improvement in the explanation AUPRC  $(10.01\%)$ , AUP  $(6.01\%)$ , and AUR  $(3.35\%)$  over the strongest baselines. Specifically, TIMEX improves ground-truth explanation in terms of AUP by  $3.07\%$  on FreqShapes,  $6.3\%$  on SeqComb-UV,  $8.43\%$  on SeqComb-MV, and  $6.24\%$  on LowVar over the strongest baseline on each dataset. In all of these settings, AUR is less critical than AUP since the predictive signals have redundant information. TIMEX achieves high AUR because it is optimized to output smooth masks over time, tending to include larger portions of entire subsequence patterns than sparse portions, which is relevant for human interpretation. To visualize this property, we show TIMEX's explanations in Appendix C.5.
Real-world datasets: arrhythmia detection. We demonstrate TIMEX on ECG arrhythmia detection. TIMEX's attribution maps show a state-of-the-art performance for finding relevant QRS intervals driving the arrhythmia diagnosis and outperform the strongest baseline by  $5.39\%$  (AUPRC) and  $9.83\%$  (AUR) (Table 3). Integrated gradients achieve a slightly higher AUP, whereas state-of-the-art time series explainers perform poorly. Notably, TIMEX's explanations are significantly better in AUR, identifying larger segments of the QRS interval rather than individual timesteps.
Ablation study on ECG data. We conduct ablations on the ECG data using TIMEX (Table 3). First, we show that the STE improves performance as opposed to soft attention masking, resulting in an AUPRC performance gain of  $9.44\%$ ; this validates our claims about the pitfalls of soft masking for time series. Note that this drop in performance becomes more significant when including direct-value masking, as shown in Appendix C.6. Second, we use SimCLR loss to align  $Z^{E}$  to  $Z$  as opposed to MBC; SimCLR loss can achieve comparable results in AUPRC and AUR, but the AUP is  $13.6\%$  lower than the base TIMEX. Third, we experiment with the usefulness of MBC and LC objectives. MBC alone produces poor explanations with AUPRC at  $65.8\%$  lower score than the base model. LC alone does better than MBC alone, but its AUPRC is still  $21.5\%$  lower than the base model. MBC and LC, in conjunction, produce high-quality explanations, showing the value in including more intermediate states for optimizing  $G^{E}(H^{E}(\cdot))$ . Extensive ablations are provided in Appendix C.6.
R2: Occlusion experiments on real-world datasets.
We evaluate TIMEX explanations by occluding important features from the reference model and observing changes in classification [63, 58, 87]. Given a generated explanation  $E(\mathbf{x}_i)$ , the bottom  $p$ -percentile of features are occluded; we expect that if the explainer identifies important features for the model's prediction, then the classification performance to drop significantly when replacing these important features (determined by the explainer) with baseline values. We compare the performance under occlusion to random explanations to counter misinterpretation (Sec. 3.1). We adopt the masking
Method	AUPRC	ECG AUP	AUR	TIMEX Ablations	AUPRC	ECG AUP	AUR
IG	0.4182±0.0014	0.5949±0.0023	0.3204±0.0012	Full	0.4721±0.0018	0.5663±0.0025	0.4457±0.0018
Dynamask	0.3280±0.0011	0.5249±0.0030	0.1082±0.0080	-STE	0.4014±0.0019	0.5570±0.0032	0.1564±0.0007
WinIT	0.3049±0.0011	0.4431±0.0026	0.3474±0.0011	+SimCLR	0.4767±0.0021	0.4895±0.0024	0.4779±0.0013
CoRTX	0.3735±0.0008	0.4968±0.0021	0.3031±0.0009	Only LC	0.3704±0.0018	0.3296±0.0019	0.5084±0.0008
SGT + Grad	0.3144±0.0010	0.4241±0.0024	0.2639±0.0013	Only MBC	0.1615±0.0006	0.1348±0.0006	0.5504±0.0011
TIMEX	0.4721±0.0018	0.5663±0.0025	0.4457±0.0018
Table 3: (Left) Benchmarking TIMEX on the ECG dataset. (Right) Results of ablation analysis.
(a) Epilepsy
(b) PAM
Figure 3: Occlusion experiments on real-world datasets. Higher values indicate better performance.
(c) Boiler
procedure described in Sec. 4.2, performing attention masking where applicable and direct-value masking otherwise.
Figure 3 compares TIMEX to Dynamask, a strong time-series explainer. On all datasets, TIMEX's explanations are either at or above the performance of Dynamask, and both methods perform above the random baseline. On the Boiler dataset, we demonstrate an average of  $27.8\%$  better classification AUROC across each threshold than Dynamask, with up to  $37.4\%$  better AUROC at the 0.75 threshold. This gap in performance between TIMEX and Dynamask is likely because the underlying predictor for Boiler is weaker than that of Epilepsy or PAM, achieving 0.834 AUROC compared to 0.979 for PAM and 0.939 for Epilepsy. We hypothesize that TIMEX outperforms Dynamask because it only considers changes in predicted labels under perturbation while TIMEX optimizes for consistency across both labels and embedding spaces in the surrogate and reference models. TIMEX performs well across both univariate (Epilepsy) and multivariate (PAM and Boiler) datasets.
R3: Landmark explanation analysis on ECG.
To demonstrate TIMEX's landmarks, we show how landmarks serve as summaries of diverse patterns in an ECG dataset. Figure 4 visualizes the learned landmarks in the latent space of explanations. We choose four representative landmarks based on the previously described landmark ranking strategy (Sec. 4.4). Every landmark occupies different regions of the latent space, capturing diverse types of explanations generated by the model. We show the three nearest explanations for the top two landmarks regarding the nearest neighbor in the latent space. Explanations  $①$ ,  $②$ , and  $③$  are all similar to each other while distinctly different from  $④$ ,  $⑤$ , and  $⑥$ , both in terms of attribution and temporal structure. This visualization shows how landmarks can partition the latent space of explanations into interpretable temporal patterns. We demonstrate the high quality of learned landmark explanations through a quantitative experiment in Appendix C.10.
Additional experiments demonstrating flexibility of TIMEX.
In the Appendix, we present several additional experiments to demonstrate the flexibility and superiority of TIMEX. In Appendix C.8, we replicate experiments on two other time series architectures, LSTM and CNN, and show that TIMEX retains state-of-the-art performance. In Appendix C.9, we demonstrate the performance of TIMEX in multiple task settings, including forecasting and irregularly-sampled time series classification. TIMEX can be adapted to these settings with minimal effort and retains excellent explanation performance.
We develop TIMEX, an interpretable surrogate model for interpreting time series models. By introducing the novel concept of model behavior consistency (i.e., preserving relations in the latent space induced by the pretrained model when compared to relations in the latent space induced by the explainer), we ensure that TIMEX mimics the behavior of a pretrained time series model, aligning
Figure 4: Landmark analysis of TIMEX on the ECG dataset. Shown is a UMAP plot of the latent explanation space along with learned landmark explanations. For two selected landmarks (in red), we show three explanation instances most similar to each landmark.
influential time series signals with interpretable temporal patterns. The generation of attribution maps and utilizing a latent space of explanations distinguish TIMEX from existing methods. Results on synthetic and real-world datasets and case studies involving physiological time series demonstrate the superior performance of TIMEX compared to state-of-the-art interpretability methods. TIMEX's innovative components offer promising potential for training interpretable models that capture the behavior of pretrained time series models.
Limitations. While TIMEX is not limited to a specific task as an explainer, our experiments focus on time series classification. TIMEX can explain other downstream tasks, assuming we can access the latent pretrained space, meaning it could be used to examine general pretrained models for time series. Appendix C.9 gives experiments on various setups. However, the lack of such pretrained time series models and datasets with reliable ground-truth explanations restricted our testing in this area. One limitation of our approach is its parameter efficiency due to the separate optimization of the explanation-tuned model. However, we conduct a runtime efficiency test in Appendix C.7 that shows TIMEX has comparable runtimes to baseline explainers. Larger models may require adopting parameter-efficient tuning strategies.
Societal impacts. Time series data pervades critical domains including finance, healthcare, energy, and transportation. Enhancing the interpretability of neural networks within these areas has the potential to significantly strengthen decision-making processes and foster greater trust. While explainability plays a crucial role in uncovering systemic biases, thus paving the way for fairer and more inclusive systems, it is vital to approach these systems with caution. The risks of misinterpretations or an over-reliance on automated insights are real and substantial. This underscores the need for a robust framework that prioritizes human-centered evaluations and fosters collaboration between humans and algorithms, complete with feedback loops to continually refine and improve the system. This approach will ensure that the technology serves to augment human capabilities, ultimately leading to more informed and equitable decisions across various sectors of society.
We gratefully acknowledge the support of the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001 and awards from NIH under No. R01HD108794, Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, Bayer Early Excellence in Science, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Pfizer Research, Chan Zuckerberg Initiative, and the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders. The authors declare that there are no conflicts of interest.
[1] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural
Appendix A Further discussion of background
Straight-through estimators. Discrete operations, such as thresholding, are often avoided in neural network architectures due to difficulties differentiating discrete functions. To circumvent these issues, [76] introduces the straight-through estimator (STE), which uses a surrogate function during backpropagation to approximate the gradient for a non-differentiable operation. STEs have seen usage in quantized neural networks [S1]. This method shows empirical performance even though there is little theoretical justification behind it [S2].
Self-supervised learning. Methods in self-supervised learning (SSL) have become a common pretraining technique for settings in which large, unlabeled datasets are available [S3, S4, S5, S6]. Common approaches for self-supervised learning are contrastive learning, which seeks to learn representations for samples under invariant data augmentations, and metric learning, which aims to generate a latent space in which a distance function captures some pre-defined relations on data [S7]. Consistency learning has emerged as another promising SSL approach; intuitively, this family of methods seeks to learn latent spaces in which similar pairs are expected to be embedded similarly, i.e., preserving some consistent properties. Consistency learning has seen use in aligning videos [S8], enhancing latent geometry for multimodal contrastive learning [79], and pretraining time series models across time and frequency domains [1].
The use of consistency for explaining machine learning models. Consistency has been considered in previous XAI literature in two ways: 1) consistency between explanations and 2) consistency as an explainability metric, as explained below.