SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ground-truth explanations for ECG datasets. We extract ground-truth explanations via a QRS detection strategy following [85] because an initial set of beat labels was produced by a simple slope-sensitive QRS detector and were then given to two cardiologists, who worked on them independently. The cardiologists added additional beat labels where the detector missed beats, deleted false detections as necessary, and changed the labels for all abnormal beats. We employ Neurokit<sup>1</sup> to extract QRS complexes and also take care to ensure that the QRS is the proper explanation for each class. We consider two types of arrhythmias: left bundle branch block beat and right bundle branch block beat, to categorize our "abnormal" class. We perform the ground-truth evaluation on only the abnormal
$^{1}$  https://github.com/neuropsychology/NeuroKit
Method	IoU	AUPRC
IG	0.3750±0.0022	0.5760±0.0022
Dynamask	0.2958±0.0014	0.4421±0.0016
TIMEX	0.5214±0.0019	0.7124±0.0017
Table 10: IoU metric compared to AUPRC on the SeqComb-UV dataset.
Method	No STE	STE
AUPRC	AUP	AUR	AUPRC	AUP	AUR
FreqShapes	0.6695±0.0038	0.6398±0.0038	0.5454±0.0026	0.8324±0.0034	0.7219±0.0031	0.6381±0.0022
SeqComb-MV	0.5694±0.0023	0.8723±0.0006	0.3229±0.0017	0.6878±0.0021	0.8326±0.0008	0.3872±0.0015
ECG	0.4014±0.0019	0.5570±0.0032	0.1564±0.0007	0.4721±0.0018	0.5663±0.0025	0.4457±0.0018
Table 11: Ablation 1: Ablation using the STE vs. no STE. "No STE" is equivalent to continuous masking, as discussed in Section 4.2.
class, as the normal class signifies negative information, which may be harder to pinpoint based on model logic.
Statistical analysis. We evaluate each experiment on a 5-fold cross-validation of each dataset. We then report average performance and standard error across all folds of evaluation for each experiment, which results in the error bars seen in all tables throughout this work.
Evaluation with IoU metric. To further demonstrate the superior performance of TIMEX and the soundness of our evaluation setup, we additionally use an Intersection over Union (IoU) metric to measure the quality of explanations produced by baseline explainers compared to that of TIMEX when compared to ground-truth explanations. We calculate the IoU score on the SeqComb-UV dataset, as shown in Table 10. For comparison to our metrics, we include the AUPRC results for the same methods. The IoU metric is highly correlated with the AUPRC metric, with each metric resulting in the same ranking of methods and TIMEX achieving the highest metric.
C.5 Visualization of explanations
Figure 6 shows an example of TIMEX explainer versus IG and Dynamask. Shown is the SeqComb-UV dataset, which has increasing and decreasing subsequences that determine the class label. Each explainer identifies the regions driving the prediction. IG identifies very sparse portions of the predictive region, choosing only one point out of the sequences for the explanation; this is not reasonable when scaling to large and noisier datasets where the signal might not be as clear. Dynamask seems to miss some important subsequences, identifying one or two subsequences. In contrast, TIMEX identifies a larger portion of the important subsequences due to the connection loss in Equation 2. This property becomes crucial when scaling to time series datasets with more noise as it becomes more difficult to intuitively deduce the causal signal through visual inspection.
C.6 Further ablation experiments
We present a more in-depth study of ablations on TIMEX on three datasets: FreqShapes (univariate), SeqComb-MV (multivariate), and ECG (real-world). This is an extension to the ablations on the ECG dataset in Section 6, R1 in Table 3.
Ablation 1: No STE. We now conduct an experiment examining the effectiveness of using the STE for training TIMEX. Table 11 shows the results of this ablation experiment. Using the STE provides over a  $17\%$  increase in AUPRC for attribution identification for every dataset. Furthermore, AUR is better when using an STE for every dataset, but the AUP is better for SeqComb-MV without the STE than with the STE. Using the STE also shows benefits in both the univariate (FreqShapes, ECG) and multivariate (SeqComb-MV) settings. In conclusion, the STE provides a noticeable benefit over a continuous masking approach, giving empirical evidence for the claims made in Section 4.2.
Ablation 2: SimCLR vs. MBC. We now test a classical SimCLR [S5] contrastive learning loss against our proposed model behavior consistency (MBC). The SimCLR objective is designed to decrease the distance between explanation embeddings and embeddings in the reference model's latent space. We do not perform data augmentations as in the original SimCLR work. The SimCLR loss that we use is given as:
Figure 6: Visualization of explanations on SeqComb-UV dataset. Each column corresponds to a unique sample. All are of Class 3, which consists of one increasing subsequence and one decreasing subsequence. The methods that generate each figure are shown for each of the rows, while ground truth explanations are provided in the top row.
$$
\mathcal {L} _ {\text {S i m C L R}} \left(Z, Z ^ {E}\right) = \frac {1}{N} \sum_ {\mathbf {z} _ {i} \in Z, \mathbf {z} _ {i} ^ {E} \in Z ^ {E}} - \log \frac {\exp \left(D \left(\mathbf {z} _ {i} , \mathbf {z} _ {i} ^ {E}\right)\right)}{\sum_ {j \neq i} \exp \left(D \left(\mathbf {z} _ {j} , \mathbf {z} _ {i} ^ {E}\right)\right)} \tag {7}
$$
For each SimCLR trial, we fixed the number of sampled negatives at 32 and kept all other parameters equal. In addition, an early stopping strategy was performed where the stopping value was based on cosine similarity between explanation embeddings and reference sample embeddings (higher similarity is better).
SimCLR loss provides a valuable objective for training TIMEX relative to baseline explainers, but MBC optimization produces more robust explanations. SimCLR delivers a slightly better AUPRC for ECG, but its AUPRC values are below that of MBC for FreqShapes and SeqComb-MV. SimCLR loss yields explanations with consistently lower AUP; AUP is closest for SeqComb-MV with only
Method	AUPRC	SimCLR	AUPRC	MBC
AUP	AUR	AUP	AUR
FreqShapes	0.7014±0.0046	0.5991±0.5915	0.5915±0.0027	0.8324±0.0034	0.7219±0.0031	0.6381±0.0022
SeqComb-MV	0.6645±0.0019	0.8148±0.0009	0.3777±0.0017	0.6878±0.0021	0.8326±0.0008	0.3872±0.0015
ECG	0.4767±0.0021	0.4895±0.0024	0.4779±0.0013	0.4721±0.0018	0.5663±0.0025	0.4457±0.0018
Table 12: Ablation 2: Ablation considering SimCLR objective for training TIMEX versus an MBC objective as outlined in the main text.
Dataset	Ablation	AUPRC	AUP	AUR
FreqShapes	MBC only	0.2316±0.0020	0.1533±0.0015	0.4763±0.0022
LC only	0.2629±0.0022	0.1850±0.0016	0.5893±0.0018
MBC + LC	0.8324±0.0034	0.7219±0.0031	0.6381±0.0022
SeqComb-MV	MBC only	0.0761±0.0008	0.0576±0.0006	0.4996±0.0019
LC only	0.0788±0.0009	0.0570±0.0006	0.5294±0.0034
MBC + LC	0.6878±0.0021	0.8326±0.0008	0.3872±0.0015
ECG	MBC only	0.1615±0.0006	0.1348±0.0006	0.5504±0.0011
LC only	0.3704±0.0018	0.3296±0.0019	0.5084±0.0008
MBC + LC	0.4721±0.0018	0.5663±0.0025	0.4457±0.0018
Table 13: Ablation 3: Effects of model behavior consistency (MBC) and label consistency (LC) losses on explanation performance.
a  $3.4\%$  drop from MBC, but it is at a  $17.0\%$  decline for FreqShapes and a  $13.6\%$  drop for ECG. It is important to note that in addition to increased performance, MBC loss is more computationally efficient than SimCLR loss, avoiding inference on negative samples.
Ablation 3: Effect of MBC and LC losses. We now examine the effectiveness of using both model behavior consistency (Eq. 3) (MBC) and label consistency (Eq. 4) (LC) losses. Table 13 shows that using LC and MBC in combination is always better than using either alone. In isolation, LC performs better than MBC, which is expected given its (obviously) higher correlation with the classification predictions than MBC, which relies on an earlier layer's embedding space. Using both losses results in a powerful explainer that achieves over  $27.5\%$  higher AUPRC than MBC or LC alone. MBC and LC work together to capture rich information about the model's behavior, allowing TIMEX to be a state-of-the-art explainer.
To justify these results, we recall an argument presented in Section 4.3, where we justify MBC. We remark that perturbation-based methods have a similar idea to TIMEX: find some sparse perturbation to the input that can preserve the output state of the model. This is often done by observing the predicted label from the model under an applied mask or perturbation, e.g., in one of our baselines, Dynamask. A perturbation that preserves the output state is said to be "faithful" to the model because it is assumed that the model is invariant to the perturbation. In a sense, MBC generalizes this idea to latent spaces, ensuring that invariances are preserved in the latent space of the model as well as the prediction space.
Beyond the introduction of MBC alone, another core contribution of our work focuses on optimizing faithfulness to predictor models on multiple levels. We use multiple hidden or output states of the model, e.g., a latent and logit space, on which the explainable surrogate should match the reference predictor. The hypothesis behind this innovation is that model behavior (the exact objective we are trying to explain) cannot be fully captured by one state, e.g., a predicted label, but rather by multiple states throughout the model. A similar assumption is made in knowledge distillation, where methods often optimize student models to match the teacher in various network layers. Therefore, MBC and LA together enforce adherence to model behavior on two fronts: the latent space and prediction space, respectively. This explains the observed behavior: MBC and LA perform poorly alone, but together, these two losses provide a powerful objective to train the model.
C.7 Implementation and computing resources
Implementation. We implemented all methods in this study using Python 3.8+ and PyTorch 2.0. In our experiments, we employed the Vanilla Transformer [88] as the classification model for all methods. We verified that the classification models achieved satisfactory performance on the testing set to ensure that the models are strong predictors, which was previously pointed out by Faber et al.
[S18] as necessary for explainability evaluation. Complete classification results are in Table 15. We followed the hyperparameters recommended by the respective authors for all baseline methods.
Computational resources. For computational resources, we use a GPU cluster with various GPUs, ranging from 32GB Tesla V100s GPU to 48GB RTX8000 GPU. TIMEX and all models were only trained on a single GPU at any given time. The average experiment runtime in this work was around 5 minutes per fold, with ECG taking the longest at approximately 13 minutes per fold when training TIMEX to convergence.
PAM	Epilepsy
Training	Inference	Training	Inference
Dynamask	N/A	102	N/A	358
WinIT	1810	5730	93.3	84.7
TIMEX	580	1.04	247	1.07
Table 14: Runtime results for Dynamask, WinIT, and TIMEX on PAM and Epilepsy datasets. Time is shown in seconds. Note that Dynamask requires no training, thus a "N/A" designation.
Runtime experiment. We conducted a runtime experiment to understand how TIMEX compares to baseline explainers. Table X shows the training and inference time in seconds of TIMEX versus two state-of-the-art time series-specific baselines, Dynamask and WinIT. We chose two real-world time series datasets, PAM and Epilepsy, which are of varying sizes. PAM contains 4266 training samples and 534 testing samples, each of 600 time steps in length. Epilepsy contains 8280 training samples and 2300 testing samples, each of 178 time steps in length. Table 14 shows the time in seconds needed to train each explainer and to perform inference on the testing set.
TIMEX is the most efficient model at inference time for both datasets. This result is expected, as Dynamask and WinIT both require iterative procedures for each sample at inference time, while TIMEX requires only a forward pass at inference. Combining training and inference time, TIMEX is the second fastest on both datasets. However, WinIT and Dynamask times vary greatly between each dataset, with Dynamask being the fastest on PAM and WinIT being the fastest on Epilepsy. WinIT scales poorly to samples with many time steps, while Dynamask scales poorly to large testing sets. TIMEX strikes a compromise between these extremes, scaling better than Dynamask to larger testing sets while scaling better than WinIT to longer time series.
C.8 Flexible use of TIMEX with different time series architectures
We now study the ability of TIMEX to work with different underlying time series architectures. This means that of the original architecture,  $G$  and  $G^{E}$  are now an alternative architecture, while  $H^{E}$  remains as described in Section 4.1. Since experiments in the main text are based on transformer architectures, we now use a convolutional neural network (CNN) and long-short term memory (LSTM) network as the underlying predictors with the following hyperparameters:
- LSTM: 3 layer bidirectional LSTM + MLP on mean of last hidden states
CNN: 3 layer CNN + MLP on meanpool
Tables 18 19 show the results of TIMEX against strong baselines with a CNN predictor. TIMEX retains the state-of-the-art prediction observed for the transformer-based architecture, achieving
Dataset	F1	AUPRC	AUROC
FreqShapes	0.9716±0.0034	0.9940±0.0008	0.9980±0.0003
SeqComb-UV	0.9415±0.0052	0.9798±0.0028	0.9921±0.0011
SeqComb-MV	0.9765±0.0024	0.9971±0.0005	0.9990±0.0001
LowVar	0.9748±0.0056	0.9967±0.0013	0.9988±0.0005
Boiler	0.8345±0.0089	0.8344±0.0071	0.8865±0.0159
ECG	0.9154±0.0134	0.9341±0.0169	0.9587±0.0111
Epilepsy	0.9201±0.0079	0.9246±0.0130	0.9391±0.0157
PAM	0.8845±0.0051	0.9251±0.0029	0.9786±0.0009
Table 15: Classification (i.e., predictive) performance achieved by transformer time series models on datasets used in this study. These models are considered time series predictors throughout experiments in this study.
Method	AUPRC	FreqShapes	AUPRC	SeqComb-MV
AUP	AUR	AUP	AUR
IG	0.9282±0.0016	0.7775±0.0010	0.6926±0.0017	0.2369±0.0020	0.5150±0.0048	0.3211±0.0032
Dynamask	0.2290±0.0012	0.3422±0.0037	0.5170±0.0013	0.2836±0.0021	0.6369±0.0047	0.1816±0.0015
WinIT	0.4171±0.0016	0.5106±0.0026	0.3909±0.0017	0.3515±0.0014	0.6547±0.0026	0.3423±0.0021
Ours	0.9974±0.0002	0.7964±0.0009	0.8313±0.0011	0.1298±0.0017	0.1307±0.0022	0.4751±0.0015
Table 16: Explainer results with LSTM predictor on FreqShapes and SeqComb-MV synthetic datasets.
Method	AUPRC	ECG AUP	AUR
IG	0.5037±0.0018	0.6129±0.0026	0.4026±0.0015
Dynamask	0.3730±0.0012	0.6299±0.0030	0.1102±0.0007
WinIT	0.3628±0.0013	0.3805±0.0022	0.4055±0.0009
Ours	0.6057±0.0018	0.6416±0.0024	0.4436±0.0017
the best AUPRC on SeqComb-MV and ECG datasets. However, the performance for FreqShapes saturates at very high values for both TIMEX and IG, making the comparison more difficult for AUPRC. Tables 16,17 show the results of TIMEX against strong baselines with an LSTM predictor. TIMEX performs very well for both FreqShapes and ECG datasets, achieving the highest AUPRC, AUP, and AUR for both datasets. For SeqComb-MV, TIMEX did not converge. However, no explainer performed well for this task, achieving lower results than for the transformer and CNN predictors.
Visualization under different architectures. We add a visualization on the FreqShapes dataset with various explainers on different underlying architectures. Figure 7 shows this visualization. Explanations are similar across models. IG outputs appear similar, but TIMEX has a higher recall for essential patterns, which is reflected in quantitative results.
C.9 TIMEX on diverse tasks and datasets
We experiment with TIMEX on various tasks and types of datasets to demonstrate the generality of the method.
Irregular time series. We experiment with TIMEX on irregular time series classification. We introduce an irregularly sampled version of SeqComb-MV, randomly dropping an average of  $15\%$  of time steps. This results in variable-length time series, addressing both W1 and W2. We then follow Zhang et al. [S23], using a time series transformer with an irregular attention mask. We train only with direct-value masking to avoid direct interference with this mechanism.
Table 20 shows the results of this experiment. We compare TIMEX to Integrated Gradients (IG) performance because, given the nuance of learning from irregularly-sampled datasets [S23], most
Table 17: Explainer results with LSTM predictor on ECG dataset.
Method	AUPRC	FreqShapes	AUPRC	SeqComb-MV
AUP	AUR	AUP	AUR
IG	0.9955±0.0005	0.8754±0.0008	0.7240±0.0015	0.5979±0.0027	0.8858±0.0014	0.2294±0.0013
Dynamask	0.2574±0.0008	0.4432±0.0032	0.5257±0.0015	0.4550±0.0016	0.7308±0.0025	0.3135±0.0019
WinIT	0.5321±0.0018	0.6020±0.0025	0.3966±0.0017	0.5334±0.0011	0.8324±0.0020	0.2259±0.0020
Ours	0.9941±0.0002	0.6915±0.0010	0.8522±0.0009	0.7016±0.0019	0.7670±0.0012	0.4689±0.0016
Table 18: Explainer results with CNN predictor on FreqShapes and SeqComb-MV synthetic datasets.
Method	AUPRC	ECG AUP	AUR
IG	0.4949±0.0010	0.5374±0.0012	0.5306±0.0010
Dynamask	0.4598±0.0010	0.7216±0.0027	0.1314±0.0008
WinIT	0.3963±0.0011	0.3292±0.0020	0.3518±0.0012
Ours	0.7844±0.0014	0.8706±0.0012	0.3972±0.0010
Table 19: Explainer results with CNN predictor on ECG dataset.
Figure 7: Visualization of FreqShapes dataset of TIMEX explainer (top row), integrated gradients (IG), and Dynamask explainers on the FreqShapes dataset with transformer, CNN, and LSTM architectures.
AUPRC	AUP	AUR
IG	0.286±0.004	0.617±0.009	0.372±0.009
TIMEX	0.434±0.007	0.666±0.010	0.196±0.003
Table 20: Performance of TIMEX and IG on a modified version of the SeqComb-MV dataset such that the samples are irregularly sampled.
bases do not apply to irregularly-sampled time series datasets without significant changes that are out-of-scope for our work. TIMEX demonstrates superior performance in this setting, outperforming IG by an over  $1.5\mathrm{x}$  improvement in AUPRC.
Figure 8: Forecasting experiment on the ETTh1 dataset with TIMEX explanations.
Forecasting. We demonstrate TIMEX's generalizability to diverse tasks by explaining a forecasting model. We use the ETTh1 dataset [S24] and a vanilla forecasting transformer. To modify TIMEX for forecasting, we first extract embeddings used for MBC by max pooling over hidden states of the decoder. Next, we used a revised LC loss that used MSE as the distance function between predictions rather than JS divergence. We show visualizations of two samples in Figure 8. Explanations are in the left column, while forecasted time steps are in the right column. A few patterns emerge:
1. TIMEX identifies late time steps as crucial for the forecast, an expected result for a dataset with small temporal variance.
2. (a): the forecast is an increasing sequence, and TIMEX identifies a region of an expanding sequence (time 450-485). This suggests the model uses this increasing sequence as a reference for forecasting.
3. (b): a sharp upward spike is forecasted at the beginning of the predicted window. Correspondingly, TIMEX identifies a local minimum around time 260 and a sharp upward spike around time 475-490.
This preliminary experiment demonstrates that TIMEX shows the potential to extract meaningful explanations from forecasting datasets.
C.10 Quantitative analysis of landmark explanations
Figure 9: Distributions of DTW distance between structures of masked-in portions of explanations.
To examine landmark quality on the ECG dataset, we compare two groups: 1) the landmark group, with the five nearest-neighbors around the top 10 filtered landmarks, and 2) the random group, with five random explanations. We then compare the structure of the most salient values in the samples. We mask in the top-10 time steps as identified by TIMEX, then compute DTW [S25] distance between samples in each group. We then plot the distribution of average within-group DTW distances in Figure 9. A lower average DTW within their groups demonstrates the high quality of learned landmarks.
Clustering method	NMI	ARI
Landmark	0.191±0.014	0.152±0.026
K-means	0.147±0.010	0.025±0.001
Random	0.109±0.030	0.027±0.042
Table 21: Quantitative experiment comparing learned landmark explanations to K-means and Random clusters in the explanation latent space.
We have run an additional experiment comparing the clustering performance of our landmarks to those learned by k-means on the final latent space. On the ECG dataset, we match embeddings to their nearest landmarks, to their nearest k-means cluster centroids, and to randomly-selected points as a baseline. For each method, we use 50 clusters. We then evaluate each clustering method's Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI), which are standard clustering metrics, against the ground-truth labels and report the standard error  $(+/-)$  computed over 5-fold cross-validation. Higher metrics for one set of centroids would indicate that proximity is more meaningful for prediction. The results are shown in Table 21. Here, we see higher NMI and ARI for landmarks, which means the landmarks are a better heuristic for forming clusters than the K-means centroids for this task.
[S18] Lukas Faber, Amin K. Moghaddam, and Roger Wattenhofer. When comparing to ground truth is wrong: On evaluating gnn explanation methods. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 332-341, 2021.
[S19] Borys Surawicz, Rory Childers, Barbara J Deal, and Leonard S Gettes. Aha/accf/hrs recommendations for the standardization and interpretation of the electrocardiogram: part iii: intraventricular conduction disturbances: a scientific statement from the american heart association electrocardiography and arrhythmias committee, council on clinical cardiology; the american college of cardiology foundation; and the heart rhythm society: endorsed by the international society for computerized electrocardiology. Circulation, 119(10):e235–e240, 2009.
[S20] Mariana Florida, Alexandra Noela Parteni, Ioana Alexandra Neagu, Radu Andy Sascau, Cristian Statescu, and Daniela Maria Tanase. Incomplete right bundle branch block: Challenges in electrocardiogram diagnosis. Anatolian Journal of Cardiology, 25(6):380, 2021.
[S21] Chirag Agarwal, Eshika Saxena, Satyapriya Krishna, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. arXiv preprint arXiv:2206.11104, 2022.
[S22] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020.
[S23] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, ICLR, 2022.
[S24] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[S25] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69-84, 2007.