SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Consistency between explanations: This notion has been introduced in previous works in explainability literature. Pillai et al. [S9] train a saliency explainer via contrastive learning that preserves consistency across the saliency maps for augmented versions of images. A few other works have explored maintaining consistency of explanations across various perturbations and augmentations, specifically in computer vision [S10, S11]. In one of the only previous works to consider explanation consistency in time series, Watson et al. [S12] train an explainer on an ensemble of classifiers to optimize the consistency of explanations generated by an explainer applied to each classifier. TIMEX does not seek to optimize consistency between explanations but rather a consistency to the predictor model on which it is explaining.
- Consistency as an explainability metric: Dasgupta et al. [S13] defines explanation consistency as similar explanations corresponding to similar predictions; this metric is then used as a proxy to faithfulness to evaluate the quality of explainers. However, Dasgupta et al. use consistency to evaluate explainers, not to train and design a new explainer method. TIMEX uses consistency as a learning objective rather than simply a metric.
Our work differs from these previous formulations of explanation consistency. We seek to optimize the consistency not between explanations directly, as mentioned in previous works, but rather between the explainer and the model it is tasked with explaining. MBC attempts to ensure that the behavior of the explainable surrogate matches that of the original model. The definition of consistency in Dasgupta et al. is the closest to our definition of MBC; however, Dasgupta et al. seek not to optimize the consistency of explainers but rather to evaluate the output of post-hoc explainers. TIMEX directly optimizes the consistency between the surrogate model and the original predictor through the MBC loss, a novel formulation that seeks to increase the faithfulness of explanations generated by TIMEX.
Appendix B Further theoretical discussions
B.1 Differentiable attention masking
As is described in Section 4.1, we use differentiable attention masking [77], which is defined as such:
$$
\alpha^ {m} = \left(\operatorname {s o f t m a x} \left(\frac {\mathbf {Q K} ^ {T}}{\sqrt {d _ {k}}}\right) \odot \mathbf {M} _ {\mathcal {X}}\right) \mathbf {V}, \tag {6}
$$
where  $\mathbf{Q},\mathbf{K},\mathbf{V}$  represent query, key and values operators,  $d_{k}$  is used as a normalization factor, and  $\mathbf{M}_{\mathcal{X}}$  is a mask with self-attention values. This procedure is fully differentiable, and given that  $\mathbf{M}_{\mathcal{X}}$  is binarized via the STE, it sets all attention values to zero that are to be ignored based on output from  $H^{E}$ .
B.2 Further discussion on the utility of model behavior consistency
The model behavior consistency (MBC) framework in TIMEX is a method to train an interpretable surrogate model  $G^{E}$ . In Section 4.3, we discuss the conceptual advances of this approach. Here, we will outline another advantage of the approach—preserving classification performance—and a brief discussion on the broader uses of MBC in other domains and applications.
Training a model with an interpretability bottleneck such as TIMEX is often challenging, as the inherent interpretability mechanism can hinder the performance and expressiveness of the method; this is an advantage of post-hoc methods. MBC allows one to preserve the performance of the underlying predictor. TIMEX, a surrogate method, allows one to keep the predictions from a pretrained time series encoder and develop explanations on top of it, which is practical for real-world use when a drop in classification performance is highly undesirable.
MBC is not limited to time series classification tasks. We demonstrate the utility of MBC for time series due to the particularly challenging nature of the data modality and the lack of available time series explainers. However, MBC gives a general framework for learning interpretable surrogate models through learning the  $H^{E}$  and  $G^{E}$  modules. MBC also has the potential to be applied to tasks outside of classification; since MBC is defined on the embedding space, any model with such an embedding space could be matched through a surrogate model as in TIMEX. This opens the possibility of learning on general pretrained models or even more complex tasks such as forecasting (as shown in Appendix C.9). Finally, we see MBC as having potential beyond explainability as well; one could imagine MBC being a way to distill knowledge into smaller models [S14, S15, S16]. We leave these discussions and experiments for future work.
B.3 Explanation landmark selection strategy
We proceed to describe how landmarks are selected for final interpretation. As described in Section 4.4, landmarks are initialized with the embeddings given by  $G$  for a random number of training samples. Practically, we stratify this selection across classes in the training set. Landmarks are then updated during the learning procedure. After learning landmarks, not every landmark will be helpful as an explanation; thus, we perform a filtration procedure. Intuitively, this filtration consists of detecting landmarks for which the landmark is the nearest landmark neighbor for many samples. This procedure is described in Algorithm 1.
Algorithm 1: Landmark filtration
Input: Landmark matrix  $\mathbf{L}\in \mathbb{R}^{n_L\times d_z}$  ; training explanation embeddings  $\{\mathbf{z}_1^E,\dots,\mathbf{z}_N^E\}$  for  $\mathbf{z}_i^E\in \mathbb{R}^{d_z}$  ; threshold number of neighbors  $n_{\epsilon}\in \mathbb{N}$ $N_{l}\gets \{\}$    
for  $i\gets 1$  to  $N$  do Compute similarity to all landmarks  $S_{i}^{l} = \mathrm{sim}(\mathbf{z}_{i}^{E},\mathbf{L})$ $j_{\mathrm{max}}\gets \mathrm{argmax}_jS_i^l [j]$  (Gets nearest landmark for sample explanation i)   
Append  $j_{\mathrm{max}}$  to  $N_{l}$    
end   
 $F_{l}\gets$  Frequency of occurrence of each unique element in  $N_{l}$ $\mathbf{L}_{\text{filter}}\gets$  every landmark in  $\mathbf{L}$  s.t.  $F_{l}\geq n_{\epsilon}$    
Return  $\mathbf{L}_{\text{filter}}$
Appendix C Additional experiments and experimental details
C.1 Description of datasets
We conduct experiments using both synthetic and real-world datasets. This section describes each synthetic and real-world dataset, including how ground-truth explanations are generated when applicable.
C.1.1 Synthetic datasets
We employ synthetic datasets with known ground-truth explanations to study the capability to identify the underlying predictive signal. We follow standard practices for designing synthetic datasets, including tasks that are predictive and not susceptible to shortcut learning [S17] induced by logical shortcuts. These principles are defined in [S18] concerning graphs, but we extend these to synthetic datasets for time series. Each time series is initialized with a non-autoregressive moving average (NARMA) noise base, and then the described patterns are inserted. We will briefly describe the construction of each time series dataset in this section, and the codebase contains full details at https://github.com/mims-harvard/TimeX. We designed four synthetic datasets to test different time series dynamics:
FreqShapes. Predictive signal is determined by the frequency of occurrence of an anomaly signal. To construct the dataset, take two upward and downward spike shapes and two frequencies, 10 and 17 time steps. There are four classes, each with a different combination of the attributes: class 0 has a downward spike occurring every 10-time steps, class 1 has an upward spike occurring every 10-time steps, class 2 has a downward spike occurring every 17-time steps, and class 3 has an upward spike occurring every 17-time steps. Ground-truth explanations are the locations of the upward and downward spikes.
SeqComb-UV. Predictive signal is defined by the presence of two shapes of subsequences: increasing (I) and decreasing (D) trends. First, two subsequence regions are chosen within the time series so neither subsequence overlaps; each subsequence is 10-20 time steps long. Then, a pattern is inserted based on the class identity; the increasing or decreasing trend is created with a sinusoidal noise with a randomly-chosen wavelength. Class 0 is null, following a strategy in [S18] that recommends using null classes for simple logical identification tasks in synthetic datasets. Class 1 is I, I; class 2 is D, D; and class 3 is I, D. Thus, the model is tasked with identifying both subsequences to classify each sample. Ground-truth explanations are the I and D sequences determining class labels.
SeqComb-MV. This dataset is a multivariate version of SeqComb-UV. The construction and class structure are equivalent, but the I and D subsequences are distributed across different sensors in the input. Upon constructing the samples, the subsequences are chosen to be on random sensors throughout the input. Ground-truth explanations are given as the predictive subsequences on their respective sensors, i.e., the explainer is required to identify the time points at which the causal signal occurs and the sensors upon which they occur.
LowVar. Predictive signal is defined by regions of low variance over time that occur in a multivariate time series sample. Similar to SeqComb datasets, we choose a random subsequence in the input and, in that subsequence, replace the NARMA background sequence with Gaussian noise at a low variance. The subsequence is further discriminated by the mean of the Gaussian noise and the sensor on which the low variance sequence occurs. For class 0, the subsequence is at mean -1.5 on sensor 0; for class 1, the subsequence is at mean 1.5 on sensor 0; for class 2, the subsequence is at mean -1.5 on sensor 1; for class 3, the subsequence is at mean 1.5 on sensor 1. This task is distinctly different from other synthetic datasets, requiring recognition of a subsequence that is not anomalous from the rest of the sequence. This presents a more challenging explanation task; a simple change-point detection algorithm could not determine the explanation for this dataset.
We create 5,000 training samples, 1,000 testing samples, and 100 validation samples for each dataset. A summary of the dimensions of each dataset can be found in Table 4.
Table 4: Synthetic Dataset Description
Dataset	# of Samples	Length	Dimension	Classes
FreqShapes	6,100	50	1	4
SeqComb-UV	6,100	200	1	4
SeqComb-MV	6,100	200	4	4
LowVarDetect	6,100	200	2	4
C.1.2 Real-world datasets
We employ four datasets from real-world time series classification tasks: PAM [82] - human activity recognition; ECG [81] - ECG arrhythmia detection; Epilepsy [83] - EEG seizure detection; and Boiler [84] - automatic fault detection.
PAM [82]. It measures the daily living activities of 9 subjects with three inertial measurement units. We excluded the ninth subject due to the short length of sensor readouts. We segment the continuous signals into samples with a time window of 600 and the overlapping rate of  $50\%$ . PAM initially has 18 activities of daily life. We exclude the ones associated with fewer than 500 samples, leaving us with eight activities. After modification, the PAM dataset contains 5,333 segments (samples) of sensory signals. Each sample is measured by 17 sensors and contains 600 continuous observations with a sampling frequency of  $100\mathrm{Hz}$ . PAM is labeled into eight classes, where each class represents an activity of daily living. PAM does not include static attributes, and the samples are approximately balanced across all eight classes.
MIT-BIH (ECG) [81]. The MIT-BIH dataset has ECG recordings from 47 subjects recorded at the sampling rate of  $360\mathrm{Hz}$ . The raw dataset was then window-sliced into 92511 samples of 360 timestamps each. Two cardiologists have labeled each beat independently. Of the available annotations, we choose to use three for classification: normal reading (N), left bundle branch block beat (L), and right bundle branch block beat (R). We choose these because L and R diagnoses are known to rely on the QRS interval [S19, S20], which will then become our ground-truth explanation (see Section C.4). The Arrhythmia classification problem involves classifying each fragment of ECG recordings into different beat categories.
Epilepsy [83]. The dataset contains single-channel EEG measurements from 500 subjects. For every subject, the brain activity was recorded for 23.6 seconds. The dataset was then divided and shuffled (to mitigate sample-subject association) into 11,500 samples of 1 second each, sampled at  $178\mathrm{Hz}$ . The raw dataset features five classification labels corresponding to different states of subjects or measurement locations — eyes open, eyes closed, EEG measured in the healthy brain region, EEG measured in the tumor region, and whether the subject has a seizure episode. To emphasize the distinction between positive and negative samples, we merge the first four classes into one, and each time series sample has a binary label indicating whether an individual is experiencing a seizure. There are 11,500 EEG samples in total.
Boiler [84]. This dataset consists of simulations of hot water heating boilers undergoing different mechanical faults. Various mechanical sensors are recorded over time to derive a time series dataset. The learning task is to detect the mechanical fault of the blowdown valve of each boiler. The dataset is particularly challenging because it includes a large dimension-to-length ratio, unlike the other datasets, which contain many more time steps than sensors (Table 5).
Table 5: Real-World Dataset Description
Dataset	# of Samples	Length	Dimension	Classes	Task
PAM	5,333	600	17	8	Action recognition
MIT-BIH	92,511	360	1	5	ECG classification
Epilepsy	11,500	178	1	2	EEG classification
Boiler	160,719	36	20	2	Mechanical fault detection
C.2 Descriptions of baseline methods
We now describe each baseline method in further detail.
IG [31]. Integrated gradients is a classical attribution method that utilizes the gradients of the model to form an explanation. The method compares the gradients to a baseline value and performs Riemannian integration to derive the explanation. Integrated gradients is a popular data-type agnostic interpretability method [S21], but it has no inductive biases specific for time series. We use the Captum [S22] implementation of this method, including default hyperparameters such as the baseline value.
Dynamask [58]. This explainer is built specifically for time series and uses a perturbation-based procedure to generate explanations. The method performs iterative occlusion of various input portions, learning a mask that deforms the input time series towards a carefully-determined baseline value.
This method is different from TIMEX in a few key ways. First, it performs continuous masking; TIMEX performs discrete masking through STEs. Second, it measures perturbation impact on the original model  $F(G(\cdot))$ ; TIMEX trains a surrogate model  $G^{E}$  to learn the explanations and measure the impact of masking the input. Third, Dynamask learns the explanations iteratively for each sample; TIMEX trains the surrogate, which can then output explanations in one forward pass of  $H^{E}$ .
WinIT [86]. This explainer is a feature removal explainer, similar to Dynamask. WinIT measures the impact of removing features from a time series on the final prediction value. It eliminates the impact of specific time intervals and learns feature dependencies across time steps. WinIT uses a generative model to perform in-distribution replacement of masked-out features. WinIT improves on a previous time series explainer, FIT [63], which is a popular baseline in time series explainability literature but is excluded in our work because WinIT is more recent and improves on FIT both conceptually and empirically.
CoRTX [87]. Contrastive real-time explainer (CoRTX) is an explainer method that utilizes contrastive learning to approximate SHAP [3] values. This method is developed for computer vision, but we implement a custom version that works with time series encoders and explanation generators. We include this method because it uses self-supervised learning to learn explanations. TIMEX also uses a self-supervised objective to learn explanations, but our method differs from CoRTX in several ways. First, CoRTX performs augmentation-based contrastive learning while we use MBC, which avoids the definition of negatives or the careful choice of augmentations specific to the data modality. Second, CoRTX fundamentally attempts to approximate SHAP values via a small number of SHAP explanations. In contrast, TIMEX includes a masking system that can produce masks without fine-tuning a model on a set of explanations derived from an external method. CorRTX parallels ours in using self-supervised learning but is fundamentally different from TIMEX.
$\mathbf{SGT} + \mathbf{Grad}$  [17]. Saliency-guided training (SGT), an in-hoc explainer, is based on a modification to the training procedure. During training, features with low gradients are masked to steer the model to focus on more important regions for the prediction. The method is not an explainer alone but requires another post-hoc explainer to derive explanations. In our experiments, we consider saliency explanations, which the SGT authors recommend. The authors found that this method can improve performance on time series data. For this reason, we include it as one of our baselines to demonstrate the effectiveness of TIMEX against modern in-hoc explainers.
C.3 Hyperparameter selection
We list hyperparameters for each experiment performed in this work. For the ground-truth attribution experiments (Section 6, results R1), the hyperparameters are listed in Table 6. The hyperparameters used for the occlusion experiment (Section 6, results R2) with real-world datasets are in Table 7. We also list the architecture hyperparameters for the predictors trained on each dataset in Tables 8-9.
A few abbreviations are used for hyperparameters that are not mentioned in the main text. "Weight decay" refers to an L1 regularization on the model weights; the value for weight decay is equivalent to the weight on that term in the loss function compared to the rest of the loss terms (Section 4.4). "Scheduler?" refers to using a learning rate scheduler that decreases the learning rate by a factor of 10 if a plateau occurs. We use a scheduler that delays decreasing learning rates until after 20 epochs; not every experiment utilizes the scheduler as it is based on which choice yields lower validation loss upon convergence. "Distance norm." refers to a normalization of the distances in  $\mathcal{L}_{\mathrm{MBC}}$ ; the loss is divided by the variance of the distances on the  $Z$  embedding space.  $\tau$  is the temperature parameter used for the Gumbel-Softmax reparameterization [73], Section 4.1.  $d_h$  refers to the dimensionality of hidden layers in the transformer predictor. Finally, "Norm. embedding" refers to an architecture choice that normalizes  $Z$  when training the predictor; this is used to prevent a poor latent space when a collapse is observed via poor latent space geometry.
A few other notes on implementation and design of TIMEX: The architecture of  $H^{E}$  uses the same size of  $G^{E}$  and encoder for  $H^{E}$  as for the predictor on each task. The number of transformer decoder layers is fixed at 2. Please reference the codebase for more details on these hyperparameters and implementations https://github.com/mims-harvard/TimeX.
Choosing the  $r$  parameter. One of the most significant parameters in training TIMEX is  $r$ , the parameter which controls sparsity of the learned masks. We conduct an experiment where we vary the parameters and measure explanation quality. We use the SeqComb-UV dataset and hold all hyperparameters constant while varying the parameter. The result is visualized in Figure 5. Low
Figure 5: Experiment on SeqComb-UV dataset varying the  $r$  parameter.
Parameter	FreqShape	SeqComb-UV	SeqComb-MV	LowVarDetect	ECG
Learning rate	0.001	0.001	0.001	0.003	0.0005
Batch size	64	64	64	64	16
Weight decay	0.001	0.001	0.001	0.0001	0.0001
Scheduler?	Yes	Yes	No	No	No
Epochs	50	50	100	100	5
r	0.5	0.5	0.5	0.5	0.5
Distance norm.	No	No	No	Yes	No
λLC	1.0	1.0	1.0	1.0	1.0
λE	2.0	2.0	2.0	2.0	2.0
λcon	2.0	2.0	2.0	2.0	2.0
τ	1.0	1.0	1.0	1.0	1.0
nL	50	50	50	50	50
values lead to a drop in explainer performance with respect to AUPRC and AUP. Importantly, for values above 0.4, the explainer performance is stable, suggesting that is robust to choice of value. Thus, we recommend choosing a value near 0.5 for experiments, as we did throughout our experiments. If the pattern is hypothesized to be sparse, one can set the value lower, and vice versa if the pattern is hypothesized to be dense.
C.4 Evaluation details
Following [58], we use AUP and AUR to evaluate the goodness of identification of salient attributes as a binary classification task, which is defined in C.1:
Definition C.1 (AUP,AUR [58]). Let  $\mathbf{Q}$  be a matrix in  $\{0,1\}^{T\times d_X}$  whose elements indicate the true saliency of the inputs contained in  $\mathbf{x}\in \mathbb{R}^{T\times d_X}$ . By definition,  $Q_{t,i} = 1$  if the feature  $x_{t,i}$  is salient
Table 6: Training parameters for TIMEX across all ground-truth attribution experiments.
Parameter	Epilepsy	PAM	Boiler
Learning rate	0.0001	0.002	0.0001
Batch size	32	32	32
Weight decay	0.001	0.001	0.001
Scheduler?	Yes	No	Yes
Epochs	50	100	50
r	0.5	0.1	0.5
Distance norm.	No	Yes	No
λLC	1.0	1.0	1.0
λE	2.0	2.0	2.0
λcon	2.0	0.0	2.0
τ	1.0	1.0	1.0
nL	50	50	50
Table 7: Training parameters for TIMEX across all real-world datasets used for the occlusion experiments.
Parameter	FreqShape	SeqComb-UV	SeqComb-MV	LowVarDetect	ECG
Num. layers	1	2	2	1	1
dh	16	64	128	32	64
Dropout	0.1	0.25	0.25	0.25	0.1
Norm. embedding	No	No	No	Yes	Yes
Learning rate	0.001	0.001	5e-4	0.001	2e-3
Weight decay	0.1	0.01	0.001	0.01	0.001
Epochs	100	200	1000	120	500
Table 8: Training parameters for transformer predictors across all ground-truth attribution experiment datasets.
Param.	Epilepsy	PAM	Boiler
Num. layers	1	1	1
dh	16	72	32
Dropout	0.1	0.25	0.25
Norm. embedding	No	No	Yes
Learning rate	0.0001	0.001	0.001
Weight decay	0.001	0.01	0.001
Epochs	300	100	500
Table 9: Training parameters for TIMEX across all real-world datasets used for the occlusion experiments.
and 0 otherwise. Let  $M$  be a mask in  $\{0,1\}^{T\times d_X}$  obtained with a saliency method. Let  $\tau \in (0,1)$  be the detection threshold for  $M_{t,i}$  to indicate that the feature  $\mathbf{x}_{t,i}$  is salient. This allows to convert the mask into an estimator  $\hat{Q}_{t,i}(\tau)$  via:
$$
\hat {Q} _ {t, i} (\tau) = \left\{ \begin{array}{l l} 1 & \text {i f} M _ {t, i} \geq \tau \\ 0 & \text {e l s e .} \end{array} \right.
$$
By considering the sets of truly salient indexes and the set of indexes selected by the saliency method:
$$
A = \left\{\left(t, i\right) \in [ 1: T ] \times [ 1: d _ {X} ] \mid q _ {t, i} = 1 \right\}
$$
$$
\hat {A} (\tau) = \left\{\left(t, i\right) \in [ 1: T ] \times \left[ 1: d _ {X} \right] \mid \hat {q} _ {t, i} (\tau) = 1 \right\}.
$$
the precision and recall curves that map each threshold to a precision and recall score:
$$
\mathrm {P}: (0, 1) \longrightarrow [ 0, 1 ]: \tau \longmapsto \frac {| A \cap \hat {A} (\tau) |}{| \hat {A} (\tau) |}
$$
$$
\mathrm {R}: (0, 1) \longrightarrow [ 0, 1 ]: \tau \longmapsto \frac {| A \cap \hat {A} (\tau) |}{| A |}.
$$
The AUP and AUR scores are the area under these curves:
$$
\mathrm {A U P} = \int_ {0} ^ {1} \mathrm {P} (\tau) d \tau
$$
$$
\mathrm {A U R} = \int_ {0} ^ {1} \mathrm {R} (\tau) d \tau .
$$