STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data
Authors: Maximilian Forstenhäusler
Daniel Külzer
Christos Anagnostopoulos
Shameem A Puthiya Parambath
Natascha Weber
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
transformer
sequential modeling
contrastive learning
semi-supervised learning
time series
Primary area: deep_learning
TL;DR: STaRFormer combines dynamic regional masking with semi-supervised contrastive learning to enable robust predictions across diverse time series types and downstream tasks.

Abstract: Understanding user intent is essential for situational and context-aware decision-making. Motivated by a real-world scenario, this work addresses intent predictions of smart device users in the vicinity of vehicles by modeling sequential spatiotemporal data. However, in real-world scenarios, environmental factors and sensor limitations can result in non-stationary and irregularly sampled data, posing significant challenges. To address these issues, we propose STaRFormer, a Transformer-based approach that can serve as a universal framework for sequential modeling. STaRFormer utilizes a new dynamic attention-based regional masking scheme combined with a novel semi-supervised contrastive learning paradigm to enhance task-specific latent representations. Comprehensive experiments on 56 datasets varying in types (including non-stationary and irregularly sampled), tasks, domains, sequence lengths, training samples, and applications demonstrate the efficacy of STaRFormer, achieving notable improvements over state-of-the-art approaches.

STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data
Maximilian Forstenhäusler1.2. Daniel Külzer Christos Anagnostopoulos
Shameem Puthiya Parambath² Natascha Weber
BMW Group
Uniieeriiyy of Glasgow
Project Page: https://star- htrmr.tttr wiw.hgi.thh.....
Abstract
Understanding user intent is essential for situational and context-aware decision- making. Motivated by real-world scenario, this work addresses intent predictions of smart device users in the vicinity of vehicles by modeling sequential spatiotem- poral data. However, in real-world scenarios, environmental factors and sensor limitations result in non-stationary and irregularly sampled data, posing signifi- cant challenges. To address these issues, we propose STaRFormer, a Transformer- based approach that can STaRFormer utilizes a new dynamic attention based regional masking scheme combined with a novel semi-supervised contrastive learning paradigm to enhance task-specific latent representations. Comprehensive experiments on 56 datasets varying types (including non-statinn stationtt and irregularly sampled), tasks, domains, sequence lengths, training samples, and applications demonstrate the efficacy of Foamerrmer, achieving notable improvements over ttat-off-the-ar approaches.
universal framework for sequential modeling.
Introduction
Advancements in machine learning architectures, such as LSTM [1] and Transformer [2], have enhanced the ability model sequential data. However, these algorithms typically assume that the data is fully observed, stationary, and sampled at regular intervals [3]. In reality. sensor technology and external conditions often influence data collection, leading to non-stationary and irregularly sampled time series. For instance, in the automotive industry, manufacturers have recently integrated Ultra- Wideband (UWB) and Bluetooth Low-Energy (BLE) technologies to enhance the Digital Key (DK) [4-8]. This integration ensures precise and secure vehicle access along with applications for connected vehicles. Precise localization is achieved by performing time-of-flight calculations between each UWB anchor in a vehicle and smart device, leveraging UWB's ' 2ns pulse duration [9]. Nonetheless, measuring algorithm for UWB ranging may yield irregularly recorded time-of-flight calculations, resulting in irregularly sampled time series. Additionally, when recording real-world data using UWB-capable ranging devices, external factors such as signal interference and device positioning can introduce non-stationarity. These conditions may ultimately affecct the overall performance of Machine Learning (ML) algorithms, the real-world Digital Key Trajectories (DKT) dataset provided by the BMW Group (Appendix A and C.I.I), we confirmed, by (KPSS) and augmented Dickey-Fulle, (ADF) tests, that approximately 79% of the sequences are non-stationary. Based on the real- world trajectories generated from the DK, focus predicting the smart device user's s intent, formulated as a specific classification task.
Email: maximilian. aannaaa:aaaillllaaffffottaaaaaa
de, m.forstenhauuslrr 1Cresearch, aetresarch.glaagovv.ooo glasgov.ac.ut
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generally, trajectories involve variables such as latitude, longitude, altitude, and speed, which are often irregular. Similarly, weather conditions. geographical barriers, sensor availability, and device malfunctions [10] can result in non-stationary characteristics, aligning with the properties found in the DKT dataset. Although several solutions exist to address these issues, they require substantial prior knowledge and effort in model selection [11-20]. To address these challenges, we propose a versatile framework, STaRFormer, designed to effectively model time series with the aforementioned characteristics while maintaining applicability to standard time series data. STaRFormer proposes dynamic regional masking to manipulate key task-specific regions within an input sequence, introducing synthetic variations in statistical properties, such mean, variance, and sampling frequency. By incorporating this masking layer during the learning process of a downstream task, STaRFormer generates masked and unmasked latent representations of the same input sequence. Building on prior work, which highlighted that the task-specific importance of elements within sequence can vary in their influence on downstream tasks [21, 22], we extend this approach by coupling representation learning with a downstream objective. This coupling allows to incorporate context-specific information that may be overlooked in decoupled self-supervised frameworks [23-27]. Through a novel combination of self-supervised and supervised contrastive learning (CL), STaRFormer creates robust task-informed latent embeddings by maximizing agreement between class-wise and batch-wise similarities of the masked and unmasked latent representation. This technique is designed enhance the model's robustness to irregularities in time series while serving as an augmentation method to improve performance for various time series types and tasks. In summary, our main contributions are:
We propose STaRFormer, a highly effective and robust approach boosting the performance of downstream tasks for diverse types of time series and tasks.
We develop a novel semi-supervised CL approach for time series analysis, leveraging batch-wise and class-wist similarities by reconstructing latent representations from masked inputs.
We design a novel Dynamic Attention- Regional Masking (DAReM) scheme that identifies task-specific important regions a sequence, allowing to embed task-specific knowledge.
We assess STaRFormer using 56 public and non-public datasets to validate its effectiveness compared to state-of state-of-the-arr the-art methods, highlighting its versatility for various types of time series.
2 Related work
Regular time series modeling for classification. Time series modeling for classification seeks to analyze and identify patterns in sequential data collected over consistent time intervals, with the goal of assigning labels to entire sequences or per elements within the sequence. It is generally assumed that the sequential data is stationary and uniformly sampled. Common ML baselines include dimension- dependent dynamic time warping (DTWD) [28. 29] and WEASEL-MUSE [30]. Deep Learning (DL) has proven powerful for time series classification by automatically extracting complex features. Unlike traditional methods that rely on handcrafted features, DI. models such as RNN [3], L.TTM [1] and GRU [14] learn hierarchical representations directly from the data. However, these models often struggle with capturing long-term dependencies and spatiotemporal patterns. ROCKET [33] and MiniROCKET [34], CNNs that have been effective in capturing local dependencies, have achieved impressive results by learning features through diverse random convolutional kernels. Transformer-based approaches have recently gained attention due to their ability to capture long- range dependencies in sequential data. Various Transformer baasdd models have been proposed for forecasting, classification, and anomaly detection [3, 21, 23, 35 35-40J. Initial approaches utilized a full encoder-decoder Transformer architecture for univariate time series forecasting [41], while TST [23] generalized unsupervised representation learning for Transformers and time series, similarly to BERT's Masked Language Modeling (MLM) [42). TARNet (21) addresses the issue of decoupling unsupervised pretraining from downstream tasks using dynamic masking and reconstruction. We address the challenge of time series classification by pairing a novel semi-supervised CL approach with a proposed generalization of the dynamic masking approach from TARNet. In doing so, we extend the proposition of coupling representation learning while learning a downstream task.
Non-stationary and irregularly sampled time series modeling. Non-stationary time series mod- eling addresses the variability in statistical properties over time, i.e., changing means and covariances [43, 44]. Traditional models often fail to capture these dynamics. While most research has focused
2

---

on forecasting, some efforts have been directed towards -tatiionny time series classification. Recent advancements include adaptive RNNs [45, 46], normalization-bart based approaches [47, 35]. and non-stationary Transformers, which incorporate non-stationary factors to improve accuracy while addressing distribution shifts [48]. Irregularly sampled time series modeling addresses sequences with varying time intervals between observations. A standard solution is converting continuous time observations into fixed intervals [13, 15]. Several models have been proposed to capture dynamics between observations such GRU-D [16] and multi-directional RNN [49]. Attention-basee models, including Transformers, [2. 23] and ATTAIN [50], incorporate attention mechanisms to handle time inregularity. Raindrop [20] uses graph neural networks to model irregular time series as graphs. Meanwhile, TrajFormer [51] introduces a Transformer architecture that generates continuous point embeddings to deal with irregularities of trajectories. Recently, ViTST [3] focused on time series in the visual modality by transforming sequences into visualized line graphs, leveraging pretrained AisionTrransfommn backhones. To handle non-stationarity and sampling irregularity, introduce dynamic regional masking strategy that perturbs input sequences by modifying their statistical and sampling properties. Coupled with our CL scheme, this representation learning approach promotes robustness to distributional shifts and irregular sampling, enhancing the latent space rather than relying solely on input reconstruction.
Time series contrastive learning. CL has proven effective in extracting high-quality, discrimina- tive features [52]. CL operates self-supervised learning paradigm, learning representations by contrasting positive and negative pairs. The goal is to bring similar (positive) pairs closer and push dissimilar (negative) pairs apart, typically using contrastive losses like NT-Xent [52]. InfoNCE [53], or triplet loss [54]. For sequential data, self-supervised CL aims to extract invariant representations from augmented views of unlabeled data through carefully designed pretext tasks. Methods such as TCL [55], and TNC [56] use subsequence- neighborhood based sampling assuming distant segments as negative pairs and neighboring segments positive pairs. Info'T [26] emphasizes appropriate augmentation selection using meta-learning, and TS2 [24] learns contextual representations across semantic levels. CoST [25] uses model inductive biases to separate seasonal and trend patterns, introducing a frequency-domain contrastive loss. However, these methods often suffer from flawed augmentations, weak negative samples, and limited information use [27]. TimesURL [27] proposes self-supervised framework that combines CL, time reconstruction, and a frequency-temporal aug- mentation with hard negative sampling to learn universal time series representations for diverse downstream tasks. While prior work applies self-supervised CL to learn universal time series repre- sentations. we propose a task-coupled approach that jointly optimizes representation learning with the downstream objective, embedding task-specific information into the representations.
3 Approach
Former adopts a Siamese network architecture [57] consisting of two 'towers' of N encoder- only Transformer blocks, f. that share common set of model parameters. STaRFormer is illustrated in Fig. 1. Without loss of generality. we consider classification, anomaly detection and regression as downstream tasks. For sequence-level classification tasks, special token is utilized facilitate the downstream predictions. The other downstream tasks utilize appropriate variations, such as pooling operations or clement-wise predictions, facilitate the computation of the desired task predictions. For detailed information, output head formulations, and related remarks, see Appendix B.
Notation. Let ((S()) ()) ii= 11=1,.... 1,..., .) denote a time series dataset containing M samples. Each sequence, S(t) N has N elements and assigned to a label ( (1,.... C}.
where C' is the number of classes. Each data point in the sequence can have an associated timestamp.
Thus, the j-th data point in S(i) can be represented as s) (z) (1) (i) E B... Therefore, S(O)
N3 E R Nx2 x2 is formed by concatenating all N elements. For multivariate time series, the dimensionality is not fixed to two; thus S(t) E NxD, where N E Noo and D E N22. A mini-batch of size B, where B < M. is defined as X C DD where X E NxBDD
{s/
(8) |j 1....,,J
M where (()) E R can be multivariate, predict the class (0)) {L, ..., CJ. for each sequence ((O in D.
Problem 1- Classification. Given a dataset D= = {(S(1), (+) i= =
3

---

Tabk
Seperisdd
Eorwed
Fatoual
Nx
Adi Nove
Adi Nara
Appopitis vi kodwer
Alsridas
Alt sine
Mireie Swar lna)
bodda
Excedaa
Supanen
Figure I: Architecture of STaRFormer; (a) High level Siamese network architecture the left tower performs the downstream task while the right tower performs the reconstruction of the masked sequence. (b) The DAReM scheme exemplified size 16 for an encoder with = 4 layers. ReM abbreviates regional mask.
single batch from the DKT dataset with batch
{(S(), y() I I...., M) where 5( E NxD can be multivariate and (() IR., predict for each element sequence S in D whether UjEN (normal observation) or UjeN E (1,...,C] (anomalous observation).
Problem Anomaly defection. Given a dataset
Problem 3 Regression. Given dataset D {SS() y())) ,,..., M), where ( RN can be multivariate, predict the continuous target value y for each sequence S(() in D.
3.1
Semi-supervised task informed representation learning
This section presents STaRFormer's components facilitating task-informed representation learning.
3.1.1
Dynamic attention-based regional masking (DAReM)
Prior work has shown that the task-specifi importance of elements within sequence varies w.r.t. their impact on downstream tasks [21, 22]. STaRFormer adopts this characteristic by dynamically masking regions around the features the model deems important. These masks force the model to learn changes in statistical properties and irregular sampling induced by the masking. Our rationale is that reconstructing key sequential regions amplifies non-stationary and irregular sampling characteristics. This enables the model to generate more effective latent representations for the downstream task. This masking scheme, termed DAReM, can be seen generalization of the masking scheme proposed in [21]. During training a downstream task, STaRFormer dynamically gathers attention weights QK?
A softmax
A Nx (left tower, Fig. 1), where L, B, N represent the number Vdd
of attention layers, the mini- batch size, and the number of elements in the sequences, respectively. The attention weights, denoted as A, essentially indicate the importance of each sequential element with respect to each other. The collected attention weights are then aggregated via attention rollout [58], refer to Eq. (39), resulting in Á E IR Bx N In order to determine the 'global' importance of specific elements within sequence, we compute the attention scores, dv,k', refer to Eq. (40), as in [21], where greater di values indicate higher importance of sequential element and vice versa. The resulting attention scores, O.K BBx, allow distinct masking scheme for each element in X, resulting in B masks per X. The creation of the regional mask, g D - R, requires three hyperparameters: ip, determines the maximum amount of elements that are masked; Ç, determines the number of sequential elements that are masked based on the attention scores a (see Eq, (40)); and y, which determines the bounds of the region to be masked. Further details of DAReM, including the implementation, are provided in Appendix B.3.
3.1.2 Semi-supervised contrastive learning
Previous work has focused on pretraining techniques aimed at creating generalizable time series representations applicable to a wide range of downstream tasks, as well as on learning sequence reconstructions both during pretraining and during training of a downstream task [24, 27, 42. 23, 21].

---

Instead, STaRFormer aims to enhance the latent space representation utilized by the model to perform downstream task. While training for a downstream task, DAReM allows the creation of two correlated latent representations, i.e., masked (Z)) and unmasked (Z()). It is well know that CL can extract high-quality, discriminative features [52-54, 59, 55, 60]. Thus, with STaRFormer, we aim to facilitate CL in optimizing the trade-off between these representations (Appendix B.3, Fig. 8) leveraging unmasked and masked embeddings of the same input sequence batch-wise and of the same class as class-wise positive pairs. This aims to: strengthen the model's robustness to perturbations, enhance generalization, reduce overfitting, and improve resilience to challenges like non-stationarity and irregular sampling. Based on these positive pairs, STaRFormer fuses two types of CL tasks: (i) self-supervised using batch-wise, and (ii) supervised using class-wise similarities. We propose three formulations: the first requiring a class label per sequence; the second requiring a label for every sequential element; and the third requiring a scalar target value per sequence.
Formulation 1 Sequence-level prediction tasks. During training, the latent spaces Z, Z E xxxx become three-diment mire-dimmensoonl tensor representations, where Z = f(X), Z f(g( X)) and the latent embedding dimension. To extract the similarity scores, by computing the inter- sequence cosine similarity (u, v) u ||u|| [/vl) between the sequences in batch, we average the latent representations along their first dimension, i.e., Z., N Σn= Zn R reducing each sequence to a single vector representation. This allows us to formulate the NT-Xent [52] inspired batch-wise contrastive loss for single positive batch-wise sample as:
exp
sim
20)0) 20)
/T
bw = log
(1)
k= [
exp
sim
2000, ((()) /T)
and the class-wise contrastive loss for single positive class-wise sample as:
exp Σ T[c,=c,]
(sim 22)0) (0) /T)
loog E [c++ exp
(2)
sim ((K)) /T)
The indicator function differs in the two cases: [eys for batch-wiss which is 1 iff + i, [[C]CC for class-wise, which is if the class of is different from the class of k, and vice versa for Iic, [[]. The complete loss the sum over all sequences in batch, where the batch-wise and class-wise
components are defined as Lbw = B Σ 1 bw and Sew B DE (5) respectively.
(7)
Formulation 2 Sequence element-level prediction tasks. The previous formulation is insufficient for element- wise prediction tasks. To address this, we introduce modifications that enable the application of our contrastive loss compositions in such settings. To create element-wise positive pairs per batch element, the first two dimensions of Z are collapsed to form Zeat, Zaat E IR N= Thus, at each position where = j. the element originates from the same sequential input element. Consequently, the element-wise contrastive loss for a single sequential element becomes:
exp sim
Baat
(c)
Baat
-(t)
/T
log
bw
k=
[
exp
sim
Z
flat
Z
Blat
/T
In the element-win wise formulation, the class-wise positive pairs allow intra- and inter-class formulations, whereas, in Formulation 1, only inter-class formulations are possible. To compute the positive pairs, we need to define left, Y) E BR x1 and a right, Y, E glxB+M label tensor as well as sequence indicator tensor, & N where i {0, 1, .... N (B 1)}. Thus, the inter-class element- contrastive loss for a single sequential element becomes:
1=
NoB ((-1
inter, Y ((())= (1))
exp
sim (1)
filat
Zfiat
=(1)
/T)
(
-lgg
(4)
ew-inter
Σ NoB ((.)k
k= inter,
xx cxp
sim (5) (A)
Zitat=
Hat
/T)
where ((5))
inter, ( is 1 iff G, + G, (is) (i.)) A x(( -1 A Y. (i.)
-1,
refer to Eq. (41). The intra-class formulation requires the cosine similarity computation between
5

---

each element of sequence, simpatra- We use a batch-wise matrix multiplication operator X bmm R RBBNNXM x RBBM + R Bxxxx to compute the hiree-dimensional similarity matrix (Eq. (42)).
perrm and Z pern are permutee equivalents of Z and Z fitted to the required shapes for X bmmm Thus, the intra-class element- wise contrastive loss for single sequential element becomes:
[((-3) intra.
exp simintra (pporm: Zporm)) /T)
(((U))= =((())
(15) cw-intra
log Σ k=
(5)
intra. (1 xx(-,)) exp simintra
Zperrm Zpprrm /T)
(3)
(()
where [(f-))
- is iff JA Y = Y!U!) Y((J)) ~ Y( -1 (Eq. (43)).
intra, (
For the element-wise formulation, the total batch-wis wise loss Low = NoB Σ
whereas the
bw
total class-wise loss is Lew N+B Di=1 "ew-iuter
N+B (
EB 1 B
((-) Ej=1 cww cotrrara"
Formulation Sequence-level regression tasks. This section outlines the formulation for the regression task, which necessitates scalar predictions rather than categorical classes. Consequently, only the self-supervised component, specifically the batch-wise formulation presented in the sequence- level prediction task (Eq. (1)), can be computed directly. To incorporate the supervised CL component, we generate pseudo labels by clustering the predictive target values into K clusters. The parameter k is a hyperparameter requiring optimization. Once the targets are clustered, each target within cluster K is assigned the same pseudo-label which is subsequently employed as supervision in Eq. (2). The clustering of target values into & clusters is achieved using the k-means algorithm [61].
Independent of the formulation used, we define the fused contrastive loss as the weighted sum of the batch-wise and class-wise contrastive losses:
LSTRRCCL Aruse-cr. + (1 Afuse-cI. Sew
(6)
Finally, FTaRFormer's loss is defined the weighted sum of Frask and the fused contrastive loss, LSTA-CCL:
EsTaRFormer Lrask + ACILSTER
(7)
where AcI. is tunable hyperparameter. In experiments, we Muse-CI. 0.5 to equally weigh batch and class-wise similarities. For further insights, see Appendix B.3 and Figures 10 and 11.
4 Experiments
This work is motivated by the challenge of predicting user intent (a classification task) from non- stationary, spatiotemporal, and irregularly sampled time series. This problem can present significant difficulties for conventional modeling techniques. Our main focus is to evaluate model performance under these conditions. To ensure a robust and comprehensive assessment, we additionally employ an irregular sampled and regular sampled time series benchmark. demonstrate broader applicability, we extend the evaluation to additional downstream tasks, i.e., anomaly detection and regression. We compare against state-of-the-art methods to evaluate STaRFormer" effectiveness and perform exhaustive ablation studies to verify the performance gains. In [62], we present a comprehensive large-scale evaluation conducted within a federated environment.
4.1
Classification results
This section reports the classification results obtained across various time series domains.
4.1.1
Non-stationary and spatiotemporal time series
First, we evaluate the performance on non- stationary spatiotemporal data using the DKT and Geolife (GL) [63] datasets. The DKT dataset consists of mixture of non-stationary, spatiotemporal, and irregularly sampled time series, encompassing 559,709 labeled and anonymized customer trajectories. These trajectories were recorded from vehicles in the BMW Group's fleet over a three-month period. The associated task is intent prediction, which framed as a binary classification problem. The DKT results in Table 1 are averaged over five seeds. We additionally use public dataset (GL) similar to the

---

Table 1: Results for spatiotemporal, non-stationary time series.
Table 2: Results for irregular sampled time series (in %).
P12
PAM
Ausuney
Fas
Accunacy
Asuraay
RNN
TijRorme:
0.754 a: 0.000
0.754 a 0,010
06643
0.555
0.561
43.8.233 3-2
43.02:2.1
42.702
533 6.T.
82.6 2.2
S1.52:21
06.32- 4.4
011
47.5 3.6
4-47
863
$18
440 3/6
06.-2:21
10.:2:12
812.1:6
51111:13
55.15:24
018-112
LSTM
Trandommr
ST-GRU"
03664
B399 +0:00
0380
40.001
0,505
0013
ITTAND
067-133
636-113
53.6-3
35.1--32
582-408
026-4:14
961116
085-4-34
171-136
026-131
763--3
765-4
45.
7566-3 9-1
765-1-18
775-1-2
11P-1-3
0684433
3131-1
7661-78
TARNe:
0.781
10.011
0.782 10,012
0,590
Timeel RT.
0.234 10/00
01911
VITST
20:
13.1
011
96.2 613
16.1 m6.1m11
The model results marked *0 are from [17]. ++ from [51]
[3].
DKT dataset to evaluate STaRFormer, Due to the environmental influences while recording GPS data, we expected some degree of non-stationary in GL [10]. KPSS and ADF tests [64, 65] confirmed that 93% of the data used for training and validation is non-stationary. Across both datasets, STaRFormer consistently outperforms state-of-the-ar approaches, including TimesURL. and other Transformer- based methods such as TARNet. The results are documented in Table 1. Additionally, we perform a robustness analysis with baseline models that achieve very similar performance ForRrrrmer on the DKT dataset. To investigate the sensitivity of the predictions to potential sensor noise, we add noise to the last 10 and 30 elements of longer sequences in the test set of DKT and evaluate the coefficient of variation (CV). The analysis reveals that all models exhibit reduced robustness as noise increases. However, STaRFormer demonstrates superior robustness by maintaining the lowest CV and moderate MAE values, indicating minimal sensitivity to noise. In contrast, the Transformer model exhibits the most significant performance degradation, underscoring the effectiveness of our approach in learning robust latent representations that enhance downstream task performance. Refer to Appendix D.2 and Table 14 for an extended analysis.
4.1.2 Irregularly sampled time series
We compare STaRFormer against state-of-the-ar methods designed for irregularly sampled time series on the PhysioNet Sepsis Early Prediction Challenge 2019 (P19) [66], the PhysioNet Mortality Prediction Challenge 2012 (P12) [67], and the Physical Activity Monitoring (PAM) [68] datasets. In real-world applications, particularly in healthcare, the times series data is often accompanied with static attributes. Following prior baseline methods, such as ViTST, we convert these static attributes into sentences and encode them using RoBERTa [69]. The resulting embeddings are concatenated with the latent embeddings from STaRFormer before being passed the output head. For consistency. static features are also used in all baseline models. The results are averaged over five data splits. Across all models, STaRFormer consistently and significantly outperforms state-of-the-an baseline models on all datasets. Furthermore, STaRFormer yields predictions with significantly smaller standard deviations across all metrics than other methods, indicating greater consistency, reliability, and reduced performance variability. The results validate that our approach works particularly well for irregularly sampled time series. These findings are summarized in Table 2.
4.1.3 Regular time series
We evaluate STaRFormer using the UEA benchmark [70] to assess its performance on regular time series. The datasets covers variety of domains, sensor types, sampling frequencies, number of samples, time series lengths, feature counts, and target classes for comprehensive evaluation. For the evaluation in Table 3, as not all models have reported results on the complete benchmark, we consider
Table Classification results on the multivariate time series UEA benchmark (30 datasets) [70].
VIES!!
DTWD-
Vicred Nixe
TST (ToeddR
TS-TDC"
TNC
beoTS"
Bader
Moe- Kicke"
UST (DAROAS"
IdiTS,"
TraoUitL"
TARNa"
Ferner
Slah.
Ang. Ancenay
07790
0,900
0.091
D517
0.558
[1668
0.63)
6/70L
3314
6:15
353
0/799
6710
0533
0.555
Ang. Scoee
154
9.2
7.4
57
3.3
24
27
25
22
US Cooe
23
30
30
30
30
30
30
30
30
30
30
Ascuaay
11000
0.535
08777
07133
5:12
8/200
3:33
0/224
6:728
0353
01700
420
Roak 26
15
11
11
40
A1
0073%
0737
0421
0307
67W
0315
676
0771
677
0312
6761
Roak
Ang Konk 0
64
10.8
003
1233
15.1
11.9
50
100
637
74
300
54
53
65
33
The model results marked with are taken from the (21). T from [27). 1.7 from [26] and from [3].
7

---

three splits of the benchmark depending on the results available in literature (UEA, UEA 28 and UEA 9). The summarized scores of the complete results (Table 30) across the UEA benchmark are displayed in Table 3. STaRFormerr achieves the highest accuracy in all three splits (0.795), improving the state-of-the-art on the complete benchmark by 4.0 percentage points; the largest number of top scores (9); and the best average rank (2.8). Furthermore, STaRFormer performs better on UEA datasets with only a few samples, achieving top scores for DDK, NT, PS, SCP2 and SWJ for example. This suggests its capability as an augmentation technique, especially for lower data regimes.
4.2 Anomaly detection results
Table 4: Anomaly detection results (univariate).
Yahuo
KPI
In this setting, we adopt the stream- ing evaluation protocol proposed in [71] and utilized by [24, 27]. The
F1 f
Precision
Recall
Fj
Precision
Recall ?
SPOT
0.338
0.269
0.454
0.217
0.786
0.126
DSPOT
0.316
0.241
0.458
0.52]
0.623
0,447
model performance is evaluated on the KPI [71] and Yahoo [72] bench- mark datasets and compared against several state-of-the-art approaches, such as TiMesURL and TS2Vec.
DONUT
0.026
0.013
0.825
0,347
0.371
0.326
SR
0.563
0,451
0.747
0.622
0.647
0.598
TS2Vec
0.745
0.729
0.762
0.677
0.929
0.533
TiIL
0.749
0.748
0.750
0.688
0.925
0,546
STaRFormer
0.772
0.807
0.830
0.852
0.811
Each time series split chronologically, where the first half is used for training and the second for evaluation. To facilitate efficient computation, we choose to segment sequences into fixed-size windows, allowing overlap between these segments during training. STaRFormer demonstrates superior performance across both datasets in the benchmark, as shown in Table 4.
4.3 Time series extrinsic regression results
Table 5: Regression results on the TSR benchmark (19 datasets) [73] reported in RMSE.
5-NN- ED=
5-NN- DTWD.
FPCR+
Optimiseds
Forest=
Boust
Rocket-
HUN+
Note
Inceptions
Net
Former
Avg. Rel. Mean
Difference.
0.028
0.208
-0.121
-0.132
0.051
-0.031
-0.215
-0.160
-0.119
-0.220
0,170
-0.254
Avg. Mean
Difference Rank
9
12
5
10
2
4
3
Top Scores
The model results marked with taken from the official benchmark (http:/ http://tttrrggiio.cooo orgg/).
The results of the Time Series Extrinsic Regression (TSR) benchmark summarized in Table 5. The complete results can be found in Table 33. The results present the Root Mean Squared Error (RMSE) of scalar regression predictions produced by each model. In order to facilitate comparative analysis of the models within this benchmark, we adhere to the evaluation metric established in [23], referred to the average relative mean difference (Eq. (48)). This metric quantitatively evaluates the deviation of each model from the mean RMSE for each dataset. Therefore, superior model performance is indicated by increasingly negative values the metric, whereas inferior performance corresponds to less negative and positive values. We implemented TARNet and utilize the model configurations provided by the authors, where available, for the respective datasets. Across the entirety of the benchmark, STaRFormer consistently achieves the greatest relative mean difference among all models, alongside the largest number of top scores.
4.4
What contributes to STaRFormer's performance?
This section investigates the source of STaRFormer's performance gains through empirical validation.
4.4.1 STaRFormer architecture ablation
To demonstrate the performance gains achieved by DAReM paired with the semi-supervised CL in STaRFormer, we train two ablations of ForRerr an encoder-only Transformer (Base), and STaRFormer with Random Masking (RM). We select the datasets from Sections 4.1.1, 4. 1.2, and 4.2, in addition to representative set from the UEA benchmark (see Appendix C.1.3), for total of 19 datasets. FoaRFrrmer outperforms ForRForee--RM in 16 out of the 19 selected datasets, while STaRFormer-RA RM outperforms STaRFormer only in one dataset, verifying that DAReM significantly
8

---

Table 6: STaRFormer archi- tecture ablation results on 19 datasets.
Table 7: Ablation results for semi-supervised CL and DAReM.
CL Method
DKT (cei in 0.796)
GLL (lcc to 0.773)
OAM in 0.566)
Acceracy 1
Post
Nocuncy T
PasT
Accuracy
Precisox
Rocall?
F
Bas
STaRFirre RM
STiAForxx
85.2
a52
50.4
97.6
974203
Aoa. Baak
wil wclt!
64.5 4 0.2
64
00001:1 1.4
962 2.1
0:14077
=7 004007
84.5 a
84.7 0.7
50.6 1.0
60.7
90.6 1.6
27.
Aug Rent
31
7.5
12
Top Scores
0.1
6..62.0.44
0000
2
000:8:00
96.7 0..5
Basc
12
Stattvvener
80.6
58.3
97004-1.0 1.6
V0.2 6.07
4.0
97.6 0.7
GL (c (0
PAM 10000
0.3)
Accunery
Fas
Accunay
Fax
Ascunny
Pressesrs
Recall
Fr
85.0
$7.
97.0-:0..
97.4
0.3
24.
20.34
58
97
0.15
85.0 = 0.2
25.0
10.34 1.5
59.2 3.7
07.1 1.1
07.5
075+10
=7.
0.20
81.1
0.1
65.1 2..11
W0. 11
1.0
16. 0.6
16.7 0.5
10.5 1.00
96.6
2.4
0.25
8533 + 01
853 -03
00.1 1.6
88.4 1.4
96
1.1
960 8-
16.5 6-5
16.7 =0
55.0
10.34
58.2 3.4
06.4
enhances the robustness of the model compared to RM, see Table 6. STaRFormer achieves the highest average accuracy (0.841) surpassing the two ablation variants by 1.5 and 1.7 percentage points respectively; the highest number of top scores (15); and best average rank (1.2).
4.4.2 Impact of semi-supervised contrastive learning and regional masking
Impact of semi-supervised CL. We examine the effect of different components of the semi- supervised CL in STaRFormer by removing the respective components from the loss function. The results in Table 7 show the advantages of maximizing agreement between both batch-wise and class-wise representations in CL. The downstream task performance (accuracy) declined by 0.4 to 1.4 percentage points when these representations were not fused in the CL approach. There is no consistent trend favoring one representation over the other; e.g., in GL, supervised CL outperformed self-supervised CL, while the opposite holds for PAM. In DKT, both methods yield comparable results. Additionally, we study the impact of combining the contrastive loss ESTAR-CL and the task loss LTask via ACT. Eq. (7). The scale difference between Taask and &STAR-CI is approximately a factor of 10 across all datasets. Consequently, values of ÅCI. > 0.1 assign greater weight to LSTAR-CI. thus increasing its impact on the overall loss and the model updates during backpropagation. Our results indicate that higher values of ÀCL lead to improved performance, with all top scores achieved at ACL > 0.1 (in some cases, large weights of and 10 yielded best results). These results indicate that emphasizing context-aware representation learning during the training of downstream task can enhance the overall performance on this task, supporting our approach.
Impact of regional masking. To examine the impact and benefit of masking regions with DAReM, we perform an one-at-a-ting analysis (OAT), where we iteratively change the region defining pa- rameter, 7, while keeping all other parameters fixed. We expect better performance when using larger masking regions (7 > 0) around the top-k compared to only masking the top-k important sequential elements (y 0), which is essentially the masking approach in TARNet. However, excessively large masked regions may degrade performance by limiting the informative context for reconstruction. The observed trend indicates on a macro scale that masking larger regions enhances the performance of STaRFormer. Masking regions larger or equal to 10% of the global sequence length around the selected elements achieves top scores for 7 out of 8 metrics (Table 7 bottom section). On a scale, performance peaks were observed at different optimal configurations with the best performance for DKT at y= 0.25, whereas for GL and PAM, the performance peaks were found for smaller region masks. Further increasing or decreasing the masked regions gradually deteriorated the results, supporting our initial hypothesis.
4.4.3 Latent space analysis
To evaluate the hypothesis that enhancing the latent space embedding improves prediction perfor- mance, we analyzed t-SNE visualizations [74] of four datasets (test sets only) where Former outperforms the Base ablation. Thus, we compare the latent space representation of Base and STaR- Former. As shown in Fig. 2, the t-SNE visualizations reveal that while Base achieves some degree of class separation, STaRFormer consistently produces distinct and well-separated clusters for each
semi semi-supervised, self self-supervised and sup supervised

---

(a) DKT, Base
(b) DKT, STaRFormer
PAM. Base
(d) PAM, STaRFormer
(c) GL, Basc
(f) GL, STaRFormer
(g) PS, Basc
(h) PS, STaRFormer
Figure 2: t-SNE visualizations (plotted with perplexity 50) of latent spaces representations for the
DKT (a, b), PAM (c, d), GL (e, f), and PS (UEA) (g. h) datasets, comparing Base and STaRFormer.
class across all datasets (classes color-coded). In DKT, the latent embeddings for both models show overlap between class clusters. However, STaRFormer can more distinctly separate clusters between the classes, whereas Base has significant of overlap. This trend is amplified by the observations PAM and GL, where FTaRrormer displays more distinct clusters with minimal overlap compared to Base. In GL., clusters for 'walk' and 'bike' well as 'car' and "bus' distinctly separated. However, distinguishing between "car' and 'bus' remains challenging due to similar traveling speeds and trajectories. In PS, the Base producing scattered clusters with significant overlap, while STaRFormer achieves distinctly separated clusters. These results align with the test accuracy reported in Table 31, where the accuracy difference between Base and STaRFormer is most significant on PS. In summary. the clusters from both models appear more similar for datasets with similar test accuracies. STaRFormer creates more discriminative latent representations, i.c., enhanced class separation. which, considering the improved test accuracy, leads improved classification performance over the Base ablation. For datasets where our CL approach is very effective, e.g., PS, the improvement through our approach is clearly visualized the t-SNE visualizations.
pronounced difference is obtained with
Limitations and conclusion
We propose task-coupled semi-supervised CL technique that jointly optimizes representation learning with the downstream objective, embedding task-specific information into the representations. By integrating embeddings that generated from masked (DAReM) and unmasked sequences, the semi-supervised CL exploits both batch-wise supfr-sueer and bliss-wise (supervised) similarities to achieve improved task-specific representations for predictions various downstream tasks. Comprehensive experiments demonstrate that STaRFormer either surpasses or is on par with state- of-the-art techniques for various time series types. We verify this performance on 55 benchmark datasets and real-world data from the BMW Group. Notable limitations include: the computational overhead of the attention-based masking with O(N2) complexity, especially for long sequences, and the additional increase in training time and complexity due to CL and DAReM, which however does affect inference time. Additionally, the task-coupled nature of STaRFormer results further limitation constraining its flexibility compared to task-agnostic models such as TimesURL or InfoTS. These models aim to learn universal representations that, theoretically, can be utilized across a variety of downstream tasks without the need of training from scratch. Future work could explore more efficient attention mechanisms, such as flash attention, to enhance the scalability and efficiency of STaRFormes for long sequences and parge-scale datasets. Moreover, exploring task-agnostic implementation of FTrmFrrmer could substantially enhance its flexibility.
10

---

Acknowledgments
This project is supported and funded by the BMW Group and has received partial funding from the European Union's ss Horizon research and innovation programme (Grant Agreement No. 101 101159667) Any opinions, findings, conclusions or recommendations expressed herein are those of the authors. They should not be interpreted necessarily representing the views, either expressed implied, of the BMW Group and its affiliates or the European Commission. Neither the European Commission the BMW Group is responsible for any use that may be made of the information contained herein.
References
[1] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory, Neural Computation, URL
(8):1735-1780, November 1997. ISSN 0899-7667. doi: 10. 10.1162/neco. 1eeeecco.1999 https://doi.org:
1162/neco 1997. .9. 8.8.11. 1735.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / //proceedings neurips. cc/paper__... hwl.ee//pppee////aaaaa/////aaaa/ ad053cic4a845er Abstract html.
[3] Zekun Li, Shiyang Li, and Xifeng Yan. Time Series as Images: Vision Transformer for Irregularly Sampled Time Series. Advances in Neural Information Processing Systems, 36: 49187-49204, December 2023.
[4] Apple. Explore UWB-based car keys WWDC21 Videos, June 2021. URL https: //developer apple. com/videos http:/pawa22//////////p
[5] BMW. BMW anmounces support for Digital Key for iPhone. A secure and easy way to use iPhone https//wwwwa.o......///////tttta
key to lock, unlock, drive, and share keys with friends., June 2020. URL
http:://wwww.rsss.aem press. https:
-hounces-support-fee http://wpp...............eeeeprr
o- -usse-iponne-ss-a- a- car wtw.::/www.........ooooooooooaaa
riends?language=en.
[6] BMW. What's the deal with Ultra Wideband technology and what will it do for your car?,
November 2021. URL attps://wwt bmw www.co/////..// httpiinnnvvtiinnnneieeeoooeo wwwooonwmwwwaeeeeeeeeeeeeeeee www-dddggialkkyyyy
s s-ultra-widebant html.
[7] Mercedes-Benz. UWB, BLE Digital Car Key E Klasse, April 2023. URL https:/meedia. 21333ddc110339222244/aa
sercedes-benze http://eess..tt////bobbbbbbbbbb oftaraaoooooooaaaa/.....//o-o--r cle/ca92e61f- 000--00000000077050000---------0
[8] Samsung. Samsung Wallet Adds Digital Key for Select Audi Vchicles, November 2024. URL
https://newt httpe:///ewessamungg..memm com/
hat.u/g./ooo/ooo////aaaaaa hglgg//lllaaauugggggoaoo waatmsunnwwalee-----eeeaa wwwlllet-ddd---ee
[9] CCC. Digital Key Release 3, 2024. URL -release- Lal.easee-----ppcciiiatatiiieeee dawnload/. Version 1.1.3.
[10] Luke Bermingham and Ickjai Lee. A probabilistic stop and classifier for noisy GPS trajectories. Data Min. Knowl. Discov, 3266)1133-1166,, November 2018. ISSN 1384-5810. doi: 1000007/10008--18-0555556 URL 8-8.
[11] M.A. Hearst, S.T. Dumais, E. Osuna, J. Platt, and B. Scholkopf. Support vector machines. IEEE Intelligent Systems and their Applications, 13(4):18-28, July 1998. ISSN 2374-9423, doi: 10. 109/5254. 7084228.. URL https:://///.... /aapp:s////eeeeoorrr//r// wetrrr.eee..... ieee. wwwpeeer/wwcccmeee///e/eee certuumbbr=70044500 Conference Name: IEEE Intelligent Systems and their Applications.

---

[12] Leo Breiman. Random Forests. Machine Learning, 45(1):5-32, October 2001. ISSN 1573-0565. doi: 10.1023/A:1010033400100 URL https: pestss//dooi //toi. org /0.g//../ 1023/A: 1010933404324.
[13] Benjamin M. Marlin, David C. Kale, Robinder G. Khemani, and Randall C. Wetzel. Un- supervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Sympo- siam, IHI '12, pages 389-398 398, New York, NY, USA, January 2012. Association for Com- puting Machinery. ISBN 978- 1888---450-0788--9 doi: 10.1 10.1145/21 (02///210333..0 21 10408. URL https://doi. org/ htt.g//......////.... 1111/111111181 2110408.
[14] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN En- coder-Decoder for Statistical Machine Translation. In Alessandro Moschitti. Bo Pang. and Walter Daelemans, editors. Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1724-1 1134, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3 1/111//111111///// URL attps://aclanthologg. org org/D14-1179.
[15] Zachary C. Lipton, David Kale, and Randall Wetzel. Directly Modeling Missing Data in Sequences with RNNs: Improved Classification of Clinical Time Series. In Proceedings of the 1st Machine Learning for Healthcare Conference, pages 253-270. PMLR, December 2016. URL https: http:////oceeeennsseeeee mar.annnsmm.......mmmmmmmmmmmm.m gress/v56/Liptont html. ISSN: 1938-7228.
[16] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports, 8(1):6085, April 2018. ISSN 2045-2322. doi: 10. 1038/s4
URL Publisher: Nature
https: //www. nature www./aoaiiiie.o....... /41ps///111/-aa/ 018 24271-9. Publishing Group.
[17] Hongbin Liu, Hao Wu, Weiwei Sun, and Ickjai Lee. Pratio-Temporal GRU for Trajectory Classification. In 2019 IEEE International Conference on Data Mining (ICDM), pages 1228- 1233, November 2019. doi: 10. 10.11099/CDM22201000000 URL https: http:////eeeeeeeeeeeeeee ieee. http:///aaa///07777/////000000/0 (ac.unnnt/////rre///mmmmmmeerrre
ISSN: 2374-8486.
[18] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set Functions for Time Series. In Proceedings of the 37th International Conference on Machine Learning, pages 4353-4363 PMLR, November 2020. URL. https://pproceedings mmlr.
press/vv19/hornp/// html. ISSN: 2640-3498.
[19] Satya Narayan Shukla and Benjamin Marlin. Multi-Time Attention Networks for Irregularly Sampled Time Series. International Conference on Learning Representations, 2021. URL
[20] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-Guided Network for Irregularly Sampled Multivariate Time Series. International Conference on Learning Representations, February 2022. URL https: http://opppeevveeee net/forum?id-Kw m8I7aU-15.
[21] Ranak Roy Chowdhury, Xiyuan Zhang, Jingho Shang, Rajesh K. Gupta, and Dezhi Hong. TARNet: Task-Aware Reconstruction Time-Series Transformer. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 212-220, Washington DC USA, August 2022. ACM. ISBN 978- 978-1-4503-9385-0 doi: 10. 10/3144/3334678 3539329, URL https::/ddd acm. org/ /14///1....1//333775556 3539329..
[22] Haoran Liang, Lei Song, Jianxing Wang, Lili Guo, Xuzhi Li, and Ji Liang. Robust unsupervised anomaly detection via multi-time scale DCGANs with forgetting mechanism for industrial multivariate time series. Neurocomputing. 423:444
January 2021. ISSN 0925-2312. doi: 10. m..101//.eeccom.2200010... URL wwtphhttps.. /www. sciencedirect. http://rrettcooom/cccn/////////e cle/ cle/ppi/50995221223355
12

---

[23] George Zerveas, Srideepika Jayaraman. Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A Transformer- maassseemee-baeedd Framework for Multivariate Time Series Representation Learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, KDD) '21, pages 21 2114-2124, New York, NY, USA, August 2021. Association for Computing Machinery. ISBN 978-1-4503--833-55. doi: 10.1145/344548.3467777 URL https: //d1. acm. ottg//oo/11...14//444475//9 0450//4444558..447400
[24] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong. and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series, February 2022. URL http://arxiv
10466. arXiv:2106.10466 [cs].
[25] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive Learning of Disentangled ceasonal-Trent Representations for Time Series Forecasting, May 2022. URL http: /ttp///axvvvvvvv org/
wwwggag//////eo...o...oo..eeeee/ a.b///22/.22 01575. arXiv:2202.01575 [cs].
[26] Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, and Xiang Zhang. Time Series Contrastive Learning with Inferm Augmentations. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4))453444542 June 2023. ISSN 2374-3468, doi: 10. w..1609/aai.v337/0 v3714.25575. URL https: //ojs aaai werg/iide.... php/ http//AAT/ariicl//// ew/25575. Number: 4.
[27] Jiexi Liu and Songcan Chen. TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 1311)))3188-1922 March 2024. ISSN 2374-3468. doi: 10.1609/aaii. il2 112.29299. URL https: h/ops:///of aaai, orgg/nndes php/ httpp//AAAaaatceeeeaeeeeeeaaeaee AAA//attiice//vie///2222. Number: 12.
[28] H. Sakoe and S. Chiba. Dynamic programming algorithm optimization for spoken word recognition. JEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1):43-49, 1978. doi: 10.1109/TASSPP199 1978. 1663055.
[29] Mohammad Shokoohi- Yekta, Jun Wang, and Eamonn Keogh. On the Non- Trivial Generaliza- tion of Dynamic Time Warping to the Multi-Dimensional Case. In Proceedings of the 2015 SIAM International Conference on Data Mining (SDM), Proceedings, pages 297- Society URL.
for Industrial and Applied Mathematics, June 2015. doi:
betps://eputee siam. org/ http///o//....///11..111 111.1111...1111111111 97811119740100
[30] Patrick Schäfer and Ulf Leser. Multivariate Time Series Classification with WEASEL+MUSE, August 2018. URL
http://axxivv..r/ab/// wwtp://///.aaaaaaaaaoab 11343. arXiv:1711.11344 [cs].
[31] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, (23(6888) October 1986. ISSN 1476-4687, doi: 10. 10.1038/323533a0. URL https: /t///wwww /www./www...oo//.. nature. com htmm/arriicleee/2553aaa Publisher: Nature Publishing Group.
[32] Jeffrey L. Elman. Finding Structure in Time. Cognitive Science, 14(2):19--211, 1990. ISSN 1551-6709. doi: 15.12077/ 1. URL / wwley. com/doi/abs/ 200.00000//115117999000444400
[33] Angus Dempster, François Petitjean, and Geoffrey I. Webb. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining
and Knowledge Discovery, 34(5): 1454-1495, September 2020. ISSN 1573-756X. doi: 10110000000000100000000000000000
10.
URL https://doi
[34] Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb. MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, pages 248-257, New York, NY, USA, August 2021. Association for Computing Machinery, ISBN 950-1-4503--332- 14/114//44444 /445//4///4444844 3467231. 1465133447788338823...
5. doi: 10.1145534455483447777 URL https:///oi.
13

---

[35] Zhiding Liu, Mingyue Cheng, Zhi Li, Zhenya Huang, Qi Liu, Yanhu Xie, and Enhong Chen. Adaptive Normalization for Non-stationary Time Series Forecasting: A Temporal Slice Perspective. Advances in Neural Information Processing Systems, 16:14273-14299 December 2023. URL. http:::tpp/ttttt httpppppp/pooccceiiigggssss http://rrrredn.n............pppp neurips www.wps....ppppppae.... e1 1195aa9480000eee0044449990ff.da3 htwp://www...ooo-aaaaaaeeea Confeetecoooeeeeceeeee html.
[36] Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting. The Twelfth International Conference on Learning Representations, October 2023. URL https: /tttps://open review. wwweewnee/fooooo wtt/forumm FoorumniddM.kerODiiee
[37] Peng Chen, Yingying Zhang, Yunyao Cheng. Yang Shu, Yihang Wang. Qingsong Wen, Bin Yang. and Chenjuan Guo. Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting. In The Twelfth International Conference on Learning Representations, October 2023. URL https: :/ http//pppprrieeeee// net. (ttp//Eorumm1dd11JoooPaaaa
[38] Yuxin Li, Wenchao Chen, Xinyue Hu, Bo Chen, Baolin Sun, and Mingyuan Zhou. Transformer- Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting. In The Twelfth International Conference Learning Representations, October 2023. URL https: hotp::///pperevie... net.iiewwweeeeeeeee.eeeee.eeeeee forum?1d=qae04YAAHH
[39] Junho Song, Keonwoo Kim, Jeonglyul Oh, and Sungzoon Cho. MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection. November 2023. URL https:
/ppnrreview.... net uttp/foorumnddddPoooduuu
[40] Shengming Zhang, Yanchi Liu, Xuchao Zhang, Wei Cheng, Haifeng Chen, and Hui Xiong. CAT: Beyond Efficient Transformer for Content-Aware Anomaly Detection in Event Sequences. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pages 4541-4550 New York, NY, USA, August 2022. Association for URL doi: 10. 10.1145/3334678.35300
Computing Machinery. ISBN 978-1-4503-9385-0. https: acm. oww//oi.//oooooooooooooo.o.o.oo / 00445//534678..8
[41] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhaneing the Locality and Breaking the Memory Bottleneck Transformer on Time Series Forecasting. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /ttttss://preet p/pooeeed////ocoeiigg/e.e
hwwpeddiinnnnnnennneeeeeee neurips cc/paper.......... hace/pppe_ffieeeeee www.fiilespppee /paper Aeto0--bbttttttttoo html.
[42] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training Deep Bidirectional Transformers for Language Understanding. May 2019. URL http:
//arxiv.org/atee httg/abs/18800 1atoos//810.00 04805. arXiv: 18.::881 181.04880 [cs] version: 2.
[43] Christopher M. Bishop. Pattern recognition and machine learning. Information science and statistics. Springer, New York, 2006. ISBN 978-0-387-31073-2.
[44] Rebecca Salles, Kele Belloze, Fabio Porto, Pedro Gonzalez, and Eduardo Ogasawara. Nonstationary time series transformation methods: An experimental review. Knowledge-Basee Systems, 164:274-291. Jamuary 2019. ISSN 0950-7051. doi: 10. 10.10166jkmooss.220888 10.041. URL. httpp::/wwwe rectt http.///caaccaaaaaatae////////// 3.
[45] Wendi Li, Xiao Yang, Weiqing Liu, Yingce Xia, and Jiang Bian. DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4):4092-41) 4100, June 2022. ISSN 2374-3468, doi: 10.1609 #36i4.20327. URL. https:tt//// //j.../..../..////./... aaai org/ index.dexxx/ phtp//AI//aaaaaaaaaaawwaa http://AAAaarticeeeeeeee//////// Number:
[46] Yuntao Du, Jindong Wang. Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. AdaRNN: Adaptive Learning and Forecasting of Time Series. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, CIKM '21, pages 402-41 411, 11 New York, NY, USA, 2021. Association for Computing Machinery. ISBN URL httph::////ool http:://dooooooooo/// org/10. 1145/34
978-1 978-1-4503-8444 doi: (0.114553459637.344888 59637 3482315.
14

---

[47] Nikolaos Passalis, Anastasios Tefas, Juho Kanniainen, Moncef Gabbouj. and Alexandros Iosifidis. Deep Adaptive Input Normalization for Time Series Forecasting. IEEE Transactions Neural Networks and Learning Systems, PP:1-6, December 2019. doi: N...1099NNNI.SO
19. 2944933.
[48] Yong Liu, Haixu Wu, Jianmin Wang. and Mingsheng Long. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 9881 19993. Curran Associates, Inc., 2022. URL http:ttss//prrceediinggg neurips cc/ feaper_filee http://////pp//oo/ooooo///////// b b-Paper- Confer-Coofeeencee pdf.
[49] Jinsung Yoon, W. Zame, and M. Schaar. Multi-directional Recurrent Neural Networks
A Novel Method for Estimating Missing Data. In Time series workshop in international conference on machine learning, 2017. URL
http///ww.eaaaicsccch.oaaaa org. pape Networs 7,3A A for- Yoon- Zame/T mame/77
r htt.:-iiireetiiiaaa---oorrrrraae fff00tterr/e11211666667777666663
[50] Yuan Zhang, Xi Yang. Julie Ivy, and Min Chi. ATTAIN: Attention-based Time-Aware
LSTM Networks for Disease Progression Modeling, pages
2019. URL https:
/ //www.ijcan org.orgg/ooooooooooooooo httpg//ooeeiiig///11991////
[51] Yuxuan Liang, Kun Ouyang, Yiwei Wang. Xu Liu, Hongyang Chen, Junbo Zhang, Yu Zheng, and Roger Zimmermann. TrajFormer: Efficient Trajectory Classification with Transformers. In Proceedings of the 3 ACM International Conference on Information & Knowledge Management, pages 1229-1237, Atlanta GA USA, October 2022. ACM. ISBN 978-1 --503- 9236-5. doi: 10.1 145/311111111 145/5511 1808. 3557441.
URL https: / /tt////lll/a/ acm. wtg//oi/1/./ooooooooo 1145/351
[52] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. Simple Framework for Contrastive Learning of Visual Representations, June 2020. httthtt/p//arxii.orr
abs/2002 05709.7709... arXiv:2002.05709 [cs, stat].
[53] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Con- trastive Predictive Coding, January 2019. URL MXXiv1180.007420 [cs, stat].
[54] Vassileios Balntas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. In Procedings of the British Machine Vision Conference 2016, pages 119.1-119.11 York, UK, 2016. British Machine Vision Association. ISBN 978-1-901725-5944 http://wwue bwva. httgg/bwwe/22180
doi: 10.5244/C.30.119 URL
[55] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. In Advances in Neural Information Processing Systems, vol- ume 29. Curran Associates, Inc., 2016. URL httphttts///rrroeeeiigg neurips. cc/paper _files httpp////0///ae////////////////a
Abstract Abstract.ttttttl...... htmll.
[56] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In International Conference on Learning Representations, October 2020. URI. https: /tttps://operreee http:///ppereeieeeeeeeee
[57] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively. with application face verification. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR volume 1, pages 539-546, 2005. doi: 10.1109/CVPR.2 005.202. URL https://ieeexppoortt ieee. perr//docmentt//m @rocummett11673144 1467314.
[58] Samira Abnar and Willem Zuidema. Quantifying Attention Flow in Transformers, May 2020.
URL http: //rxivv http:/awwww....aaaaaa//a www///r///////// /t0.://00/000000...... arXiv:2005.0092 [cs].
[59] Ling Yang and Shenda Hong. Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion. In Proceedings of the 39th International Conference on
Machine Learning, pages 25038-25054. PMLR, June 2022. htl.:tttt::/pprceeeee
mlr, grressvv62//ae www16//y/gg2ee...m html. ISSN: 2640-3498.
15

---

[60] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised Scalable Repre- sentation Learning for Multivariate Time Series. Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htwp::::t::: http://ppppp////ceedaagg...s... neurip.. Setaper_filssppppeeeee act html.
[61] Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129-137, March 1982. ISSN 1557-9654, doi: 10.1 (0.119//1T.19922115555 URL ittps://eeeexpoors ieee. http::pplll...eeeeer//oooooooooo wwgg/occmeet/1155e60e0
[62] Maximilian Forstenhäusler, Christos Anagnostopoulos. Shameem Parambath, Daniel Külzer, and Natascha Weber. Leveraging Federated Learning for Decentralized Semi- Suni-Supervised Task-Informed Representation Learning Sequential Data. In 2025 IEEE 45th International Conference on Distributed Computing Systems Workshops (ICDCSW), pages 207-212, IEEE Computer Society, July 2025, ISBN 1---331---725--11 doi: 10.1 10.109//CDDSSS623330022 .00041. URL https: //ww.. computer hwtpppppe...o..caa.////reeeeeiie http:///w/..ooooceconaaatttteeea 25/1 255172500a207/222 2cc5aa2222722255225112
[63] Yu Zheng, Hao Fu, Xing Xie, Wei- Wei-Ying Ma, and Quannan Li. Geolife GPS trajectory datasel User Guide. Geolife gps trajectories 1.1 edition, July 2011. URL https: /ttp///wwwww.// www.microooof.tttomow.owww com/ /e/eusseeeeerrchh de/.
[64] Denis Kwiatkowski, Peter C. B. Phillips, Peter Schmidt, and Yongcheol Shin. Testing the null hypothesis of stationarity against the alternative of unit root: How sure are we that economic time series have unit root? Journal of Econometrics, 54(1):159-177, October 1992. ISSN 0304-4076. doi: 10.1016/0004444776699000
URL httht::/://wwr http::/wwww.cccennereeee.ccc
[65] Yin-Wong Cheung and Kon S. Lai. Lag Order and Critical Values of the Augmented Dickey-Fuller Test. Journal of Business & Economic Statisties, 13(3):277-280) July
doi: 10. 10.000035550015 1995. 10524601. URL https: //doi orgo/////// 10. 1000007500015 1995. 10055..155555 10524601. Publisher: ASA Website.
1995. ISSN 0735-0015.
[66] Matthew A. Reyna, Christopher S. Josef, Russell Jeter, Supreeth P. Shashikumar, M. Brandon Westover, Shamim Nemati, Gari D. Clifford, and Ashish Sharma. Early Prediction of Sepsis in Cardiology Challenge 2019. Critical Care Medicine, 48(2):110, February 2020. ISSN 1530-0293. doi: 10. 10.10977CCCM.00000000000
From Clinical Data: The PhysioNNe//Computiee
www com ccmj 4145. URL sers__sepss___eee from www.iiiiiiiiiio latpaiicllaaaa___aaaaaaaaaaaeaee the. 10. aspx.
y_ of
[67] A. Goldberger, Luis A. N. Amaral, I.. Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals, January 2012. URL https::/pphyiioonet peg/connent// challenge-2012/1...00
[68] Attila Reiss and Didier Stricker. Introducing New Benchmarked Dataset for Activity Monitoring, In 2012 16th International Symposium on Wearable Computers, pages 108-109, June 2012. doi: 10. C09/1SWC.201..13. URL https: sttpp::/ieeexppllee ieee, wwweee.rg///oooeooo docume nt/6246152. ISSN: 2376-8541.
[69] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. July 2019. URL http: org/abs/1907 110007...699 11692.
arXiv:1907.1 1692 [cs].
[70] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018, October 2018. URL http:///a///ati wwwlloorggaaaee www.//o//////..//.ee 00075. 181 111118117007 [es,
stat].
16

---

[71] Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing. Mao Yang, Jie Tong, and Qi Zhang. Time-Series Anomaly Detection Service at Microsoft. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3009-3017, July 2019. doi: 10. 145/3292500033306888 URL. org/ absa 1906. 03821. arXiv:1906.0382) [es].
[72] Yahoo Labs I Webscope. S5 A Labeled Anomaly Detection Dataset, 2015. URL https:
/webscope.... sandbox yaandboxx.hhoo.... com camalla pippddaaattpppessdddatt
[73] Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I, Webb, Time series extrinsic regression. Data Mining and Knowledge Discovery, 35(3):1032-1060, May 2021. ISSN 1573-756X. doi: 111100//1161-001---000 s10618- 02000001-00000 0074-999
URL
[74] Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, ((86))2579-2605, 2008. ISSN 1533-7928. URL http: //jmrr org/ paapers/ve/aandrnneee
html.
[75] G. E. P. Box, G. M. Jenkins, and G. C. Reinsel. Time Series Analysis. Prentice Hall, 1994.
[76] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. URL wwwpleeeedeeeeeeobbooo..o org/.
[77] Zachary C. Lipton, John Berkowitz, and Charles Elkan. A Critical Review Recurrent Neural Networks for Sequence Learning, October 2015. URL http:// arrg/bbs/1500 000 19. arXiv:1506.00019
[78] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. JEEE Transactions on Neural Networks, 5(2):157-166, March 1994. ISSN 1941- 0093. doi: 10.1 10.1109/72.279181 URL pestps:/ieeew/ 1. Conference Name: IEEE Transactions Neural Networks.
ieee org/doccuent//2711
[79] Sepp Hochreiter, Yoshua Bengio, P. Frasconi, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001. URL httpp://vvet http://wwwbbbboonntt///e bibssnomy. org/ bttb ex 22122755t07711111100066622222222
schaul.
[80] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Eval- uation of Gated Recurrent Neural Networks on Sequence Modeling, December 2014. URL fttp://arxiie org/ abs/ / 11/.//111... 15452...55,. arXiv: 411.35555 [cs].
[81] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre- Pre-Trainne prearinnnn June 2018. URL https::t openai com/r pdff.
esear ch @searchh-overrs/aauue /wwoverrslaaggugeeeeee
[82] Tom B. Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Perbert-Vosss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Leamerr, July 2020. URL http: ptstp://arxiin http///axvvvvv. org/ abs/ 2005 14165. 14X1v
[83] OpenAI. ChatGPT, July 2024. URL
[84] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, March 2020. URL http:///axxiv.org/bbbb//
10555. arXiv:2003.10555.
[85] J Wang. Articulary Word Recognition. Utt.ttppp////w..tieeeerrccccccct com/ hom//escrrptiion.o
[86] Ben H. Williams. Character Trajectories Dataset.
fication, hoo.aaeccpppoooppppaaeeeeearrrrr
17

---

[87] Time Series Classification Webpage. Cricket Dataset, URL http:://ww.iimseereeell www.seiiiimmmeeclll
assification, com/description php7Datast--ricket. ort.r
[88] Xenoxanto. Duck Duck Geese Dataset. URL http::::www...innsseeeeccaasaaaa
con//eescription.pppp/oo ort.apcpppp.opppppaaaaaeeeeeeeen eppplpaattteeeeeccccceeeeeeeeeee
[89] Andre Brown. Eigen Worms Dataset, 2013. https://www.w. http://wwww..mmmmmcccccaaasssaia tion.com/ceiic wwwp///wawp.piionnooooo htt.ap/cppppppooppppppaaaaaaaaaa pit.pppaaattteeetgieeoooora
[90] José R. Villar, Paula Vergara, Manuel Menéndez, Enrique De La Cal, Victor M. González, and Javier Sedano. Generalized Models for the Classification of Abnormal Movements in Daily Life and its Applicability to Epilepsy Convulsion Recognition. International Journal Neural Systems, 26(06): 1650037, September 2016. ISSN 0129-0657, 1793-6462. doi: 10. 126677/2002 URL http:///.....ttttt//////oo/..... ctt.tttpp////.....raaaaaaaaaaaaa 1142/5012906571660033
[91] M. Wilhelm. FRing Dataset. URL. htt.://ppp//m.ttiaarrrraaaaaaaaa
[92] James Large. Ethanol Concentration Dataset, 2018. httptttppee///w...i.eieeeeeeeeee eification. cwtccom///////// http:///////aaaae.o/o////attatte
[93] Rik Henson. Face Detection Dataset, 2014. URL http://tptt///....ttteeeeeeeeaee tion. http://///eeeorrpropppraaaaaaaaa pup.T/aataaeett-ccccceeeeci
[94] Benjamin Blankertz. Finger Movements Dataset. URL /tt.ttggg///....mmmerreeee..aaa
f fication.c ceter/iin..ommdsssiipiiiic httppp/wwwpppppaaaaaaaeoeaeeeeee
[95] Tephan Waldert. Hand Movement Direction Dataset. URL wtt.Lttts//wwwtiieesrerreee timeseriesclsss sification. haa.ariii..oo.ppppPaaatsaat-aaao
[96] Mohammad Shokoohi-Yekta. Handwriting Dataset, 2017. URL URL.tttpss//ww.ihheeerrr classification wtt.:/oo..comm//ere/ppeee /
[97] A Goldberger. Heartbeat Dataset, 2016. URL htt.tttppp.///...tteeeeeeeeeeeet
[98]
Time Series Classification Webpage. Insect Wingbeat Dataset, URL https//ww//
timese
r httpa//ssssaaatttooocoo///eeeers http://ccpppppppppoocaacceeeaaaa
[99] M Kudo. Japanese Vowels Dataset, 1999. URL https: http::tttttttttttteeeeeeeeeeeeee / on www//essrriptionnccooo phappp/btttttttt-aaaaeeeeeree.s. ottp////.rrrpoooo...aaaaaaaaaaaa
[100] D. Dias. Libras Dataset. htt.h//twwwwoo....eereeeeeeeeeee iption. wttp:paaastt--ibbrr...... ippiio..pppPPataaettLbaaa
[101] Kaggle. LSST Dataset. URL http:p//www.ttteeeecccecsssssoee tion. phppppapataass.-SS...
[102] Time Series Classificatiinn Webpage. Motor Imagery Dataset, URL https://www timese
[103] Time Series Classiffication Webpage. NATOPS Dataset, URL. https:////// timeseries
[104] Marco Cuturi. PEMS-SF, 2009. URL httpppp/wwww..teeer.rcceeaaaaaaa com
Dlecrriptionn.hhp7latttsssttt
[105] F Alimoglu. Pen Digits Dataset, 1996. URL httpp://ww..iineeccicclllaaaaaaa
[106] Hamooni. Phoneme Spectra Dataset. URI. http::::/www..iimeeeeellaaaaaaie
com/ wwt./o/ooo/iiie com./esccrptionnoo htt.::/ppp...pppppooaataaaaaapap
18

---

[107] Phillip Perks. Racket Sports Dataset. http:///pp//////.oooooooaaaaaaa.
om/description. pippoaataacc--ccccttttt..r......
[108] Thilo Hinterberger. Self Regulation SCP Dataset, 1999. URL wttp.ttttss////wtttteeesese.
classification http:///w.......o.oooo////aaaaaa
[109] Thilo Hinterberger. Self Regulation SCP / Dataset, 1999. URL http:///tttttttt/////teteeeeeeee
[110] N Hammami. Spoken Arabic Digits ShppTTattssttspppkeearrabaaagggt
2010. sification. arrficaaion..oodddessteeee www.///crrpptioonnnoooomo htt.p///ww...aaaaoo..oooooaaaaaa
[111] Jiayang Liu. U Wave Gesture Library, 2009, URL http://www/./////////oaaaaaaaaaa
tion. wom/descrippionn www.aaasetppaaaaaaette...... hagp:/aattaaetieaavveoeeeeterlar
[112] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb, Appliances
Energy Dataset, June 2020. URL httLttt::/eenooo.rrr//eeee org/recoodss/390267.
[113] Australia Rainfall Dataset. URL htw..i/pps///eaoooo.orr//e//////
[114] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Beijing PM10 Dataset, June 2020. URL
[115] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Beijing PM2.5 Dataset, June 2020. URL https://zenods: org/recoords/390277.
[116] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webh. Benzene Concentration Dataset, June 2020. URL hal..ttggs//eeaodd...r/reeeeee/s
[117] Chang Wei Tan, Christoph Bergmeir, Francois Petitjcan, and Geoffrey I. Webb. BIDMC Heart Rate Dataset (32 seconds window), June 2020. URL https:://enoo:d http:///oooooorg/eeerr////////// 1456.
[118] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. BIDMC Respiratory Rate Dataset (32 seconds window), June 2020. URL https:://eeeoii org/r ecords/4001463.
[119] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. BIDMC Blood Oxygen Saturation Dataset (32 seconds window), June 2020. URL http:: /ttp::dd////oooooooo org/r ecords/4001464.
[120] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Covid-19 Death Rate Dataset, June 2020. URL hirkllttts:///ndddooorggg http://nww......eeeeerra///////0
[121] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Flood Modeling Dataset June 2020. URL h
[122] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Flood Modeling Dataset June 2020. URL http:::://zennooo.ogr//eeeeeeee/
[123] Chang Wei Tan, Christoph Bergmeir, Francois and Geoffrey I. Webb. Flood Modeling Dataset June 2020. URL
[124] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Houschold Active Power Consumption Dataset, June 2020. URL https: 2704.
[125] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Household Reactive Power Consumption Dataset, June 2020. URL ott.i//oee///ooooooooooorooooooo wwwp:/oodeooeeeeeeee/e/oo 902706.
org/records/3
[126] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. IEEEPPG Dataset, June 2020. URL https: ptttps://zendda /ttp///eooo/////////////////oo./ org/records//9027100
19

---

[127] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. Live Fuel Moisture Content Dataset, June 2020. URL
6
[128] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffryy Webb. News Headline Sentiment Dataset, June 2020. URL httpp:://znooo.orrg//eeeeee otg//eecordds/3992/11
[129] Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey I. Webb. News Title Sentiment Dataset, June 2020. URL https://e/eooooo http::///emnoooorrrrrrroo//////
[130] Chang Wei Tan, Christoph Bergmeir, Francois Peritjean, and Geoffrey I. Webb. PPGDalia
Dataset, June 2020. URL cett:a///eaodo.ooggr/ero////////
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions and scope by clearly stating the claims in Section I, 2, and 3, aligning with results in Section 4, and acknowledging limitations, ensuring transparency and consistency with the findings in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The key limitations the proposed approach are briefly discussed in Section with more detailed analysis provided in Appendix B.4. We support our claims with diverse set of experiments and extensive ablation studies aiming to rigorously validate all claims made throughout the paper, refer to Section 4, Section 4.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not discuss any theoretical results, so we cannot provide corresponding
proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive documentation, including the necessary infor- mation and code, to enable the reconstruction of the results, refer to Appendix Upon acceptance of the paper, we will additionally publish online report detailing all doc- umented runs referenced within the paper. Due to anonymity concerns, this release is contingent upon acceptance to comply with double-bline review restrictions. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide access to most of this work's code and data. The code and data related to the DKT dataset, which is only small percentage, cannot be made public due to publishing restrictions of our sponsor. All other code and data are accessible. The code, data, and instructions on how to run and access the data are placed in the anonymous repository (https: /ttp:///noooouuu.// 4open. sciisiiiinnn ence// SSTaRFormer (armer-7788/) Most of our dataset classes will automatically download the respective raw data and apply pre-processing steps automatically. The data used is extensively documented in the appendix. We also provide all the links to access the public dataset used there: see Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines tattps:/nniss.co
wewgddeeeeoeeeiiieeeeeeeeeeeeeee http://eeuubmmmssiooooooi.ioo//o onPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nnis... cc/ccc/pp// http/uullcggguddee /esegguuddds//oeSuubiiii
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have documented configuration extensively using 'hydra' and corre- sponding 'yaml' files that should allow users to understand the full specification required to reproduce the training and test details. This is accessible in (https: hatps://anooymmuut 4open. wttteee////aaaarreeeeeeeo
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
23

---

7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In most of our experiments and ablation studies, we employed five distinct seeds to quantify uncertainty scores in the reported results. These are reported. For datasets and benchmarks where prior methodologies did not incorporate this approach, we align with existing literature practices. All runs, along with corresponding statistical measures such as the mean and standard deviation, where suitable, are documented in Appendix D.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details about the GPU types used for training. the training times, and the corresponding AWS cluster utilized. For additional information, please refer to Appendix D and the detailed documentation in Table 15 and 16.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: // http.ccccbpcc/cccicccc..
Answer: [Yes]

---

Justification: To the best of our knowledge, this work adheres to the NeurIPS Code of Ethics and complies with other relevant ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: At present, we do not foresee any direct negative societal impacts resulting from this work. Given its foundation in a real-world application, the proposed approach has the potential to yield positive societal benefits, such enhancing customer experience. However, at this time, further specifics cannot be disclosed due to confidentiality agreements with the BMW Group.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g., dep.lymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: At the current time, do not see any potential for misuse of our work, if pretrained models where to be released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
25

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We thoroughly ensure that all prior work referenced in this study is appro- priately credited and cited. For public data assets, all URLs are provided in the appendix, specifically in the dataset overview in Table 8, well in the anonymous repository https:/ //an:////nnomuus 4open
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcon com/ /nnaattsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This work introduces a new asset, the DKT dataset, which is comprehensively examined and detailed the extent permitted by the sponsor. Due its ongoing develop- ment, the asset remains confidential, and exhaustive details cannot be disclosed publicly. The asset will not be released to the public. Please refer Section 1, Appendix A and Appendix C.I.I.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
26

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLM tools were used for editing purposes only, such as checking grammar or paraphrasing elements and sentences of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non stannaandard components.
Please refer to our LLM policy for what should or should not be described.
https///ececoeeeeeeeeeeeeeeeeeee 220iceee////////// /LLM)
27

---

Glossary
? Determines the bound of the region be masked. 4, 39
X Mini-batch. 3, 4, 32, 39-41
Attention scores computed from the attention weights A. 4,
Number of Elements that can be masked. 4, 39, 40
Determines the number of sequential elements that are masked based on the attention scores a.
39
ADF Augmented Dickey-Fuller, 1, 31, 32
AE
Appliances Energy. 43, 53, 54, 62
AF Atrial Fibrillation. 43, 53, 60
AR Australia Rainfall. 43, 53, 54, 62
AWR Articulary Word Recognition. 43, 53, 60
BC Benzene Concentration. 43, 53, 54, 62
BCE Binary Cross-Entropy. 38
BIDMCHR BIDMC32HR, 43, 53, 54, 62
BIDMCRR BIDMC32RR. 43, 53, 54, 62
BIDMCSPO2 BIDMC32SpO2. 43, 53, 54, 62
BLE Bluetooth Low-Energy. 3,
BM Basic Motions. 43, 43,53.6 60
BPM10 Beijing PM10 Quality. 43, 53, 54, 62
BPM25 Beijing PM25 Quality. 43, 53, 54, 62
C3M Covid3Month. 43, 53, 54, 62
CCC Car Connectivity Consortium. 31
CE Cross-Entropy, 38
CK Cricket. 43, 53, 60
CL Contrastive Learning. 1-6, 57,
CNN Convolutional Neural Network. 2,
CoST Contrastive Learning of Disentangled Beasonal-Trend Representations for Time Series Fore-
casting. 3
CT Character Trajectories. 53, 60
DAReM Dynamic Attention-basee Regional Masking. 1, 5, 8-10, 38, 39, 42, 55,
DDK Duck Duck Geese. 8, 43, 53, 60
DK Digital Key. 1, 31, 43
DKT Digital Key Trajectories. 1. 2, 7,9, 10, 23, 26, 31, 42-44, 47-51, 55, 56, 61
DL Deep Learning. 2, 34, 37
DTWD Dimensiion-dependen dynamic time warping. 2, 7. 60
EC Ethanol Concentration. 43, 46, 52, 60, 61
EP Epilepsy. 43, 53, 60
ER ERing. 43, 53, 60
EW Eigen Worms. 43, 46, 52, 60,
FD Face Detection. 43, 46, 52, 60, 61
28

---

FM Finger Movements. 43, 53, 60
FMI Flood Modeling 1. 43, 53, 54, 62
FM2 Flood Modeling 2. 53, 54, 62
FM3 Flood Modeling 43, 53, 54, 62
GL Geolife. 6, 7,9, 10, 51, 61
GRU Gated Recurrent Unit. 2, 36, 49, 51, 55
HB Heartbeat, 43, 47, 52, 60, 61
HMD Hand Movement Direction. 43, 53, 60
HPC1 Household Power Consumption 1. 43, 53, 54, 62
HPC2 Household Power Consumption 2. 43, 53, 54, 62
HW Handwritting. 43, 47, 52, 60,
L.d.. Independent and identically distributed. 33
IEEEPPG IEEEPPG. 43, 53, 54, 62
InfoTS Time Series Contrastive Learning with Information-Awart Augmentations. 3, 7, 10, 60
IW Insect Wingbeat. 43, 53, 60
JV Japenese Vowels. 43, 47, 50, 52, 60, 61
KPI KPI. 8, 43, 47, 52, 61
KPSS Phiiatows---hlllls-Shmit------ 1, 7,
LFMC Live Fuel Moisture Content. 53, 54, 62
LI Libras. 43, 53, 60
LSST LSST. 43, 53, 60
LSTM Long Short-Term Memory.
51,
MI Motor Imagery. 43, 53, 60
ML Machine Learning. 1, 3,
MLM Masked Language Modeling. 2
MLP Multilayer Perceptron. 34-36, 38
NHS News Headline Sentiment. 43, 53, 54, 62
NT NATOPS. 43, 53, 60
NT-Xent Normalized temperature-scaled cross entropy. 3,
NTS News Title Sentiment. 43, 54, 62
OAT One-at-a-time analysis.
P12 PhysioNet Mortality Prediction Challenge 2012. 46, 51, 60, 61
P19 PhysioNet Sepsis Early Prediction Challenge 2019. 7343, 45, 51, 59, 61
PAM Physical Activity Monitoring. 9, 10, 40, 43, 46, 51, 59, 61
PD Pen Digits. 43, 47, 52, 60, 61
PPG PPG Dalia. 43, 53, 54, 62
PS PEMS-SF: 10, 43, 47, 52, 60, 61
PSp Phoneme Spectra. 53, 60
29

---

RM Random Masking. 8, 52, 53, 57-61
RMSE Root Mean Squared Error. 48
RNN Recurrent Neural Network. 3, 7, 51,
RS Racket Sports. 43, 53, 60
SAD Spoken Arabic Digits. 43, 47, 52, 60, 61
SCP1 Self Regulation SCPI. 43, 47, 52, 60,
SCP2 Self Regulation SCP2. 8, 43, 47, 52, 60, 61
STaRFormer Semi- Superviuee Informee Representation Learning Transformer. 1-10, 36,
38 38-42, 48-53, 55-62
SVM Support Vector Machines. 7, 56
SWJ Stand Walk Jump. 8, 43, 53, 60
t-SNE t-Distributed Stochastic Neighbor Embedding. 9, 10
TARNet Task-Awar Reconstruction for Time Series Transformer. 2, 7-9, 51, 54-56, 60
TCL Time-CL. 3
TRL SeperSuuee Contrastive Learning for Universal Time Series Representation Learn ing. 3, 7, 8, 10, 51, 55, 56,
TNC Temporal Neighborhood Coding. 3, 7, 60
TS2Vec Towards Universal Representation of Time Series. 3, 8, 60
TSR Time Series Extrinsic Regression. 8, 42, 62
TST Time Series Transformer. 2. 7, 46, 51, 60
UEA UEA. 10, 42, 46, 50, 52, 53, 60
UW UWave Gesture Library. 43, 47, 52, 60, 61
UWB Ultra- Wideband. 1, 31, 32, 43
VITST Vision Time Series Transformer. 7, 60
Yahoo A Labeled Anomaly Detection Dataset. 8, 4,43,,45 47, 52,6 61
30

---

Appendix Supplementary Material
A
Localization and Tracking via UltraWiideband Technology and the Digital Key
The digital key (DK) enables the use of a smart device as 'vehicle key' [5. 7]. facilitating handsfree or passive access to vehicle through the smart device. The DK technology is standardized by the Car Connectivity Consortium (CCC), led by Apple, the BMW Group, Ford, Google, Mercedes, Xiaomi, and other global corporations [9]. In recent years, car manufacturers have started to incorporate (UWB) and Bluetooth Low-Energy (BIF) technologies to enhance the capabilities of the DK [4-8]. This allows for precise and secure vehicle access while paving the way for the creation of additional applications for connected vehicles.
A.1 Localization Non--tationary Characteristics
A Bluetooth connection is initially established between the smart device and the vehicle to detect a paired personal smart device nearby. Following the exchange of security protocols, an UWB connection is set up enable secure ranging of the smart device. The vehicle is equipped with multiple UWB anchors. Between each UWB anchor and the smart device, time-of-flight measurements executed, allowing precise localization due to UWB's pulse duration of 2ns [9]. When the localization is recorded, one is able to track the smart device around the vehicle, enabling intent predictions based on the sequentially collected localization measurements. However, various external factors can influence the localization accuracy, including materials of different vehicle models, external environments like weather conditions, interference from other signals, and the position of the smart device (e.g., in hand, front pocket, or handbag). These interferences can introduce non-stationary characteristics to the sequential data. To verify this hypothesis, we compute the KwiatkowskiiPhiile Sitii----iiipss--iii----- Schmidt-Shin (KPSS) and augmented Dickey-Fuller (A (ADF) tests [64, 65]. We consider a time series non-stationary if both tests agree, i.e., the p-values for KPSS are smaller than a
(b)
07
20-
(c)
(d)
5
Figure 3: Example plots visualizing the non-stationary characteristics of the sequential data in the DKT dataset. The red or orange line visualizes the mean and the green dashed lines the standard deviation of segment. Multiple mean and standard deviation lines per plot indicate changes in the underlying generative distribution of the visualized data. These plots only serve as a demonstration and visualization of the non-stationary characteristics of data samples from the DKT dataset.
31

---

Ry (288mss)
10
10
11
A, 120me
A.R] = 381ms
Figure 4: An illustration demonstrating the collection and utilization of UWB signal measurements to localize a smart device around the vehicle using a multiplier RAN = Due to the continuous hopping strategy for ranging, fixed ranging round indices set before the data is collected. In this case, ranging round indices [9, 2, 6] lead to a difference in the time delta between three following 264ms [9].
Rn Ln'ss.e e.g. R A.R
significance level, i.e., PKPSS < 0.05, whereas the p-values for ADF are larger than significance level, i.e. PADF > 0.05. The results suggest that 79% of the data provided us by the BMW Group used for training and validation is indeed non-stationary. Fig. 3 depicts six examples visually. which display non-stationary characteristics the time series data the DKT.
A.2
UWB Ranging and Measuring
The measuring algorithm employed for ranging introduces another level uncertainty. The UWB signal collection for localization occurs within time window referred to ranging block. A ranging block, Rg, defined as Rg = multiplier RAN x Tmin, where multiplier E {1...., 255} and Twin = 96ms [9]. The Rß,, where i = 0, - 1, is divided into M ranging rounds R where 0, 1..... M 1. Although these RR fixed, the specific Ri-- -iddxx used for recording varies per ranging block, to reduce the probability of interference in multi-device scenarios. Different strategies have been proposed for selecting a -inddeex;; in the data provided by the BMW Group, a continuous strategy is employed such that the R2 index for each Rig is predetermined. However, the differences between consecutive R 's A, R R do not necessarily match the differences in the following window, & = R R [9]. Fig. 4 visualizes the UWB ranging algorithm and the resulting irregular sampled sequential data conceptually.
B Approach
Notation, We extend the formulation from Section In this work, the following notation is used: tensors and matrices are represented by capital letters A and column vectors are represented as Given matrix A, one can access the i-th, j-th element as Ai,. To access a row A, one can slice the matrix A,; = (+) and vice versa A,; to access column. For 3-D tensor A, element (i, j, k) is Aij.k. The i-th element of column vector is 0g Scalars are depicted
The dataset used to train machine learning algorithm is noted as D, where in general (x((), (9) is the i-th sample-label pair of D in the supervised setting. ÿ denotes the predicted labels by a function f. As previously defined, mini-batch (X), X 'D and X Bxx is 3-D tensor where B is the batch-size, N the sequence length the feature dimension and the B < M. Hence, one can access the i-th element of the mini-batch as x. which is the same as S(t), A set denoted as A. where set of real numbers for example. Special notations are the indicator function, II,
1 if A 0f A
(x)
8
and the identity matrix L, (n m))matrix. The symbol O is the Hadamard (elementwise) product operator, and the symbol @ is a tensor product operator.
B.1 Baseline-Models
In machine learning, sequential data requires specific modeling techniques because the order and context of the data points significantly influence the overall meaning and patterns, necessitating mod- els that can effectively capture and utilize temporal dependencies and relationships. The sequences'
32

---

(a)
(b)
Figure 5: Illustration of (a) a sequence using a first-order Markov chain and (b) a sequence using Markov chain of latent variables.
intrinsic order is crucial for conducting analysis and making predictions. Therefore, the generally applicable independent and identically distributed (i.i.) assumption, is not suitable [43], as a current state in sequence depends on its preceding states. Hence, accurately model a sequence, one would aim to compute the joint probability of all elements in the sequence, i.e.
P (s) 82
(1) SN
P(s)) ( II (s
( (1) 18 ,2
Naturally, recent observations will most likely provide more insights into future predictions than historically older observations. Additionally, it is not feasible to assume that future observations depend on past observations [43]. The simplest formulation, Fig. 5(a), and hence ignoring the intrinsic order of the sequence, applies the Markov assumption stating new state is only dependent on the current state, i.c.,
(1) 8 8--1
(10)
j+1
Therefore, the joint probability is
N
P (
( P(s
II P j=2
(s
( ()))
(11)
To enable prior observations to impact the modeling, one can transition to utilizing higher-order Markov chains, considering greater number of preceding states. However, this will lead to exponentially growing number of parameters the model requires, rendering it impractical.
To consider the more complex intrinsic order in sequences, latent variable model that is not limited by the Markov assumption can be used, refer to Fig. 5(b). latent variable model permits the creation of rich model out of simple components. In this approach, a latent variable or hidden state, - stores the information of the sequential steps up to j 1, while still satisfying the conditional h such that the joint probability distribution is h
independence property [43], h h
I1
(6)
P
22
(8) N
h 10)
h N
P (h (8)
II P h
b
h
I (8
h
(8)
1=1
(12)
Intuitively, when examining sequence, one goal might be to predict the next value that might occur in the sequence. This can achieved, for example, by evaluating the expected value of the likelihood of new state s) Let's define random variables for the subsequent state (Y())) as ( sj and
the sequence of preceding states (X (i) as X (i)
13-1.
Y X
E
1
(13)
Then, for example, linear regression model can be employed to estimate the conditional expectation, E, as follows,
(
x (i)
f(x (0) (0,0) + +e",
(14)
where is Gaussian white noise, N'O,, 3-1 [43]. The model f.
N
f(x (1) () Σ
rT 1 (1)
(15)
a linear combination of its parent nodes, which is known autoregressive model [75].
33

---

B.1.1 Recurrent Neural Network (RNN)
Elman [32] developed a unique modeling technique, referred to RNN, which is specifically designed to capture and utilize the temporal dependencies and relationships inherent in such data. Note that RNNs are increasingly being replaced by Transformer-based architectures, which can be more efficient at processing sequential data due to their parallel processing capabilities.
A RNN [31, 32] is a deep learning model that is trained to process a sequential input and convert into a specific sequential output. Sequential data refers data, such as words, sentences, or time series data, where sequential components are linked together based on complex semantic and syntactic rules. RNNs manage sequence dynamics through recurrent connections, which function like cycles within the network recursively evaluating the sequential elements. These recurrent connections are unrolled across sequential steps, applying the same parameters each step [76], as illustrated in Fig. 6(a). While standard connections propagate activations synchronously within the same sequential steps, recurrent connections transmit information across adjacent sequential steps, also shown in Fig. 6(a). RNNs can be seen feed-forward networks or multilayer perceptrons (MLPs) with shared parameters across sequential steps, typically representing steps in time. Sequentiality is not exclusive to RNNs; for instance, Convolutional Neural Networks (CNNs) can be adapted for data with varying lengths, such as images of different resolutions. Although RNNs have recently been overshadowed by Transformer models, they remain essential for complex sequential modeling, For more detailed discussion on RNNs, refer to the comprehensive reviews by [76] and [77].
h
Au
A
tanh
unroll
A1-1
(a) Unrolling computational cycles in a RNN,
(b) Computational logic in a RNN cell.
Figure 6: Recurrent Neural Network: (a) illustrates the high level computational concept of cycles in the networks used in RNN and how they are unrolled. Recurrent connections are highlighted in red. (b) illustrates the computational logic of each cycle applied as neural network for an adjacent element of the input sequence.
Adopting a Deep Learning (DL) perspective, the sequence in Fig. 5(a) could also be considered a computational graph, for which Eq. (16) describes the recurrent recursive computation [76].
5) (
f()
(16)
This builds the foundation of the RNN architecture. In a similar fashion to the latent variable model described previously, a RNN often formulated using a hidden state, h which stores the information of the preceding states in a higher dimension. This allows the hidden state to be calculated any step by the hidden state of the previous step and the current state, i.e.
h = (5)) 10 0)))
(17)
the function f is sufficiently powerful, the latent variable model can be exact, as h can store all previously observed data. However, this can lead high computational and storage costs. A simple deep neural network, MLP layer, is used in RNNs to describe the function f that allows to compute
(6) hidden states h that are used to approximate the likelihood, P
(s} [s]
(6) (4) (6)
| )/11)
a
OT N,n
s (1) + bia + 0n,h + by.a
(18)
h tanh
a
(19)
As defined in Section 3, where D is the dimensionality of sequential element of the input sequence and H the dimensionality
( R hence, Oia E DDH Or,a R and b bia E R
34

---

should affect of the hidden state. Based on the hidden state, simple MLP layer can be applied to compute an output.
0f' = % h bb,o:
(20)
where On.o E HxF and ba. E R F defines the dimensionality of the output. This allows to compute the conditional expectations of the current state s For example, to compute a classification,
U one would apply the softmax the outputs, úن softmax(o, ). The computational logic of
the aforementioned mathematical formulations are schematically displayed in Fig. 6(b).
B.1.2 Long Short-Term Memory (LSTM)
The Elman-style RNNs encounter difficulties in learning long-term dependencies, identified by Elman [32]. These challenges, articulated by [78] and [79], arise due to vanishing or exploding gradients during backpropagation. In lengthy sequences, recurrent computations
repeatedly applied to the weights. Since these weights are shared across sequential steps, if O2 < 1, it results in vanishing gradients, whereas if i. 1, it leads to exploding gradients [76]. While gradient clipping mitigates exploding gradients, vanishing gradients require more sophisticated solutions. One of the earliest and most effective methods to address vanishing gradients is the LSTM model introduced by [1]. LSTMs are similar to standard RNNs but replace each recurrent node with a memory cell. Each memory cell includes an internal state with a self-connected recurrent edge, allowing gradients to propagate across many time steps without vanishing or exploding. The term 'long short-term memory' reflects the model's ability maintain both long-tent memory, through slowly changing weights that encode general data knowledge, and short-term memory, through transient activations passed between nodes.
LSTM cell contains an internal cell state, denoted as c,"
It includes several multiplicative gates:
the input gate, 1/0 decides an input, determined by the input node gate, Bj (1) the internal cell state, of
the forget gate, r3 determines if the internal cell state, oj should be reset, and
the output gate, o controls whether the internal cell state should influence the cell's output. h hn)
As in the RNN, the input gate, the forget gate, the input node gate and the output gate are latent variable model in the form of MLP layer, with fully connected layers for the input s/ and the hidden state follows:
(4) - where denotes sigmoid activation function. Hence, the gates are described as
0 ( ( + ba. + Hn. + bb.r)
(21)
1 a (8) + b,, On, 1:) by.))
(2)
(22)
8
( tanh (0
(1) + b + On.hh + bl..))
(23)
( (
+ b + Onoh( 1- + bn,o)
(1)
(24)
( r
(1) og
(1)
(25)
h i
o (
tanh(c
(26)
DD. hence, Or,,, Or,, Oigg ii, DDH Or,f, On.s, On, On, H and ba bi. bi.gs D R and bp b biogs bb E RR The computational logic of the afore- mentioned mathematical formulations are schematically displayed in Fig. 7()).
Given 8/
35

---

+
Fanhy
Az42
tumb
0
tanh
A
A
(a) Computational logic in LSTM cell.
(b) Computational logic a GRU cell.
Figure 7: Illustration of the computational logic of each cycle applied as a neural network for an adjacent element of the input sequence for LSTM cell (a) and GRU cell (b).
B.1.3 Gated Recurrent Unit (GRU)
The Gated Recurrent Unit (GRU) [14] provides a streamlined architecture of the LSTM memory cell, retaining internal state and multiplicative gating mechanisms while speeding up computation and, often, achieving comparable performance [80]. In GRU, the LSTM's three gates replaced by two: the reset gate, (6) and the update gate, u
The reset gate controls how much of the previous state to remember, while the update gate manages how much of the state is copy of the old one. As in previous RNN models, the gates' outputs are produced by MLP layer with sigmoid activation function. Similar to the input node gate in the LSTM memory cell, the new gate, n (4) computes the temporary hidden state. The information used from this state is determined by the reset gate. The final hidden state is determined
by incorporating the update gate, u,") and the new temporary hidden state, n, Specifically, the
(6) and the final hidden state, h. (t) is computed weighted sum of the temporary hidden state, n)
old hidden state, h with the update gate, u (i)
(i)
serving the weight.
(0)
O. (1) -birr On, + bb.r)
(27)
u
(1) OT, ))
(1) bi,u + On, (3)
j-1 + bb,u
(28)
n (i) tanh
(eT S
(8)
+ b + +ry
(1)
(O).n (2)
bb.n))
(29)
h = (1
① n
+ 1hh j-1
(30)
Given 8) R. thus, Or.r, Oz.. O RDDH Ob.r,, Ou, On,n E #RH and b bi b R and ba bb. Da,n H The computational logic of the aforementioned mathematical formulations are illustrated in Fig. 7(b).
B.1.4 Transformer
The seminal Transformer architecture in [2] includes encoders and decoders. Since then, there has been tendency to use decoder architectures mainly for generative tasks [81-83] and encoder architectures for tasks requiring understanding [42, 84, 23, 21]. STaRFormer follows this trend and chooses only Transformer encoder layers as the central component. Additionally, encoders allow for general framework for learning task-specific reconstructions that can be applied to wide range of tasks. allows one to handle any task, such as classification, regression, generative forecasting, or anomaly detection, by simply adjusting the output layer the latent embedding gets passed to.
36

---

As introduced in [2]. the encoder layer consists of two sub-layers, a multi-heat self self-attention mechanism, and fully connected neural network. Both layers are followed by a residual connection and a normalization layer. The self-attention in [2] is mechanism that allows each element of sequence to consider the entire sequence when computing
representation. This capability helps the model to grasp the context surrounding each token in sequence, making it highly effective in sequential data tasks. This allows the model to effectively capture both long-term and short-term dependencies within the sequence, This feature addresses the limitations of previous DL approaches like LSTM [1] or RNN [32], which can struggle with capturing such dependencies.
In order for attention to work, each sequence has been embedded as vector representation. Then, series of query (Q), key (K) and value (V) terms are formed: Q is representation the model focuses on, K determines the relevance of each element, and V is representation used to form output scores. The Scaled Dot-Product Attention approach in [2] computes weighted sum of the input values with the attention weights, where the weights are determined by the similarity between input elements computed via the softmax function. The attention scores are normalized by the square root of the dimension of the key vectors to stabilize gradients during training, i.e..
QK? VTx
Attention( (Q, K. V) softmax
V
(31)
where Q. E RNNxBxD and dg is the dimension of key vectors. Often, not just single self-attention mechanism is performed, but rather mechanism referred to multi-head attention. In multi-head attention, the queries, keys, and values are linearly projected Thead times and then concatenated and projected to the model's embedding dimension, i.e.,
Abbond
MultiffeadAlleentionn (Q, V) =
OHH H
W
(32)
where Whead is a tunable hyperparameter and H;; = Attention (QW? KWK, VWY) represents 8 an attention head. Here, W Rddast dd, w.k E Rddms Xdy, Wy E Rdmessd and w. R Maradd denedd are projection matrices where d, indicate the respective dimensions.
necessity for Transformer models is the encoding process of the sequential inputs. When the sequential input is vectorized, the imput representation loses the sequential information, i.e.. the order of the sequence. Hence, an underlying property of the data type lost. This is why it is essential to inject the sequential information about the relative or absolute sequential position into the vector representation. To do so, [2] introduces sinusoidal positional encodings, which are added to the encoded sequence before it is passed to the Transformer encoder layer, i.e.:
(p),,,,)) 21) sin ppos/100002/ dmonde)
(33)
P (p((p cOs (pos/ 10000 25/0 3/damodel
(34)
As mentioned before, in certain time series, the sequence lengths N can vary. Consequently, to process batches of sequences with differing lengths, padding is necessary. To ensure that padded elements are not considered during the attention mechanism, it crucial to introduce a batch-wise masking strategy. This mask is passed the attention process prevent artificially padded elements from being attended to, thereby preserving the integrity of the sequential data. This mask might further be required in different output heads or loss formulations.
37

---

B.2 Downstream Tasks
Classification. Instead of performing autoregressive predictions [21] or predictions based on the concatenation of the entire embedded representation [23], special token is used ins FTaRFrrmer for classification tasks. This is a design choice and no requirement. The token effectively captures the dependencies between sequential elements via the encoder's self-attention mechanism. When performing classification tasks, a multilayer perceptron (MLP) performs the classification based on this token. The reduced version of the MLP layer used in STaRFormer, is fully connected neural network followed by sigmoid or softmax activation, depending if multi-class or binary predictions are required, i.c.,
y = oZers + b),
(35)
while the default version adds an additional activation layer and a normalization layer, i.e.,
ÿ ((Norm(((Zecss)) + b))) b).
(36)
where ÿ E BRBK ZCLS Zo.:: E RBF is the specialized token of the latent embedding and O E RPX and b E R are the model weights and bias, where K defines the output dimension of the final layer. The inner activation function, is a tuneable hyperparameter. The output of the MLP layer, i.e., the prediction (00, is passed through either the cross-entropy (CE) loss function for multi-class predictions, Ltask = Die ( log()()), or the binary cross-entropy (BCE) loss for binary predictions, Ltask = y0d (og((()) + (1 - (o)((())
Anomaly Detection. The same output head configuration is employed for anomaly detection, with only the output dimensionality K adjusted accordingly. In this setup, dedicated classification token is unnecessary, predictions are made at the element level. Therefore, the raw latent embedding E RBxNxP serves as the input to the output head. Moreover, as the sequences possess element- wise anomalous labels, anomaly detection tasks requires predictions at each time step. Consequently, the model must generate N predictions for an input sequence (()) E ND hence ý E RR BBN
Regression. The objective to predict scalar value for each sequence, as defined in Section 3. We adopt the output-head configuration used for classification, but fix the output dimension to K = 1. Moreover, the model output does not require an outer activation function, such as softmax, since we aim to predict scalar values. Accordingly, the reduced version is
20 + b.
(37)
The default version incorporates an additional hidden layer, i.e.,
Norm (a (Norm(a b)) + D))0 + b.
(38)
Here, 2 E R Rx is computed via an average pooling operation over each sequence, i.e., the mean per sequence, and ý E R denotes the predicted scalar value per sequence in batch.
B.3 Semi-supervised Task Informed Representation Learning in STaRFormer
Fig. display the Contrastive Learning (CL) approach applied in FTaRFormer. The CL approach ultimately depends on the accumulated latent representations, Z and Z, that are created via the Dynamic Attention-based Regional Masking (DAReM) scheme. During the computation of the multi- head attention, while executing training of a downstream task, STaRFormer dynamically collects the attention weights, A. To aggregate the attention, STaRFormer employs a slightly modified attention rollout technique [58] rather than mere summation or aggregation. This approach enables better consideration of the flow of information within the Transformer layers. The operation accounts for padded sequences by masking the attention if necessary, i.e.,
(A O M+ 1IN) @ A A
if i 0, i
(39)
O M
where A E B represents the aggregated attention, and M the mask accounting for the padded input sequences. STaRFormer then computes the attention scores as in [21], i.e.,
Σ/ Arj.
Oi,k'
Σ
N
N Σ A A.jjk
(40)
38

---

Z
Maximize hutch-
Z
0.00
wise & class wiss agreement
f(-)
f()
0.10
0.15
X
X
0.20
9(C, 7.4;))
0.25
7 0.30
Figure 8: FTaRFormer's CL approach: The masking, generates two correlated views. The encoder, f, is trained maximize the trade-off between batch- and class-wise agree- ment of the latent embeddings and Z while training for downstream task.
Figure 9: Seven different regional masks for the same batch, with sequences aligned horizontally and stacked vertically (per X). The masked re- gions in DAReM are defined by different values of 7, with yp 0.2 and 0.3 held constant and depicted in yellow.
Algorithm 1 Dynamic Attention-based Regional Masking (DAReM)
Require: A. n. 7.
A t attention-rollout (A)
+ attention-scores(AA mask-indices {-
for i in range (B) do
Otop +- topk(Ç, n;)
Otopl +- topp [0]
btop1 <- n;
mask-indices, +
m, <- range(max(0, topl btop1), min(n;; Stop1 + btop1 1))
if (mmask-indiccss.) n, then
for i, Took in enumerate(Orop [1:) do if 1en (1e(maak-indices) S n; (p then
beop f n1 -
mask-indices,.xteeddd/m
end if end for end if'
if [mnamask- indices,) < n, +p then
my + random (aaaamaalaallliiiiiiiio nlait) mask-indices,.extenddl end if
mask-indices.appe appeedd (nakkiiiiiss end for return mask-indices
where As..k is the attention weight assigned s during the update of s. in Eq. (31). greater oi value indicates higher importance of the k-th element in S()) Algorithm 1 implements the arithmetic's 's of the DAReM scheme introduced in STaRFormer. In the algorithm, A refers to the attention weights collected from the multi-head attention layer in the encoder, n refers to an array stating the sequence lengths per element the mini-batch and B refers to the batch-size of the mini-batch. The masking parameters sp, and ( are introduced in Section 1. We illustrate several regional masks for the same batch, with the region parameter ? varied while yp and Ç remain fixed in Fig. 9. To note is that if the region of the most important sequential element is already
39

---

(a) batch-wise
(b) class-wist
(c) batch-wise
(d) class-wisc
Figure 10: Example visualizations from two different batches of the PAM dataset: images (a) and (b) are from one batch, while images (c) and (d) are from another batch, both have a mini-batch of size 32. Positive pairs within each batch are color-coded. The darkest shade represents negative pairs.
(a) semi- supervised
(b) supervised
(c) self- supervised
(d) semi- supervised
(c) supervised
(f) self- supervised
(g) semi- supervised
(h) supervised
(i) self- supervised
(j) semi- supervised
(k) supervised
(l) self-
supervised
Figure 11: Similarity heat maps illustrating the contrastive loss formulation in STaRFormer for two mini-batches of size B 32 from the PAM dataset. The top row displays similarities between latent embeddings ()) and (x) for untrained model, while the bottom
shows similarities between (0) and (1) for trained model. Plots (a)-(c) and (g)-(i) pertain one batch (same batch as in plots (a) and (b) Fig. 10), whereas plots (f)-(f) and (j)(() pertain to another batch (same batch as in plots
(c) and (d) in Fig. 10).
greater than threshold s., only that region is masked, and the other selected values are dropped. If all important sequential regions are masked and the threshold still allows samples be masked, then random samples are selected using the available indices of a. We opt to select the following bounds for the masking parameters: & E (0.0, 0.5), 7 (5j x 10-2 E {0, 1, 4,4, 5)5 and ç= = 10-1 | j
In Section 3.1.2, we introduce the implementation of semi-supervised CL paradigm as employed the STaRFormer framework. This methodology exploits the inherent batch-wise and class-wise agreement between masked (2) and unmasked (Z))) latent representations allowing to facilitate semi-supervised CL. Fig. 10 provides visual illustration of positive pair selection under the semi-supervised CL framework, using two distinct batches from the Physical Activity Monitoring (PAM) dataset. It demonstrates the construction of batch-wise and class-wise positive pairs for contrastive learning. For batch-wise pairs, the corresponding diagonal elements are selected (Fig. 10(a) and 10(c)), whereas for class- wise pairs, the corresponding diagonal and off-diagonal elements are selected.
In Fig. 11, the results of deploying different possible CL. paradigms are graphically represented. Specifically, as discussed in Section 4.4.2, we analyze three different learning paradigms: semi- supervised, supervised, and self-supervised. Fig, 11 is continuation of the visual analysis initiated
40

---

in Fig. 10, employing the same two batches for consistency. The upper row of Fig. 11 presents the similarity heat maps for a model prior to training, while the lower row illustrates the heat maps post-training. A color-coded scheme is utilized to convey similarity levels, with yellow indicating high similarity and purple denoting low similarity. The matrices' diagonal entries quantify the self- similarity among batch elements. In contrast, the off- entries measure the degree of similarity between disparate batch elements. The two rows within Fig. 11 visualize the learning objective of CL as described in Section 3.1.2 and Fig. 8, where similar samples are pulled closer together, while dissimilar samples are pushed further apart. In the top row, the untrained model evaluates relatively high similarity across all element pairs within the batch, regardless of the CL paradigm used, whereas the trained model clearly distinguishes between similar and dissimilar samples in the batch in accordance to the contrastive paradigm applied.
When the model is trained to prioritize batch-wise similarity under the self- eeff-superviee contrastive learning paradigm, the heat maps reveal, as expected, brightly colored diagonal entries in yellow, indicating the intended emphasis on self-similarity. This illustrated in Fig. ii and 1 11)... Con- versely, the off-diagonal entries are predominantly cast in darker shades ranging from blue to purple, indicating a stark contrast in similarity and, thus, a clear distinction between different elements.
Training with an emphasis on class-wise similarity yields a different pattern; refer to the heat maps in Fig. 11(h) and 11(k). Here, addition to self-similar elements, the heat maps distinctly accentuate high similarity among elements belonging to the same class, while elements of disparate classes are clearly differentiated by lower similarity scores, reflecting the model's class-wiss learning.
The semi-supervised training paradigm offers a composite view, where the model demonstrably assimilates both batch-wise and class-wise similarities (see Fig. (1(g) and 11))). This dual learning is evidenced by the pronounced similarity not only along the self-similar diagonal entries but also between class-aligned, diagonal, and off-diagonal elements. However, the off-diagonal entries not associated with class similarity do not display the darkened hues observed in the strictly supervised model. This absence of dark hues suggests more tempered and generalized learning process, where the model avoids overfitting to specific batch-wise or class-wise similarities, potentially achieving a more holistic representation of the data.
B.3.1 Additional Information Formulation
The resulting cosine similarity matrix computed from the reduced latent space representation Z and É., represents the inter-batch similarity between all sequences in a batch. Hence, elements at in the similarity matrix originate from the same input sequence S Thus, position where given Zi.j E IR BxF, STaRFormer can form B positive and B(B 1) negative batch-wise pairs.
By having C' classes per X, we obtain Ee n2 positive and [Ee class-wise pairs; ne the number of samples per class per X.
c==1 ne) - (Ee n negative
B.3.2 Additional Information Formulation 2
As stated in Section 3.1, the element-wise formulation allows to create intra- and inter class positive pairs. Intra-class positive pairs are created between elements within a sequence whereas inter-class positive pairs are created between elements with other sequences in mini-batch.
We chose to select an inter-class positive pair if ((.))
Inter-class.
inter. [x(")__((")) is 1, i.c. the class
in the left and right label is equal each other, non-negative and the elements from the same sequence. The following equation, for completeness, defines the indicator function applied for the inter-class class-wise contrastive loss formulation in Section 3.1.2.
if #6, Y( A A((0)>> > - Y -1 (41)
(b.t) inter, (x
otherwise
This definition accounts for padded elements, where the label tensor equals -1.
41

---

Intra-class. We chose to select an intra-class positive pair if ((G.))
intra, [x( = is 1, i.e., the
class in the left and right label is equal to each other, non-negative and the elements are from the same sequential input element. In this case, a few modifications are necessary. The left and right label tensors are created as Y1 E BxxNx1 cosine similarity needs to be computed between each element a sequence; thus, we require a three-dimensional similarity matrix. As described in Section 3.1, this requires X bmm in the similarity computation. Thus, the cosine similarity is defined as:
x1 and Y, E R respectively. Additionally, the
simintra (U,V) = U hmm V
(42)
V
The latent embeddings Zerrm and Zperm are permuted equivalents of Z and Z. where Zperm B and Zperm E RBXDxN For completeness, the following equation defines the indicator function applied for the intra-class class-wise contrastive loss formulation in Section 3.1.2.
1
if = Y(UJ) A A -1AY -1
(6.5) intra. ( vlal]
(43)
otherwise
This definition accounts for padded elements, where the label tensor equals -1.
B.4 Limitations of STaRFormer
A key limitation of the proposed approach is the computational overhead introduced by the attention- based masking mechanism (DAReM), which requires the computation of attention weights scaling with O(N2) complexity, which becomes increasingly computationally expensive proportional to the length of the sequences. Additionally, integrating CL and DAReM during training further increases training time and computational demands, as each input must be processed two times top of the increased workload by applying CL in the first place. However, these overheads are confined to the training phase and do not impact inference performance, where only the downstream prediction task is executed. Despite the additional computational overhead introduced by CL and DAReM, STaRFormer maintains comparable batch sizes during training. For instance, ForRer trains with batch sizes of 512 and 256 on DKT and Geolife (GL), respectively, matching those of the baseline transformer model. In practice, batch size limitations are primarily constrained by sequence length due to the quadratic complexity of attention- weight computation.
The outlined limitations become more pronounced at scale, particularly when training on large datasets such as DKT. While the additional computational cost negligible for small datasets, becomes significant factor during large-scale training, especially in the context of hyperparameter tuning. Although this work prioritizes predictive performance over computational efficiency, we acknowledge the potential for optimizing the implementation of DAReM to mitigate training overhead and more efficient attention computation to improve scalability.
C Datasets
In Table 8, we display the different attributes of each dataset used in this work. For GL and DKT, we use five different seeds to ensure fair evaluation of STaRFormer's performance, For DKT, we keep the test set fixed across all splits. As default, we set the seed to 42 and use 123, 0, 63, and 2024 additionally. For the benchmarks of GL, the UEA benchmark (UEA) [70] and the TSR benchmark (TSR) [73], we only report the best performing model in the paper of a single seed, be consistent with previous literature.
42

---

Table 8: Time Series Datasets Overview
Litesuae
Lask
Nev
Dignal Roy Trgectarles (DKT)
447.705
111.544
677
Suskiray
Goille
6.434
1.556
7,590
10
[63]
gectib-link
krozelaly
Sarpted
Phyeral Actiity Meesioiig (HAM)
Ply Phyandeen Menulty Predction Chaleags 2012 (P12)
toely
10,200
4799
1.199
335
5001
36
17
[67]
[64]
p12-ank
Acteciary Recogation (WWR)
275
[85)
Fibtiliatia
Baak Motiso (BXG
40
40
100
[67]
lick
Dhanzam Trjearnes (UT))
1,223
1/416
20
182
[88)
10
108
1.97
1871
12
Worms
131
12K
17:86
189)
ensiin
15
Eppllpoy (BIF)
137
138
216
190]
op lnt
Eittin (Bi))
1911
17
Closii
Flaget Miserionn (FM)
Face Detoction (FD)
5890
316
3524
100
62
50
28
1941
1931
fatick
18
Hand Moxcenous Dioction (HMD)
300
14T
400
10
1951
21
Regiar
Irose Wingher (TW)
Houtheat (718)
30
204
70,00
106
10
75
700
61
1981
1971
Shilik
Inclic
Seporse Vioels (IV)
770
70
17
1991
Jelnic
190
180
10000
(1.55))
25
Meku lasgery (MI)
278
100
3600
64
[1621
26
NATOPS INT)
180
180
51
24
[163]
PEMS (PS)
20
Ploncre Spe.ru PPp))
3
3,353
30
217
11
1106
pop lad
10
Racket Spone (8S)
151
182
30
[107]
morkk
Set Regulatton (S111)
263
203
[10N]
spi-ink
15
Spokin Amiiz Dhyin (SAD))
6,599
2119
11100
16
Stndd Welk jump (SWI)
15
2300
(69)
ony cey-ink
UWasc Geatee Library (UW)
2,298
2241
[113]]
Labelel Assecal Desection Dutact
37
Detoction
KM
54
56
74,581
[71]
lick
Applonces Energy (AF)
74
[11:1
Rojjin PATOO Qualit (MPMI))
13,037
5,00
74-
[114]
41
Reijnng PXMS Quality (RPPEES))
17.039
5100
74-
[115]
42
Benocae Concaatation (BC)
3,23
5.455
240
[115]
lank
BEDACEERS (SIDMCRR)
4,000
45
BEDNCC2S,02 (SIEMCSPOO)
5,500
2.30
4,000
[119]
46
CinallMouth (C334)
140
61
84
[1208
Regresiss
Mookligg
300
Fladd Mookling (FM))
420
186
1828)
30
Ekzzschild Power Cresampion (UPC
296
696
1,40
1134)
tpe Frink
Honetbelld Poweer Commempton (IPC2)
198
1440
1125]
Live Manutarz Coston (LJML)
3
1310
112/1
36
Nevs Lamdliee Sentinent (h01))
55.13
26355
266
[128)
the Inc.
Nivss Sentrnent (N15)
54213
24951
[129)
th-iikk
MPC Tick
C.1 Classification Time Series Datasets
This section introduces the datasets used to evaluate the performance for time series classification.
C.1.1
Non-Stationary Spatiotemporal Time Series Datasets
In this section, we provide details about the non-stationary spatiotemporal datasets used in Section 4.
C.1.1.1 Real-World Digital Key Trajectories (DKT) Dataset
The DKT dataset comprises multivariate time series data, capturing x- and y-positions sequentially to predict the intent of the smart device carrier. In total, the DKT dataset comprises 559,709 anonymized customer trajectories, recorded over a span of three months from a subset of BMW's fleet of vehicles. This dataset of labeled trajectories was obtained using high-precision localization with UWB technology and the DK. It includes various vehicle types, from small hatchbacks to large SUVs. Each trajectory is associated with binary label, y E {0, 1}. indicating whether a specific action is
Table 9: DKT label distribution (in %) and number of samples per data-subset for seed 42.
Label Distribution (%)
Num. of Samples
Train Dataset 48.50
51.50
358.211
Val Dataset
48.81
51.19
89,554
Test Dataset
48.67
51.33
111,944
Total
48.65
51.34
599.709
43

---

taking place (1) or not (0). The label distribution is approximately 48/52, with 52 % corresponding label However, localization accuracy can be affected by various external factors and ranging algorithms, as discussed in Appendix A. Consequently, the DKT data includes irregularly sampled and non-stationary sequential data. Table 9 displays the label distribution the DKT dataset.
C.1.1.2 Geolife (GL) Dataset
The GL GPS trajectory dataset collected by 182 Geolife project over span more than five years (from April 2007 to August 2012) [63]. Each GPS trajectory in this dataset is sequence of time-stamped points, providing information on latitude. longitude, and altitude. The dataset comprises 17,621 trajectories, covering total distance of approximately 1.2 million kilometers and total duration exceeding 48,000 hours. These trajectories were recorded using various GPS loggers and GPS- GPS-enabled phones, featuring range of sampling rates. This dataset captures wide array of users' outdoor movements, including everyday activities like commuting to work or home, as well as recreational and sports activities such shopping, sightseeing. dining, hiking, and cycling. The GL trajectory dataset is valuable for research in multiple fields, including mobility pattern mining, user activity recognition, location-based social networks, location privacy, and location recommendation [63].
part of the Microsoft Research Asia
Our pre pre-processing of the data before it is usable in training includes:
filtering for labeled and unlabeled samples
removing samples that have fewer than five sequential elements
the data is restricted the Beijing metropolitan area, which upproximately 96% of the entire labeled data
converting longitude, latitude, and altitude to SI units, i.e., meters (m)
considering the imbalance in data distribution, all trajectories labeled airplane, boat, mo- torcycle. subway and train are dropped, runs are considered a walk and taxis considered car
filtering trajectories that surpass certain speed limit, indicating that the label is wrong. As walks and runs are combined, we consider an average pace of 12 km/h, i.e., 5 min/km, as the speed boundary for the walk-run class, 60 km/h for bikes, 100 km/h for busses, and 120 km/h for cars. The last two correspond to the speed limits in China.
removing outliers
After the data has been preprocessed, consists 7,990 samples. use a ratio of 7/1/2 to split the data into training, validation, and testing respectively, following [17]. The breakdown of the class distribution is provided Table 10.
Table 10: Label distribution (in %) of pre-processed GL dataset of seed 42.
Label Num. of Samples
Label Distribution (%)
bike 1,534
19.20
bus
1,745
21.84
car
1,186
14.84
walk
3,525
44.12
Total
7,990
In Fig, 12, the trajectories of the dataset visualized for the Beijing metropolitan area.
44

---

Bam
Matooyele
Anpplane
Traks
Subway
Car
Vide
(a) Overview
(b) Zoomed View
(c) Legend
Figure 12: This figure presents two images depicting the collected trajectories from the GL dataset in the Beijing Metropolitan Area. Image (a) provides a complete overview, image (b) shows zoomed-in version, and image (c) includes the legend that explains the color coding used for the trajectories in both (a) and (b). Images are taken from https: httpp::/heeemaps .aaipp...tthte github. ho/pptk//utooraall..... ewer/geol wfe........ html.
C.1.2 Irregular Sampled Time Series Datasets
In this section, we provide details about the irregular sampled time series datasets used in Section 4.
C.1.2.1 PhysioNet Sepsis Early Prediction Challenge 2019 (P19) Dataset
The PhysioNet Sepsis Early Prediction Challenge 2019 (P19) dataset comprises time series records from 38,803 ICU patients, each monitored via 34 physiological sensors. From the original 40,336 patients, samples with extremely short or long sequences (fewer than 2 or more than 60 observations) were excluded. Each patient is also associated with a static feature vector encoding demographic and clinical attributes, including age, gender, ICU type, ICU stay duration, and time elapsed between hospital and ICU admission. The prediction task involves binary label indicating whether sepsis will occur within the subsequent hours [20]. The dataset is highly imbalanced, as displayed in Table 11. To ensure fair evaluation, the performance is averaged over five consistent data splits.
Table 11: P19 Label Distribution in percentage (%).
Split 0
Split I
Split
Split 3
Split 4
Num. of Samples
0
0
0
Training Dataset
95.78
4.22
95.84
4.16
95.83
4.17
95.80
4.20
95,83
4.17
31,042
Validation Dataset
96.29
3.71
96.01
3.99
95.88
4.12
95.80
4.20
95.46
4.54
3,380
Test Dataset
95.54
4.46
95.34
4.66
95.54
4.46
95.88
4.12
96.01
3.881
Total
38,803
C.1.2.2 PhysioNet Mortality Prediction Challenge 2012 (P12) Dataset
After filtering out 12 entries lacking time series data, the PhysioNet Mortality Prediction Challenge 2012 (P12) contains data from 11,988 ICU patients. Each sample includes multivariate time series from 36 sensors (excluding weight), collected over the first 48 hours of the ICU stay. A static feature vector with nine demographic and clinical variables (c.g., age, gender) accompanies each sample. The binary prediction target denotes the ICU length of stay, where < days is the negative class and > 3 days the positive class. The dataset is heavily imbalanced, as displayed in Table 12. To ensure a fair evaluation, the performance is averaged over five consistent data splits.
45

---

Table 12: P12 Label Distribution in percentage (%)
Split
Split
Split
Split
Split
Num. of Samples
Training Dataset
85.60
14.40
85.99
14.01
85.77
14.23
85.83
14.17
86.08
13.92
9,590
Validation Dutaset
85.65
14.35
83.99
16.01
85.40
14.60
85.32
14.68
85,82
14.18
1.199
Test Dutuset 87.16
12.84
85.74
14.26
86.07
13.93
85.65
14.35
83.15
16.85
1,199
Total
11.988
C.1.2.3 Physical Activity Monitoring (PAM) Dataset
The PAM dataset, derived from PAMAP2 (Physical Activity Monitoring), records physical activities of nine subjects using three inertial measurement units. To adapt for irregular time series classification, the ninth subject, due to insufficient sensor readout length, is excluded. The continuous signals are segmented into samples with a time window of 600 and 50% overlap rate. Initially, PAM includes 18 daily activities, but those with fewer than 500 samples are excluded, leaving eight activities. After these modifications, the PAM dataset comprises 5,333 segments (samples) of sensory signals. Each sample is captured by 17 sensors and contains 600 continuous observations at sampling frequency of 100 Hz. To simulate irregular time series data, 60% of the observations are randomly removed. For fairness in comparison, the removed observations are randomly selected but consistent across all experimental settings and approaches. The PAM dataset labeled into 8 classes, each representing a physical activity, and does not include static attributes. For more detailed descriptions please refer to [20]. The samples are roughly balanced across the categories, as displayed in Table 13. ensure a fair evaluation, the performance is averaged over five consistent data splits. The pre- pre-processed data of PAMAP2 as well as the data splits can be accessed via the link provided in Table 8.
Table 13: PAM Label Distribution in percentage of Split (in %).
6
7
8
Num. of Samples
Training Dataset
22.08
11.77
10.31
11.86
15.56
5.91
11.67
10.83
4,266
Validation Dataset
24.95
11.26
9.01
12.20
15.20
4.69
11.44
11.26
533
Test Dataset
23.22
12.17
11.05
13.67
15.73
6.37
7.49
10.30
534
Total
5,333
C.1.3 Regular Time Series Datasets
This section provides details about the regular sampled time series datasets used Section 4.
C.1.3.1 UEA Benchmark Datasets
The datasets from the UEA Archive [70] commonly used to benchmark machine learning models on time series classification tasks. For a detailed overview of the datasets, please refer to Table 8. For all datasets separate testing and training datasets provided, hence only the training set is split with a ratio of 9/1 into training and validation. This is executed consistently for all datasets from the UEA benchmark mentioned below. For all other datasets, the test is also used for validation.
C.1.3.1 Dataset Selection
We follow the curation of diverse subset from Time Series Transformer (TST) [23] for the ab- lation study in Section 4.4.1. The diverse selection of datasets from the UEA benchmark ensures variability across key characteristics: sample dimensionality, sequence length, dataset size, and task difficulty. Our selection encompasses both high-performing ("easy") ) and low-performing ('challeng- ing') ) datasets, referenced by the baselines employed. Below is brief justification for each selected multivariate dataset:
1. Eigen Worms (EW): Low dimensionality, few samples, very long sequence length, moderate class count, relatively challenging dataset.
2. Ethanol Concentration (EC): Low dimensionality, few samples, moderate sequence length, moderate class count, challenging dataset [23].
3. Face Detection (FD): Very high dimensionality, large sample size, short sequences, binary classification [23].
46

---

4. Handwritting (HW): Low dimensionality, limited samples, moderate sequence length, many classes [23].
5. Heartheat (HB): High dimensionality, small sample size, moderate sequence length, binary classification [23].
6. Japenese Vowels (JV): Variable sequence lengths, moderate dimensionality, few samples, moderate class count, baselines perform well [23].
7. Pen Digits (PD): Low dimensionality, many samples, short sequence length, many classes, baselines perform well.
8. PEMS-SF (PS): Extremely high dimensionality, few samples, moderate sequence length, moderate class count, baselines perform well [23].
9. Self Regulation SCP1 (SCP1): Low dimensionality, few samples, long sequences, binary classification; baselines perform well [23].
10. Self Regulation SCP2 (SCP2): Similar to SCP1 but with increased task complexity [23].
11. Spoken Arabic Digits (SAD); Moderate dimensionality, large sample size, heterogeneous sequence lengths, moderate class count, baselines perform well [23].
12. UWvve Gesture Library (UW): Low dimensionality, few samples, moderate sequence length, moderate class count, baselines perform well [23].
C.2 Anomaly Detection Time Series Datasets
This section introduces the benehmark datasets used evaluate the performance for univariate time series anomaly detection.
C.2.1 A Labeled Anomaly Detection Dataset (Yahoo) Webscope
Yahoo created comprehensive public dataset, aiming to aid anomaly detection research [72]. This dataset includes both synthetic and real internet traffic data, with the latter manually labeled, acknowledging potential human error. Further it includes variety of anomaly types such as outliers and change-points [24]. The dataset encompasses 367 hourly sampled time series with tagged anomaly points. The sequences are split as described in Section 4.2.
C.2.2 KPI
The KPI dataset was released in an AIOPS challenge [71]. includes multiple minutely sampled real KPI curves from many internet companies [24]. In total, it has 58 sequences, with the longest sequences exceeding 70,000 elements. The sequences are split as described in Section 4.2.
C.2.3 Window Creation for Long Sequences
To create a sliding window mechanism that creates instance segments of a sequence, two variables defined. W defines the size of the window and S the size of the stride. If W S, there is no overlap between segments. The total number segments is computed as:
N N W
(44)
S
where N is the length of sequence ( E R NxD, Then, window can be defined as
w ((t)
jeS:j+StW:
(45)
where j {0, I,..,,.,,, N and W RxDD
D Experiments
D.I Evaluation Metrics
Typically, for benchmarking classification tasks, the accuracy on the test set is reported. In addition, for the DKT dataset, we want to focus on minimizing false positive predictions and thus record the Fg-score, Eq. (46) and (47), explicitly.
47

---

(1+ TP
F Pg-score-
(46)
(1+B8)TPP4 FP DR.FN (1+ (1+19)- -TP TPP
TP
(1+ TP FP FN (1+33) B- TTP
TP
1/(TP (TTP-+FF)) FPP+ 1/(TP+ FN) (TP+FP)
TTP.. (TP FN) (TP+ TP
TP Procision Recall
(1 (+8))
PPTFP TPOPPEEN
PPIPN
82. (TP PPITRAPNT (TLFPTATP+EN)
TPIPT*PI
Precision Recall
(1+ + B8))
TP Precision TP Recall
32
TP4FP
4P4FN
Precision Recall
= (1+ )² Precision + Recall
(47)
The F3-score balances precision and recall through the weighting parameter 3. For B = 1, it equals the Fscrr... A ß-value < emphasizes precision, reducing false positives, while 3 prioritizes recall, reducing false negatives. We choose / 0.5. Excellent Fy-scores range from 0.8 0.9,
whereas scores below 0.5 are considered poor.
Depending on the dataset, other metrics used for classification and anomaly detection tasks include Fr-Score, Precision, Recall, Area Under the Receiver Operating Characteristic (AUROC), Area Under the Precision-Recct Curve (AUPRC) and Mean Absolute Error (MAE).
For regression, follow the 'average relative mean difference', ry the evaluation metric used in previous literature (21, 23). For each model j over N datasets, the average relative mean difference defined as:
N
R(i,j) R,
(48)
R4
and
M R(i,)),
R:
(49)
M km
where M is the number of models, R) is the Root Mean Squared Error (RMSE) of the model on dataset i.
D.2 DKT Robustness Analysis
Table 19 documents the complete results presented in Table 1, detailing performance across five training seeds for various baseline models on the DKT dataset. Although STaRFormer is able to outperform the baseline models, some achieve nearly similar performance. Thus, conduct further examinations to evaluate the performance.
We discovered that the labeling process during the trajectory recording leads to overlap of positive and negative labels for some visually similar trajectories. This overlap creates a performance ceiling that we believe is inherent to the dataset. Despite efforts to overfit the model during training, the maximum accuracy attained was approximately 90%. This suggests that the performance metrics are approaching the upper limit, given the current data collection methods. Consequently, we performed a robustness analysis to explore not only the defined metrics but also sensitivity of model predictions to potential noise from the sensors used for the data collection. In this analysis, utilized the coefficient of variation (CV), as shown in Eq. (50), to assess the variability in the model's predictions.
σ
CV
(50)
48

---

Experimental Setup. We selected longer sequences from the DKT test set, specifically those where ()n > 100 elements. Then, Gaussian white noise is added to the final 10 and 30 elements of the selected sequences. Consequently, for each sequence, we obtained 10/30 additional corrupted sequences in the respective setups, We then evaluated all sequences and calculated their corresponding CV values.
Results. Table 14 presents the results of the robustness analysis, comparing four models under varying levels of input noise. The evaluation focuses on two primary metrics: CV, which reflects the stability of model predictions, and the Mean Absolute Error (MAE) between original and noisy predictions, which quantifies sensitivity perturbations. A lower CV value is indicative of better inherent robustness, implying more consistent predictions across samples. Conversely, a lower MAE signifies that the model's outputs are less influenced by noise. However, excessively low MAE values may also suggest that model is insensitive to real-world variability, potentially leading to underfitting or lack of generalization.
The results clearly show that all models experience a degradation in robustness as the noise level increases from 10 to 30 corrupted sequential elements. This trend is evident in the positive deltas across all metrics (Aoriginal. Anolay, and AMAE), signifying increased variability and eтror due to noise. Among the models evaluated, STaRFormer demonstrates superior robustness characteristics: yielding the lowest CV values for both original and noisy data for both perturbations (10 and 30), exhibiting the smallest deltas (Auriginal 0.063, moissy 0.098) which indicates that its predictions are least affected by increased noise, and reporting moderate MAE values (0.039 at 10 and 0.074 at 30) suggesting a balanced trade-off between robustness and sensitivity, avoiding excessive rigidity.
In contrast, the Transformer model shows the highest deltas and CV values across the evaluation, indi- cating the most substantial degradation in performance under noise. This emphasizes the advantages of our approach, demonstrating that our method facilitates the creation of more robust latent represen- tations, which consequently enhances the overall robustness of the downstream task performance. even if the improvements in downstream task metrics are small. Additionally, it is notable that the LSTM model consistently exhibits the lowest MAE between the original and corrupted sequences under both perturbation levels (10, 30). This suggests high degree of robustness to input noise. However, the minimal deviation in predictions may also reflect an excessive rigidity or insensitivity to input variability, potentially harming the model's 's ability to generalize effectively.
Overall, Former emerges as the most robust model, maintaining consistent prediction quality while allowing for some degree of variability, which is critical for generalization.
Table 14: Robustness analysis results.
CV
Deltas (A)
Method
10
30
original
noisy
MAE
original
noisy
MAE
A Aoriginal
Dpisy
A AmA
LSTM
0.845
0.847
0.003
0.959
0.993
0.035
0.114
0.146
0.032
GRU
0,810
0.846
0.036
0.949
1.020
0.071
0.139
0.174
0.035
Transformer
1.018
1,068
0.050
1.223
1.309
0.086
0.205
0.241
0.036
STaRFormer
0.765
0.804
0.039
0.828
0.902
0.074
0.063
0.098
0.035
D.3 Experiment Runs
We use a Rate Scheduler (Reduce Learning Rate on Plateau Learning), Early Stopping, and the Adam optimizer for all experiments. All configurations of the model, the datasets, and all other relevant hyperparameters are extensively documented in the accompanying GitHub repository, https: //github. faccuu..com//55 com/STaR htmm//STaR-From//strrrrre Former/starforrer, and can be found in the 'experiment// /finnn///nnlll sub- folder in "configs'.
D.3.1 Compute Resources and Execution Times
We conducted all experiments using Amazon EC2 instances pottps://awws amazon com/cc/ /instance Linstancettppes//nott [s), which offer broad range of instance types with configurable
49

---

combinations of CPU, memory, storage, and networking capabilities. These allow for flexible resource allocation tailored specific computational requirements. For this study, we primarily utilized AWS accelerated computing instances, specifically the P3 and G5 families.
Due to the asynchronous scheduling of experiments and the dynamic nature of cloud resource avail- ability, we employed different GPU and CPU configurations depending on instance accessibility the time of execution. Nevertheless, all experiments were run exclusively on AWS EC2 P3 or GS instances, utilizing either an NVIDIA Tesla V100-SXM2-16GB or an NVIDIA A A10G-24GB GPU. During the course of experimentation, AWS deprecated the P3 instance family, rendering them inaccessible for future runs. As result, we transitioned to the G5 instance family. For compre- hensive overview of G5 instance specifications, please refer to the official AWS G5 documentation (https: / aws. amazon. com/ ec2/ instiinnteee waaanceettpeee/ee (oppe//5/),
Hyperparameter Tuning. We performed extensive hyperparameter tuning for all experiments involving STaRFormer to identify optimal configurations, For large-scale datasets, such as DKT, this process was computationally intensive and required sustained multi-GPU workloads over several days. Specifically, for DKT, we executed multiple sweep agents on 8-GPU EC2 P3 instance over the span of one week to converge on the final configuration. In contrast, tuning for smaller datasets, particularly those in the UEA benchmark benchmark, required significantly less time. For example, the case of Japenese Vowels (JV), single sweep was completed approximately five hours. hyperparameter sweep configurations used in this study available in the accompanying GitHub repository.
Execution Times. The following table reports training durations, memory usage, and resource allocation for all executed runs, including both our model and baseline implementations. Note that many ablation studies reused already documented configurations with slight changes. Given the high similarity in resource profiles across these repeated runs and the large number of total ablations (approximately 210), only representative exccution times documented from the original configu- ration. Find the complete documentation for the individual execution times in Tables 15, 16, 17 and 18.
Note: some instances, multiple experiments were simultaneously executed on the same GPU to optimize memory utilization. Consequently, resource contention led to increased training dura- tions, with some recorded runtimes exceeding those expected under isolated, single-worklone workload GPU exccution.
50

---

Table 15: Combined training and testing times for non-stationary (DKT, GL) and irregularly sampled (P19, P12, PAM) datasets. All training times for a single GPU are reported to ensure a fair comparison. However, some are estimates (indicated by *), as they were trained using data-distributed parallel strategies on multi-GPU workloads.
Dateset
Mathod
Hurdwaee
Memoryy (GB)
Splits (Sood)
Average
Std. Dev.
1(123)
3(663)
(22024)
3,600
1,713
2.157
2.805
2.749
2.605
Epocts
54
Memury Allocation
0.115
0.089
0.107
0.167
0.164
0.123
0.015
Time (h)
4.773
5.225
4.588
4.041
4.672
4,660
0474
LSTM
A10G
24
Epocits
98
63
102
105
68
$7,200
20.012
Memory Allocizion
1,080
0.125
0.128
0.122
0.122
0,115
0.000
Time (h)
4,805
3,180
4.7%8
4.630
2.851
4,053
AIOG
24
Eguchs
94
124
192
132.000
39.013
Momory Allocation
0.129
0.118
0.118
0.072
0.113
0.024
DKT
Transfommrr (TST)
Tosla V100- SXM2
16
Time (h)
Fipcchs
3,670
98
1.846
65
2,900
105
2.315
83
2,455
87
87.600
2.637
15.356
0.680
Memury Allocation
0
0.118
0.103
0.118
0.118
0,115
0.006
Time (h)
28.318
14.239
22.575
17.661
18:343
20.347
5.317
TARNet
AIOG
24
Epuchts
300
300
300
300
300
300,000
0.000
Memory Allocation
0,574
0.448
0.554
Time (h)
118.398
115.005
115.954
1,493
Time
A
Fyechss
100
100
100
100.000
Memury Allocation
0,681
0.706
0.756
0.530
0.555
0.645
0.088
STaRForases-MM
Testa V100 SXM2
16
Time (h)
Eppcitss
112.921
95
80.068
7X
50.440
80
66.424
89
61,878
60
82.200
74,346
25.004
10.055
Memory Allocation
0,939
0.484
0.516
0.939
0,516
0.679
0.338
73.729
58.723
58:15
68
62:267
64.395
6 6,696
STaRFormerr'
SXM2
Momory Allocution
Epochs
0,846
0.566
0.941
0.039
0.484
91.200
0.755
0.315
Transfommer (TST)
Tesa V100-
SXM2
16
Momary Allocizion
Time (h)
Epochs
0.439
1.112
115
0.344
0.099
61
0.112
0.272
69
0.029
0.320
84
0.204
0.112
74
80.600
0,099
0.310
20,959
0.067
0.018
Time (h)
1.54L
1.541
TARNet
1OG
Eguchs.
200.000
Momory Allocation
0.574
0.574
Time (h)
9.901
9.903
URL
A10G
24
Epochts
300
300,000
Memury Allocation
0,383
0,383
STaRFurmer- RM
Testa V100- SXM2
Time (h)
Eguchs
0.854
57
1.287
62
1.343
1,210
56
0.838
56
60.000
1.107
0.342
5,612
Memory Allocation
0.174
0.254
0.254
0.254
0.174
0.222
0.044
STaRFormerr
V10K- SXM2
Time (h)
Fiochss
1.112
70
1.752
96
1.442
91
1,299
13,099
0.293
Memury Allocation
0,878
0.674
0.639
0.732
0.611
0,713
0.100
Hose
Testa V100 SXM2
16
Time (h)
Epocits
1,077
125
2,328
86
2,217
74
2.475
86
2,305
77
39,600
2,080
20.501
0.569
Memory Allocation
0.549
0,978
0.978
0.978
0.978
0.852
0.170
STaRFormeer
SXM2
V1000
Momory Allocution
Epochs
0.186
1,501
72
0.188
1.080
0.188
0.186
1.705
2.236
0.189
0.187
1.554
18.144
0.001
STuRFurmer
Testa V100- SXM2
16
Time (h)
Epucits
3:445
106
4.247
136
4.021
124
5.613
76
4,953
158
140,000
4.456
27,604
0.343
Memary Allocizion
11259
0.260
0.260
0.259
0.250
0,260
0.000
Testa V100- SXM2
Time (h)
Fgochss
1.199
0.519
0.285
0.323
53
0.286
64.400
0.523
0.390
633
Memory Allocation
0,928
0.791
0.658
0.807
P12
STaRFormer RM'
Tesla V100- SXM2
16
Time (h)
Epochs
0,930
82
97
0.441
0.516
62
0.405
49
68.600
0.625
20.251
0.340
Memory Allocation
0.984
0
0.564
0.590
0.746
0.693
0.179
SToRFurmer"
Testa V100-
SXM2
16
Memory Allocation
Time (h)
Egochs
0.820
0.515
85
0.579
1.161
117
0.708
2.417
259
0.670
1.149
122
0.585
1.640
163
149.200
0.671
1.438
67.351
0.138
0.621
Time (h)
0.166
0.322
0.100
0.166
0:138
Rose
SXM2
Memury Allocation
Fiocchs.
0.234
86
0.186
0.234
0.234
74
0.234
73,800
0.224
0.021
PAM
S HMM
Testa V100
SXM2
16
Memory Allocstion
Time (h)
kppochss
0.368
0.559
3D
0:368
0.675
101
0.099
0.308
100
0.099
0.308
103
0.366
0.663
101
97.000
0.200
0.503
0.183
0.002
9.566
STaRFormer'
V1000
Epochs
0.569
0,702
103
0.422
0.561
132
119.200
0.505
32.568
0.130
Momory Allocation
0.371
0.371
0.371
0.311
0.371
0.571
0.000
51

---

Table 16: Combined training and testing times for datasets from the UEA and the anomaly detection benchmarks (Yahoo, KPI). All training times are reported for a single GPU to ensure a fair comparison.
Memory (GB)
Time (h)
Memory Allocation
Dataset
Hardware
Method
Epochs
Base
0.015
44
0.097
Tesla V100- SXM2
EW
16
STaRFormer-
0.016
37
0,030
STaRFormer
0.033
0.180 87
Basc
0.119
65
0.559
Tesla V100 SXM2
EC
16
STaRFormer-RM
0.296
73
0.700
STaRFormer
0.035
0,557
Base
0,416
87
0.559
Tesla V100 SXM2
FD
16
STaRFormer-RM
0.201
32
0,700
STaRFormer
0.342
102
0.557
Base
0.051
111
0.131
Tesla V100 SXM2
HW
16
STaRFormer- RM
0,104
197
0,142
STaRFormer
0.114
162
0.277
Base
0.018
50
0.345
Tesla V100
HB
16
STaRFormer-RM
0.023
48
0.130
SXM2
STaRFormer
0.023
34
0.481
Base
0.019
108
0.701
Tesla V100
STaRFormer-RM
0.068
232
0.122
SXM2
STaRFormer
0.041
103
0.126
Base
0.199
113
0,031
Tesla V100- SXM2
PD
16
STaRFormer-RM
1.110
262
0,031
STaRFormer
2.017
266
0.427
Basc
0.058
195
0.119
Tesla V100 SXM2
PS
16
STaRFormer-RM
0.123
177
0.154
STaRFormer
0.159
291
0.237
Base
0.049
87
0.788
Tesla V100 SXM2
SCP1
16
STaRFormer-RM
0.118
112
0.609
STaRFormer
0.043
103
0,245
Base
0.026
69
0,279
Tesla V100- SXM2
SCP2
16
STaRFormer-RM
0,052
32
0,831
STaRFormer
0.054
42
0.442
Basc 0.143
59
0.102
Tesla V100 SXM2
SAD
16
STaRFormer-RM
0.690
51
0.175
STaRFormer
1.567
105
0.145
Base
0.034
142
0.838
Tesla V100
UW
16
STaRFormer-RM
0.090
98
0,523
SXM2
STaRFormer
0.037
96
0,295
Base
0.036
39
0.601
Yahoo
A10G
24
STaRFormer-RM
0.141
46
0.904
STaRFormer
0.461
61
0.660
Base
0.967
102
0.422
KPT
A10G
24
STaRFormer-RM.
1.429
54
0.422
STaRFormer
2.374
100
0.415
52

---

Table 17: Combined training and testing times of STaRFormer of remaining datasets from the UEA
and TSR benchmarks. All training times are reported for a single GPU ensure fair comparison.
Memory Allocation
Memory (GB)
Time (h)
Benchmark
Dataset
Hardware
Epochs
(max 1)
AWR
A10G
24
0.227
145
0.472
AF
A10G
24
0.009
53
0.010
BM
A10G
24
0.011
76
0.010
CT
A10G
24
1.221
163
0.526
CK
A10G
24
0.085
0.934
DDK
A10G
24
0.024
36
0.634
EP
A10G
24
0.045
56
0,397
ER
A10G
24
0.103
113
0.383
FM
A10G
24
0.052
36
0.619
UEA
HMD
A10G
24
0.049
54
0.320
IW
A10G
24
8.768
84
0,743
LI
A10G
24
0.170
189
0.657
LSST
A10G
24
0.360
45
0.284
MI
A10G
24
0.031
17
0.927
NT
A10G
24
0.117
156
0,265
PSp
A10G
24
0.859
64
0,417
RS
A10G
24
0.124
122
0.597
SWJ
A10G
24
0.024
96
0.902
AE
A10G
24
0.085
827
0.089
AR
A10G
24
22.039
200
0.298
BPM10
A10G
24
1.663
0.191
BPM25
A10G
24
1.108
127
0.150
BC
A10G
24
2.144
263
0.529
BIDMCHR
A10G
24
3,998
104
0.510
BIDMCRR
A10G
24
1.249
110
0.460
BIDMCSPO2
A10G
24
2.733
104
0,340
C3M
A10G
24
0.014
175
0.037
TSR
FMI
A10G
24
0.965
419
0.664
FM2
A10G
24
0.249
157
0.320
FM3
A10G
24
0.388
294
0.368
HPC1
A10G
24
0.786
146
0.670
HPC2
A10G
24
0.643
159
0,513
IEEEPGG
A10G
24
2,439
173
0.705
LFMC
A10G
24
1.650
190
0,358
NHS
A10G
24
10.479
0.326
NTS
A10G
24
9,910
107
0,586
PPG
A10G
24
17.886
276
0.314
53

---

Table 18: Combined training and testing times of TARNet for the datasets from the TSR benchmarks.
All training times reported for single GPU to ensure fair comparison.
Memory
Memory (GB)
Time (h)
Benchmark
Dataset
Hardware
Epochs
Allocation
(max 1)
AE
A10G
24
0.011
200
A10G
24
1.291
BPM10
A10G
24
0.336
200
BPM25
A10G
24
0.262
200
BC
A10G
24
0.529
200
BIDMCHR
A10G
24
2.094
BIDMCRR
A10G
1.264
BIDMCSPO2
A10G
24
1.781
300
C3M
A10G
24
0.017
300
TSR
FMI
A10G
24
0.108
300
FM2
A10G
0.070
FM3
A10G
24
0.065
300
HPCI
A10G
24
0.462
100
HPC2
A10G
24
0.465
100
IEEEPG
A10G
24
1.745
200
LFMC
A10G
0.921
NHS
A10G
24
4.581
300
NTS
A10G
24
3.709
300
PPG
A10G
24
4.773
300
54

---

D.3.2 DKT
Table 19: Run documentation of the methods trained on the DKT dataset. The respective seeds for each stated in the parenthesis in the header row.
Method
Metrics
(42)
(123)
2.00)
(63)
4 (2024)
Median
Avemge
Std. Dev.
Accuracy
0.74794
0.76239
0.76147
0.75659
0.73939
0.755
0.754
0.010
Fo.r-Score
0.74776
0.76224
0.76288
0.75698
0.73917
0.755
0.754
0.010
LSTM Baseline
Accuracy
0,83982
0,84202
0.84576
0.84732
0.8438
0.844
0.844
0.003
Foor-e
0.84046
0,84194
0.84581
0.84359
0.84383
0.844
0.843
0.002
GRU Baseline
Accuracy
0.84088
0,8399
0,83505
0.84337
0.84284
0.841
0.840
0,003
Fo.s Score
0.84097
0.8399
0.83547
0.84323
0.84282
0.841
0.840
0.003
Transformer
Accuracy
0.84847
0,84589
0.84962
0.84851
0,85059
0.849
0.849
0,002
Fo.s Score
0.84855
0,84577
0.84953
0,84844
0.8505
0.849
0.849
0.002
Accuracy
0,78050
0.76269
0,78482
0,78121
0.79360
0.781
0.781
0.011
TARNet
Fo.s Score
0.78408
0.76311
0.78469
0.78152
0.79545
0.784
0.782
0.012
TRL
Accuracy
0.72720
0.7261
0.7244
0.7205
0.7207
0.724
0.724
Fo.
STaRFormer with RM
Accuracy
0.84535
0,84509
0.84687
0.84477
0,84408
0.845
0.845
0.001
Fo.s-Score
0.84535
0.8448
0.84675
0.8447
0,84405
0.845
0.845
0.001
STaRFormer with DARcM
Accuracy
0.85498
0.85069
0.8493
0.85366
0.84931
0.851
0.852
Fo.s-Score
0,8549
0,8507
0.84916
0.85355
0.84952
0.851
0.852
0,003
Table 20: Run documentation of the ablation study on the DKT dataset evaluating the impact of our semi-supervised CL approach the model performance. The respective seeds for each run are stated the parenthesis in the header row.
Ablatica
Metries
(42)
1 (123)
2(00)
(63)
(2024)
Median
Average
Sid. Dev.
suprvisee (Acr. ss 0.796)
Accuracy
0,85498
0,85069
0,85366
0.84931
0.852
0.005
Fo.s-Soore
0.8549
0.8507
0.84916
0.85355
0.84952
0.851
0.852
0.003
wro self supervised (Acs. 0.796)
Accuracy
0.84916
0.34950
0.8445
0.84680
0,84917
0.848
0.848
0.002
Foor-Score
0.84663
0.002
wlo supervised (ACL. 0 0.796)
Accuracy
0.84782
0.84824
0,84742
0.84762
0,84865
0.848
0,848
0.001
Fo.r-Soore
0,84770
0.84818
0.84371
0.84760
0,84854
0.848
0.847
0.002
semi-supervised (Act. 0.1)
Accuracy
0.84943
0.84861
0.84736
0.84613
0.84908
0.848
0.848
0,001
Score
0.84000
0.84874
0.84761
0.84593
0.84897
0.847
0.846
0.004
semi superviper (AcL 1.0)
Accuracy
0.85223
0.85337
0,85004
0,85030
0.84819
0.851
0.851
0,002
Four-Sorre
0.85023
0,002
semi-supervised (eet. 5.0)
Accuracy
Score
0.84629
0.84900
0.84887
0.84900
0.85023
0.85030
0.85123
0,85132
0.849
0.849
0.002
0.002
semi-supervised 10.0)
Accuracy
0.84525
0.84455
0,84567
0.84309
0,85009
0.845
0.846
0.003
RoorS
0.84517
0.84454
0.84553
0.84299
0.85000
0.846
0.003
55

---

Table 21: Run documentation of the ablation study evaluating how the size of the masked regions affects the model performance on the DKT dataset. The respective seeds for each run are stated in the parenthesis in the header row.
Ablation
Metrics
(42)
(0)
(63)
(2024)
Average
default
0.427
0.2
0.25
Accurncy Fo Scoore
0.85498
0.85069
0.84931
0.8549
0.8507
0.84916
0.85355
0,84952
0.852
0.852
0.003
Accuracy Fo Szore
0.84898
0.85157
0.84685
0,85129
0.427
0.2
0.0
0.84895
0.85139
0.84671
0.84883
0.8518
0.850
0.850
0.002
0.427
0.2
0.05
Accuracy
0.84958
0.84961
0.84611
0.85328
0.8494
0.850
0.850
0.003
0.84593
0.84594
0.84508
0.85324
0,84937
0.848
0.848
0.003
Accuracy
0,84890
0,85015
0.84488
0.85408
0.84799
0.849
0,849
0.003
0.427
0.10
Po.seScore
0.84886
0.85005
0.8488
0.85403
0.84794
0.850
0.849
0,002
Accuracy
0.84839
0.85171
0.427
0.2
0.15
Soore
0.85216
0.85059
0.84849
0.84854
0,8516
0.850
0.850
0.002
4
0.427
0.2
0.20
Fo S-core
Accuracy
0.85223
0.85221
0.84957
0.84962
0.85114
0.85106
0.84978
0.84994
0.85132
0.85138
0.851
0.851
0.851
0.851
0.001
0.001
0.427
0,25
Accuracy
0.85498
0.85069
0,8493
U.85366
0.84931
0,852
0.852
0.003
Fo S-orr
0.8549
0.8507
0.84916
0.85355
0.84952
0.852
0.852
0.003
0.427
0.2
0.30
Accuracy 0.8512
0.84934
0.84974
0,8518
Fo.s-S
0.84929
0.84963
0.84888
0,851/6
0.850
0,850
0.001
D.3.3
Baseline Implementation
We selected two distinct state-of-the-art methodologies from literature to serve as additional baseline methods on the DKT dataset. Specifically, chose Task-Aware Reconstruction for Time Series Transformer (TARNet) [21] and Sepf-Suuee Contrastive Learning for Universal Time Series Representation Learning (TimesURII) [27]. To utilize the official code baselines, we adapted our data loading procedures accordingly. Due to the absence of specified hyperparameters, for the Transformer backend in TARNet, we applied the same hyperparameters URL, we employed the model's default parameters. It is important to note that TiMesURL inherently utilizes a grid search strategy find the optimal the Support Vector Machines (SVM) for the downstream task. For the DKT dataset, as the grid search is quite expensive, we limited it to two folds instead of the default five.
those used for STaRFormer. For
56

---

D.3.4 Geolife
Table 22: Run documentation of the ablation study on the three ablations of STaRFormer on the GL dataset [63]: (i) (Base), (ii) STaRFormer-RM and (iii) STaRFormer. Here only the architecture of the model is changed, keeping everything else fixed. The respective seeds for each run are stated in the
parenthesis in the header row.
Ablation
Metrics
(42)
(123)
(0)
(63)
(2024)
Median
Average
Std. Dev.
Accuracy
0.88614
0.89796
0.87687
0.86696
0.87750
0.878
0.881
Base
Fo.r-Score
0.86227
0,87591
0.84978
0.84574
0.85735
0.857
0.858
0.012
Accuracy
0.91518
0,90074
0.88235
0.89093
0.8825
0.891
0.894
0.014
STaRFormer with RM
Fo.r-Score
0.90047
0.88006
0.85443
0.8704
0.85923
0.870
0.873
0.018
STaRFormer with DARcM
Accuracy
0.95238
0.89904
0.89904
0.89183
0.89625
0.899
0.904
0.016
Fo.o-Score
0.91589
0,87505
0.87469
0.87114
0,87595
0.875
0.883
0.019
Table 23: Run documentation of the ablation study evaluating the impact of our semi-supervised CL
approach on the model performance the GL dataset [63]. The respective seeds for each run stated in the parenthesis in the header row.
Ablation
Metrics
(42)
(123)
(()
(63)
4 (2024)
Median
Average
Sid. Dev.
Accumecy
0.3238
0.89904
0.89904
0,89183
0.89625
0.899
0.904
semi-supervised (Acr z 0.773)
Soore
0.91586
0,87505
0.87469
0.87114
0,87595
0.875
0.883
0.019
w/o sull-supervieed (Aci, 0.773)
Accuracy
092188
0,88882
0,88882
0.89964
0.90125
0.900
0.900
0.014
Fo.s-Score
0.89874
0.86552
0.86048
0.87804
0.88177
0.878
0.877
0.015
who supervised (Aca, Ru 0.773)
Accurayy
092175
0.87921
0,89423
0,88942
0.89125
0.891
0.895
0.016
Fo.s-Score
0,90169
0.87604
0.86763
0. 0.86696
0.87048
0.870
0.877
0.014
semi-vueeriised (Acu, 0.1)
Accuracy
0.93125
0.89904
0,89363
0,88522
0,89063
0.894
0,900
0.018
Fuor-coree
0.11877
0,87578
0,86639
0,86519
0,87052
0.871
0.879
0.021
semi-oupervised (Acu, 1.0)
Fa.s-Score
Accuracy
0.92452
0.90499
0.87011
0.89483
0,87759
0.90144
0.87942
0 0,89844
0.86601
0,89000
0.878
0.898
0.902
0.880
0.013
semi-supervised (Aex. 5.0)
Accuracy
0.92925
0.90264
0.90505
0. 0.89663
0:90438
0.904
0.908
0.013
0.91513
0.88069
0,88002
0.87347
0.88488
0.881
0.887
somi-ssperrised (Act. 10.0)
Accuracy
0,91838
0.90565
0.90385
0.90925
0,89125
0.906
0.906
0,010
Fo.s-Score
0.90368
0,88400
0.87978
0.88987
0.86998
0.884
0.885
0.013
Table 24: Run documentation of the ablation study evaluating how the size of the masked regions affects the model performance the GL dataset [63]. The respective seeds for each run are stated in the parenthesis in the header row.
Ablation
Metries
(42)
(123)
(0)
4 (2024)
Median
Average
Std. Dev.
y
7
default
0.399
0.05
Accurncy Fo s-Score
0.93238
0,89904
0.89904
0.89183
0,89625
0.904
0,904
0.016
0.91586
0,87505
0.87469
0,87114
0.87595
0.883
0.883
0.019
0.399
0.0
Accuracy Fo s-Score
0.91007
0.92875
0,88041
0,87835
0.89183
0.86425
0.89844
0.87641
0,88875
0.86787
0.898
0,879
0,898
0.879
0.018
0.019
2
0.399
0.1
0.05
Accuracy Fo S-Sore
0.93238
0.89904
0.89904
0,89183
0,89625
0.904
0.904
0.016
0.91586
0,87505
0.87469
0.87114
0.87595
0.883
0.883
0.019
Accuracy
0.92375
0.90352
0.89543
0.89663
0,893755
0.903
0.903
0.012
0.399
0.1
0.10
Soooe
0.90316
0.88335
0.86971
0.87761
0.87655
0.882
0.882
0.399
0.15
Accuracy
0.92875
0,90445
0.89543
0.89603
0,89063
0,903
0,903
0.015
Fg S-oore
0.90977
0.88539
0.86977
0,87752
0.86884
0.882
0,882
0.017
Accuracy
0.91750
0.90565
0.89483
0,89363
0,89312
0.901
0.901
0.011
0.399
0.1
0.20
SSoore
0.89494
0.8839
0.86964
0.87391
0.87221
0.879
0.879
0.910
Accuracy
0.92813
0.88642
0.89663
0.89724
0,89625
0.901
0.901
0.016
0.399
0.1
0.25
Score
0.88427
0.8712
0.87805
0,87629
0.884
0.014
Accurncy
0.92287
0.90204
0.90144
0.89784
0,88875
0.903
0.903
0.013
0.399
0.30
Fo. Score
0.90571
0.88161
0.87608
0.87777
0.86723
0.882
0.882
0.014
57

---

D.3.5 PAM
Table 25: Run documentation of the ablation study on the three ablations of STaRFormer on the PAM dataset [68]; (i) Base, (ii) STaRFormer-RM and (iii) STaRFormer. Here only the architecture of the model is changed, keeping everything else fixed.
Ablation
Metrics
0
3
4
Median
Average
Std. Dev.
Accuracy
0.97917
0,96875
0.96181
0,96544
0.94476
0.965
0.964
0.013
Base
Precision
0.98309
0.97635
0,96303
0,97274
0,94997
0,973
0.969
0.013
Recall
0,97998
0,96654
0.97068
0.97579
0,95619
0,971
0.970
0.009
Fi-Score
0.98184
0.9705
0.96647
0.97414
0,95184
0,971
0.969
0.011
Accuracy
0,97812
0,96122
0.97031
0.9625
0,94901
0,963
0.964
0.011
STaRFormer with RM
Precision
0,978
0,96542
0.9654
0.9687
0,96077
0.968
0.006
Recall
0.97711
0,97067
0,97067
0,96127
0,95837
0,971
0.968
0.007
Fi- Fi-Score
0.97735
0.96777
0,96777
0.96591
0.95893
0.968
0.968
0.007
Accuracy
0.98307
0.98047
0.97786
0.96011
0.97917
0.979
0.976
0.009
STaRFormer with DAReM
Precision
0.97796
0,97683
0,97132
0.97032
0,96945
0,971
0.973
0.004
Recall
0.98016
0.9745
0.97467
0.97275
0.97717
0.975
0.976
0.003
Fi-Score
0.97893
0,97542
0.97278
0.97146
0.97299
0.974
0.003
Table 26: Run documentation of the ablation study evaluating the impact of our semi-supervised CL approach on the model performance on the PAM dataset [68].
Ablation
Metrics
2
3
4
Median
Average
Std. Dev.
Accuracy
0.97786
0.96011
0.97917
0.979
0.009
semi-supervissd (Aer. 0.567)
Precision
0.97796
0.97683
0,97132
0,97032
0.96945
0.971
0.973
0.004
Rocall
0.98016
0.9745
0,97467
0,97275
0.97717
0.975
0.976
0.003
Fi-Seore
0.97893
0.97452
0,97278
0,97146
0.97299
0.973
0.974
0.003
Accuracy
0,97786
0,96402
0,97656
0,96532
0.9272
0.965
0.962
0.021
wro suleppprreee (Acu. su 0.567)
Precision
0.97599
0.97403
0,97192
0,97513
0.95906
0.974
Recall
0.97187
0,9711
0.96882
0.97989
0. 0.96069
0.971
0,970
0.007
Sc-Se
0.93499
0.97201
0,97013
0,97708
0.95936
0.972
0,970
0.007
Accurncy
0.98047
0,95017
0,98177
0.96271
0.95017
0.963
0.968
0.016
wlo supervised (Azu. eu 0.567)
Precision
0.97531
0,97483
0.9708
0.975
Recall
0,97677
0.97115
0.98065
0.97062
0.97006
0.971
0.974
0.005
Fi-Score
0.97222
0.97388
0.9777
0.97265
0.96993
0.974
0,974
0.003
Accuracy
0,95147
0,94709
0.97526
0.95881
0.97786
0.959
0.962
0.014
semi-supervised (Acc 0.1)
Precision
0,96899
0,95585
0 169666
0.96658
0.97207
0.969
0.966
0.006
Recall
0.97509
0.95055
0,97504
0,97247
0.96874
0.972
0.968
0.010
Fj Score
0.97161
0,95235
0,97129
0. 1.96895
0.97003
0.970
0.967
0.008
Accuracy
0.97135
0.97786
0.981N7
0.96532
O. 0,95402
0.971
0.972
0.007
semi-supervised (Aer. L.0)
Precision
0,97248
0.97765
0.97733
0.97445
0.96996
0.974
0.974
0.003
Recall
0.96366
0.96661
0.97731
0,97883
0.97242
0.972
0.972
0.007
Se-re
0.96764
0,97155
0
0.97635
0.97107
0.972
0.973
0.004
Accuracy
0.92803
0.96922
0.98438
0.96792
0.98307
0.969
0.967
0.023
semi-supervised (Acr 5.0)
Precision
0.95442
0,98047
0.98276
0.97708
0.97937
0.979
0.975
0.012
Recall
0.97688
0,97828
0,9801
0.97473
0.977
0.970
0.017
Score
0.94606
0.9785
0.98039
0.97846
0.97682
0.978
0.972
0.015
Accuracy
0.94886
0.96141
0.98698
0.96922
0.98568
0.969
0.970
0.016
semi-supervised (AcL 10.0)
Precision
0,96938
0.96931
0.98449
0.97912
0.98321
0.979
0.977
0.007
Recall
0.96927
0.96652
0.98293
0.98193
0. 0.97697
0.977
0.976
0.007
St-ree
0.96907
0.96733
0.98352
0.98043
0.97947
0.979
0.976
0.007
58

---

Table 27: Run documentation of the ablation study evaluating how the size of the masked regions affects the model performance on the PAM dataset [68].
Ablation
#
Metries
0
2
4
Median
Average
Std. Dev.
Accuracy
0.98307
0.98047
0.97786
0,9601
0.97917
0.976
0,976
0.009
default
0.207
0.3
0.10
Precision
0.97796
0.97683
0.97132
0,97032
0,96945
0,973
0.973
0.004
Recall
0.98016
0.9745
0.97467
0,97275
0,97717
0.976
0,976
0.003
Fi Score
0.97893
0.97542
0.97278
0.97146
0.97299
0.974
0.974
0.003
Accuracy
0.97656
0,96251
0,97786
0,96662
0,96402
0.967
0,970
0.007
0.207
0.3
0.0
Precision
Recall
1.97116
0.97485
0.96728
0.9759
0.97289
0.9759
0,97544
0,98193
0,96955
0.97176
0.971
0.975
0.973
0.974
0.002
0.006
Fi-Score
0.97282
0.97135
0.9742
0,97817
0.97045
0.973
0.973
0.003
Accuracy
0.96875
0.97656
0.96141
0,94105
0.94233
0,961
0.958
0.016
0.207
0.05
Precision
0.96279
0.97400
0,96650
0,96174
0.96014
0,965
0.965
0.006
Recall
0.96229
0,96816
0.97287
0,96853
0.95972
0
0.966
0.005
Score
0.96450
0.96917
0,96938
0,96449
0.95961
0,965
0.965
0.004
Accuracy
0.98307
0,98047
0,97786
0,96011
0.97917
0.976
0,976
0.009
3
0.207
0.3
0.10
Precision
0.97796
0.97683
0.97132
0,97032
0.96945
0.973
0.973
0.004
Recall
0.98016
0.9745
0,97467
0.97275
0,97717
0.976
0,976
0.003
Fi -core
0.97893
0.97542
0.97278
0.97146
0.97299
0.974
0.974
0.003
Accuracy
0,97266
0.96271
0,98828
0,95881
0,97183
0.972
0.971
0.011
0.207
0.15
Precision
0,9703
0.9747
0,98254
0,96929
0,97944
0.975
0,975
0.006
Recall
0.96576
0,96958
0.98727
0,96638
0,9842
0,970
0.975
0.010
Fi-Score
0.96763
0.97153
0,98478
0.96737
0.98167
0.972
0.975
0.008
Accuracy
0.96532
0,96122
0,97031
0.9625
0,94901
0.963
0,962
0.008
5
0.207
0.3
0.20
Precision
Recall
0.97354
0.96551
0.96209
0.97312
0,97067
0,96542
0,96372
0.9687
0.96077
0,95837
0,964
0
0,966
0.967
0.006
0.005
Fi Fi-Score
0.96855
0.96711
0.96777
0.96591
0.95893
0.967
0.966
0.004
Accuracy
0.97917
0.96122
0,97031
0.9625
0.94901
0,963
0.964
0,011
0.207
0.25
Precision
Recall
0,97103
0.97775
0,97312
0.96209
0,96542
0,97067
0,96372
0.9687
0.96077
0,95837
0.969
0,964
0.965
0,969
0,006
0,007
Fi-Score
0.9742
0.96771
0,96777
0,96591
0.95893
0.968
0,967
0,005
Accuracy
0.97266
0,96122
0.97031
0.9625
0.94901
0.963
0,963
0.009
7
0,207
0.3
0.30
Precision
0.96562
0.97312
0,96642
0.9687
0.96077
0
0.967
Recall
0.96632
0,96209
0.97067
0,96372
0.95837
0.964
0,964
0.005
Fi-Score
0.96585
0.96711
0.96777
0,96591
0.95893
0.966
0.965
0.004
D.3.6 P19 Runs
Table 28: Run documentation of the ablation study the three ablations of FTamerrmer the P19 dataset [66]; (i) Base, (ii) STaRFormer-RN and (iii) STaRFormer. Here, only the architecture of the model is changed, keeping everything else fixed.
Ablation
Metries
4
Median
Average
Std. Dev.
AUROC
0.9095
0.88573
0.88623
0.8879
0.86183
0.886
0.886
0.017
Base
AUPRC
0.63754
0,57881
0.60019
0.58167
0.55528
0,582
0.591
0.031
Accuracy
0.9714
0.9665
0.97114
0.97011
0.96856
0.970
0.970
0.002
Fi-Score
0.80053
0.77317
0.78357
0.76393
0.75725
0.776
0.017
Recall
0.87247
0,84609
0.89798
0,85501
0.81718
0,855
0.858
0.030
Precision
0.75358
0.72758
0.72314
0.71227
0.71775
0.723
0.727
0.016
AUROC
0.921
0.87785
0.88049
0.88521
0,86792
0.880
0.886
0.020
STaRFormer with RM
AUPRC
0.67494
0,57197
0.57515
0.58078
0.5798
0,580
0.597
0.044
Accuracy
0.9714 0.96702
0,96882
0.97088
0.75293
0.969
0.926
0.097
Fi-Score
0.81059
0.76449
0.74701
0.76374
0.97063
0.764
0.811
0.092
Recall
0.85729
0.86743
0.91123
0.8728
0.86125
0,867
0.874
0.022
Precision
0.77563
0.70946
0.6806
0.70699
0.69718
0.714
AUROC
0.91218
0.89776
0.89279
0,89397
0.87469
0.894
0.013
STaFFormer with DARcM
AUPRC
0.66579
0.60643
0.61391
0.60891
0.57196
0.609
0.613
0.034
Accuracy
0.97346
0.96212
0.96985
0.97243
0.97114
0.971
0.970
0.005
F Score
0.81241
0.77328
0.77309
0.78401
0.77452
0.775
0.783
0.017
Precision
0.89539
0.79238
0.886
0,85432
0.87199
0,872
0.860
0.041
Recall
0.76017
0.75682
0.7142
0.73374
0.71976
0.737
0.021
59

---

D.3.7
P12 Runs
Table 29: Run documentation of the ablation study on the three ablations of STaRFormer on the P12 dataset [67]; (i) Base, (ii) STaRFormer-RM and (iii) FTaRFormer. Here, only the architecture of the model is changed, keeping everything else fixed.
Ablation
Metries
0
2
3
4
Median
Average
Std. Dev.
AUROC
0.85655
0.85965
0,7408
0.85859
0.83593
0.85655
0.830
0.051
AUPRC
0.51476
0.55478
0.33127
0.54069
0.52082
0.52082
0.492
0.092
Base
Accuracy
0,60384
0.86822
0.86322
0.82402
0.84487
0,84487
0.801
0.111
Fi-Score
0,54218
0.57294
0,55269
0.7109
0.60415
0,57294
0.597
0.068
Recall
0.60725
0.79105
0.70883
0.68606
0.74284
0.70883
0.707
0.068
Precision
0.73951
0,55995
0,54662
0.76901
0.585
0.585
0.640
0.106
AUROC
0.85727
0.86518
0.83434
0.86184
0.82935
0.857
0.850
0.017
STaRFormer with RM
AUPRC
0,51466
0,5758
0.47908
0,53086
0,49449
0.515
0.519
0.037
Accuracy
0,8849
0.8799
0,86072
0,8824
0,77815
0.880
0.857
0.045
F,-Soree
0.61662
0.63967
0,86072
0.8824
0.68701
0.687
0.737
0.125
Recall
0,79122
0,82993
0.70894
0,77173
0.66954
0.772
0.754
0.064
Precision
0,58794
0.60576
0.70578
0.69176
0,75015
0.692
0.668
0.069
AUROC
0.85853
0.85989
0,84508
0,86469
0.83435
0.859
0.853
0.012
STaRFormer with DAReM
AUPRC
0.51325
0.53224
0.51175
0.5426
0,50128
0.520
0.017
Accuracy
0.88657
0.87239
0,86986
0.88407
0.84257
0.872
0.871
0.018
Fy-Score
0.67798
0.61871
0.65169
0.70709
0.63258
0.652
0.658
0.035
Recall
0.64426
0.59163
0.62328
0.67095
0.60915
0.623
0.628
0.031
Precision
0.75933
0.78038
0.72938
0.78793
0.71802
0.759
0.755
0.031
D.3.8 UEA Benchmark
Table 30: Complete results of the UEA benchmark for 30 multivariate time series datasets [70].
Dessel
Maacc
Bllow"
15-100*
TNC-
3d615"
Bocid"
Raciet
InfiTS,
TincAULL
DA00K"
Penser
AF
03.13
0.353
11061
0.333
01360
0.135
0.210
0.310
01557
01133
0.51
0:261
0.400
1/66
0.557
0.50
3.500
0.581
01.85
6167
0.911
11911
(194)
0971
0166
0.500
0.61
01041
CK
03.10
1160
1100
0.533
01307
0088
0301
0.566
1
01355
0.41
1
1,000
1160
1
0.878
0.508
8.000
5728
0.80
07774
0.00
0981
6711
0650)
(3740
0130
6716
6.571
0.30
0.89)
0.333
2374
0.33
0,804
0.52
0.59
0.599
01356
0555
0053
0585
0910
0330
0.154
0.33
0305
03385
0797
0.511
01591
01330
0.11
PM
0.530
3.400
3360
0.80
01460
6470
0480
0.000
0553)
11.00
0350
0920
0,600
0.60
0541
EW
0.255
0.61
0.29
0.51
0.59
0355
0.564
0.62
0.281
03325
0.768
0.517
5746
0.31
676
6122
6151
0.711
0.33
0771
0746
6750
0772
0.56
TV
0.94
0.049
257
0291
0133
007
0164
C.84
0575
06860
8555
0.89
0162
0.90
LSST
0.551
B350
5466
0.59
0.41
0.35
8531
5.59
0625
0.53
0576
0361
0/00
8.5
0.59
0.00
14300
0600
5660
13581
5350
0317
6911
0961
6911
0389
IS
0.913
0.511
B240
0354
0.34
0,90
0331
053
0.99
0550
675
0201
5506
0343
P50
0.141
B180
0336
0355
635
0331
C319
037
1.31
0111
8351
6351
01/4
0174
0.31
6776
033
SCP1
01944
03335
5751
0911
050
0915
6561
6581
SCP
(1561
0.514
0114
0351
6.0
257
0555
0.56
0101
0/00
660
SWI
0.300
5331
8367
0111
0330
8/00
9-67
B467
0.330
8660
6:11
Ang Anaray
0.990
0.605
0.555
01565
0600
0.064
0.144
0215
0.79
0:125
8/30
0.03
6.235
0795
Arg Rak
105
8.6
5.9
65
100
49
58
Top Sonze
26
54
27
DS Cozn
10
29
26
30
30
30
30
30
30
30
30
30
30
30
1.:54
D:691
D/631
0.255
BA77
0717
0.52
02233
04335
9:334
677A
6,50
6.71
0.73
Rack 25
15
14
12
75
52
A1
Aucumcy
0.328
0.502
0337
0.335
6715
0734
0.277
07766
0351
0371
6736
0,710
6:17
0200
Ang. Raak9
11.86
9.0
123
11.1
11.3
10.
9.0
100
67
74
35
6.4
53
63
33
The model results marked with 9 taken from the [21]. from [27]. 11 from [26] and from
60

---

D.3.9 STaRFormer Approach Ablation Runs
Table 31: STaRFormer architecture ablation study results (Accuracy).
Base
STaRFormer- RM
STaRFormer
DKT
0.849 = 0.002
0.845 # 0.001
0,852 = 0.003
GL
0.881 + 0.012
0.894 + 0.014
0.904 0.015
P19
0.970 + 0.002
0.970 = 0.002
0.970 = 0.005
P12
0.801 1 0.111
0.857 1 0.045
0.871 0.018
PAM
0.964 0.013
+: 0.011
0.976 0.009
EW 0.752
0.799
0.850
EC
0.402
FD
0.687
0.673
0.697
HW 0.336
0.327
0.373
IIB 0,786
0.772
0.772
JV
0.990
0.982
0,990
PD 0.982
0.980
0,983
0.909
0.922
0,943
SCPI
0.906
0.891
0,913
SCP2
0.630
0.620
0,635
SAD
0,990
0,983
0.989
UW
0.881
0.838
0.894
Yahoo
0.988
0.991
0.992
KPI
0.982
0.980
0.981
Avg. Acc.
0.824
0.826
0.841
Rank
3
2
1
Avg Rank
2.1
2.5
1.2
Top Scores
2
15
1-v-1
Base
12
3
1
STaRFormer
14
16
In Table 31, display the complete metric scores summarized in Table 6. To ensure consistence, report the accuracy for every dataset, as available for each dataset. However, this is not the idcal metric for many datasets. Heavily skewed datasets like P19 and P12 anomaly detection datasets, where anomalous elements appear much less frequently than regular elements, F1 would be better score consider.
D.3.10 Univariate Anomaly Detection Benchmarks
Table 32: Run documentation of the ablation study the three ablations of STaRFormer the uni- variate anomaly detection benchmark datasets; (i) Base, (ii) STaRFormer RM and (iii) FTaRFormer. Here, only the architecture of the model is changed, keeping everything else fixed.
Yahoo
KPI
Method
F
Precision
Recall
Accuracy
F
Precision
Recall
Accuracy
Base
0.685
0.671
0.767
0.988
0.814
0.857
0.780
0.982
STaRFormer-RNy
0.737
0.801
0.696
0.991
0.737
0.910
0,670
0.980
STaRFormer
0.789
0.772
0.807
0.992
0.830
0.852
0.811
0.981
61

---

D.3.11 TSR Benchmark
Table 33:
Complete results of the TSR Benchmark for 19 time series datasets.
SYI
Bondern
Inerp-
STAR-
Al
3.485
3,603
3:555
2.233
2.004
5,252
4.221
0.33
4.100
2.299
23000
1J85
6.433
3.81
1.844
09.75
110.53
110.544
00.07
93.138
125
115.00
120058
96.109
95.05
96
116.7)
115.421
SPM55
06:375
09.37
75334
71,47
(3.301
99.066
18.194
16.15
88.255
12.713
02770
96.227
06,853
03228
85.371
$4.004
3.35
BIDAKCRA
1365
3.365
4.100
3.774
4.30
1.333
6.357
0.135
3.500
3,452
1.93
1578
1931
300
3487
3.913
SITANCSPOO
8819
4.50
5.13
5.23
596
1983
5331
6.130
1311
0033
0127
01966
0305
0315
0.015
16012
11000
6J812
03877
03323
0053
0107
0633
1382
0.0
0003
0126
8.06
20114
0J15
0.09
00019
14016
0.006
03877
0404
0000
0340
R886
HICC
147.50
147.500
519.1%
501
345.899
331,990
412.995
427
320
112.99
162.44
153.00
155.16
518
147.20
HPCZ
463235
40 104
57:340
$5.991
66.02
66.773
11,479
04.775
393913
51,83
32.00
46.2
21410
30011
42.00
31.245
LEMI
$7,681
15.03
30:731
33.443
03:87
39-973
35.85
35,35
30.553
35.7%
NBS
0.142
0.843
0.143
0.188
8.142
0.30
0.157
0.196
116
6,841
6.18
0.190
0.10
016
0.187
6.138
0.13
6.138
0.000
Arg. Mel Monn
4.121
4.133
758
0.125
A.M
100
-0119
10.30
40,38
Moua
tiitornne Ltank:
of Tip
The results marked with official benchmark huttp://taergggeeeiiio org/).
62


Reviews:
[{"review_id": "isBmcqySjn", "reviewer": "Reviewer_nw2T", "summary": "This paper proposes STaRFormer, a Transformer-based framework designed to handle irregularly sampled and non-stationary sequential data effectively. The key contributions include a dynamic attention-based regional masking (DAReM) mechanism and a semi-supervised contrastive learning (CL) method, explicitly coupling representation learning with downstream tasks. Experiments conducted across 19 diverse datasets, including irregularly sampled and real-world non-stationary datasets, demonstrate that STaRFormer achieves competitive  performance relative to current state-of-the-art (SOTA) methods.", "questions": "Is there a reason for putting the supplementary material in the main paper? I think the authors didn't follow the submission form properly.", "limitations": "The manuscript adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Novel Dynamic Masking Approach (DAReM): The proposed attention-based regional masking strategy is innovative and addresses the critical challenge of handling irregular sampling and non-stationarity effectively by adaptively perturbing key regions of the sequence based on learned attention scores.\n\n2. Effective Semi-Supervised Contrastive Learning: The proposed contrastive learning paradigm combines supervised (class-wise) and self-supervised (batch-wise) signals, explicitly linking representation learning with downstream tasks, contributing to robust representations and improved performance.\n\n3. Comprehensive Experimental Validation: The extensive evaluation across multiple datasets (regular, irregularly sampled, and non-stationary time series) provides strong empirical support for STaRFormer's versatility and effectiveness in diverse real-world scenarios.\n\n4. Thorough Ablation Studies: The paper provides extensive ablations and clearly demonstrates the incremental contributions of each proposed component (DAReM, semi-supervised CL), supporting the methodological design choices.\n\nWeaknesses\n\n1. Limited Computational Efficiency: The proposed dynamic masking strategy, relying on full attention mechanisms, inherently incurs computational complexity of 𝑂(𝑁^2), potentially hindering scalability for very long sequences, as acknowledged by the authors. Although this complexity affects only training, it can limit practical usability on larger datasets.\n\n2. Marginal Improvements on Certain Datasets: Despite consistent outperformance across most datasets, performance gains are modest in some cases, which may raise concerns regarding whether the additional complexity introduced by DAReM and contrastive learning fully justifies the marginal performance gains.", "was_revised": true, "final_justification": "Most concerns from the initial review have been addressed in the rebuttal. The authors clarified the computational efficiency limitation, noting potential future improvements (e.g., flash attention), and explained modest gains on certain datasets with additional robustness analysis and broader benchmark results. The performance improvements, especially on the full UEA benchmark and regression tasks, further support the method’s effectiveness. Formatting issues are minor and easily addressed. Overall, the rebuttal reinforces the initial assessment, and I maintain my weak accept recommendation.", "rebuttal": "We express our gratitude to the reviewers for their careful reading and critical assessment of our work. We are greatly encouraged by the overall positive reception of our work, reflected in the reviewers' consensus across most reviewing criteria. We are glad to hear they found our approach of addressing irregularly sampled and non-stationary time series 'innovative' (nw2T), 'novel' (KH4k, 8jfK) and 'effective' (nw2T, 7K1F). Moreover, we are pleased that the reviewers acknowledged our efforts to evaluate our approach through 'comprehensive' (nw2T), 'thorough' (nw2T), and 'extensive' (7K1F, 8jfK) experiments and ablation studies. We are satisfied that these experiments demonstrate the effectiveness (nw2T, 7K1F) and strong performance (KH4k) of our approach compared to state-of-the-art baseline models. We will address your comments below and incorporate your feedback accordingly.\n\n**Q1)** _*Limited Computational Efficiency: The proposed dynamic masking strategy, relying on full attention mechanisms, inherently incurs computational complexity of $𝑂(𝑁^2)$, potentially hindering scalability for very long sequences, as acknowledged by the authors. Although this complexity affects only training, it can limit practical usability on larger datasets.*_ \n\nAs you noted, we briefly acknowledge this limitation in the main paper. The computational complexity associated with transformers is a recognized concern, which is the reason we aim to explore methodologies that are more computationally efficient, such as flash attention in future work. Our initial objective was to establish a performance-effective methodology before potentially enhancing computational efficiency.\n\n**Q2)** _*Marginal Improvements on Certain Datasets: Despite consistent outperformance across most datasets, performance gains are modest in some cases, which may raise concerns regarding whether the additional complexity introduced by DAReM and contrastive learning fully justifies the marginal performance gains.*_\n\nIn the DKT dataset, the initial motivation (task), we aim to address this concern through an additional robustness analysis and further elaborate on the challenges associated with the ground truth labeling in Appendix Section D.2. Regarding the numerous other datasets, the model performances are generally robust, achieving accuracy rates up to 99% in certain instances. Consequently, only marginal improvements are feasible in many situations. During the review phase, we computed the entire UEA benchmark, as presented in Table 3, comprising 30 datasets, along with an additional extrinsic regression task involving 19 datasets. On the complete UEA benchmark, STaRFormer (0.795) surpasses the next best models, TARNet (0.755) and TimesURL (0.752), by approximately 4 percentage points. In the regression task, see the tseregression (dot org) website, we also achieve state-of-the-art performance in terms of average relative mean difference, with STaRFormer (-0.254) outperforming the next best model, Rocket (-0.245), by approximately 1 percentage point. We are ready to incorporate these findings upon your suggestion to substantiate performance scores and address any further potential evaluation concerns.\n\n**Q3)** _*The authors did not follow the format of NeurIPS and included supplementary material in the main paper.*_\n\nAs stated in the criteria for paper formatting instructions on the NeurIPS website in the Call for Papers section: \"The authors may optionally choose to include some or all of the technical appendices in the same PDF above. But those included parts cannot be changed after the full submission deadline.\" We understood this as an option to attach the supplementary material to the main PDF. Upon your recommendation, we are happy to attach it as an additional file.\n\nIf our responses have adequately addressed your questions and concerns, we would appreciate your consideration in updating your recommendation score. Otherwise, we welcome further discussion."}, {"review_id": "TzhMEZvAm6", "reviewer": "Reviewer_KH4k", "summary": "The paper proposes a time-series classification model combining multiple task objectives during training to learn better representations that support better classification performance. Among these multiple tasks are: (1) the classification loss, (2) the contrastive loss on instances produced via (a) task-aware masking and (b) class labels. Their results and ablations show a better performance over baselines.", "questions": "See Cons.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Pros\n\n1. The idea of combining masking and contrastive loss in a semi-supervised setup on time-series data seems novel.\n2. The experiments show good performance over the baselines.\n3. Ablation shows benefits of DAReM over random masking.\n4. Visualization shows a good separation of clusters.\n\n## Cons\n\n1.  I feel that there needs to be a better utilization of the space in the main paper. \n    1. The paper shares too few details about DAReM in the main paper. For instance, it is not clear what φ, γ, ζ mean in more exact terms.\n    2. It may be a good idea to perhaps offload less essential related work (which takes a whole page) to the appendix.\n    3. The font size of the tables is too small.\n2. In the ablation table (Table 6), the variations in the scores are too tiny. I wonder if this is expected or informative.\n3. Mainly, I don’t see a connection between their initial motivation of tackling irregularly sampled time series and the main novelty of their method (DAReM + CL). \n4. Also, I don’t see why irregular sampling would be a concern for, say, TARNet or SLOTS, since one can simply construct a positional embedding using the actual time-stamp.", "was_revised": true, "final_justification": "One of my remaining concern (and justification for my score) is the repeated mention of an *industrial partner*. Ideally, mentioning such details should be reserved for the final / camera ready version – and in my mind, it (softly) hurts the pure double-blind nature of the review process.", "rebuttal": "We express our gratitude to the reviewers for their careful reading and critical assessment of our work. We are greatly encouraged by the overall positive reception of our work, reflected in the reviewers' consensus across most reviewing criteria. We are glad to hear they found our approach of addressing irregularly sampled and non-stationary time series 'innovative' (nw2T), 'novel' (KH4k, 8jfK) and 'effective' (nw2T, 7K1F). Moreover, we are pleased that the reviewers acknowledged our efforts to evaluate our approach through 'comprehensive' (nw2T), 'thorough' (nw2T), and 'extensive' (7K1F, 8jfK) experiments and ablation studies. We are satisfied that these experiments demonstrate the effectiveness (nw2T, 7K1F) and strong performance (KH4k) of our approach compared to state-of-the-art baseline models. We will address your comments below and incorporate your feedback accordingly.\n\n**Q4)** _*Space utilization and offload less essential work (1.1, 1.2).*_\n\nDue to space constraints, we have chosen to offload specific information regarding the DAReM procedure. We explicitly provide the formal definition of each hyperparameter used to create the regional masks in the main document, on page 4, line 153. We consider it essential to refer to the related work, especially for readers who do not possess a similar background. Therefore, we have chosen to offload certain mechanics to the supplementary section, which interested readers can access, while maintaining the entirety of the related work. It is noteworthy that offloading or reducing the related work section can often lead to criticism from reviewers concerning missing related works; thus, a balanced trade-off is necessary. However, upon your recommendation, we are more than willing to accommodate your suggestion in the camera-ready version.\n\n**Q5)** _*The font size of the tables is too small (1.3).*_\n\nIn the paper, we exclusively use the standard font size from the LaTex template. To retain as much information as possible and maintain transparency when selecting baseline models, we aimed to include as many baselines as feasible to avoid scrutiny for not including enough baselines. To ensure the tables fit within the page limit requirements, we enforced the tables to be text-width size using the command \\adjustbox{max width=1.0\\textwidth}{...}. In the camera-ready version, upon your recommendation, we would be happy to remove some baselines from the main document and include them in a comprehensive table in the supplementary information. As stated in our response to **Q2)**, we are also able to include additional experiments for Table 3, along with an additional task, which would necessitate reformatting Table 3 anyway.\n\n**Q6)** _*In the ablation table (Table 6), the variations in the scores are too tiny. I wonder if this is expected or informative.*_\n\nIndeed, the results for DKT vary only minimally. This is somewhat expected, considering the underlying data constellation of the DKT dataset, as explained in **Q2)** and Section D.2. When examining the other two datasets, we observe a deviation of about 1.3 to 2.7 percentage points, which is significant, especially considering that the experiments are conducted using 5 different seeds and data splits. Additionally, given that we are only testing different configurations while maintaining the overall modeling architecture, we do not expect to see large variations. We believe that the table provides reasonable evidence to conclude that: 1. semi-supervised CL is beneficial compared to self-supervised or supervised-only approaches by themselves and 2. masking regions is advantageous over masking individual elements, with the region parameter needing to be fine-tuned for optimal performance, as stated in the paper. This evidence supports our architectural choices, hence the ablations are also informative.\n\n**Q7)** _*Mainly, I don’t see a connection between their initial motivation of tackling irregularly sampled time series and the main novelty of their method (DAReM + CL).*_\n\nIn the introduction, we clearly state the initial motivation for modeling specifically for non-stationary and irregularly sampled time series based on observations from our industrial partner. In the related work section, at the end of each paragraph, we describe how our proposed approach connects to potential challenges and how our modeling approach differs from others. This is followed by a distinct statement on page 4, line 145, explaining our rationale and how our approach sets itself apart. To summarize, by introducing artificially masked regions in sequences, we emphasize the sensation of irregular sampling (and non-stationarity). The rationale is that this emphasis will focus the learning on these irregularities when the transformer model forms the embedding representations during training in the latent space. Via our contrastive formulation, we then aim to align even disparate embeddings in the latent space to make the latent space, and hence the model, more robust to these irregularities. It is relatively common practice to mask sequences to create irregularly sampled data, as is done to achieve the datasets in Table 2 for example. To demonstrate this, we conduct an ablation study where we perform no masking and random masking (see Section 3.1, Table 5 and Table 27), showing that this approach outperforms the no masking and random masking approaches.\n\n**Q8)** _*Also, I don’t see why irregular sampling would be a concern for, say, TARNet or SLOTS, since one can simply construct a positional embedding using the actual time-stamp.*_\n\nTo the best of our understanding, TARNet does not construct positional embeddings based on actual timestamps; at least, this is not achieved in their code implementation. We assume you might mean TimesURL? In the context of irregularly sampled time series, transformer models may face challenges due to their inherent reliance on regular positional encoding (which assumes equally spaced time intervals). Modeling approaches can indeed mitigate the issue of irregular sampling by using positional embeddings based on actual timestamps. However, this would require access to the actual timestamps, which in many datasets, including the dataset from our industrial sponsor, is not the case. Hence, we do not know/have real access to the exact timestamps and cannot account for the temporally contextualized informed embeddings.\n\nIf our responses have adequately addressed your questions and concerns, we would appreciate your consideration in updating your recommendation score. Otherwise, we welcome further discussion."}, {"review_id": "jyHi7yfpk8", "reviewer": "Reviewer_7K1F", "summary": "This paper propose a Transformer-based framework called STaRFormer for sequential data modeling. They state that traditional LSTM and Transformer models do not perform well on non-stationary and irregularly sampled time series. And to tackle these challenges, they introduce a novel semi-supervised, task-informed representation learning technique using a dual-encoder (towers) architecture: (1) In one tower, the model utilize masked token modeling during training for representation learning to improve the robutness of the model in handling potential non-stationary data during application, and (2) in another tower, the model perform supervised downstream task prediction. A contrastive loss is then used maximizes the agreement between the latent representations from both towers, improving the model's robustness and generalization. Experiments demonstrate the effectiveness of the model.", "questions": "I have attached my questions in the Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Disclaimer:** I am not an expert in this area, but will do my best to provide helpful evaluation:\n\n## Strengths\n\n1. The paper addresses the important and practical challenge of modeling sequential data that is non-stationary and irregularly sampled. This sounds to me a common issue in many real-world applications, such as digital car keys prediction that the authors used to motivate this work, making the problem highly relevant.\n\n2. The proposed model, STaRFormer, is shown to be highly effective through extensive experiments. The improvement is not aways significant, but they are quite consistent across tasks.\n\n3. The claims are well-supported by a comprehensive set of ablation studies.\n\n## Weaknesses\n\nI mainly have some concerns on the motivation on some design choices of the model:\n\n1. The paper states that LSTMs and Transformers \"typically assume that the data is fully observed, stationary, and sampled at regular intervals.\" This might be a reasonable assertion for LSTMs, which inherently process inputs in a sequential order. However, it is not clear to me why this will apply to Transformers as well. For instance, with architectural variants like relative positional embeddings, Transformers can be adapted to handle signals sampled at dynamic intervals. It would be nice to elaborate on why standard Transformers are still considered ill-suited for such data.\n\n2. The proposed method dynamically masks regions that are deemed important based on attention scores, which is different from the random masking used in many self-supervised methods. My understanding is that the purpose of masking, as seen in paradigms like Masked Image Modeling (MIM), is to incentivize representation learning by forcing the model to learn the correpondance between regions. From this perspective, masking the *most important* regions seems counter-productive, as it removes the most informative signals for reconstruction. Could the authors provide more intuition on why this strategy is more effective than random masking?", "was_revised": true, "final_justification": "The authors provided detailed clarifications on both points I raised: (1) why standard Transformers are less suited to irregularly sampled time series without timestamp information, and (2) the rationale behind importance-based masking in the context of contrastive learning. These responses have partially addressed my concerns, though some minor issues remain. In particular, for the second point, while the authors’ contrastive-learning-based justification is clear, the potential trade-off of masking highly informative features for representation quality remains a conceptual concern. \n\nI would also like to note that works proposing new architectures should be encouraged when they provide reasonably solid empirical evaluation. Therefore I will maintain my positive rating.", "rebuttal": "We express our gratitude to the reviewers for their careful reading and critical assessment of our work. We are greatly encouraged by the overall positive reception of our work, reflected in the reviewers' consensus across most reviewing criteria. We are glad to hear they found our approach of addressing irregularly sampled and non-stationary time series 'innovative' (nw2T), 'novel' (KH4k, 8jfK) and 'effective' (nw2T, 7K1F). Moreover, we are pleased that the reviewers acknowledged our efforts to evaluate our approach through 'comprehensive' (nw2T), 'thorough' (nw2T), and 'extensive' (7K1F, 8jfK) experiments and ablation studies. We are satisfied that these experiments demonstrate the effectiveness (nw2T, 7K1F) and strong performance (KH4k) of our approach compared to state-of-the-art baseline models. We will address your comments below and incorporate your feedback accordingly.\n\n**Q9)** _*The paper states that LSTMs and Transformers \"typically assume that the data is fully observed, stationary, and sampled at regular intervals.\" This might be a reasonable assertion for LSTMs, which inherently process inputs in a sequential order. However, it is not clear to me why this will apply to Transformers as well. For instance, with architectural variants like relative positional embeddings, Transformers can be adapted to handle signals sampled at dynamic intervals. It would be nice to elaborate on why standard Transformers are still considered ill-suited for such data.*_\n\nPrimarily, this is not feasible for us because, as in many of the underlying datasets, we are only provided with the time series features, not the actual timestamps a priori. This includes the main motivation (DKT) for the work from our industrial sponsor. Hence, it is not feasible to construct exact time intervals to effectively apply temporally contextualized embeddings. Upon your recommendation, we are happy to elaborate on this point in the camera-ready version, if you deem it necessary. For additional information, please refer to the answer to **Q8)**.\n\n**Q10)** _*The proposed method dynamically masks regions that are deemed important based on attention scores, which is different from the random masking used in many self-supervised methods. My understanding is that the purpose of masking, as seen in paradigms like Masked Image Modeling (MIM), is to incentivize representation learning by forcing the model to learn the correspondence between regions. From this perspective, masking the most important regions seems counter-productive, as it removes the most informative signals for reconstruction. Could the authors provide more intuition on why this strategy is more effective than random masking?*_\n\nThank you for the insightful comment, reflecting the core arguments regarding the dynamic masking method presented in the paper. We believe your understanding to be indeed accurate. The key distinction in our approach lies in the fact that we do not base our model updates on the reconstruction term between the correlated views, as is commonly done in approaches like Masked Language Modeling (MLM) or Masked Image Modeling (MIM). Instead, we employ CL to align the two distinct latent spaces derived from masked and unmasked sequences. By forming positive pairs between batch-wise and class-wise samples, we consider the correspondence not only to the actual sequence but also to other sequences within the same class. In the class-wise case, this implies that there is no direct correspondence in the traditional sense. Our formulation differs from the 'classical' approach by not directly modeling the correspondence between two regions, as seen in methods like MIM, as you mentioned. Instead, we focus on the degree of difference in the latent space created by the masking, with the contrastive formulation aiming to align even disparate embeddings in the latent space. Thus, we aim to create more disparate representations by specifically masking their key important features. Through the creation of more diverse and challenging positive pairings, we strive to enhance and develop a more robust latent space, which the model uses to perform downstream tasks effectively.\n\nIf our responses have adequately addressed your questions and concerns, we would appreciate your consideration in updating your recommendation score. Otherwise, we welcome further discussion."}, {"review_id": "tdM3alSGXY", "reviewer": "Reviewer_BjfK", "summary": "The paper proposes STaRFormer, a unified framework for time series modeling that addresses challenges such as non-stationarity and irregular sampling. It introduces a novel Dynamic Attention-based Regional Masking (DAReM) module that adaptively masks task-relevant regions based on attention scores. The framework employs a semi-supervised contrastive learning approach, combining self-supervised batch-wise contrastive loss with supervised class-wise loss to improve representation learning for downstream tasks. STaRFormer is evaluated on multiple benchmark datasets across different tasks, demonstrating state-of-the-art performance.", "questions": "As weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The framework composes formulations for both sequential-level and element-level supervision, enabling adaptability to diverse downstream tasks such as classification and anomaly detection.\n\n 2. The Dynamic Attention-based Regional Masking (DAReM) module introduces a novel masking strategy that leverages attention scores to identify and focus on task-relevant regions.\n\n3. The paper presents extensive experiments and ablation studies, and the results show improvements compared to previous methods.\n\nWeaknesses:\n1.  It is unclear whether the trained STaRFormer model can generalize across multiple downstream tasks or if it must be trained separately from scratch for each task, limiting its flexibility and generability compared to task-agnostic frameworks.\n\n2. The proposed DAReM masking method is not compared against other baseline masking strategies (e.g., random, uniform, or fixed-region masking), making it difficult to isolate and verify its effectiveness.\n\n3. Figure 1(b) is not well explained in the main paper and is difficult to interpret without first reading Supplementary Section B.3.\n\n 4. Important figures such as Fig. 8 (which is referred to in the main text) are only included in the supplementary material, reducing accessibility to explanations within the main paper.", "was_revised": true, "final_justification": "After reviewing the rebuttal and additional clarifications, I acknowledge that several of my earlier concerns, such as the readability of certain sections and the clarification of the task-specific design, were addressed.  I want to note that my evaluation is not intended to penalize the authors for being transparent about limitations, but on weighing the trade-offs between robustness/performance and computational efficiency/flexibility. While the efficiency limitation is acknowledged in the paper, the runtime and memory usage still indicate substantial overhead relative to lighter baselines for marginal accuracy improvements in certain settings. For these reasons, I believe my current rating reflects both the contributions of the work and the remaining concerns.", "rebuttal": "We express our gratitude to the reviewers for their careful reading and critical assessment of our work. We are greatly encouraged by the overall positive reception of our work, reflected in the reviewers' consensus across most reviewing criteria. We are glad to hear they found our approach of addressing irregularly sampled and non-stationary time series 'innovative' (nw2T), 'novel' (KH4k, 8jfK) and 'effective' (nw2T, 7K1F). Moreover, we are pleased that the reviewers acknowledged our efforts to evaluate our approach through 'comprehensive' (nw2T), 'thorough' (nw2T), and 'extensive' (7K1F, 8jfK) experiments and ablation studies. We are satisfied that these experiments demonstrate the effectiveness (nw2T, 7K1F) and strong performance (KH4k) of our approach compared to state-of-the-art baseline models. We will address your comments below and incorporate your feedback accordingly.\n\n**Q11)** _*It is unclear whether the trained STaRFormer model can generalize across multiple downstream tasks or if it must be trained separately from scratch for each task, limiting its flexibility and generability compared to task-agnostic frameworks.*_\n\nYou raise a valid point that highlights a limitation inherent to any non-task-agnostic model. Thus, this limitation applies to all non-task-agnostic models and is not exclusive to STaRFormer. However, we do not believe this is unclear in the paper, as we explicitly state in lines 81-82 on page 2 and lines 117-119 on page 3 that we propose a task-coupled approach, thus implying that STaRFormer cannot be task-agnostic. Moreover, the flexibility within this context is constrained, as, to the best of our knowledge, all existing task-agnostic models have been evaluated on tasks for which they were previously trained to extract universal representations. Therefore, from the perspective of this study, these models merely decouple the training process between representation and task, which can actually introduce an additional layer of complexity, as it requires two distinct training pipelines. For instance, in the case of TimesURL, optimizing a Support Vector Classifier (SVC) for large datasets, such as DKT, can be quite challenging and time-consuming, as scikit-learn does not natively support GPUs. This is evident in Table 14 (Appendix Section D.3.1) for the DKT dataset, where the runtimes of TARNet, TimesURL, and STaRFormer can be compared. The runtime of TimesURL is significantly longer. Furthermore, it should be noted that task-agnostic models may experience reduced performance and effectiveness as a trade-off for their increased flexibility. Within this scope, the extent of performance degradation is unclear, and we are currently unable to validate or refute the degree of degradation due to the lack of supporting evidence.\n\n**Q12)** _*The proposed DAReM masking method is not compared against other baseline masking strategies (e.g., random, uniform, or fixed-region masking), making it difficult to isolate and verify its effectiveness.*_\n\nWe would like to point out that, indeed, we have compared the masking scheme against random masking (RM) and no masking across all datasets used in our study. These results are summarized in section 4.3.1 and in Table 5, while extensive results can be found in Table 27. Considering the other three ablation studies, we deemed two variations sufficient in this ablation to convince the reader; however, upon your recommendation, we could add additional variations for the camera-ready paper.\n\n**Q13)** _*Figure 1(b) is not well explained in the main paper and is difficult to interpret without first reading Supplementary Section B.3.*_\n\nThank you for providing this perspective. As mentioned in the response to **Q4)**, we had to transfer some contents to the appendix. This necessitates finding the appropriate balance between what to retain in the main body and what to relocate to the supplementary information. At the time of submission, we believed the explanation provided was sufficient for comprehension, and for readers with a particular interest in the masking procedure, we directed them to the relevant section in the appendix. Considering your suggestion, we could elaborate on this section in greater detail with additions from the supplementary material for the camera-ready version.\n\n**Q14)** _*Important figures such as Fig. 8 (which is referred to in the main text) are only included in the supplementary material, reducing accessibility to explanations within the main paper.*_\n\nFrom our perspective, Fig. 8 in particular, is a generic schematic that can also be found in other CL-related publications, such as the original SimCL (NT-Xent) paper [1]. Hence, although we originally had it included in the main PDF, we didn't deem it important enough to relocate any other content to the appendix. That is why we chose to link it to the main PDF so that it would be easily accessible to readers seeking additional information. Upon your recommendation, we would be happy to include Fig. 8 in the main PDF.\n\nIf our responses have adequately addressed your questions and concerns, we would appreciate your consideration in updating your recommendation score. Otherwise, we welcome further discussion.\n\n[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, June 2020. arXiv:2002.05709 [cs, stat]."}]

Author rebuttal:

Meta-review: The authors propose a Transformer-based framework designed to handle irregularly sampled and non-stationary sequential data effectively. The key contributions include a dynamic attention-based regional masking (DAReM) mechanism and a semi-supervised contrastive learning (CL) method, explicitly coupling representation learning with downstream tasks. Experiments conducted across 19 diverse datasets, including irregularly sampled and real-world non-stationary datasets, demonstrate that STaRFormer achieves competitive performance relative to current state-of-the-art (SOTA) methods. During rebuttal, most concerns are addressed, including clarifying the computational efficiency limitation, the motivation and details of the proposed components, and providing additional robustness analysis/broader benchmark results. The final ratings are three borderline acceptances and one borderline rejection. However, the borderline rejection reviewer's final concern is "the repeated mention of an industrial partner". The AC checked that the double-blind requirement has not been breached. Hence, it is not clear what the remaining key concern is. As a result, the AC recommends acceptance.
Decision: Accept (poster)