Title: STaRFormer: Semi-Supervised Task-Informed Representation Learning via Dynamic Attention-Based Regional Masking for Sequential Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning

transformer

sequential modeling

contrastive learning

semi-supervised learning

time series
Abstract: Understanding user intent is essential for situational and context-aware decision-making. Motivated by a real-world scenario, this work addresses intent predictions of smart device users in the vicinity of vehicles by modeling sequential spatiotemporal data. However, in real-world scenarios, environmental factors and sensor limitations can result in non-stationary and irregularly sampled data, posing significant challenges. To address these issues, we propose STaRFormer, a Transformer-based approach that can serve as a universal framework for sequential modeling. STaRFormer utilizes a new dynamic attention-based regional masking scheme combined with a novel semi-supervised contrastive learning paradigm to enhance task-specific latent representations. Comprehensive experiments on 56 datasets varying in types (including non-stationary and irregularly sampled), tasks, domains, sequence lengths, training samples, and applications demonstrate the efficacy of STaRFormer, achieving notable improvements over state-of-the-art approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes STaRFormer, a unified framework for time series modeling that addresses challenges such as non-stationarity and irregular sampling. It introduces a novel Dynamic Attention-based Regional Masking (DAReM) module that adaptively masks task-relevant regions based on attention scores. The framework employs a semi-supervised contrastive learning approach, combining self-supervised batch-wise contrastive loss with supervised class-wise loss to improve representation learning for downstream tasks. STaRFormer is evaluated on multiple benchmark datasets across different tasks, demonstrating state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework composes formulations for both sequential-level and element-level supervision, enabling adaptability to diverse downstream tasks such as classification and anomaly detection.\n\n 2. The Dynamic Attention-based Regional Masking (DAReM) module introduces a novel masking strategy that leverages attention scores to identify and focus on task-relevant regions.\n\n3. The paper presents extensive experiments and ablation studies, and the results show improvements compared to previous methods.\n\nWeaknesses:\n1.  It is unclear whether the trained STaRFormer model can generalize across multiple downstream tasks or if it must be trained separately from scratch for each task, limiting its flexibility and generability compared to task-agnostic frameworks.\n\n2. The proposed DAReM masking method is not compared against other baseline masking strategies (e.g., random, uniform, or fixed-region masking), making it difficult to isolate and verify its effectiveness.\n\n3. Figure 1(b) is not well explained in the main paper and is difficult to interpret without first reading Supplementary Section B.3.\n\n 4. Important figures such as Fig. 8 (which is referred to in the main text) are only included in the supplementary material, reducing accessibility to explanations within the main paper.\n\nQUESTIONS:\nAs weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a Transformer-based framework called STaRFormer for sequential data modeling. They state that traditional LSTM and Transformer models do not perform well on non-stationary and irregularly sampled time series. And to tackle these challenges, they introduce a novel semi-supervised, task-informed representation learning technique using a dual-encoder (towers) architecture: (1) In one tower, the model utilize masked token modeling during training for representation learning to improve the robutness of the model in handling potential non-stationary data during application, and (2) in another tower, the model perform supervised downstream task prediction. A contrastive loss is then used maximizes the agreement between the latent representations from both towers, improving the model's robustness and generalization. Experiments demonstrate the effectiveness of the model.\n\nSTRENGTHS AND WEAKNESSES:\n**Disclaimer:** I am not an expert in this area, but will do my best to provide helpful evaluation:\n\n## Strengths\n\n1. The paper addresses the important and practical challenge of modeling sequential data that is non-stationary and irregularly sampled. This sounds to me a common issue in many real-world applications, such as digital car keys prediction that the authors used to motivate this work, making the problem highly relevant.\n\n2. The proposed model, STaRFormer, is shown to be highly effective through extensive experiments. The improvement is not aways significant, but they are quite consistent across tasks.\n\n3. The claims are well-supported by a comprehensive set of ablation studies.\n\n## Weaknesses\n\nI mainly have some concerns on the motivation on some design choices of the model:\n\n1. The paper states that LSTMs and Transformers \"typically assume that the data is fully observed, stationary, and sampled at regular intervals.\" This might be a reasonable assertion for LSTMs, which inherently process inputs in a sequential order. However, it is not clear to me why this will apply to Transformers as well. For instance, with architectural variants like relative positional embeddings, Transformers can be adapted to handle signals sampled at dynamic intervals. It would be nice to elaborate on why standard Transformers are still considered ill-suited for such data.\n\n2. The proposed method dynamically masks regions that are deemed important based on attention scores, which is different from the random masking used in many self-supervised methods. My understanding is that the purpose of masking, as seen in paradigms like Masked Image Modeling (MIM), is to incentivize representation learning by forcing the model to learn the correpondance between regions. From this perspective, masking the *most important* regions seems counter-productive, as it removes the most informative signals for reconstruction. Could the authors provide more intuition on why this strategy is more effective than random masking?\n\nQUESTIONS:\nI have attached my questions in the Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes a time-series classification model combining multiple task objectives during training to learn better representations that support better classification performance. Among these multiple tasks are: (1) the classification loss, (2) the contrastive loss on instances produced via (a) task-aware masking and (b) class labels. Their results and ablations show a better performance over baselines.

STRENGTHS AND WEAKNESSES:
## Pros

1. The idea of combining masking and contrastive loss in a semi-supervised setup on time-series data seems novel.
2. The experiments show good performance over the baselines.
3. Ablation shows benefits of DAReM over random masking.
4. Visualization shows a good separation of clusters.

## Cons

1.  I feel that there needs to be a better utilization of the space in the main paper. 
    1. The paper shares too few details about DAReM in the main paper. For instance, it is not clear what φ, γ, ζ mean in more exact terms.
    2. It may be a good idea to perhaps offload less essential related work (which takes a whole page) to the appendix.
    3. The font size of the tables is too small.
2. In the ablation table (Table 6), the variations in the scores are too tiny. I wonder if this is expected or informative.
3. Mainly, I don’t see a connection between their initial motivation of tackling irregularly sampled time series and the main novelty of their method (DAReM + CL). 
4. Also, I don’t see why irregular sampling would be a concern for, say, TARNet or SLOTS, since one can simply construct a positional embedding using the actual time-stamp.

QUESTIONS:
See Cons.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes STaRFormer, a Transformer-based framework designed to handle irregularly sampled and non-stationary sequential data effectively. The key contributions include a dynamic attention-based regional masking (DAReM) mechanism and a semi-supervised contrastive learning (CL) method, explicitly coupling representation learning with downstream tasks. Experiments conducted across 19 diverse datasets, including irregularly sampled and real-world non-stationary datasets, demonstrate that STaRFormer achieves competitive  performance relative to current state-of-the-art (SOTA) methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Novel Dynamic Masking Approach (DAReM): The proposed attention-based regional masking strategy is innovative and addresses the critical challenge of handling irregular sampling and non-stationarity effectively by adaptively perturbing key regions of the sequence based on learned attention scores.\n\n2. Effective Semi-Supervised Contrastive Learning: The proposed contrastive learning paradigm combines supervised (class-wise) and self-supervised (batch-wise) signals, explicitly linking representation learning with downstream tasks, contributing to robust representations and improved performance.\n\n3. Comprehensive Experimental Validation: The extensive evaluation across multiple datasets (regular, irregularly sampled, and non-stationary time series) provides strong empirical support for STaRFormer's versatility and effectiveness in diverse real-world scenarios.\n\n4. Thorough Ablation Studies: The paper provides extensive ablations and clearly demonstrates the incremental contributions of each proposed component (DAReM, semi-supervised CL), supporting the methodological design choices.\n\nWeaknesses\n\n1. Limited Computational Efficiency: The proposed dynamic masking strategy, relying on full attention mechanisms, inherently incurs computational complexity of 𝑂(𝑁^2), potentially hindering scalability for very long sequences, as acknowledged by the authors. Although this complexity affects only training, it can limit practical usability on larger datasets.\n\n2. Marginal Improvements on Certain Datasets: Despite consistent outperformance across most datasets, performance gains are modest in some cases, which may raise concerns regarding whether the additional complexity introduced by DAReM and contrastive learning fully justifies the marginal performance gains.\n\nQUESTIONS:\nIs there a reason for putting the supplementary material in the main paper? I think the authors didn't follow the submission form properly.\n\nLIMITATIONS:\nThe manuscript adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}