Title: TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: time series analysis

multimodal retrieval
Abstract: The ubiquity of dynamic data in domains such as weather, healthcare, and energy underscores a growing need for effective interpretation and retrieval of time-series data. These data are inherently tied to domain-specific contexts, such as clinical notes or weather narratives, making cross-modal retrieval essential not only for downstream tasks but also for developing robust time-series foundation models by retrieval-augmented generation (RAG). Despite the increasing demand, time-series retrieval remains largely underexplored. Existing methods often lack semantic grounding, struggle to align heterogeneous modalities, and have limited capacity for handling multi-channel signals. To address this gap, we propose TRACE, a generic multimodal retriever that grounds time-series embeddings in aligned textual context. TRACE enables fine-grained channel-level alignment and employs hard negative mining to facilitate semantically meaningful retrieval. It supports flexible cross-modal retrieval modes, including Text-to-Timeseries and Timeseries-to-Text, effectively linking linguistic descriptions with complex temporal patterns. By retrieving semantically relevant pairs, TRACE enriches downstream models with informative context, leading to improved predictive accuracy and interpretability. Beyond a static retrieval engine, TRACE also serves as a powerful standalone encoder, with lightweight task-specific tuning that refines context-aware representations while maintaining strong cross-modal alignment. These representations achieve state-of-the-art performance on downstream forecasting and classification tasks. Extensive experiments across multiple domains highlight its dual utility, as both an effective encoder for downstream applications and a general-purpose retriever to enhance time-series models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a multimodal framework that aligns time series data with corresponding textual context for cross-modal retrieval, forecasting, and classification. It designs tokens during the encoder pretraining stage amortized for downstream tasks. It also introduces a dual-level alignment strategy using channel identity tokens and contrastive learning at both the sample and channel levels. As such, it supports Text-to-TS and TS-to-Text cross-modal retrieval and also acts as a plug-and-play module for retrieval-augmented forecasting and classification, where the efficacy is validated in the experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed framework formulates and tackles cross-modal retrieval between time series and texts, which enables multiple downstream tasks. The experiments with ablations demonstrate its efficacy. \n\n2. The idea of dual-level contrastive learning for cross-modal alignment, including the design of hard negative mining is well-motivated and somewhat novel, in terms of enabling fine-grained semantic retrieval between time series and texts.\n\n\n\nWeaknesses:\n\n\n1. The availability of aligned time series and textual data (including not only global descriptions but also channel-specific descriptions) is assumed, which limits its deployment in real-world scenarios such as finance where texts and the stock prices of finance entities are not highly correlated, and industrial sensors .\n\n2. This paper evaluates its method and baselines on multiple multi-modal time series datasets. However, it lacks discussions and baseline comparisons for a few methods with multi-modal modeling in forecasting and classification subtasks. For example, the TimeCMA [1] method also proposes channel-wise similarity retrieval in cross-modality alignment that facilitates forecasting performance. \n\n3. The design of channel-biased attention and RoPE for channel dependencies encoding is not easy to read, and it is somewhat not well-motivated in terms of multi-modal modeling.  \n\n[1] TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment\n\nQUESTIONS:\n1) Please refer to weaknesses 1 and 2:\n\n1-1) The assumption regarding highly aligned time series - text pairs, is a bit too strong in real-world time series contexts. Do the authors consider any countermeasures to mitigate this issue?\n\n1-2) Could the author provide more evaluations regarding the aforementioned baselines that shares similar idea in terms of cross-modal retrieval? \n\n2) For the evaluation in Table 1, the encoder of TRACE is replaced by other foundation models with frozen parameters, demonstrating significant performance gain. Did the authors consider other moderated time series encoder, such as PatchTST, iTransformer during the pre-training stage?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a new multimodal retrieval system **TRACE**, which implements `Channel Identity Tokens (CIT)` within a masked autoencoder framework to capture multivariate time series patterns while maintaining channel disentanglement and also implementing hierarchical hard negative mining at both channel and sample levels to employ cross-modal alignment. This work addressed the critical gap between time series analysis and their semantic grounding. The key technical contributions of the paper are:\n1. The framework introduces a comprehensive system for cross-modal retrieval between time series and text. \n2. The framework introduces RAG capabilities for time series foundation models through soft token interfaces\n\nThe paper provides a comprehensive evaluation across customer weather datasets, TimeMMD benchmark subsets, multiple baseline comparisons against both traditional time series models and recent foundational models. The inclusion of extensive ablation studies that validate architectural choices and hyperparameter sensitivity instills confidence in the system's design and performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. TRACE can be used for downstream tasks (forecasting and classification) in both scenarios - external data is available (RAG mode), or no external database exists (standalone decoder mode). This dual functionality implies that the learned representations are robust and can be helpful in multiple deployment scenarios. \n2. TRACE allows frozen base models to incorporate retrieved context without architectural modifications. \n3. The learned multimodal embedding space is already aligned with textual descriptions reducing the dependency on additional textual signals. \n4. TRACE demonstrates superior performance across multiple evaluation dimensions: Cross-modal retrieval, RAG enhancement, Standalone\n\nQUESTIONS:\n1. What is the optimal ratio between the flattened sequence length `L` and the original temporal length `T` for different types of time series data? How does this ratio affect model performance, and are there recommended guidelines for practitioners? Do you observe performance degradation when `L >>T` (over-segmentation) or when `L <<T` (under-segmentation)?\n\n\n2. In `Figure 4`, Why are Relative PE results missing at `d=768`?\\\n       a. Were the results excluded due to poor performance?\\\n       b. Were there implementation difficulties? \n\n3. How is `CLS` token initialized and learned to aggregate global information?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TRACE, a multimodal retrieving framework designed for aligning and embedding time-series data with textual context. Recognizing that time-series data in real-world scenarios (e.g. healthcare, weather, and energy) frequently accompanies domain-specific textual information, TRACE aims to bridge this multimodal gap through semantically grounded embeddings. TRACE uses a two-stage training process: first pre-training a time-series encoder via masked reconstruction with specialized Channel Identity Tokens (CITs) and Channel-biased Attention, followed by fine-grained cross-modal alignment using dual-level hard negative mining. The model demonstrates two functions. One, as a standalone encoder offering state-of-the-art performance for forecasting and classification, and two, as a general-purpose retriever enabling retrieval-augmented generation (RAG) for enhancing downstream models. Extensive experiments confirm TRACE’s effectiveness across multiple retrieval and forecasting scenarios, surpassing several state-of-the-art methods.\n\nContributions:\n* Novel Multimodal Retriever between TS-Text: TRACE is a novel multimodal retriever explicitly designed to ground time-series embeddings with fine-grained textual context at both channel and sample levels.\n\n* Superior Performance: Extensive empirical validation demonstrates that TRACE consistently achieves state-of-the-art results. It outperforms existing SOTA models in cross-modal retrieval tasks (both text-to-time-series and vice versa). In retrieval-augmented generation setups, it boosts downstream model performance.\n\n* Hierarchical Hard Negative Mining: The paper introduces a unique hierarchical mining strategy that effectively discriminates between semantically similar but contextually divergent negatives, enhancing the semantic precision of embeddings.\n\n* Versatile Integration: TRACE’s architecture enables straightforward and efficient integration with existing time-series foundation models through retrieval-augmented generation, requiring minimal additional fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* Clear Problem Definition and Proposed Solution: Paper identifies the gap that time-series foundation models are usually unimodal and NLP‑style RAG cannot ingest continuous signals, then supplies a two‑stage pipeline and an additional RAG stage that closes it.\n\n* Architectural & Training Novelty: Channel‑identity tokens + channel‑biased attention give each variable a fixed semantic slot. Proposed dual‑granularity InfoNCE aligns both pairs of CLS and report text, CIT and channel text. Soft‑prompt RAG injects retrieved multimodal context into frozen decoders with a small number of extra parameters.\n\n* Broad empirical coverage. Tests three tasks (retrieval, forecasting, classification) on four different domains and compares against both scratch networks and six public foundation models (Chronos, Timer‑XL, Time‑MoE, Moment, etc.).\n\n* Achieves substantially superior empirical performance compared to prior state-of-the-art: Retrieval precision significantly improved from best prior 55.73% to 90.08% P@1 for TS-to-Text and 64.67% to 89.63% for Text-to-TS. It has the smallest forecasting error (MAE and MSE) across datasets from different domains most of the time. By fine-tuning a pretrained model, TRACE makes classification accuracy raised notably by more than 10%, surpassing other prior methods.\n\nWeaknesses:\n* Synthetic or Overly Simple Channel Texts: The channel-level textual descriptions are either artificially generated or probably extremely short (sometimes just sensor IDs), reducing external validity. Real-world scenarios involving richer, natural-language texts might challenge the proposed dual-level alignment method.\n\n* Lack of Purpose-built Cross-modal Baselines: Some comparisons in the experiments are restricted to adapted models (e.g., Timer-XL, Moment) not originally designed for cross-modal alignment. Missing are purpose-built baselines (e.g., TimeRAF), potentially inflating reported retrieval performance gains.\n\n* Quadratic Complexity of Hard-negative Mining: The hierarchical negative mining strategy seems to scale quadratically with batch size (O(B²)), potentially becoming a computational bottleneck for large-batch training scenarios. No acceleration methods or approximations are discussed.\n\n* No Support for Irregular or Sparse TS Data: The Rotary Positional Encoding (RoPE) approach implicitly assumes regular global time steps, and there is no experimental validation provided for event-driven or sparsely sampled time series. It may limit TRACE's applicability.\n\nQUESTIONS:\nQ1: Channel‑Identity Tokens scale linearly with the number of variables. How does training/inference memory behave if C >> 1000 (e.g. high‑frequency trading data)? Is it possible that CITs be compressed (hashing, shared prototypes) without losing retrieval fidelity?\n\nQ2: Please correct me if I am wrong. Rotary PE assumes a single global sampling interval. How would the encoder handle irregularly‑sampled or event‑driven series. Can RoPE be replaced by other position embedding techniques to address this issue?\n\nQ3: In Stage 1, the masked‑reconstruction loss predicts raw values (MSE). Would frequency‑domain or season‑trend decomposition losses produce richer features and better alignment downstream?\n\nQ4: Channel‑biased attention forbids CITs from attending to other channels. Have you tried soft gating instead of a hard mask so causal inter‑dependencies aren’t completely blocked?\n\nQ5: Instead of averaging retrieved CLS/text vectors, have you explored attention‑based fusion or set transformers so the model can weigh items by relevance?\n\nLIMITATIONS:\nThe authors have partially addressed the limitations and potential negative societal impact of their work.\n\nLimitations\n* Quantitative Robustness to Noisy Texts: Missing explicit quantitative analysis demonstrating how performance degrades when textual labels are noisy or auto-generated.\n* Recommendation: Provide clear experiments quantifying robustness or degradation under noisy or missing text conditions in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generic multimodal retriever, TRACE, that grounds time series in aligned textual descriptions. It supports text-to-ts and ts-to-text retrieval modes. By retrieving relavant pairs, TRACE improves predictive accuracy in downstream applications. Besides, it can also serve as a encoder of multimodal time series data. Experiments across several datasets demonstrate the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea of retrieving across time series and textual data is compelling.\n2. To enhance the alignment of time series and textual data, it introduces channel-level aligment and hard negative mining strategies.\n3. To evaluate the performance of TRACE, the paper curated a new dataset, with multivariate time series.\n4. The experiments are comprehensive and the results demonstrate the effecitiveness of TRACE.\n\nWeaknesses:\n\n1. The proposed method requires curated multimodal datasets, which are not easy to achieve in time series domain.\n2. Although the paper have taken much efforts to curate Weather dataset, it seems no metrics to demonstrate the quality of the dataset intuitively.\n3. More baselines should be included in Table 4, especially multimodal time series models, e.g., TimeCAP [1], Multimodal PatchTST, Multimodal iTransformer from [2]\n\n    [1] TimeCAP: Learning to contextualize, augment, and predict time series events with large language model agents.\n\n    [2] Time-mmd: A new multi-domain multimodal dataset for time series analysis.\n\nQUESTIONS:\n1. It says that the study uses ChatGPT to generate channel-level textual descriptions for selected TS samples. Have you evaluated the overall quality of the descriptions? How do you ensure the quality of the generated descriptions?\n2. How to obtain the soft token of a time series snippet when TRACE is used as a general-purpose retriever? Does the soft token affect the performance of TRACE?\n3. I also wonder how information leakage is avoided during experiments or dataset construction. For instance, textual descriptions of events may include the ground truth of the forecasting target. Does TRACE implement any mechanisms to prevent such leakage?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}