TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval
Authors: Jialin Chen
Ziyu Zhao
Gaukhar Nurbek
Aosong Feng
Ali Maatouk
Leandros Tassiulas
Yifeng Gao
Rex Ying
Venue: NeurIPS 2025 poster ()
Keywords: time series analysis
multimodal retrieval
Primary area: deep_learning

Abstract: The ubiquity of dynamic data in domains such as weather, healthcare, and energy underscores a growing need for effective interpretation and retrieval of time-series data. These data are inherently tied to domain-specific contexts, such as clinical notes or weather narratives, making cross-modal retrieval essential not only for downstream tasks but also for developing robust time-series foundation models by retrieval-augmented generation (RAG). Despite the increasing demand, time-series retrieval remains largely underexplored. Existing methods often lack semantic grounding, struggle to align heterogeneous modalities, and have limited capacity for handling multi-channel signals. To address this gap, we propose TRACE, a generic multimodal retriever that grounds time-series embeddings in aligned textual context. TRACE enables fine-grained channel-level alignment and employs hard negative mining to facilitate semantically meaningful retrieval. It supports flexible cross-modal retrieval modes, including Text-to-Timeseries and Timeseries-to-Text, effectively linking linguistic descriptions with complex temporal patterns. By retrieving semantically relevant pairs, TRACE enriches downstream models with informative context, leading to improved predictive accuracy and interpretability. Beyond a static retrieval engine, TRACE also serves as a powerful standalone encoder, with lightweight task-specific tuning that refines context-aware representations while maintaining strong cross-modal alignment. These representations achieve state-of-the-art performance on downstream forecasting and classification tasks. Extensive experiments across multiple domains highlight its dual utility, as both an effective encoder for downstream applications and a general-purpose retriever to enhance time-series models.

TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval
Jialin Chen', ; Ziyu Zhao", Gaukhar Nurbek', Aosong Feng', Ali Maatouk' Leandros Tassiulas', Yifeng Gao*, Rex Ying' Yale University, MccGill University, UUniversity of Texas Rio Grande Valley
(jialin. chen, aosong feng, maatouk, leandros. tassiulas, rex. vex.ying)@yale.edd ziyu. wrw.zhhao@@mil. mmcgiml ca; {gaukhar nurbbkk11 yifeng gut)
Abstract
The ubiquity of dynamic data in domains such as weather, healthcare, and energy underscores growing need for effective interpretation and retrieval of time-series data. These data are inherently tied to domain-speciinn contexts, such as clinical notes weather narratives, making moos- retrieval essential not only for downstream tasks but also for developing robust time-series foundation models by retrieval-augmented generation (RAG). Despite the increasing demand, time-series retrieval remains largely underexplored. Existing methods often lack semantic grounding. struggle align heterogeneous modalities, and have limited capacity for handling multi-channel signals. To address this gap, we propose TRACE, a generic multimodal retriever that grounds time-series embeddings in aligned tex- tual context. TRACE enables fine-grained channel-level alignment and employs hard negative mining facilitate semantically meaningful retrieval. It supports flexible cross-modal retrieval modes, including Text-to- -ii-----ri and Timeseries-to Text, effectively linking linguistic descriptions with complex temporal patterns. By retrieving semantically relevant pairs, TRACE enriches downstream models with informative context, leading to improved predictive accuracy and interpretabil- ity. Beyond a static retrieval engine, TRACE also serves as powerful standalone encoder, with lightweight task-specific tuning that refines context-aware represen- tations while maintaining strong cross-modal alignment. These representations achieve state-of-the-art performance on downstream forecasting and classification tasks. Extensive experiments across multiple domains highlight its dual utility, as both an effective encoder for downstream applications and a general- -prrpse retriever to enhance time-series models ?
Introduction
Time-series data is prevalent across critical domains such as healthcare, weather, and energy 2, 3,4 Crucially, such data rarely exists in isolation in real-world applications. is typically accompanied by rich, domain-specific textual context, egg. clinical notes and weather reports [5, 6, 7J. This inherent multimodality necessitates shift beyond unimodal time-series analysis towards multi-modal frameworks that seamlessly integrate these heterogeneous data types.
Cross-modal modal retrieval between time series and text is not only natural but necessary. As shown in Figure 1, given a flash flood report describing extreme rainfall and high wind gusts, retrieving historical time series that exhibit similar patterns support downstream tasks such weather forecasting and disaster warning. Such retrieval also enables the integration of semantically aligned
Cross
Botth authors contributed equally this paper. Codes are available at http://giitub..oo//rpaaaaatt
Cow.d--ommmii--LLannnnnRAACEEEE MA TSEncoder.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Flash Flood Event Report:
A flash floo ocoured Extremery moist air and weak shortwave trough riagered persistent heavy showers, Rainfall totals reached 6-10- inchos, with recont- brankiiiniinng 18-1 inches. The fooding cuused videsproed demage before subsiding aoonn midday.
Channel-evel Description:
temperatura ranged
Br..... were sporadio instances preciptiation with significatt peak
Redative humidity fuchuseed between 0S 100 0%. higher valurs Vicblity remained relationly high, mostly around kmm with ozzssionoll
drope.... Wnd diretionn showed variebilly... douds....
velocity varidd, with notable gusts neaching up mis... The sky osver rangee from clear acathered
What weather conditions are likely to follow this flash flood event?
Top-1 Retrieved Time Series (Label: Flash Flood)
Cross-modal Retriever
temperature
precipitation
relative humidity
visibiity
wind
wind
sky code
00
Figure I: A Use Case of -ii------ieeeeiess Retrieval
external knowledge into time series foundation models [8, 9, 10), guiding model attention to relevant segments, and facilitating more generalizable inference via retrieval- -atiiveell-umeeeeed generation (RAG).
Despite the clear demand, time-series retrieval, particularly in a cross-modal context, remains significantly underexplored. Existing approaches often fall short several ways [11, 12, 13, 14, 15]. They overlook the rich textual context within time-series data and rely on shallow similarity measures rather than contextual understanding, leading to a lack of effective cross-modal alignment between time-series signals and their associated textual descriptions. Moreover, they struggle with the multi- channel nature of real-world time series, where each channel can encode distinct yet interrelated information [16, 17, 18]. Importantly, prior work rarely explores retrieval-augmented generation (RAG) for time series foundation models, restricting their utility augmenting downstream models.
To address this gap, we introduce TRACE, a novel multimodal Time-series Retriever with Aligned Context Embedding. As illustrated in Figure 2, TRACE adopts a two-stage training: a pre-training stage for the time-series encoder, followed by cross-modal aligmment. To address the challenge of modeling multivariate time series, we introduce Channel Identity Tokens (CTTs) into a masked autoencoder framework pre-trained at both the token level and channel level in Stage 1. CITs guide the model to attend to unique channel behaviors and enable the learning of channel disentangled representations, overcoming the limitation of conventional decoder-only foundation models which often yield embeddings lacking discriminative power for retrieval and classification. In Stage 2, we propose novel component for effective cross-modal alignment between time-series embeddings and their textual counterparts through a hierarchical hard negative mining strategy. At the channel level, we identify distractor single-channe segments that exhibit misleadingly similar patterns. At the sample level, we dynamically mine hard negatives by selecting highly similar text descriptions but with divergent semantics. This dual- contrastive learning encourages the model to learn both local precision and global consistency, leading to strong generalization downstream tasks.
Time-series Foundation Model
TRACE: Multimodal Time-series Retriever
d Frozen
Stage I: Pre-training
Stage 2: Alignment
4 Training
Soft token
forecasting classification prediction
token-level
channel-level
sample-evel
MMTS
Database
CIT
CIT
CIT
A. Generalpurpose
B. Standafone Encoder
Retriever
MMTS
Temperature ranges...
Prediitation is
Humidity Bluduottes
Dataset
TRACE
(1) TRACE Two-stage Training
(2) TRACE Two-fold Utility
Figure 2: Overview of TRACE, CIT stands for Channel Identity Tokens, which serve as a key bridge to connect two stages. MMTS denotes multimodal time series.
TRACE is designed with two-fold utility. It acts as a general-purpor retriever, which provides relevant information via soft token interface. The soft token summarizes retrieved time-series snippets into a latent vector, which is then prepended as conditioning token, guiding frozen time-series foundation model towards more context-awand aware predictions. Moreover, TRACE serves as a powerful standalone encoder, producing rich embeddings that achieve state-of saae-oof-the-ar the- performance on downstream forecasting and classification tasks. Extensive experiments on both public benchmarks and our curated multimodal dataset validate the effectiveness of TRACE, demonstrating superior retrieval accuracy. The retrieved context substantially boosts downstream time-series models in retrieval-augmie augmented settings, with up to 4.56% increase in classification accuracy and 4.55% reduction

---

in forecasting error. In addition, TRACE produces high-quality time-series embeddings that achieve state-of-the-arg results on wide range of forecasting and classification benchmarks.
The contributions of this paper are: (1) we propose the first multimodal retriever, TRACE, that learns semantically grounded time-series embeddings through fine-grained dual-level alignment; (2) we establish new benchmarks on cross-modal retrieval between time series and text, and (3) extensive validation showcases that TRACE consistently delivers state-of-the-art performance both as a general-purpose retriever for time-series models and powerful encoder for time series analysis.
2 Related Work
Time Series Forecasting. Recent work on time-series forecasting has led to a range of model architectures, each emphasizing different inductive biases. Transformer-based models leverage self- attention to capture long-range dependencies and flexible temporal dynamics [19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. Lincar-based models assume time-series signals can be effectively decomposed and modeled with simple linear projections [29, 30]. brequency-domain and mixing- based approaches aim to model periodicity and multi-scale temporal structures using Fourier transforms or token mixers [31]. Recently, variety of time series foundation models have emerged. Timer-XL, [9] leverages Kronecker attention and is pre-trained with multivariate next-token prediction to enable unified, long-context forecasting. Chronos [32] tokenizes time series via scaling and quantization, and trains a T5-style model for zero-shot probabilistic forecasting. Time-MoF [10] introduces a sparse mixture-of-experts architecture to support variable horizons and input lengths. TimesFM [33] uses input patching and is pre-trained on large-scale data for strong zero-shot performance. Moment [8] and Moirai [34] adopt masked prediction pretraining enable generalization across diverse multivariate forecasting tasks. While these models perform well on forecasting tasks, they are generally unimodal and not designed for retrieval or integration of external context, highlighting gap addressed by cross- modal retrieval framework.
Time Series Language Models. Recently, several multimodal encoders have been proposed to integrate time series and text [35, 36, 37, 38, 39, 40], which aim to leverage the generalization capabilities of large language models by reprogramming time series into token-like representations or textual prototypes. [ha[]] models time series as a foreign language by normalizing and discretizing continuous signals into token sequences, which are then processed by a large language model (LLM). CharTS[44] supports both understanding and reasoning by fine-tuning on synthetic datasets generated via attribute- based sampling. TimeXL[433 combines a prototype- -based time series encoder with a multimodal prediction framework to capture explainable temporal patterns guided by aligned textual cues. However, they primarily treat text as global context and lack fine-grained alignment between structured time series components and textual semantics, leading to suboptimal cross-modal embedding or retrieval.
Time Series Retrieval System. Recent work has explored retrieval systems for time series data, primarily within a unimodal setting [44, 13, 45, 15]. CTSR [11] supports content- based time- retrieval using contextual metadata. TimeRAF [12] integrates trainable retriever with task-specific time-series knowledge bases for downstream augmentation. TS- RAG [14] retrieves relevant time series segments using pre-trained encoders and combines them via mixture-of-experts module to improve forecasting. However, all of these methods rely solely on time series embeddings and do not incorporate textual signals, limiting their ability to support multimodal and context-aware retrieval.
3 Proposed Method
As shown in Figure 3. TRACE learns robust time series representations through a masked reconstruction objective with channel-biased attention in the pre-training stage (Sec. 3.2). Then, each time series channel is aligned with its corresponding textual description via fine- -ee-ggraind contrastive learning in the cross-modal alignment stage (Sec. 3.3). We further propose a novel strieval-auugented generation strategy for time series foundation models, where TRACE retrieves relevant context for downstream tasks (Sec. 3.4). This modular design enables both strong standalone performance and effective integration with existing time series foundation models.
3

---

Reconstruction Forecasting Classification
Mext
Head
Head
Head
CLS
Fext
Projection
Sentence Enco der
Evant Raport: A strong storm system slowly moved... The low- level wind shear had become strong the which supported the development
Linear
CIT,
front itself, of few tornadoes....
Encoder-only Transformer
73
Channel-biased Attention
Cross Attention
Temperalure ranges from a high of 21 21.42"G to low of 9 9.24%C
Pracipitation generally low, with several days no rain, but
RoPE
attoad
CITY
2
repel
Humidity fluctuates significantly.
CIS
CIT
CIT.
X3 Mask Taken
Z2
Fatch Taken
averaging around 75%...
Figure 3: Illustration of TRACE, which encodes multivariate time series using channel-biased attention and aligns token embeddings with its corresponding textual description (e.g., Z, and Zext) through cross-attention and dual-Ievel contrastive learning. Zex indicates an in-batch hard negative sample.
3.1 Problem Definition
Multimodal Time-series. Let X E RCT denote a multivariate time series instance, where C' is the number of channels (or variables) and T is the number of time steps. We assume the availability of two types of textual information aligned with X. First, for each channel e in an instance X, there is corresponding textual description T. that summarizes the behavior trend of X, over the time window [0, 7). These descriptions are denoted as T = {T>lc = 1, ... C). Additionally, there is a sample-level context Tex summarizing the overall condition occurring during the same time window, which could be weather reports or clinical narratives, depending on the application domain.
Task Objectives. The goal is to jointly embed the multivariate time series X and its corresponding textual context T = Thh U [Feu into a shared space that supports multiple downstream tasks, including: (1) forecasting future values XT.T+ H RC*H for the next H time steps; (2) classification, where the model predicts a categorical label for each time series instance; and (3) cross-modal retrieval, where the goal to retrieve relevant time series X based on a text query Tex or retrieve historical relevant reports from T given a time series query, etc.
3.2 Stage 1: Time Series Encoder Pre-training
Time Series Tokenization. Given an input multivariate time series X E RCXT, we divide the temporal dimension into non-overlapping (or strided) patches of length P, resulting in f = patches per channel. Each patch is flattened and linearly projected into d-dimensional embedding space using a learnable linear projection. This converts each channel into a sequence of patch tokens XFalch R for Ve E {1....,C) C). To capture localized semantics within each channel, we prepend a learnable channel identity token [CIT] E IR Ixd to the patch token sequence of each channel. These tokens serve as explicit representations of channel-level summaries. Each token is uniquely indexed and not shared across channels, initialized from a standard Gaussian distribution, and trained jointly with the model. This design allows the model to differentiate between channels and effectively aggregate channel-wise patterns. We then concatenate all tokenized channels into single sequence and insert a global learnable [CLS] token at the beginning of the full sequence. The final token sequence for multivariate instance is structured as:
H [CLS]; [CIT] ( Xpatch [CIT] x xpptch,...... ....; [CIT] o5 pracch)
E Rxd
(1)
where L C(f + 1) is the total sequence length after flattening all channel in 1. This tokenization strategy preserves both temporal and structural granularity: patchification encodes token-level patterns; [CIT] summarizes intra-channel dynamics; and [CLS] provides a global and sample-level embedding that can be used for downstream retrieval and classification tasks.
Channel-biased Attention and Rotary PE. To encode channel dependencies in multivariate time series, we introduce a novel Channel-biased Attention (CbA) mechanism that incorporates both induc- tive bias for channel disentanglement and temporal order encoding via rotary positional embeddings (RoPE) [46]. In our CbA, we design a biased attention mask M E {0, 1 to prevent unintended semantic entanglement across heterogeneous variables. Specifically, for each channel identity token [CIT], located at index i, in the flattened sequence, we define Mic.i 0 if token j & channel
and 1 otherwise, and Maj if token k is not [CIT] Let Q K. V E plxd be the learned

---

linear projections of the input token embedding H. We apply RoPE to the query (Q) and key (K) vectors before computing attention. RoPE is applied independently within each channel to the T temporal tokens, and is not applied to the channel identity tokens, which act as position-agnostic aggregators. The attention weight between tokens i and in RoPE-enhanced attention is given by Oij = softmax, (Q, Roar. Kj/Vā+ + log Mts). where Roasy (.) denotes a rotation by angle A and Atu is the relative time difference between tokens i and j in their original unflattened sequence. This is crucial in the multichannel setting, as two tokens that are close in actual time may appear far apart in the flattened sequence. Using Alij ensures that the position encoding remains consistent with the true temporal structure rather than the flattened channel order. Mij mask enforces channel disentanglement, while still allowing rich token-level interactions across the full sequence.
Pre-training Setup. We adopt encoder-only Transformer [47] with multi- head Mtannel-based attention layers in TRACE. We apply reversible instance normalization [48] to multivariate time series before tokenizing and embedding. A fixed proportion of these tokens is randomly masked with mask ratio of y, and the model is pre-trained to reconstruct the missing values based on the unmasked context. We use mean squared error (MSE) loss to supervise pre-training, encouraging the model to capture cross-channel dependencies while learning transferable representations for downstream tasks.
3.3 Stage 2: Multimodal Alignment Learning
Motivation. Standard contrastive learning methods typically rely on sample-level random negatives. However, textual descriptions frequently reference specific variables (e.g., temperature spikes, wind gusts), which cannot be precisely aligned using a single global embedding. To address this, we introduce channel-level alignment that explicitly models the interaction between individual time- series channels and their corresponding textual context. This not only enhances semantic precision but also promotes modularity in representation learning and enables variable-spp speebifc interactions.
Cross-attention Between Modalities. After pre-training the time-series encoder via masked recon- struction, we obtain hidden embedding H RLxd from the final transformer layer, where I. is the full sequence length after flattening all channels. From this, we extract the [CLS] token embedding hcaus] Rdd, and the set of channel identity token embeddings H tcm [h, [h...... hc] Cxdd each corresponding to [CIT] token and serving as fine-grained anchors that enable structured reasoning the channel level. Let Text and T. denote the sample-level and the e-th channel textual context for time series instance, respectively. The textual inputs are first encoded using a pre- trained language Transformer [49]), followed by a learnable linear layer that projects them into the same d-dimensional embedding space as the time series representations, collectively denoted as fi( This yields semantic embeddings Zext = fi(Text) E RRd for the sample-level context and ze = f((T.) E Rdd for each channel-level description. We further apply a cross cross-attention between H(em R Cxd and channel text embeddings Ze [zz..... zc] E Cxd, allowing information to be fused across aligned channels. This interaction allows the model to refine its channel-wise time-series representations using semantically aligned textual information.
model (e.g.. frozen Gentence-Transformmer
Dual-level Hard Negative Mining. To enhance the discriminative capacity of the model, we develop dual-level hard negative mining strategy that introduces fine-grained contrastive pressure at both the sample and channel levels. This approach enables the model to distinguish not only between unrelated time series and text, but also between subtly confusable pairs that share superficial temporal similarity but diverge semantically. For each time series instance i, we mine negative candidates from all other sample-level reports in the same batch based on embedding cosine similarity. For a certain channel, we mine channel-level negatives from broader candidate pool that includes both intra-instance distractors (other channels within the same sample) and inter-instance distractors (same-indexed channels across different samples). Specifically, for the e-th channel of the i-th instance, we define the sample-level and channel-level negative candidate set as s((() ext = Topx (iim(h (4)
[CL.] : z ( l + i) _N/hie) = Topx (sim ( 200 d' + c j + #}
where K is number of negative samples at each level. Symmetric negative sets are defined in the
reverse direction for 2 and (2) by swapping the roles of time series and text. We then compute (lext-ats Cts-atext and similarly for channel-level losses. bidirectional InfoNCE loss at sample levels: global Lglobal
The total alignment objective is the average of both directions (Formulations detailed in Appendix C):
5

---

is + L aalign (Eglobal
global
Xch
2 chamnel
ts +
channel
lext
where Adh controls the contribution of channel-level alignment. The entire alignment objective is optimized jointly with the trainable parameters of the time series encoder in the pre-training stage and the linear projection head in fi, while keeping the backbone language model frozen.
3.4 Retrieval-augmenter Generation with Time Series Foundation Models
As shown in Figure 2. TRACE enables retrieval-augmentte generation (RAG) for time series foundation models, inspired by the success of RAG in NLP [50, 13J. Given query time series, TRACE computes its [CLS] token embedding and retrieves the top-f most relevant multimodal pairs (X', réx) from the built multimodal database based on the embedding similarity, where X is historical multivariate time series and Fext is the associated sample-level context. Specifically, the time series component is encoded h Rdd and the textual context 76 is encoded to Zext TRd (as in Sec. 3.3). These representations are concatenated, stacked, and mapped through a single trainable projection layer to generate final, dense soft token P. which serves as a continuous prompt that is prepended to the query sequence input. This design allows the downstream forecaster to incorporate external knowledge without architectural modification. Importantly, the base time-series foundation model remains frozen during training; only the projection layer and a lightweight task- speccsic head are updated. This approach ensures efficiency and model-agnosticism. enabling plug-an ahdu-and-pa -lay integration across diverse backbone architectures. In effect, TRACE acts as a structured, external memory, enriching the model's input with historically grounded and semantically aligned context.
4 Experiments
We evaluate TRACE from three key perspectives: (1) its effectiveness in cross-modal retrieval (Sec. 4.2) and time-sereis retrieval (Sec. 4.3) compared to strong baselines, (2) its utility as a retriever in retrieval- standalone encoder for forecasting and classification (Sec. 4.5). Experiments are conducted on public benchmarks and curated multimodal dataset designed to assess cross-modal alignment and retrieval performance.
augmented forecasting pipelines (Sec 4.4), and (3) its generalization ability
4.1 Experimental Setting
Dataset. To support real-world multimodal time series applications, we construct new dataset in the weather domain with three aligned components: multivariate time series, sample-level event reports, and synthetic channel-level descriptions, specifically for downstream forecasting and event-type classification tasks. The event reports are sourced from the NOAA Events Database [51], while the associated time series data are retrieved from the NOAA Global Historical Climatology Network (GHCN) [52]. We focus on stations and time windows characterized by frequent severe weather events and extract historical multivariate time-series segments at multiple temporal resolutions, anchored at event onset. To enhance data diversity and model robustness, we also sample non-event (i.e., typical) periods from the same stations, as well as from geographically distinct locations. Each time-series segment includes seven variables (e.g., temperature, relative humidity, precipitation) and is annotated with either specific event type non-event label. To evaluate performance in the univariate setting, we further incorporate the three largest Heaeett-Healthh Energy, and Environment from TimeMMD [5]. multimodal benchmark designed for time series forecasting, where each single- variate instance is aligned with a sample-level textual report (e.g., clinical notes, incident logs). This setting allows
assess the model's generalization across diverse domains and varying channel configurations. Full dataset details and illustrative examples are provided in Appendix B.
Baselines. We evaluate against the state-of-ther the--rtt-rrtttt traditional time series models and recent time series foundation models. Traditional baselines include DLinear [29], iTrrnsformer [24], PatchTST [22], TimesNet [53], TimeMixer [54], and multimodal model FSCA [37]. These models are trained from scratch on each task. For foundation models, we include Chronos[32], TimesFM [33], Timer-XL [9], Time-MoE [10], Moirai [34] and Moment [8]. We refer to Appendix D.I for baseline details.
Implementation Details. The default TRACE consists of a 6-layer Transformer encoder with hidden dimension of 384 and attention heads. We use the AdamW [55] optimizer with a linear warmup followed by a cosine decay schedule. Pre-training is conducted with mask ratio of 0.3, and runs for

---

up to 400 epochs. We take 32 in-batch negative samples at each level in the alignment stage and run for up 300 epochs. All experiments are conducted over five runs with different random seeds on NVIDIA A100 40GB GPUs. We refer to Appendix D.2 for experiment configurations and details.
4.2 Cross-modal Retrieval
Alignment Setup. To evaluate the model's retrieval performance, we conduct controlled comparison by replacing the encoder in TRACE with several strong time series foundation models that produce fixed-length embeddings. Each encoder is jointly fine-tuned end-to-end with lightweight projection layer following the sentence encoder, using a contrastive learning objective. While TRACE leverages [CLS] and [CIT] embeddings for dual-level alignment, other baselines use mean pooling over the sequence due to their architectural constraints.
Evaluation Metrics. TRACE supports flexible retrieval modes, including cross moddll (Text-to-T3 to-TS and Texto-Text) and unimodal TS-to-TS retrieval. For cross-modal retrieval, a query in one modality is used retrieve its corresponding counterpart in the other modality based on embedding cosine similarity. The evaluation includes several metrics:
Label Matching uses P@k k to measure the precision of correctly labeled items among the top-k retrieval, and Mean Reciprocal Rank (MRR) to assess the rank of the first correct item.
Modality Matching evaluates whether query retrieves its paired instance from the opposite modality, using p@k for top-k precision and MRR for the rank of the true counterpart.
Text Similarity uses ROUGE between the query text and the text paired with the top-1 retrieved time series (for text-to-ts scenario), or between the top-1 retrieved and the original text paired with the query time series (for ts-to-text scenario).
Time Series Similarity computes MAE and MSE between the time series linked to the query and that of the top-1 retrieved pair, defined similarly to Text Similarity.
Results, As shown in Table 1, TRACE consistently achieves oftt--of-the-art performance in two retrieval settings with approximately 90% top-1 label matching and 44% top-1 modality matching. Notably, this retrieval precision surpasses the classification accuracy of all trai-froo-scratee models reported in Table 4, highlighting the strength of alignment supervision in learning discriminative representations. Among baselines, Moment outperforms other foundation models, suggesting that encoder-only architectures are better suited for dense retrieval tasks. In contrast, TRACE provides fine-grained embeddings for cross-modal alignment, enabling it to recover semantical counterparts with high precision.
Table 1: Retrieval results on 2,000 bidirectional Text-Timeseries query pairs. "Random" indicates a non -on-informative retriever that ranks candidates uniformly at random.
Label Matching
Modality Matching
Text ROUGE (t)
Time Series
Retriever P@1 (*)
(
MRR (t)
P@1 (1)
P@5 (?)
MRR (t)
MAE (l)
MSE (L)
Random
42.61
47.50
0.583
0.00
0.00
0.00
0.416
0.874
1,653
w/ Time-MoB
46.46
43.98
0.612
1.79
5.93
0.052
0.482
0.837
1.607
Text-to-IS TS-to-Test
Timer-
36.34
38.16
4.29
0.482
1.493
w/ TS2Vec
50.47
48.72
0.651
4.37
14.57
0.112
0.503
0.784
1.462
w/ Moment
55.73
53.18
0.691
7.78
21.68
0.154
0.515
0.747
1.415
90.08
77.60
70.24
0.717
0.403
Time- MoB
$7.08
52.22
0.656
0.75
2.89
0.460
0.857
1.578
w/ Timer-XL.
63.91
58.71
0.731
2.94
9.47
0.073
0.463
0.821
1.568
w/ TS2Vec
60.28
56.41
0.706
7.42
23.70
0.184
0.471
0.806
1.490
Moment
64.67
59.53
0.740
5.83
18.15
0.133
0.488
0.778
1.467
TRACE
89.63
78.39
0.938
43.72
69.84
0.557
0.713
0.411
0.793
4.3
Retrieval
To further assess the effectiveness of TRACE, we conduct a TS-to-TS retrieval task where each query is matched against all other time series identify the most semantically similar ones. The evaluation is performed using the label matching metrics (Sec. 4.2), including Precision 1, Precision 5,, and Mean Reciprocal Rank (MRR), alongside query time as proxy for computational efficiency.
Baseline Setup. We compare TRACE, against several representative time series retrieval methods. Euclidean Distance (ED) serves simple statistical baseline based mcan-pooled raw time
7

---

series. Dynamic Time Warping (DTW), a classic elastic matching method, evaluates similarity by aligning sequences with potential shifts. SAX-VSM [56] leverages symbolic aggregation and vector space modeling to convert time series into symbolic representations for efficient textual retrieval. CTSR [11] a learnable baseline that uses contextual metadata to enhance retrieval.
Analysis. As shown in Table 2, TRACE substantially out- performs all baselines across accuracy metries while main- laining the lowest retrieval latency. Notably, despite the design simplicity of SAX-V VSM and its moderate perfor- mance gains over raw ED, it fails to capture deep temporal or semantic patterss. CTSR, while benefiting from struc- tured cues, struggles to generalize as effectively in purely time-series scenarios. The results suggest that TRACE, when equipped with task-driven objectives and textual alignment-awas training, provides not only superior re- trieval quality but also enables scalable and efficient re- trieval pipelines. A detailed case study on TS-to-TS TS retrieval is presented in Appendix D.8, illustrating how TRACE's s structured aggregation across all channels effectively captures global semantics and highlights the most semantically dominant channels contributing to retrieval relevance.
Table 2: TS-to-TS Retrieval perfor- mance comparison. Evaluation is con- ducted over 1000 randomly sampled weather time-series queries.
Method
P@1
P@5
MRR
Time
ED
0.548
0.762
0.644
0.083
DTW
0.380
0.770
0.543
2273.93
0.551
0.769
01649
0.343
CTSR
0.682
0.893
0.802
0.057
TRACE
0.900
0.986
0.938
0.045
4.4 Retrieval-augmented Time Series Forecasting
Setup. We use TRACE retrieve the most relevant timeseries-tex, text pairs from the curated corpus based on time-series em- bedding similarity, which is then passed through trainable linear layers to produce soft prompt. For TS-only setting, the prompt derived solely from the retrieved raw time series, denoted as hu; for TS+ Text, we concatenate lus and semantic embed- ding Zext. from the retrieved text to form the prompt. This soft prompt is then prepended to the query for the downstream forecasting layer, without fine-tuning the pre-trained model weights. We refer to Appendix D.5 for implementation details. We test two architecture families: (1) decoder-only models, including Timer-XL and Time-MoE, where the prompt is prepended at every autoregressive generation step, and (2) encoder-only models, Moment and TRACE, where the prompt is prepended to the encoder's 's hidden states and followed by a trainable forecasting head. In all settings, only the linear projection layers for prompt generation and the forecasting head (for encoder-only)
Table 3: Forecasting performance on Weather dataset for next 24 steps under different retrieval-augmented generation settings.
Thmer-XI
Time-Mol
Moment
TRAGF
Setting
MSE
MAE
MSE
wlo RAG
0.729
1.088
0.635
0.03
0.645
0.816
0.576
0.718
w/ TS-only
0.720
1.009
0.621
0.801
0.628
0.797
0.556
0.698
TS=Text
0.984
0.611
0.787
0.696
trained.
Results. Table presents the forecasting results across decoder-only and encoder-only models under different RAG settings, augmented by top-/t retrieved instances. We refer to Figure 4 (d) for ablation on R. The results reveal that retrieval augmentation consistently improves forecasting performance across all models, and the TS+ Text setting leads to the most significant gains for decoder-only models like Timer-XL, and Time-MoE. Notably, TRACE shows marginal improvement when moving from TS-only to TS+ Text retrieval. which can be attributed to that its multimodal embedding space is already aligned with textual descriptions. This alignment reduces the dependency on additional textual signals and justifies TRACE 's design lightweight, general- purpose retriever for RAG pipelines. Moreover, these results indicate decoder-only models are more sensitive to the richness of retrieved modalities, whereas encoder-only models exhibit more stable and better capacity for internalizing and utilizing structured representations. While our RAG design adopts a simple embedding concatenation strategy, it primarily validates the general utility of retrieved content across different model families. We leave optimizing augmentation architectures for future work.
general-purpose
4.5 Standalone Time Series Encoder
Setup. To evaluate TRACE as a standalone encoder, we conduct experiments on forecasting and classification tasks. We compare TRACE against full-shot models all trained from scratch, and time series foundation models. All foundation models are evaluated in a zero-shot setting, except for Moment and TRACE, which are fine-tuned the forecasting head following the official protocol for
8

---

Table 5: Forecasting results (MAE and MSE) of full-shot models and time series foundation models multi-variate (M) and univariate (U) datasets. Red: the best, Blue: the 2nd best.
Weather (M)
Health (U)
Energy (U)
Environment (U)
Model
H=7
=4
H 12
=
H
=
H=48
=
MAB
MSE
MAB
MSE
MAB
MSE
MAE
MSE
MAB
MSE
MAE
MSB
MAE
MSE
MAE
MSE
Chronos
0.560
0.937
0.646
1,094
0.650
1.106
0.987
2.019
0.263
0.148
0.554
0.553
0.536
0.612
0.583
0.671
Zero-shot
Time-MolF:
0:579
0.803
0.635
0.903
0.604
0.981
0.832
1.697
0.205
0.089
0.451
0.396
0.562
0.508
0.836
0.969
TimesFM
0.550
0.859
0.640
1.034
0.610
0.913
0.865
16685
0.248
0.137
0.499
0.452
0.503
0.532
0.531
0.569
Timer XL
0.645
0.912
0.729
1.055
0.741
12255
0.988
1.892
0.236
0.118
0.460
0.424
0.549
0.564
0.565
0.574
Moirai
0.593
1.001
0.675
1.135
0.976
3.029
1.569
8.125
0.318
0.273
0.692
1.455
0.935
12.428
2.237
25,011
Moment
0.572
0.732
0.645
0.816
0.988
1.824
0.997
1.902
0.471
0.411
0.542
0.542
0.449
0.375
0.554
0.502
DLinear
0.593
0.778
0.691
0.884
1.178
2.421
1.132
2.256
0.410
0.273
0.546
0.512
0.561
0.515
0.581
0.534
irrnsformrr
0.707
0.814
0.676
0.911
0:267
0.124
0,487
0.486
0.458
Pall- shot
PatchTST
0.529
0.723
0.599
0.826
0.656
1.034
0.902
1.708
0.263
0.121
0.489
0.407
0.493
0.462
0.525
0.511
TimesNet
0.497
0.654
0.581
0.786
0.820
1.376
0.969
1,903
0.270
0.127
0.496
0.398
0.520
0,486
0.489
0.430
TimeMixer
0.667
0.787
1.091
2.215
1.126
0.376
0.246
0.538
0.568
FSCA
0.496
0.642
0.780
0.762
0.756
1.200
0.969
1.904
0.278
0.136
0.520
0.466
0.497
0.462
0.511
0.496
TRACE
0.501
0.623 0.576 0.718
0.547
0.768
0.827
1.435
0.230
0.448
0.389
0.455
0.475
0.413
forecasting. For classification, we evaluate on our curated weather dataset and fine-tune all foundation models in the same setting to ensure fair comparison (detailed in Appendix D.6).
Results. As shown in Table 5, TRACE outperforms baselines across different datasets and showcases capability on longer forecasting horizons (H), whereas the performance of baselines exhibits con- siderable variation. This observation justifies the cross-modal de- sign behind TRACE, which equips the model with stronger semantic grounding and context-aware forecasting. In the event-type clas- sification task (as shown in Table 4), we observe that fine-tuned foundation models underperform traditional siiin-fro--scattte base- lines, suggesting that their embeddings may be overgeneralized and poorly adapted to domain-specinf classification signals. In con- trast, TRACE achieves significantly higher accuracy and F1 without RAG. and benefits further from the retrieval-augne augmeeted setting. This demonstrates TRACE's ability to retain discriminative structure while maintaining broad semantic alignment, which is essential for robust downstream deployment. Full results of other foundation model variants are in Appendix D.4.
Table 4: Weather Event Clas- sification Results.
Model
Accuracy F1
Train-from-scratchh Model
DLinear
82.37
65.78
iTransformer
84.99
68.29
PatchTST
84.78
69.13
TimesNet
86.09
68.97
TimeMixer
84.78
68.66 68.65
FSCA
85.62
69.41
Finetune a Pre-trained Model
Mome-MeEamge
59.09
19.74
Momenfaase
65.43
28.29
Timer-XI.
72.38
33.45
Chronosin
74.79
40.21
TRACE w/o RAG
85.20
69.98
TRACE RAG
89.76
72.36
5 Ablation Studies
Hyper- prreee- Sensitivity. Figure 4 presents a comprehensive ablation study investigating the effects of patch length P. positional embedding (PE) types, and the number of retrieved instances R used in our RAG setup. Rotary PE consistently outperforms Relative PE by achieving lower reconstruction and forecasting MSEs as well as higher classification accuracy, particularly when using a smaller model size (d 384). Notably, increasing the model size to d 768 does not yield significant improvements, especially for downstream forecasting and classification tasks, suggesting that careful architectural design and PE choice may matter more than simply scaling parameters. Across tasks, mid-range patch lengths (e.g.. P = 6) offer the best trade-off between local and global temporal resolution. In Figure 4 (d), we observe that time series foundation models are relatively robust to the choice of R, and models augmented with aligned text generally outperform their TS-only counterparts, highlighting the benefit of cross-modal retrieval in improving forecasting performance.
0.295
0.76
Rebriie Pi, Wee
Roonsrructer
0,290
MCC
0.74
Reary X. model 184
0.72
0.265
Ganiittalion
0.70
0.280
Peatioe 4 heee Eabry Gondeeee
0.275
[a] Putch Langth
(b) Patch Langth
(c) Patch Length
Figure 4: Ablation stuiies on patch length, positional embedding, and hidden dimension for (a) Reconstruction MSE, (b) Classification Accuracy (%), and (c) Average Forecasting MSE. (d) shows ablation studies on the number of retrieved instances (R) in the RAG pipeline.

---

Attention Variants. Table 6 assess the impact of key architectural choices in TRACE, including channel identity token (CIT) and different attention mechanisms. Removing CIT results notable increase in average MSE, indicating its importance for capturing fine-grained temporal dependencies. We also replace the channel-biased attention (CbA) with two alternatives: full attention, similar to multivariate variant of Moment [8], and causal attention, analogous to decoder-only designs like Timer-XL [9], Both alternatives yield degraded performance. These results highlight the effectiveness of the architectural design in TRACE, particularly the synergy between CIT and CbA in achieving outstanding performance. We refer to Appendix D.9 for runtime and efficiency evaluation.
Cross-Attention and Hard Negative Sampling. Fig- ure presents an ablation study on key components in TRACE for retrieval precision under different num- bers of negative samples (K). "all" indicates using the entire batch (excluding the paired counterpart) as negatives. The default model, using nomic text encoder [57], consistently achieves the highest per- formance, especially when is small, highlighting its efficacy in low-computation settings. Removing the final cross- cross-attention module between time series and text leads to notable performance degradation under small K, suggesting that cross-modal fusion becomes especially crucial when fewer negatives available. Similarly, eliminating channel-level align- ment yields consistent drop, confirming the strength of the proposed dual-level contrastive mechanism. Substituting nomic with weaker text encoders like bge or MiniLM LM results in worse performance, imply- ing that high-quality embeddings are necessary for discriminating harder negatives. Overall, these trends support the effectiveness of our hard negative mining strategy and emphasize the importance of dual-level aligmment retrieval performance. We provide em- pirical case studies in Appendix D.7.
Table 6: Ablation study on attention variants in pre-training architecture.
Avg. MSE
Acc, (%)
TRACE who CIT
0.67011.013
85.20:10.13
0.713+00016
85.04++0.26
CbA = Full Attn
0.705=00013
84.18+0.1
Cbλ Causal Attn
0.682.10.015
83.72.10133
Precisieng (%)
30
20
w/o croes-atte creet-ane-ttoon
10
w/b charnel aligmmett aet bge base en.51.5
-l-.mii....... 5-12
64 Number Negative Samples
Figure Retrieval performance under vary- ing numbers of negative samples. The best is indicated by
6 Conclusion and Future Work
We introduce TRACE, a multimodal framework that aligns time series with textual descriptions at both channel and sample levels. Extensive experiments demonstrate that TRACE outperforms strong baselines across retrieval, standalone encoding, retrieval-augmented settings, and generalizes well across model families. One limitation is that TRACE relies supervised textual alignment, which may not be readily available in all domains. future work, we plan to extend TRACE to support weakly supervised and semi- supervsuee settings, where textual context is partially missing or noisy. Another promising direction is integrating domain adaptation techniques to improve generalization across unseen domains and sensor modalities (e.g., image, video). Moreover, exploring autoregressive generation conditioned on retrieved time series-text pairs may further enhance understanding tasks in temporal modeling. We refer to Appendix for detailed discussion on social impact.
Acknowledgments and Disclosure of Funding
This research was supported in part by the National Science Foundation (NSF) Division of Computer and Network Systems (CNS-2431.504) Army Research Office contract W91 1NF-23-11111 23-1--088.- the Yale AI Engineering Research Grant from the Yale Office of the Provost, the LEAP-U Sponsored Research from Samsung Research America, the Roberts Innovation Award 2025, and the AWS Research Awards. We would also like to thank the anonymous reviewers for their insightful and constructive feedback.
10

---

References
[1] Francisco Martinez Alvarez, Alicia Troncoso, Jose Riquelme, and Jesus S Aguilar Ruiz. Energy time series forecasting based pattern sequence similarity. IEEE Transactions Knowledge and Data Engineering, 23(8): 1230-1233222 2010.
[2] Irena Koprinska, Dengsong Wu, and Zheng Wang. Convolutional neural networks for energy time series forecasting. In 2018 international joint conference on neural networks (IJCNN), pages 1-8. IEEE, 2018.
[3] Rafal A Angryk, Petrus Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajann Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, et al. Multivariate time series dataset for space weather data analytics. Scientific data, 7(1):1-13, 2020.
[4] Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 382(6677))1416-1422 2023.
[5] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Prabhakar Kamarthi, Aditya Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, et al. Time-mmd Multi-domain multimodal dataset for time series analysis. Advances Neural Information Processing Systems, 77.77888-779333 2024.
[6] Jialin Chen, Aosong Feng, Ziyu Zhao, Juan Garza, Gaukhar Nurbek, Cheng Qin, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, and Rex Ying. Mtbench: multimodal time series benehmark for temporal reasoning and question answering, 2025.
[7] Geon Lee, Wenchao Yu, Kijung Shin, Wei Cheng, and Haifeng Chen. Timecap: Learning to contextualize, augment, and predict time series events with large language model agents, 2025.
[8] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint wrXi::2402.03885 2024.
[9] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer-xl: Long- context transformers for unified time series forecasting. arXiv preprint arXhv:2410.04803, 2024.
[10] Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time-moe: Billion-scale time series foundation models with mixture of experts. arXiv preprint arXiv:2409.16044 2024.
[11] Chin-Chia Michael Yeh, Huiyuan Chen, Xin Dai, Yan Zheng, Jumpeng Wang, Vivian Lai, Yujie Fan, Audrey Der, Zhongfang Zhuang, Liang Wang, al. An efficient content- based time series retrieval system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4909-4915, 2023.
[12] Huanyu Zhang, Chang Xu, Yi-Fan Zhang, Zhang Zhang, Liang Wang, and Jiang Bian. Timeraf: Retrievallaugmennedd foundation model for zero-shot time series forecasting, IEEE Transactions on Knowledge & Data Engineering, (01):1-12, 2025.
[13] Jingwei Liu, Ling Yang, Hongyan Li, and Shenda Hong. Retrieval -uggeeeee diffusion models for time series forecasting, Advances in Neural Information Processing Systems, 37:2766-2786. 2024.
[14] Kanghui Ning. Zijie Pan, Yu Liu, Yushan Jiang, James Y Zhang, Kashif Rasul, Anderson Schneider, Lintao Ma, Yuriy Nevmyvaka, and Dongjin Song. Ts-rag: Retrieval- generation based time series foundation models are stronger zero-shot forecaster. arXiv preprint arXiv:2503.07649 2025.
[15] Silin Yang, Dong Wang, Haoqi Zheng, and Ruochun Jin. Timerag: Boosting Ilm time series forecasting via aetreneal-uummented generation. ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
11

---

[16] Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. arXiv preprint arXiv:2304.05200 2023.
[17] Jongseon Kim, Hyungjoon Kim, HyunGi Kim, Dongjun Lee, and Sungroh Yoon. A compre- hensive survey of deep learning for time series forecasting: architectural diversity and open challenges. Artificial Intelligence Review, 58(7):1-95, 2025.
[18] Jialin Chen, Jan Eric Lenssen, Aosong Feng, Weihua Hu, Matthias Fey, Leandros Tassiulas, Jure Leskovec, and Rex Ying. From similarity to superionity: Channel clustering for time series forecasting, Advances in Neural Information Processing Systems,
2024.
[19] Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0476.. 2020.
[20] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in neural information processing systems, 35:9881-9893, 2022.
[21] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long, Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 14:22419-224435 2021.
[22] Y Nie. A time series is worth 64words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14738 14730, 2022.
[23] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
[24] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
[25] Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems, 34:23592-23608, 2021.
[26] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. arXiv preprint arXiv:2201. /2740 2022.
[27] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2021.
[28] Aosong Feng, Jialin Chen, Juan Garza, Brooklyn Berry, Francisco Salazar, Yifeng Gao, Rex Ying, and Leandros Tassiulas. Efficient high-resolution time series classification via attention kronecker decomposition. arXiv preprint arXiv:2403.00 048822 2024.
[29] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 11121-11 128, 2023.
[30] Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303. 23Xvv22333.06053 2023.
[31] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. arXiv preprint arXiv:2405. /4616, 2024.
[32] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, al. Chronos: Learning the language of time series. arXiv preprint 400 2024.
12

---

[33] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou, A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[34] Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. arXiv preprint arXiv:2410. 10469, 2024.
[35] Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-Ilm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310001728, 2023.
[36] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:45322-43335,, 2023.
[37] Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, and Yuntian Chen. Context-alignment: Activating and enhancing Ilm capabilities in time series. arXiv preprint arXiv:2501.032: 037447. 2025.
[38] Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang. Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards li-eemoweredd multivariate time series forecasting via cross-modality alignment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 18780-18788, 2025.
[39] Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. s² ip-lim: Semantic space informed prompt learning with Ilm for time series forecasting. In Forty-first International Conference on Machine Learning, 2024.
[40] Taibiao Zhao, Xiaobing Chen, and Mingxuan Sun. Enhancing time series forecasting via lulti-level text alignment with llms. arXiv preprint arXiv:2504.000 2000000000000,, 2025.
[41] Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. In Proceedings of the AAAI Conference on Artificial Intellignce, volume 39, pages 12694-12702, 2025.
[42] Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. Chatts: Aligning time series with Ilms via synthetic data for enhanced understanding and reasoning. arXiv preprint arXiv:2412.03104 2024.
[43] Yushan Jiang, Wenchao Yu, Geon Lee, Dongjin Song, Kijung Shin, Wei Cheng, Yanchi Liu, and Haifeng Chen. Explainable multi-modal time series prediction with tn-inn-he--loop. arXiv preprint a:Xivv2503.01013, 2025.
[44] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 8980 8987, 2022.
[45] Baoyu Jing, Si Zhang, Yada Zhu, Bin Peng, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Retrieval based time series forecasting. arXiv preprint arXiv:2209.13525, 2022
[46] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568: 127063, 2024.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International conference on learning representations, 2021.
[49] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint 1008:41908.10084, 2019.
13

---

[50] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval sarreval-augmenter generation for large language models: A survey. arXiv preprini arXiv: 10997, 2023.
[51]
DOCIOAANESDISSCCCC and National Climatic Data Center, NESDIS, NOAA, U.S. De- partment of Commerce. Storm Events Database, 2023. Accessed: February 21, 2025.
[52] Matthew Menne, Simon Noone, Nancy W. Casey, Robert H. Dunn, Shelley McNeill, Diana Kantor, Peter W. Thome, Karen Oreutt, Sam Cunningham, and Nicholas Risavi. Global Historical Climatology Network-Hourly (GHCNh), 2023.
[53] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Times- net: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022.
[54) Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. arXiv preprint 24X.v::2005.46666 2024.
[55] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[56] Pavel Senin and Sergey Malinchik. Sax-vsm: Interpretable time series classification using sax and vector space model. In 2013 IEEE 13th international conference on data mining, pages 1175-1180. IEEE, 2013.
[57] Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder. arXiv preprint 3rX1v:2402.01613, 2024.
[58] Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Chen Wang. Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. arXiv preprint arXiv:2407.13277 13278, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Section 3,4 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer this question will be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 'tent acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 and Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

reproducible, they If the paper includes experiments, No answer to this question will not be perceived
well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be a way to access this model for reproducing the results or a way reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for repro- ducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code link will be put for the camera-ready copy
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/nipps.c/ppbbicl Baides/CCadeeubmissioooooiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: httpppiicggiide/ooeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
17

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Ye]]
Justification: Section 4 and Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Section 4 and Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eттor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required each of the individual experimental runs as well as estimate the total compute.
18

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Question:
Answer: [Yes]
Justification: All section
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are properly cited
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode wtw.oodddd.o//mmatt...s.o. has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htttss//eeeriiaccccooooereeeeess for what should or should not be described.
21

---

Appendix
A
Notations
The main notations used throughout this paper are summarized Table
Table 7: Summary of the notations used in this paper.
Notation
Description
X CCxT
Input multivariate time series with C' channels and T time steps Patch length for time series tokenization
Number of temporal patches per channel (LT/P))
C
Number of channels (variables)
Number of time steps in the original sequence Forecasting horizon
Embedding dimension
Length of the flattened token sequence
Patth
Embedding of the i-th patch token
M {0, 11
Biased attention mask for the flatten token sequence Output token embeddings from the Transformer encoder Embedding of the global [CLS] token
H R Lxd
[cL.)] E Rd
Harr E RRxd
Embeddings of Channel Identity Tokens (CITs) Sample-level textual context associated with time series Channel-level textual description for the e-th variable
Text
Zext E IRd
Semantic embedding of the sample-level text Semantic embedding of the channel-level text
Ze E Rd
N Nex,, Nah
Sample-level and channel-level hard negative candidate sets
B Dataset Curation
We curate new multimodal time series dataset in the weather domain by extending MTBench [6]. It built from two primary sources:
Event reports from the NOAA Storm Events Database(51] which contains detailed narratives of severe weather occurrences across the U.S.
Weather Time Series (TS) data from the NOAA Global Historical Climatology Network Hourly (GHCN-hJ[52), covering multiple metcorological variables.
When applying TRACE to our curated dataset, the sample-level context is event report, while the channel-level description synthetically generated by LLMss.
B.1 Station and Event Selection
We begin by selecting over 100 U.S. locations frequently affected by severe weather events and associated with long narrative reports. This yields approximately 5,000 event entries. For each event location, identify nearby GHCN-h weather stations and extract multivariate TS data anchored the start time each event.
B.2 Time Series Sampling
Each event treated as an anchor point extract TS data at three resolutions:
Hourly for 7 days
Every 4 hours for 28 days
22

---

Daily for 180 days
This results in approximately 15,000 TS samples from event-associated windows. To balance the dataset, we sample additional 30,000 TS sequences from the same stations random non-event times, ensuring no overlapping event narratives. To enhance weather diversity, we also sample 30,000 TS sequences from geographically distant stations without any event association, using randomly selected anchor times. See Figure 6. All time series instances contain seven channels: temperature, humidity, wind_ u, wind v, visibility, precipitation, and sky code. The curated weather dataset contains total of 74,337 time series instances, and the lengths have a mean of 169.25 and median of 168.0.
Figure 6: The red points are locations with event reports and the blue points are locations without event reports
Example: An Event Report of Debris Flow
"event type": "Debris Flow". "state": "CALIFORNIA".
"cz name": "TULARE",
"begin datetime": "2021-12-14 13:14:00".
"end datetime": "2021-12-14 16:14:00",
"narrative": "A strong low pressure system dropped southeast of the Gulf of Alaska on December 12 and intensified off the Pacific Northwest coast on December 13 pulling up some deep moisture which was pushed into central California during the afternoon. The precipitation intensified during the evening of December 13 through the morning of December 14 as the low carved out a deep upper trough which pushed across California during the afternoon of December 14. This system produced 2 to 4 inches of liquid precipitation over the Sierra Nevada from Sequoia National Park northward and south of Sequoia Park. The precipitation fell mainly in the form of snow above 5500 feet and several high elevation SNOTELs estimated to 4 feet of new snowfall. The snow level lowered to as low 1500 feet during the evening of December 14 as the cooler airmass behind the system pushed into central California. Much of the San Joaquin Valley picked up between I to 2 inches of rainfall while the Kern County Mountains picked up between 0.75 and 1.5 inches of liquid precipitation. The Kern County Desert areas only picked up between quarter and half inch of rain at most locations due to rain shadowing. The storm produced
inches of liquid precipitation
23

---

widespread minor nuisance flooding in the San Joaquin Valley and Sierra foothills with few rock slides noticed. Several roads were closed as precaution and chain restrictions were implemented some roads in the Sierra Nevada. The storm also produced strong winds over the West Side Hills as well as in the Grapevine and Tehachapi areas in Kern County. Several stations in these areas measured wind gusts exceeding 50 mph with few locations near the Grapevine measuring brief gusts exceeding 70 mph. California Highway Patrol reported mud, rock and dirt covering most of North Plano St. near Lynch Dr.",
B.3 Synthetic Description Generation
We use ChatGPT to generate channel-level textual descriptions for selected TS samples, where all TS samples linked event reports are included. We also randomly select 50% of TS samples from both the non-event windows at event-associated stations and the non-event-associatee stations to generate channel-level descriptions for event-label balance. The generated descriptions follow the style of TimeCap[7], [7] but each is additionally annotated with one or more keywords selected from the set as auxiliary information: (Clear, Cloudy, Rainy, Snowy, Windy, Foggy, Hot, Cold, Humid, Stormy). We use consistent meta meta-prompt to elicit both descriptive and label-aligned outputs. A full example of the meta-prompt and generated description provided in B.4.
B.4 Prompt for Weather Description Generation and example synthetic description
Weather Summary Prompt
You are daily weather reporter, asked to summarize the past seven days of hourly weather (or the past 28 days of 4-hourly weather, or the past months daily weather, depending on the selected mode).
will be multichannel with temperature, precipitation, relative. ree_hmmidity visibility, wind_u and wind_v aspects. Summarize these channels and label the overall weather with one or more keywords from the set:
{Clear, Cloudy, Rainy, Snowy, Windy, Foggy, Hot, Cold, Humid, Stormy).
You are not expected to report each time point individually. Instead, analyze the entire period whole. Additionally, for temperature, precipitation, and relative feedimiiiiitt ty, identify any noticeable trends, potential periodicities (e.g. daily or weekly patterns), overall volatility, and any clear outliers that stand out. You do not need to analyze other channels for these advanced statistics.
The input includes:
Location
Date
Temperature time series
Precipitation time series
Relative humidity time series
Visibility time series
Wind u time series
Wind v time series
Sky cover codes
Sky cover codes are interpreted as follows:

---

Code
Meaning
Sky Fraction Covered
00
CLR (Clear)
0/8 or 0%
01
FEW
1/8 12%)
02
FEW
2/8 3/8 (25%-37%)
03
SCT (Scattered)
4/8 50%)
04
SCT
5/8 62%)
05
BKN (Broken)
6/8 (75%)
06
BKN
7/8 8/8 (87%-100%)
07
BKN
~9/10
08
OVC (Overcast)
10/10 (100%)
09
VV (Vertical Visibility)
Sky obscured
10
X (Unknown)
Partially obscured
Please summarize the data using the following format:
Date: (sentence of date)
Location: (sentence of location)
Temperature: [sentence of temperature]
Precipitation: (sentence of precipitation]
Relative Humidity: (sentence of relative humidity)
Visibility: (sentence of visibility)
Wind V: [sentence wind_
Wind U: [sentence of wind_ut
Sky Cover: (sentence of sky cover)
Keywords: (list of keywords from label set)
No additional explanation or commentary should be included in the output.
Example: Generated Weather Summary
Date:
The past 28 days from January 30, 2021, February 26, 2021.
Location:
The weather data is from Pike, Kentucky.
Temperature: The temperature ranged from low -10.77 "C to a high of 20.25 PC with noticeable fluctuations and general upward trend towards the end of the period, indicating warming conditions.
Precipitation: There was no recorded precipitation throughout the 28 days, indicating dry weather.
Relative Humidity: Relative humidity varied significantly, peaking at 100% on multiple occasions, with general trend of higher humidity levels during the earlier part of the period and lower levels towards the end.
Visibility: Visibility remained consistently high 14.58 km throughout the reporting period. Wind V: Wind velocity showed variability, with occasional gusts and a general trend of calm conditions.
Wind U: Wind direction fluctuated, with both positive and negative values indicating changes in wind patterns.
Sky Cover: The sky was consistently clear with no significant cloud cover reported. Keywords: [Clear, Cold, Humid]
B.5
Dataset Details
Our curated weather dataset contains total of 74,337 time series instances. We allocate 9,561 of these exclusively for the forecasting task, ensuring this subset is disjoint from the pretraining and classification data to avoid any potential label leakage or information overlap. The classification task is formulated as multi-class event prediction, where each time series instance is annotated by the NOAA System with a corresponding weather event type from nine common severe weather events, and one special category for non-events. The event labels are follows: Lightning (0), Debris
25

---

Table 8: Dataset size for each task.
Dataset Type
Train
Test
Val
Total
Newly Curated Weather Dataset
Forecasting (H=7)
6,690
957
1,914
9,561
Pretraining Classification
45,339
6,484
12,953
64,776
Public Dataset from TimeMMD
[5]
Health (H=12)
929
266
129
1,324
Energy (H=12)
992
284
138
1,414
Environment (H=48)
7,628
2,173
1,064
10,865
Flow (1), Flash Flood (2), Heavy Rain (3), Tornado (4), Funnel Cloud (5), Hail (6), Flood (7), Thunderstorm Wind (8). Instances that do not correspond to any specific event are labeled as None. This setup ensures the model learns to distinguish between distinct event types while being robust to trivial (non-event) data. We follow the original split to create train/test/val set for TimeMMD forecasting tasks [5].
C Alignment Objective: Full Formulation
To fully capture the structured alignment between multivariate time series and text, employ dual-level contrastive learning strategy, consisting of sample-level and channel-level hard negative mining.
C.1 Hard Negative Candidate Sets
Given a time series instance with C' channels, we define the following negative sets:
Sample-level negative sets. For aligning the global [CLS] embedding h (cis) instance with its (6)
corresponding sample-level textual embedding ext we mine hard negatives from other samples in the batch. Specifically:
A ext
Topx sim
(0) (cLs) 2ext | + $
(3)
and symmetrically,
(i,tee))) Topx (imm( (zexi h () (CLS) + $
(4)
Channel-level negative sets. To align each channel-specific CIT embedding h.S with its corre- sponding channel-level text embedding (!) we mine two types distractors:
Intra-instance negatives: embeddings from other channels within the same instance, i.e., (
where d' e;
Inter-instance negatives: same-indexed channel embeddings across different instances, i.e..
( where j #
Formally, the channel-level negative set is defined as:
iie) Topx sim ( 2(") corj #1)
(5)
and similarly in the reverse direction:
A (i,e,es)) TTopx { (iim(z) (1, hh))) + c +1).
(6)
ch
26

---

C.2
Contrastive Alignment Loss
We adopt bidirectional InfoNCE loss at both the sample and channel levels. For each alignment direction, the objective maximizes the similarity between the positive pair and minimizes similarity with hard negatives.
Sample-level loss.
exp(sim Z Zext h [CLs] /rr)
global
log
exp(sim exxt tass )/7
(
(
(7)
je (E( ext
exp(sim(h [LS)] (ext)/T)
Aes- global
log
(8)
(8) exp(sim [CLs] 2xx) (T)
jE(1)uu
Channel-level loss.
C
exp(sim Zo .h (b("))) )/T)
Le channel
1
Σ
- -log
(9)
C
Σ
exp(sim(2,) (1)) )/T)
c=1
(iel) (iet) ((ic))uutiion
0
exp(sim(h,. (3) 1/T)
Als-tlext
-
-log Σ
(10)
0
exp(sim(h," (1) 220 227
c=1
(ie') ((te))uuu
C.3
Total Loss Objective
The total alignment loss is the average of both sample-level and channel- contrastive losses:
Lalign
global + Gglobal + Ach
(Clext-its + Channell
(11)
2
where the temperature hyperparameter, and Ach is hyperparameter, controlling the contribution of channel-level alignment. We set Auh 1.0 as default in experiments.
D Experiments
D.1 Baselines
D.1.1 Full-shot Time Series Models
DLinear [29]is lightweight time-series forecasting model that decomposes the input into trend and seasonal components, and applies simple linear layers to each component separately. Despite its simplicity, DLinear has demonstrated strong performance on both long- and short-term forecasting tasks by effectively capturing linear temporal patterns without relying on complex neural architectures.
PatchTST [22] reformulates time-series forecasting as patch-based sequence modeling problem. splits the input time series into non-overlapping patches and applies Transformer encoder to model inter-patch dependencies. The design removes positional encoding and avoids decoder layers, making the model more suitable for forecasting tasks while benefiting from the global receptive field of Transformers.
TTransformer [24] (Instance-aware Transformer) extends Transformer-based forecasting by modeling instance-wise variations. It introduces a shared backbone Transformer and an instance-specific modulation mechanism, enabling the model to better adapt diverse temporal dynamics across different time-series samples. This design improves generalization and robustness, particularly for multivariate forecasting.
TimesNet [53] proposes a novel temporal block that captures multi-frequency patterns time-series data using learnable convolutions in the frequency domain. By combining time and requencyydomaan
27

---

features, TimesNet achieves strong performance across a variety of datasets. It is particularly effective modeling both short-term and long-tew term temporal dependencies.
TimeMixer [54] employs a structured state-space-inspiree architecture where time mixing and channel mixing operations alternate. It replaces self-attention with parameter-efficien mixing blocks that blend information across the temporal and feature dimensions. TimeMixer is designed for scalable forecasting and excels in low-resource regimes due to its compact architecture and efficient training.
FSCA [37] introduces the language environments familiar to LLMs to enable LLMs to contextualize and comprehend TS data, thereby activating their capabilities. FSCA uses Dual-Scale Context-Alignmen Graph Neural Networks (DSCA-GNNs) framework to achieve both structural and logical alignment, demonstrate good performance in few-shot and zero-shot settings.
paradigm that aligns time series (TS) with linguistic component in
D.1.2
Time Series Foundation Model
Table 9: Comparison of time-series foundation models.
Method
Chronos
Time-MoE
TimesEM
Moirai
Moment
Timer-XL
Architecture
Decdder
Decoder-Daly
Decoder-Onf Only
Encodcr-Only
Encoder-Onb
Decodcr-only
(Max) Model Size
710M
2.4B
200M
311M
385M
84M
Input Token
Point
Point
Patch
Patch
Patch
Patch
Max Length
512
4096
512
5000
512
1024
FEN Cross-channel
Dense
Sparse
Dense
Dense
Dense
Dense
x
We test several recent time-series foundation models that have been pretrained on large-scale datasets from relevant domains, including weather, healthcare, energy, and environment. These include Chronos [32], Time-MoE [10], TimesFM [33], Moirai [34], Moment [8], and Timer-XL [9], which offer strong generalization through large-scale pretraining. A comparison given in Table 9, To performance on diverse real-world domains, we integrate our retriever with three publicly available time-series foundation models: Time-MoE, Timer-XL and Moment, which are selected based on the availability of stable, open-source implementations that support customization and downstream fine-tuning. We leave the adaptation of our retriever to additional proprietary or closed-source foundation models, as well as its integration into unified pretraining pipelines, for future work.
evaluate
Comparison of TRACE with Time-series Foundation Models. It is important to note that model is not itself a cross-doman foundation model, but rather modular encoder based retriever capable of enhancing such models. Architecturally, model adopts encoder-only design with flexible point- and satch-based tokenization, supports input sequences exceeding 2,048 tokens, and enables effective cross-channel interactions through channel-biased attention mechanisms.
D.2 Experiment Configurations
All models are implemented in PyTorch and trained on NVIDIA A100 40GB GPUs. For most time series models, we adopt the implementation from TSLib [ The sequence length is fixed at 96 for both prediction horizons of and 24. We use mean squared error (MSE) as the loss function for forecasting tasks, and accuracy for classification. Forecasting models are trained for 10 epochs, while classification models are trained for up to 150 epochs with early stopping. We follow the official code to implement other baselines [37]*. All other hyperparameters follow the default settings in TSLib, except for those explicitly tuned to achieve the best performance, as reported in Tables 10. For our model. the initial learning rate is tuned from {10 10 10-33. The number of attention layers is tuned from {6, 12}. and the hidden dimension is from (384, 768} with the number of heads {6, 12}.
Library
28

---

Table 10: Best hyperparameters per model
Model Name
Learning Rate
Encoder Layers
Hidden Dimension
DLinear
0.0010
2
32
PatchTST
0.0050
4
64
TimeMixer
0.0100
4
64
TimesNet
0.0010
64
iTransformer
0.0100
4
64
PSCA
0.0001
4
256
D.3 Embedding Visualization
Figure 7 presents the cosine similarity matrix between text and time series embeddings across the test set. The diagonal dominance indicates that TRACE successfully aligns each time series with its corresponding textual description, suggesting strong one-to-one semantic matching in the shared embedding space. Off-diagonal similarities remain low, demonstrating the model's 's ability to distinguish unrelated instances. Figure visualizes the joint embedding space using UMAP. Each color represents distinct event category, where circles (о) denote time series instances and crosses (x) denote their corresponding textual descriptions. line connects each text time series pair. We observe clear clustering by event type, with paired modalities positioned closely in the embedding space. Notably, for some events (e.g., "Flood" and "Debris Flow"), clusters partially overlap. reflecting shared underlying dynamics. The tight alignment between paired points validates the effectiveness of our dual-level alignment strategy, and the modality-ming mixing within clusters suggests successful fusion of structured and unstructured signals.
1.0
Flash
Heavy
Time Series Embeddings
Funncll Cloud
Hall
Food
Thunderdtom
Time Series
0.4
Text
0.2
0.0
-0.2
-0.4
Text Embeddings
Figure 7: Cosine Similarity Matrix Between Text and Time Series Embeddings.
Figure 8: Umap Visualization of Aligned Text and Time Series Embeddings.
D.4 Classification Task
Table 11 reports the classification accuracy and Fl scores of different size variants of time-series foundation mod- els the weather event classification task. We observe that a larger model size does not necessarily lead to bet- ter performance. For example, Moment's base model achieves a higher Fl score than the large model despite a lower accuracy. contrast, Chronos exhibits more stable performance across scales, with the tiny and mini vari- ants achieving the best Fl scores, outperforming even the larger variants, These results suggest that, in domain- specific classification tasks with relatively limited super- vision, scaling up foundation models may not always be beneficial, and smaller models can offer a better balance between accuracy and efficiency.
Table 11: Weather Event Classification Accuracy and F1 Score (%).
Model
Size
Accuracy
F1
small
56.27 59.09
16.56
Time-MoB
large
19.74
base
65.43
28.29
Moment
large
64.94
26.35
tiny
74.79
40.21
mini
73.89
37.98
Chronos
small
71.07
35.39
base
71.42
36.40
large
71.97
36.30
29

---

D.5 RAG Setting
In our retreeval-augmented generation (RAG) framework, given a query time series g.. we compute its [CLS] token embedding as h2 I2 using the frozen encoder from TRACE. Based on cosine similarity, we retrieve the top-ft most relevant multimodal pairs (X' Tox)=== from the corpus, where X is historical multivariate time series and rox: is the associated sample-level context. Each retrieved pair is transformed into soft prompt vector using trainable linear projection layer. Specifically, the time series component is encoded to (0) IRd, and the textual context T is encoded to zext(() E R" using a frozen concatenate each pair as p(o) = [A (1) (6)
followed by shared projection. For the TS+ Text setting, we eext E R2dd and stack all R vectors to form the final prompt:
P Proj (D .......pp))) :D! E Rdy,
where Proj feedforward layer mapping from I2add + Ikd,, and dy is the hidden dimension of the downstream time series foundation model. For the TS-only setting, we omit the text component and instead concatenate [hg ()...... R I. Rd and project into Rd accordingly.
This dense prompt P is prepended to the query sequence during inference. For decoder-oni models (e.g., Timer-XI,, Time-MoE), P is appended to the autoregressive context at each decoding step. For encoder-only models (e-g., Moment, TRACE), is inserted as prefix to the encoder input, i.e.,
Head g)),)))
where H, E RR4xd is the encoded query and Head is forecasting head trained from scratch. In all configurations, only Proj and Head are updated during training RAG framework, while the backbone foundation model remains frozen.
D.6 Standalone Time Series Encoder
To evaluate the classification capabilities of time series foundation models, finetune multi-layer perceptron (MLP) classifier on top of each model's final output representation, as most existing time series foundation models do not support classification task by design, except Moment [8], The MLP consists of four hidden layers with sizes [256, 128, 64, 32]. followed by softmax output layer corresponding weather event categories. This architecture was selected based on empirical our classification task. We include all available variants from
tuning for optimal performance four foundation model families: Time-MoF, Timer-XI., Moment, and Chronos. All backbone parameters of the time series foundation models fully activated and updated during training to ensure consistency and fair evaluation. Each model is finetuned for 100 epochs using the Adam optimizer. The training batch size is set to 256 for small and mid-sized variants, and reduced 128 for larger models to accommodate memory constraints.
For full-shot time series models, we train them from scratch using unified training and evaluation protocol with time series foundation models. Results are shown in Table 4 and Table 11.
D.7 Empirical Case Study
Figure iilustrates the capability to align detailed textual context with corresponding multivariate time series. The retrieval pool is constructed by excluding the query's paired time series instance. This setup ensures that retrieved results are non-trivial and reflect the model's ability to identify semantically similar yet distinct examples. TRACE leverages both high-level and fine-grained semantic cues to retrieve the most relevant time series from the curated candidate pool. The top-1 retrieved sequence closely reflects key patterns in the query text, which can serve valuable reference for downstream forecasting, scenario simulation, or contextual explanation.
D.8 Timeseries-to- Timeserress Retrieval
TS-to-TS Case Study. Figure illustrates a case study of TS-to-TS retrieval using TRACE. Given a query time series (top row) labeled as Flood, the system retrieves the top-3 most similar samples from the corpus based on embedding similarity in the shared representation space. The similarity score for each retrieved sample is shown on the left, with per-channel similarity values annotated below
30

---

Flash Flood Evert Report:
tough rigpored subsiling midday
record-crook crooking 16.17 in Nighest 24-hour
Channellavel Description:
The temperature ranged from low of 6 to high ss showing a noticable le daiyy pottorn with nameer ter perpertlars during the day and coolor tomorttture at 2ight accosional sugpesiig typ
There wor sporodic instanocs pesciptationn
significant poak rim Jil inclozting a goner ally weather coed tore.
warl
Visibiity remained relatively high,
around 16 D5 im occasional
lower valune
Wind direction showed variaility, with some poriods caln windd and others with deonger gusts from various direcions.
Wind volocity varied, with notable gusts maching up to m/s indicatig some windy contitions lines.
any cover ranged rrom cue acateres couts with few reranoes of broken couds, and cating moaely clear conditions throughout the weak
Top-1 Retrieved Time Series (Label: Flash Flood:: tomporature
procipitation
relative hamidity
visiiiity
wind
wind
sky code
22.5
Figure 9: A case study of cues are highlighted in color for clarity.
retrieval of flash flood-related time series, The key textual
temperature
precisitation
relative humidRy
visiblityy
windi
wind
dky code
Quary
1400
Figure 10: Visualization of Timescries-to Timeeseries Retrieval by TRACE
each plot. We observe that all retrieved samples have high overall similarity scores (approximately 0.79-0.81), reflecting strong semantic alignment. The top-2 retrievals are also labeled as Flood, while the third belongs a semantically related event, Flash Flood, suggesting that TRACE is capable of retrieving contextually relevant samples even across closely related labels. Notably, TRACE enables fine-grained channel-level similarity assessment by leveraging its Channel Identity Tokens (CIT), which allow independent embedding of channel-specific signals.
However, we also find that high similarity in individual channels (e.g.. temperature or precipitation) does not always guarantee high overall semantic alignment. For instance, the first retrieval shows moderate similarity across channels but still achieves high overall semantic score. This highlights the benefit of TRACE's structured aggregation over all channels to capture global semantics and reveal the most semantically dominant channels that contribute most to the retrieval relevance. This capability enables TRACE to go beyond surface-level similarity, retrieving samples that share latent event signatures rather than merely matching patterns across all channels uniformly.
D.9 Complexity and Efficiency
D.9.1
Computational Complexity
We analyze the computational complexity of the main components in TRACE, including the encoder stage, the dual-level contrastive alignment, and the oetrieval-augmentt generation (RAG) setup.
1. Encoder Pre-training Complexity. Let X E RCT be the input multivariate time series with C channels and T time steps. The sequence is tokenized into / [T/P] patches per channel, each projected to d-dimensional embedding. The total token length after flattening is 1 + C(î + 1).
31

---

This includes one global [CLS] token, one [CIT] token per channel, and T patch tokens per channel. The encoder is N llyer Transformer with multi-head channel-bias -iised attention. The complexity per attention layer is ( = d)) Note that channel- -haanell-based attention applies a sparse mask M E {0, 1 to restrict certain attention to within-channel interactions, which effectively reduces the constant factors in practice but not the asymptotic complexity.
2. Dual-level Contrastive Alignment. Let B be the batch size. For each time series, the alignment stage computes:
Sample-level similarity: O(B²d) for all [[cLs] -ext
Zext pairs.
Channel-level similarity: For C' channels and B instances, total cost is d( for he-
pairs.
Negative mining selects top-R R hardest negatives per instance and per channel, which costs O(B log R + BC'
BC'log log R), and is negligible compared to similarity computation.
Retrieval-Augmentedd Generation. During inference, retrieval selects top-R neighbors for query based on cosine similarity:
Retrieval cost: (Rd) using approximate methods (e.g., FAISS) from database.
Prompt generation: if soft prompt dimension dy. and each retrieved pair contributes d-dim vector, this yields a projection cost of (.
The forecasting model remains frozen; only the soft prompt (a single vector of shape [1, d,,) is appended, incurring no extra Transformer-layer cost.
Summary. Pre-training (Transformer encoder) yields O(L2d) per layer. Alignment yields O(B2d B²Cad) and RAG inference yiclds O ((Rdyy) for retrieval and projection.
D.9.2 Empirical Runtime
We report the model size and empirical runtime of TRACE and other baselines in Table 12, including FSCA [37], which the second-best train-from-seratch time series model, and time series foundation models with the availability of open-source implementations. TRACE activates only 0. 1M parameters during finetuning with lightweight linear head, which is nearly 200x fewer than FSCA and over 700x fewer than Hime-MoEsmar This lightweight design results in substantially faster training and inference speed. Compared Moment, TRACE achieves faster training time with significantly fewer trainable parameters and better performance, which can be attributed to its multichannel modeling with channel- attention. While slightly slower than Timer-XI., which a decoder-only model with causal attention, TRACE offers an acceptable overhead given its significantly stronger retrieval performance and the high quality of embeddings produces for cross-modal and TS-to-TS retrieval. It is worth noting that for Timer-XL XL and Time-MoE, despite their strong generalizability, parameter- efficient finetuning strategies are relatively underexplored, as all model parameters must be activated and updated during finetuning for reasonable performance in domain-specift tasks.
Table 12: Comparisons of model efficiency. Activated Params indicates the number of parameters activated during finetuning for 7-step forecasting on the weather dataset. Training and inference time are seconds per epoch on the forecasting dataset. Device is single A100 40GB GPU.
Total Params
Activated Params
Training Time
Inference Time
FSCA
82.35M
22.68M
1249.701
1.589
TRACE
10.78M
0.12M
6.054
0.955
Momentuse
109.87M
0.24M
11.706
1.691
Timer-XI L
84.44M
84.44M
3.392
0.685
Time-MoEsmal
113.49M
113.49M
106.308
15.545
E
Discussion
Limitation. While TRACE demonstrates strong performance multimodal retrieval and retrieval- augmented forecasting, it currently assumes the availability of aligned time series-text pairs during training. some domains, such alignment may be noisy or incomplete. Additionally, although channel-level alignment improves interpretability and fine- fine-grained matching, it introduces modest
32

---

increase in computational overhead during training. We believe these trade-off's are justified by the performance gains but acknowledge that further optimization may enhance scalability.
Future Work. In future work. we plan to extend TRACE to support weakly supervised and semi- supervised settings, where textual context is partially missing or noisy. Another promising direction is integrating domain adaptation techniques to improve generalization across unseen domains and sensor modalities (e.g,, image, video). Moreover, exploring autoregressive generation conditioned on retrieved time series text pairs may further enhance understanding tasks in temporal modeling.
Broader Impact. TRACE offers general framework for cross-modal reasoning in time series applications, with potential benefits domains such healtheare monitoring, disaster forecasting, and industrial diagnostics. By improving retrieval and interpretation of structured temporal data, our approach may enhance decision support and model transparency. However, we encourage responsible deployment and emphasize the importance of auditing training data and retrieval outputs to avoid amplifying biases present in either modality.
33


Reviews:
[{"review_id": "ZeKQFTr7Zu", "reviewer": "Reviewer_3mFi", "summary": "This paper proposes a generic multimodal retriever, TRACE, that grounds time series in aligned textual descriptions. It supports text-to-ts and ts-to-text retrieval modes. By retrieving relavant pairs, TRACE improves predictive accuracy in downstream applications. Besides, it can also serve as a encoder of multimodal time series data. Experiments across several datasets demonstrate the effectiveness of the method.", "questions": "1. It says that the study uses ChatGPT to generate channel-level textual descriptions for selected TS samples. Have you evaluated the overall quality of the descriptions? How do you ensure the quality of the generated descriptions?\n2. How to obtain the soft token of a time series snippet when TRACE is used as a general-purpose retriever? Does the soft token affect the performance of TRACE?\n3. I also wonder how information leakage is avoided during experiments or dataset construction. For instance, textual descriptions of events may include the ground truth of the forecasting target. Does TRACE implement any mechanisms to prevent such leakage?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of retrieving across time series and textual data is compelling.\n2. To enhance the alignment of time series and textual data, it introduces channel-level aligment and hard negative mining strategies.\n3. To evaluate the performance of TRACE, the paper curated a new dataset, with multivariate time series.\n4. The experiments are comprehensive and the results demonstrate the effecitiveness of TRACE.\n\nWeaknesses:\n\n1. The proposed method requires curated multimodal datasets, which are not easy to achieve in time series domain.\n2. Although the paper have taken much efforts to curate Weather dataset, it seems no metrics to demonstrate the quality of the dataset intuitively.\n3. More baselines should be included in Table 4, especially multimodal time series models, e.g., TimeCAP [1], Multimodal PatchTST, Multimodal iTransformer from [2]\n\n    [1] TimeCAP: Learning to contextualize, augment, and predict time series events with large language model agents.\n\n    [2] Time-mmd: A new multi-domain multimodal dataset for time series analysis.", "was_revised": true, "final_justification": "The authors' rebuttal has effectively addressed my initial concerns. The paper presents a novel and compelling idea. Another significant contribution is the curation and introduction of a new multimodal time-series dataset, filling an important gap. The experimental evaluation, now strengthened by the inclusion of additional baselines during rebuttal, provides convincing evidence for the method's effectiveness. Therefore, I increase my rating and recommend acceptance.", "rebuttal": "Thanks for your positive feedback on the idea and experimental results. We have addressed your concerns as follows.\n\n> ### Weakness 1: The proposed method requires curated multimodal datasets, which are not easy to achieve in the time series domain.\n\nWhile perfectly aligned multimodal datasets may require additional effort, such data are increasingly accessible in practice recently. For instance, the TimeMMD [2] already provides multi-domain, multimodal time series data across Health, Energy, and Environment sectors. Furthermore, many time series datasets (e.g., weather, finance, traffic) can be enriched with textual context through simple and scalable methods, including web scraping, report parsing, or LLM-based description generation (as we did in our curated NOAA dataset).  Therefore, multimodal alignment is practical with modest preprocessing (We refer to Appendix B for our detailed preprocessing steps on weather dataset).\n\nIn addition, we note the growing trend in both academia and industry towards integrating language and time series for improved performance in forecasting and reasoning tasks. As the benefits of multimodal modeling become more evident, we expect more such datasets to emerge. TRACE offers a general and extensible framework that not only works with existing aligned datasets but also encourages scalable dataset construction and integration of textual context into time series modeling.\n\n>### Weakness 2: The paper presents a curated Weather dataset but lacks intuitive or quantitative metrics to demonstrate its quality or reliability.\n\nWe provide multiple forms of evidence to support the quality of the curated Weather dataset, detailed in **Appendix B**. For your reference: \n- **Coverage**: The dataset includes diverse weather events (e.g., flash floods, heatwaves, snowstorms) across different regions and time periods.\n- **Examples**: Realistic channel-level textual descriptions are paired with corresponding time series, demonstrating semantic alignment and meteorological plausibility.\n- **Statistics**: With over 74,000 samples across 7 variables, it is among the largest publicly available multimodal weather datasets.\n\n\n\n> ### Weakness 3: More baselines should be included in Table 4, especially multimodal time series models.\n\n- For **TimeCAP**: We have already reached out to the authors to request the official code and are awaiting a response to ensure a fair and reproducible evaluation.  \n- For **Multimodal PatchTST and iTransformer**: We have implemented the MM-TSlib of these models and included initial results on the Weather dataset (see below). It is worth noting that models in Table 4 do not take textual context as the input for time series forecasting. Therefore, we also implemented TRACE w. Text by concatenating the textual embedding with the time series for downstream forecasting. Our experiments show that TRACE still outperforms them across multiple forecasting horizons (H=7 and H=24) in multimodal scenarios.\n\n\n| Model               | H=7 MAE ↓ | H=7 MSE ↓ | H=24 MAE ↓ | H=24 MSE ↓ |\n|--------------------|-----------|-----------|------------|------------|\n| Multimodal PatchTST | 0.478    | 0.589    | 0.523     | 0.636     |\n| Multimodal iTransformer | **0.453**    | 0.561    | 0.508     | 0.616     |\n| TRACE w. Text | 0.458| **0.557** |**0.500** | **0.609** |\n\nWe commit to including comprehensive comparisons with all three models, including TimeCAP, once the official implementation becomes available. These results will be added to the camera-ready version of the paper.\n\n\n\n> ### Question 1: Have you evaluated the overall quality of the descriptions? How do you ensure the quality of the generated descriptions?\n\nTo ensure both factual accuracy and semantic utility, we applied a controlled generation pipeline with multiple layers of validation:\n\n- **Structured Prompting with Constraints**: As detailed in Appendix B.4, we designed a strict template that limits generation to factual, numeric summaries per channel (e.g., min, max, volatility, trends), and enforces keyword annotations from a closed set (e.g., {Clear, Rainy, Foggy}). The prompt explicitly avoids speculative or narrative content.\n\n- **Grounded Generation Using Raw Data**: The full time series values are provided as input to ChatGPT, which ensures that every generated description is directly derived from actual statistics, thus enabling traceability and reproducibility.\n\n- **Qualitative Examples**: Appendix B.4 provides representative samples. For instance:  \n  *“The temperature ranged from a low of -21.57°C to a high of 26.44°C, with noticeable trends showing a gradual warming as the months progressed from winter to summer, indicating a clear seasonal transition. ”* These examples illustrate coherent, granular, and faithful text grounded in underlying trends.\n\n- **Empirical Validation**: The alignment between text and time series is quantitatively validated in Section 4.2. TRACE achieves high cross-modal retrieval accuracy (e.g., Text-to-TS P@1 of 89.63%), and the UMAP plots in Appendix D.3 show tight clustering between paired text and time series embeddings, hence indirectly confirming consistency.\n\n- **Scale and Balance**: As shown in Appendix B.5, over 74,000 multivariate samples were processed. Channel-level text was generated for all event-labeled samples and half of the non-event samples to ensure diversity and reduce leakage.\n\nWe agree with the reviewer that showcasing a side-by-side comparison with real descriptions would further strengthen the case, and we will include such qualitative comparisons in the camera-ready version.\n\n\n\n> ### Question 2: How to obtain soft tokens from time series and how these tokens affect TRACE’s performance?\n\nThe process of obtaining the soft tokens from a set of retrieved time series snippets (with their associated textual descriptions) is detailed in Appendix D.5 in the supplementary. After retrieving the top relevant time series–text pair, TRACE encodes both modalities into embeddings: $h_{\\text{ts}}^{(i)}$ for the time series (from [CLS] token) and $z_{\\text{text}}^{(i)}$ for the text using a frozen SentenceTransformer. These are fused via a trainable MLP over $[h_{\\text{ts}}^{(1)};z_{\\text{text}}^{(1)};......;h_{\\text{ts}}^{(R)}; z_{\\text{text}}^{(R)}]$ to form the final soft tokens.\n\nImportantly, this **soft token is not part of TRACE itself**, but serves as a lightweight augmentation signal for any downstream time series foundation model. Therefore, the **performance of TRACE itself is not affected by the learned soft token**. The purpose of soft tokens is to demonstrate that TRACE, as a retriever, can produce **semantically rich, task-relevant representations** to condition external models in retrieval-augmented settings. As shown in Table 2, stronger soft tokens provide richer semantic cues, leading to better predictions, particularly in models like Time-MoE and Timer-XL. This highlights TRACE’s utility as a plug-and-play retriever that delivers highly informative prompts for downstream time series models.\n\n> ### Question 3: How does TRACE prevent information leakage in textual description?\n\nYes. TRACE implemented specific safeguards during dataset construction and modeling to prevent any information leakage within the textual description as follows.\n- **Time Series Anchoring**: As described in Appendix B.1–B.2, each input time series is truncated to end exactly at the event start time (or the reference time for non-events). Forecasting targets lie strictly in the future, beyond the input window.\n\n- **No Use of Post-Event Text in Training**: While NOAA event reports may contain post-hoc narrative content, TRACE does **not** use these raw reports directly in retrieval or training. Instead, the only text used during alignment and RAG stages are:\n  - (1) **Channel-level descriptions** synthesized from the observed input window only, and  \n  - (2) **Sample-level context summaries**, similarly generated or aligned to pre-event data.\n\n- **Synthetic Description Control**: As detailed in Appendix B.4, LLM-generated descriptions are constrained to summarize only the observed (pre-event) portion of the time series. No future values are included in the generation prompt.\n\nThis setup ensures strict temporal alignment and rules out any leakage of ground truth labels or future signals into the input space. TRACE’s design adheres to the forecasting causality constraint throughout training and evaluation.\n\n\n### We thank the reviewer for constructive comments. We hope the above responses clarify your questions. We are more than happy to provide more explanations if needed."}, {"review_id": "q86JUDqoDR", "reviewer": "Reviewer_Lbrb", "summary": "The paper proposes TRACE, a multimodal retrieving framework designed for aligning and embedding time-series data with textual context. Recognizing that time-series data in real-world scenarios (e.g. healthcare, weather, and energy) frequently accompanies domain-specific textual information, TRACE aims to bridge this multimodal gap through semantically grounded embeddings. TRACE uses a two-stage training process: first pre-training a time-series encoder via masked reconstruction with specialized Channel Identity Tokens (CITs) and Channel-biased Attention, followed by fine-grained cross-modal alignment using dual-level hard negative mining. The model demonstrates two functions. One, as a standalone encoder offering state-of-the-art performance for forecasting and classification, and two, as a general-purpose retriever enabling retrieval-augmented generation (RAG) for enhancing downstream models. Extensive experiments confirm TRACE’s effectiveness across multiple retrieval and forecasting scenarios, surpassing several state-of-the-art methods.\n\nContributions:\n* Novel Multimodal Retriever between TS-Text: TRACE is a novel multimodal retriever explicitly designed to ground time-series embeddings with fine-grained textual context at both channel and sample levels.\n\n* Superior Performance: Extensive empirical validation demonstrates that TRACE consistently achieves state-of-the-art results. It outperforms existing SOTA models in cross-modal retrieval tasks (both text-to-time-series and vice versa). In retrieval-augmented generation setups, it boosts downstream model performance.\n\n* Hierarchical Hard Negative Mining: The paper introduces a unique hierarchical mining strategy that effectively discriminates between semantically similar but contextually divergent negatives, enhancing the semantic precision of embeddings.\n\n* Versatile Integration: TRACE’s architecture enables straightforward and efficient integration with existing time-series foundation models through retrieval-augmented generation, requiring minimal additional fine-tuning.", "questions": "Q1: Channel‑Identity Tokens scale linearly with the number of variables. How does training/inference memory behave if C >> 1000 (e.g. high‑frequency trading data)? Is it possible that CITs be compressed (hashing, shared prototypes) without losing retrieval fidelity?\n\nQ2: Please correct me if I am wrong. Rotary PE assumes a single global sampling interval. How would the encoder handle irregularly‑sampled or event‑driven series. Can RoPE be replaced by other position embedding techniques to address this issue?\n\nQ3: In Stage 1, the masked‑reconstruction loss predicts raw values (MSE). Would frequency‑domain or season‑trend decomposition losses produce richer features and better alignment downstream?\n\nQ4: Channel‑biased attention forbids CITs from attending to other channels. Have you tried soft gating instead of a hard mask so causal inter‑dependencies aren’t completely blocked?\n\nQ5: Instead of averaging retrieved CLS/text vectors, have you explored attention‑based fusion or set transformers so the model can weigh items by relevance?", "limitations": "The authors have partially addressed the limitations and potential negative societal impact of their work.\n\nLimitations\n* Quantitative Robustness to Noisy Texts: Missing explicit quantitative analysis demonstrating how performance degrades when textual labels are noisy or auto-generated.\n* Recommendation: Provide clear experiments quantifying robustness or degradation under noisy or missing text conditions in the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n* Clear Problem Definition and Proposed Solution: Paper identifies the gap that time-series foundation models are usually unimodal and NLP‑style RAG cannot ingest continuous signals, then supplies a two‑stage pipeline and an additional RAG stage that closes it.\n\n* Architectural & Training Novelty: Channel‑identity tokens + channel‑biased attention give each variable a fixed semantic slot. Proposed dual‑granularity InfoNCE aligns both pairs of CLS and report text, CIT and channel text. Soft‑prompt RAG injects retrieved multimodal context into frozen decoders with a small number of extra parameters.\n\n* Broad empirical coverage. Tests three tasks (retrieval, forecasting, classification) on four different domains and compares against both scratch networks and six public foundation models (Chronos, Timer‑XL, Time‑MoE, Moment, etc.).\n\n* Achieves substantially superior empirical performance compared to prior state-of-the-art: Retrieval precision significantly improved from best prior 55.73% to 90.08% P@1 for TS-to-Text and 64.67% to 89.63% for Text-to-TS. It has the smallest forecasting error (MAE and MSE) across datasets from different domains most of the time. By fine-tuning a pretrained model, TRACE makes classification accuracy raised notably by more than 10%, surpassing other prior methods.\n\nWeaknesses:\n* Synthetic or Overly Simple Channel Texts: The channel-level textual descriptions are either artificially generated or probably extremely short (sometimes just sensor IDs), reducing external validity. Real-world scenarios involving richer, natural-language texts might challenge the proposed dual-level alignment method.\n\n* Lack of Purpose-built Cross-modal Baselines: Some comparisons in the experiments are restricted to adapted models (e.g., Timer-XL, Moment) not originally designed for cross-modal alignment. Missing are purpose-built baselines (e.g., TimeRAF), potentially inflating reported retrieval performance gains.\n\n* Quadratic Complexity of Hard-negative Mining: The hierarchical negative mining strategy seems to scale quadratically with batch size (O(B²)), potentially becoming a computational bottleneck for large-batch training scenarios. No acceleration methods or approximations are discussed.\n\n* No Support for Irregular or Sparse TS Data: The Rotary Positional Encoding (RoPE) approach implicitly assumes regular global time steps, and there is no experimental validation provided for event-driven or sparsely sampled time series. It may limit TRACE's applicability.", "was_revised": true, "final_justification": "Thanks the author for the detailed rebuttal. Most of concerns have been answered or addressed. Raising the rating to Accept.\n\nBest, Reviewer Lbrb", "rebuttal": "Thanks for your constructive feedback. We have addressed your concerns as follows.\n\n\n>### W1: The channel-level descriptions may be overly simplistic, potentially limiting realism and their alignment quality.\n\nTo ensure the accuracy and usefulness of LLM-generated channel-level descriptions, we applied a carefully controlled and validated generation process. As detailed in Appendix B.4:\n- **Factual Grounding**: Each prompt includes raw time series data, and the model is instructed to summarize statistical properties (e.g., min/max, volatility) only from the observed input. This ensures the descriptions are grounded in actual trends, not speculative narratives.\n- **Prompt Structure and Constraints**: We use a strict prompt format that enforces concise, factual summaries, reducing the chance of hallucination.\n- **Qualitative and Empirical Validation**: TRACE achieves strong retrieval performance (Text-to-TS P@1 = 89.63%) and high cross-modal alignment in embedding space (Appendix D.3), demonstrating that the synthetic texts are semantically informative and align well with the time series.\n- **Scalability and Diversity**: As described in Appendix B.5, we generated over 74,000 such descriptions with balanced event/non-event coverage, confirming both scale and robustness.\n\n> ### W2: The experimental comparisons rely on adapted unimodal models rather than purpose-built cross-modal baselines like TimeRAF.\n\nWhile TimeRAF is a strong retrieval-augmented forecasting method, it is *not* a cross-modal model, and the method does not involve **text encoders** or **cross-modal contrastive alignment**. In contrast, TRACE is explicitly designed for **bidirectional and fine-grained** alignment between time series and text, with a dual-level (sample + channel) contrastive objective that supports both *text-to-TS* and *TS-to-text* retrieval. These tasks are outside the scope of TimeRAF and thus not directly comparable.\n\nTo further strengthen our evaluation, we implemented Multimodal PatchTST and Multimodal iTransformer from MM-TSlib and compared them against TRACE w Text on the Weather dataset:\n\n| Model| H=7 MAE ↓ | H=7 MSE ↓ | H=24 MAE ↓ | H=24 MSE ↓ |\n|--|--|---|--|---|\n| mm-PatchTST | 0.478| 0.589| 0.523| 0.636|\n| mm-iTransformer | **0.453**| 0.561| 0.508| 0.616|\n| TRACE w. Text | 0.458 | **0.557** |**0.500** | **0.609** |\n\nWe appreciate the suggestion and will explore adapting retrieval-augmented forecasters like TimeRAF into TRACE’s **multimodal** retrieval framework in future work.\n\n> ### W3: The hierarchical hard-negative mining strategy has quadratic complexity for batch size ($\\mathcal{O}(B^2)$), raising concerns about its scalability and computational cost. \n\nWhile the full similarity computation for hard-negative mining does scale as $\\mathcal{O}(B^2d)$ (sample-level) and $\\mathcal{O}(B^2C^2d)$ (channel-level), we mitigate the practical overhead in several ways:\n\n- **Efficient Hard-Negative Filtering**: As described in Appendix D.9.1, we use a hierarchical mining strategy that combines lightweight prototype filtering with partial sorting. This reduces the actual mining cost to $\\mathcal{O}(B\\log R + BC \\log R)$, which is negligible compared to full pairwise similarity computation and does not scale quadratically in practice.\n\n- **Empirical Efficiency**: Table 7 (Appendix D.9.2) shows that TRACE trains faster than larger baselines like FSCA and Moment (6.05s vs. 11.7s for Moment), while activating only **0.12M** parameters during fine-tuning. This confirms that our dual-level contrastive training, including hard-negative mining, is not a bottleneck at realistic batch sizes.\n\n- **Future Scalability Plans**: We plan to integrate scalable approximations for even larger settings, including:\n  - **Memory-bank-based mining** (e.g., MoCo-style gradient caching),\n  - **Approximate nearest neighbor (ANN)** search using tools like FAISS,\n  - **Stochastic sub-batch or masked-channel mining**, reducing sampling complexity to $\\mathcal{O}(B\\log B)$ or better.\n\nThese extensions can be integrated modularly into TRACE's framework without architectural changes.\n\n\n> ### W4 and Q2:  RoPE assumes regular sampling intervals. This may limit its applicability to irregularly sampled or event-driven time series. \n\nIt is correct that RoPE, in its standard form, assumes regularly sampled sequences. However, we would like to emphasize:\n\n- **Common Assumption in the Literature**: This is a shared limitation among most Transformer-based time series models (e.g., TSFormer, FSNet, Timer-XL), which use absolute or relative positional encodings tailored to uniformly sampled data. Our focus on regular time series aligns with the standard evaluation setting adopted in prior works, including those in TimeMMD and weather forecasting.\n\n- **Modular Design of TRACE**: TRACE is designed with modularity in mind. RoPE is not a fundamental dependency—it was selected due to its superior performance in structured, densely sampled domains (e.g., weather, energy). The architecture can be extended to support irregular or sparse data by swapping in alternative temporal encoding mechanisms.\n\n- **Future Extensions**: TRACE can readily incorporate time-aware mechanisms such as:\n  - Delta-timestamp attention bias (e.g., Informer, RAIM),\n  - Learned timestamp embeddings or time-to-vector encoders (e.g., Neural ODE-style Time2Vec),\n  - Adapted RoPE with relative phase shifts or interpolation strategies to handle sparse or non-uniform time steps.\n\nWe plan to explore these directions in future work to broaden TRACE’s applicability to irregularly sampled or event-driven settings, and appreciate the reviewer’s suggestion to highlight this in the final version.\n\n\n> ### Q1: How does training/inference memory behave if C >> 1000? \n\nWe thank the reviewer for the insightful question on scalability. While our CITs scale linearly with the number of channels, handling extreme-channel scenarios (C >> 1000) is a broader challenge for the entire time series field. Such inputs create ultra-long token sequences that exceed the capacity of current Transformer-based time-series models. We agree this is a crucial area for future research. Fortunately, TRACE's modular design is well-suited for future extensions to address this, such as by compressing channels via clustering or sharding them into manageable groups. We consider this a promising direction for future work.\n\n\n> ### Q2: Please refer to our response to Weakness 4\n\n> ### Q3: Would incorporating frequency-domain or season–trend decomposition losses help the downstream alignment?\n\nThis is an insightful question. Indeed, frequency-domain or season–trend-aware objectives can sometimes promote better modeling of temporal structures. However, we note the following:\n\nIn many models (e.g., PatchTST, DLinear), seasonal–trend decomposition (e.g., STL, Fourier transforms) is applied as a preprocessing step rather than as part of the loss function. This modularity allows for flexible architecture choices and avoids hard-coding inductive biases into training.\n\n- TRACE’s masked MSE loss operates directly on raw values and does not assume any specific temporal structure (e.g., periodicity or stationarity). This makes it more broadly applicable across diverse domains and datasets, especially where such assumptions may not hold (e.g., medical signals, financial trends).\n\n- As shown in Table 1 (retrieval) and Table 3 (classification), as well as the UMAP visualization in Appendix D.3, TRACE achieves strong representation quality and alignment performance with the current loss setup. The simplicity and generality of the masked MSE loss proved to be both stable and effective in our experiments.\n\nThat said, exploring season–trend-aware objectives as auxiliary tasks or plug-in modules could be a promising extension, and we thank the reviewer for suggesting this direction.\n\n\n> ### Q4: Would soft gating allow better modeling of causal interdependencies?\n\nWe thank the reviewer for this observation. As correctly noted, our current design employs a **hard attention mask** where each **Channel Identity Token (CIT)** is restricted to attend only to patch tokens within its own channel. This enforces **channel-specific abstraction**, which is beneficial for alignment and interpretability.\n\nHowever, the reverse is not restricted—**patch tokens are allowed to attend to all CITs**. This design enables **implicit cross-channel interactions** through shared CITs, allowing the model to propagate interdependencies while maintaining a disentangled structure. We agree that introducing **soft gating mechanisms**, such as learnable attention masks, cross-channel adapters, or attention dropout, could allow more flexible modeling of causal relationships across channels without fully discarding the benefits of CIT-driven modularity.\n\nWe consider this an avenue for future work and thank the reviewer for the constructive suggestion.\n\n\n\n> ### Q5: Have you considered using attention-based fusion to better weigh relevance among retrieved items?\n\nWe appreciate the reviewer’s suggestion. While more expressive fusion mechanisms (e.g., Set Transformers, gated attention, or cross-attention pooling) may give certain benefits, we chose **mean pooling** for its simplicity, training stability, and computational efficiency. This choice aligns with our paper’s core focus on **retrieval and alignment quality**, rather than optimizing the fusion architecture.\n\nImportantly, TRACE is designed to be **modular**. The aggregated embedding (i.e., soft prompt) is decoupled from the encoder and can be easily replaced with more advanced fusion modules, such as attention-based pooling or set-wise encoders, **without retraining** the time-series or text encoders. We consider exploring such plug-and-play fusion enhancements a promising direction for future work."}, {"review_id": "3Pnk2FtUPb", "reviewer": "Reviewer_JN3H", "summary": "The author proposes a new multimodal retrieval system **TRACE**, which implements `Channel Identity Tokens (CIT)` within a masked autoencoder framework to capture multivariate time series patterns while maintaining channel disentanglement and also implementing hierarchical hard negative mining at both channel and sample levels to employ cross-modal alignment. This work addressed the critical gap between time series analysis and their semantic grounding. The key technical contributions of the paper are:\n1. The framework introduces a comprehensive system for cross-modal retrieval between time series and text. \n2. The framework introduces RAG capabilities for time series foundation models through soft token interfaces\n\nThe paper provides a comprehensive evaluation across customer weather datasets, TimeMMD benchmark subsets, multiple baseline comparisons against both traditional time series models and recent foundational models. The inclusion of extensive ablation studies that validate architectural choices and hyperparameter sensitivity instills confidence in the system's design and performance.", "questions": "1. What is the optimal ratio between the flattened sequence length `L` and the original temporal length `T` for different types of time series data? How does this ratio affect model performance, and are there recommended guidelines for practitioners? Do you observe performance degradation when `L >>T` (over-segmentation) or when `L <<T` (under-segmentation)?\n\n\n2. In `Figure 4`, Why are Relative PE results missing at `d=768`?\\\n       a. Were the results excluded due to poor performance?\\\n       b. Were there implementation difficulties? \n\n3. How is `CLS` token initialized and learned to aggregate global information?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. TRACE can be used for downstream tasks (forecasting and classification) in both scenarios - external data is available (RAG mode), or no external database exists (standalone decoder mode). This dual functionality implies that the learned representations are robust and can be helpful in multiple deployment scenarios. \n2. TRACE allows frozen base models to incorporate retrieved context without architectural modifications. \n3. The learned multimodal embedding space is already aligned with textual descriptions reducing the dependency on additional textual signals. \n4. TRACE demonstrates superior performance across multiple evaluation dimensions: Cross-modal retrieval, RAG enhancement, Standalone", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback. We have addressed your concerns as follows.\n\n> ### Question 1: What is the optimal ratio between $L$ and $T$ for different types of time series? How does this ratio affect performance, and are there any practitioner guidelines?\n\nThe ratio between the flattened token sequence length $L$ and the original temporal length $T$ is controlled by the patch length $P$ (i.e., $L = T/P$). As shown in **Figure 4**, we conduct a detailed ablation over $P$ and observe that **mid-range patch lengths ($P = 6–8$)** consistently yield the best performance across tasks. This corresponds to a **flattened ratio $L/T ≈ 1/6$ to $1/8$**, though the exact value depends on the number of channels and sequence length.\n\n**Key observations:**\n\n- **Over-segmentation ($L$ ≫ $T$ / very small $P$)**: Leads to overly fine-grained tokenization, exposing the model to high-frequency noise and increasing computational cost. This often results in overfitting and unstable training.\n  \n- **Under-segmentation ($L$ ≪ $T$ / very large $P$)**: Causes excessive compression and loss of temporal detail, harming both reconstruction quality and fine-grained alignment.\n\nWe recommend tuning $P$ based on the **dominant temporal resolution** of the dataset (e.g., shorter $P$ for hourly signals, longer $P$ for daily/monthly). Moderate patching offers a good trade-off between **local detail and global context**, and we found it to be robust across diverse domains (weather, energy, health).\n\n>### Question 2: In Figure 4, why are the results for Relative Positional Encoding (PE) missing at embedding dimension d = 768?  Were they excluded due to poor performance?  Were there implementation difficulties?\n\nThank you for pointing this out. The omission of Relative PE results at d = 768 was due solely to space and figure clarity constraints, not performance concerns or technical limitations. They show **slight improvement over Relative PE at $d = 384$ but still underperform Rotary PE**, which remains our default due to its consistent superiority across tasks.\n\nWe appreciate the reviewer’s attention to detail and will make sure these results are explicitly presented in the final version.\n\n> ### Question 3: How is the [CLS] token initialized and trained to capture global information in TRACE?\n\nWe follow standard practice and initialize the [CLS] token using `nn.init.normal_`, with parameters learned during pretraining. The [CLS] token is fully trainable and optimized end-to-end as part of the masked reconstruction and contrastive objectives.\n\nImportantly, during encoding, **the [CLS] token is allowed to attend to the entire flattened sequence**, including all channel identity tokens (CITs), which summarize per-channel information, and all patch tokens, which encode localized temporal patterns. This design enables the [CLS] token to act as a **global aggregator**, capturing context across time and variables. Its resulting embedding is used for sample-level alignment, retrieval, and forecast conditioning."}, {"review_id": "sAUdXAm0HT", "reviewer": "Reviewer_aNW1", "summary": "The paper proposes a multimodal framework that aligns time series data with corresponding textual context for cross-modal retrieval, forecasting, and classification. It designs tokens during the encoder pretraining stage amortized for downstream tasks. It also introduces a dual-level alignment strategy using channel identity tokens and contrastive learning at both the sample and channel levels. As such, it supports Text-to-TS and TS-to-Text cross-modal retrieval and also acts as a plug-and-play module for retrieval-augmented forecasting and classification, where the efficacy is validated in the experiments.", "questions": "1) Please refer to weaknesses 1 and 2:\n\n1-1) The assumption regarding highly aligned time series - text pairs, is a bit too strong in real-world time series contexts. Do the authors consider any countermeasures to mitigate this issue?\n\n1-2) Could the author provide more evaluations regarding the aforementioned baselines that shares similar idea in terms of cross-modal retrieval? \n\n2) For the evaluation in Table 1, the encoder of TRACE is replaced by other foundation models with frozen parameters, demonstrating significant performance gain. Did the authors consider other moderated time series encoder, such as PatchTST, iTransformer during the pre-training stage?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed framework formulates and tackles cross-modal retrieval between time series and texts, which enables multiple downstream tasks. The experiments with ablations demonstrate its efficacy. \n\n2. The idea of dual-level contrastive learning for cross-modal alignment, including the design of hard negative mining is well-motivated and somewhat novel, in terms of enabling fine-grained semantic retrieval between time series and texts.\n\n\n\nWeaknesses:\n\n\n1. The availability of aligned time series and textual data (including not only global descriptions but also channel-specific descriptions) is assumed, which limits its deployment in real-world scenarios such as finance where texts and the stock prices of finance entities are not highly correlated, and industrial sensors .\n\n2. This paper evaluates its method and baselines on multiple multi-modal time series datasets. However, it lacks discussions and baseline comparisons for a few methods with multi-modal modeling in forecasting and classification subtasks. For example, the TimeCMA [1] method also proposes channel-wise similarity retrieval in cross-modality alignment that facilitates forecasting performance. \n\n3. The design of channel-biased attention and RoPE for channel dependencies encoding is not easy to read, and it is somewhat not well-motivated in terms of multi-modal modeling.  \n\n[1] TimeCMA: Towards LLM-Empowered Multivariate Time Series Forecasting via Cross-Modality Alignment", "was_revised": true, "final_justification": "Some of my concerns are resolved during the rebuttal. The discussion of related methods and the clarity needs improvements.", "rebuttal": "Thanks for your constructive feedback. We have addressed your concerns as follows.\n\n> ### Weakness 1: The availability of aligned time series and textual data may limit its deployment in real-world scenarios.\n\nWhile perfectly aligned multimodal datasets may require additional effort, such data are increasingly accessible in practice recently. For instance, the TimeMMD [2] already provides multi-domain, multimodal time series data across Health, Energy, and Environment sectors. Furthermore, many time series datasets (e.g., weather, finance, traffic) can be enriched with textual context through simple and scalable methods, including web scraping, report parsing, or LLM-based description generation (as we did in our curated NOAA dataset).  Therefore, multimodal alignment is practical with modest preprocessing (We refer to Appendix B for our detailed preprocessing steps on weather dataset).\n\nIn addition, we note the growing trend in both academia and industry towards integrating language and time series for improved performance in forecasting and reasoning tasks. As the benefits of multimodal modeling become more evident, we expect more such datasets to emerge. TRACE offers a general and extensible framework that not only works with existing aligned datasets but also encourages scalable dataset construction and integration of textual context into time series modeling.\n\n\n\n> ### Weakness 2 and Question 1.2: The paper omits comparison with multimodal methods like TimeCMA, which proposes channel-wise retrieval for cross-modal alignment and improved forecasting. \n\nThe core objectives, architectures, and applications of TRACE and TimeCMA are fundamentally different. Specifically, TimeCMA's reliance on LLM-generated synthetic prompts for multivariate time series forecasting presents a key limitation: the lack of genuine alignment between different modalities. TimeCMA generates its own textual modality by wrapping numerical data in a prompt. This means the \"textual\" information isn't independently sourced or inherently meaningful in the same way an actual weather narrative would be. TimeCMA is inherently built for forecasting and classification tasks. Achieving cross-modal retrieval with TimeCMA would require substantial model modifications, making direct comparisons to dedicated retrieval models challenging.  We will add a more detailed discussion on TimeCMA as relevant work in our revision.\n\n\n\n> ### Weakness 3: The design of channel-biased attention and RoPE for modeling channel dependencies is not easy to read and not clearly motivated in the context of multimodal modeling.\n\nDue to space constraints, we used compact in-line formulations for both the channel-biased attention mask and RoPE integration. We acknowledge this and will provide detailed formulation and motivational context in our revision.\n\nHere, we elaborate on the motivation behind both components. Our goal is to build *interpretable and modular representations*  by encouraging each Channel Identity Token (CIT) to first abstract information from its own channel (via masked attention to intra-channel patch tokens). This ensures that each CIT captures localized semantics independently, which is crucial in multimodal alignment where channel-level descriptions may map to different textual spans.  The global CLS token, which is allowed to attend across all tokens, then aggregates these per-channel abstractions to enable inter-channel reasoning. This staged structure balances local disentanglement (for channel grounding) and global integration (for sample-level alignment and retrieval).\n\nRoPE injects relative positional information while preserving sequence length generalization, and we apply it **independently within each channel**. This ensures that each CIT benefits from phase-aware encoding of local temporal dynamics, which is especially important when channel-level trends vary (e.g., temperature vs. humidity). Unlike absolute or learned position embeddings, RoPE is scale-invariant and better suited for cross-modal alignment across variable-length sequences.\n\nWe will better communicate these motivations to aid readability in our revision.\n\n\n> ### Question 1.1: The assumption regarding highly aligned time series - text pairs, is a bit too strong in real-world time series contexts. \n\nWhile full alignment is ideal, TRACE is designed to be robust in semi- or weakly-aligned settings, and we already implement several countermeasures:\n- Synthetic descriptions from raw TS: In cases where aligned text is unavailable, channel-level descriptions can be automatically generated using LLMs with structured prompts based on raw statistics (e.g., min/max/mean/volatility). This is how we constructed our NOAA dataset (see Appendix B.4).\n- Optional text conditioning: TRACE can operate in TS-only mode during inference, or use retrieved text from similar samples, even if not perfectly aligned. This makes it applicable to domains like finance or industry where explicit text pairs are sparse.\n- Flexible supervision: Our contrastive training allows text–TS alignment even with partial or noisy matches, encouraging shared semantics without requiring perfect ground-truth pairing.\n\n\n> ### Question 2: Did the authors consider other moderated time series encoders in Table 1?\n\nYes, we did explore alternative time series encoders like PatchTST and iTransformer during the pre-training stage, and our architectural design in TRACE is, in fact, informed by their respective strengths. Specifically, given a multivariate time series input of shape $[N, T]$, where $N$ is the number of channels and $T$ is the temporal length, PatchTST applies token-wise attention across time (i.e., $T \\times T$), treating each channel independently. In contrast, iTransformer focuses on channel-wise attention ($N \\times N$), but ignores temporal locality. Both approaches are partially expressive. Instead, TRACE simultaneously attends over channel identity tokens (CITs) and temporal tokens, which is a practice increasingly standard in time-series foundation models.\n\nThis architectural choice is empirically validated. Our early experiments demonstrate that token-only variants (akin to PatchTST) or channel-only variants (akin to iTransformer) struggle to learn discriminative representations for downstream tasks (also shown in Tables 2 and 3). Moreover, **dual-level cross-modal alignment consistently fails** under these alternatives, as the resulting encoders cannot generate sufficiently distinct channel-level and sample-level embeddings, which are critical for our contrastive alignment framework. We will add these discussions and experimental results to the appendix in our final version.\n\n### We thank the reviewer for constructive comments. We hope the above responses clarify your questions. We are more than happy to provide more explanations if needed."}]

Author rebuttal:

Meta-review: TRACE proposes a multimodal retriever that grounds time-series representations in aligned text and supports both text-to-TS and TS-to-text retrieval. It appears the method also works as a context-aware encoder and a plug-and-play RAG module, using channel-level alignment and hard-negative mining to boost forecasting and classification across domains. Reviewers say the idea is novel and the results are strong across retrieval, forecasting, and classification, with a useful curated weather dataset. It seems concerns on data alignment, missing baselines, and efficiency were addressed in rebuttal, and all reviewers end up recommending acceptance.

I recommend accept. All reviewers recommend accept and raise or keep positive scores after rebuttal, and the paper seems technically solid with clear novelty and broad empirical gains. The remaining concerns on realism of texts and scalability seem manageable and are acknowledged with concrete plans. The contribution seems timely and likely impactful for multimodal time-series learning and RAG.
Decision: Accept (poster)