From Values to Tokens: An LLM-Driven Framework for Context-aware Time Series Forecasting via Symbolic Discretization

## Abstract

Abstract Time series forecasting plays a vital role in supporting decision-making across a wide range of critical applications, including energy, healthcare, and finance. Despite recent advances, forecasting accuracy remains limited due to the challenge of integrating historical numerical sequences with contextual features, which often comprise unstructured textual data. To address this challenge, we propose TokenCast, an LLM-driven framework that leverages language-based symbolic representations as a unified intermediary for context-aware time series forecasting. Specifically, TokenCast employs a discrete tokenizer to transform continuous numerical sequences into temporal tokens, enabling structural alignment with language-based inputs. To bridge the semantic gap between modalities, both temporal and contextual tokens are embedded into a shared representation space via a pre-trained large language model (LLM), further optimized with autoregressive generative objectives. Building upon this unified semantic space, the aligned LLM is subsequently fine-tuned in a supervised manner to predict future temporal tokens, which are then decoded back into the original numerical space. Extensive experiments on diverse real-world datasets enriched with contextual features demonstrate the effectiveness and generalizability of TokenCast. The code is available 1 1 1 https://github.com/Xiaoyu-Tao/TokenCast .

## Introduction

Time series forecasting (TSF) is critical for decision-making in domains such as energy (Jin et al. 2024), healthcare (Qiu et al. 2024), and finance (Feng et al. 2019). The goal is to predict future values based on historical observations and associated contextual features. In practice, accurate forecasting requires not only modeling temporal dependencies in numerical sequences, but also understanding how they interact with external contextual factors—such as static attributes or dynamic events (Liu et al. 2024b). Fundamentally, TSF can be viewed as learning a mapping from past values and contextual features to future outcomes (Liu et al. 2024e).

Figure: Figure 1: Multimodal modeling integrates time series with contextual features, using a shared cross-modality representation to fuse the modalities.
Refer to caption: x1.png

To learn this mapping, researchers have proposed a wide range of methods, ranging from classical statistical models to modern data-driven approaches. Traditional methods, such as ARIMA (Hyndman and Khandakar 2008) and state-space models (Winters 1960), rely on strong assumptions about data generation and often incorporate domain-specific priors. In contrast, data-driven approaches such as deep learning models aim to learn patterns directly from data without handcrafted assumptions. Architectures based on RNNs (Lai et al. 2018), CNNs (Cheng et al. 2025b), Transformers (Zhou et al. 2022), and MLPs (Challu et al. 2023) have been widely adopted, each capturing different aspects of temporal dependencies. However, most of these models assume homogeneous numerical inputs and struggle to incorporate complex contextual features, particularly those with heterogeneous modalities.

Beyond capturing temporal dependencies, there is a growing emphasis in recent research on incorporating contextual features to enhance forecasting performance (Liu et al. 2024a; Williams et al. 2024; Liu et al. 2024b). These features typically fall into two categories: dynamic exogenous variables (e.g., weather conditions) and static attributes (e.g., product types, patient demographics, market segments). When contextual features share the same numerical modality as the target series, they can be directly modeled as additional input channels. However, many high-value contextual features—such as clinical notes, policy texts, or user logs—are expressed in unstructured textual form. This heterogeneity poses significant challenges for aligning and integrating information across modalities.

To address these challenges, some studies have explored shallow fusion strategies to incorporate contextual features. Models such as DeepAR (Salinas et al. 2020) and Temporal Fusion Transformer (TFT) (Lim et al. 2021) typically concatenate external variables with time series or introduce gating mechanisms. While offering basic integration, these methods often rely on weak alignment and struggle to capture deep semantic interactions across modalities (Huang et al. 2025). More recently, LLMs have been introduced into time series forecasting (Liu et al. 2024c; Sun et al. 2023; Cheng et al. 2024). Methods like Time-LLM (Jin et al. 2023) inject time series features into LLMs using linear adapters (Fig. [1](https://arxiv.org/html/2508.09191v1#Sx1.F1)(a)) or soft prompts (Fig. [1](https://arxiv.org/html/2508.09191v1#Sx1.F1)(b)). Although promising, these approaches fall short in resolving the structural discrepancies between numerical sequences and unstructured context. Moreover, they fail to fully leverage the generative and reasoning capabilities of LLMs, which are pretrained on large-scale corpora. This observation raises a fundamental question: Can time series be effectively modeled in a discrete token space to unlock the full potential of LLMs?

Motivated by this question, we explore a more expressive yet under-explored paradigm that formulates time series forecasting as a multimodal discrete context understanding and generation problem, powered by pre-trained LLMs, as illustrated in Fig. [1](https://arxiv.org/html/2508.09191v1#Sx1.F1)(c). The key idea is to transform continuous numerical sequences into discrete tokens and embed them into the same semantic space as contextual language inputs. This formulation enables the full use of LLMs’ capabilities in semantic understanding, contextual reasoning, and autoregressive generation. However, this paradigm introduces several non-trivial challenges. First, discretizing dynamic time series is more difficult than compressing static data, as it requires preserving temporal dependencies while reducing granularity. Second, even with symbolic representations, semantic misalignment between temporal tokens and contextual features may hinder effective fusion. Finally, it remains unclear whether time series forecasting can be effectively addressed through autoregressive generation over discrete tokens—a direction still largely unexplored.

Based on the above analysis, we propose TokenCast, an LLM-driven framework for context-aware time series forecasting via symbolic discretization. TokenCast begins with a time series tokenizer that converts continuous sequences into temporal tokens, mitigating structural discrepancies across data modalities. To bridge the semantic gap, temporal and contextual tokens are jointly embedded into a shared representation space using a pre-trained LLM, optimized via an autoregressive objective while keeping the backbone frozen and tuning only the embedding layer. Building on this unified semantic space, the aligned LLM is further fine-tuned with supervised forecasting signals to enhance predictive performance. We evaluate TokenCast on diverse real-world datasets enriched with contextual features. Experimental results show that TokenCast achieves strong accuracy and generalization across domains. We also conduct comprehensive ablation and qualitative studies, offering insights into the flexibility of symbolic, LLM-based time series forecasting.

## Related Work

Time series forecasting (TSF) is a fundamental task across various domains.
Traditional approaches typically rely on statistical assumptions such as stationarity and linearity, and often depend on handcrafted assumptions that limit their flexibility (Holt 2004; Kalekar et al. 2004).
Alternatively, data-driven methods (Chen and Guestrin 2016), particularly those based on deep learning, have advanced TSF by learning temporal patterns directly from data. RNN-based models (Wang et al. 2019) capture dependencies through recurrence, CNN-based models (Wang et al. 2023) enhance local pattern extraction, and Transformer-based architectures (Tang and Zhang 2025) are well-suited for modeling long-range interactions. Furthermore, MLP-based approaches (Wang et al. 2024c) demonstrate that simple architectures can achieve competitive performance with improved computational efficiency.
These models mainly focus on numerical data, with less emphasis on unstructured context.

In addition to modeling temporal dependencies, recent research increasingly emphasizes the integration of contextual features for accurate forecasting (Chang, Peng, and Chen 2023; Liu et al. 2024d; Zhao et al. 2025). Two major lines of research have emerged in this direction. One line of research focuses on deep learning architectures that explicitly model feature interactions (Gasthaus et al. 2019).
For example, TimeXer (Wang et al. 2024d) employs cross-attention mechanisms to fuse dynamic and static modalities.
Another line of research leverages pre-trained LLMs for multimodal modeling (Cheng et al. 2025a; Liu et al. 2025).
Some approaches, such as TEMPO (Cao et al. 2023), utilize linear adapters to project time series features into the LLM’s semantic space. Others, like Promptcast (Wang et al. 2024b), employ soft prompts to guide the frozen LLM’s behavior.
However, these promising approaches fail to bridge the structural gap between numerical and textual modalities.

## Methodology

In this section, we present the formal problem definition, clarify the key concepts and notations used throughout the paper, and provide an overview of the TokenCast. We also detail the implementation process.

Figure: Figure 2: Overview of the TokenCast for context-aware time series forecasting: (a) time series tokenizer to address the structural differences between modalities, (b) cross-modality alignment with an autoregressive objective to bridge the semantic gap, and (c) generative fine-tuning and context-aware forecasting through time series decoding for horizon prediction.
Refer to caption: x2.png

### Problem Formulation

We consider a dataset $\mathcal{D}=\{(X_{i},T_{i},P_{i})\}_{i=1}^{N}$ of $N$ multimodal time series instances. For each instance, $X\in\mathbb{R}^{L\times C}$ represents the multivariate time series over $L$ time steps and $C$ channels, $T$ denotes the contextual features, and $P\in\mathbb{R}^{L_{P}\times C}$ is the ground-truth future sequence over a horizon $L_{P}$. The contextual features $T$ are tokenized to tokens $Y$ using the tokenizer of a pre-trained LLM, while the time series $X$ is converted into discrete tokens $Z_{q}$ via a learnable mapping $f_{\theta}:X\mapsto Z_{q}$. These two token sequences are then concatenated to form a token sequence $Z=[Z_{q};Y]\in\mathcal{V}^{T^{\prime}}$.
We use boundary markers to delimit the temporal tokens of $\hat{Z}$. Finally, a decoding function $g_{\phi}:\hat{Z}\mapsto\hat{P}$ is applied to reconstruct the raw time series $\hat{P}\in\mathbb{R}^{L_{P}\times C}$.

### Framework Overview

Fig. [2](https://arxiv.org/html/2508.09191v1#Sx3.F2) illustrates the overview of our TokenCast, which consists of three main stages. The process begins with the time series tokenizer, which transforms continuous time series into a sequence of discrete tokens via a dynamical and decoupled vector quantization tokenizer. Subsequently, both the temporal and contextual tokens are then jointly processed by a pre-trained LLM, which performs cross-modality alignment under autoregressive objectives. Following this alignment, the aligned LLM is adapted to the forecasting task via generative fine-tuning, enabling token prediction. The predicted tokens are decoded to raw time series using a frozen time series de-tokenizer. The following sections elaborate on the principal stages of the TokenCast.

### Discretized Representation

#### Time Series Tokenizer.

The fundamental structural differences between time series and contextual features pose significant challenges to multimodal modeling. In this work, we decide to pursue a simple yet effective solution—time series discretization. We note that existing work has explored this direction. For example, some methods employ efficient numerical binning strategies (Ansari et al. 2024). However, such discretization is irreversible, thereby hindering the reconstruction of the original time series. Alternatively, while VQ-VAE-based (Feng et al. 2025; Rasul et al. 2022, 2024) approaches enable learnable encoding and reconstruction, they are optimized for compression, resulting in symbolic representations that lack awareness of temporal dynamics.

Based on the above analysis, we design a dynamic and decoupled tokenizer to perform time series discretization. To achieve dynamic modeling, we encode both historical and predicted time series within a shared latent space, allowing the tokenizer to learn temporal dynamics rather than static representations.
We recognize that time series often suffer from distribution shifts due to non-stationary characteristics. To mitigate this, we introduce a history-based reversible normalization strategy that decouples temporal dynamics from static values, enabling stable modeling of non-stationary time series without future information leakage.

As illustrated in Fig. [2](https://arxiv.org/html/2508.09191v1#Sx3.F2)(a), the tokenizer consists of a history-based reversible instance normalization layer, a causal temporal convolutional network (TCN) encoder, a vector quantization layer, and a Transformer-based decoder. The core mechanism begins by computing the mean and standard deviation from the historical time series and applying them to normalize the input sequence.
Formally, our tokenizer processes a multivariate time series $X=[H;P]\in\mathbb{R}^{L\times C}$, where $H\in\mathbb{R}^{L_{H}\times C}$ and $P\in\mathbb{R}^{L_{P}\times C}$ denote the historical and predicted time series, respectively. The process begins with a history-based reversible instance normalization (RIN) layer. We compute the mean $\mu(H)$ and standard deviation $\sigma(H)$ solely from the historical time series $H$, and use them to normalize the time series $X$. These statistics are retained for inverse transformation during decoding.

The normalized time series is then passed through a causal temporal convolutional network (TCN) encoder $f_{\text{enc}}$, yielding a sequence of continuous latent representations $Z=f_{\text{enc}}(X)\in\mathbb{R}^{T\times d}$, where $T$ is the number of latent vectors and $d$ is the feature dimension. To discretize the latent representations, we apply a vector quantization layer. For domain $i$, a learnable codebook $C_{i}=\{e_{i,k}\}_{k=1}^{K}\subset\mathbb{R}^{d}$ is maintained, containing $K$ embedding vectors. Each latent vector $z_{t}\in\mathbb{R}^{d}$ is mapped to its nearest neighbor in the codebook as $z_{t}^{q}=e_{i,k^{*}}$, where $k^{*}=\arg\min_{k}\|z_{t}-e_{i,k}\|_{2}^{2}$. The output of this layer is a quantized sequence $Z_{q}=(z_{1}^{q},\dots,z_{T}^{q})$, and the corresponding sequence of indices $\{k^{*}\}$ serves as the discrete tokens for downstream modeling. These tokens are subsequently decoded by a causal Transformer decoder $f_{\text{dec}}$, and the final reconstruction $\hat{X}$ is obtained by applying the inverse RIN operation using the stored statistics $\mu(H)$ and $\sigma(H)$, i.e., $\hat{X}=f_{\text{denorm}}(f_{\text{dec}}(Z_{q}))$.

#### Training Objective.

The tokenizer is optimized by minimizing the objective function defined as follows:

$$ $\mathcal{L}=\mathcal{L}_{\text{recon}}+\beta\left(\mathcal{L}_{\text{commit}}+\mathcal{L}_{\text{codebook}}\right)+\gamma\mathcal{L}_{\text{diversity}},$ (1) $$

where $\mathcal{L}_{\text{recon}}=\|\hat{X}-X\|_{2}^{2}$ is the reconstruction loss that optimizes both the encoder and decoder. Due to the non-differentiability of the $\arg\min$ operation in quantization, we employ the straight-through estimator (STE) during backpropagation.
To train the vector quantizer, we include:
$\mathcal{L}_{\text{codebook}}=\|\text{sg}[Z]-Z_{q}\|_{2}^{2}$,
$\mathcal{L}_{\text{commit}}=\|Z-\text{sg}[Z_{q}]\|_{2}^{2}$,
where $\text{sg}[\cdot]$ denotes the stop-gradient operator, which prevents gradients from flowing into its argument during backpropagation.
To avoid codebook collapse and promote diverse usage of codebook entries, we add a diversity loss $\mathcal{L}_{\text{diversity}}$, which encourages the embeddings in $C_{i}$ to be utilized more uniformly. We set $\beta=0.25$, $\gamma=0.25$ by default.

### Representation Modeling

#### LLM Backbone.

Following the discretization of time series into discrete tokens, the next challenge is to model the complex dependencies embedded in these sequences. While architectures like TCNs or Transformers can be trained from scratch, we argue that a pre-trained LLM serves as a more effective backbone. This is supported by two observations:
(1) pre-trained LLMs possess strong semantic understanding and contextual reasoning capabilities acquired from large-scale corpora, and (2) the structure of discrete time series tokens closely resembles that of language tokens (Zhao et al. 2023). By casting forecasting as a generative task, we directly leverage the LLM’s autoregressive generation ability.
To guide LLM reasoning and incorporate contextual features, we employ a structured prompt template, as shown in Fig. [2](https://arxiv.org/html/2508.09191v1#Sx3.F2)(b). This prompt template consists of four essential components: domain knowledge, task instructions, statistical properties, and discrete time series tokens. This design ensures token-level consistency with language tokens and introduces task-specific descriptions alongside context-aware statistical attributes, enabling the LLM to perform instruction-driven generation.

#### Cross-Modality Alignment.

While discretization aligns time series structurally with language tokens, a semantic gap remains between time series and contextual features. Existing methods often introduce projection modules (e.g., MLPs) to map time series into the LLM’s latent space for fusion (Liu et al. 2025). Although effective in downstream tasks, these strategies rely on external transformation modules for alignment, which bypass the language model’s native vocabulary modeling mechanism.
To this end, we implement a more direct vocabulary-level alignment strategy. As illustrated in Fig.  [2](https://arxiv.org/html/2508.09191v1#Sx3.F2)(b), we construct a unified vocabulary by directly appending $K$ temporal tokens and $S$ task-specific special tokens to the original vocabulary $V_{\text{orig}}$ of the pre-trained LLM, forming an extended vocabulary $V$. Correspondingly, a shared embedding matrix $E\in\mathbb{R}^{|V|\times d}$ is used to encode all tokens, regardless of their modality origin.
This unified embedding mechanism enables seamless fusion of time series and contextual features while maintaining alignment with the pre-trained model.

To ensure distributional alignment with pre-trained embeddings for stable fine-tuning, the embedding of the newly introduced time series tokens is initialized by sampling from a multivariate gaussian distribution defined by the mean $\mu$ and covariance $\Sigma$ of the original word embeddings. During prompt construction, temporal tokens $Z_{q}$ and contextual tokens $Y$ are concatenated at the token level and jointly transformed into embeddings via the shared embedding layer:
$\text{Embed}([Z_{q},Y])=E([z_{1},\dots,z_{n},y_{1},\dots,y_{m}]),$
where $E$ denotes the unified embedding matrix. This unified embedding process enables the LLM to reason over concatenated sequences without requiring architectural modification.

To optimize cross-modality token representations within the shared embedding space, we adopt an autoregressive training objective. Specifically, we freeze all parameters of the pre-trained LLM and update only the shared embedding matrix $E$, which is responsible for encoding both temporal and contextual tokens. Given a concatenated token sequence $[Z_{q},Y]$, the training objective is formulated as a next-token prediction task over the combined sequence:

$$ $\mathcal{L}_{\text{align}}=-\sum_{t=1}^{T}\log p(z_{t}\mid z_{1},\dots,z_{t-1};E),$ (2) $$

where $z_{t}\in V$ denotes the $t$-th token in the sequence, and $p(\cdot)$ is the conditional probability predicted by the frozen language model given the embedding vectors from $E$.

#### Generative Fine-tuning.

We now detail the procedure for adapting the aligned LLM for forecasting tasks. As illustrated in Fig.  [2](https://arxiv.org/html/2508.09191v1#Sx3.F2)(c), we employ a generative fine-tuning strategy to specialize the model for context-aware time series forecasting. This process consists of two primary stages: (1) structured prompt-based generative fine-tuning; and (2) multimodal forecasting with token-based decoding.
In the first stage, prompt-based generative fine-tuning is designed to teach the aligned LLM to perform complex reasoning and forecasting based on contextual features and time series. The model is then fine-tuned using an autoregressive objective, where it learns to generate a structured response containing both natural language analysis and the sequence of future time series tokens. This phase is crucial as it tailors the model’s autoregressive generative capabilities

In the second stage, the fine-tuned model is utilized for context-aware forecasting and decoding. During inference, the model receives a prompt with historical data and contextual features, and autoregressively generates a complete response. The key component of this generated output is the sequence of discrete tokens, which represents the model’s prediction of future time series values. To translate this symbolic representation back into a continuous predicted time series, these tokens are processed by a frozen time series de-tokenizer. We use boundary markers to explicitly delimit the temporal tokens within the generated sequence. This procedure leverages the LLM’s reasoning and in-context learning capabilities, enabling reliable forecasting grounded in the semantic and statistical features provided in the prompt.

**Table 1: All reported results are average over four horizons and three trials on various context-rich benchmark datasets. Lower values indicate better performance. The best results are highlighted in bold, and the second-best are underlined.**
| Model | TokenCast | Time-LLM | GPT4TS | TimeDART | SimMTM | Crossformer | Autoformer | DLinear |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| Economic | 68.911 | 1.701 | 81.542 | 1.760 | 85.947 | 1.716 | 86.029 | 1.771 | 90.351 | 1.672 | 406.418 | 4.074 | 116.745 | 2.088 | 122.216 | 2.070 |
| Health | 2.525 | 0.081 | 2.823 | 0.104 | 2.565 | 0.083 | 2.623 | 0.088 | 2.720 | 0.088 | 1644.745 | 2.504 | 2.617 | 0.265 | 28.587 | 0.455 |
| Web | 497.410 | 1.246 | 557.833 | 1.751 | 540.492 | 1.458 | 773.635 | 1.369 | 847.649 | 1.327 | 698.316 | 1.963 | 722.506 | 3.303 | 632.301 | 1.398 |
| Stock-NY | 0.482 | 0.455 | 0.662 | 0.510 | 0.638 | 0.502 | 0.776 | 0.606 | 0.613 | 0.585 | 1.111 | 0.912 | 0.676 | 0.573 | 0.999 | 0.754 |
| Stock-NA | 1.134 | 0.780 | 1.200 | 0.925 | 1.272 | 0.880 | 1.409 | 0.883 | 1.343 | 0.834 | 1.913 | 1.053 | 1.558 | 0.914 | 1.710 | 0.958 |
| Nature | 0.269 | 0.297 | 0.258 | 0.283 | 0.274 | 0.299 | 0.243 | 0.273 | 0.259 | 0.286 | 0.735 | 0.511 | 0.508 | 0.481 | 0.369 | 0.436 |
| 1^st Count | 5 | 5 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

## Experiments

In this section, we conduct comprehensive experiments to evaluate our TokenCast’s performance on diverse real-world datasets enriched with contextual features for time series forecasting. Additionally, we perform extensive ablation studies and exploration analysis.

### Experimental Setup

**Table 2: Diverse real-world datasets from various domains and with distinct characteristics.**
| Dataset | Domain | Frequency | Length | Variables |
| --- | --- | --- | --- | --- |
| Economic | Economic | 1 day | 728 | 107 |
| Health | Health | 1 day | 1,392 | 948 |
| Web | Web | 1 day | 792 | 2,000 |
| Stock-NY | Stock | 1 day | 1,243 | 5 |
| Stock-NA | Stock | 1 day | 1,244 | 5 |
| Nature | Nature | 30 mins | 19,934 | 11 |

#### Datasets.

As shown in Table [2](https://arxiv.org/html/2508.09191v1#Sx4.T2), we evaluate our framework on six real-world datasets from diverse domains enriched with contextual features: Economic (McCracken and Ng 2016), Health (Panagopoulos, Nikolentzos, and Vazirgiannis 2021), Web (Gasthaus et al. 2019), two subsets of Stock data (Feng et al. 2019) and Nature (Poyatos et al. 2020). These datasets, spanning various temporal patterns and contextual dependencies, serve as a comprehensive benchmark for context-aware forecasting. Data preparation involves imputing missing values and applying z-score normalization to all datasets, ensuring stable convergence and comparability. A detailed description is provided in the Appendix A.

#### Baselines.

We compare our proposed framework against eight strong baselines, grouped into four representative categories for comprehensive evaluation. For LLM-based models, we include Time-LLM (Jin et al. 2023) and GPT4TS (Zhou et al. 2023), which adapt pre-trained LLMs for time series forecasting using modality-aware prompting and reprogramming. In the self-supervised frameworks category, we evaluate TimeDART (Wang et al. 2024a) and SimMTM (Dong et al. 2023).

Additionally, we include Transformer-based methods like Autoformer (Wu et al. 2021) and Crossformer (Zhang and Yan 2023). Finally, we consider the MLP-based method DLinear (Zeng et al. 2023). Further details are provided in the Appendix B.

#### Implementation Details.

For each baseline, we search over multiple input lengths and report the best performance to avoid underestimating its capability. The historical length is set to $L=96$ for the Nature dataset and $L=36$ for the other five datasets, based on the data volume and temporal resolution. The forecasting horizons are set to {24, 48, 96, 192} for Nature and {24, 36, 48, 60} for the other dataset. We adopt two widely used evaluation metrics in time series forecasting: mean absolute error (MAE) and mean squared error (MSE). We report average results for the main and ablation studies. For exploratory analysis, we use 96-to-24 on Nature and 36-to-24 on the other datasets. Complete results for the main experiments, ablation studies, and exploratory analysis are included in the Appendix C. All experiments are implemented in PyTorch and conducted on a distributed setup with 8 NVIDIA A100 GPUs.

**Table 3: Ablation study on the effects of cross-modality alignment and generative fine-tuning across multiple datasets.**
| Dataset | Economic | Health | Web | Stock-NY | Stock-NA | Nature |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| w/ Cross-modality Alignment, w/o Generative Fine-tuning | 406.418 | 4.074 | 2.875 | 0.084 | 555.375 | 1.447 | 0.556 | 0.479 | 1.317 | 0.813 | 0.378 | 0.357 |
| w/o Cross-modality Alignment, w/ Generative Fine-Tuning | 72.292 | 1.690 | 2.783 | 0.079 | 504.740 | 1.264 | 0.515 | 0.478 | 1.181 | 0.804 | 0.305 | 0.318 |
| w/ Cross-modality Alignment, w/ Generative Fine-tuning | 68.961 | 1.695 | 2.524 | 0.081 | 497.410 | 1.246 | 0.482 | 0.455 | 1.134 | 0.780 | 0.269 | 0.297 |

### Main Results

Table [1](https://arxiv.org/html/2508.09191v1#Sx3.T1) comprehensively compares forecasting performance across six benchmark datasets. TokenCast demonstrates superior performance in most scenarios, further confirming previous empirical findings (Zhou et al. 2023) that no single model performs best across all settings.
Notably, LLM-based baselines like Time-LLM also show competitive results, particularly on context-rich datasets such as Economic and Stock-NY. This further validates the potential of leveraging large language models in time series forecasting. However, these models often lack the structural alignment mechanisms introduced by our framework, limiting their consistent performance.
Conventional baselines such as TimeDART perform well on datasets with strong periodicity and weak contextual dependence (e.g., Nature), but their performance drops significantly on complex datasets rich in contextual features (e.g., Economic and Web). This contrast underscores the importance of contextual feature modeling and cross-modal interaction.
In summary, our framework delivers state-of-the-art results with high consistency. This is attributed to its core design: discretizing time series into discrete tokens and aligning them with contextual features. This unified token-based paradigm effectively captures multimodal dependencies and addresses real-world context-aware time series forecasting challenges.

### Ablation Studies

#### Ablation on Alignment and Fine-tuning.

We conduct the ablation study on two crucial training steps: the cross-modality alignment and generative fine-tuning. The comprehensive results in Table [3](https://arxiv.org/html/2508.09191v1#Sx4.T3) clearly demonstrate their indispensable contribution to the overall framework performance. The model equipped with the cross-modality alignment stage consistently achieves lower MSE scores across all six datasets. Without this alignment, contextual features risk being misinterpreted by the time series backbone, leading to suboptimal forecasts. This highlights its critical role in effectively integrating contextual information by bridging structural and semantic discrepancies between time series and contextual features, thus facilitating meaningful feature interaction. This alignment thus acts as a foundational step, ensuring the subsequent fine-tuning stage operates on a semantically rich and coherent feature space.

Concurrently, Table [3](https://arxiv.org/html/2508.09191v1#Sx4.T3) vividly illustrates the pivotal contribution of the generative fine-tuning stage. Across all six benchmark datasets, the model employing generative fine-tuning consistently and substantially outperforms its counterpart that omits this crucial step. The performance degradation when omitting this stage is notable across various datasets, underscoring the general applicability and importance of the fine-tuning process. This drop is particularly stark on datasets like Stock-NA, where the complex, non-stationary patterns demand task-specific adaptation. Ultimately, these findings emphasize that generative fine-tuning is essential for adapting the pre-trained LLM’s general capabilities to generative time series forecasting.

Figure: Figure 3: Ablation study on multiple datasets on the contribution of multimodal context in time series forecasting.
Refer to caption: x3.png

#### Ablation on Multimodal Contributions.

Fig.  [3](https://arxiv.org/html/2508.09191v1#Sx4.F3) analyzes how different types of contextual features affect forecasting performance. Incorporating any contextual features yields substantial improvements, as the variant without contextual input consistently performs worse. We further divide the input into general info, which provides high-level context such as domain knowledge and task instructions, and local info, which offers event-specific details like static statistical attributes. While both types contribute to performance, general info typically brings larger improvements. The strong results of the full model indicate its ability to effectively combine the broad context from general info and the specific cues from local info to enhance prediction accuracy.

**Table 4: Study on the number of tokens in the codebook across multiple datasets. We report predicted reconstructed MSE (Recon.), downstream MSE, and downstream MAE.**
| Dataset | Economic | Stock-NA | Nature |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | Recon. | MSE | MAE | Recon. | MSE | MAE | Recon. | MSE | MAE |
| 32 | 190.371 | 50.234 | 1.372 | 0.244 | 0.794 | 0.636 | 0.134 | 0.233 | 0.281 |
| 64 | 141.852 | 37.699 | 1.293 | 0.213 | 0.690 | 0.616 | 0.158 | 0.241 | 0.296 |
| 128 | 170.630 | 39.379 | 1.251 | 0.205 | 0.571 | 0.600 | 0.104 | 0.203 | 0.265 |
| 256 | 191.937 | 39.309 | 1.339 | 0.209 | 0.646 | 0.593 | 0.114 | 0.248 | 0.288 |

### Exploration Analysis

#### Codebook Size.

We conduct a study to assess the impact of codebook size on model performance, as summarized in Table [8](https://arxiv.org/html/2508.09191v1#A1.T8).
The results highlight the importance of selecting an appropriate codebook size for time series forecasting.
Specifically, a size of 128 achieves state-of-the-art results on the Nature and Stock-NA datasets, while a smaller size of 64 excels on the Economic dataset. Interestingly, both smaller (32) and larger (256) codebook sizes fail to produce better results and often lead to significant performance degradation.
This suggests that for our framework, simply increasing token granularity is not always beneficial. Instead, a moderate codebook size strikes a more effective balance between reconstruction fidelity and the complexity of the downstream generation task.

Figure: Figure 4: Forecasting with uncertainty on Stock-NY (left) and Economic (right) datasets. The plots compare the ground truth trajectories with the model’s mean predictions, along with the 50% and 80% predictive intervals.
Refer to caption: x4.png

#### Generative Uncertainty.

To validate the uncertainty modeling capabilities of our TokenCast, we conduct experiments on both the Economic and Stock-NY datasets. As shown in Fig. [4](https://arxiv.org/html/2508.09191v1#Sx4.F4), our method produces predictive distributions that closely track the ground truth, with 50% and 80% prediction intervals capturing the inherent variability in the data. By adjusting the temperature during sampling, we observe that the model can flexibly modulate the spread of the predictive intervals, indicating its potential for controllable uncertainty-aware forecasting. This demonstrates that our model not only provides accurate mean predictions but also yields well-calibrated uncertainty estimates.

#### LLM Backbone.

We evaluate four LLM backbones to identify the optimal architecture for our TokenCast. As summarized in Table [9](https://arxiv.org/html/2508.09191v1#A1.T9), the Qwen2.5-0.5B-base models consistently demonstrate superior performance. Specifically, the base version achieves state-of-the-art results on the Nature and Stock-NA datasets, while the instruct-tuned version excels on the more complex Economic dataset.
Interestingly, larger models like Qwen2.5-1.5B-instruct fail to yield further gains and often underperform. This suggests that for our tasks, simply scaling up model size is not beneficial. Instead, the 0.5B models strike a balance between representational capacity and generalization.

**Table 5: Performance comparison of different backbone models and their variants (base/instruct) across varying model scales and multiple datasets.**
| Dataset | Economic | Stock-NA | Nature |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE |
| Qwen2.5-0.5B-base | 37.164 | 1.301 | 0.668 | 0.605 | 0.180 | 0.246 |
| Qwen2.5-0.5B-inst. | 36.744 | 1.299 | 0.695 | 0.614 | 0.187 | 0.253 |
| Qwen2.5-1.5B-inst. | 38.549 | 1.283 | 0.722 | 0.611 | 0.229 | 0.270 |
| Qwen3-0.6B-inst. | 39.629 | 1.315 | 0.936 | 0.715 | 0.236 | 0.281 |

#### Embedding Layer Initialization.

We investigate three initialization strategies for our model’s embedding layer. As shown in Table [10](https://arxiv.org/html/2508.09191v1#A1.T10), mean initialization consistently provides the most robust performance. Specifically, it achieves the best results on the Nature and Economic datasets.
Specifically, word initialization refers to initializing new embeddings by randomly sampling existing vectors from the original vocabulary, whose performance is less consistent across other domains. Notably, standard random initialization suffers a significant performance degradation on Stock-NA, highlighting its instability. These findings suggest that initializing embeddings with meaningful prior information provides a better starting point for optimization. Therefore, we adopt mean initialization as the default.

**Table 6: Study on different initialization methods on the embedding layer. We compare mean initialization, word initialization, and random initialization.**
| Dataset | Economic | Stock-NA | Nature |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE |
| Mean Initialization | 36.744 | 1.299 | 0.695 | 0.614 | 0.187 | 0.253 |
| Word Initialization | 39.680 | 1.261 | 0.667 | 0.602 | 0.224 | 0.264 |
| Random Initialization | 36.744 | 1.299 | 1.101 | 0.725 | 0.189 | 0.256 |

#### Qualitative Analysis of Tokenization.

To better understand our discretization module, we conduct an in-depth analysis on the Nature dataset, focusing on two key aspects: token usage patterns (Fig. [5](https://arxiv.org/html/2508.09191v1#Sx4.F5)) and reconstruction fidelity (Fig. [6](https://arxiv.org/html/2508.09191v1#Sx4.F6)).
The token usage heatmap in Fig. [5](https://arxiv.org/html/2508.09191v1#Sx4.F5) reveals a high utilization rate across the 64-token vocabulary, avoiding codebook collapse. Frequent token appearances suggest that the learned symbols capture rich and meaningful temporal structures, rather than relying on a limited subset of tokens.
Fig. [6](https://arxiv.org/html/2508.09191v1#Sx4.F6) shows that the reconstructed time series closely follow the original time series, maintaining low MSE and MAE values, even for high-frequency sequences.
These results show that our discretization process achieves a diverse and expressive symbolic vocabulary and accurate reconstructions for effective downstream forecasting task.

Figure: Figure 5: Token usage statistics over the Nature codebook. The heatmap shows the usage frequency of all 64 tokens, with color intensity reflecting how often each token appears.
Refer to caption: x5.png

Figure: Figure 6: Visualizing the reconstruction of the Nature dataset in the vector quantized networks.
Refer to caption: x6.png

## Conclusion

We proposed TokenCast, a context-aware time series prediction framework based on a pre-trained LLM. This approach first converts a continuous time series into discrete tokens. Leveraging a pre-trained LLM, it aligns the temporal and contextual tokens through an autoregressive objective, achieving unified modeling of both modalities. The model is then further fine-tuned to generate future token sequences.
We evaluate TokenCast on multiple real-world datasets rich in contextual information. Experimental results demonstrate that TokenCast achieves superior accuracy. We also conduct comprehensive ablation experiments and qualitative analysis to validate the framework’s adaptability and flexibility for symbolic, LLM-driven time series forecasting. Looking ahead, we believe that leveraging language as a symbolic intermediary will have the potential to advance time series forecasting towards a multimodal and multi-task level.

## References

- Ansari et al. (2024)
Ansari, A. F.; Stella, L.; Turkmen, C.; Zhang, X.; Mercado, P.; Shen, H.; Shchur, O.; Rangapuram, S. S.; Arango, S. P.; Kapoor, S.; et al. 2024.
Chronos: Learning the Language of Time Series.
*Transactions on Machine Learning Research*, 2024.
- Cao et al. (2023)
Cao, D.; Jia, F.; Arik, S. O.; Pfister, T.; Zheng, Y.; Ye, W.; and Liu, Y. 2023.
Tempo: Prompt-based generative pre-trained transformer for time series forecasting.
*arXiv preprint arXiv:2310.04948*.
- Challu et al. (2023)
Challu, C.; Olivares, K. G.; Oreshkin, B. N.; Ramirez, F. G.; Canseco, M. M.; and Dubrawski, A. 2023.
Nhits: Neural hierarchical interpolation for time series forecasting.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 37, 6989–6997.
- Chang, Peng, and Chen (2023)
Chang, C.; Peng, W.-C.; and Chen, T.-F. 2023.
Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms.
*CoRR*.
- Chen and Guestrin (2016)
Chen, T.; and Guestrin, C. 2016.
Xgboost: A scalable tree boosting system.
In *Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining*, 785–794.
- Cheng et al. (2024)
Cheng, M.; Chen, Y.; Liu, Q.; Liu, Z.; and Luo, Y. 2024.
Advancing time series classification with multimodal language modeling.
*arXiv preprint arXiv:2403.12371*.
- Cheng et al. (2025a)
Cheng, M.; Tao, X.; Liu, Q.; Zhang, H.; Chen, Y.; and Lian, D. 2025a.
Cross-domain pre-training with language models for transferable time series representations.
In *Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining*, 175–183.
- Cheng et al. (2025b)
Cheng, M.; Yang, J.; Pan, T.; Liu, Q.; Li, Z.; and Wang, S. 2025b.
Convtimenet: A deep hierarchical fully convolutional model for multivariate time series analysis.
In *Companion Proceedings of the ACM on Web Conference 2025*, 171–180.
- Dong et al. (2023)
Dong, J.; Wu, H.; Zhang, H.; Zhang, L.; Wang, J.; and Long, M. 2023.
Simmtm: A simple pre-training framework for masked time-series modeling.
*Advances in Neural Information Processing Systems*, 36: 29996–30025.
- Feng et al. (2019)
Feng, F.; He, X.; Wang, X.; Luo, C.; Liu, Y.; and Chua, T.-S. 2019.
Temporal relational ranking for stock prediction.
*ACM Transactions on Information Systems (TOIS)*, 37(2): 1–30.
- Feng et al. (2025)
Feng, S.; Zhao, P.; Liu, L.; Wu, P.; and Shen, Z. 2025.
Hdt: Hierarchical discrete transformer for multivariate time series forecasting.
*arXiv preprint arXiv:2502.08302*.
- Gasthaus et al. (2019)
Gasthaus, J.; Benidis, K.; Wang, Y.; Rangapuram, S. S.; Salinas, D.; Flunkert, V.; and Januschowski, T. 2019.
Probabilistic forecasting with spline quantile function RNNs.
In *The 22nd international conference on artificial intelligence and statistics*, 1901–1910. PMLR.
- Holt (2004)
Holt, C. C. 2004.
Forecasting seasonals and trends by exponentially weighted moving averages.
*International journal of forecasting*, 20(1): 5–10.
- Huang et al. (2025)
Huang, Y.-H.; Xu, C.; Wu, Y.; Li, W.-J.; and Bian, J. 2025.
Timedp: Learning to generate multi-domain time series with domain prompts.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 17520–17527.
- Hyndman and Khandakar (2008)
Hyndman, R. J.; and Khandakar, Y. 2008.
Automatic time series forecasting: the forecast package for R.
*Journal of statistical software*, 27: 1–22.
- Jin et al. (2024)
Jin, M.; Koh, H. Y.; Wen, Q.; Zambon, D.; Alippi, C.; Webb, G. I.; King, I.; and Pan, S. 2024.
A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*.
- Jin et al. (2023)
Jin, M.; Wang, S.; Ma, L.; Chu, Z.; Zhang, J. Y.; Shi, X.; Chen, P.-Y.; Liang, Y.; Li, Y.-F.; Pan, S.; et al. 2023.
Time-llm: Time series forecasting by reprogramming large language models.
*arXiv preprint arXiv:2310.01728*.
- Kalekar et al. (2004)
Kalekar, P. S.; et al. 2004.
Time series forecasting using holt-winters exponential smoothing.
*Kanwal Rekhi school of information Technology*, 4329008(13): 1–13.
- Lai et al. (2018)
Lai, G.; Chang, W.-C.; Yang, Y.; and Liu, H. 2018.
Modeling long-and short-term temporal patterns with deep neural networks.
In *The 41st international ACM SIGIR conference on research & development in information retrieval*, 95–104.
- Lim et al. (2021)
Lim, B.; Arık, S. Ö.; Loeff, N.; and Pfister, T. 2021.
Temporal fusion transformers for interpretable multi-horizon time series forecasting.
*International journal of forecasting*, 37(4): 1748–1764.
- Liu et al. (2024a)
Liu, H.; Xu, S.; Zhao, Z.; Kong, L.; Prabhakar Kamarthi, H.; Sasanur, A.; Sharma, M.; Cui, J.; Wen, Q.; Zhang, C.; et al. 2024a.
Time-mmd: Multi-domain multimodal dataset for time series analysis.
*Advances in Neural Information Processing Systems*, 37: 77888–77933.
- Liu et al. (2024b)
Liu, H.; Zhao, Z.; Wang, J.; Kamarthi, H.; and Prakash, B. A. 2024b.
Lstprompt: Large language models as zero-shot time series forecasters by long-short-term prompting.
*arXiv preprint arXiv:2402.16132*.
- Liu et al. (2025)
Liu, P.; Guo, H.; Dai, T.; Li, N.; Bao, J.; Ren, X.; Jiang, Y.; and Xia, S.-T. 2025.
Calf: Aligning llms for time series forecasting via cross-modal fine-tuning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 18915–18923.
- Liu et al. (2024c)
Liu, X.; Hu, J.; Li, Y.; Diao, S.; Liang, Y.; Hooi, B.; and Zimmermann, R. 2024c.
Unitime: A language-empowered unified model for cross-domain time series forecasting.
In *Proceedings of the ACM Web Conference 2024*, 4095–4106.
- Liu et al. (2024d)
Liu, Y.; Qin, G.; Huang, X.; Wang, J.; and Long, M. 2024d.
Autotimes: Autoregressive time series forecasters via large language models.
*Advances in Neural Information Processing Systems*, 37: 122154–122184.
- Liu et al. (2024e)
Liu, Z.; Yang, J.; Cheng, M.; Luo, Y.; and Li, Z. 2024e.
Generative pretrained hierarchical transformer for time series forecasting.
In *Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining*, 2003–2013.
- McCracken and Ng (2016)
McCracken, M. W.; and Ng, S. 2016.
FRED-MD: A monthly database for macroeconomic research.
*Journal of Business & Economic Statistics*, 34(4): 574–589.
- Panagopoulos, Nikolentzos, and Vazirgiannis (2021)
Panagopoulos, G.; Nikolentzos, G.; and Vazirgiannis, M. 2021.
Transfer graph neural networks for pandemic forecasting.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, 4838–4845.
- Poyatos et al. (2020)
Poyatos, R.; Granda, V.; Flo, V.; Adams, M. A.; Adorján, B.; Aguadé, D.; Aidar, M. P.; Allen, S.; Alvarado-Barrientos, M. S.; Anderson-Teixeira, K. J.; et al. 2020.
Global transpiration data from sap flow measurements: the SAPFLUXNET database.
*Earth System Science Data Discussions*, 2020: 1–57.
- Qiu et al. (2024)
Qiu, X.; Hu, J.; Zhou, L.; Wu, X.; Du, J.; Zhang, B.; Guo, C.; Zhou, A.; Jensen, C. S.; Sheng, Z.; et al. 2024.
Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods.
*arXiv preprint arXiv:2403.20150*.
- Rasul et al. (2024)
Rasul, K.; Bennett, A.; Vicente, P.; Gupta, U.; Ghonia, H.; Schneider, A.; and Nevmyvaka, Y. 2024.
Vq-tr: Vector quantized attention for time series forecasting.
In *The Twelfth International Conference on Learning Representations*.
- Rasul et al. (2022)
Rasul, K.; Park, Y.-J.; Ramström, M. N.; and Kim, K.-M. 2022.
Vq-ar: Vector quantized autoregressive probabilistic time series forecasting.
*arXiv preprint arXiv:2205.15894*.
- Salinas et al. (2020)
Salinas, D.; Flunkert, V.; Gasthaus, J.; and Januschowski, T. 2020.
DeepAR: Probabilistic forecasting with autoregressive recurrent networks.
*International journal of forecasting*, 36(3): 1181–1191.
- Sun et al. (2023)
Sun, C.; Li, H.; Li, Y.; and Hong, S. 2023.
Test: Text prototype aligned embedding to activate llm’s ability for time series.
*arXiv preprint arXiv:2308.08241*.
- Tang and Zhang (2025)
Tang, P.; and Zhang, W. 2025.
Unlocking the Power of Patch: Patch-Based MLP for Long-Term Time Series Forecasting.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 12640–12648.
- Wang et al. (2024a)
Wang, D.; Cheng, M.; Liu, Z.; Liu, Q.; and Chen, E. 2024a.
Timedart: A diffusion autoregressive transformer for self-supervised time series representation.
*arXiv preprint arXiv:2410.05711*.
- Wang et al. (2023)
Wang, H.; Peng, J.; Huang, F.; Wang, J.; Chen, J.; and Xiao, Y. 2023.
Micn: Multi-scale local and global context modeling for long-term series forecasting.
In *The eleventh international conference on learning representations*.
- Wang et al. (2024b)
Wang, J.; Cheng, M.; Mao, Q.; Liu, Q.; Xu, F.; Li, X.; and Chen, E. 2024b.
Tabletime: Reformulating time series classification as zero-shot table understanding via large language models.
*arXiv e-prints*, arXiv–2411.
- Wang et al. (2024c)
Wang, S.; Wu, H.; Shi, X.; Hu, T.; Luo, H.; Ma, L.; Zhang, J. Y.; and Zhou, J. 2024c.
Timemixer: Decomposable multiscale mixing for time series forecasting.
*arXiv preprint arXiv:2405.14616*.
- Wang et al. (2019)
Wang, Y.; Smola, A.; Maddix, D.; Gasthaus, J.; Foster, D.; and Januschowski, T. 2019.
Deep factors for forecasting.
In *International conference on machine learning*, 6607–6617. PMLR.
- Wang et al. (2024d)
Wang, Y.; Wu, H.; Dong, J.; Qin, G.; Zhang, H.; Liu, Y.; Qiu, Y.; Wang, J.; and Long, M. 2024d.
Timexer: Empowering transformers for time series forecasting with exogenous variables.
*Advances in Neural Information Processing Systems*, 37: 469–498.
- Williams et al. (2024)
Williams, A. R.; Ashok, A.; Marcotte, É.; Zantedeschi, V.; Subramanian, J.; Riachi, R.; Requeima, J.; Lacoste, A.; Rish, I.; Chapados, N.; et al. 2024.
Context is key: A benchmark for forecasting with essential textual information.
*arXiv preprint arXiv:2410.18959*.
- Winters (1960)
Winters, P. R. 1960.
Forecasting sales by exponentially weighted moving averages.
*Management science*, 6(3): 324–342.
- Wu et al. (2021)
Wu, H.; Xu, J.; Wang, J.; and Long, M. 2021.
Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.
*Advances in neural information processing systems*, 34: 22419–22430.
- Zeng et al. (2023)
Zeng, A.; Chen, M.; Zhang, L.; and Xu, Q. 2023.
Are transformers effective for time series forecasting?
In *Proceedings of the AAAI conference on artificial intelligence*, volume 37, 11121–11128.
- Zhang and Yan (2023)
Zhang, Y.; and Yan, J. 2023.
Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.
In *The eleventh international conference on learning representations*.
- Zhao et al. (2023)
Zhao, W. X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. 2023.
A survey of large language models.
*arXiv preprint arXiv:2303.18223*, 1(2).
- Zhao et al. (2025)
Zhao, Z.; Wang, P.; Wen, H.; Wang, S.; Yu, L.; and Wang, Y. 2025.
STEM-LTS: Integrating Semantic-Temporal Dynamics in LLM-driven Time Series Analysis.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 22858–22866.
- Zhou et al. (2022)
Zhou, T.; Ma, Z.; Wen, Q.; Wang, X.; Sun, L.; and Jin, R. 2022.
Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.
In *International conference on machine learning*, 27268–27286. PMLR.
- Zhou et al. (2023)
Zhou, T.; Niu, P.; Sun, L.; Jin, R.; et al. 2023.
One fits all: Power general time series analysis by pretrained lm.
*Advances in neural information processing systems*, 36: 43322–43355.

## Appendix A Appendices

### Appendix A Datasets Descriptions

In this study, we utilize six diverse real-world datasets enriched with contextual features spanning various domains, including economics, health, web, stock markets, and natural sciences. Each dataset exhibits unique temporal characteristics and varying degrees of contextual dependency, offering a comprehensive benchmark.

- •
Economic (FRED-MD): A monthly macroeconomic dataset consisting of 107 indicators across sectors such as production, labor, and inflation. It supports empirical studies requiring rich contextual interpretation.
- •
Health (Covid-19): Released by Facebook’s “Data for Good” initiative, this dataset tracks human mobility patterns across regions during the COVID-19 pandemic, offering policy-driven contextual signals.
- •
Stock-NY (NYSE): Similar in structure and period to NASDAQ, this dataset provides daily time series from the New York Stock Exchange, facilitating comparative financial forecasting studies.
- •
Stock-NA (NASDAQ): A daily stock dataset collected from the NASDAQ exchange between 2013 and 2017, containing representative securities with dynamics heavily influenced by external news and events.
- •
Web (Wike2000): A high-dimensional dataset recording daily page views of 9,013 Wikipedia articles. We select the top 2,000 pages to capture volatile, event-driven user behavior shaped by external textual contexts.
- •
Nature (CzeLan): A 30-minute resolution dataset capturing natural environmental signals with strong periodic patterns and low contextual dependence. It serves as a representative benchmark for low-context forecasting.

### Appendix B Additional Implementation Details

In this appendix, we provide comprehensive descriptions of the baseline methods used for comparison in the main paper. We also detail the additional configuration parameters and training setups specific to our proposed model to ensure full reproducibility and transparency.

#### Compared Baselines.

We first provide a detailed overview of the baseline models employed for comparative analysis in the main manuscript. These models are grouped into four distinct categories, each reflecting a key methodological paradigm in contemporary time series forecasting: LLM-based approaches, self-supervised frameworks, Transformer-based architectures, and a straightforward yet effective linear model. Below, we present concise descriptions of each model, emphasizing its core techniques and underlying conceptual foundations.

- •
Time-LLM: This is a reprogramming framework that transforms time series into text-based representations for input into a frozen large language model (LLM), guided by a Prompt-as-Prefix mechanism to enable reasoning and achieve general-purpose time series forecasting.
- •
GPT4TS: This work proposes the Frozen Pretrained Transformer (FPT), a framework that repurposes language or vision transformers for general time series analysis by freezing their core layers and fine-tuning only task-specific components, leveraging large-scale pretraining without requiring extensive time series data.
- •
TimeDART: This self-supervised pre-training framework addresses the challenge of modeling long-term dynamics and local patterns by combining Transformer encoding with a denoising diffusion process, yielding more transferable representations for downstream tasks.
- •
SimMTM: This is a masked time series pre-training framework that addresses the challenge of disrupted temporal semantics by reconstructing masked points through weighted aggregation from multiple complementary series, preserving temporal variations and learning manifold structures for improved downstream performance.
- •
Autoformer: This addresses the challenge of long-term time series forecasting by introducing a novel decomposition-based architecture with an Auto-Correlation mechanism, which replaces traditional self-attention to capture periodic dependencies and progressively model complex temporal patterns.
- •
Crossformer: This addresses the challenge of capturing temporal and inter-variable dependencies in multivariate time series forecasting using a Dimension-Segment-Wise embedding and Two-Stage Attention within a hierarchical encoder-decoder architecture.
- •
DLinear: This work challenges the effectiveness of complex Transformer-based models for long-term time series forecasting by demonstrating that a simple one-layer linear model can outperform them, highlighting limitations of self-attention in capturing temporal order and calling for renewed exploration of alternative approaches.

**Table 7: All reported results are the average of three trials on various context-rich benchmark datasets. Lower values indicate better performance. Best results are in bold and second-best results are underlined.**
| Model | TokenCast | Time-LLM | GPT4TS | TimeDART | SimMTM | Crossformer | Autoformer | DLinear |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| Economic | 24 | 38.946 | 1.188 | 38.235 | 1.379 | 40.546 | 1.289 | 38.172 | 1.392 | 42.856 | 1.198 | 386.326 | 3.892 | 66.721 | 1.714 | 69.693 | 1.621 |
| 36 | 56.116 | 1.488 | 64.829 | 1.626 | 67.560 | 1.569 | 66.131 | 1.632 | 70.291 | 1.511 | 401.198 | 4.036 | 95.088 | 1.953 | 99.743 | 1.905 |  |
| 48 | 77.678 | 1.767 | 92.481 | 1.882 | 97.990 | 1.851 | 100.061 | 1.861 | 103.499 | 1.820 | 415.148 | 4.156 | 130.566 | 2.195 | 137.429 | 2.220 |  |
| 60 | 102.904 | 2.140 | 130.623 | 2.153 | 137.690 | 2.156 | 139.751 | 2.199 | 144.757 | 2.159 | 423.001 | 4.212 | 174.605 | 2.489 | 181.999 | 2.531 |  |
| Avg | 68.911 | 1.646 | 81.542 | 1.760 | 85.947 | 1.716 | 86.029 | 1.771 | 90.351 | 1.672 | 406.418 | 4.074 | 116.745 | 2.088 | 122.216 | 2.069 |  |
| Health | 24 | 1.699 | 0.065 | 2.322 | 0.081 | 1.961 | 0.063 | 2.030 | 0.084 | 1.828 | 0.069 | 1644.541 | 2.363 | 2.097 | 0.276 | 24.717 | 0.426 |
| 36 | 2.228 | 0.073 | 2.702 | 0.103 | 2.384 | 0.080 | 2.405 | 0.094 | 2.400 | 0.083 | 1645.156 | 2.536 | 2.407 | 0.256 | 29.370 | 0.463 |  |
| 48 | 2.607 | 0.085 | 2.916 | 0.114 | 2.710 | 0.091 | 2.747 | 0.083 | 2.980 | 0.094 | 1645.237 | 2.588 | 2.770 | 0.253 | 16.203 | 0.381 |  |
| 60 | 3.563 | 0.102 | 3.353 | 0.118 | 3.201 | 0.096 | 3.309 | 0.092 | 3.672 | 0.107 | 1645.053 | 2.529 | 3.193 | 0.277 | 44.057 | 0.551 |  |
| Avg | 2.524 | 0.081 | 2.823 | 0.104 | 2.564 | 0.083 | 2.623 | 0.088 | 2.720 | 0.088 | 1644.997 | 2.504 | 2.617 | 0.266 | 28.587 | 0.455 |  |
| Web | 24 | 453.609 | 1.169 | 524.512 | 1.699 | 510.602 | 1.345 | 695.235 | 1.264 | 1794.692 | 1.638 | 644.061 | 1.881 | 671.749 | 3.912 | 561.505 | 1.277 |
| 36 | 480.019 | 1.204 | 542.815 | 1.720 | 525.713 | 1.441 | 780.234 | 1.339 | 508.610 | 1.162 | 697.898 | 1.930 | 706.661 | 3.302 | 630.056 | 1.362 |  |
| 48 | 518.164 | 1.281 | 571.482 | 1.783 | 551.430 | 1.504 | 779.730 | 1.398 | 531.377 | 1.224 | 719.304 | 2.002 | 736.918 | 3.076 | 649.944 | 1.439 |  |
| 60 | 537.828 | 1.330 | 592.523 | 1.803 | 574.221 | 1.543 | 839.340 | 1.474 | 555.918 | 1.283 | 732.002 | 2.040 | 774.696 | 2.921 | 687.700 | 1.514 |  |
| Avg | 497.410 | 1.246 | 557.833 | 1.751 | 540.492 | 1.458 | 773.635 | 1.369 | 847.649 | 1.327 | 698.316 | 1.963 | 722.506 | 3.303 | 632.301 | 1.398 |  |
| Stock-NY | 24 | 0.289 | 0.350 | 0.351 | 0.383 | 0.342 | 0.380 | 0.499 | 0.477 | 0.332 | 0.377 | 1.100 | 0.909 | 0.427 | 0.466 | 0.600 | 0.573 |
| 36 | 0.372 | 0.401 | 0.403 | 0.412 | 0.401 | 0.409 | 0.661 | 0.551 | 0.398 | 0.406 | 1.032 | 0.886 | 0.526 | 0.508 | 0.856 | 0.702 |  |
| 48 | 0.538 | 0.479 | 0.599 | 0.569 | 0.576 | 0.523 | 0.852 | 0.643 | 0.552 | 0.477 | 1.061 | 0.889 | 0.757 | 0.605 | 1.112 | 0.802 |  |
| 60 | 0.727 | 0.588 | 1.293 | 0.674 | 1.232 | 0.697 | 1.092 | 0.753 | 1.170 | 0.719 | 1.251 | 0.962 | 0.994 | 0.714 | 1.447 | 0.938 |  |
| Avg | 0.482 | 0.455 | 0.662 | 0.510 | 0.638 | 0.502 | 0.776 | 0.606 | 0.613 | 0.495 | 1.111 | 0.912 | 0.676 | 0.573 | 1.004 | 0.754 |  |
| Stock-NA | 24 | 0.661 | 0.600 | 0.725 | 0.700 | 0.796 | 0.689 | 0.796 | 0.711 | 0.867 | 0.677 | 1.773 | 0.996 | 1.171 | 0.811 | 1.498 | 0.920 |
| 36 | 0.887 | 0.694 | 0.962 | 0.828 | 1.122 | 0.828 | 0.983 | 0.737 | 1.281 | 0.827 | 1.921 | 1.046 | 1.494 | 0.912 | 1.755 | 0.974 |  |
| 48 | 1.473 | 0.886 | 1.385 | 1.004 | 1.481 | 0.956 | 1.822 | 1.028 | 1.577 | 0.907 | 1.975 | 1.072 | 1.712 | 0.958 | 1.852 | 0.982 |  |
| 60 | 1.515 | 0.941 | 1.729 | 1.166 | 1.688 | 1.045 | 2.036 | 1.055 | 1.646 | 0.924 | 1.980 | 1.095 | 1.608 | 0.930 | 1.733 | 0.955 |  |
| Avg | 1.134 | 0.780 | 1.200 | 0.925 | 1.272 | 0.880 | 1.409 | 0.883 | 1.343 | 0.834 | 1.912 | 1.052 | 1.496 | 0.903 | 1.710 | 0.958 |  |
| Nature | 24 | 0.179 | 0.246 | 0.212 | 0.254 | 0.226 | 0.270 | 0.170 | 0.245 | 0.205 | 0.255 | 0.623 | 0.453 | 0.325 | 0.383 | 0.288 | 0.396 |
| 48 | 0.252 | 0.293 | 0.251 | 0.281 | 0.271 | 0.303 | 0.261 | 0.271 | 0.262 | 0.295 | 0.724 | 0.513 | 0.556 | 0.509 | 0.364 | 0.448 |  |
| 96 | 0.291 | 0.307 | 0.275 | 0.293 | 0.286 | 0.305 | 0.254 | 0.274 | 0.268 | 0.288 | 0.734 | 0.516 | 0.502 | 0.487 | 0.354 | 0.423 |  |
| 192 | 0.355 | 0.340 | 0.295 | 0.302 | 0.311 | 0.318 | 0.286 | 0.313 | 0.301 | 0.304 | 0.849 | 0.562 | 0.648 | 0.564 | 0.470 | 0.475 |  |
| Avg | 0.269 | 0.297 | 0.258 | 0.283 | 0.274 | 0.299 | 0.243 | 0.276 | 0.259 | 0.286 | 0.733 | 0.511 | 0.508 | 0.486 | 0.369 | 0.436 |  |
| 1^st Count | 17  11 | 2  1 | 0  1 | 4  5 | 0  5 | 0  0 | 1  0 | 0  0 |  |  |  |  |  |  |  |  |  |

#### Model Configurations.

Next, we present the implementation details of our TokenCast framework, with a special focus on its three core stages: (a) time series discretization, (b) cross-modality alignment, and (c) generative fine-tuning.
We design a specialized time series tokenizer to bridge structural differences across modalities. It consists of a Causal TCN encoder that extracts contextualized embeddings and a Causal Transformer decoder that reconstructs the original sequence. The embeddings are quantized into discrete tokens, producing compact and informative representations. Specifically, the encoder comprises 3 layers for effective feature extraction, with an embedding size of 64 and a uniform patch size of 4.
The second stage aligns time series data with a pre-trained LLM by expanding its vocabulary to include time series tokens and introducing a unified projection layer for shared semantic space. The model is trained with an autoregressive objective using contextual features and historical tokens. Key hyperparameters, such as a learning rate of $5\times 10^{-5}$ and batch size of 16, are carefully tuned to ensure stable alignment.
For the final forecasting task, we utilize the aligned LLM in a generative manner. The model takes historical time series and relevant context as input to predict the sequence of future tokens. These generated tokens are then passed to the time series de-tokenizer to be converted back into a continuous predicted time series. For the optimization settings in this phase, we employ the Adam optimizer with a fine-tuning learning rate set to $1\times 10^{-5}$. All parameters of the aligned model are updated during the fine-tuning process to adapt its generative capabilities specifically for multi-step horizon prediction, while retaining the same architectural configuration as the alignment phase.

### Appendix C Full Results

Due to space limitations, the complete results of all experiments are presented in the appendix. The main experimental results are summarized in Table [7](https://arxiv.org/html/2508.09191v1#A1.T7). To further analyze the impact of each component, we conduct ablation studies on the codebook size, the LLM backbone, and the embedding layer initialization. The corresponding results are reported in Tables [8](https://arxiv.org/html/2508.09191v1#A1.T8), [9](https://arxiv.org/html/2508.09191v1#A1.T9), and [10](https://arxiv.org/html/2508.09191v1#A1.T10), respectively.

#### Main Results.

Table [7](https://arxiv.org/html/2508.09191v1#A1.T7) provides a comprehensive performance comparison across six benchmark datasets, evaluating models on both MSE and MAE metrics. Our model, TokenCast, demonstrates state-of-the-art performance, securing 17 first-place finishes and establishing itself as a top-tier method alongside the leading baseline. This aligns with findings that no single model universally excels, yet it highlights the advantages of our approach.
Notably, other LLM-based baselines like Time-LLM and GPT4TS also deliver competitive results, which further validates the potential of leveraging large language models for time series forecasting. However, the performance of these models often varies significantly by dataset. For instance, while Time-LLM is highly effective on the Economic dataset, TokenCast shows a clear advantage on the Stock-NA benchmark, consistently outperforming all other models across nearly all forecasting horizons. This variability suggests that while powerful, generic LLM baselines may lack the specialized architecture needed to explicitly ground and adapt to diverse time-series dynamics.
In stark contrast, earlier architectures like Crossformer and Autoformer consistently underperform, particularly on complex, non-stationary datasets such as Web and Economic. Their limitations are evident in the quantitative results; for example, on the Economic dataset, the average MSE for Crossformer (423.001) and Autoformer (174.605) is substantially higher than that of TokenCast (68.911). This large performance gap underscores the difficulty their feature interaction mechanisms face in capturing intricate time-series patterns.
In summary, TokenCast achieves not only state-of-the-art but also highly consistent results across a wide range of scenarios. We attribute this success to its core design: discretizing the time series into a unified token-based paradigm. By modeling time-series forecasting as a sequence-to-sequence task in this discrete space, TokenCast effectively captures the intricate dependencies and dynamics that challenge other methods, proving its robustness and effectiveness across diverse forecasting scenarios.

#### Codebook Size.

To investigate the impact of the number of tokens in the codebook, we conducted an ablation study with codebook sizes of 32, 64, 128, and 256. Table [8](https://arxiv.org/html/2508.09191v1#A1.T8) presents the results, including the reconstruction MSE from the time series discretion stage (Recon.) and the downstream forecasting performance (MSE and MAE) across all six datasets.
The key finding is that the optimal codebook size is highly dependent on the specific characteristics of the dataset. There is no single size that universally performs best. For instance, on the Economic dataset, a relatively small codebook of 64 tokens achieves the best performance across all three metrics. In contrast, for the downstream task on the Stock-NA dataset, a larger codebook of 256 tokens is optimal.
Furthermore, we observe an interesting trade-off between reconstruction fidelity and downstream task performance. On the Health and Stock-NY datasets, the codebook size that yields the best reconstruction error is not the one that results in the best downstream forecasting accuracy. For example, on Health, a size of 256 is best for reconstruction, but a size of 32 is superior for the downstream task. This suggests that a better reconstruction of the original signal does not always translate to better forecasting performance, and the ideal level of data quantization can vary.
Given that a codebook size of 64 or 128 provides a strong and balanced performance across most datasets, we selected one of these values (e.g., 64) as the default for our main experiments to ensure robust and generalizable results.

**Table 8: Study on the number of tokens in the codebook across multiple datasets. We report predicted reconstructed MSE (Recon.), downstream MSE, and downstream MAE.**
| Dataset | Economic | Health | Web | Stock-NY | Stock-NA | Nature |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | Recon. | MSE | MAE | Recon. | MSE | MAE | Recon. | MSE | MAE | Recon. | MSE | MAE | Recon. | MSE | MAE | Recon. | MSE | MAE |
| 32 | 190.371 | 50.234 | 1.372 | 207.459 | 1.772 | 0.065 | 731.474 | 451.827 | 1.165 | 0.569 | 0.325 | 0.377 | 0.244 | 0.794 | 0.636 | 0.134 | 0.233 | 0.281 |
| 64 | 141.852 | 37.699 | 1.293 | 101.652 | 2.714 | 0.072 | 664.501 | 529.401 | 1.228 | 0.573 | 0.339 | 0.381 | 0.213 | 0.690 | 0.616 | 0.158 | 0.241 | 0.296 |
| 128 | 170.630 | 39.379 | 1.251 | 186.619 | 2.622 | 0.070 | 3924.953 | 1743.889 | 1.539 | 0.518 | 0.730 | 0.604 | 0.205 | 0.671 | 0.600 | 0.104 | 0.203 | 0.265 |
| 256 | 191.937 | 39.309 | 1.339 | 69.035 | 2.413 | 0.070 | 5062.452 | 899.202 | 1.385 | 0.572 | 0.384 | 0.424 | 0.209 | 0.646 | 0.593 | 0.114 | 0.248 | 0.288 |

#### LLM Backbone.

To determine the optimal Large Language Model (LLM) backbone for our framework, we conducted a comparative study of different Qwen models, focusing on the impact of model scale and fine-tuning variants (base vs. instruct). The results across six benchmark datasets are detailed in Table [9](https://arxiv.org/html/2508.09191v1#A1.T9).
Our analysis reveals two key insights. First, for the task of time-series forecasting, the base model consistently outperforms its instruction-tuned (instruct) counterpart. As shown with the Qwen2-0.5B model, the base version achieves superior or equal performance on all six datasets, with notably lower error rates on Web, Stock-NY, Stock-NA, and Nature. This suggests that fine-tuning for instruction-following, while beneficial for conversational AI, may not be suitable and can even slightly degrade performance on specialized numerical prediction tasks.
Second, forecasting performance does not necessarily improve with a larger model scale. The Qwen2-1.5B-inst model, despite being the largest, did not yield the best results and was generally outperformed by the much smaller Qwen2-0.5B-base model. This indicates that for time-series analysis, simply increasing the parameter count is not a guaranteed path to better performance, and other factors like architecture and training data are more critical.
In conclusion, the Qwen2-0.5B-base model demonstrated the strongest overall performance, securing the best results on four of the six datasets.
Based on this evidence, we selected it as the default backbone for our main experiments.

**Table 9: Performance comparison of different backbone models and their variants (base/instruct) across varying model scales and multiple datasets.**
| Backbone | Economic | Health | Web | Stock-NY | Stock-NA | Nature |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| Qwen2.5-0.5B-base | 37.164 | 1.301 | 2.492 | 0.068 | 586.793 | 1.271 | 0.297 | 0.355 | 0.668 | 0.605 | 0.180 | 0.246 |
| Qwen2.5-0.5B-inst. | 36.744 | 1.299 | 2.493 | 0.068 | 586.780 | 1.271 | 0.353 | 0.391 | 0.695 | 0.614 | 0.187 | 0.253 |
| Qwen2.5-1.5B-inst. | 38.549 | 1.283 | 2.471 | 0.069 | 589.843 | 1.273 | 0.329 | 0.372 | 0.722 | 0.611 | 0.229 | 0.270 |
| Qwen3-0.6B-inst. | 39.629 | 1.315 | 2.320 | 0.068 | 588.379 | 1.272 | 0.405 | 0.417 | 0.936 | 0.715 | 0.236 | 0.281 |

#### Embedding Layer Initialization.

We investigated the impact of different initialization methods for the embedding layer, as this can significantly affect model convergence and final performance. Table [10](https://arxiv.org/html/2508.09191v1#A1.T10) compares three distinct strategies: Mean Initialization, Codebook Sampling, and Random Initialization.
The results clearly indicate that Mean Initialization is the most robust and effective method across the majority of the datasets. It achieved the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) on four of the six benchmarks: Health, Web, Stock-NY, and Nature. It also tied for the best MSE on the Economic dataset.
While Codebook Sampling delivered the best performance on the Stock-NA dataset, it did not consistently outperform Mean Initialization elsewhere. Random Initialization, which serves as a standard baseline, was generally the least effective method. Its performance was notably weaker on the Stock-NA dataset, suggesting that a more structured initialization approach provides a significant advantage.
Given its consistent and superior performance, this study validates the effectiveness of Mean Initialization. We therefore adopted it as the default initialization strategy for the embedding layer in all other experiments conducted in this paper.

**Table 10: Study on different initialization methods on the embedding layer. We compare mean initialization, codebook sampling, and random initialization.**
| Method | Economic | Health | Web | Stock-NY | Stock-NA | Nature |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| Mean Initialization | 36.744 | 1.299 | 2.357 | 0.068 | 585.064 | 1.256 | 0.319 | 0.371 | 0.695 | 0.614 | 0.187 | 0.253 |
| Codebook Sampling | 39.680 | 1.261 | 2.574 | 0.068 | 585.665 | 1.265 | 0.337 | 0.380 | 0.667 | 0.602 | 0.224 | 0.264 |
| Random Initialization | 36.744 | 1.299 | 2.493 | 0.068 | 586.780 | 1.271 | 0.353 | 0.391 | 1.101 | 0.725 | 0.189 | 0.256 |

#### Visualization

Fig. [7](https://arxiv.org/html/2508.09191v1#A1.F7) provides a qualitative analysis of the 36-to-36 forecasting results on the Stock-NA dataset.
A visual inspection reveals that the LLM-based models demonstrate a strong ability to model the series. Their predictions mirror the overall trend and direction of the ground truth. While they may not capture the exact magnitude of every peak and trough, they correctly anticipate the major turning points and replicate the essential high-frequency fluctuations, which are crucial for any meaningful financial forecasting. This indicates a deep understanding of the underlying patterns in the data.
In stark contrast, the traditional baselines exhibit significant weaknesses.
The failure of Crossformer is even more pronounced. Its forecast begins to track the series for a short period but then rapidly collapses towards zero, indicating a catastrophic failure to model the time-series dependencies over the 36-step horizon. This type of model degeneration highlights its lack of robustness on this challenging dataset.
This qualitative comparison provides compelling visual evidence for the superiority of the LLM-based methods. Their ability to produce granular, responsive forecasts that respect the non-stationary and volatile nature of the financial data stands in sharp contrast to the limitations of earlier models, underscoring the effectiveness of the unified token-based paradigm for capturing complex temporal dynamics.

Figure: Figure 7: Visualize the 36-to-36 prediction results of different models on the Stock-NA dataset.
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Could the authors provide clearer **evidence that multimodal context actually improves forecasting performance**?  
   For example, are there quantitative comparisons between using and omitting textual/contextual inputs, or analyses showing which modalities contribute the most?  
2. The vector quantization approach appears similar to that used in **Chronos**. Could you clarify the methodological or empirical differences?  
3. Are the reported results averaged across **multiple random seeds** for reliability?  
4. Can you provide **runtime, memory, or parameter comparisons** to support the claimed efficiency?  
5. How sensitive is performance to the size of the token vocabulary or the choice of LLM backbone?
soundness: 3
strengths: - **Proper positioning within current research trends.**  The paper is aligned with the recent movement toward symbolic or token-based time-series modeling, showing that the authors are aware of ongoing developments in the field.
- **Well-organized framework.** The three-stage pipeline (tokenization, alignment, and generative prediction) is logically structured and easy to follow.  
- **Readable presentation.** The writing is clear, and figures effectively illustrate the workflow.
summary: This paper proposes TokenCast, an LLM-driven framework for context-aware time series forecasting based on symbolic discretization.
Instead of processing continuous numerical values directly, the authors convert time-series data into discrete temporal tokens via vector quantization and reversible normalization, enabling the model to operate in the same token space as textual inputs.
By extending the vocabulary of a pre-trained LLM, the method aligns time-series and text representations within a shared semantic space, allowing joint reasoning through next-token prediction.
Extensive experiments on six context-rich datasets (economic, health, web, and stock domains) show that TokenCast achieves competitive or superior results compared with strong baselines such as Time-LLM, GPT4TS, and Crossformer.
Ablation and sensitivity studies confirm the effectiveness of the proposed tokenization and alignment strategies.
Overall, the paper offers a novel perspective on unifying numerical and textual modalities under the LLM generative paradigm, though the baseline coverage could be broader and the efficiency analysis remains limited.
weaknesses: - **Lack of novelty relative to existing work.**  
  The proposed vector quantization and tokenization strategy is highly similar to the approach used in Amazon’s Chronos model, which also discretizes numerical sequences into symbolic tokens for autoregressive forecasting.  Several recent works (e.g., Chronos, Chronos-Bolt, SymbolicTS, and TokenTS) have already explored nearly identical ideas.  The paper does not clearly differentiate itself in methodology or theoretical contribution, making the innovation appear incremental.

- **Lack of clear evidence for multimodal gains.**  
  Although the paper emphasizes context-aware forecasting, it does not clearly show how textual or non-temporal modalities enhance numerical prediction. Many so-called multimodal datasets contribute little meaningful contextual signal, and in some cases may even introduce data leakage risks.

- **Overreliance on existing LLM architecture.**  
  The contribution lies primarily in applying tokenization to an existing LLM rather than introducing a new modeling principle or objective.

- **Efficiency and scalability not evaluated.**  
  Tokenization and vocabulary extension introduce additional computation, but the paper provides no analysis of training or inference cost.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can you provide robustness or failure-case analysis, for example, regimes such as market shocks, policy changes, or abrupt environmental shifts where the approach does not reduce error relative to baselines?
2. The method conditions on contextual features. For identical historical numeric input, can you show how adding and removing specific contextual signals changes the generated forecast and explain how those changes reflect the contextual content?
soundness: 2
strengths: 1. The paper formulates context-aware forecasting as conditional sequence generation by mapping multivariate time series into discrete tokens, aligning them with text tokens in a shared LLM vocabulary, and autoregressively generating future trajectories.  
2. The method includes reversible instance normalization using only historical context and a shared codebook, encoder-decoder, which keeps the tokenization invertible.  
3. Experiments span six real-world domains and compare against LLM-based, Transformer-based, linear, and self-supervised forecasting baselines, reporting averaged MSE/MAE over multiple horizons.
summary: The paper studies context-aware time series forecasting, where the goal is to predict future multivariate trajectories from historical signals together with auxiliary contextual information such as textual event or domain descriptions. The proposed framework, TokenCast, discretizes time series via a VQ-style tokenizer with reversible instance normalization (to avoid future leakage), injects these discrete indices into the shared vocabulary of a frozen large language model through a learned unified embedding layer that aligns time-series tokens and text tokens, and then generatively fine-tunes the model to autoregressively produce future tokens that are decoded back to continuous values. The approach is presented as a unified pipeline that allows an LLM backbone to consume numeric history and contextual signals without altering its core architecture beyond the shared embedding layer. The method is evaluated on six real-world datasets spanning economics, public health/mobility, web traffic, stock markets, and environmental sensing using MSE/MAE across multiple horizons and baselines, and the paper reports lower errors on most datasets plus ablations linking gains to alignment, generative training, and contextual conditioning.
weaknesses: 1. Dataset descriptions contain internal inconsistencies (e.g.,  the Economic dataset describes as daily in the main text but as monthly macroeconomic data in the appendix), which obscures the exact sampling frequency and temporal structure assumed in training and evaluation.
2. The reported MSE/MAE averages lack standard deviations, confidence intervals, or significance tests, which limits assessment of robustness when baseline performance is numerically close.
3. The paper only sketches how contextual features are constructed, temporally aligned, and used at inference time, and this under-specification affects reproducibility and the scope of claims about context-driven forecasting.
4. Figure/table references are inconsistent. In Section 4.1.1, the panel summarizing domains, frequencies, lengths, and variable counts is captioned as Figure 3 but referred to as Table 3.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. I am interested in how the model's performance would change if it only outputs time series tokens, instead of a mixture of time series and textual tokens.
2. I am confused by the organization of the input tokens. In the text, the paper states that time series tokens are placed in front of textual tokens. However, in Figure 2, the textual tokens appear in front of the time series tokens, which seems inconsistent.
3. In the stage of the time series tokenizer, TokenCast employs a TCN as a causal encoder. The choice of convolution kernel length is likely to have a significant impact on performance, and it would be helpful to include an ablation study to examine this effect.
soundness: 2
strengths: 1. The paper introduces a novel LLM-driven framework, named TokenCast, for time series forecasting by leveraging LLMs to utilize unstructured contextual information.
2. The paper is clearly written and well-organized, making it easy to follow the main ideas. The methodology is technically sound and clearly explained.
summary: The paper proposes TokenCast, an LLM-driven framework for context-aware time series forecasting, which consists of three stages: time series tokenizer, modality alignment, and supervised fine-tuning. Experimental results show that TokenCast achieves strong performance.
weaknesses: 1. The discussion of related work on contextual information integration could be strengthened. While many existing approaches incorporate numeric contextual signals to enhance forecasting, the integration of unstructured contextual information requires cross-modal alignment strategies. Several recent studies have explored this direction; however, this emerging line of work is not sufficiently discussed or contrasted with TokenCast.
2. In line 181, the paper states that RevIN may risk leaking future information. However, this claim might not be fully justified, as RevIN typically computes normalization statistics (e.g., mean and standard deviation) based only on the lookback window within the input sequence.
3. In line 82, the paper states that it is unclear whether time series forecasting can be addressed through autoregressive generation over discrete tokens. However, this direction has been explored in prior work. For example, Chronos and AutoTimes both employ a decoder-only architecture and transform numeric time series into discrete tokens via value-based quantization.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: - In Line 191, there is a T that denotes the number of latent vectors, but there is no explanation of how T is determined or computed. Could the author provide more details of the encoder?
soundness: 2
strengths: - The framework is well-motivated and clearly presented.
- The proposed method is extensively evaluated on various real-world datasets, covering diverse domains such as healthcare, finance, and environmental monitoring. TokenCast consistently outperforms existing baselines.
summary: This paper introduces TokenCast, a novel framework for context-aware time series forecasting via symbolic discretization. By transforming continuous time series data into discrete tokens and embedding them into a semantic space shared with contextual features, TokenCast leverages the generative and reasoning of pre-trained LLM. The proposed approach demonstrates superior performance across various real world datasets and provides a new perspective on integrating time series data with contextual information.
weaknesses: - The paper claims to leverage the modeling and reasoning capabilities of LLMs, which are generally associated with larger-scale models. However, the experiments primarily rely on a relatively small LLM (Qwen2.5-0.5B). This raises questions about whether the claimed reasoning capabilities are being fully utilized and whether such a small-scale LLM can truly demonstrate the generative and reasoning power the framework aims to exploit. The choice of model size contradicts typical expectations for LLM usage and requires further explanation.

- The multi-stage training process (e.g., symbolic discretization, cross-modal alignment, and generative fine-tuning) introduces significant computational overhead. It seems that each stage requires separate optimization. The training cost and efficiency of this approach compared to existing baselines are not adequately discussed.

- The overall design of the framework lacks novelty. For example, as mentioned in Line 186, normalization is a standard component in many existing models. Additionally, the contextual feature selection is similar to existing TimeLLM.