Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Authors: Siru Zhong
Weilin Ruan
Ming Jin
Huan Li
Qingsong Wen
Yuxuan Liang
Venue: ICML 2025 poster (2025)
Keywords: Time Series Forecasting
Vision-Language Models
Primary area: deep_learning->sequential_models_time_series
TL;DR: A novel multimodal framework that leverages pre-trained Vision-Language Models (VLMs) to bridge temporal, visual, and textual modalities for augmented time series forecasting.

Abstract: Recent advancements in time series forecasting have explored augmenting models with text or vision modalities to improve accuracy. While text provides contextual understanding, it often lacks fine-grained temporal details. Conversely, vision captures intricate temporal patterns but lacks semantic context, limiting the complementary potential of these modalities. To address this, we propose Time-VLM, a novel multimodal framework that leverages pre-trained Vision-Language Models (VLMs) to bridge temporal, visual, and textual modalities for enhanced forecasting. Our framework comprises three key components: (1) a Retrieval-Augmented Learner, which extracts enriched temporal features through memory bank interactions; (2) a Vision-Augmented Learner, which encodes time series as informative images; and (3) a Text-Augmented Learner, which generates contextual textual descriptions. These components collaborate with frozen pre-trained VLMs to produce multimodal embeddings, which are then fused with temporal features for final prediction. Extensive experiments demonstrate that Time-VLM achieves superior performance, particularly in few-shot and zero-shot scenarios, thereby establishing a new direction for multimodal time series forecasting. Code is available at https://github.com/CityMind-Lab/ICML25-TimeVLM.

Time-VLM: Exploring Multimodal Vision-Languaae Models for Augmented
Time Series Forecasting
Siru Zhong' Weilin Ruan
Ming Jin
Huan Li
Qingsong Wen Yuxuan Liang'
Abstract
Tempocal to Visual Trarsfommtion
Unage
Recent advancements in time series forecasting have explored augmenting models with text vision modalities to improve accuracy. While text provides contextual understanding. it often lacks fine-grained temporal details. Conversely, vision captures intricate temporal patterns but lacks se- mantic context, limiting the complementary po- tential of these modalities. To address this, we propose Time-VLM, novel multimodal frame- work that leverages pre-trained Vision-Language Models (VLMS) to bridge temporal, visual, and textual modalities for enhanced forecasting. Our framework comprises three key components: (I) Setrieval-Augmentt Learner, which ex- tracts enriched temporal features through mem- ory bank interactions; (2) Vision-Augmented Learner, which encodes time series as informa- tive images; and (3) Text-Augmented Learner, which generates contextual textual descriptions. These components collaborate with frozen pre- trained VLMMs to produce multimodal embed- dings, which are then fused with temporal fea- tures for final prediction. Extensive experiments demonstrate that TiMe-VI.M achieves superior performance, particularly in few-shot and zero- shot scenarios, thereby establishing new di- rection for multimodal time series forecasting. Code is available https: : / //t.thb.b...om com/ CityMind Laainndd-ab///// Lab/ ICML25- rimeVLM.
CNN
Day's
Reskie
Grach
Day'
Eidia
Viddon
VIT
2000
Lony [Context]: Time are residental electticlls usage pateess [Dataset]: Hledractty consumction posks. on wonkiay wenings drsps durtng dayi mn Paral riverses res days. with
Cora Net
Saboraztr
Cook NeXt
[Statistic) The
Graysacle Image
metian smediats, The overall supwardl
Visio-TTesl
Text
Figure 1: Our Time-VLM combines text (Right) and vision (Left) modalities augment time series forecasting.
(Idrees et al., 2019), climate (Karevan & Suykens, 2020), en- ergy (Deb et al., 2017), and transportation (Zheng & Huang, 2020). Accurate forecasting supports proactive risk mitiga- tion, efficient resource allocation, and data-driven decision- making. Traditional models like ARIMA, while historically dominant, struggle to capture complex nonlinear patterns. In contrast, deep learning methods-from recurrent neural networks (RNNs) (Medsker et al., 2001) to Transformer- based architectures (Li et al., 2019; Wu et al., 2021; Zhou et al., 2021; Liu et al., 2022a; Zhou et al., 2022; Nie et al., leverage innovations such as patch-based feature extraction, auto-correlation mechanisms, and frequency de- composition model complex temporal dynamics. Despite their success, these models often fail to generalize across domains or adapt to data-limited scenarios, particularly few- shot and zero-shot settings (Liang et al., 2024).
1. Introduction
To overcome these issues, researchers have turned to aug- menting time series forecasting with additional modalities, such as text and images, which provide complementary in- formation that can enhance predictive accuracy:
Time series data captures temporal signal evolution and underpins forecasting in diverse domains such as finance
'The Hong Kong University of Science and Technology (Guangzhou), China Grrffftth University, Australia *Zhejiang Uni- versity, China 'Squirrel Ai Learning, USA. Correspondence to: Yuxuan Liang Zyuxliang@outlook.oo @omm.....com...
Text-Augment Augmentedd Models: Textual data offers valuable semanties crucial for accurate forecasting. For instance, contextual descriptions or dataset statistics can signifi- cantly enrich the understanding of time series patterns (Figure 1, right). Methods like Time-LLM (Jin et al., 2024) and UniTime (Liu et al., 2024b) leverage the supe- rior pre-trained inference capabilities of Large Language
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TIMc-VLM:
Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Models (LLMs) by mapping time series into textual repre-
We propose the first framework that unifies temporal, vi- sual, and textual modalities by leveraging their comple- mentary strengths for enhanced time series forecasting.
sentations. However, these approaches encounter two key
challenges: (I) the modality gap between continuous time
series and discrete text leads to information loss during
We introduce a retrieval- -arieeea-- learner for hierarchi- cal temporal feature enhancement, vision-augmented learner for adaptive seue--seres-to-imaaee transformation, and a text-augmented learner for contextual prompt gen- eration, enabling seamless integration with VLMs.
representation alignment, and (2) pre-trained language
knowledge is rare for capturing fine-grained temporal
patterns, limiting their ability to learn nuanced dynamics.
Vision-Augmented Models: Visual representations of time suries-such as line graphs, recurrence plots, or grayscale images-enable models to exploit spatial pat- terns embedded within temporal data. By transforming sequences into images, methods such as CNNs. ViTs, and MAEs can extract hierarchical features that reveal latent temporal relationships (Figure 1, left). Recent studies (Wu et al., 2023b; Wang et al., 2024; Chen et al., 2024) have demonstrated the natural alignment between time series and vision, both are continuous and share struc- tural similarities, allowing pre-trained vision models to effectively encode temporal hierarchies. However, these methods lack semantic interpretabillty, restricting their capacity to incorporate domain-specifit knowledge,
Extensive evaluations show T'me-VIMM's strong perfor- mance, especially under data-scarce conditions, offering a brand new paradigm for multimodal time series research.
2. Related Work
Text-Augmented Models for Time Series Forecasting.
The success of LLMs has inspired their application to time series forecasting. Methods like LLMTime (Gruver et al., 2023) and LLM4TS (Chang al., 2023) tokenize time se- ries for autoregressive prediction but inherit limitations such as poor arithmetic and recursive reasoning. Approaches including GPT4TS (Zhou et al., 2023) and TimeLLM (Jin et al., 2024) project time series into textual representations to leverage LLMs' reasoning capabilities, yet face challenges like the modality gap and limited temporal adaptability of word embeddings. UniTime (Liu et al., 2024b) and Tim- eFFM (Liu et al., 2024a) incorporate domain knowledge and federated learning, respectively, but remain constrained by their exclusive dependence on textual modeling.
Despite advancements in text- and rision-augmente mod- els, integrating both modalities with time series remains underexplored. Current approaches often focus on single modalities, failing to harness their combined strengths. To address this gap, we propose Time-VLM, a novel frame- work that leverages pre- trained VLMs enhance time se- ries forecasting by unifying temporal, visual, and textual information. VLMS provide a promising foundation, as they excel at aligning visual and textual modalities, making them well-suited for incorporating temporal information to unify the three modalities. By projecting time series into a uni- fied vision-language semantic space, Time-VLM enables rich cross modal interactions, combining the strengths of both modalities while mitigating their individual limitations. In this paradigm, each modality contributes uniquely: text provides semantic context, vision captures spatial-tempont patterns, and time series encodes sequential dynamics.
Vision-Augmented Models for Time Series Forecasting.
Vision provides a natural way to preserve temporal pat- terns in time series data. Early approaches apply CNNs to matrix-formed time series (Li et al., 2020; Sood et al., 2021), while TimesNet (Wu et al., 2023b) introduces multi- periodic decomposition for unified 2D modeling. Vi- sionTS (Chen et al., 2024) pioneers the use of pre-trained visual encoders with grayscale time series images, and TimeMixer++ (Wang et al., 2024) advances the field through multi-scale frequency-based time-image transformations. Despite their effectiveness in temporal modeling, these methods often lack semantic context, limiting their ability to utilize high-level contextual information for prediction.
Specifically, Time VLM introduces three key components: (1) Augmentdl- Learner that processes raw time series data through patch-based feature extraction and mem- ory bank interactions to generate enriched temporal repre- sentations, capturing both local and global dependencies: (2) a Vision-Augmented Learner that adaptively transforms time series into images using multi-scale convolution, fre- quency encoding, and periodic encoding, preserving both fine-grained details and high-level structures; and (3) a Text- Augmented Learner that generates rich textual context (e.g., statistics and dataset descriptions) to complement the visual representations. These modules collaborate with VLMM to integrate temporal, visual, and textual modalities, producing accurate forecasts through a fine-tuned predictor.
Vision-Language Models.
VLMs like ViLT (Kim et al., 2021), CLIP (Radford et al., 2021), and ALIGN (Jia al., 2021), have transformed mul- timodal understanding by aligning image and text repre- sentations. Recent progress, like BLIP-2 (Li et al., 2023) and LLaVA (Liu et al., 2023), further enhance multimodal reasoning. However, VLMs remain underexplored for time series analysis. Our work bridges this gap by leveraging VLMs to integrate temporal, visual, and textual modalities, addressing the limitations of unimodal approaches.
Our key contributions can be summarized as follows:
2

---

Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Predictur
Gate
Erused
Lasmable [dass] takmn
postzon enooding
Outper Poston Emboosdings
Production Besults
Model: oype mark
Patci postiio encoding
Multimodal Fusion Network
000
000
Mull Head Selt-A Mention Pooling
[CLS]
VIM Vision Anooder
[CLS] S]
Text Encoder
Memnoy
Image Interpolation & Nomalizatonn
Electriityy
consunptton
peak
200n
Honk
[Tadd]: the nett (pmd, Jan] aags using past (sog lm) stepe. [Domsing: Electiiily comuumptin tpicalll peeks as noon, drep at right. [Statiated]: Ingut value mages from
stali-scaleeeee Coavolution Projection
cmaast with median of aredarr> and an overt sdowneerds tran,, and tug lags (lags). [imagge The time
Patch Frididitings
Fost Foccion Transfom
Progunccy Racuder
converted into on image usiny multi scale com olutinaal layers and pertodacity encoding highlighttng trnnk, pertodic pattoms, multi frutures for tommosting
Pos tinnl Embolding
Porodkiity
Periodicily Preanding
Cosite Pesodacity
Lansee
N Nw. / MM
W
Palchily
Restugee
MV M
Domazn Contect
Stalistic Coumpete
Image Desorigtion
Seane
522, s00, Im, vam))
Time Scnes
Retrieval-A Aggrleeel-uummeteed ugmented L Learner (RAL)
Vision-A ugmented Learner (VAL)
c) Text-A ugmented Learner (TAL)
Figure 2: Overview of the Time-VL. framework.
3. Methodology
3.1. Retrieval-Augmented Learner (RAL)
To address the limitations of single-modality approaches and leverage the complementary strengths of visual, tex- tual, and temporal modalities, we propose TLMe-VL,, unified framework that integrates these modalities for en- hanced time series forecasting. As illustrated in Figure 2, the framework comprises three core components:
The RAL module extracts high-level temporal features via patch-based processing and serieval-augmeent memory mechanisms. It dynamically retrieves historical patterns and integrates them with current observations, adapting to complex time series structures. It operates in two key stages,
The input time series Zene E Bx is divided into overlapping patches of length pl with stride st, where B, L, and D denote the batch size, sequence length, and number of variables, respectively. Each patch is linearly projected into a dmoda -iit--dimminonall latent space, and positional embeddings are added to preserve temporal order. This yields patch embeddings E, E X where N₂ = 4ーي! + represents the total number of patches.
Patch Embedding:
Oetrieval-Augmente Learner (RAL): Extracts tempo- ral features from raw time series patches and maintains a memory bank to refine patch embeddings through multi- head self-attention and pooling mechanisms, thereby pre- serving rich temporal representations and enhancing long- term dependency modeling for robust forecasting.
Vision-Augment Learner (VAL): Transforms time se- ries into informative threc-channel images through multi- scale convolutions, frequency and periodic encoding. The images are processed by a frozen VLM vision encoder to extract hierarchical visual features, capturing both fine- grained details and high-level temporal patterns.
Retrieval-Augmentted Memory: A memory bank with maximum capacity M stores historical patch representa- tions M E RR x During each forward pass, the cur- rent patch embeddings are averaged across the temporal dimension and added to the memory bank using a circular buffer update strategy, ensuring that the most recent pat- terns are retained. To better capture temporal dynamics, we introduce a hierarchical memory structure:
Text-Augmented Learner (TAL): Generates contextual textual prompts for input time series, including statistical features (e.g., mean, variance, trends), domain-specific context (c.g., electricity consumption patterns), and image descriptions. These prompts are encoded by frozen VI.M text encoder to produce textual embeddings.
Local Memory: Given current patch embeddings P R Bx N, Xomael we retrieve top-k similar patches from the memory bank M based on cosine similarity:
The image and text embeddings, extracted by the VLM, are integrated with temporal memory features via gated fusion mechanism, effectively capturing complementary in- formation to improve forecasting accuracy. These enriched multimodal features are then processed by a fine-tuned pre- dictor to generate precise and reliable forecasts.
(P.MM) =
(1)
where M E R X stores historical patch represen- tations. The retrieved patches are processed through a
3

---

Time-VLMs Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
two-layer MLP to extract local memory features:
The concat tensor is processed through multiple convolutional layers to extract hierarchical temporal patterns. A ID convolutional layer captures local dependencies, transforming the input into R n Dx Hastes X J, where Haiden is the hidden dimension. Averaging along D yields xg XL, Two 2D convolutional layers follow: the first halves the channel dimension, and the second maps features C' output channels, producing the final output capturing both local and global temporal structures.
Multi-scale Convolution:
M local = (L,(ttpk(E))))),
i= i=1....,B B.
(2)
These features are averaged across patch dimension and combined with the original P via a residual connection.
Global Memory: To capture long-range dependencies, we apply multi-head self-attention over the current patch embeddings P. yielding contextualized representations:
Attn P) = MultiHead(Q.. V),
(3)
Image Interpolation & Normalization: is resized to the desired image dimensions (H, W) using bilinear interpolation. For a target pixel (zr, y), the interpo- lated value I(x, y) is computed as follows:
The output tensor
where Q, K, V are linear projections of P. The global memory is obtained by temporal averaging:
1
Mg Mglabal
N Aun (P)
(4)
2
I v)
I (1is uz) Wajs
(8)
The two memories are fused via gated mechanism:
1=1
(5) where a is a learnable gating parameter that adaptively balances the contribution of local and global memory.
Minset- Moral + (1 - a). Mglnhul,
I Iaw
Min 1rw
Inorm = 255
Max (I
Min
where (æi, Mj) are the coordinates of the four nearest neigh- bors, Wy are weights based on relative distances, and <= =0 prevents division by zero. Pixel values are scaled to [0. 255] via min-max normalization, producing the nor- malized image Inorn E B (C is the number of channels). This ensures alignment with the VLM vision encoder's 's input distribution for effective feature extraction. Example images and descriptions can see Appendix C.
The output Mused E BxN, x dhnnks captures high-level tem- poral patterns. This fused representation supports dynamic retrieval and integration with other modalities (e.g., vision or text), enabling adaptive context-aware forecasting.
3.2. Vision-Augmentted Learner (VAL)
The VAL module adaptively transforms the input time series
Tene RR Bxxxx I.x into image representations, enabling fine- grained and high-level temporal pattern extraction via the VLM vision encoder. The process is in three steps:
3.3. Text-Augmented Learner (TAL)
The TAL module provides contextual textual representa- tions, either pre-defined (e.g.. expert annotations) or dynam- ically generated, offering flexibility across diverse scenarios.
To capture spectral and temporal dependencies, the VAL module applies two complementary encoding techniques the input time series, explicitly adding frequency and time-domain information.
Frequency and Periudicity Encoding:
For dynamically generated prompts, TAL extracts key sta- tistical properties from the input time series, including:
Statistical Properties: Value range (min//mxx, central tendency (median), and overall trend direction.
1. Frequency Encoding: A Fast Fourier Transform (FFT) extracts frequency components from raw input Tenc as:
Contextual Information: Periodic description, task-
FFT (Zene)
Fene (t) 2
(6)
specific parameters (input window length and forecasting horizon), and domain-specifee dataset characteristics.
These features are formatted into structured textual prompts, such as the example shown in Figure 2(c). When domain- specific knowledge is available (e.g., in medical diagnostics or financial analysis), TAL incorporates pre-defined tex- tual descriptions, which are combined with the dynamically generated prompts to enhance contextual understanding.
where k is the frequency index. The resulting frequency features are concatenated with the input time series, re- sulting in a tensor of shape Bx
2. Periodicity Encoding: Temporal dependencies are en- coded using sine and cosine functions for each time step:
1mT encoding(() sin (2m²)
2tt
coS
The final textual inputs are processed by the VLM text encoder, producing contextual embeddings that complement both visual and temporal features. This dual-path design supporting both static and dynamic textual ensures strong generalization across a wide range of applications, from generic forecasting tasks to specialized domain scenarios.
where P is the periodicity hyperparameter. These encod- ings are concatenated with the input time series, resulting in a tensor of shape IR. Dxx3 Complete periodic pa- rameter settings can be found in Appendix A.
4

---

Time-VLA VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
3.4. Multimodal Fusion with VLMs
3.5. Optimization
The multimodal fusion pipeline integrates visual (VAL), textual (TAL), and temporal (RAL) information, leverag- ing their complementary strengths to enhance time series forecasting. It consists of three key steps:
The model is trained end-to-end using mean squared error (MSE). Given historical observations X R RNX (with N variables and T time steps), the objective is to predict future values Y E IR. NxH over H steps:
The generated im- ages and text are processed by a frozen VLM (e.g, ViLT or CLIP), producing multimodal embeddings of shape B xdy, where B is the batch size, L is the sequence length, and d, is the VLM's hidden dimension. These em- beddings capture visual and textual context, leveraging the VLM's 's pre-trained multimodal understanding capabilities.
Multimodal Embeddings Extraction:
1 11 H
L
Σ 1=1
Y - YAII,
(15)
where Y, and y, denote predicted and ground-truth values at step h. The pre-trained VLM is kept frozen, and only lightweight components are optimized during fine-tuning:
RAL: Patch embedding, memory retrieval, and attention modules for temporal pattem learning;
Temporal Feature Fusion:
To address the distribution shift
between temporal and multimodal features, both modal- space.
VAL: Freqqeency/periodiiitt encoding and multi-scale
ities are projected into a shared dueder- Temporal memory embeddings F Fm from RAL encode high- level temporal patterns and serve queries in cross- modal multi-head attention (CM-MHA) mechanism, while the mul- timodal embeddings F mm from the VLM serve as keys and values. The CM-MHA is defined as:
CNNs for visual representation generation;
Prediction Head: A gate network and linear projection
fuse multimodal features to generate final forecasts.
This strategy adapts the VLM to time series forecasting, achieving robust performance with minimal overhead.
CM-MHA K. V) = -Cat(head,,... headd,,,,,,,,,,,,,,, W (10)
4. Experiments
QQW? (KW.KT ddx
Datasets and Metrics. We evaluate Time- Mime-VLM on seven widely used time series datasets across diverse domains: en- ergy (ETTh1, ETTh2, ETTm1, ETTm2), weather, electricity (ECI.), and traffic (Zhou et al., 2021; Lai et al., 2018). These datasets are commonly used for benchmarking long-term forecasting models (Wu et al., 2023a), and vary in frequency, dimensionality, and temporal characteristics. For short-term forecasting, we use the M4 benchmark (Makridakis et al., 2018), which includes marketing data across multiple fre- quencies. Performance is measured using Mean Absolute Error (MAE) and Mean Squared Error (MSE), following standard evaluation practices in this field. Additional details are provided in Appendices A. and A.3.
head;- = softmax
vWY. (11)
where Q = From WQ, K = Fmm W and V = mm WV Here, W.O. W.K, W., and WO are learnable projection matrices. dę = dmodel/h h is the head dimension, and h is the number of attention heads. This mechanism aligns and integrates temporal and multimodal features, capturing both Line-grained patterns and high-level context. A residual connection and layer normalization stabilize training:
Fam = LayerNorm (Fo F + CM-MHA(Q, K)). (12)
Baselines. We compare Time-VLM with state-of-the-art time series models, including augmennedt methods like TimeLLM (2024), GPT4TS (2023), and LLMTime (2023); vision-augmented methods like TimesNet (2023b); tra- ditional deep models like PatchTST (2023), ESTformer (2022), Non-Stationary Transformer (2022b). FEDformer (2022), Autoformer (2021), Informer (2021), and Reformer (2020); and recent competitive models like DLinear (2023), LightTS (2022), N-HiTS (2023), and N-BEATS (2020). No- tably, Time-VLM is the first framework combining three modalities for time series forecasting. Performance results for some baselines are cited from (2024a) where applicable.
A gated fusion mechanism further enhances the output by dynamically weighting each modality:
G o (W Femm F mm]] + by),
(13)
F Frused G C F Fann (1 G) O F mm:
(14)
where Wg and by are learnable parameters, and (-)) is the sigmoid function. This gated mechanism adaptively bal- ances temporal and multimodal features for robust fusion.
The fused embedding is processed by a fine- tuned predictor, consisting of fully connected layers, to gen- erate forecasts ÿ E R Bx D By combining visual, tex- tual, and temporal modalities, the pipeline captures both de- tailed patterns and high-level context, leveraging pre-trained VLMs for enhanced forecasting across diverse scenes.
Forecasting:
Implementation Details. We compare Time- -Lnee-LM&&e against superior baselines using a unified evaluation pipeline un- der the same configurations as (Wu et al., 2023a) to ensure a fair comparison. ViLT (Kim et al., 2021)
5

---

Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Table 1: Few-shot learning on 5% training data. Results are averaged over forecasting horizons H ( 192, 336, 7203. Lower values
indicate better performance. Full results see Section B.I. Red: best, Blue: second best.
Metnods
Time VLMsow
Time LLM
GPT4TS
DLinnar
PachTTT (2023)
TimesNet
FiiDtoemer
Autotormer
Stationryy (2022h)
ETSfommer
LightTS
Informer
Refermer (2020)
(Ours)
(2024)
(2023)
(2023)
(2023a)
(2022)
(2021)
(2022)
(2022)
(2021)
Mesie
MSE
MAE
MSB
MAE
MSB
MAE
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
67TR1
0,442
0.453
0.227
0.543
[0668]
0,500
0.750
0.611
1.094
0.569
0,925
0.647
0.65
0.562
0.722
0.598
0.943
0.646
1,189
0.839
1,451
0,903
1,225
0.817
1.241
0,835
ETT12
0.354
0.402
0.382
0.418
00,000
0,433
00,094
0.577
10:27
0.615
0.439 02,48
0.463
0.454
0.441 0.457
0,470
0,489
00,009
0.081
3,206
1,268
3,922
1,653
3,527
1,472
F77ml
0.364
0.385
0.425
0.434
[0,472
0.450
[0,600
0.417
[0.52
0.476|
0
0.717
0.561 0.730
0.592
0.796
0.630
0.857
0.598
1,125
0.782
1.123
0.765
1,363
0.791
1,264
0,826
FTTm2
0.262
0.323
0.274
0.325
[0.308
0,346
[0.999
0.426
[0.314
0.352
0.344 0,372 0.388
0.404
0.368
0,433
00.341
0.312
01:344
0.547
1,415
0,871
3,658
1,489]
3,561
1,437
Weather
0.240
0.280
0.260
0.309
[0.263
0.301
|0.263
0.308
[0.269
0.303 0.299 0,318 0.309
0.353
0,310
0.353
0.327
0.326
0.333
0.371
0,,00
0.345
0.584
0.527
0.447 0.453
BCL
0.218
0.315
0.114
0.268
[0.178
0.203
[0.176
0.275
[0.181 0.277
[0.002 0:453
0.266
0.353
0.346
0,404
00.22
0.603
0.300
0.688
[08878
0,725
1,281
0.929)
1,289 0,904
Traffic
0.558
0.410
0.423
0.208
[0.434
0.305 |0.450 0.317 [0.418 0.296
0 0.867 0.403 0.666 0.423 0.833 0.502 1,536 0.839 1,859
0,927
1.557
0.795
1.591 0.832]
1.618 0.851
Table 2: Few-shot learning on 10% training data. We use the same protecol in Table Full results see Section B.
Metaods
Time-VI Masi
Time-I
GPT4TS
innar
PaccTTT
TimexNat
PRDifomer
Astoformer
Stationory (20226)
RTSfomner (2022)
LighTS (2022)
Informer
Reformer
(Ours)
(2024)
(2023)
(2023)
(2023)
(2023a)
(2022)
(2021)
(2021)
(2020)
Menric
MSB
MAE
MSB
MAE
MSE
MAB|
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE|
MSE
MAE
MSB
MAB|
MSE
MAE
0.431
0.442
0.556
0.522
0.590
0,525
00,691
0.600
[.633
0,542
0.899
0,628
0.639
0.561
0.702
0,596
0,915
0.639
1.180
0.834
1,375
0.877
1.199
0.809
1.249 0,833
BTTR2
0.361
0.405
0.370
0.394
|0:397
0.431
|0.605
0.538
[0.45
0.43|
|0:49 0.465
[0:466 0:475
0.488
0.499
0.460
0.455
0.894
0.713
2.655
1.160)
3,872
1.513]
3.485 1.456
FT
0.360
0.382
0.404
0.477
[0.464
0.441
[0.411
0.429
p.501
0.466
(.677 0.537
[0.22
0.605
0.802
0.628
0.7997
0.578
0,980
0.714
0.971
0.708
1.192
0.821 [ |1426 0.856
KTTw2
0.263
0.323
0.322
0.323
|02203
0.335
00336
0.368
[0:206
0
0.330 0.353
[0:463
0.488
0.930
0.332
0:366
0.447
0.487
0.9967
0.756
3.370
1.400|
3,978 1.587
Wootter
0.233
0,274
0.238
0.273
[0.238
0.275
[0.241
0.283
[.242
0.279
[0.279 0.301
[0.284
0.324
0.300
0.342 0.318
0.325 0.318
0.360
00.289
0.322
0.597
0.495
[0,565 0.469
RCL
0.198
0.291
0.175
0.230
|0.125
0.209
[0.150
0.280
[0.180
0,273
0.323 0,392
|0.366
0.427
(0451
0
00:544
0.480
0.660
0.617
00:441
0.489
1.195
0.891
[0,955 0,768
Truffic
0.484
0.357
0.429
0.306
[0:400
0,510
(0,447
0.315 0.430 0.305 0.951 0,535
0.563
0:425
0.749
0,446|
1,453 0.815|
1.914
0.936
1,248
0.684
1.534
0
1.551 0,821
Table 3: Zero-shot learning results. Full results see Section B.2.
MAE. On Weather with 5% data, Time- VLM outperforms TimeLLM by 7.7% in MSE and 9.4% in MAE. The perfor- mance gap between Time- VLM and traditional models (e.g.. PatchTST, FEDformer) is particularly pronounced in few- shot settings, demonstrating the effectiveness of multimodal integration when data is scarce. This performance gain stems from the model's 's ability to leverage rich multimodal priors from pre-trained VLMs, while effectively capturing temporal patterns through memory-enhanced attention.
Methcds
Time VLM uos
Time Ma
LLMTiae (2023)
GPT4TS
DLisear
PaaTST (2023)
(Owrs)
(2004)
(2003)
(2023)
Menic
MSS
MAB
MSE
MAE
MAE]
MAE
MAE
MSE
MAE
ETTML 67712
10,35
0.365
10335
1.902
|.422/0.903
0442
0.s8 (0.80 0.405
BTTAL
BTT
10203
0.380
66275
11967
(0.41 0,422
R173R- 8371
10:00
0:000
66.77
11961
20.00 0.51
01733- 6377uu
1.297
0.353
6.222
0.341
1.507
(a335 0.370
0.370/0.300
(0.328 0,366 10.225
617mm 60772
[0,54
0.397
[0:61]
0.122
[0002
0.205
(443 0439
(0.44 0,45 (0:40 0.43
E77mi -e 877m2
[1364
0.319
10268
0.320
1.807
0860
pa01 0:448 (0.30 aaso (0.06 0:34
637m2- R8772
[1302
0.399
[0.354
0.30
[a592
0.705
jauts 0.445 jo.ss
0,411
(o.0s 0a25
BTTm22 B.T7ml [1432
0.426
|0.414
0.48
|1983
0084/a..0 0.567 josao 0,537 0.564 0.492
("vilt-b32 wineetned--nnee coco") serves as the de- fault vision- languaee backbone; CLIP and BLIP-2 are also supported. All models are trained with Adam (10-3 initial learning rate, halved per epoch), batch size 32, for up to 10 epochs with early stopping. Experiments run on Nvidia RTX A6000 GPU (48GB). More details are in Appendix A.2.
4.2. Zero-shot Forecasting
Setting. We evaluate the zero-shot forecasting capability of Time-VIM in cross-domain settings, where the model pre- dicts on unseen datasets by effectively transferring knowl- edge from unrelated domains. To ensure a rigorous com- parison, we conduct experiments using the ETT datasets as source and target domains, following previous setup (Jin et al., 2024). Results are summarized in Table 3.
4.1. Few-shot Forecasting
Setting, We evaluate the few-shot long-term forecasting capabilities of TiMe-VLM by testing its performance us- ing only 5% or 10% of the training data. This setting as- sesses how effectively Hime-VLM integrates pre-trained multimodal knowledge from the VLM with time series- specific features under minimal task-specific supervision.
Results. Time-VLM demonstrates strong generalizability, consistently outperforming or matching state-of of-the-art base- lines while using fewer parameters. For example, in the transfer setting. Time-VI.M achieves a 4.2% lower MSE and 0.5% lower MAE than TimeLLM. In ETTm1->ETTHE it outperforms TimeLLM by 7.1% in MSE and 3.6% in MAF, In ETTM2->ETTh2, T'me-VI.M performs competitively, closely matching TimeLLM with only 1.4% difference in MSE and 0.3% in MAE. These results highlight Time- VLM's ability to generalize across domains without fine-tuning, leveraging pre-trained vision- language priors for effective knowledge transfer.
Results. As shown in Table 1 and Table 2, Time Time-VLM VLM con- sistently outperforms most baselines across datasets. For example, on ETThl with 5% training data, Time-VLM re- duces MSE by 29.5% and MAE by 16.6% compared to the second-ber best model, TimeLLM. On E with 10% data, it surpasses TimeL.LM by 11.1 % in MSE and 10.5% in
6

---

Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Table 4: Short-tern term time series forecasting results on M4. The forecasting horizons are in [6, 48] and the three rows provided are weighted
averaged from all datasets under different sampling intervals. Full results see Section B.3.
Methods
TLMesLLMaasss (2024)
GPT-TS
TimesNet
PachTST
N-HHS
N-BEATS
ETStomer
LighTTS
DLinear
FEDformer (2022)
Stationay/ (2022h)
Autafoomer (2021)
Informr (2021)
Reformer
(Ours)
(2023)
(20238)
(2025)
(2023)
(2020)
(2022)
(2022)
(2023)
(2020)
SMAPE]
11,894
11.983
12,690
12,880
12,059
12.035
12:250
14.718
13,525
13,699
13.160
12.780
12,909
14.086
18.200
MASE
1.592
1.595
1.808
1.836
1.623
1.625
1,698
2,408
2.111
2.005
1.775
1.756
1.771
2.718
4.223
OWA
0,85
0,859
0,940
0.955
0.869
0.869
0.896
1.172
1,051
1.051
0.949
0.930
0.939
1,230
1.775
Table 5: Long-term forecasting results. We use the same protocol in Table 1. Full results see in Section B.4.
Methods
Time MLM
(Ours)
(2024)
LLM
GPTHTS (2023)
(2003)
PachTTT (202.))
TimesNer (20239)
D'Dioommer (2002)
Autodormer (2021))
Stationary (20226)
ETSformer (2022)
LigheTS (2022)
(2021)
Reformer
(2020)
Matric
MSE
MAE
MSS
MAE
MSE
MAE]
MSE
MAE|
MSE
MAE|
MSS MAE|
MSE
MAE
MSE
MAE
MAE
MSE
MAE
MSE
MAE
MSE
MAE]
MSE
MAE
ET751
00,405
0.420
10.00
0.425
0.465
0.45500422
J04422
0.437 [0.43 0.430
0:458 0.400
(0.400 0.450 [0.496 0.487 0.570 0.537 [0.542 0.510 [0491
0.479 1.040
0.705] 1,020 0.305
FT752
[0:34]
0.991
00.394
0.983
|0.38
0.41
0.446 (0.300 0.379 [0:414 0.47700.47
[0.437 0.449 [0:450 0.459 00.526 0,516
[0.490 0.452
[0/02) 0.543
4.43) 1,729 667% 2.191
FTTml
10.347
0.977
00.329
0.392
10.388
0.453
0:378
(8785 0.380|
(0:400
0.406]0:448
(0:448 0.452 [0.588
0,517 00488 0:456
[0:479 0:425
[0:45 0.437
(0.66) 0.734
(,799 0.671
FTTm2
90.248
0.311
[0.25]
0.313
|0.281
0:39900222
(02267 0:333
0.33310.255 0.315 (0.29] 0.333|0.905
10.005 0.349 [0:327 0,371 0.306 0,347 [0:293 0.542 [0:49 0.436 1:410 0.810] 1.479 0915
Weather
19.224
0.263
10.225
0.287
[0.25]
0220000.248
0.300 00.225 0.264 (0.259 0.267 10,009 0.360 (0,338 0,382 0.288 0,314 [0:27] 0.334 [0:26] 0,312 (0634 0.548 (.800 0.656
Blecivicily
10.172
0.273
[0.188
0.252
[0.167
(.263
[.266:0.166
10.166
0.263 [0.16
0.252
10.192 0.205 10.214 0.927 [0.227 0,338 0.193 0.206
[0.208 0,323
(0.229
0.329 (0:31) 0.397]
(.335 0:422
Frnffär
10.419
0.303
|0.388
0.364
|0414
0:354
0.394 (2.500
0.263 (0.000 o 0.356 jo.sin j.866
0.376 (0:628 0,379 0.624 0,540
(a.221 0,906
(0,222 0,392 (0.764 0.416
(3744) 0.422
4.3. Short-term Forecasting
4.5. Model Analysis
Setting. For short-term forecasting, we evaluate Time- VLM on the M4 benchmark, which includes marketing data at various sampling frequencies. Performance is measured using SMAPE, MASE, and OWA metrics, averaged across datasets and sampling intervals (see Table 4).
Ablation Studies. Table 6 evaluates the contributions of key components of Time-VLM including the RAL (with Local and Global Memory), VAL., and TAL.. The study highlights the importance of each module. Removing the RAL causes significant performance drop (35.6% in MSE), with its local (RAL.L) and global (RAL.G) branches contributing 17.2% and 4.3%, respectively, validating our hierarchical memory design. The VAL is also essential-removing it in- creases MSE by 9.0%, demonstrating its ability to preserve fine-grained temporal patterns via the VLMM vision encoder. In contrast, removing the TAL results in only minor degra- dation (2. 12.%% in MSE), likely due to the sparsity of textual tokens in the VLM output; for example, ViLT produces just 11 textual tokens out of 156 total, the rest being visual embeddings. While the TAL provides useful semantic con- text, its impact is limited by the scarcity of textual signals. Future work may explore VLMs with stronger language capabilities for better temporal- -emporal-semantic alignment.
Results. Time- demonstrates strong performance, con- sistently outperforming of baselines across all metrics. For instance, it surpasses the second-best model, Time-LLM, with improvements of 0.7% in SMAPE, 0.2% in MASE, and 0.5% in OWA, all while utilizing significantly fewer parameters and computational resources. Compared to traditional models like PST and N-HiTS, the perfor- mance gains more, highlighting the benefit of multimodal knowledge in short-term forecasting. These gains stem from Time- VLM's integration of temporal, visual, and textual data, capturing richer features for improved accuracy.
4.4. Long-term Forecasting
Table 6: Ablation study on multimodal components over forecast- ing horizons H E (96, 192, 336, 720] on Weather dataset, with MSE performance degradation (DDeg) measured for each variant.
Setting. We evaluate the long-term forecasting capabilities of Time- VLM across multiple horizons and datasets.
Results. As shown in Table 5, Time-VLL achieves com- petitive performance compared to state-of-the-art base- lines. On ETTh1, MiLee-MLMiii improves upon TimeLLM by 0.7% in both MSE and MAE, On ETTm2, it outperforms TimeLLM by 1.2% in MSE and 0.6% in MAE. However, on the Weather dataset, TIM slightly underperforms TimeLLM, with a 0.4% higher MSE and 2.3% higher MAE.
Horizon
Full
w/o RAI.
w/o RAI.J.
wo RAI_G
w/o VAL
w/o TAI.
0.160
0.273
0.185
0.165
0.213
0.165
192
0.203
0.297
0.235
0,210
0.237
0.208
336
0.253
0.325
0.295
0.265
0.255
0.258
720
0,317
0.369
0.375
0.330
0.309
0.322
Avg
0.233
0,316
0.273
0.243
0.254
0.238
DDeg
35.6%
17.2% 1
4.3% f
9.0%
2.1% 1
Multimodal and Few-/Zero-shot Analysis. To under- stand the source of Time- VLM's strong performance in data-scarce scenarios, we examine the relationship between RAL (temporal) and TAL/VAL (multimodal) embeddings. As shown in Figure 3, their complementary nature is evident. The left panel illustrates balanced gate weight distributions, indicating effective fusion of temporal and multimodal sig-
Overall, Time-VLM demonstrates robust performance across diverse tasks and datasets, highlighting its generaliza- tion and efficiency. By leveraging multimodal knowledge, it consistently outperforms state-of-the-ar baselines with sig- nificantly fewer parameters (143M vs. TimeLLM's 3405M), making it a practical solution for real-world applications.
Time-VLM's
7

---

Time-VLM, Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
nals. The right panel presents UMAP visualization reveal- ing distinct yet partially overlapping clusters, confirming successful integration of multimodal information while pre- serving
UMAP Visualization All Features
COLO
characteristics. The robust few- shot and zero-shot capabilities of TiMe-VI.M stem from its integration of temporal, visual, and textual modalities. Specifically, the RAL models temporal dependencies via memory bank interactions, enabling robust feature extrac- tion even with limited data. The VAL captures interpretable visual sath as trends, seasonality, and period- icity-in domain-agnostic representations, while the TAL. generates semantic descriptions that enhance generalization. Together, these components allow Time- VLM to leverage pre-trained multimodal knowledge, making it highly adapt- able to new tasks and domains with minimal finettunig.
UMAP
Interpretabilityy visualization of mul- timodal feature alignment via UMAP.
Figure 4:
UMAP Clasterng Visuatization
bj Pusion Gate Weight Distributions
Sup ailii dudlin
Val anda Fan Siiglt Weighs
pure image representations retain modality-specific charac- teristics. Similarly, COCO forms a completely separate cluster, highlighting sigmificant modality gaps. In contrast, COCO-Pair exhibits maximal overlap with time-series data, demonstrating strong cross-modal complementarity. The textual semantics in COCO-Pair bridge visual and temporal modalities, enhancing their alignment. Notably, COCO- Pair is positioned near the center of time-series clusters, suggesting its role as a key mediator between modalities.
Frequctyy
UMAP Dicannion
Figure 3: UMAP visualization (left) and gate weight distri- butions (right) of of multimodal and temporal embeddings.
These observations of relying on motivate Time-VLM's design: instead projections, we embed time series into multimodal space for richer semantic understand- ing. The model's strong performance in data-scarce settings stems from pre-trained VLM knowledge. With more time series data, better alignment of multimodal embeddings is expected, further enhancing performance.
Interpretability Analysis. We investigate how pre-trained VLMs can be leveraged for time series forecasting by ana- lyzing the alignment between visual, textual, and temporal representations. To this end, we sample 400 image-text pairs from MSCOCO, the primary pre-training dataset for VLMs, and 200 samples each from time series datasets: ETT, Traffic, Weather, and ECL. Using UMAP, visualize four types of embeddings in a shared 2D space:
Computation Studies. Time-VLM demonstrates strong computational efficiency, as shown in Table 7. With only 143.6M parameters (1/20 of Time-LLM's 3404.6M), mem- ory usage scales from 1968 MiB (Weather) to 24916 MiB (Traffic), adapting to dataset complexity. Inference speed ranges from 0.2057s/iter (ECL) to 0,4809s/iter (ETThI), efficiently handling varying loads. In contrast, Time-LLM requires over 37GB of memory even for smaller datasets like ETTh1 and ETTh2, making infeasible for larger datasets such as Weather, ECL, and Traffic. This highlights VLM's lightweight design and practical scalability.
multimodal embeddings derived from COCO- sam- ples using VLM, reflecting the model's general pretrained cross-modal knowledge, which VLM'ss main mo- tivation of the time series project to.
multimodal embeddings from time series- seffes-gnnerate image-text pairs processed through the same VLM, repre- senting Time-VLM task-specific augmentation.
Visual-only embeddings from COCO- Image samples tracted via VIT, reflecting pure visual knowledge.
Table 7: Computational efficiency comparison between Time-VLM and Time- LLM across datasets. "_" denotes memory exceeds 49GB, infcasible on a single GPU. Results are averaged over multiple prediction steps under consistent conditions.
Text-only embeddings from COCO- coco-Text samples encoded with BERT, capturing linguistic knowledge.
Method
Metic
ETIai
ETTh2
BTTml
ETiml
Weather
Traftic
As shown in Figure 4, both Imao-Image and COCO-Text form distinct, separate clusters, remaining isolated from time-series features. This suggests that while low-level vi- sual patterns COCO-Image resemble temporal dynamics,
Parun.
143.6
143.6
143.6
143.6
143.6
Time-VIM
(MiR)
2630
2630
2640
1968
24916
Spasi (xSten)
0.481
0.38
0.210
0.296
0.206
0.323
Parmm (M)
1404.
$409.6
$404.
5484.6
Time LLM
Mem. (MID)
37325
37723
37849
37849
Speed (witse)
0.507
0.553
0.9
0.205
8

---

Time- VL:: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Hyperparameter Studies. We analyze the impact of key hyperparameters on performance, shown in Figure 5. Se- quence length performs best between 96 and 1024 timesteps, with 512 being optimal for most datasets. Longer input introduce noise without significant gains, indicating that lo- cal temporal patterns are sufficient for accurate forecasting, The normalization constant peaks at 0.4, reflecting a balance between feature scaling and training stability. Model dimen- sion shows dataset-dependee behavior: values of 128-256 256 suffice for short-term datasets like ETThl and ETTh2, while longer horizons and nigher-dimensional data (e.g.. Traffic, Weather) benefit from larger dimensions (up to 512). sug- gesting greater capacity is needed to model complex dynam- ics and variable interactions. Similarly, the gate network ceeenniion-rsppossiiie for multimodal fusion-achiever optimal performance 256 for medium-range forecasts. For more challenging settings like long-horizor or high- variable inputs, increasing it to 336 or 512 further improves results, highlighting the importance of adaptive fusion in capturing complex cross- modal relationships.
evaluate the benefit of VLMs' pre-trained cross-modal align- ment, we construct modular baseline using separately trained vision and language encoders: ViT-B/16 and BERT- Base. As shown in Table 8, this custom combination un- derperforms all pretrained VL.S across metrics, achieving an average MSE of 0.348 versus 0.336 for ViLT, with no compensating gain in speed (0.17 s/iter). These findings highlight that the cross-modal alignment in VLMs offers a key inductive bias for time series modeling. Modular ap- proaches, lacking such a unified space, fail to match this the value of pre-aligned mul- timodal representations for efficient fusion and forecasting,
Table &: Comparison of different VLM variants on ET in terms of performance and computational efficiency.
VLM Type
Params (M)
Mem. (MiR)
Speed (sfiter)
MSE (xvg)
|MAE (avg)
VILT
128.9
1346
0.36
0.336
0.388
CLIP BLIP-2 Custom
168.4
1174
0.12
0.339
0.391
3763.1
252001
0.98
0.342
0.391
213.2
1474
0.17
0.348
0.397
5. Conclusion
B01
17Tl
BITS
BTTm
Worder
Isput Largth
We presented VLM, a novel framework that lever- ages pretrained VLMs to unify temporal, visual, and textual modalities for time series forecasting. By integrating the RAL, VAL, and TAL modules, Time-VIN, bridges modal- ity gaps and enables rich cross-modal interactions. Notably, time series data without requir- ing external information, enabling fair comparisons and demonstrating the ability to generate textual and visual rep- resentations internally for self-augmeenation. This design not only improves accuracy but also highlights the frame- work's robustness parieesss-atttlllrrly in domains where auxiliary data is scarce. Extensive experiments show that Time-V Vime-VLMM achieves superior performance across diverse datasets, espe- cially in few-shot and zero-shot settings, outperforming ex- isting methods while maintaining computational efficiency. Our work establishes a new direction in multimodal time series forecasting by highlighting the potential of VLMs in capturing both temporal dynamies and semantic context.
1.55
it operates solely
0:22
112
(O Medel Dincosion
(8) Gate Network Dimexsien
1.5
1.2
Cane
Figure 5: Hyperparameters sensitivity analysis on input length, normalization constant, dimension of model and dimension of gate network, reflected by MAE
Limitations. Despite its strengths, Time- has sev- eral limitations. First, the TAL module provides semantic context but has limited impact due to current VLMs' con- strained understanding of time series semantics. Second, while excelling in low-data regimes, full-shot performance slightly lags behind specialized unimodal models on cer- tain tasks (e.g., ECL, Traffic), suggesting room for domain- specific adaptation. Third, although computationally ef- methods, deployment on
VLM Variants Analysis. We conduct ablation studies on different VLM backbones and custom combinations to assess their impact on forecasting performance and computational efficiency. We evaluate three widely-used VLMs-ViLT, CLIP, and BLiP-2-with varying size, and find that increased model size does not improve accuracy. Notably, although BLIP-2 is the largest (3.7B parame- ters, 25GB+ memory), it underperforms ViLT and CLIP in terms of MSE (0.342 vs. 0.337) and exhibits slow train- ing speed (0.98 s/iter), limiting its practical use. In con- trast, lightweight models like ViLT (128.9M parameters, 1346 MiB memory) and CLIP (168.4M, 1174 MiB) achieve comparable or better accuracy at a fraction of the cost. To
ficient compared to resource-constraines devices remains challenging. These limitations suggest promising directions for future work, including temporally aware VLMs, improved time series imaging, visual distillation, and enhanced text encoders with stronger temporal reasoning. For details, see Appendix D.

---

Time- VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Acknowledgements
Jia, C., Yang. Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z.,, and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
This work is mainly supported by the Guangdong Basic and Applied Basic Research Foundation (No. 19254)515011994). This work is also supported by the Na- tional Natural Science Foundation of China (No. 62402414, No. 62402420), Guangzhou Municipal Science and Tech- nology Project (No. 2023A03J001 I), the Guangzhou In- dustrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), a grant from State Key Laboratory of Resources and Environmental Information System, Guang- dong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things (No. 2023B RE53B12120I10 101000007). and the CCF-DiDi GAIA Collab- orative Research Funds for Young Scholars.
Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.FF, Pan, S., and Wen, Q- Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR), 2024.
Karevan, and Suykens, J. A. Transductive Istm for time- series prediction: An application weather forecasting. Neural Networks, 125:1-9, 2020.
Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pp. 5583- 5594. PMLR, 2021.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning by integrating temporal, visual, and textual modalities for time series forecasting. While our ap- proach improves accuracy and cross-domain generalization, we acknowledge potential risks such as data privacy con- cerns, algorithmic bias, and increased computational costs. We encourage further research into mitigating these risks to ensure responsible deployment in high-stakes applications.
Kitaev, N., Kaiser, E.,, and Levskaya, A. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 international ACM SIGIR confer- ence on research development information retrieval, pp. 90-95--444 2018
References
Challu, C., Olivares, K. G., Oreshkin, B. N., Ramirez, G., Canseco, M. M., and Dubrawski, A. Nhits: neural hi- erarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 6989-6997, 2023.
Li, J., Li, D., Savarese, S., al. Blip-2: Bootstrapping language-ing eeguage--mage pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19742. PMLR, 2023.
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang. Y.-X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series fore- casting. In Advances in neural information processing systems, 2019.
Chang, C., Peng, W.-C., and Chen, T.-F. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained Ilms. arXiv preprint 2xXiv:2308.08466 2023.
Chen, M., Shen, L., Li, Z., Wang, X. J., Sun, J., and Liu, C. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters, 2024. URL https: // //axiiv.. org/ abs 2408. 17253.
Li, X., Kang, Y., and Li, F. Forecasting with time series imaging. Expert Systems with Applications, 160:1 13680, 2020.
Deb, C., Zhang, F., Yang, J., Lee, S. E., and Shah, K. W. A review on time series forecasting techniques for building energy consumption. Renewable and Sustainable Energy Reviews, 74:902-9955 924, 2017.
Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., and Wen, Q. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 6555-6565, 2024.
Gruver, N., Finzi, M.. Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. In Advances in Neural Information Processing Systems, 2023.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning, arXiv preprint trXiv:2304.08485 2023.
Liu, Q., Liu, X., Liu, C., Wen, Q., and Liang, Y. Time-ffm: Towards Im-empowered federated foundation model for time series forecasting. arXiv preprint arXiv:2405.1423 14252, 2024a.
Idrees, S. M., Alam, M. A., and Agarwal, P. A prediction approach for stock market volatility based on time series data. IEEE Access, 7:17287-17298, 2019.
10

---

Time-VLM, Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Liu, S., Yu, H., Liao, C., L.,,, Lin, W., Liu, X., and Dust- dar, S. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022a.
series forecasting. Advances in Neural Information Pro- cessing Systems, 11-22219-224330 2021.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d- 2drvitiatoon modeling for gen- eral time series analysis. In International Conference on Learning Representations, 2023a.
Liu, X., Hu, J., Li, Y., Diao, S., Liang, Y., Hooi, B., and
Zimmermann, R. Unitime: A language-empoweree uni-
fied model for cross-domain time series forecasting. In Proceedings of the ACM on Web Conference 2024, pp. 4095-4106 2024b.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang. J., and Long, M. Timesnet: Temporal 2d- 2d-vaiiation modeling for gen- eral time series analysis. In International Conference on Learning Representations, 2023b.
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-988 19893,, 2022b.
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11122 2023.
Makridakis, S., Spiliotis, E., and Assimakopoulos, V. The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting, 34(4): 802-808, 2018.
Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng. S., and Li, J. Less is more: Fast multivariate time series forecasting with light sampling-orient mlp structures. arXiv preprint 2rX 2022.
Medsker, L. R., Jain, L., al. Recurrent neural networks. Design and Applications, 5(64-67):2, 2001.
Zheng, J. and Huang, M. Traffic flow forecast through time series analysis based on deep learning. IEEE Access, 82562 82570, 2020.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, A time series is worth 64 words: Long-tern forecasting with transformers. In International Conference Learning Representations, 2023.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on Artificial Intelligence, pp. 11106-
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference Learning Representations, 2020.
11115, 2021.
Zhou, T., Ma. Z., Wen, Q., Wang. X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed trans- former for long-term series forecasting. In International Conference on Machine Learning, pp. 27268-27286. 2022.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763 PMLR, 2021.
Zhou, T., NIu, P., Wang, X., Sun, L., and Jin, R. One fits all: Power general time series analysis by pretrained Im. In Advances in Neural Information Processing Systems, 2023.
Sood, S. Zeng, Z., Cohen, N., Balch, T., and Veloso, M. Visual time series forecasting: an image-driven approach. In Proceedings of the Second ACM International Confer- ence on Al in Finance, pp. 1-9, 2021.
Wang, S., Li, J., Shi, X., Ye, Z., Mo, B., Lin, W., Ju, S., Chu, Z., and Jin, M. Timemixer++: general time series pattern machine for universal predictive analysis. arXiv preprint arXiv:2410.16032. 2024.
Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, Eis- former: Exponential smoothing transformers for time- series forecasting. In arXiv preprint arXiv:2202.01381. 2022.
Wu, H., Xu, J., Wang, J.. and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term

---

Time-VIA Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
A. Experimental Details
A.1. Dataset Details
Table 9: Summary of benchmark datasets. Each dataset includes multiple time series (Dim.) with varying sequence lengths, split into training, validation, and testing sets. Data are collected different frequencies across various domains.
Tasks
Datasct
Dim.
Series Length
Dataset Size
Frequency
Domain
Periodicity
ETTml
7
(96, 192. 336, 720)
(34465, 11521, 11521)
15 min
Temperature
96
Long-term
ETTm2
7
(96, 192, 336, 720)
(34465, 11521, 11521)
15 min
Temperature
96
Forecasting
ETTh1
7
(96, 192, 336, 720)
(8545, 2881, 2881)
hour
Temperature
24
FTTh2
7
(96, 192, 336, 720}
(8545, 2881, 2881)
hour Temperature
24
Electricity
321
[96, 192, 336, 720)
(18317, 2633, 5261)
hour
Electricity
24
Traffic
862
(96, 192, 336, 720}
(12185, 1757, 3509)
hour
Transportation
24
Weather
21
(96, 192, 336, 720)
(36792, 5271, 10540)
10 min
Weather
144
M4 Yearly 1
6
(23000, O, 23000)
Yearly
Demographic
1
M4 Quarterly
1
8
(24000, 24000)
Quarterly
Finance
4
Short-term
M4 Monthly
1
18
(48000, 0, 48000)
Monthly
Industry
3
Forecasting
M4 Weekly
13
(359, 359)
Weekly
Macro
4
M4 Daily
1
14
(4227, 0, 4227)
Daily
Micro
1
M4 Hourly
(414, 414)
Hourly
Other
24
The benchmark datasets used our experiments are summarized in Table These datasets span diverse domains, including temperature monitoring (E7TmI, ETTm2, ETThl, ETTh2), electricity consumption (Electricity), transportation (Traffic), and weather forecasting (Weather). Each dataset contains multiple time series with varying sequence lengths, split into training, validation, and testing sets. The datasets are collected at different frequencies, ranging from 15 minutes yearly intervals, and exhibit distinct periodic patterns. For short-term forecasting, we utilize the M4 benchmark, which includes datasets with yearly, quarterly, monthly, weekly, daily, and hourly frequencies, covering domains such as finance, industry, and demographics. This diverse collection of datasets ensures a comprehensive evaluation of our method.
A. DATASET DESCRIPTION
The datasets used in our experiments are described below:
ECL. Measurements of electric power consumption in one household with a one-minute sampling rate over 4 years. includes various electrical quantities and sub-metering values, totaling 2,075,259 measurements from house in Sceaux, France (December 2006 November 2010).
ETT: The Electricity Transformer Temperature (ETT) dataset, crucial for electric power deployment, contains years of data from two counties in China. Subsets E7Th/ and F77h2 provide I-hour-level data, while F77m/ offers 15-minute-level data. Each point includes the target "oil temperature" and 6 power load features, with 12/4/4 month train/val/test split.
Traffic: Hourly data from the California Department of Transportation, describing road occupancy rates measured by
sensors on San Francisco Bay area freeways.
Weather: Recorded every 10 minutes throughout 2020, this dataset includes 21 meteorological indicators, such as air temperature and humidity.
M4: A collection of 100,000 time series from the Makridakis Forecasting Competition, including yearly, quarterly, monthly, weekly, daily, and hourly data. The training sets have minimum observations of 13 (yearly), 16 (quarterly), 42 (monthly), 80 (weekly), 93 (daily), and 700 (hourly). Forecasts required are 6 (yearly), 8 (quarterly), 18 (monthly), 13 (weekly), 14 (daily), and 48 (hourly).
1.1. PERIODICITY PARAMETER
The Periodicity column in Table 9 specifies the periodicity hyperparameter P used in the periodicity encoding process. This parameter derived from the inherent characteristics of each dataset and reflects the dominant temporal patterns, such as
12

---

Time- VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
daily, weekly, or seasonal eycles. For example, in the ETTm/ and ETTm2 datasets, which are sampled every 15 minutes, the periodicity P 96 corresponds
daily cycle (24 hours x samples per hour). Similarly, for the E7Th/ and E77h2 datasets, sampled hourly, P 24 represents daily cycle. The Weather dataset, sampled every 10 minutes, has P = 144, reflecting daily eycle (24 hours x 6 samples per hour). For the M4 benchmark datasets, the periodicity values are set based on their sampling frequencies: P = 1 for yearly data, P = for quarterly and weekly data, P = 3 for monthly data, and P = 24 for hourly data. These values are used in the periodicity encoding formula:
{ sin
2
2 P
encoding(()
cos
(16)
where & is the time step and P is the periodicity hyperparameter. The resulting encodings are concatenated with the input time series, enriching the model's 's ability to capture temporal dependencies and periodic patterns.
A.2. Optimization Settings
A.2.1. MODEL ARCHITECTURE PARAMETERS
Time- VL.M consists of several key components, each with specific parameter configurations. Image representations are set to a size of 64 x 64, balancing computational efficiency and temporal information preservation. The model backbone utilizes a hidden dimension of d_model = 128, while the encoder-decoder structure comprises elaayers = 2 encoder layers and dlaayers = 1 decoder layer. A dropout rate of 0.1 is applied to mitigate overfitting during training. For efficient data loading, the model employs num_workers = 32 to parallelize data preprocessing tasks.
The gated fusion module is designed with dimension of duf'iioon 256, facilitating the effective integration of multimodal features. The VLM component generates multimodal embeddings with token length of vlm_,used_ 156 and hidden dimension of hidddedde_ddmm = 768, ensuring seamless compatibility with the pre-trained VLM's architecture.
Table 10: Default Model Architecture Parameters
Parameter
Default Value
Description
image size
64
Size of generated image representation
d model
128
Dimension of hidden embeddings
fusion
256
Dimension of gated fusion module
num_ workers
32
Number of data loader workers
laayers
2
Number of encoder layers
d.layers
Number of decoder layers
dropout
0.1
Dropout rate
vlm vlm_fused
156
Token length VLM multimodal embedding
vlm_hidden_dim
768
Hidden dimension of VLM
A.2.2. TRAINING PARAMETERS
We adopt comprehensive training strategy with both general and task-specific parameters. The model is trained with a batch size of 32 and an initial learning rate of 0.001, using the A optimizer. Early stopping with a patience of 3 epochs implemented to prevent overfitting. The training process employs Mean Squared Error (MSE) as the primary loss function and runs for maximum of 10 epochs. For time series processing, we use an input sequence length of 512 and prediction lengths of 96, 192, 336, or 720, depending on the task. The output dimension (c_out) varies by dataset: 7 for 1TTh1/h2mm1/// 1/m2, 21 for Weather, 321 for Electricity, and 862 for Traffic. The periodicity parameter is set to 24 for ETTh 1/h2, Electricity, and Traffic; 96 for ETTm 1/m2; and 144 for Weather, ensuring alignment with dataset-specific temporal patterns. A normalization coefficient of 0.4 is applied to stabilize training dynamics. The patch embedding module uses patch length of 16, a stride of 8, and padding of to process the input sequences. The temporal memory mechanism employs 8 learnable queries and 4 attention heads to capture high-level dependencies. Additionally, the training process leverages automatic mixed precision (AMP) to accelerate training while maintaining numerical stability.
13

---

Time-VLM, Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Table II: Default Training Parameters
Parameter
Default Value
Description
batch size
32
Training batch size
learning rate
0.001
Initial learning
training epochs
10
Number of training epochs
patience
3
Early stopping patience
loss
MSE
Mcan square error
seq_len
512
Input sequence length
7 (ETThl/t2/mm//n)
21 (Weather) 321 (Flectricity) 862 (Traffic)
out
Output dimension (dataset-specific)
pred_len
96/192/336/720
Prediction length
24 (ETThh//h///leccccttt//aaaaoo 96 (ETTm1/m2)
periodicity
Dataset periodicity (dataset-specific)
144 (Weather)
norm_const
0.4
Normalization coefficient
patch_len
16
Patch length
padding
8
Padding length
stride
8
Stride length
num.queries
8
Number of learnable queries for temporal memory
n.heads
Number of attention heads
A.3. Evaluation Metrics
For evaluation, we utilize mean squared error (MSE) and mean absolute error (MAE) for long-term forecasting. For short-term forecasting on the M4 benchmark, we adopt symmetric mean absolute percentage error (SMAPE), mean absolute scaled error (MASE), and overall weighted average (OWA), following the evaluation protocol N-BEATS (Oreshkin et al., 2020). OWA is specific metric used the M4 competition. The metrics are calculated as follows:
T MSE Σ h=1
H Ynh.
H (Y
E(Y,
Y
MAE
Σ
Yr
H
h=1
SMAPE 200
H
Y YR
MAPE 100
H
Y
YA
H
YN + [Yml
H
Y
h=1
h=1
MASE 1
Σ
H
Yn Y, 115 Y, Y s-
OWA
SMAPE
MASE MASEN
SMAPE
H-8
where is the periodicity of the time series, H is the prediction horizon, and Yn, and Yd the ground truth and prediction at time step h. respectively.
14

---

Time-VLMe Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
B. Complete results
B.I. Few-shot Forecasting
Table 12: Full few-shol learning results on 5% training data with forecasting horizons H E99, 192, 336, 720). A lower value indicates better performance. '-" means that 5% time series is not sufficient to constitute a training set. Red: the best, Blue: the second best
Methods
Time
VLM|
Time LLM|
GPT-TS
DLincur
PancTST
TimesNet
|
FEDDoemerr
Autaformr
Stationary
|
ETStomeer
Lightt'
Informer
Reformer
Merric
MSF
MAR
MSF
MAR]
MSF
MAF
MSF
MAF
MSF
MAF
MSF
MAF] MSE
MAF]
MSE
MAF
MSE MAF | MSF MAF]
MSR
MAF
MSF
MAF
MSF
MAF
96
0.437
01.35
0.563
0.664
0.543
0.508
0.547
0.503
11.557 0,519
0,8922
0,625
0,593
0.529
01.01
0.570
0.952
0.600
1.169
0.852
1.483
0.910
1.225
01812
1.198
0.795
ET7m2 E1/wl E77Y3 E77
192
0.450
0.458
0.639
01.40
0.748
0.580
0.730
01604
0.711
0.570
0.940
0.665
0662
0.563
0.725
0.402
0.941
0.645
1.221
0.853
1.525
0.990
1.249
0.878
1,273
0.853
336
0.460
0.465
0.768
0.626
0.754
0.595
0.984
0.727
0.816
0.619
0.945
0.653
0.731
0.391
0.761
0.624
0,935
0,641
1.179
0.832
1.347
0.870
1.202
0.311
1.254
0.857
Avg
0.442
0,627
0.543
0.681
0.560
0.750
0.61
0.604
0,560
0.925
0.647
0.656
0.562
0.732
0.508
0,943
0.616
1.169
0.839
1.451
0.903
1.225
0.817
1.241
0.835
0.302
0.336
0.397
0.376
0.421
0,442
0.445
(1,401
01421
00,409
0.420
0.390
0.424
0.423
0.468
0,408
0:423
0.675
0.619
2,022
1.006
3,837
1.508
3:753
1.538
192
0.361
0.406
0.400
0.418
0.441
0.617
0.542
0.452
0.455
0.483
0:464
0:457
0.465
0.496
0.504
0.497
0:468
0.845
0.697
3.534
1.348
3.975
1.933
3,516
1.473
336
0.398
0.434
0.905
0.432
0.405
0.439
1,424
0.849
0.464
0.469
0.499
0.479
0:477
0.183
0.486
0.496
0.507
0:451
0.905
0.727
4.063
1.451
3.956
1.520
3.312
1.427
720
Avg
0.354
0.402
0.382
0,418
0.400
0.433
0.694
0.577
0.827
0,615
0,439
0,448
0.463
0.454
0.441
0.457
0,470
0,489
0.809
0.681
3.206
1.268
3.922
1.653
3.527
1,472
96
0.314
0.357
0.316
0.317
0.286
0.415
0.332
01.77
10.999
0.414
0.606
0,518
0.628
0.544
0.726
0.578
0.823
02557
1.031
0.747
1.04K
0.735
1.130
0.775
1.234
0.7798
192
0.343
0.353
0,450
0.464
0.440
0.438
0.358
0.390
0:441
0.416
0.681
0.539
0.666
0.566
0.750
0.591
0.844
0.591
1.087
0.766
1.097
0.756
1.150
0.788
1.287
0.839
336
0.373
0.391
0,450
0,424
0,485
0.459
0.402
0.11
0.499
0.467
0.786
0.597
0.807
0.62%
0.851
0.689
0.870
0.600
1.138
0.767
1.147
0,775
1.198
0.304
1.288
0.842
0.425
11420
0.461
0.471
0.577
0.499
0.511
0-489
0.767
0.567
0.796
0.593
0.822
0.633
0.655
0.898
0.611
1.245
0.851
1.200
0.799
1.175
0.70
1.347
0.858
Avg
0.364
0.025
0,434
0.472
0.450
0.400
0.417
0.506
0,476
0.717
0.56]
0.730
0.592
0.796
0.620
0.596
1.125
0.763
1.123
0.765
1.163
0.791
1.364
0.626
0.169
0.360
0.174
0.261
0:192
0.280
0.236
0.326
11206
0 0.268
0.220
0.299
0.229
0.320
0.232
0.322
0.238
0:316
0,414
0.485
1.10K
0.772
3.509
1477
3.883
1,545
192
0.224
0.298
0.215
0-28
0.255
0.316
0.306
0.373
0.264
0.324
0.31
0.361
0.394
0.361
0.291
0.357
0.296
0.389
0.179
0.521
1.317
0.850
3:578
1475
3.553
1,484
330
0.222
11:38
0.271
0.330
0.318
0.353
0.380
0/425
0.734
0.367
0,338
0.356
0.378
0.427
0.478
0.517
0,383
0.380
0,552
0.555
1.415
0.879
3.501
1,473
3/460
1,400
220
0:375
11.97
12412
0.460
0.456
0.674
0.581
11454
0.432
0.500
0:465
0.525
0.510
0.553
0.538
0.475
0:445
0.701
0.627
1.822
0.984
3.806
1:531
1:445
1,400
Avg 0,262
0.323
0.274
0.333
0,308
0.366
0.399
0.426
0.314
0,352
0,364
0,372
0.381
0.104
0.388
0.433
0.341
0.373
0.534
0,547
1.415
0.871
3.658
1.489
3.581
1.487
Tou Fir Eledrinily Weather
96
0..176
0.231
0.172
0.263
0.175
0.330
0.18
0.242
00.17 0.224
0.2077
0.253
0.229
0.309
0.227
0.209
0.215
0:252
0.215
0.295
0.230
0.285
0.407
11497
11,406
0.435
192
0.316
0.363
0.234
0.271
0.227
0.276
0.228
0.283
0,230
0.277
0.271
0.307
0.265
0.317
0.278
0.333
0.250
0.307
0.294
0.331
0.274
0.323
0.620
0.545
0,446
0.450
336
0.364
01238
0.282
18:22
0.286
0.322
0.274
0.312
0.294
0.326
0.328
0.392
0.251
0.393
0,353
0.34M
0.359
0,398
0.318
0,355
0.649
01547
0,465
0.459
0.327
0.342
0.179
0.388
0.584
0.367
0-400
0.365
0:387
0.389
0.450
0.407
0.461
0.461
0.401
0.418
0.570
0.502
0.471
0.468
Avg
0.346
0,360
0.309
0.263
0.301
0.265
0.308
0.259
0.303
0.298
0,318
0.309
0.353
0.310
0.353
0.328
0.371
0.305
0.345
0.584
0.527
0.:47
0.453
0.185
0.296
0:147
0,242
0.143
6.241
0.150
0.251
0.145
0.344
0.315
0.389
0:235
0.322
0.297
0.367
0.454
0.518
0.697
0.638
0.639
0.689
1,365
0,919
1.414 0.855
192
0.191
0.500
0.188
0:341
0.159
0.255
0.163
0.263
0.163
0.260
0.318
0.396
0.247
0.34]
0.308
0.375
0.501
0.531
0.718
0.648
0.772
0.678
1.258
0.909
1,260
0,919
336
0.210
0.3115
0.117
0.277
0.129
6.274
0.375
0.27%
11183
0.261
0.340
0.415
0.267
0.356
0.354
0.411
0.574
0.57M
0.758
0.667
0.501
0.745
1,302
(1842
1,253
0
730
0.251
0.346
1:224
0.117
0.233
0.323
0.219
0.311
0.231
0.323
0.635
0.613
0.118
0 0.994
0.426
0.466
0.953
0.786
1.028
0.788
1,200
0.871
1,259
0.919
1.349
0.921
Avg
0.2118
0.315
0.179
0.365
0.178
0.273
0.176
0.275
0.181
0,277
0,402
0.453
0.266
0.353
0.346
0.404
0.627
0,600
0,800
0.685
0.878
0.725
1,281
0.929
1.259
0,904
96
0,550
0.408
0.414
0.201
00419
0.298
0.427
0.304
10,004
0,256
0.854
0.492
0.670
0-421
0.795
0.481
1,468
0.821
1.643
0.855
1,157
0.636
1.557
0,821
1,586
0.841
192
0.552
0.408
0.419
0.291
0.434
0.305
0.417
0.315
0,412
0.284
0.891
0.517
0.653
0.405
0.837
0.503
1,508
0.838
1.856
0.928
1.688
0.848
1.596
0.834
1.802
0.844
336
0.572
0.414
0.457
0.314
0.449
0,313
0.478
0.335
0,310
0,853
0.471
0.707
0.445
0.867
0.523
1,602
0.800
2.080
0,999
1,826
0.905
1.621
0.841
1,668
0.868
Avg
0.358
0.410
0.423
0.298
0.434
0.305
0.450
0.317
0.418
0.296
0.867
0.493
0.676
0.423
0.833
0.502
1,526
0.839
1.859
0.927
1.357
0,795
1.591
0,832
1.618 0.851
Table 13: Full few-shot learning results on 10% training data.
Metiods|
Tiae-VLM
Time-LLM
|
GPTTTS
DlLinesc
TamesNee
|
PhDMormerr
Autafonaer
Statioaary
ETSComee
Lightts
Informer
Relonner
Mesric
MSE
MAB
MSE
MAB|
MSE
MAE
MSS
MAE
MSE
MAE
MSC
MAE| | MSE MAD| MSB
MAE
MSE
MAE
MSC
MAB|
MSB
MAE
MSB
MAE
MSE
MAB
96
11.391
0.404
0.448
0.460
0.458
0.45
0.492
01495
0.516
0.485
0.861
0.628 0.512 0.499
0.613 0.552
11.988
03639
1.112
0.806
1,298 0.838
1.179
0.792
1.184
0.700
192
0.420
0.431
0.484
0.483
0.570
0,516
0.565
0.538
0.588
0.524
0.797
0.593
0.64
0.555
0.722
0.398
0.915
0.629
1.155
0.823
1.322
0.854
1.199
0.506
1.295
0,850
336
0.439
0.448
0.569
0.540
0.608
0.535
0.721
0.622
0.657
0.550
0.941
0.648
0.691 0,576
0.350
0.619
0,939
0.644
1.179
0.852
1.347
0.870
1,202
0.811
1.294
0.854
730
11476
0.484
0,200
0.604
0.725
0.591
0.986
0.745
0.762
0.610
0.877
0.641
0.728
0.614
0.721
0.616
0.887
0.645
1,273
0.874
1.534
0944
1.317
0.825
1,213
0.838
Avg
0.431
0.556
0.522
0.590
0.525
0.691
0.600
0.633
0.542
0.869
0.628
0.639
0.56]
0.702
0.596
0.915
0.639
1.180
0.634
1.375
0.877
1.199
1,219
2:3 912
192
96
11284
0.349
0.308
0.275
0.374
0.326
0.373
0.351
0.402
0.374
0:411
0.357
0.569
0411
0.519
0.353
0,403
0.389
0.444
0.37%
0.690
0.407
0.467
0.382
0.478
0.466
0.474
0.413
0:474
0.451
0.477
10.300
0.473
DA11
0.455
0.678
0.785
0.619
0.666
2,022
2.329
1.104
1.00
5.837
3.856
1.508
1.513
3.788
3.552
1,533
1,683
11570
0.412
0.429
0.400
0.453
0.671
0.572
0.426
0.441
0.537
0.494
0.504
0.501
01547
0.543
01.07
0.430
0.859
0.684
2.453
1.122
3.952
1.526
3.395
1.526
730
[1441
0.466
11427
0.44
(1447
0.46
0.824
0.648
0.477
0.480
0.510
0.401
0.499
0.509
0.516
0.523
0.477
0.472
1,273
0.874
3,816
1.407
3.842
1.503
1.205
1.401
Avg
0.361
0.405
0.300
0.394
0.397
0.421
0.605
0.538
0.415
0.431
0:479
0,965
0.166
0.475
0.488
0.499
0.462
0.455
0.894
0.713
2.655
1.600
3,872
1.513
3.685
1,586
96
10.10
1.354
0.346
0.388
0.390
0.404
0.352
0.392
0.410
0.419
0.553
0.501
0.5778
0:518
0.774
0.614
[0.761
0.568
00.911
0.688
0.921
0.682
1.162
0.785
1.442 0.847
192
0.340
0.370
0.533
0,416
0.129
0.423
0.382
0.112
0.437
0.434
0.630
0.528
0617
0,586
0.754
0.592
0.781
0.574
0.955
0.700
0.957
0.701
1.172
0.793
1.444
0,862
336
11369
0.367
0.4%
0.439
0.419
0.452
0.476
0.454
0.725
0.508
0.99%
0.775
0.669
0.677
01.00
0.587
0,991
0.998
1.227
0.500
1.450
0.866
0,423
1.455
0.476
03569
0.496
0.490
0.477
0.681
0.556
0.760
0.540
0.693
0.579
0.810
0.844
0.5&1
1.062
0.747
1.007
1.307
1.366
0.850
Avg
0.360
0.362
0.404
0.421
0.464
0.461
0.111
0.129
0.466
0.677
0.537
0.722
0.605
0.802
0.628
0.797
0.578
0.980
0.714
0.971
0.705
1.192
0.521
1.426
0.856
gr7m2
[1160
0.360
0.177
0:261
08188
0.769
0.13
0.303
0.191
0.374
0.211
0.285
(0221
0,399
0.352
0.454
[0.229
0,
[0.33]
0.430
(0813
0.688
3.303
1,407
4.195
1.638
182
0122
0.296
0.241
0:51
0251
0.309
0.278
0.345
0.252
0.317
0,270
0.323
0.307
0.379
0.681
0,691
0.291
0.343
0.500
0,664
1.008
0.768
3.112
1.387
4,012
1,501
320
11277
01:35
1274
0274
0:327
0ENT
0.346
0.33%
0.385
0.300
0.353
0.321
0.355
0.543
0,559
2.408
11:448
11576
0.469
0,48%
1.031
0.775
5:255
1.421
3:963
1,565
230
0-401
0:300
0:417
0.436
0.440
0.433
0.437
0.474
0.440
0.712
0.614
1.91.
16666
0.461
0.589
0.557
1.096
0.791
3,900
1.543
3.711
1,530
Avg
0.263
0.333
0.277
0.323
0.293
0.335
0.316
0.268
0.296
0.343
0.330
0.353
0.463
0.485
1.342
0.930
0.332
0.306
0.447
0.487
0.987
0.756
3.370
1.440
3.978
1.587
Tro Fiee Elostrioka Weather
96
0.174
0 0.308
0.161
0.210
0.163
0.215
0.171
0.334
0.165
0.215
0.114
0,250
0.188
0.253
0
01221
0.207
[0. 100
0.234
0.199
0.272
0.217
0.250
0.369
0.374
0.401
0.335
0.350
192
0.217
0.262
0.501
0.348
0.10
0.251
0.155
0.263
0.210
0.257
0.245
0.283
0.250
0,304
0.270
0.322
0.269
0.298
0.279
0.332
0.259
0.304
0.552
0.478
0.522
0.462
336
10564
1.286
0.501
03:00
0.36
0.202
0.25%
0.299
0.259
0.257
0.305
0.321
0.312
0.346
0.320
0.351
0.570
0.357
0.356
0.3%6
0.305
0.334
0.724
01:41
0.715
0.535
730
0.340
0,509
0.551
0.15
0.220
0.346
0.302
0.346
0.361
0.371
0.367
0.393
0.300
0.437
0.448
0.177
0.739
0.558
0.611 0.500
Aug
1.245
0.282
0.234
0.275
0.258
0.275
0.241
0.283
0.342
0.229
0.279
0.301
0.284
0.324
0.300
0,342
0.308
0.323
0.318
0.350
0.289
0.322
0.597
0.495
0.546
0.469
96
0,160
0.769
0.139
0.241
0.139
0.237
0.550
0.253
0.140
0.218
0.200
0.373
00233
0.323
0.261
0.348
0.440
0:466
10.999
0.587
0.3550
0.425
1.359
0.919
0.903
0.784
192
0.174
0.279
0,151
0.248
0.155
0:252
0.164
0.264
0.160
0.255
0.505
0.379
0.261
0.356
0.338
0.406
0.411
0.459
0.620
0.588
0.376
0.448
1.160
0.873
0.918
0.753
336
11.100
0.204
0.169
0.270
01175
0.270
0.181
01282
0.150
0.277
0.319
0.391
0.360
0.445
0.410
0.474
11.444
0.473
0.662
0.619
0.428
0.485
1.157
0.572
0.925
0.745
720
0.229
0:323
0.340
0:322
0.255
0.317
0.223
0.321
0.241
0.303
0.369
0.426
0.530
0.585
0.715
0.685
0.510
0.521
0.757
0.664
0.611
0.597
1.203
0.598
1.004
0.750
Avg
[1.198
0.291
0.175
0.270
0.172
0.269
0.180
0.280
0.180
0.273
0.323
0,392
0.345
0.427
9.431
0.478
11,444
0.430
0.660
0.617
0.441
0.489
1.395
0.391
0.905
0.768
96
0.465
0.349
0.418
0.291
0.414
0.297
0.419
0.298
0,403
0.289
0.719
0,416
0.639
0.400
0.672
0.465
1.412
0.802
1.643
0.855
1.157
0.636
1.557
0.821
1.527
0.815
192
0.468
0.350
0.414
0.29%
0.426
0.301
0.434
0.305
0.415
0.296
0.748
0.428
0.637
0.416
0.727
0.426
1.419
0,306
1,641
0.854
1.207
0.661
1.454
0.365
1.538
0.817
136
01483
0.356
0.421
0.311
0.452
0.303
0.440
0.11
[1426
0.304
0.853
0.471
0.655
0.427
0.749
0.454
1.443
0.515
1.711
0.878
0.713
1.521
0,12
1,550
0.819
710
0.520
0.373
0.462
0.327
0.167
0.337
0.336
0.331
1,485
0.625
0.722
0.456
0.547
1,539
2,680
1.605
1.558
0,833
Avg
0.484
0.357
0.429
0.300
0.440
0,310
0.447
0.313
0.430
0.305
0.951
0.535
0.663
0.425
0.749
0.446
1,433
0..15
1.914
0.936
1.248
0.684
1.534
0.311
1.551
0.821
15

---

Time- VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
B.2. Zero-shot Forecasting
Table 14: Full zero-shot learning results on ETT datascts. A lower value indicates better performance.
Methods
VL.M
Time-I..M
L.MTime
GPT4TS
DLinear
PatchTST
TimesNat
Autoformer
Metric
MSE
MAB|
MSE
MAE
MSE
MAE|
MSE
MAE
MSE
MAE
MSE
MAE|
MSE
MAB
MSE
MAE
96
0.277 0.338
0.279
0.337
0.510
0.576
0,335
0.374
0.347
0.400
0.304 0,350
0.358
0.387
0.469
0.486
192
0.333
0.378
0
0.374
0.523
0.586
0,412
0.417
0.447
0.460
0.386
0,400
0.427
0.429
0.634
0.567
ETTà1 + ETTh2
336
0.360
0.399
0.388
0,415
0.640
0.637
0.441
0.444
0.515
0.505
0.414
0.428
0.449
0.451
0.655
0.588
720
0.383
0.425
0.391
0.420
2.296
1.034
0.438
0.452
0.665
0.589
0.419
0.443
0.448
0.458
0.570
0.549
Avg
0.338
0.385
0.353
0.387
0.992
0.708
0.406
0.422
0.493
0.488
0.380
0,405
0:421
0.431
0.582
0.548
96
0.207
0.297
0.189
0.293
0.646
0.563
0.236
0.315
0.255
0.357
0.2115
0.304
0.239
0.313
0.352
0.432
192
0.258
0.329
0.233
0.312
0.934
0.654
0.287
0.342
0.338
0.413
0.275
0,339
0.291
0.342
0.413
0.460
ETTM1-EETH
336
0 0.310
0.291
0.365
1.157
0.728
0.341
0,374
0,425
0.465
0.334
0.373
0.342
0,371
0,465
0.489
720
0 0.398
0.412
0.372
0.390
4.730
1 1.531
0.435
0.422
0.640
0.573
0.431
0.424
0:434
0.419
0,599
0.551
Avg
0.273
1.867
0.869
0.325
0.363
0.415
0.452
003144
0.360
0.327
0.361
0.457
0.483
96
0,434
0.441
0,450
0,452
1.130
0.777
0.732
0.577
0.689
0.555
0,485
0,465
0.848
0.601
0.693
0.569
192
0.464
0.454
0.465
0.461
1.242
0.820
0,758
0.559
0.707
0.568
0.565
0,509
0.860
0.610
0.760
0.601
ETT22 ETTh1
336
0.489
0.481
1.328
0.864
0.759
0.578
0.577
0.581
0.515
0.867
0.626
0,781
0.619
720
0.595
0.543
0.501
0,501
4.145
1 1.461
0.781
0.597
0.704
0.596
0.628
0.561
0.887
0.648
0.796
0.644
Avg
0.496
0 480
0.479
0.474
1.961
0.981
0.757
0.578
0.703
0.574
0.565
0.513
0.865
0.621
0.757
0.608
96
0.204
0.297
0.174
0.276
0.646
0.563
00.25
0.329
0,240
0.336
n..26
0.309
0.248
0.324
0.263
0.352
192
0.255
0. 0.328
0.233
0.315
0.934
0.654
0,293
0,346
0.295
0.369
0,289
0.345
0.296
0.352
0,326
0,389
ETT12 + mT
336
0.311
0 0.362
0.291
0.337
1.157
0.728
0,347
0,376
0.345
0.397
0.348
0,379
0.353
0.383
0,387
0.426
720
0.420
0.425
0.392
0.417
4.730
1.531
0.446
0.429
0.432
0.442
0.439
0.427
0.471
0.446
0.487
0.478
Avg
0.297
0.353
0.272
0.341
1.867
0.869
0.335
0.370
0.328
0.386
0.325
0.365
0.342
0.376
0.366
0.411
96
0.297
0.356
0.321
0,369
0.510
0.576
0,353
0.392
0.365
0.415
0.354
0,385
0.377
0.407
0.435
0.470
192
0.349
0
0.389
0,410
0.523
0.586
0.,44
0.437
0,454
0.462
0.447
0.434
0.471
0.453
0.495
0.489
ETTml + ETT12
336
0,374
0.409
0.408
0.433
0.640
0.637
0,469
0.461
0.496
0.494
0.481
0.463
0.472
0.484
0.470
0.472
720
0.396
0.433
0.406
2.296
1.034
0,466
0,468
0.541
0.529
0.474
0,471
0.495
0,482
0,480
0.485
Avg
0.354
0.397
0
0.412
0.992
0.708
0.443
0.439
0.464
0.475
0.439
0.438
0.457
0.454
0.470
0.479
96
0.178
0. 267664
0.169
0.257
0.646
0. 0.563
10.217
0.294
0.221
0.314
0.195
0.271
0.222
0.295
0.385
0.457
192
0.226
0.29%
0,227
0,318
0.934
0.654
0,277
0.327
0.359
0.258
0.311
0.288
0.469
8T7ml - k77m2
336
0.2799
0.329
0,290
0.318
1.157
0.728
0,,31
0,360
0.357
0.406
0.317
0.348
0.341
0.367
0.476
0.477
720
0.373
0.
0.375
0.367
4.730
1.531
0,429
0,413
0.476
0.476
0.416
0,404
0.436
0.418
0,582
0.535
Avg
0.264
0.319
0.268
0,320
1.867
0.869
0.313
0,348
0.335
0.389
0.296
0.334
0.322
0.354
0.469
0.484
96
0.285
0.347
0,298
0.356
0.510
0.576
0,360
0.401
0.333
0.391
0.327
0,367
0.360
0.401
0.353
0.393
192
00.348
0.384
0.359
0.397
0.523
0.586
0.434
0.437
0.441
0.456
0.411
0.418
0.434
0.437
0,432
0.437
ETTm2 - ETT12
336
0,380
0.415
0.367
0.412
0.640
0.637
0.460
0.459
0.505
0.503
0.439
0.447
0.460
0.459
0,452
0.459
720
0.424
0.451
0.393
0.434
2.296
1 1.034
0.485
0.477
0.543
0.534
0.459
0.470
0.485
0.477
0.453
0.467
Avg
0.359
0.399
0.354
0.400
0.992
0.708
0,435
0.443
0.455
0.471
0.409
0,425
0.435
0.443
0.423
0.439
96
0.370
0.390
0.359
0,397
1.179
0.781
0.747
0.558
0.570
0.490
0.491
0.437
0.747
0.558
0,735
0.576
192
0,400
0.409
0.390
0.420
1.327
0.846
0,781
0.560
0.590
0.506
0,530
0.470
0.781
0.560
0.753
0,586
KF7m2 - k77ml
336
0.426
0.420
0.421
0.445
1.478
0.902
0.778
0.578
0.706
0.567
0.565
0.497
0.778
0.578
0.750
0.593
720
0.531
0.487
0.487
0.48%
3.749
1.408
0,769
0.573
0.731
0.584
0.686
0,565
0.769
0.573
0,782 0.609
Avg
0.432
0.426
0.414
0.984
0..69
0.567
0.649
0.537
0.568
0.492
0.769
0.567
0.755
0.591
B.3. Short-term Forecasting
B.4. Long-term Forecasting
C. Visualizations
C.1. Visualization of Generated Time Series Images
The image generation module employs advanced techniques freequuee--eequencyy and periodicity Encoding, multi-scale convolution, interpolation and normalization-tto to create informative and discriminative image representations of time series data. These representations enhance downstream VLMs for improved forecasting. As shown in Figure 6, the generated images capture key temporal characteristics through the following features:
Frequency-Domain Information: FFT integration captures recquencydoomain characteristics, visualized as distinct textures exturee-finegeee fine- line-grained for high-frequency components and broader color regions for low-freuueccy components.
Multi-scale Periodic Encoding: Temporal dependencies at multiple scales (e.g., daily, weekly) are encoded, visible as regular patterns such as repeating vertical bands for daily cycles or broader horizontal patterns for weekly cycles.
Image Interpolation: Bilinear interpolation ensures smooth and coherent images, preserving essential time series characteristics through seamless transitions between color intensities.
16

---

Timc-VLM: VLM: Exploring Multimodal Vision- Vision-Language Models for Augmented Time Series Forecasting
Table 15: Full short-term time series forecasting results. The forecasting horizons are in [6, 48] and the last three rows are weighted averaged from all datasets under different sampling intervals. A lower value indicates better performance.
Methools
Time- VLM
Time-LMM
GPT-TT
TimesNet
PaccTTST
N-HITS
N-BEATS
ETSformer
LighTT
DLincar
FEDiomeer
Stationary
Auteformer
Informer
Reformer
Yearly
SMAPE
13.285
13.419
15,110
15.378
13.477
13.422
13.487
18.009
14.247
16.965
14,021
13.717
13,974
14.727
16.169
MASE
2.993
3.005
3.565
3.554
3.019
3.056
3.036
4.487
3,109
1,283
3.036
3,078
3.134
3.118
3,800
OWA
0.783
0.789
0.911
0,918
0.792
0.795
0.795
1.115
0.827
1.058
0.811
0,807
0.822
0
0.973
Quanecty
SMAPE
10.218
10.110
10.597
10:165
10.380
10.185
10.564
13.376
11.364
12.145
11.100
10.958
11.338
11,360
13.313
MASE
1203
1.178
1,253
1,227
1.233
1.100
1.252
1,906
1.328
1.520
1,350
1.325
1,365
1,401
1.775
OWA
0.903
0,889
0.938
0,923
0.921
0.893
0.936
1.302
1,000
1.106
0.996
0.981
1.012
1.027
1.252
Monthly
SMAPE
12.788
12.990
13:258
13.513
12,959
13:059
13.089
14.588
14.014
13.514
14.403
13:917
13,958
14.062
20.128
MASE
0.963
1,003
0.970
1.013
0.996
1.368
1.053
1.037
1.117
1.097
1.103
1111
2.614
OWA
0
0.903
0.931
0.957
0,905
0.929
0.922
1,149
0.981
0.956
1,038
0.998
1,002
1.024
1.927
Otbers
SMAPE
1.915
4.795
6.124
6,913
4.952
4,711
6,599
7.267
15,880
6,709
7.148
6,302
5.185
21,160
32:191
MASE
3.257
3.178
4.116
4.507
3,547
3.054
4.430
5,240
11:434
4.953
4.041
4,064
3.865
20,960
33.355
OWA
1.006
1.259
1,049
1.393
1.591
1.487
1.389
1.304
1.187
5,879
8.679
Avenge
SMAPE
11.894
11.983
12,690
12,880
12.059
12.035
12.250
14.718
13.525
13.639
13.160
12.780
12,909
14.086
18.200
MASE
1,592
1.595
1.808
1.836
1,623
1.625
1.698
2/408
2.111
2.095
1.775
1.756
1.771
2.718
1.223
OWA
0.855
0.859
0.940
0.955
01889
0.860
0.896
1.172
1,851
1.051
0.949
0.930
0.939
1,230
1.775
Table 16: Full long-term forecasting results. We use the same protocol as in Table 1.
Methods
Time
VLM
Time LLM
GPT-TS
DLinsar
ParcTTS
TimesNet
FEDiormer
Autoformer
Stationary
ETSformer
LightTS
Informer
Reformer
Metric
MSF
MAF
MSF
MAF
MSF
MAF
MSF
MAF
MSE
MAE]
MSF
MAF]
MSF
MAF
MSF
MAF
MSF
MAF
MSF
MAF
MSF
MAF
MSE
MAF
MSF
MAF
96
0.361
0.386
0.362
0,392
0.376
0,397
0.375
0.359
0.570
0.399
0.384
0.402
0.376
0.419
0.449
0.459
0.513
0.691
0.494
0.479
0.424
0.432
0,963
0.713
0.837
0.728
0.415
0.395
0.418
0.416
0.405
0.413
0.421
0.4%
0.420
01500
0.482
0.554
0.504
0.538
0.514
0.475
0.462
0.792
0.923
0.766
0,421
0.627
0.433
0.443
0.622
0.459
0.521
0,588
0.574
0.909
1.097
0.835
730
0.45%
0.457
0.477
0.456
(1472
0.490
00.47
0.406
0.521
0,500
0.506
0.507
0.514
0.643
0.616
0.562
0.535
0.547
0.533
1,181
0.885
1.257
(1889
Avg
0..08
0.420
0.408
0.423
0.465
0.455
0.422
0.47
0:413
0.430
0.458
0.450
0.440
0.460
0.406
0.487
0.570
0.537
0.542
0.510
0.491
0:479
1.040
0.705
1.029
0.805
96
0.267
0.335
0.265
0.325
0.285
0.342
0.2299
0.353
0.277
0.336
0.340
0.574
0.35M
0.347
01346
0.38H
0.576
0,458
0.360
0.391
0.297
0.437
3.755
1.525
2.626
1.317
K77
193
0.326
0.373
0.329
0.375
0.354
0.389
01383
0418
0.330
0.379
0.402
0.414
0.429
0.439
01456
0.452
0.512
0.493
0.430
0.489
0.520
0.504
5,600
1.931
11.120
2,979
336
0.352
0.406
0.365
0.809
0.373
0.407
0.488
0.465
0.329
0.350
0.452
0.052
0.196
0.167
0.482
0.486
0.552
0.551
0.485
0.479
0.626
0.559
4.721
1,835
9.323
2.769
730
0,412
0.445
0.372
0.420
0.406
0.441
11:05
0.551
11.379
01422
0.462
0.468
0.463
0.474
11.155
11.11
0,502
0.560
0.500
0.497
0.565
0.672
3.647
1,625
3.874
1.697
0,341
0.301
0.334
0.363
0.381
0.412
0.41
0.446
01,300
0.379
0.414
0:427
0.437
0.449
0.450
0.459
0,526
0,516
0.439
0.452
0.602
0.543
4.431
1.739
6.736
2.191
FTTw2 E7Pml
0.304
0.346
0.366
0.310
0.358
0.334
0:292
0,346
0.315
0.299
0.290
0.332
369
0.333
0.374
0.379
0.426
0.505
0.553
0,386
0.45)
0.375
0.408
0.398
0.374
0.400
0.407
0.660
0.538
0.528
0.592
0.385
0,381
0.366
0.294
0.369
0
0.366
0,410
0.411
0.445
0.621
0.695
0.435
0.425
0.138
1,212
0.87
0.898
0.721
0.410
0.383
0.411
0.477
0,421
01425
0/21
D446
0,420
0.478
0.543
0.671
0.561
0.585
0.516
0.499
0.462
0.527
0.502
1,166
0.:23
1.102
0.841
Avg
0.350
0.372
0.329
0.372
0.388
0.403
0.357
0.378
0.351
0.380
0:400
0,406
0.448
0.452
0.588
0.517
0.481
0,456
0.429
0.425
0.435
0.437
0.961
0.734
0.799
0.671
96
0.160
0.259
0.161
0.255
0.173
0.262
0.367
0.269
00.165
0.255
0.187
0.267
0.2015
0.287
01:255
1.339
0.192
0.274
0.189
0.260
0.209
0.30M
0,365
0.453
0.65%
01.699
192
0.215
0.290
0.219
0.201
0.229
0.00
0.224
0000
0:220
0.202
0.249
0. 10309
0.769
0.328
0.281
0. 1.540
0.280
0.350
0.255
0.319
0.311
0.382
0.513
0.563
1.078
0.827
336
0.270
0.325
0.286
0.341
0.281
0.342
0.274
0.329
0.321
0,35]
0.325
0.366
0.339
0.352
0.334
0.361
0.314
0.357
0.442
0.466
1.363
0.887
1.549
0,972
730
0.345
0.35%
0.382
0.376
0.401
0.397
0.421
0.362
0.385
0.408
0.403
0.421
0.415
0.435
0.432
0.417
0.413
0.414
0.113
0.675
0.587
3.379
1.338
2.631
1.242
Arg
0.344
0.511
11251
0:315
0.284
0.339
01267
0.333
11.255
0,315
0.291
0,333
0.305
0.349
0.327
0.371
0.306
0.347
0.295
0.342
0.409
0.436
1.410
0,800
1,479
0,915
Westher
0,148
0.200
0.147
0.201
0.162
0,212
0.176
0.237
0.149
0.196
0.172
0.220
0.217
0
0.296
0.266
0.336
0.173
0.223
0.197
0.281
0.182
0.242
0.300
0.384
0.889
0.596
0.189
0.234
0.204
0.248
0.220
0.262
0.194
0.241
0.219
0.226
0.307
0.345
0.257
0.312
0.267
0,598
0.544
0.752
0.262
0.279
0.254
0.265
0:345
0.280
0.306
0.330
0.359
0.321
0.208
0.353
0.334
0.503
0.630
0.506
0.312
0.304
0,316
0.326
0.237
0.333
0,262
0.314
0,334
0.365
0.403
0.419
0.814
0,410
0.252
0.288
0.386
1,059
0:741
1.130
0.792
Aug
0.225
0.257
0.237
0.270
11:248
0.300
10:225
0.264
0.259
0.287
0.309
0.3
361
0.338
0.582
0.288
0.514
0.271
0.354
0.261
0.312
0.634
0.548
00:00
13666
Fe Elvetriclly
96
0.142
0.245
0.131
0.224
0.1.9
0.238
0..10
0.237
0.129
0,232
00.188
0.272
0.193
0 0.308
0.201
0.317
0.169
0,273
0.187
0.304
0.207
0.307
0.274
0.368
0.312
0,402
192
0.157
0.260
0.152
0.241
0.153
01251
0.153
01249
0.157
0.240
0.184
0.289
0.201
0.315
11.223
0.334
0.182
0.286
0.199
0.315
0.213
0.316
0.256
0.386
0.:45
01,433
336
0.174
0.276
0.160
0.245
169
0.266
0 166
0.267
0.163
02250
0.198
0.300
0.214
0.33 0.320
0.231
0.318
0.200
0.304
0.212
0.329
0.230
0.333
0.300
0.364
0.350
0.433
720
0.234
0,308
0.192
0.295
0,206
0.297
0.203
0.201
0.197
0.290
0.220
0.320
0.246
0.285
0.254
0.361
0.222
0.321
0.253
0.345
0.265
0.260
0.373
0,429
0,340
0.420
Ang
0,172
0.273
0.158
0.252
0.167
0.263
00166
0.363
161
0.252
0.192
0.295
0.214
0.327
01.27
0,338
0,193
0.296
0.208
0.323
0.229
0.329
0.311
0,307
0,338
11.422
96
0.393
0.290
0.362
0.245
0.388
0.282
0.410
0.282
0.360
0.249
0.593
0.321
0.587
0.366
0.613
0.388
0.612
0.338
0.607
0.392
0.615
0.291
0.719
0.391
0.732
0.423
0.405
0.236
0.247
0.407
0.290
0.423
0.287
0.375
0.246
0.61
0.504
01616
0.613
0.340
0.621
0.299
0.501
0.382
0.696
0.379
0.733
0.420
0.305
0.271
0.12
0.294
0.476
0.397
0.629
0.621
0.622
0.618
0,622
0.396
0.386
0.420
0.742
0.420
0.323
0.288
0.450
0.312
0.466
0.315
0.432
0.640
0.626
0.600
0.653
0.632
0.396
0.658
0.407
0.964
0,472
0.755
0,423
Avg
0.419
0.303
0.388
0.204
0.444
0.204
01433
0.205
0.263
0.620
0.336
0.600
0.376
0.628
0.379
0.624
0.340
0.621
0.996
0.622
0.392
0.764
0.416
0.741
0,422
Color Trends: Color intensity corresponds to time series malues-darked regions (e.g., deep blue) indicate lower values, while brighter regions (e.g., yellow) represent higher values, enabling easy identification of trends.
Abrupt Changes and Anomalies: Sudden shifts in color intensity (e.g., sharp transitions from dark to bright) highlight abrupt changes or anomalies, crucial for identifying irregular events like traffic spikes or weather shifts.
C.2. Visualization of prediction results
The prediction results in Figures 7, 8, 9, and 10 demonstrate Time- VLM's ability to accurately forecast time series across diverse datasets and prediction horizons. For datasets with clear periodic structures, such as the daily cycles in ETThl and ETTml, Time- VLM captures both global trends and fine- one-grained temporal patterns effectively. This is evident in the close
17

---

Time- VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
Figure 6:
Time series transformed images, capturing key temporal characteristics, including trends, stationarity, scasonality, sudden changes, and frequency-domanl patterns.
alignment between the true values (solid lines) and predicted values (dashed lines) across all horizons. Similarly, for the ECL dataset, which exhibits regular consumption patterns, Time-VLM delivers highly accurate forecasts, showcasing its strength in handling structured environments.
a) ETTh1
b) ETTml
C) ECL
c) Traffic
Figure 7: Prediction results visualization for ETTh1, ETTm1, ECL, and Traffic datasets at 96 prediction lengths. True values (solid line) and predicted values (dashed line) are shown for each dataset and horizon.
a) ETTh1
b) ETTml
c) ECL
c) Traffic
Vole
Tree Vilee
Tru Vab
Figure 8: Prediction results visualization for ETTh1, ETTm1, ECL, and Traffic datasets at 192 prediction lengths. True values (solid line) and predicted values (dashed line) shown for each dataset and horizon.
However, performance varies for datasets with irregular or abrupt changes. On the Traffic dataset, which is characterized by non-stationary patterns, Time- VLM shows slight deviations in capturing sudden fluctuations, particularly at longer horizons (e.g., 336 and 720). These deviations highlight the challenges of modeling highly irregular data and suggest opportunities for refining the time series-to-image transformation process to better handle such scenarios.
18

---

Time- VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
a) ETTh1
b) ETTm1
c) ECL
c) Traffic
Polkcu Vaite
Teae
1.5
10-
02
Figure 9: Prediction results visualization for ETTh1, ETTml, ECL, and Traffic datasets at 336 prediction lengths. True values (solid line) and predicted values (dashed line) shown for each dataset and horizon.
a) ETTh1
b) ETTml
c) ECL
c) Traffic
Figure 10: Prediction results visualization for ETTh1, ETTm1, ECL, and Traffic datasets 720 prediction lengths. True values (solid line) and predicted values (dashed line) are shown for each dataset and horizon.
D. Future Work
D.1. Limitations
While Time-VIL VLM demonstrates significant improvements in time series forecasting by integrating temporal, visual, and textual modalities, it has some limitations.
First, the framework performs less robustly on datasets with highly volatile or irregular patterns, such as those with sudden changes or non-stationary trends, compared to datasets with periodic structures. This limitation may arise from the current visual transformation techniques, which may not adequately capture abrupt temporal dynamics or sudden shifts. Future work could refine these transformations to better handle such irregularities.
Second, the current implementation relies on pre-trained VLMs like ViLT and CLIP, which are optimized for natural lision-language tasks rather than time series forecasting. While these models excel in visual understanding, their textual capabilities are limited, often supporting only shorter text inputs and lacking domain-specinf knowledge relevant to time series. This restricts their ability to fully utilize textual context for forecasting. Future work could involve developing larger, domain-specific VLMs trained on multimodal time series datasets to address these limitations.
D.2. Future Work
Building on the current framework, several promising directions for future research emerge:
Optimizing Visual Transformations: Future work could focus on developing adaptive visual transformation techniques that better preserve temporal dynamics, especially for datasets with irregular or non-stationary patterns, to more effectively highlight sudden changes and complex trends.
Scaling Multimodal VLMs for Enhanced Forecasting: While the current framework uses smaller pre- mre-trained Vision- Language Models (VLMs), scaling to larger models could improve forecasting accuracy. Investigating trade-offs between model size, computational efficiency, and performance is a promising direction for future research. Additionally, studying different VLM architectures could identify optimal designs for temporal modeling.
Interpretable Multimodal Learning for Time Series Analysis: Understanding the contributions of visual and textual modalities in time series forecasting crucial for improving model transparency. Future work could explore the
19

---

Time-VLA Exploring Multimodal Vision-Language Models for Augmented Time Series Forecasting
interpretability of multimodal features, analyzing how different types of information contribute performance gains. This would provide deeper insights into temporal dependencies and enhance trust in multimodal forecasting models.
Pre-training Multimodal Foundation Models for Time Series Analysis: Existing VLMs are not designed handle time series data, limiting their ability capture domain-specifie temporal context. Future research could focus on constructing large-scale multimodal datasets that pair time series data with rich textual and visual annotations, enabling the development of models specifically optimized for time series forecasting. Additionally, this multimodal framework could be extended to support multi-task learning, enhancing the model's versatility for tasks such as anomaly detection, classification, or imputation. This would allow the model to capture broader range of temporal patterns and dependencies, improving its applicability across various domains.
By addressing these directions, future research can build the foundation laid by Time-VLMN advancing the field of multimodal time series forecasting while ensuring responsible and ethical deployment in real-world applications.
20


Reviews:
[{"review_id": "75KfUYRJ7Q", "reviewer": "Reviewer_6Pe4", "summary": "The paper introduces Time-VLM, a multimodal time series forecasting framework that leverages pre-trained Vision-Language Models (VLMs) to integrate temporal, visual, and textual information. By combining retrieval-augmented learning, vision-based encoding, and text-based contextualization, Time-VLM enhances forecasting performance, particularly in few-shot and zero-shot settings.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the clear motivation, method design, and rigorous evaluation of our paper. Below we endeavor to address your questions.\n\n> **Q1: Comparison with Time Series Foundation Models**\n> \n\n**A1:** We appreciate your suggestion. However, ​Time-VLM fundamentally **differs** from foundation models (e.g., CHRONOS, TimesFM, MOIRAI) in its **​learning paradigm**: While the latter rely on ​**external datasets** for pre-training, Time-VLM ​uses only the target dataset. By generating text and visual modalities, it augments raw time series ​**without external knowledge**, enabling efficient fine-tuning and cross-dataset zero-shot learning.\n\nHowever, we added a comparison of Time-VLM (5% few-shot) vs. foundation models (zero-shot) on ETT datasets. Results show Time-VLM achieves lower MSE across all datasets, validating its effectiveness under data scarcity.\n\n| Methods | Time-VLM | Moirai | Chronos | TimesFM |\n| --- | --- | --- | --- | --- |\n| ETTh1 | 0.442 | 0.475 | 0.560 | 0.489 |\n| ETTh2 | 0.354 | 0.379 | 0.392 | 0.396 |\n| ETTm1 | 0.364 | 0.714 | 0.636 | 0.434 |\n| ETTm2 | 0.262 | 0.343 | 0.313 | 0.320 |\n\n> **Q2: Instance Normalization and Misalignment**\n> \n\n**A2:** We added ablation experiments on Weather dataset to evaluate text normalization’s impact. Results show that normalized text **reduces MAE by 1.29%**, underscoring the importance of cross-modal alignment. We will fix this in final version.\n\n| Horizon | Time-VLM (Raw) | Time-VLM (Normalized) |\n| --- | --- | --- |\n| **96** | 0.160 | 0.159 |\n| **192** | 0.203 | 0.201 |\n| **336** | 0.253 | 0.249 |\n| **720** | 0.317 | 0.312 |\n| Avg | 0.233 | 0.230 |\n\n> **Q3: Role of the Vision Encoder and Freezing Rationale**\n> \n\n**A3:** We conducted an analysis to evaluate how pre-trained VLMs can be applied to time series forecasting. Specifically, we sampled 200 **image-text pairs from MSCOCO** (VLM’s pre-training dataset) and 60 samples each from **time series dataset (ETT, Traffic, Weather, ECL)**. Using t-SNE, we visualized four embedding types in 2D space:\n\n1. Multi-modal embeddings from COCO-Pair samples through VLM ⇒ representing **VLM’s pre-training knowledge**\n2. Multi-modal embeddings from time series-generated image-text pairs through VLM ⇒ representing **Time-VLM’s augmented knowledge**\n3. Visual embeddings from COCO-Image samples through ViT ⇒ representing **visual knowledge from a single VM**\n4. Text embeddings from COCO-Text samples through BERT ⇒ representing **text knowledge from a single LM**\n\nKey findings from the visualization https://anonymous.4open.science/r/Time-VLM/ts_embeddings_with_coco_samples.png:\n\n1. **COCO-Image features (green)** fcluster centrally, **surrounded by time-series features** (ETT: yellow/orange, ECL: purple, etc.), confirming intrinsic visual-temporal similarity. \n    \n    This aligns with VisionTS [1], where **visual features** (pixel intensity variations, repeating patterns, color consistency, and edge transitions) **directly map** to **temporal behaviors** (value fluctuations, periodicity, stable segments, and outliers/abrupt changes, respectively). This explains why time series imaging [2] has emerged as a hot research direction. Our image generation process further advances this explicitly encoding frequency/periodicity and using interpolation/color semantics to preserve temporal information.\n    \n2. **COCO-Text features (blue)** show **clear separation** with time-series clusters, highlighting modality gaps.\n3. **COCO-Pair features (red)** achieve **maximal overlap** with time-series data, demonstrating **cross-modal complementarity** - text semantics enhance visual-temporal alignment.\n\nThis **motivates** Time-VLM. Compared to Time-LLM and VisionTS which utilize a single modality projection, we argue that **projecting time series into multi-model embeddings** is more plausible. Time-VLM's remarkable zero-shot and few-shot learning capabilities primarily stem from the pre-trained VLM's knowledge.\n\n**Regarding freezing the visual encoder**, we did extensive experiments, which revealed that unfreezing led to training instability (e.g., overfitting on small datasets like ETT). Since the VLM has already achieved cross-modal alignment, freezing the encoder allows us to reuse this capability while avoiding excessive training overhead, we only add a simple fusion layer that is time-efficient for alignment.\n\n[1] M. Chen et al. VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters\n\n[2] J. Ni et al. Harnessing Vision Models for Time Series Analysis: A Survey.\n\n> **Q4: Lack of Component Analysis**\n> \n\n**A4:** We expanded ablation studies on the memory bank. Results on the Weather dataset (MSE) confirm **both local and global memory are critical**:\n\n| Horizon | Full Model | w/o Local Mem | w/o Global Mem |\n| --- | --- | --- | --- |\n| 96 | 0.160 | 0.185 | 0.165 |\n| 192 | 0.203 | 0.235 | 0.210 |\n| 336 | 0.253 | 0.295 | 0.265 |\n| 720 | 0.317 | 0.375 | 0.330 |\n| Avg | 0.233 | 0.273 | 0.243 |", "claims_and_evidence": "Overall, the claims made in the paper are clear.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate for the problem of time series forecasting.", "theoretical_claims": "The paper does not contain any theoretical claims that require formal proof verification.", "experimental_designs_or_analyses": "I have reviewed the experimental design and analyses. To further establish the validity of the results, additional comparisons with other models that perform well in zero-shot and few-shot forecasting would be necessary.", "relation_to_broader_scientific_literature": "This paper contributes to the broader scientific literature by exploring multimodal time series forecasting with pre-trained Vision-Language Models (VLMs). While previous works have integrated either text or vision with time series forecasting, this study is one of the first to leverage both modalities simultaneously.", "essential_references_not_discussed": "The paper would benefit from citing and discussing prior works on zero-shot and few-shot forecasting, particularly those related to foundation models in time series forecasting. Several recent studies, such as CHRONOS, TimesFM, and MOIRAI, have explored the use of large-scale pre-trained models for improved generalization in forecasting tasks.", "other_strengths_and_weaknesses": "Strengths  \n1. The paper presents a clear motivation and is well-structured, making it easy to follow.  \n2. The approach of transforming time series into visual cues and leveraging a memory bank for enriched feature extraction is novel.  \n3. The proposed method demonstrates strong performance across various time series forecasting scenarios, highlighting its effectiveness.  \n\nWeaknesses  \n1. Comparison with Time Series Foundation Models: Recent foundation models for time series forecasting (e.g., CHRONOS, TimesFM, MOIRAI) have demonstrated strong zero/few-shot forecasting performance. A direct comparison with these models is necessary to validate the effectiveness of the proposed approach.  \n\n2. Instance Normalization and Misalignment: Since raw time series data undergoes normalization, but textual information is extracted from the unnormalized version, this may cause misalignment in feature representations. The paper should include experiments addressing this issue to ensure robustness.  \n\n3. Role of the Vision Encoder: It is unclear whether a pre-trained vision encoder can effectively process images generated from time series data. A more thorough evaluation is needed to justify its role. Additionally, the decision to freeze the vision encoder should be explained, and results from fine-tuning should be provided for comparison.  \n\n4. Lack of Component Analysis: While the paper provides extensive forecasting results, there is a lack of detailed ablation studies on key components. For example, the memory bank is a core part of the method, yet its contribution is not analyzed enough. Further component-wise evaluations would enhance the credibility of the approach.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "juR4pN0HBO", "reviewer": "Reviewer_gzkF", "summary": "This paper proposes Time-VLM, a multimodal framework that leverages vision-language models to encode temporal, \"visual\" and textual modalities for enhanced time series forecasting. Specifically, RAL encodes and saves time series into a memory bank for further interaction with multi-modal embedding. VAL encodes time series to multi-dimensional representations and TAL generates time series descriptions. \"vision\" and textual features are processed with frozen pretrained VLMs to obtain informed meta representations. Fused with temporal features, the Time-VLM achieves superior performance on various experiment settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the method design, complete evaluation and unique innovation of our paper. Below are our responses to your major questions.\n\n> **Q1: Textual Encoder Limitations Analysis**\n> \n\n**A1:** We systematically investigated the limitations of textual encoders in VLMs through three complementary analyses:\n\n1. **Input Length Analysis**: We evaluated three VLMs with progressively longer maximum text input lengths—**ViLT (40 tokens), CLIP (77 tokens), and BLIP-2 (512 tokens)**—and observed no improvement in forecasting performance despite increased model capacity. Instead, larger models exhibited lower efficiency, with significantly slower inference and training speeds.\n2. **Custom VLM Analysis**: We implemented a Custom implementation on single VM (ViT-B/16) + LM (BERT-Base) on traffic datasets. However, experiments revealed that it underperformed native VLMs.\n    \n    \n    | Horizon | Time-VLM (ViLT) | Time-VLM (Custom) |\n    | --- | --- | --- |\n    | 96 | 0.148 | 0.158 |\n    | 192 | 0.193 | 0.206 |\n    | 336 | 0.243 | 0.258 |\n    | 720 | 0.312 | 0.334 |\n    | Avg | 0.224 | 0.239 |\n3. **Embedding Space Divergence**: As we discussed in **Q3/A3 of Reviewer 6Pe4**, The t-SNE visualization demonstrates that:\n    - **Text embeddings (blue)** from COCO-Text form distinct clusters separated from time-series data distributions (ETT/Traffic/ECL), confirming poor semantic alignment.\n    - **Visual embeddings (green)** from COCO-Image naturally encompass time-series features, validating that pixel-space representations better preserve temporal dynamics.\n    \n    This reveals a lack of effective cross-modal alignment. Crucially, the inherent divergence between image, text, and time-series modalities—compounded by their distinct training data distributions—hinders performance without explicit alignment mechanisms.\n    \n4. **Future Potential**: Time-VLM is a **self-enhanced** paradigm that operates without real image/text data—both are generated from raw time series. We believe that with a high-quality dataset covering real-world multimodal time-series data (e.g., ECG time series paired with medical images and diagnostic reports), Time-VLM’s potential could be fully unlocked. At that stage, modalities could complement each other more effectively. \n\n> **Q2: The n-dimensional representation of time series and its description are loosely connected. Directly utilizing a pretrained VLM may reduce effectiveness.**\n> \n\n**A2:** Your concern is right. However, our approach does not aim to map time series directly to text. Instead, we project them into a joint text-image multi-modal space, leveraging the complementary strengths of both modalities. For a deeper discussion on this methodology, please refer to **Q3/A3 (Reviewer 6Pe4)**.\n\n> **Q3: The “visual” representation of time series is difficult for human perception to comprehend. The visualization in C.1 lacks meaningful interpretation.**\n> \n\n**A3:** I fully understand your concerns. In response, we added an insightful analysis in **Q3/A3 (Reviewer 6Pe4)**, showing that the pre-training knowledge from the VLM **aligns** with the time series domain. Additionally, our Time-VLM's image generation framework integrates innovative techniques to better align the distribution of the VLM’s input **while maximizing** temporal information retention:\n\n1. **Frequency Domain Information Injection:** \n    - High-frequency components **appear as** fine-grained textures\n    - Low-frequency components **form** broad color gamut distributions\n2. **Multi-Scale Period Encoding**:\n    - Cosine/sine functions encode periodic patterns (e.g., daily/weekly cycles)\n    - **Represented** through distinct spatial patterns in the image\n3. **Interpolation Alignment Mechanism**:\n    - Bilinear interpolation ensures smooth pixel transitions\n    - Sudden time-series changes **correspond to** sharp color intensity variations\n4. **Color Semantic Mapping**:\n    - Dark blue → low values; light yellow → high values\n    - Gradient transitions **directly indicate** trend directions\n\nWe acknowledge that visual interpretability remains a challenge and plan to explore this further in future work.\n\n> **Q4: Would using a pretrained VM combined with a pretrained LM, instead of a pretrained VLM, better enhance your visual and textual inputs?**\n> \n\n**A4:** This is **an interesting idea**, and we **have explored it experimentally**. You can find details in **Q1/A1** and **Q3/A3 (Reviewer 6Pe4)**.\n\n> **Q5: Since you mentioned foundation models for time series, could they offer stronger encoding capabilities and enhance your pipeline?**\n> \n\n**A5:** As discussed in **Q1/A1 (Reviewer 6Pe4)**, Time-VLM and time-series foundation models follow **different paradigms**. However, **we may explore building a pre-trained vision-language-enhanced foundation model** once high-quality multimodal time-series datasets become available.", "claims_and_evidence": "Yes. Claims are clear and well supported.", "methods_and_evaluation_criteria": "Yes, methods and evaluation criteria make sense in this area.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, experients are sound and comprehensive.", "relation_to_broader_scientific_literature": "Contribute to multi-modal time series analysis via combining temporal, visual and textual modalities", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n\n1. The n-dimensional representation of time series is good. Instead of merely plotting the time series, it allows for a more effective utilization of the vision encoder. \n2. Comprehensive and complete experiments. \n3. The framework is new to this field. \n\nWeakness:\n\n1. The textual encoder component follows Time-LLM, and based on your ablation study, its contribution appears marginal. There is room for improvement in this area.\n2. The n-dimensional representation of time series and its description are loosely connected. Directly utilizing a pretrained VLM may reduce effectiveness.\n3. The “visual” representation of time series is difficult for human perception to comprehend. The visualization in C.1 lacks meaningful interpretation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Would using a pretrained VM combined with a pretrained LM, instead of a pretrained VLM, better enhance your visual and textual inputs?\n2. Since you mentioned foundation models for time series, could they offer stronger encoding capabilities and enhance your pipeline?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "yBL5DQdiRy", "reviewer": "Reviewer_Cr31", "summary": "Time-VLM is a groundbreaking framework that leverages pre-trained VLMs to unify temporal, visual, and textual data for time series forecasting. Key innovations include adaptive time-series-to-image conversion (VAL), memory-enhanced temporal modeling (RAL), and contextual prompt generation (TAL). The experiments demonstrate superior performance in few-shot and zero-shot settings, with efficiency gains compared to Time-LLM. This work opens a new paradigm for multimodal time series forecasting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Scalability: Exploring larger VLMs (e.g., LLaVA, GPT-4V) could enrich textual context and further improve performance, offering a promising avenue for future work.**\n> \n\n**A1:** Thank you for your suggestion. We empirically evaluated VLMs across different scales, ranging from smaller architectures like ViLT (143M) to larger ones such as BLIP-2 (3.75B). Our experiments revealed two key findings:\n\n1. **Diminishing Returns on Scale**: Larger models demand substantially more computational resources without delivering performance gains.\n2. **Overfitting Tendency**: On benchmarks like the ETT datasets, we observed rapid training loss reduction alongside increasing validation loss, indicating overfitting.\n\nBased on these results, we adopted a more compact VLM to optimize the trade-off between efficiency and effectiveness. We hypothesize that the rich **multimodal priors from large-scale pretraining** may be unnecessarily **complex** for our current datasets (ETT, Traffic, Weather, Electricity), which are **relatively small-scale**.", "claims_and_evidence": "The claims are compelling. Ablation studies (Table 6) strongly support the multimodal synergy claim, as removing any modality degrades forecasting accuracy. Time-VLM's efficiency is notable, reducing parameters 20× compared to Time-LLM (Table 7) yet maintaining competitive performance, with inference speed metrics further affirming it. Results on zero-shot cross-domain tasks (Table 3) validate its generalization ability. Overall, the evidence substantiates the approach's effectiveness.", "methods_and_evaluation_criteria": "The methods are technically sound. VAL's time-series-to-image conversion retains temporal relationships, suiting VLMs. TAL's hybrid prompts combine statistical and domain knowledge practically. RAL's memory-enhanced modeling uses historical patterns for forecasting. Multimodal fusion of RAL, VAL, and TAL modalities captures diverse patterns, improving accuracy. Evaluation is comprehensive, with standard metrics and diverse datasets.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The experiments are comprehensive and well-executed, spanning key aspects such as long/short-term forecasting, few/zero-shot prediction, ablation studies, model analysis, visualization case studies, and hyperparameter sensitivity.", "relation_to_broader_scientific_literature": "This work aligns with recent efforts to integrate foundation models into time series analysis (e.g., Time-LLM, LLMTime) but stands out by uniquely addressing multimodal fusion. It advances the field by demonstrating how VLMs can effectively adapt to time series forecasting, bridging gaps between modalities.", "essential_references_not_discussed": "1) Vision Language Models Are Few-Shot Audio Spectrogram Classifiers (NeurIPS Workshop 2024):\nThis work evaluates zero-shot capabilities of LMMs for audio classification using spectrogram images and textual prompts. While related, Time-VLM uniquely addresses forecasting-specific challenges (e.g., periodicity-aware image encoding in VAL), highlighting its broader applicability.\n\n2) Training-Free Time-Series Anomaly Detection: Leveraging Image Foundation Models (Arxiv 2024):\nThis work focuses on zero-shot anomaly detection using visual representations. Discussing it would further emphasize Time-VLM’s unique contributions in multimodal forecasting and its efficiency compared to larger LMMs.", "other_strengths_and_weaknesses": "Strengths:\n\n1) Novel use of VLMs for time series, enabling semantic context injection.  \n\n2) Strong empirical validation across diverse domains. \n\nWeaknesses: \n\n1) Scalability: Exploring larger VLMs (e.g., LLaVA, GPT-4V) could enrich textual context and further improve performance, offering a promising avenue for future work.", "other_comments_or_suggestions": "Please refer to the weakness.", "questions_for_authors": "Please refer to the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1VqHKEVJm", "reviewer": "Reviewer_wRhk", "summary": "This paper proposes Time-VLM, a multimodal framework using pre-trained VLMs for time series forecasting. It introduces VAL, RAL, and TAL modules to consider the information from three different views. Experimental results show that it achieves good performance with high efficiency. It contributes to a possible new direction in multimodal time series modeling.\n\n## Update After Rebuttal\n\nI've read the rebuttal and other reviewers' comments, and my final rating is weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Add the latest references related to multi-modal time series**\n> \n\n**A1:** Thank you for your reminder. Our manuscript now includes two recent comprehensive surveys on multi-modal time series analysis [1,2], which we have integrated into both the **Introduction** and **Related Work** sections. These references help contextualize our work within current research trends and highlight important methodological connections.\n\n[1] Y. Jiang et al. Multi-modal Time Series Analysis: A Tutorial and Survey.\n\n[2] H. Liu et al. How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook.\n\n> **Q2: The projection from the text/images to the time series space needs more interpretation. Can the paper provide some case studies for a better illustration?**\n> \n\n**A2**: Thank you for mention this important question. We have discussed a lot about this point, you can refer to **Q3/A3 (Reviewers 6Pe4 / gzkF)**, hope to address your concern.\n\n> **Q3: According to Table 1 to Table 3, the proposed model can achieve good performance in zero/few-shot settings. However, the results in Table 4 and 5 are not significant, i.e., only limited improvements on the full-shot setting. Can the author clarify this point? In other words, if we have enough data, we don’t need other modalities anymore?**\n> \n\n**A3:** Thank you for your question. The experimental results indeed support this observation. However, it is worth noting that Time-VLM still **outperforms** Time-LLM despite having significantly fewer parameters. Our conclusion is that **when time series data is limited**, visual and textual transformations can **compensate for the lack of information**, though time series data remains the most critical factor. When the time series data is **sufficiently diverse and abundant**, the model can learn **effectively from the time series alone**.\n\nCurrently, Time-VLM research is **constrained** by the lack of high-quality multi-modal time series datasets. Existing text and image data are generated, with real-world multi-modal datasets—such as medical scenarios combining ECG time series, textual diagnoses, and other modalities—the fusion of these data types could yield better performance even in full-shot settings, as different modalities would **complement each other** more effectively.\n\n> **Q4: How about the efficiency of the proposed model? Do the authors evaluate different model size of Time-VLM on these datasets? How is the sensitivity to the parameter size? Generally the model size should largely impact the model performance.**\n> \n\n**A4:** The model size of Time-VLM is determined by its **underlying VLM architecture**. We evaluated four variants: ViLT-, CLIP-, BLIP-2-based implementations, and a custom model.\n\nExperiments show that larger models do not perform better. Since time series data is **relatively sparse** compared to multimodal image-text data, **excessively large VLMs** are prone to **overfitting** and **slower efficiency**. Among the tested versions, the ViLT- and CLIP-based configurations achieve the best trade-off between efficiency and performance.", "claims_and_evidence": "The claim that fusing temporal, visual, and textual modalities improves forecasting accuracy is supported by ablation studies. The claims are almost correct. I don’t identify crucial technical errors within the paper.", "methods_and_evaluation_criteria": "The methods of Time-VLM are primarily divided into the following components:\n- VAL’s Image Encoding: The frequency/periodicity-based time-series-to-image conversion aligns well with VLM requirements while retaining temporal relationships.\n- TAL’s Hybrid Prompts: Combining statistical features with domain-specific knowledge provides a pragmatic design for real-world deployment.\n- RAL’s Memory-Enhanced Modeling: The retrieval-augmented mechanism leverages historical patterns to enhance temporal understanding, offering a robust foundation for forecasting.\nThese three components are fused via the vision-language model to produce the predictive output. The evaluation metrics used are standard in this field (MSE, MAE, SMAPE, MASE, OWA), and the datasets (ETT, Weather, Electricity, Traffic, M4) are comprehensive and widely recognized.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Overall, the experiments are thorough and well-structured, addressing key aspects of time series forecasting and multimodal learning. While the results are promising and the approach demonstrates strong potential, further exploration in more datasets and diverse real-world scenarios could provide additional insights. Nonetheless, the evaluation is solid and offers a convincing case for the effectiveness of the proposed framework.", "relation_to_broader_scientific_literature": "This work aligns with recent efforts to integrate foundation models into time series analysis (e.g., Time-LLM, LLMTime) but stands out by uniquely addressing multimodal fusion. It advances the field by demonstrating how VLMs can be effectively adapted for time series forecasting, bridging gaps between modalities.", "essential_references_not_discussed": "To the best of my knowledge, there are a series of existing studies discussing multimodal time series analysis [1, 2]. It would be good to discuss several of them for a more thorough comparison.\n\n[1] Y. Jiang et al. Multi-modal Time Series Analysis: A Tutorial and Survey. arXiv.\n\n[2] H. Liu et al. How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook. arXiv.", "other_strengths_and_weaknesses": "Overall, the insight of fusing textual and visual information with time series is interesting. The Time-VLM model can effectively address the complicated relationships among these three modalities. The experiments can verify the efficacy of these modalities. \n\nDespite the above merits, the paper can be improved in the following aspect:\n1. The projection from the text/images to the time series space needs more interpretation. Can the paper provide some case studies for a better illustration?\n2. According to Table 1 to Table 3, the proposed model can achieve good performance in zero/few-shot settings. However, the results in Table 4 and 5 are not significant, i.e., only limited improvements on the full-shot setting. Can the author clarify this point? In other words, if we have enough data, we don’t need other modalities anymore?\n3. How about the efficiency of the proposed model? Do the authors evaluate different model size of Time-VLM on these datasets? How is the sensitivity to the parameter size? Generally the model size should largely impact the model performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please make rebuttal to the weaknesses. I’ve listed several questions there. Thank you.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a multimodal framework using pre-trained vision-language models (VLMs) for time series forecasting.

After the rebuttal, three reviewers (Cr31, gzkF, and wRhk) gave positive ratings (3, 3, 3), recognizing the paper’s strong performance and novel ideas. Reviewer 6Pe4 also acknowledged the novelty and good performance of the proposed method but raised concerns about the lack of comparison with foundation models and the absence of certain analyses. The authors provided a thorough response; however, Reviewer 6Pe4 did not reply to the feedback.

The Area Chair (AC) agrees with most of the reviewers and recommends accepting the paper. Additionally, it is strongly suggested that the authors incorporate the content discussed in the feedback into the revised version to make the paper more comprehensive.
Decision: Accept (poster)