TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop
Authors: Yushan Jiang
Wenchao Yu
Geon Lee
Dongjin Song
Kijung Shin
Wei Cheng
Yanchi Liu
Haifeng Chen
Venue: NeurIPS 2025 poster ()
Keywords: multi-modal time series prediction
explanations
prototypes
large language models
Primary area: deep_learning

Abstract: Time series analysis provides essential insights for real-world system dynamics and informs downstream decision-making, yet most existing methods often overlook the rich contextual signals present in auxiliary modalities. To bridge this gap, we introduce TimeXL, a multi-modal prediction framework that integrates a prototype-based time series encoder with three collaborating Large Language Models (LLMs) to deliver more accurate predictions and interpretable explanations. First, a multi-modal prototype-based encoder processes both time series and textual inputs to generate preliminary forecasts alongside case-based rationales. These outputs then feed into a prediction LLM, which refines the forecasts by reasoning over the encoder's predictions and explanations. Next, a reflection LLM compares the predicted values against the ground truth, identifying textual inconsistencies or noise. Guided by this feedback, a refinement LLM iteratively enhances text quality and triggers encoder retraining. This closed-loop workflow---prediction, critique (reflect), and refinement---continuously boosts the framework's performance and interpretability. Empirical evaluations on four real-world datasets demonstrate that TimeXL achieves up to 8.9\% improvement in AUC and produces human-centric, multi-modal explanations, highlighting the power of LLM-driven reasoning for time series prediction.

TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop
Yushan Jiang' Wenchao Yu Geon Lee², 3 Dongjin Song Kijung Shin?,
Wei Cheng', Yanchi Liu", Haifeng Chen School of Computing, University of Connecticut 2 Data Science & System Security Department, NEC Labs America
Kimm Jaechul Graduate School of AI, KAIST
Abstract
Time series analysis provides essential insights for real-world system dynamics and informs downstream decision-makinn yet most existing methods often overlook the rich contextual signals present in auxiliary modalities. To bridge this gap, we introduce TimeXI., multi-modal prediction framework that integrates a prototype- based time series encoder with three collaborating Large Language Models (LLMM) to deliver more accurate predictions and interpretable explanations. First, a multi- modal prototype-based encoder processes both time series and textual inputs generate preliminary forecasts alongside case-based rationales. These outputs then feed into prediction LLM, which refines the forecasts by reasoning over the encoder's predictions and explanations. Next, reflection LLM compares the predieted values against the ground truth, identifying textual inconsistencies noise. Guided by this feedback, a refinement LLM iteratively enhances text quality and triggers encoder retraining. This closed-loop workfloo--rrediction, critique (reflect), and definementttconiinuuuu boosts the framework's performance and interpretability. Empirical evaluations four real-world datasets demonstrate that Ti achieves up to 8.9% improvement in AUC and produces human-centric, multi-modal explanations, highlighting the power of LLM-driven reasoning for time series prediction.
Introduction
In the modern big-data era. time series analysis has become indispensable for understanding real- world system behaviors and guiding downstream decision- making tasks across numerous domains, Although deep learning models have
including healthcare, traffic, finance, and weather [ demonstrated success in capturing complex temporal dependencies (3] 6 7 8) real-world time series are frequently influenced by external information beyond purely temporal factors. Such additional context, which may come from textual narratives (e.g., finance news [9] or medical reports [I0],, can offer critical insights for more accurate forecasting and explainability.
Recent multi-modal approaches for time series have shown promise by integrating rich contextual signals from disparate data sources, thereby improving performance across range of tasks includ- ing forecasting, classification, imputation, and retrieval [I] [2| [3, [4]. While these approaches utilize supplementary data enhance predictive accuracy, they often lack explicit mechanisms to systematically reason and explain about why or how contextual signals affect outcomes. This gap in interpretability poses significant barriers for high-stakes applications such as finance and healtheare, where trust and transparency are paramount.
Most of this work was done during the internship NEC Labs America. "Correspondncc to: Wenchao Yu wwwu@ncc-labs.com Dongjin Song <dongjin song @ @uco.n.edu>.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Meanwhile, Large Language Models (LLMs) [ 16 have risen to prominence for their remarkable ability process and reason over textual data across domains, enabling tasks like sentiment analysis, question answering, and content generation in zero- and few-shot settings [7
18 19 Recent
advancements agentic designs further augment L.LM capabílities, facilitating structured reasoning
and iterative decision-makinn for context-rich scenarios [20) [2]] Their encoded domain knowledge makes them natural candidates for supporting real-world multi-modal time series analyses, where
textual context (e.g., news or expert notes) plays vital role 22 23 24
Motivated by these observations, we intro- duce TimeXL, a novel framework that adopts a closed-loop workflow of prediction, cri- tique (reflect), and refinement, and unifies a prototype-driven time series encoder with LLM--based reasoning to deliver both accurate and interpretable multi-modal forecasting (Fig- ure D. Our approach first employs multi- modal prototype-based encoder to generate preliminary time series predictions alongside suman-readable explanations, leveraging case- based reasoning [25 26, (7] from both the tem- poral and textual modalities. These explana- tions not only justify the encoder's 's predictions but also serve as auxiliary sigmals to guide an LLM-powered component that further refines the forecasts and contextual rationales.
Feedbask that detects nenqual nolio
Input
Refinement LLM
Enhances text quality and biggers ancoddr retraining
Roflection LLM
Prototype- based Explainable Encoder
Prelimnary prenkction rationales
Chrpue
(2)
Tertual exparationss
Prediction from refinee rationalis, ground
Time series explnnations
Prediction
Prediction LLM
Figure An overview of the TimeXL workflow. A prototype-basee explainable encoder first produces predictions and case-based rationales for both time series and text. The prediction LLM refines fore- casts based on these rationales (Step 1). reflection LLM then critiques the output against ground truth (Step 2), providing feedback detect textual noise (Step 3). Finally, refinement LLM updates the text accordingly, triggering encoder retraining for improved accuracy and explanations (Step 4). Note that the predictions from the encoder and the LLM are also fused to enhance overall accuracy.
Unlike conventional methods that merely fuse multi-modal inputs for better accuracy, TimeXL iterates between predictive and re- finement phases to mitigate textual noise, fill knowledge gaps, and produce more faithful planations. Specifically, a reflection LLM di- agnoses potential weaknesses by comparing predictions with ground-truth signals, while refinement LLM incorporates these insights to update textual inputs and prototypes itera- tively. This feedback loop progressively im- proves both the predictive and explanatory capabilities of the entire system. Our contributions are summarized as follows:
We present a prototype-basee encoder that combines time series data with textual context, producing transparent, case-based rationales.
We exploit the interpretative prowess of LLMs reason over the encoder's outputs and iteratively refine both predictions and text, leading to improved prediction accuracy and explanations.
Experiments four real-worle benchmarks show that TimeXL consistently outperforms baselines, achieving up to a 8.9% improvement in AUC while providing faithful, human-centric multi-modal cxplanations. Overall, TimeXL opens new avenues for explainable multi- modal time series analysis by coupling prototype-based inference with L.LM-driven reasoning.
2 Related Work
Our work is primarily related to three key areas of time series research: multi-modality, explanation, and the use of LLMs. A more detailed discussion of related methods is provided in Appendix H
Multi-modal Time Series Analysis. In recent years, modti-modal time series analysis has gained increasing attention in diverse domains such as finance, healthcare and environmental sciences 228 13 Multiple approaches have been proposed to model interactions across different modalities,
including fusion and alignment through concatenation 29), attention mechanism [12 30 B1, gating
functions (32) and contrastive learning [33 34]. These methods facilitate tasks beyond forecasting

---

and classification, such as retrieval [14. imputation 13 and causal discovery [33] Recent research efforts have also advanced the field through comprehensive benchmarks and studies B5| 36 37].
highlighting the incorporation of new modalities to boost task performance. Meanwhile, others explore transforming time series to other modalities [8 39 40
42 or generating synthetic for effective analysis. Nevertheless, these studies primarily focus on improving numerical performance while the deeper and tractable rationale behind why or how the textual or other contextual signals influence time series outcomes remains underexplored.
ones [32 4
Time Series Explanation. Recent studies have explored diverse paradigms for time series inter- pretability. Gradient- based and perturbation- based saliency methods, for example, highlight important features different time steps [44] 445. where some works explicitly incorporate temporal structures
into models and objectives [66 47]. Surrogate approaches also offer global or local explanations,
such as applying Shapley values to time series [48), enforeing model consistency via self-supervised objectives [49], or using information -tteorrticc strategies for coherent explanations [50] In contrast to saliency or surrogate-base which end-to-end generates predictions and built-in explanations from learned prototypes. Our work extends this approach to multi-modal time series by producing human-readable reasoning artifacts for both the temporal and contextual modalities.
explanations, we adopt a case-based reasoning paradigm 25 26 277.
LLMs for Time Series Analysis. The rapid development of LLMs 15 16 has begun to inspire new directions in time series research [51] [2]. Many existing methods fine-tune pre-trained LLMs on time series tasks and achieves promising results [53] 54 551, where textual data (e.g., domain instructions, metadata, summaries) are often encoded as prefix embeddings to enrich time series 58 These techniques also contribute to the emergence of time series An alternative line of research leverages the zero-shot or few- shot capabilities of LLMs by directly prompting pre-trained language models with text-converted time series [63] or context-laden prompts representing domain knowledge [24], often yielding surprisingly strong performance in real-world scenarios. Furthermore, LLMs can act as knowledge inference modules, synthesizing high-level patterns explanations that augment standard time series pipelines [64] 633 66). Building on this trend, recent works have explored time series reasoning with problems, including time series understanding & question answering 66 68 69 77, 7 72], and language-guiin guided inference [67 69). Compared with existing
representations [56 57
foundation models 55
60
61
LLMs by framing tasks as natural-langueee
works, our work provides unique synergy between an explainable model and LLMs, enhancing explainable prediction through iterative, grounded, and reflective interaction.
3 Methodology
In this section, we present the framework for explainable multi-modal time series prediction with LLMs. We first introduce the problem statement. Next, we present the design of time series encoder that provides prediction and multi-modal explanations as the basis. Finally, we introduce three language agents interacting with the encoder towards better prediction and reasoning results.
3.1 Problem Statement
In this paper, we consider multi-modal time series prediction problem. Each instance is represented by the multi-modal input (xt 8). where x = (I),
.T) E N denotes time series data with N variables and T historical time steps, and denotes the corresponding text data describing the real-world context. The text data & can be further divided into T. meaningful segments. Based on the historical time series and textual context, our objective is to predict the future outcome y. either as a discrete value for classification tasks, or as a continuous value for regression tasks. We focus on the classification task reflect discrete decision-making scenarios common in real-world multi-modal applications, where interpretability is often essential for reliable decision support. We also include an evaluation of the regression setting in the Appendix G There are four major components in the proposed TimeXL framework, a multi-modal prototype encoder Mene that provides an initial prediction and case-based explanations, a prediction LLM Mprra that provides a prediction based on contextual understanding with explanations, a reflection LLM Mref that generates feedback, and refinement I.IM Mrefine that refines the textual context based on the feedback. Below, we introduce each component and how they synergize toward better prediction and explanation.
3

---

Time Series
Text input
Finetume
Prozen
Time
Once after training
Projection
Righ
Input:
Time Series Encoder
have rises infuenced bty..
K(Pt.)
Multi-modal
Probomype: indicating an incress of fron are price
Prototype-based Encoder
Series Explanation
Time series
Kipt.
Prediction Pras
Input
Relevance Fusion
Yen
Text Inport
Fusion
Language
Text Text Sequence Encoder
King
PEIM
Kin,
Prediction LLM
embedding
Kug
Rellection LLM
Refinement LLM
Projection
Multi-modal
Prototype- Encoder
Ground Truth y
The Training Protocol of TimeXL
Training
Text prototypes
Figure 2: The training protocol of TimeXL (left), and multi modal prototype-based encoder (right).
3.2 Multi-modal Prototype-basel Encoder
design multi-modal prototype-based encoder that can generate predictions and explanations across different modalities in end-to-end manner, as shown in Figure We introduce the model architecture, the learning objectives that yield good explanation properties of prototypes, and the pipeline of case-based explanations using prototypes.
3.2.1 Multi-modal Sequence Modeling with Prototypes
Sequence Encoder. To capture both temporal and semantic dependencies, we adopt separate encoders for time series (E&) and text (Ep). For NxT, the time series encoder Es maps the entire sequence into one or multiple representations, which serve as candidates for prototype learning. Simultancously, the text input is first transformed by frozen pre-trained language model, PLM (e.g.. BERT [73 or Sentence-BERR
74 ), to produce embeddings e, E RRd x These embeddings are then processed by separate encoder & to extract meaningful text features. It is worth noting that the choice of & and Eg also affects the granularity of explanations. As we will introduce shortly, the prototypes are learned based on sequence representations and associated with the counterparts in the input space, where the correspondences are determined by the encoders. In this paper, we choose convolution-baser encoders for both modalities capture the fine-grained sub-sequence (i.e., segment) patterns:
Z zime (Z #+1
Es(x),
Ztext (,1....,(_-w'4) = (e,es)
(I)
where 2; E T and z' E R denote segment-leve representations learned via convolutional kernels of sizes w and w', respectively.
Prototype Allocation. To establish interpretability, we learn set of time series protolypes and text prototypes for each class {1, C). Specifically, we introduce: plc) time E k pp()) text
(c) RR xh' so that each prototype p E " (time series) or p, (()) E TR (text) resides in the same feature
space the relevant encoder outputs. For input sequence, we measure the similarity between each prototype and the most relevant segment the corresponding modality:
Sim max(Sim (e).... .... Sim - -1)), where Sim = exp (-110,0 --, 23 =112²) E [0 (2)
We aggregate similarity scores across all prototypes for each modality, yielding Sim E B&C
and Simeet E R Finally, we jointly consider the cross-modal relevance and use a non-negative fusion weight matrix W E Cx++*') that translates these scores into class probabilities:
Jenc = Softmax W [Simtime (immtxxt
E [0, 1]c.

---

3.2.2 Learning Prototypes toward Better Explanation
Learning Objectives. The learning objectives include three regularization terms that reinforce the interpretability of multi-modal prototypes. In this paper, we focus on a predicting discrete label, where the basic objective is the cross-entropy loss for the prediction drawn from multi-modal explainable artifacts LCR = Em....y y log(Dene) (1 - y) 1 Denc). Besides, we encourage a clustering structure of segments the representation space by enforcing each segment representation to be adjacent to its closest prototype. Reversely, we regularize each prototype to be close to segment representation as possible, to help the prototype locate the most evidencing segment. Both regularization terms are denoted as Le and Le, respectively, where we omit the modality and class notations for ease of understanding:
L
Σ ()
min 1|z -Pill Le P.EP
Σ P.EP()
min lpp 2
2,112
Moreover, we encourage diverse structure of prototype representations to avoid redundancy and maintain compact explanation space, by penalizing their similarities via a hinge loss Ca, with threshold dminn
La = ΣΣ max (o, J#L
dmin dmin
|p: p)l12]
(5)
The full objective written as: LCE + A1 + X2h. + AzCa, with hyperparametens λι, 22, and A that balance regularization terms towards achieving an optimal and explainable prediction.
Prototype Projection. After learning objectives converge, the multi-modal prototypes are well- regularized and reflect good explanation properties. However, these prototypes are still not readily explainable as they are only close to some exemplar segments in the representation space. Therefore, we perform prototype projection to associate each prototype with training segment from its own class that preserves C. in the representation space, for both time series and text:
P +- arg min
p
2/133
23
VP E (()
By associating each prototype with a training segment in the representation space, the multi-modal physical meaning is induced. During testing phase, multi-modal instance will be compared with prototypes across different modalities to infer predictions, where the similarity scores and prototypes' class information assemble the explanation artifacts for reasoning.
3.3 Explainable Prediction with LLM-in-the-Loop
To further leverage the reasoning and inference capabilities of LLMS in real-world time series contexts, propose framework with three interacting LLM agents: a prediction agent Mpred, reflection agent Myen. and a refinement agent Myefine- These LLM agents interact with the multi-modal prototype-based encoder Mene toward better prediction accuracy and explainability.
3.3.1 Model Synergy for Augmented Prediction
Prediction with Enriched Contexts. The prediction LLM agent MMpred generates predictions based on the input text To improve prediction accuracy, the encoder Mone supplements with case-based explanations. Specifically, Mene selects the w prototypes that exhibit the highest relevance to any of the textual segments within s. Relevance is determined by the similarity scores used in Equation 2 These selected prototypes are then added to the input prompt of Mpred as explanations, providing richer real-word world context and leading more accurate predictions. The prototype-segne pairs, which construct the explanation expl, of the input text 8, are retrieved as follows:
expl, = (s)
(i.j.c) e Top-w((immexi)) ss)
}
= argTTop- (i.j.c)
Sim
Note that here p: ((0) denotes a raw text segment, and i, denote the prototype, segment, and class indices, respectively. As expl, contains relevant contextual guidance, it augments the input space and removes semantic ambiguity for prediction agent Mpread Therefore, the prediction is drawn as
ÔLLM Mprea(s, expl,). The prompt for querying M Mpred is provided in Appendix E Figure [4
5

---

Fused Predictions. We compile the final prediction based on fusion of both the multi modal encoder Mene and prediction LLM Mpred. Specifically, we linearly combine the continuous prediction probabilities Denc and discrete prediction ULLM ú o' + (1 D)LL.MM where a E [0. 1] is the hyperparameter selected from validation data. The encoder Meue and prediction agent M pred enhance each other based on their unique strengths. The Mene is fine-tuned based on explicit supervised signals, ensuring accuracy in capturing temporal and contextual dependencies of multi-modal time series. On the other hand, Mpree contributes deep semantic understanding drawn from extensive text corpora. By fusing predictions from two distinct perspectives, we achieve synergistic augmentation toward more accurate predictions for complex multi- molli-modal time series.
3.3.2 Iterative Context Refinement via Reflective Feedback
While the prediction agent Mpredd leverages the explainable artifacts to make informed predictions, it not inherently designed fit into the context of multi-modal time series data, which could lead to inaccurate predictions when the quality of textual context is inferior. To tackle this issue, we exploit another two language agents Myetl and Mrefine to generate reflective feedback and refinements on the context, respectively, for better predictive insights.
Given the prediction ÚLLM generated by the prediction agent Mpred. the reflection agent Mrea aims to understand the reasoning behind the implicit prediction logic of Mpred. Specifically, it generates a reflective feedback, Refl, by analyzing the input text s and its prediction UIIIM- against the ground truth y, to provide actionable insights for refinement, i.e., Refi = MMett (y. 91.I. s). Guided by the feedback, the refinement agent Mrefine refines the previous text 8; into 8 by selecting and empha- sizing the most relevant content, ensuring that important patterns are appropriately contextualized, which is similar how domain expert would perform, Le., 8
Mrefie (Refl, 8,). The prompts
for querying Mrefi and Mrefine are provided in Figures 16
18 19 and discussed in Appendix
We finally integrate the refinement via re- flection into the optimization loop of our proposed TimeXL, which is summarized in Algorithm||] [1] Once the textual context is improved. it used to retrain the multi- modal prototype-based encoder Mone for the next iteration. As such, the explana- tion (e.g., quality of the prototypes) and predictive performance of Mene can be improved through this iterative process. Consequently, the prediction agent Mpree could yield better prediction with more in- formative inputs, further enhancing the accuracy of ÿ. We evaluate the predictive performance of LLM and save the encoder and reflection when an improvement is ob- served (Fval(-) pass) when applied to the validation set. In the testing phase, we se- lect the reflections with the best validation performance to guide Mrefine in context refinement, mimicking how an optimized deep learning model is applied to unseen testing data.
Algorithm Iterative Optimization Loop of TimeXL
Inputs: Multi-ooal time series (æ, 8. y) with Derain. Dval. Dreat prototype-based encoder Menes prediction agent Mpred, reflection agent Mrea, refinement agent Mrefines fu- sion parameter a, max iteration т, improvement check Eval(.)
Training & Validation;
Initialize 80 = 8, 0, Mine + 2, Refl" <
while < t do
Train Mone using Dernin = {(x, 8;, y),
Dene expl, = Mene (Ir, 81)
ÚLLM = -MMrped -Mpr=Meaa(s expl,;)
aBanc + (1- @)
if Exal(Reft, Deai) pass then
L Monc +- -Moe RRel" <- Refl
Refl -Mea S.)
$141 = Mreane (Refl, s.)
increment i
return Miner Refl"
Testing:
s' Mrefine (Reff", 8) for Drest
i Done: expl, Mine (xx, 8')
LLMM = -Mrred (s', exple)
1 = aggnn + (1 a #LLM
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate methods on four multi-modal time series datasets from three different real- world domains, including weather, finance, and healthcare. The detailed data statistics are summarized in Table 3 of Appendix A.1 The Weather dataset contains meteorological reports and the hourly time series records of temperature, humidity, air pressure, wind speed, and wind direction in New York City. The task is to predict if it will rain in the next 24 hours, given the last 24 hours of weather

---

Table 1: The Fl score (F1) and AUROC (AUC) for TimeXL and state-of -fate-of-the-art baselines multi- modal time series datasets.
Weather
Finance
Hcalthcare (TP)
Healthcare (MT)
Datasets
Methods +
F1
AUC
F1
AUC
F1
AUC
F1
AUC
DLinear [75]
0.540
0.660
0.255
0.485
0.393
0.500
0.419
0.388
Autoformer 76
0.546
0.590
0.565
0.747
0.774
0.918
0.683
0.825
Crossformer 777
0,500
0.594
0.571
0.775
0.924
0.984
0.737
0,913
TimesNet [78]
0.494
0.594
0.538
0.756
0.794
0.867
0.765
0,944
Trranformer
0.541
0.650
0.600
0.783
0.861
0.931
0.791
0.963
TSMixer [80
0.488
0.534
0.465
0.689
0.770
0.797
0.808
0,931
TimeMixer 81
0.577
0.658
0.571
0.776
0.822
0.887
0.824
0.935
FreTS [82]
0.623
0.688
0.546
0.737
0,887
0.950
0.751
0.762
PatchTST [5]
0.592
0.675
0.604
0.795
0.841
0.934
0.695
0.928
LLMTime 83
0.587
0.657
0.519
0.643
0.802
0.817
0.769
0,803
PromptCast 63
0.499
0.365
0.418
0.607
0.727
0.768
0.696
0,871
OFA [53]
0.501
0.606
0.512
0.745
0.774
0.879
0.851
0.977
FSCA [S
0.563
0.647
0.592
0.790
0.820
0.891
0.872
0.977
Time-LLM 56
0.613
0.699
0.589
0.792
0.671
0.864
0.733
0,912
TimeCMA 57
0.636
0.731
0.559
0.727
0.729
0.828
0.693
0.843
MMM-Trannformer 335
0.608
0.689
0.605
0.793
0.926
0.986
0.901
0,990
MM-PatchTST
0.621
0.718
0.619
0.812
0.863
0.968
0.780
0,929
TimeCAP 65
0.668
0.742
0.611
0.801
0.954
0.983
0.942
0.988
TimeXL
0.696
0.808
0.631
0.797
0.987
0.996
0.956
0.997
records and summary. The Finance dataset contains the daily record of the raw material prices together with 14 related indices from January 2017 to July 2024. Given the last business days of stock price data and news, the task is to predict if the target price will exhibit an increasing, decreasing, or neutral trend on the next business day. The Healtheare datasets include Test-Positive (TP) and Mortality (MT), both comprising weekly records related to influenza activity. TP contains the number of respiratory specimens testing positive for Influenza A and B, while MT reports deaths caused by influenza and pneumonia. For both datasets, the task is to predict whether the respective target ratio, either the test-positive rate or the mortality ratio the upcoming week will exceed the historical average, based records and summaries from the previous 20 weeks.
Baselines, Evaluation Metrics, and Setup We compare TL with state-of- of-the-art baselines for time series prediction, including Autoformer (76). Dlinear (75), Crossformer (77), TimesNet (78), PatchTST (5) iTransformer (79], FreTS [82). TSMixer 80), TimeMixer [81] (classification imple- mented by TSlib (78) ) and LLM-based methods like LLMTime 83 PromptCast [63], OFA [53]. Time-LLM (56). TimeCMA (57 and FSCA (58). where LLMTime and PromptCast don't 't need fine-tuning. While some methods primarily used for time series prediction with continuous values, they can be casily adapted for discrete value prediction. We also evaluate the multi-modal time series methods. Besides the Time-LLM, TimeCMA and FSCA where input text is used for embedding re- programming and alignment, we also evaluate Multi-modal PStchTST and Multi-modal i'rrasformer from (35. as well as TimeCAP (65) We evaluate the discrete prediction via F1 score and AUROC (AUC) score, due to label imbalance in real-world time series datasets. All datasets are split 6:2:2 for training, validation, and testing. All results averaged over five experimental runs. We alternate different embedding methods for texts based on its average length (Sentence-BERY 77) for finance
dataset and BERT 73 for the others). More details provided in Appendix A.2
A.3
A.4
4.2 Performance Evaluation
The results of predictive performance are shown in Table [] It is notable that multi-moda methods generally outperform time series methods across all datasets. These methods include LLM methods (e.g., Time-LLM [56), TimeCMA [57] FSCA (58) that leverage text embeddings to enhance time series predictions. Moreover, the multi-modal variants (MM-iTrrnsformmer and MM-PatchTST (35)
improve the performance of of-e-off-hhe-rr time series methods, suggesting the benefits of integrating
7

---

integrates the predictions from both modalities, further improving the predictive performance. TimeXL constantly achieves the highest Fl and AUC scores, consistently surpassing both time series and multi-modal baselines by up to 8.9% of AUC (compared to TimeCAP on the Weather dataset). This underscores the advantage of TLmeXL., which synergizes multi-modal encoder with language agents to enhance interpretability and thus predictive performance in multi-moda time series.
real-world contextual information. Besides, TimeCAP 665|
Rain
Multi-model Prototypes
Rain
Humiiity (96)
Tempeneture
Wind
Wind dircction
Humidiiy
Temperaturn
Wind apwedd (m/9)
Wind direction
suggest likelihood dry westher, with stable pressure
signel the spproach of weather system low-pressure system that leed worsening westher
the wind direction primarily been westerly shifts, which typically bring cooler, drier air indicating strengthening high- pressure system
humidity levels were predominently high, wind speeds were
Air pressure bean dacreasing overall, which wind direction started westerly, became variable, and
drier moving
Air pressure remained relatively
Key time series prototypes and text prototypes learned on the Weather dataset. Each row in the figure represents a time series prototype with different channels.
Figure 3:
4.3 Explainable Multi-modal Prototypes
Next, we present the explainable multi-modal prototypes rendered by TimeXL, which establishes the case-based reasoning process. Figure 3 shows a subset of time series and text prototypes learned on the weather dataset. The time series prototypes demonstrate the typical temporal patterns aligned with different real-world weather conditions (i.e.. rain and not rain). For example, a constant or decreasing humidity moderate level, combined with high and steady air pressure, typically indicates non-rainy scenario. The consistent wind direction is also a sign of mild weather conditions. On the contrary, high humidity, low and fluctuating pressure, along with variable winds typically reveal an unstable weather system ahead. In addition to time series, the text prototypes also highlight consistent semantic patterns for different weather conditions, such the channel-specifie (e.g., drier air moving into the area, strengthening of high-pressure system) and overall (e.g., a likelihood of dry weather) descriptions of weather activities. In Appendix D.I] we also present more multi-modal prototypes for the weather dataset in Figure / for the finance dataset in Figure TO, and for healthcare datasets in Figures| I1 and 12, TimeXL provides coherent and informative prototypes from the exploitation of time series and its real-world contexts, which facilitates both prediction and explanation.
4.4 Multi-modal Case-based Reasoning
Building upon the multi-modal prototypes, we present a case study on the testing set of weather data, comparing the original and TimeXL's reasoning processes highlight its explanatory capability, as shown in Figure 4 In this case, the original text is incorrectly predicted as not rain. We have three key observations: (1) The refinement process filters the original text to emphasize weather conditions more indicative of rain, guided by reflections from training examples. The refined text preserves the statement on stability while placing more emphasis on humidity and wind as key indicators. (2) segment-prototype Accordingly, the matched segment- prototype pairs from the original text focus more on temperature stability and typical diurnal variations, while the matched pairs in the refined text highlights wind variability, moisture transport, and approaching weather system, aligning more with rain conditions. (3) Furthermore, the reasoning on time series provides a complementary view for assessing weather conditions. The matched time series prototypes identify high humidity and its drop-and-rise trends, wind speed fluctuations and directional shifts, and the declining phase of air pressure fluctuations, all of which are linked to the upcoming rainy conditions. The matched multi-modal prototypes
8

---

Original Reasoning
Rain
Prodiction: main
Timexi:
Original Text: Over the 24 hours, New York City experienced gradual. increase tampereture throughout the day, peaking in the late afternoon befors a modest decline towerds the evening. Humidity levels started high in the esrly hours, dropped sighlicannly during the day, and then showed rising trend and the period, indicating a possibla increase in moisture content in the air. Air pressure remained relatively stable with only minar fluctustions, suggesting pariod of stable weather conditions. Wind speeds varied, with a noticeable increese during the late afternoon, which
Pressuug
contributed period of cooler and more turbulent conditions. Wind direction was predominentl from the south- southwest, shifting siihtly
south southes orientation later the day, which is typical for the region's westher patterne during this time.
Prototypes:
Not min: City has experienced reletivaly stable temperatures with a slight warming trend abserved (0.78)
Temperature (1)
Not rain: peak during the late afterooon before beginning decrease slightly into the (0.54)
Not rain: dropping again, indicative of typical diumal variation (0.51)
214-
TimeXL: Text Reasoning
Prediction: Rain
Wind spaed (irie)
past hours, New City experienced a air preesure pattern with minor fluctuations, indicating stable weather conditions. The day sew gradual increase in temperature, peaking the evaning. Humidity levals high early dropped
Refined Text:
in the late afternaon before daclining significantly during the day. and rose again later, suggesting incressed moisture content. Wind direction shifted from south - southwest to south - southeast, bringing moisture laddturr--lden air, which could increase the
likelihord of rain.
Wort coreettettn pr
Prototypes:
Rain: direction wes variable without a consistent pettern. These indicators suggest (0.47)
Rain: direction started westerly, became variable, and (0.64)
Rain: which could signal the spproach of weather system (0.53)
Figure 4: Multi-modal case-based reasoning example on the Weather dataset. The left part illustrates the reasoning process for both the original and refined text in TimeXL, with matched prototype-ing pairs highlighted in the same color along with their similarity scores. The right part presents the time series reasoning in TimeXL, where matched prototypes are overlaid on the time series.
from TimeXL demonstrate its effectiveness in capturing relevant information for both predictive and explanatory analysis. We also provide a case study on the Finance dataset in Figure 13 where textual explanations generated at the granularity of half-sentence.
4.5 Iterative Analysis
Prediction LLM with Text: F1 Score
Prediction LLM with Text: AUC Score
To verify the effectiveness of overall workflow with reflection and refinement LLMs shown in Figure [ we conduct an iterative analysis of text quality and TimeXL performance, as shown in Figure 5 Specifically, we evalu- ate the text quality based on its zero-shot pre- dictive accuracy using an LLM. Notably, the text quality benefits from iteration improve- ments and mostly saturates after one or two iterations. Correspondingly, TimeXL perfor- mance quickly improves and stabilizes with very minor fluctuations. These observations underscore how TimeXL alternates between predictive and reflective refinement phases mitigate textual noise, thus enhancing its pre- dictive capability.
Weather
Weather
Finance
Healthcare(TP)
cealth
0.8
Healthcare/MTT
coalth
0.6
0.7
0.5
Original
Ieer
Ber2
Iters
Original
Iter]
Iter2
herd
TimeXL: F1 Score
TimeXL: AUC Score
1.0
Weathor
Weather
Anance
0.8
caalth-areITP
Hes (TC
Hea Mi
0.6
Original
Itor
Roa2
Iters
Original
Itor)
Ioor
Figure 5: The text quality and TimeXL performance over iterations.
4.6 Ablation Studies
We present the component ablations of TimeXL in Table 2 where the texts are refined using the reflections with the best validation performance. Firstly, the multi-modal encoder consistently outperforms single-modally series and textual modalities. Secondly, the prediction LLM exhibits better performance than text-only encoder, underscoring the advantage of LLM's contextual understanding and reasoning. Furthermore, the text prototypes consistently improve the predictive performance of LLM, underscoring the effectiveness of explainable artifacts from the multi-modal encoder, in terms of providing relevant contextual guidance. Finally, the fusion of prediction L.M and multi-modal encoder further enhances the predictive performance. surpassing the best results achieved by either component alone. These
variants across all datasets, highlighting the benefit of integrating time

---

Table 2: Ablation studies of TimeXL on multi-modal time series datasets.
Ablation 1
Variants
Weather
Finance
Healtheare (Tesi-Posilive
Healtheare (Murtality)
F1
AUC
F1
AUC
F1
AUC
F1
AUC
Encoder
Time Sories
0.602
0.691
0.585
0.751
0.889
0.957
0.86
0.965
Text
0.567
0.658
0,472
0.636
0.871
0,941
0.840
0.887
mulii-modal
0.674
0.767
0.619
0.791
0.934
0.974
0,937
0,988
0.645
0.724
0.496
0.627
0.967
L.M
Text Prototype
0.667
0.739
0.544
0.662
0.987
0.983
0,952
0.976
Select Best
0.674
0.767
0.619
0.791
0.987
0.983
0.952
0.988
Fosion
TimeXL
0.696
0,808
0.631
0.797
0.987
0.996
0,956
0.997
observations demonstrate the advantage of our framework synergizing the time series model and LLM for mutually augmented prediction. In Appendix [B] additional ablation and model studies are presented, including analyses of the multi-modal encoder (Figure Table 4 5 6 guantitative evaluation of case-based explanations (Figure 77. and comparisons of base LLMS (Table Figure K).
5 Conclusions
In this paper, we present TimeXL, an explainable multi-modal time series prediction framework that synergizes designed prototype-based encoder with three collaborative LLM agents in the loop (prediction, reflection, and refinement) to deliver more accurate predictions and explanations. Experiments on four multi-modal time series datasets show the advantages of TimeXL over state-of- the-art baselines and its excellent explanation capabilities.
6 Acknowledgments
Dongjin Song gratefully acknowledges the support of the National Science Foundation under Grant No. 2338878, as well as the generous research gifts from NEC Labs America and Morgan Stanley.
References
[1] Bo Jin, Haoyu Yang, Leilei Sun, Chuanren Liu, Yue Qu, and Jianing Tong. A treatment engine by predicting next-period prescriptions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1608-1616, 2018
[2] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 922-929, 2019.
[3] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 2141-2149, 2017.
[4] Yao Qin, Dongjin Song, Haifeng Cheng, Wei Cheng, Guofei Jiang, and Garrison W Cottrell. A dual-stage attention tresee recurrent neural network for time series prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 0627-2633, 2017.
[5] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. The Eleventh International Conference on Learning Representations, 2023.
[6] Ailin Deng and Bryan Hooi. Graph neural network- anomaly detection in multivariate time series. Proceedings of AAAI conference on artificial intelligence, volume 35, pages 2021.
[7] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35:3988-4000 2022.
10

---

[8] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long, Koopa: Learning non non-stationary time series dynamics with koopman predictors, Advances Neural Information Processing Systems, 36, 2024.
[9] Zihan Dong, Xinyu Fan, and Zhiyuan Peng. Fnspid: A comprehensive financial news dataset in time series. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4918 -927,, 2024.
[10] Ryan King, Tianbao Yang, and Bobak J Mortazavi. Multimodal pretraining of medical time series and notes. In Machine Learning for Health (MLAH), pages 244-255. PMLR, 2023.
[11] Ke Niu, Ke Zhang, Xueping Peng, Yijie Pan, and Naian Xiao. Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction. Frontiers Molecular Biosciences, 10:1 10:1136071, 2023.
[12] Geon Lee, Wenchao Yu, Wei Cheng, and Haifeng Chen. Moat: Multi-modal augmented time series forecasting. 2024.
[13] Xiaohu Zhao, Kebin Jia, Benjamin Letcher, Jennifer Fair, Yiqun Xie, and Xiaowei Jia. Vimts: Variational- imputation for multi-modal time series. In 2022 IEEE International Confer- ence on Big Data (Big Data), pages 349-358. IEEE, 2022.
[14] Tom Bamford, Andrea Coletta. Elizabeth Fons, Sriram Gopalakrishnan, Svitlana Vyetrenko, Tucker Balch, and Manuela Veloso. Multi-modal financial time-series retrieval through latent space projections. In Proceedings of the Fourth ACM International Conference on Al in Finance, pages 498-506, 2023.
[15] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[16] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 18805, 2023.
[17] Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881-3906 2024.
[18] Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 5591-5606, 2023.
[19] Zixuan Wang, Qinkai Duan, Yu-Wing Tai, and Chi-Keung Tang. C3Imm: Conditional multimodal content generation using large language models. arXiv preprint arXiv:2405.161 2024.
[20] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[21] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[22] Xin Liu, Daniel McDuff, Geza Kovacs, Isaae Galatzer-Levy, Jake Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners.
[23] Kelvin JL Koa, Yunshan Ma, Ritchie Ng, and Tat-Seng Chua. Learning to generate explainable stock predictions using self-reflective large language models. In Proceedings of the ACM on Web Conference 2024, pages 4304-4315, 2024.

---

[24] Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. Where would i go next? large language models as human mobility predictors. arXiv preprint arXiv:2308. 2023.
[25] Yao Ming. Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learn- ing via prototypes. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 903-913, 2019.
[26] Jingchao Ni, Zhengzhang Chen, Wei Cheng. Bo Zong, Dongjin Song, Yanchi Liu, Xuchao Zhang, and Haifeng Chen. Interpreting convolutional sequence model by learning local proto- types with adaptation regularization. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1366-1375, 2021.
[27] Yushan Jiang, Wenchao Yu, Dongjin Song, Lu Wang, Wei Cheng, and Haifeng Chen. Fedskill: Privacy preserved interpretable skill learning via imitation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1010-1019, 2023.
[28] Geri Skenderi, Christian Joppi. Matteo Denitto, and Marco Cristani. Well googled is half done: Multimodal forecasting of new fashion product sales with image-based google trends. Journal 2024.
of Forecasting, 43(6):1982-1999
[29] Zihao Li, Xiao Lin, Zhining Liu, Jiaru Zou, Ziwei Wu, Lecheng Zheng, Dongqi Fu, Yada Zhu, Hendrik Hamann, Hanghang Tong, and Jingrui He. Language in the flow of time: Time-series- paired texts weaved into unified temporal narrative, 2025.
[30] Sameep Chattopadhyay, Pulkit Paliwal, Sai Shankar Narasimhan, Shubhankar Agarwal, and Sandeep Chinchali. Context matters: Leveraging contextual features for time series forecasting. arXiv preprint (rXXi:2410.12672. 2024.
[31] Chen Su, Yuanhe Tian, and Yan Song. Multimodal conditioned diffusive time series forecasting. arXiv preprint arXiv:2504. 19669,
[32] Siru Zhong, Weilin Ruan, Ming Jin, Huan Li, Qingsong Wen, and Yuxuan Liang. Time-vim: Exploring multimodal vision-languag models for augmented time series forecasting. arXiv preprint arXiv:2502.04395 2025.
[33] Lecheng Zheng, Zhengzhang Chen, Jingrui He, and Haifeng Chen. Mulan: Multi-modal causal structure learning and root cause analysis for microservice systems. In Proceedings of the ACM on Web Conference 2024, pages 4107-41 16, 2024.
[34] Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong. Frozen language model helps eog zero-shot learning. In Medical Imaging with Deep Learning, pages 402-415. PMLR, 2024.
[35] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, al. Time-mmd: A new multi-domain multimodal dataset for time series analysis. arXiv preprint 27X70.2406.08627. 2024.
[36] Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Cha- pados, et al. Context is key: benchmark for forecasting with essential textual information. arXiv preprint arXiv:2410. 18959, 2024.
[37] Yushan Jiang, Kanghui Ning. Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, and Dongjin Song. Multi-modas time series analysis: A tutorial and survey. arXiv preprint arXiv:2503. 13709, 2025.
[38] Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. Advances in Neural Information Processing Systems, 000-9187--49244 2023.
[39] Shi Bin Hoo, Samuel Müller, David Salinas, and Frank Hutter. The tabular foundation model tabpfn outperforms specialized time series forecasting models based on simple features. arXiv preprint arXiv:2501.0294. 2025.
12

---

[40] Jingchao Ni, Ziming Zhao, ChengAo Shen, Hanghang Tong, Dongjin Song, Wei Cheng, Dongsheng Luo, and Haifeng Chen. Harnessing vision models for time series analysis: A survey. arXiv preprint 98X8v:2502.08869, 2025.
[41] Xiongxiao Xu, Yue Zhao, S Yu Philip, and Kai Shu. Beyond numbers: A survey of time series analysis in the era of multimodal llms. Authorea Preprints, 2025.
[42] Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, and Chenghao Liu. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters. arXiv preprint arXiv:2408.17255. 2024.
[43] Xu Liu, Taha Aksu, Juncheng Liu, Qingsong Wen, Yuxuan Liang, Caiming Xiong, Silvio Savarese, Doyen Sahoo, Junnan Li, and Chenghao Liu. Empowering time series analysis with synthetic data: A survey and outlook in the era of foundation models. arXiv preprint 1rXiv:2503.11411, 2025.
[44] Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi Bench- marking deep learning interpretability in time series predictions. Advances in neural information processing systems, 33:6441 11644116452,, 2020.
[45] Sana Tonekaboni, Shalmali Joshi, Kieran Camphell, David K Duvenaud, and Anna Goldenberg. What went wrong and when? instance-wis, feature importance for time-series black-box models. Advances in Neural Information Processing Systems, 33:799-809, 2020.
[46] Kin Kwan Leung, Clayton Rooke, Jonathan Smith, Saba Zuberi, and Maksims Volkovs. Tempo- ral dependencies in feature importance for time series prediction. In The Eleventh International Conference on Learning Representations.
[47] Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks. In International Conference on Machine Learning, pages 2166-2177 PMLR, 2021.
[48] João Bento, Pedro Saleiro, André F Cruz, Mário AT Figuciredo, and Pedro Bizarro. Timeshap: Explaining recurrent models through sequence perturbations. In Proceedings of the 27th ACM SIGKDD conference knowledge discovery & data mining, pages
2021.
[49] Owen Queen, Tom Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, and Marinka Zitnik. Encoding time-series explanations through self-supervieed model behavior consistency. Advances in Neural Information Processing Systems, 36, 2024.
[50] Zichuan Liu, Tianchun Wang, Jimeng Shi, Xu Zheng, Zhuomin Chen, Lei Song, Wenqian Dong, Jayantha Obeysekera, Farhad Shirani, and Dongsheng Luo. Timex++: Learning time-series explanations with information bottleneck. In Forty-first International Conference Machine Learning, 2024.
[51] Yushan Jiang, Zijie Pan, Xikun Zhang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. Empowering time series analysis with large language models: survey. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 0095-8103, International Joint Conferences on Artificial Intelligence Organization, 2024. Survey Track.
[52] Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6555-6565, 2024.
[53] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322-43355 2023.
[54] Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, and Qiang Xu. Multi-patch prediction: Adapting language models for time series representation learning. In Forty-first International Conference Machine Learning.
13

---

[55] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815, 2024.
[56] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yo Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024.
[57] Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang. Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. Timecma: Towards IIm-empowered multivariate time series forecasting via cross-modalityy alignment. In AAAI, 2025.
[58] Yuxiao Hu, Qian Li, Dongxiao Zhang, Jinyue Yan, and Yuntian Chen. Context-alignment Activating and enhancing LLMs capabilities in time series. In The Thirteenth International Conference on Learning Representations, 2025.
[59] Chenxi Liu, Shaowen Zhou, Hao Miao, Qianxiong Xu, Cheng Long, Ziyue Li, and Rui Zhao. Efficient multivariate time series forecasting via calibrated language models with privileged knowledge distillation. arXiv preprint arXiv:2505.02 2025.
[60]
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou, A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688, 2023.
[61] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592, 2024.
[62] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang. Jianmin Wang, and Mingsheng Long, Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02330 2024.
[63] Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
[64] Zihan Chen, Lei Nico Zheng, Cheng Lu, Jialu Yuan, and Di Zhu. Chatgpt informed graph neural network for stock movement prediction. Available at SSRN 4464002, 2023.
[65] Geon Lee, Wenchao Yu, Kijung Shin, Wei Cheng, and Haifeng Chen. Timecap: Learning to contextualize, augment, and predict time series events with large language model agents. In AAAI, 2025.
[66] Xinlei Wang. Maike Feng. Jing Qiu, Jinjin Gu, and Junhua Zhao. From news to forecast: Integrating event analysis in Ilm-based time series forecasting with reflection. arXiv preprint wwXiv:2409.1751. 2024.
[67] Jialin Chen, Aosong Feng, Ziyu Zhao, Juan Garza, Gaukhar Nurbek, Cheng Qin, Ali Maatouk, Leandros Tassiulas, Yifeng Gao, and Rex Ying. Mthench: A multimodal time series benchmark for temporal reasoning and question answering. arXiv preprint arXiv:2503. 16858, 2025.
[68] Yaxuan Kong, Yiyuan Yang, Shiyu Wang, Chenghao Liu, Yuxuan Liang, Ming Jin, Stefan Zohren, Dan Pei, Yan Liu, and Qingsong Wen. Position: Empowering time series reasoning with multimodal 1lms. arXiv preprint arXiv:2502.01477, 2025.
[69] Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. Time-mqa: Time series multi-task question answering with context enhancement. arXiv preprint arXiv:2503.0 2025.
[70] Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, and Jianxin Liao. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. arXiv preprint arXiv:2412. 2024.
[71] Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. Chatts: Aligning time series with Ilms via synthetic data for enhanced understanding and reasoning. arXiv preprint 2:10:22412.03104, 2024.
14

---

[72] Yifu Cai, Arjun Choudhry, Mononito Goswami, and Artur Dubrawski. Timeseriesexam: A time series understanding exam. arXiv preprint arXiv:2410.14752, 2024.
[73] Jacob Devlin Ming-Was Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacL-HLI, volume 1, page 2. Minneapolis, Minnesota, 2019.
[74] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Association for Computational Linguistics, 11 2019.
Processing.
[75] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
[76] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419-22444 2021.
[77] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting, In The eleventh international conference on learning representations, 2023.
[78] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2023.
[79] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu. Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
[80] Si-An Chen, Chun-Liang Li, Sercan O Arik, Nathanael Christian Yoder, and Tomas Pfister. Tsmixer: all-mlp architecture for time series forecast-ing. Transactions on Machine Learning Research, 2023.
[81] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and JUN ZHOU Timemixer: Decomposable multiscale mixing for time series forecasting, In The Twelfth International Conference on Learning Representations.
[82] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He. Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency- mips are more effective learners in time series forecasting, Advances in Neural Information Processing Systems, 36, 2024.
[83] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large Language Models Are Zero Shot Time Series Forecasters. In NeurlPS, 2023.
[84] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142. 2014.
[85] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.1553 155594 2024.
[86] Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, Yucong Luo, and Enhong Chen. Instructime: Advancing time series classification with multimodal language modeling. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM '25, page New York, NY, USA, 2025, Association for Computing Machinery.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction in this paper clearly state the scope and main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper provides discussion of limitations in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

This paper does not include theoretical analysis, instead focusing on the novel framework design and its empirical advantages.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
This paper provides detailed methodology, with clear experimental setting, which are also reproducible with the provided code and dataset.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
This paper provides code and dataset in the supplementary materials with instructions.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: This paper provides experimental settings with these details the manuscript and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We indicate that all results in this paper are reported as the average of five experimental runs. The paper also provides explanations of the statistical methods used for metric computation to ensure the robustness of reported findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information on the computer resources in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
This paper aligns with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper contains balanced discussion of the potential socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We tried our best to review and monitor the LLMs responses, where the prompt templates are provided, and the datasets are from general domain, which poses minimal risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
This paper clearly credits the creators of any used assets in the manuscript.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces new assets along with documentations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper doesn't It contain crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects, IRB approvals (or equivalent) are not required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The process of our method development doesn't involve LLMs. Regarding the method itself, it is multi-modal prediction framework integrates prototype-base -based time series encoder with three collaborating LLMs to deliver more accurate predictions and interpretable explanations.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: /tepriip/neurps... hwtp://///oooeeeenc//////////// for what should or should not be described.
22

---

A Experimental Settings
A.1 Dataset Statisties
In this subsection, we provide more details of the real- weal-world datasets we used for the experiments. The data statistics are summarized in Table including the meta information (e.g., domain resolution, duration of real-world time series records), the number of channels and timesteps and so on. We use the Weather and Healthcare datasets in TimeCAP (65) and a Finance dataset extended from [12].
The Weather dataset contains the hourly time series record of temperature, humidity, air pressure, wind speed, and wind direction and related weather summaries in New York City from October 2012 to November 2017. The task is to predict if it will rain in the next 24 hours, given the last 24 hours of weather records and summary.
JI
The Finance dataset contains the daily record of the raw material prices together with 14 related indices ranging from January 2017 to July 2024 with news articles. The task is to predict if future prices will increase by more than 1,, decrease by more than 1,, exhibit neutral trend on the next business day, given the last 5 business days of stock price data and news.
The Healthcare datasets are related testing cases and deaths of influenze Positive) dataset consists of the weekly records of the number of positive specimens for Influenza A and B, and related healthcare reports. The task is to predict if the percentage of respiratory specimens testing positive in the upcoming week for influenza will exceed the average value, given the records and summary in the last 20 weeks. Similarly, the Healtheare (Mortality) dataset contains the weekly records and healthcare reports of influenza and pneumonia deaths. The task is to predict if the mortality ratio from influenza and pneumonia will exceed the average value, given the records and summary in the last 20 weeks.
| The Healthcare (Test-
Table 3: Summary of dataset statistics.
Domain
Dataset
Resolution
Channels
Timesteps
Duration
Ground Truth Distribation
Weather
New York
Hourly
5
45,216
2012,10 - 2017.11 Rain (24.26%) / Not rain (75.74%)
Finance
Raw Material
Daily
15
1,876
2012.09 - 2022.02
Inc. (36.7% Dec. (34,1%) Neutral (29.25)
Healthcare
Test-Positive
Weekly
6
447
2015.
2024.04
Not exesed (65.77%) Exceed (34.23%)
Healthcare
Mortality
Woekly
4
395
2016.07 2024.06
Not excsed (69.33%) Exceed (30.67%)
A.2 Hyperparameters
First, we provide the hyperparameters of baseline methods. Unless otherwise specified, we used the default hyperparameters from the Time Series Library (TSLib) 77]. For I.LMTime [33 OFA 53
Time-ILM (56), TimeCMA [57]. TimeCAP (65) FSCA (58), we use their own implementations. For all methods, the dropout rate {0.0, 0.1, 0.2). learning rate E {0.0001, 0.0003, 0.001 For
21. the number of attention heads E {4, 8, 16). For Dlinear (75, moving average attentionlayers layers (3. For TimesNet (78) the number of layers E {1, 2}. For PStchTST (3) MS-PatchTST 35) and FSCA [58) the patch size E {3.5) 5} for the finance dataset. For TimeCAP (5),
transformer-basee and LLM fine-tuning methods [76 77 79
5 57 (56), the number of attention
we use the encoder with the prediction LLM, given the inherent multi-modat time series input. Next, we provide the hyperparam- eters of Ti.eXI. The numbers of time series prototypes and text prototypes are h (5, 10, 15, 20) and K'' E [5. 10), respectively. The hyperparameters controlling regularization strengths are A Xz, As E [0.1, 0.3] with interval 0.05 for individual modality, dmin E (1.0, 1.5, 2.0} for time series, dmin E {3.0, 3.5, 4.0} for text. Learning rate for multi-modal encoder E (0.0001, 0.0003, 0.001), } using Adam [84 as the optimizer. The number of case-based explanations fed to prediction LLM w E {3, 5,8, 10). All results are reported as the average of five experimental runs.
4 htaps:/wwww.inneemmmdii..oo///// http:///emmeectcooooooooooiiiiee
23

---

A.3 Large Language Model
employed gp1-40-2024-08-06 version for GPT-4o in OpenAl API by default. We use parameters nax_tokens=2048, top_p=1, and temperature=0.7 for content generation (self-reflection and text refinement), and 0.3 for prediction. We keep the same setting for Gemini-2.0 2omimi-2.0-Flash and GPT-4o-mini due to the best empirical performance.
A.4 Environment
We conducted all the experiments a TensorEX server with 2 Intel Xeon Gold 5218R Processor (each with 20 Core), 512GB memory, and 4 RTX A6000 GPUs (each with 48 GB memory).
B More Ablation Studies and Model Analysis
Ablations of learning objectives: We provide an ablation study on the learning objectives of TimeXL encoder, as shown in Figure The results clearly show that the full objective consistently achieves the best encoder prediction performance, highlighting the necessity of regularization terms that enhance the interpretability of multi modal prototypes. The clustering (A1) and evidencing (A2) objectives also play crucial role in accurate prediction: the clustering term ensures distinguishable prototypes across different classes, while the evidencing term ensures accurate projection onto training data.
Weather
AUC
AUC
0.75
0.725
0.700
0.65
0.755
0.53
0.55
0.625
0.600
Fill objective
Time
line
Tiree
Ject
Hest
Helt
Fil1
Time
lime
Time
bect
Iest
Gg
1Ay 9)
(Ay 0)
(A-
(Ay
(TP) Performance
(MT)
1,000
0.98
0.975
0.96
0.925
0.900
0.90
0.88
0.88
0.825
FUl1 obseciie
Time Us 0)
Time Or -01
Time 0)
Toit
Text (A) 0)
Text
0.800
Fall objettve
Timee
Time
Time (k-
Text
Toxt
Rat
M4 0]
IAv 6)
Us U5-00
0)
Figure 6: Ablation study of multi-modal encoder learning objectives.
Sensitivity of regularization strengths: To quantitatively evaluate the sensitivity of regularization strengths in the learning objectives, we report the performance of the multi-modal encoder on the Weather dataset (using the refined test texts), detailed in Appendix A.2 As shown in Table model performance remains relatively stable across different values. For both modalities, moderate values of X and A tend to yield better performance, suggesting the clustering and evidencing effectively guide meaningful prototype learning. A slight diversity constraint (Az) also helps to keep a compact interpretation prototype space.
Sensitivity prototype lengths: We evaluate the sensitivity
multi-modal encoder to prototype length by varying the single-kernel sizes for both time-series and text modalities on the Weather and Finance datasets (using the refined test texts), as shown Tande We observe that performance is relatively stable across range of prototype lengths the Weather dataset, with optimal results

---

Table 4: Sensitivity analysis of different À values for multi-modal encoder on the Weather dataset.
Modality
0.1
0.15
0.2
0.25
F1 AUC
0.030
0.0077
0.603-10.029
0.627
0.625 100030
A1
0.731=0.022
0.706
0.736=20.026
0.021
0.729::00023
Time
0.022-0.028
0.634.10.028
0.627 0.007
0.626_10.028
0.628.10.023.
A2
AUC
0.734=0.024
0.736=0.022
0.733=00.028
0.729=0.021
0.733=+0.025
FI
0.63==0.028
0.607
0.630=0.028
0.624=0.032
0.628=0.029
Aa
AUC
0.736=0.022
0.735=00024
0,733=0.024
0.727=0.026
0.733=#0.025
FI
0.629=0.027
0.629=0.022
0.635=0.027
0.634=0.028
A
ALC
0.722
0.006
0.740
0.738 0.023
0.705
Text
F1
0.632=0.031
0.631 0.002
0.605:00.024
0.029
0.630:10.028
A2
AUC
0.027
0.736
0.740
0.735
0.736.10.028
X
FI
0.026
0.003
0.628:10.025
0.625=0.029
0.627+40.077
ALC
0.740
0.736.1..025
0.0003
0.729
0.0011
typically achieved at moderate lengths for both modalities. For time series, the 8-hour segments are often sufficient to capture the typical temporal patterns of weather conditions, and a 12-token text window (with BERT) provides enough local contexts for weather prediction. For the Finance dataset, which operates on a business-day granularity, performance improves with longer lengths of time series prototypes that capture important financial trends. For the textual modality, we use Sentence-BFiR to embed financial news the half-sentence granularity, and shorter text windows (1-2 segments) typically provide compact and indicative market conditions. These results suggest that our model maintains robust to prototype length selection across reasonable range, guided by the temporal characteristics of each dataset.
Table 5: Sensitivity analysis of prototype lengths for multi-modal encoder on the Weather dataset.
Table 6: Sensitivity analysis of prototype lengths for multi-modal encoder on the Finance dataset.
Modality
Length
F1
AUC
Modality
Length
AUC
0.602:10.018
0.748:10.011
1
0.430:10.002
0.595=10.002
0.642.10.011
0.004810.008
0.454110.028
0.630.00.02
Time
12
0.625::0.013
0.746=20.007
Time
0.475:1:0.066 3
0.662:10.000
0.621:10.012
0.743
0.526.10.014 4
0.714:10.006
0.588:10.029 5
0.769-10018
0.619:20.014
0.743-20.008
Text 12
0.626.10.015
0.004
0.508 0.003
0.666
0.647:10.008
0.757:10.004
Text
0.496:1:0:061 2
0.673-10.067
0.627:1.0009
0.748:10.007
3
0.458.10.044
0.664:10.056
Effect of case-based explanations on LLM predictions and other quality assessments: Moreover, we assess how the number of allocated case-based explanations enhances the prediction LLM, as shown in Figure 7 We conduct experiments on the Weather and Finance datasets, demonstrating that incorporating more relevant case-based explanations consistently improves prediction performance. This trend highlights the effectiveness of explainable artifacts in providing meaningful contextual guidance. Importantly, this analysis also serves as proxy for quantitatively evaluating explanation quality, where retrieving higher-quality base- examples tends to yield greater performance gains. Beyond this quantitative assessment, alternative strategies can be employed to evaluate explanation quality more explicitly, for example, using LLM as a judge [85) to rate the relevance and helpfulness of explanations in a human-understandabhe manner, with respect to ground truth.
Effect of different base LLMs: To explore the effect of different base LLMS (Gemini-2.0 Flash and GPT-4o- mini), we provide the experiment results and analysis on the Healthcare (Test-Positive) dataset. For fair compar- isons, we used the same prompts and ran the same number of iterations and followed the same settings detailed in Appendix A.2 A.3 and A.4 The results are shown in Table 7 It can be observed that GPT-4o clearly outper- forms both GPT-4o-mini and Gemini-2.00 Flassh,,, due to better reasoning and contextual understanding capabilities
Base LLM
F1
AUC
GPT-4o-mini
0.932
0.981
Gemini-2.0-Flash
0.937
0.983
GPT-4o (default)
0.987
0.996
Table Performance of different base LLMs on Healthcare (Test-Positive)
25

---

(larger model size & pre-training corpus). Both GPT-4o-mini and Gemini-2.00 2.0-ii-- are still competi- tive compared with baselines listed in T reveals the impact of the LLM capability on the effectiveness of our framework, especially in tasks demanding real-world context understanding. We also provide the plot of iterative analysis in Figure where we can also observe the performance improvement over iterations for different base L.LMs.
Weather
Finance
0.755
0.555
0.659
0.650
0.646
0.724
722
0.724
0.415
0.72
0.625
0.627
0.600
0.70
Score
F1
AUC
0.575
0.541
0.667
0.662
0.66
0.525
3.648
0,506
0.645
0.500
DAVA
10
Number of Based Explanations
Number of Case-Based Explanations
Figure Effect of case-based explanations on L.M prediction performance.
TimeXL: F1 Score over Iterations (Healthcare TP)
TimeXL: AUC Score over Iterations
(Healthcare TP)
1.00
1.00
0.987
0.989
0.998
0,875
0.861
0.065
0.90
0.974
0.90
0.966
0.878
0.575
0.96
0.96
0.038
0.852
0.506
0.94
0.824
0.602
0.004
0.92
0.924
0.92
GPT-4o-mini
GPT-4o-mini
0.90
Gomini-2.0 D-F ash
0.90
Gemini-2.0-F ash
GPT-40
GPT-4o
0.88
0.88
Origina
Iter1
Iter2
Iter3
Original
Iter1
Iter2
Iter3
Iterations
Iterations
Figure 8: Iterative study of different base LLMs on the Healthcare (Test-Positive) dataset.
C Computation Cost for Iterative Optimization and Inference
We quantitatively analyze the token usage and runtime of the iterative process. Following the prompts and strategy detailed Appendix [P] each iteration over 4000 training and validation samples (average length of 400 tokens, around 300 words) requires about 9.74 M input tokens and yield 1.64 M output tokens. With GPT-4o (with input latency as TTFT x 0.45 seconds and output speed 156 tokens/second), it corresponds around 3.9 hours serially per iteration (prediction takes approximately 0.5 hours, reflection 0.08 hours, and refinement 3.35 hours). As prediction and refinement steps can also be parallelized, we can reduce the iteration time about 12 minutes with 20 concurrent calls. Using a more efficient LLM base model seconds and ~ 236 output tokens/second, the iteration time be further reduced to 8 minutes.
with TTFT x 0.3
In the testing stage, we apply the fixed reflections selected during validation refine the test texts. Based on our prompt lengths, the refinement and prediction steps can take down to 3.46 seconds per sample with GPT-4o, and 2.29 seconds per sample with Gemini-2.0- 2.assh,,h, which is reasonable for real-world deployment.
26

---

In addition to efficient LLM backbones and parallelized querying, we discuss further strategies to lower computational cost. As shown in Figure 1-2 iterations often yield clear performance gains. In practice, the iteration loop can be adaptively controlled by monitoring validation improvements and stopping early once gains fall below predefined threshold. To further reduce the cost, we can also do selective refinement by skipping training examples that were already predicted accurately in previous iterations. These strategies help make our method more computationally tractable.
D Explainable Multi-modal Prototypes and Case Study
D.1 Multi-modal Prototypes for All Datasets
We present the learned multi-modal prototypes (Figure TO), Healthcare (Test-Positive) (Figure [1). and Healthcare (Mortality) (Figure 12 is noticeable that the prototypes from both modalities align well with real-world ground truth scenarios, ensuring faithful explanations and enhancing LLM predictions.
datasets including Weather (Figure 9. Finance
D.2 Case-based Reasoning Example on the Finance Dataset
We provide another case-based reasoning example to demonstrate the effectiveness of TimeXL in explanatory analysis, as shown Figure B In this example, the original text is incorrectly predicted as neutral instead of a decreasing trend of raw material prices. We have few key observations based on the results. First, the refinement L.L filters the original text to emphasize economic and market conditions more indicative of a declining trend, based on the reflections from training examples. The refined text preserves discussions on port inventories and steel margins while placing more emphasis subdued demand, thin profit margins, and bearish market sentiment as key indicator of prediction. Accordingly, the case-based explanations from the original text focus more on inventory management and short-term stable patterns, while those in the refined text highlight demand contraction, production constraints, and macroeconomic uncertainty, which more consistent with decreasing trend. Furthermore, the reasoning time series provides complementary view for predicting raw material price trends. The time series explanations identify declining price movements across multiple indices. In general, the multi-moda explanations based on matched prototypes from TimeXI. demonstrate effectiveness in capturing relevant raw material market condition for both predictive and explanatory analysis.
Protatypes
Wid Sieiieen
"Suggest a likelihood dry weather with stable essuuue westtrrly shilts, which typically bring cocler, drer ai and wind direction his primarlly been from wast indicating
which could signal the approach of weather system low pressure system that could lead worsening weather
humid levels predominantly high,
speeds
moving engthennng pressure
has been
relatively
direction started westerty,
vartable,
moseebie din follmwed by gradusl ietunn higher lnveis
briel incresse, wind dincttio voss predominantly noth northesst receent hours. Air pressure e has been decressing ovarall which direction was variable without consistent partem. These incicatoons suggest
stable with slight cownward trend. Wind speeds vere mostly hours New York City, air pressure remaine relatieely stable
Figure 9: More multi-modal prototypes learned from the Weather dataset. Each row in the figure represents a time series prototype.
27

---

Mutti-modal Prototypes
Intuet
Ponert
supported by material inventories Limited spot offers
In conclusion while recent restocking activities provide some support
The seeborne raw material market is experiencing upward price movemente
Anticipated economic stimulus
China
Intensifled restocking activities ahead of the Lunar New Year have led to increased demand and price
Increasee
E.S.A
Overall the combination of weak demand
Raw material prices have fallen below $40 per dry metric ton
causing cautious purchasing behavior
The raw material market is experiencing a downturn
operations are by stricter environmental regulations
Decrease
Proteatt
Indicating balanced material
with notable volatiliy driven by various factors
Geopolitical sessonal factors
Indian material supply might increase
eased export restrictions A proposed weekly spot CFR China blast fumace sosessment aims to enhance pricing transparency
Figure 10: Key multi-modal prototypes learned from the Finance dataset. Each row in the figure represents a time series prototype.
Protatypes
growth positive specimens Risuggentsa gritiican
suggnets a warinng inluueze nesson, which could potertialy reduree
hunden heatthraa reosurces mixned flu treetment hospinal
hos boen signtficont upword frend both the number respiratory
and 68 16 respcotively. This declining siond suggcoto decressed influenzo activity, poteenially eastng hestthorre resource oloccrio on.
spociimons tested for infuente and the number of posttive casos
sesson but continued
persistent presence of inflerna The
rescning
hcolthiars system should maintain vig lance and resourtes
docino.
that
with intluanuu becoming prevstent later. musics, there
average
mostable docinee, with ovenal:
Figure 11:
Key multi-modal prototypes learned from the Healthcare (Test-Positive) dataset. Each row in the figure represents time series prototype.
28

---

Mutiimodial Protosypes
Faceed
New Biagh
Sast Nu vraar Lnghtar PONEEE
Math AMM
Bigh Mant
Drdt Non Reasee
rnsousstating close monitoring and potenbatty incresaed heatticare insourcee White infurets snems less conosming Pres mnnumoriammmiin remaine. a siggyots panaer amerging twith threse The hestfraaee system may reed preparc incrovaed comand for medioal care and healtharee intovnntion
signifcam reduction infuerze preumonia- related montllty, Indicoting effective resgonse pous by trough nation natton public
influenz
continued survllince
and poternal outhresks.
and miigon mortally
Interantoos,
the
thee intections, whith
These trones sugaest potenn roduction in hoolthear system burdono rnq a particutarly natewerthy suggasting succesatul commanment natural decline ofinduenee dares, whichooudd sllevione some precsure heolthoare
This escalation suggests glowing burdon on hoattiicre system and
Figure 12: Key multi-modal prototypes learned from the Healtheare (Mortality) dataset. Each row in
the figure represents a time series prototype.
Origonal Sect Ressoning
Poutty Decreasce
Prodiction
Reazoning
Original Teat... Recoveding glooll demand has led. to sigher Poolff mo material pricee, furmer indeaaing a
(eaurgence indusinat octidity as bust furnaces rezume operation gibally Addi ondlly port
are benng magis, onitical lictios kodiionally
earofully managed due import qustas and vessel intovs, infusnonn broader supply chain and price cynamics.
syrwm compounder prauuinggneeettinn construction
funter
aconomic stimutte
polantisl supply chain daruptuns
aggiticnttly influence future and production lavta, Asssn nes materal, brerith condiments infureninn buyent restocking innersst Some prod producers are shitting from swee-quility to to mid- quality materita due to cest natt crencies in piocossing, particuaryy among smaler faci luss... partcuiar the rest extato and construction secters continus to be key influc reers industrial domand and, consqquently ram material pricee, Anditionlly woonomic semulux mesuumes medt bozating induamee cound to trowseed demand material, Howovor goopol toco tono ons coonomic policies. panicutory y these roloted to mode, could create voludilty many
cortinue dectins persattent
Foreed
pood modionemargns,
shappng
tmgaccoory
suses while momtte constra continue due decays neactvuting production facitities On the togatical side the freight rates climping to incroased domand 10.2)
Frass
Neuntll: : peceeniallyy depiessing new manwrne pricest agan Thus desplts short torr struugh in now matwink prices driwen actire production and di sruptions in sccondary mitor ol supolyy j6, 39)
Deczwanec fao materall nel prices have been dectining dua wark marber, leading to dem peoee brying interset (8 (0)30
Neutiat: Producers are capitaliiig on high stack lenea, buying wes now matarats primanty using chasper stocks (3.27)
TimoKL Test Reaacring
Prediction Decroase
Betined ... Desprie this glseal demand recovery has boatered Asw--aache new materas al prices, sugraing a activit as
mes-rgence
gobally openatios. inventories aftecting supply dymamios
managrmmrt import quotas
gride
mil-gaate matwits command high premeume, thin producer mo giss. doteningg buyero from facd piloz optiens. Production cornicts have furter redused demand. corolbbdingto to boarish market condmmnnt Scasonal feonors, such as the sporoaching winter heating sceson, are expected to lighten production comnecs, indineclly impacting caw matenal demand dynamics. Poarmohiie geopotitiaal beratons and economic policies, especisly tooe reltte to made could Inmoduce voladinn inoo now material pricee, Econamic etmuuss menoume almed at boweting ind unnia artidly could lead incrassdd demand. Downi, producer margins, poil Inventories, and production levelo will or Book decermning the luturo srjectory of the sae materlal
heaw decrorred dia subcued demand and
Prototypes:
Decrussnc char materia marbet is reperiecingg deomium [0.43]
Decrusssc Charaal the combination weak damandd (0.22)
Decowane: rig coudious wooiiin
(0.48)
posentully copping
intuaeninggov
(0.54)
Figure 13: Multi-modal case-based reasoning example on the Finance dataset.
E Designed Prompts for Experiments
In this section, we provide our prompts for prediction LLM in Figure 14 (and a text-only variant for comparison, in Figure 15). reflection LLM in Figures T6 17 18 as well as refinement LLM in Figure 19. Note that we adopt generate-updat--summarri reflective thoughts from training samples with class imbalances, which is more structured and scalable. We make the whole training texts into batches. First, the reflection L.LM generates the initial reflection (Figure 16 by extracting key insights from class-specific summaries, highlighting text
strategy to effectively capture the
29

---

patterns that contribute to correct and incorrect predictions. Next, it updates the reflection (Figure) 17 by incorporating new training data, ensuring incremental and context-awand refinements. Finally. summarizes multiple reflections from each class (in Figure T8) into comprehensive guideline for downstream text refinement. This strategy consolidates knowledge from correct predictions while learning from incorrect ones, akin to the training process of deep models.
System Prompt
Your job is to act as [specific role]. You will be given summary of [data description] and related prototypes that you can refer on information, your to predict (task description).
User Prompt
Your task [task description). First, review the following [number of prototypes] prototype text segments and outcomes, so that can refer to when making predictions.
Prototype #1: [text protolype]
Corresponding Segment#11: [input text segment]
Relevance Score: [similarity score]
Outcome #1: [options]
Next, review the [situation]:
Summary: [text input]
Based on your understending, predict the outcome of [situation]. Respond your prediction with [options]. Response should inciuceude other terms.
Figure 14: Prompt for prediction LLM
System Prompt
Your job is [specific role]. You will be given summary of (data description). Based this information, your task is to predict (task description).
User Prompt
Your task [task description] First, review the [situation]:
Summary: [text input]
Based on your understanding, predict the outcome of [situation]. Respond prediction with [options]. Response should not include other terms.
Figure 15: Prompt for prediction with text only
System Prompt
You are an advanced reasoning agent thet cen improve the quality [domain] summary based on self reflection, You will be the summaries and [correct flag] predictions of [situation]. Your task to learn some reflections that guides the refinement of [domain] summaries.
Prompt
Your task is anelyze the provided [domain] summaries with [correct tlag] predictions, order reflection report improving its quality for [situation] prediction.
Review the following [number summaries] [domain] summaries with [ground truth] actual
[prediction] predictions.
Summary #1: [text input]
Actual Outcome #1: [ground truth) Prediction #1: [prediction]
Based an your analysis, write a high-quality reflection report that summarizes key phrases or predictions [situation] commonly misinterpreted and overlooked phrases sentences thatled led to incorrect predictions (situation).
precise terms to convey clear professional analysis, avoid overly general statements. The report should be comprehensive and infermative paragraph, which generalized refine similar [domain) summaries. Your response should not include other terme.
Figure 16: Prompt for reflection LLM: reflection generation
30

---

System Prompt
You are an advanced reasoning agent that can improve the quelity of [domain] summary based on refiection. You will receive reflection report up to this point. will also be given the summaries and [correct fing] predictions of [situation]. Your task to learn some reflections and update the current report that guides the refinerment of [domain] summaries.
User Prompt
Your analyze provided [domain] summaries with [correct flag] predictions, order update refiection report improving its quality for [situation] prediction.
First, following reflection report up point: [current reflection report]
Next, review the fallowing [number of summaries] [domain] summaries with [ground truth] actual [prediction] predictions.
Summary #1: [tex input]
Outcome (ground truth]
Prediction #1: [prediction]
Based on your analysis, write high-quality reflection report that summarizes key phrases sentences that lad correct predictions [situstion] commonly misinterpreted and overlooked phrases sentences that led incoreent predictions [situation].
precise terms to convey clear and professional analysis, and avoid overly generel statements. The should contain incremental and context- updates, and be generalized refine similar [domain] summaries. Your response should not include other terms.
Figure 17: Prompt for reflection LLM: reflection update
System Prompt
You are advenced summerizztion agent that can generate high-quality summarization. You will given previously generated reflections for refinement, from and incorrect predictions of [domain] texts. Your current task is to summarize these long reflections to better guide financial text refinement.
User Prompt
Your task is contents. The goal is to generate high-quallty report aimed at improving the [domain] text quality for better predictive accuracy.
reflections derived from previous predictions of [domain]
First, review the reflections from all combinations of possible predictions and actual outcomes: [reflection reports]
Based your analysis, summarize the reflections of different scenarios and write comprehensive report that provides guidelines select the most important content new [domnai] texts where the actual
unknown. Your responee should the enough details, yet effective, improve the text quality for downstream prediction. Your response should not include other terms.
Figure 18: Prompt for reflection LLM: reflection summarization
System Prompt
You are en advanced refinerent agent designed to enhance the quelity of [domsin] summary. You will be provided with reflective thoughts analyzed from other summaries, and summary that requires refinement. Your task to generate refined [domain] summary, by examining how reflective thoughts applied the current summary.
User Prompt
Your task generate refined weather summary predictions of [situation]. First, review the following reflections that provide guidelines for refinement:
summary
[final reflection report]
Next, review the current [domain] summary that describes [situation]:
Summary #1: [text input]
your understanding, generate new summary by selecting content the current summary, which provides insights fou understanding [situstion]. Response should not include other terms.
Figure Prompt for refinement LLM
31

---

F Reflection Reports for Text Refinement
In this section, we provide the reflection reports after the first iteration, to demonstrate the reflective thoughts by accessing real-world contexts.
Reflection Summary Weather (Now York)
Gomprehensive Report on Refining Weather Prodictoons
The tonew ol weather summar es scuse vahous prodiciv acorurcs rovedaa criicall ro gita that cen ethanne the pccuricy and roluballly of weath er foresisis By inkog utiing gueea improsiig
Indicatore Bry Conditions ("Het Ratsed Predictions Outcameej:
1. Stuble and Sighhty tncressing Air Pressure: Coraskont reedings with rinor upeand nds suggest high pressuss syskems, and oulive day weather. Spoeess Varaall Wind Directions Whotthwertery Westerty Stntee hro esutnaaewyy noethowbberggwwweeee diwectoe oooler, other sir, reduriing rain loo ceilhoodd
Moderatte
speeds nist gista support sreap fal foresaus.
Decraainng Daylime Humidity: ligh exty humatty helnwad dayimee decwosee consatue with cry conatiions
Typioal Dumal Temperation autre Patterrs Norma comperatare vartutims batter supprr cry farcorat,
pabeen chargeo
subbl, conditors.
2indictom for Rainy Conditions (Raiiee" Prodictions and Guncomeej
incresiing itamddty Leoale dipeticatt humaltyy incersss aee stoong min indicaton.
signals
incominn prossare syskers.
Wind Conditons.
Sight Wind Speed incresse: Othen precedis oen part cubety cooweeed inter
3 3.Oonmoon prvinieranitation (Raineed Predictiona with "Nol Ramned Ostcoreca::
Prossure Staliity Wissterrrelaton: Niw prossum fuduuatoons wibourchie ind cators a woak rain prodetora. Fag
log
auggeesing
Overemphask Wind Changes: Foras
wind patterss otber miner variaions
Majuudged Terpenduur Plactuadions: Evalatto 301 perpertus changla do goide comdbosiinn ino ind calers like presouro drapos Dyneric Weather System Wainterretttinn Recogrics soble urrosstiioe
lote undid roin predictions
4.verlooia Conditioe| Natt
with
Ouscomenj:
Hlamistty Peaks and Varabilly high paoka Buctuttoons oten precece nann they shouddn be understimate.. scalhert coolely
Wird Drection
signal mokauro flx.
Conbroed
ind
Stable Pressure with Other Vartables: Consddrr presaure entilly by with high suriiily and wint changes collccioely
tocuiing on hass dendle rediatore and milnnng the inteppotation on weather wements fstire weather mode can improve prediction qpanity providing mow sozumds
Figure 20: Reflection summary for text refinement on the Weather dataset.
Reflection Summary- Finance (Row Material;
Corpratonssive Guidntines for Eshancing Financial Toxt Quality Predictive Anslysic
the
prodctins
elemanta winne the
eriphssi content Snancial suminares. on reflections foom paet preditions.
guiaaimes tighaghl the
opaniinng taol
Kay Indicators for Price Trends:
major coneu mors saring aoft nargits key monurlecuting sectors and proferences noterias Also considr suth chails inventorie, For decresses, err prrptente raing inventory levela, week demand, and sperational aqj aduatments Deamish mareet sentmment and gecpotiical tara lone anould be noled well
prod by incresses for
seutral
idently market Rtrl indicaios like talanoed demand
stake production
moderate boyer behawiorr
2. 2.Cortontall Iifueeoss:
Gate der the inpect senscred brene,, rastocking act au, and ecan stimal demand stadustion. Racoggits how these factora may cause temporary
2. Andyyn glonal inde supely shain fits prione to assess thair porentaal to couse shoreerm in wo fry or soat by nather then sustaine changes.
3 Rogguaatory and Pakey Factors:
geopoliical par supply
appelicanty yy
shout carefully intggated into analyass.
Markot fardinent and Rolated Markets
recogriinng nn pos skg merrs
underty ing deerand interooneeced
relabed nav rusaritt
for ther intuance demand
3. 3.Stratogic and Operational Adjustments:
stretion procucers buyers sejust odions procumeeent storegles recocnee maroor cond adjurments demand
rsiiiity
&Avolding Common Mainterpretabirn
Avoid coermmphentnng shori-am zem as indicaton of austained market shitss Be wary of speculative infueroos and temponary pallyy ancouncement that may
2. Renogsire righ ineiitey went ne octenta atring: bu bulless and oone dow that mai in soel briigggg prios during of pock periods
intaguniigg these gudde mes, francia summerss provide more compnchenaive and ba sneed ynts entencing cnedicive sccurscy uninown market dynanics
locua
underslancing artizisting
Figure 21: Reflection summary for text refinement on the Finance dataset.
Reflection Summary- - Healthoare (Test-Positvee
Comprehensive Report on improving Healhhars Text Quality for Predictivs Accuracy
Inteduction
This yertwanees mntcciin paet aealywos of hrad
conswiing infowers pealiiity creec The gool enhonos quaiity teate
morov pooi cha storrey by krum seowomeees when ackue a roomen are unkrown. Key indlcatoro Accurata Predictions
Deekiiing Trends
Predectoos
Emphasix on Decine Comeed prodicions for ou sooomes that del not sonnedt the swwrage oten hight ihhte a dear dedine the poetivity crler. Terma such
"octaale decine "seendy decresss Regratzari reaudd are masee
paricciaryy inlunna
Testing Volume and Ration dicrusse in boking uno, coopled with post vity raloa romaninn below the averago, strnnghaoaa prodicionn.
Low Portbvily Parcentagae: Expliot wwwwoos recert peatictyy mise consuderty tiiking belnw the long wer aoerage sen bi.
Murked Incrossca Suburartia incresseds in pos tuly rakos, High fightedd apuailic numericll competsors, and oate an "ceoood" oupoome. Ieflunnca Strin Doninance Dosulo on the prodonenatio of innueraa or with thik imsuut on onoull pio by sakos are sigee foar. speciment Haatthaar System Rehwwooe inchossed traphmikertion and meciral earvice derand proviee contect vuistirs easeeding predictoons
tulty
suggese
5. Tempordl Puturns. Trasking workly pools da undo olur ding and prod caing hoipt nighteeccc in fllenna sctliiy.
-Datdtvve Abselute Increanee Mid
oten aree contating inowsses excweding the poativity te soneed
irchessse eppicity wit etorica avenagee
-Comottiall Misaggement Pieases fegit fing rloes poaiiiiity sako without contontual of aignneen overage tresholic can rislood prediciinns.
ontegrate Claar Comparteone Creas aummrrrie inceppnnte drsct compun toone to is etviccl awaragee to sonid miguaemente
sternen Trand Clartty 3ituretiate deantyy betoown inchessin frence and these that actuadl nurpaes the aue aga poetivityyo one
Proviss
+Hihlight Tempordd Duta incoponnt lompondl patoono and works posko lo in ruanood prodicivo undlyds. stharin in thece gultelreee heatt sedee can relieed to athanca pre aezurnyy. altrsteey aading in pronciive heatthaae planinng and neeo soe adiccation.
Figure 22: Reflection summary for text refinement on the Healthcare (Test-Positive) dataset.
32

---

Rellection Summmary- Healthoare (Morialll)
Reflcction Summary Report Heafthcare Teat Refieement:
S0omect Esseed' Prediction:
such
parliclllty foo Infurraa Preamoma.
Terms Theloar acarae und"
benchmani effcchvely dignal sutcamee the auppoee suerage montatyyy ortoe hight get
management
1guncc.
Ackroon wdyr g. "orriston bunden Purmicoia unida des iroo providio en mucrccd undertarn drng staulity in disese mooot.
Arzurtte forccaal marked shrsene auch hooncaming "meed "signfrcast incresee and testabls inckcrtnn momalit ritioe.
Quer Miikin oup vosiors to live "iiamats rio" in doaths and oo sono escoeding rig specril anoragis offocdivty onty contontaakio light monullly beerds
posts
outbresks Outcomast
urgancy resourse koations eritoriing prodictiv actunaty.
3inssmectt Deceed Predictions with Net
Mectappeetttrr arise from serohwerne upoordt rrend wfthout al gring them with the natox belcw the 7.84% thrsehold.
Pronangnaaa bodlted posto constziing tlicr
raturs
relatvve
barchyracce
AJncorrect Exceed' Predictions "Dooeed Outcomes
incomee suidiioo from nog jooting conductvv offors of infueria and oni. isuccially when ona disane hadone other..
acausey antunned phosting moart pasks sussonsl patterms, and camp ienaattive diessse impacte..
<Phor15zz derdfoottoo boncs ratios, crouiing they comostaaiioo againg Nitorroal evergees treshhrdss
-Jassiguash betweon phot torn luctuaionns and susta tonds, enphuszinng comuulttvv mocets over do lited poinss
management potentta
<Conaiber sessoral patterss
rnttrrca dats
shelts
prevllence.
-Ercure quenttal oo spar-sone sono are explot and suppornbl comprehor to dota ar elgli to ovold mide orrposatoon spuctuee approach
cutling accuntto
sureririon, enhanding
acourscy improwing quality heothhore anelyses.
Figure 23: Reflection summary for text refinement the Healthcare (Mortality) dataset.
G TimeXL for Regression-based Prediction: A Finance Demonstration
In this section, provide demonstration of using TimeXL for regression tasks. Two minor modifications adapt TimeXL for continuous value prediction. First, we add regression branch in the encoder design, as shown in Figure 24 On the top of time series prototype layers, we reversely ensemble each time series segment representation as a weighted sum of time series prototypes, and add regression head fully connected layer with non-negative weights) for prediction. Accordingly, we add another regression loss term (i,e., MSE and MAE) to the learning objectives. Second, we adjust the prompt for prediction LLM by adding time series inputs and requesting continuous forecast values, as shown in Figure 25 As such, TimeXL is equipped with regression capability.
Training time series
Time series prototypes
Projection
P1
P2
Weighted
K(pL.) :)
S
S1
Non-negative FC
Time Series Encoder
Forecast
Kípz.
S²
series X
K(pr. +)
S
Text input
Cross-modal Relevance Fusion
Language Model
Text Sequence Encoder
Kipt.> -)
S's
223
K(ps.> -)
Trend
Text embedding
K(PL. ))
k
Projection
P
Training & Testing Testing
Training texts
Text prototypes
Figure 24: Multi-modal pototype-based encoder design in TimeXI. for regression tasks.
33

---

System Prompt
Your job is to act as (specific role]. You will be given a summary of [data description] and related prototypes you can refer to. Based on this information, your task is to predict [task description).
User Prompt
Your task is [task description]. First, review the following [number of prototypes] prototype text segments and outcomes, so that you refer to when making predictions.
Prototype #1: [text prototype]
Corresponding Segment#1: [input segment]
Relevance Score: [similarity score]
Outcome #1: [options]
Next, review the [situation]: Summary: [text input]
Finally, review the [domain] record [situation] [time series values]
Based on your understanding, predict the outcome [situation], followed by value [domain] record. Respond your prediction [options] and [numerical value]. Response should not include other terms.
Figure 25: Prompt for prediction LLM in regression tasks.
Table 8: Performance evaluation for regression tasks.
Model
RMSE
MAE
MAPE(
DLinear 75
7.871
6.400
4.727
Autoformer
7.215
5.680
4.263
Crossformer
7.205
5.313
3.808
TimesNet 778
6.978
4.928
3,512
iTransformer 79
5,877
4.023
2.863
TSMixer [80]
7.447
5.509
3.911
TimeMixer 81
6.651
4.703
3.349
FreTS [22
7.098
4.886
3,460
PatchTST 5
5.676
4.042
2.853
LLMTime [33
11.545
5.300
3.774
PromptCast 0%)
4.728
3.227
2.306
OFA [53]
6.906
4.862
3,463
Time-LLM [56
6.396
4.534
3.238
TimeCMA [57
7.187
5.083
3,620
FSCA (58)
5,511
3.873
2.720
MM--Transformer 35
5.454
3.789
2.687
MM-PatchTST 35
5.117
3,493
2.491
TimeCAP 65
4.456
3.088
2.196
TimeXL
4.161
2.844
2.035
We conduct the experiments the same Finance dataset, where the target value is the material stock price. The performance of all baselines and TimeXI. is presented in Table Consistent with the classification setting, TimeXL achieves the best results, and the multi-modal variants of state- of-the-art baselines (MM-iTransformer and (IM-PatchTST) benefit from incorporating additional text data. This observation is further supported by the ablation study in Table 9 We also note that the text modality offers complementary information for LLMs, enhancing numerical price predictions. Furthermore, the identified prototypes provide contextual guidance, leading additional performance gains. Overall, TimeXL outperforms all variants, underscoring the effectiveness of mutually augmented prediction. We also provide an iteration analysis to show effectiveness of reflective and refinement process, as shown in Table 10, The prediction performance quickly improves and stabilizes over iterations, which underscores the alternation steps between predictions and reflective refinements.
34

---

Originai
TireXi:
Original The bey lectors infuuncing the futurn price of material include new production restrictions, which have siighty improved market contmentss desptt a generally weak morket and bearish ouilook. Production lovel adjastments paricularly in highly industiial regins, tand to have sigrificant impocts the demand and pricing of row material. Furthermore the nenowing spread betleeer the hightt- and sedeee meteral indiow indicates potertiel shift markel preferencss towards dittrentt quality gades
in matertals, directly spreting pricing influence the supply chain movemeets, prices are decine as buyers show weak interest favor discounted products, signoling suppresstvv outtokk on demand u. Furthermore geopolitical tensions and regulatory changes continue to riject volatiloy, inflwencing the trado flows and pricing of raw materials globaly. Supply chain mansghra ploralyy menitor thase dynamical especiallyy with gadusl incresoes in rsport cxpabslitio from major procucers, which could further infusence global cupply...
Additooally
aeopoliccull
policies could
pedumtio
adding complecty precicting
Packad
iurthe-infuence
Prototypea:
Neutral-atthough the overall market contmeent remoins coutious (0.39)
Neutrel: All these factora combined sighal patential fluctuationa material prioss moving forward (0.40]
Decrease: eiement
oynamics
Timoxi: Text Reasoning
Truth Decreass, 139.10 Prodiction: Decreaso, 136 84
Retined Text: Despito expectations for incraseed demand durng the September-Octete pesk
seasen, tight conditinna particu partiuulrrty for highar-quility materials have resulted in price incresses
for athrr nation domestic reanuross. Genpoittiall twnalons and acmpoii policies could further dynamica, Production restrictiona sapecially in between higher- mid-grade materall indices. Indicate towards diffarent quality godes of materiats, Bustained produccion curbs antelpoteed to esert dnserwadd pressume new materall demand. declive prices further compourded by incoassed port iwentories in China, higher shipments from Brazil and roduced Chinese incustriaal output due to stricter environmenaal regulations. The downttu in the Chinase conaat conatruction mar hat and slugfah real estate sector are adotiional factors contributing weakened damand. Consocuently the cutlook for raw material prices remains bearish
regions
spread
terrs Inf ailluenedd
supply levels and economic shilts.
Prototypes:
Decrease:: This rediuet inductionn in production has reutted decrassed demand for raw materiala (8.56)
Decrease: Chinese
market signtiiantt voriable trend
facing
TimaXXL: Cormilttion in Analypia betwaen Time Series Imput and Protosypsst
Podeed
Teare??
Figure 26: Multi-modal case- aase-based reasoning example on the Finance dataset for regression tasks.
Table 9: Ablation studies of XI. for regression tasks.
Variants
RMSE
MAE
MAPE( %)
Time Series Prototype-based Encoder Multi-modal Prototype-bas based Encoder
4.287
3.001
2.140
4.198
2,891
2.064
Prediction LLM Time Series + Text
4.600
3.121
2.226
Prediction LLM Time Series + Text + Prototype
4,352
3.003
2.165
TimeXL
4.161
2.844
2.035
Table 10: Iterative analysis of TimeXL for regression tasks.
Iteration
RMSE
MAE
MAPE(%)
Original
4.344
2.951
2.103
1
4.224
2.883
2.069
2
4.161
2.844
2,035
3
4.174
2.849
2.036
Finally, we provide case study on the test set of the Finance dataset that demonstrates the effec- tiveness of TimeXL's reasoning process for regression tasks, as shown in Figure 26 We present
35

---

the top three most similar prototypes allocated for both time series and text modalities, where we have two key observations. 1) Even if the trend predictions align correctly in both original and refined texts, the prototype allocation varies regarding market sentiments. In the original texts, prototypes from the neutral class are also allocated, highlighting a mixed sentiment and potential fluctuations, consequently deteriorating numerical prediction accuracy from both the encoder and the LLM. In the refined texts, prototypes instead capture the underlying bearish market sentiment for raw materials, emphasizing decreased demand and weakened market conditions, effectively aligning bearish market sentiment with numerical predictions. 2) The allocated top similar time series prototypes also exhibit highly similar trends compared to the original input series, offering robust reference points for numerical prediction. To quantitatively assess it, we aggregate the retrieved prototypes using their similarity scores and compute the correlation between the input time series and the aggregated prototype series. The observed strong correlations underscores TimeXI's efficacy in capturing intrinsic temporal patterns and providing predictive insights into future numerical stock values based on similar historical patterns.
H More Detailed Discussions of Related Work
Due to space limitations, we provide additional details on related methods here, expanding on the overview in the main text.
H.1 Multi-modal Time Series Analysis.
In recent years, multi-modal time series analysis has gained significant traction in diverse domains such as finance, healthcare and environmental sciences [28] II [3]. Multiple approaches have been proposed model interactions across different modalities, such as multi-modal fusion and cross- modal alignment and so on. Multi-modal fusion captures complementary information to enhance time series modeling, which are commonly implemented via addition (35) and concatenation [29 at the multiple levels. For example, Liu el al. (35) fuse predictions from both time series and text modalities via learnable linear weighted addition, further enhancing the predictive performance of state-of-the-art forecasters. In parallel, cross-modal alignment aims to capture the association between time series and other modalities for downstream tasks, where common techniques include attention mechanism [12] 30 86 31), gating functions [32), and contrastive learning [33] 34). For instance, Bamford et al. 14 align time series and text in shared latent space using deep encoders, enabling retrieval of specific time series sequences based on textual queries. In addition, Zheng er al. (33 perform causal structure learning multi-modal time series by separating modality-invarian and modality -ppecfic components through contrastive learning, enabling root cause analysis. Besides method developments, recent research efforts have also advanced the field through comprehensive benchmarks and studies [35] 36 37 highlighting the incorporation of new modalities to boost task performance. Meanwhile, others explore representing time series as other modalities (e.g., image, text, table, graph) [38 39
42) generating synthetic modalities [32 43 to facilitate downstream tasks. Nevertheless, these studies tend to focus primarily on improving numerical accuracy. The deeper and tractable rationale behind why or how the textual or other contextual signals influence time series outcomes remains underexplored.
40
41
H.2 Time Series Explanation
Recent studies have explored diverse paradigms for time series interpretability. Gradient-based and perturbation-base explanations leverage saliency methods to highlight important features at different time steps [44] 45). where some methods explicitly incorporate temporal structures into models
and objectives 46 47 Meanwhile, surrogate approaches also offer global or local explanations
by evaluating the importance of time series features with respect to prediction. These methods include the the generalization of Shapley value to time series [48), the formulation of model behavior consistency via self sulf-super learning [4),, and the usage of information -thoorettcc guided objectives for coherent explanations [50]. In contrast to saliency or surrogate- based explanations, we adopt a case-based reasoning paradigm (25 26 27 which end-to-end generates predictions and built-in explanations from learned prototypes. Our work extends this approach to multi-modal time series by producing human-readable reasoning artifacts for both the temporal and contextual modalities.
36

---

H.3 LLMs for Time Series Analysis
The rapid development of LLMs [3 16 has begun to inspire new directions in time series re- search [51] Many existing methods fine-tune pre-trained LLMs on time series tasks and achieving -fate-of-the-ar results in forecasting, classification, and beyond [53] 54 [55]. The textual data (e.g.. domain instructions, metadata, and dataset summaries) are often encoded prefix embeddings to enrich time series representations [56
These techniques also contribute to the emergence An alternative line of research leverages the zero-shot few-shot capabilities LLMS by directly prompting pre-trained language models with text-converted time series [63] context-laden prompts representing domain knowledge [24], often yielding surprisingly strong performance real-world scenarios. Furthermore, LLMs can act as knowledge inference modules, synthesizing high-level patterns or explanations that augment standard time series pipelines [64] 63 [66]. For instance, Wang et al. [66] construct a large news database for LLM with dynamically selected news, which uses reflective selection logic to incorporate matched social events and thus aug- ment numerical prediction. Building this trend, recent works have explored time series reasoning with LLMs by framing tasks as natural-language problems [68], including time series understanding & question answering (e.g., trend and seasonal pattern recognition, similarity comparison, causality analysis, etc.) [67 69 70 7 [72]. as well as language-guided inference for standard zero-shot time series tasks [67 69. Compared existing LLM-based methods, our approach provides unique synergy between an explainable model and LLMs through iterative, grounded, and reflective interaction. It explicitly provides case-based explanations from both modalities, highlighting key time series patterns and refined textual segments that contribute to the prediction.
57
58
59
of time series foundation models 55
60,
61
62
method development and perform text-to-text prediction by fine-tuming
I Limitations
Our main focus has been on classification-base time series prediction to emphasize the explanation capabilities in real-world multi-modal applications, while we also provide a detailed regression- prediction setting the appendix. Extending our framework broader regression- the multi-mo modal contexts, such as long-term time series forecasting, and time series forecasting with domain shifts, remains an important direction for us to explore in the future work.
tasks under
J Broader Impacts
Our paper presents advancements in explainable multi-modal time series prediction by integrating time series encoders with large language model-based agents. The broader impact this work multifaceted. It has the potential support high-stakes decision-making in domains such as finance and healthcare by delivering more accurate predictions accompanied by reliable case-based explanations that lead more robust analyses. The social impact of this work stems from its potential to provide a new paradigm for analyzing real-world multi-modal time series through the integration of emerging AI tools such language agents. We emphasize the importance responsible and ethical deployment of LLMs in time series analysis to ensure that such advancements lead to positive societal outcomes.
37


Reviews:
[{"review_id": "MJOwbIxQRk", "reviewer": "Reviewer_83We", "summary": "The authors built a closed-loop prediction framework, consisting of a time-series encoder to provide initial predictions and explanations, a prediction LLM to update predictions through reasoning, and a reflection LLM to compare the results with the ground truth. This workflow is multimodal, allowing for time-series and textual inputs, and provides interpretable explanations. This setup was tested on 4 real-world datasets and showed 8.9% AUC improvement.", "questions": "* How did the authors choose to use GPT 4o, Gemini-2.0-Flash and GPT-4o-mini over other LLMs? What was the performance across other models?\n* Could the authors explain why a reflection LLM is needed? Especially compared to just using a metric to evaluate the differences?\n* What’s the computational overload for this closed loop? When do you stop looping? How do you determine this stop?\n* Discussion on broader impact and limitations should be put in the main text\n* Adding a qualitative evaluation of faithfulness from domain experts would support the downstream utility of this tool", "limitations": "* There are no limitations noted in the paper nor future directions.\n* Novelty regarding methodology is low since the main contribution is combining and prompting GPT/Gemini models\n* The authors mention their contribution as “deliver both accurate and interpretable multi-modal forecasting” but yet they do not show experiments looking at forecasting in the main text (rather focused on classification tasks)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* Paper is overall clear and easy to read\n* Addresses gap in interpretability of multimodal LLMs, which is an important and relevant challenge\n* The iterative closed-loop workflow allows for continued performance enhancement\n* Provides human-interpretable explanations from multiple modalities\n* Multiple ablation studies, real-world experiments and baselines helps justify the utility and generalizability\n\nWeakness:\n* Computational overload for the closed loop may be an issue depending on stop function\n* No discussion of results, goes straight from results to short conclusion \n* No qualitative evaluation of faithfulness/utility", "was_revised": true, "final_justification": "Thank you to the authors for addressing my comments. I will keep my score, subject to the revisions being added to the final paper.", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback. Below we provide our responses to the concerns and questions.\n\n**W1 and Q3: Computation cost for closed loop optimization** \n\nWe sincerely thank the reviewer for raising this practical concern and provide our responses as follows. \nWe first quantitatively discuss the computation cost of LLMs in the closed loop, considering a dataset with 5,000 samples (4,000 for training and validation), and average length of 400 tokens (~300 words based on OpenAI tokenizer). Following the prompts and strategy detailed in Appendix D (Pages 22-24), we calculate the token usage for each LLM per call and per iteration (configured with 10 × 20‑token prototypes for Prediction LLM and a batch size of 50 for Reflection LLM).  \n\n(1) Tokens per call for each LLM: the Prediction LLM uses ~950 input tokens per sample (output negligible); the Reflection LLM uses ~20,730 input tokens per batch and ~500 output tokens; and the Refinement LLM uses ~970 input tokens and fewer than 400 output tokens.\n\n(2) Token cost per iteration (4,000 training and validation samples): Prediction takes ~3.80 M input tokens (output negligible), Reflection ~1.66 M input and ~0.04 M output tokens, and Refinement ~4.28 M input and ~1.60 M output tokens. Total: 9.74 M input tokens and 1.64 M output tokens per iteration.\n\nWe now calculate the time cost per iteration. Using GPT‑4o (Input latency as TTFT ~ 0.45 s per call; Output speed ~156 tokens/sec), prediction takes 0.50 h, reflection 0.08 h, and refinement 3.35 h, yielding ~3.9 h serially in total. Note that the prediction and refinement steps can be parallelized; for example, 20 concurrent calls reduce GPT‑4o’s iteration time to ~12 min. Using Gemini-2.0-Flash (TTFT ~ 0.30 s, ~236 output  tokens/sec) further reduces it to ~8 min.\n\nWe also provide the test-time latency as follows. Note that we apply the fixed reflection selected during validation to refine the testing texts (as noted in lines 245-250), reducing inference cost and latency. For GPT‑4o, the refinement and prediction steps take ~3.46 s per sample, and for Gemini-2.0-Flash, ~2.29 s per sample. In general, the computation cost per iteration and testing latency are reasonable.\n\nNext, we elaborate the iteration strategy. As shown in Figure 5 and discussed in Section 4.5, 1-2 iterations are sufficient to see clear performance improvements. In our study, we set a fixed number of iterations to explore the further potential of improvement from iterative reflective reasoning on texts. In real-world applications, the number of iterations can be adaptively controlled by balancing performance gains against computational budget. For example, one may monitor the performance improvement (or its average) after each iteration on validation data, and terminate the loop early if it is below a predefined threshold. To further reduce the cost, we can also do selective refinement by skipping training examples that were already predicted accurately in previous iterations. These strategies help mitigate the issue of computational cost, making our method more tractable.  We will provide a detailed discussion in the appendix.\n\n**W2: Lack of results discussions** \n\nWe would like to kindly clarify that we provided discussions for each of the results presented, including key observations and how they validate our proposed components.\n\nIn Section 4.3, we discussed the real-world implications of multi-modal prototypes across both rain and not rain classes on weather datasets, showing these prototypes capture key contexts for the following case-based explanations. In Section 4.4, we further provided a case study and elaborated how the rendered multi-modal prototypes highlight contextual information linked to the upcoming weather event through the refinement process. More results are discussed in Appendix C (Pages 20-22).\n\nTo further validate the workflow, we provided and discussed iterative analysis in Section 4.5 and ablation study in Section 4.6 (More in Appendix B, Pages 18-19). We specifically discussed the performance improvement behavior through iterative reflective refinement, as well as the benefit of integrating time series and textual modalities, the effectiveness of explainable artifacts regarding providing relevant contextual guidance, and the advantage of augmented prediction between multi-modal encoder and LLM. \n\n**W3 and Q5: Qualitative evaluation of faithfulness from domain expert**\n\nWe appreciate the reviewer’s helpful suggestion and kindly provide our justifications as follows. Firstly, our current framework is primarily task-performance driven, where explanations serve to facilitate accurate predictions. Moreover, we focus on general-domain datasets including weather, finance and influenza surveillance (domain and task descriptions are provided in lines 253-266, Section 4.1), where the contexts are relatively accessible to AI practitioners. The explanations including multi-modal prototypes and case studies (provided in Figures 3,4, 9-13) are relevant and generally intuitive without requiring deep domain expertise. Nevertheless, we acknowledge the value of involving domain experts to ensure the clarity and usefulness of explanations, and we agree that qualitative evaluation from domain experts would further support the downstream utility. We will consider domain expert evaluation in specialized applications like clinical narratives for future work. \n\n**Q1: Choices of LLMs**\n\nWe appreciate the reviewer’s interest in our choice of LLMs. Our framework is designed to be model-agnostic and can accommodate LLMs with sufficient context reasoning capabilities. As such, we selected GPT‑4o, GPT‑4o-mini, and Gemini-2.0-Flash based on their strong reasoning and inference  performance on diverse benchmarks, their diversity in model scale and architecture, and their accessibility at the time of our experiments. \n\nSpecifically, GPT-4o was selected for its strong general-purpose reasoning. GPT-4o-mini and Gemini-2.0-Flash were included as more cost-efficient variants to assess the framework’s performance under limited computation and model capacity, within and beyond OpenAI. We provided performance evaluation and iterative study, with corresponding discussions in Appendix B (Pages 18-19). The results show how LLM capability influences TimeXL in understanding real-world context, and validate its consistent effectiveness across different LLM base models.\n\n**Q2: The necessity of reflection LLM**\n\nWe appreciate the reviewer’s question. The key reason for using a reflection LLM rather than relying solely on scalar evaluation metrics is to enable generalization to the testing phase, where ground truth labels are unavailable to guide textual refinement.\n\nDuring training, the reflection LLM compares predicted values against ground truth labels to identify issues such as textual noise or misalignment. This reflection process, detailed in our prompting strategy (lines 792–800) and template designs (Figures 16-18, Appendix D, Pages 22-24), generates targeted feedback that is used to refine the input text.\n\nAt test time, since ground truth labels are not available, metrics cannot provide usable refinement signals. Instead, we apply the reflection strategy selected based on validation performance (Section 3.3.2 and Algorithm 1) to refine test samples. This mimics how a trained model generalizes to unseen data and enables label-free inference-time improvements.\n\nMoreover, compared to scalar metrics (e.g., accuracy, distance), the reflection LLM provides richer, context-aware feedback by reasoning over the input text, model prediction, and training outcomes. This leads to more meaningful and interpretable refinements. Examples of such reflection reports guiding refinement are shown in Figures 20-23 (Appendix E, Pages 25-26).\n\n**Q4 and L1: Broader impact, limitation and future directions**\n\nWe thank the reviewer for this suggestion. We acknowledge the importance of clearly articulating the broader impact and limitations. While these discussions are currently placed in Appendix H (limitations and future work, Page 30) and Appendix I (broader impact, Page 31), we agree that it could be better to incorporate them into the main text of the paper. We will appropriately integrate these sections in the updated version.\n\n**L2 Method novelty**\n\nWe appreciate the reviewer’s comment and would like to clarify the novelty of our method as follows. Our core contribution lies in a structured and iterative synergy between a multi-modal time series encoder and multiple role-specific LLMs, designed to improve both predictive accuracy and interpretability. This end-to-end reflective refinement workflow, including error diagnosis and correction, moves beyond traditional prompting. We designed tailored prompt templates and modular reasoning flows (Appendix D, Pages 22-24), and validated them across multiple tasks and datasets, showing that our synergistic design goes beyond simple prompting or model combination.\n\n**L3 Experiments of forecasting** \n\nWe appreciate the reviewer’s observation and would like to clarify as follows. As noted in lines 129-132, we focus on discrete prediction settings to reflect real-world multi-modal decision-making scenarios, and we demonstrated regression-based forecasting through experiments on Finance dataset in Appendix F (Pages 26-29). This includes a model and prompt design tailored for regression tasks (Figures 24-25), along with performance evaluation against state-of-the-art forecasting baselines (Table 6), regression visualizations (Figure 26). We also provide an ablation study, iterative refinement analysis, and a case study (Tables 7-8, Figure 27). We chose to include these results in the appendix due to space limitations, but we believe they clearly support our claim of delivering both accurate and interpretable multi-modal forecasting. We will try to highlight these results more prominently in the main text."}, {"review_id": "QkyYd4E2K5", "reviewer": "Reviewer_srsZ", "summary": "This paper introduces TimeXL, a novel multi-modal time series prediction framework that integrates a prototype-based encoder with a closed-loop system of three Large Language Models (LLMs): prediction, reflection, and refinement. The approach enhances both predictive accuracy and interpretability by iteratively improving explanations and model outputs through LLM-driven feedback. TimeXL was evaluated on four real-world datasets spanning weather, finance, and healthcare, showing improvements in AUC by up to 8.9% compared to state-of-the-art baselines.", "questions": "where are the initial time series prototypes and text prototypes from? Random initialization?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe framework, which combines the prototype-based encoder and iterative LLM agents (predict-reflect-refine), is well-motivated.\n2.\tStrong empirical results and Explainability: Demonstrates significant performance improvements across diverse datasets, highlighting robustness. Case-based reasoning with multi-modal prototypes offers transparent and human-understandable predictions.\n3.\tComprehensive evaluation: Includes ablation studies, qualitative case studies, and comparisons with numerous baselines.\nWeaknesses\n1. Computational cost: Running multiple LLMs iteratively may be resource-intensive, which might limit real-time deployment.\n2. Ambiguity in prototype learning: The clarity of how well the prototypes generalize across different tasks or domains is underexplored.\n3. Unreliable on time series regression. This method heavily relies on time series segmentation, which may introduce bias based on arbitrary windowing choices. Prediction LLM returns “increase”, “decrease”, or “neutral” for prediction, which is unreasonable for a long-horizon time series regression, where granularity and uncertainty quantification are essential. \n4. No sensitivity analysis to the subsequence length.", "was_revised": true, "final_justification": "I keep the \"borderline accept\" rating.", "rebuttal": "We sincerely appreciate the reviewer’s constructive comments. Our responses to the concerns are listed below. \n\n**W1 Computational cost for real-time deployment**\n\nWe thank the reviewer for pointing out this practical concern and address it as follows. \n\nFirst, we provide an example demonstrating the computational cost of three LLMs in the iterative process. Following the prompts and strategy detailed in Appendix D (Pages 22-24), each iteration over  4000 training and validation samples (average length of 400 tokens or ~300 words) requires ~9.74 M input tokens (Prediction LLM 3.80 M + Reflection LLM 1.66 M + Refinement LLM 4.28 M) and yields ~1.64 M output tokens. With GPT‑4o (with input latency as TTFT ~0.45 s and output speed ~156 tokens/s), it corresponds to ~3.9 hours serially (prediction takes ~0.5 h, reflection ~0.08 h, and refinement ~3.35 h). The prediction and refinement steps can also be parallelized, reducing time cost per iteration to ~12 min with 20 concurrent calls. For a more efficient variant Gemini-Flash-2.0 (evaluated in Appendix B, Pages 18-19) with TTFT ~0.3 s and ~236 output tokens/s, the iteration time can be further reduced to ~8 min.\n\nWe would like to emphasize that we do not run reflection in the testing/deployment stage, but instead apply the fixed reflection selected during validation to refine the test texts (as noted in lines 245-250), reducing inference cost and latency. Based on our prompt lengths, the refinement and prediction steps take ~3.46 s per sample using GPT‑4o, and ~2.29 s per sample for Gemini-2.0-Flash, which is generally reasonable for real-time deployment. \n\nSecond, we would like to discuss more strategies for reducing computational cost in the iterative process. As shown in Figure 5 and discussed in Section 4.5, 1-2 iterations are often sufficient to achieve clear performance improvements. In real-world applications, the number of iterations can be adaptively controlled to balance performance gains and cost. For example, one can monitor performance gains on validation data after each iteration and stop early if the improvement falls below a predefined threshold. To further reduce the cost, we can also do selective refinement by skipping training examples that were already predicted accurately in previous iterations. These strategies help make our method more computationally tractable.  We will include an extended discussion in our updated version.\n\n\n**W2 Clarity regarding generalization of prototype across tasks and domains**\n\nWe would like to clarify that we have evaluated the learned prototypes across four datasets and two tasks.  Besides the multi-modal prototypes (Figure 3) and the case-based reasoning example (Figure 4) on the Weather dataset in Section 4.3-4.4, we also provided additional results for all datasets, including multi-modal prototypes and examples in Figures 9-13 of Appendix C (Pages 20-22). These results demonstrate that our prototypes capture meaningful and context-specific patterns across diverse domains. Beyond classification, we also evaluated the regression setting on the Finance dataset. This includes task-specific design, performance evaluation, prediction visualization, ablation study, iterative analysis, and case study, which are presented in Tables 6-8 and Figures 24-27 in Appendix F (Pages 26-29). \n\n**W3 Time series segmentation for long-term times series regression**\n\nThanks for raising this concern. We would like to clarify that time series segmentation is an effective strategy for time series regression, as the state-of-the-art forecasters PatchTST and TimeLLM, also rely on patching operations (essentially segmentation). In the regression setting presented in Appendix F (Pages 26-29), we also let prediction LLM output a numerical value in addition to the trend outcome (prompt in Figure 25). While multi-modal long-term time series prediction is not our primary focus, we would like to elaborate how our method can be adapted to this setting. One solution is to perform the single-step prediction and iterate over the forecast horizons, which provides fine-grained information for analysis. It is also possible to use a larger context window and let LLM generate multi-steps predictions in one-shot, while using the trend classification over window statistics as an auxiliary guidance signal to enhance temporal reasoning. As noted in Appendix H (Page 30), we will explore this setting with more viable solutions in our future work.  \n\n**W4 Sensitivity analysis of prototype (subsequence) length** \n\nWe thank the reviewer for pointing this out. To assess the sensitivity of prototype (subsequence) length, we conducted additional experiments on the multi-modal encoder by varying the lengths for both time series and text modalities across the Weather and Finance datasets (based on the refined testing texts). The results of their combinations and summarized statistics are presented below.\n\nWe observe that performance is relatively stable across a range of prototype lengths in the Weather dataset, with optimal results typically achieved at moderate lengths for both modalities. For time series, the 8-hour segments are often sufficient to capture the typical temporal patterns of weather conditions, and the 12-token window (where we used BERT for text embedding, as indicated in line 280) provides enough local contexts for weather prediction. For the Finance dataset, which operates on a business-day granularity, performance improves with longer lengths of time series prototypes that capture important financial trends. For the textual modality, we use Sentence-BERT to embed financial news at the half-sentence granularity (as noted in line 324), and shorter text windows (1-2 segments) typically provide compact and indicative market conditions for prototypes, making them effective for trend prediction. \n\nThese results suggest that our model is relatively robust to the choice of prototype length within a reasonable range that aligns with the temporal characteristics of the data. We will include an extended analysis in our updated manuscript. \n\nWeather Dataset \n||||||||||||||||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Time Series|4|4|4|4|8|8|8|8|12|12|12|12|16|16|16|16|\n|Text|4|8|12|16|4|8|12|16|4|8|12|16|4|8|12|16|\n|F1|0.605|0.644|0.650|0.627|0.640|0.633|0.659|0.635|0.613|0.614|0.641|0.633|0.618|0.612|0.639|0.614|\n|AUC|0.731|0.754|0.758|0.749|0.755|0.746|0.763|0.753|0.741|0.739|0.755|0.750|0.743|0.737|0.753|0.738|\n\n|Time Series - Overall|F1|AUC|\n|---|---|---|\n|4|0.632 ± 0.018|0.748 ± 0.011|\n|8|0.642 ± 0.011|0.754 ± 0.008|\n|12|0.625 ± 0.013|0.746 ± 0.007|\n|16|0.621 ± 0.012|0.743 ± 0.007|\n\n|Text - Overall|F1|AUC|\n|---|---|---|\n|4|0.619 ± 0.014|0.743 ± 0.008|\n|8|0.626 ± 0.015|0.744 ± 0.008|\n|12|0.647 ± 0.008|0.757 ± 0.004|\n|16|0.627 ± 0.009|0.748 ± 0.007|\n\n\nFinance Dataset\n\n|||||||||||||||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Time Series|1|1|1|2|2|2|3|3|3|4|4|4|5|5|5|\n|Text|1|2|3|1|2|3|1|2|3|1|2|3|1|2|3|\n|F1|0.428|0.433|0.429|0.476|0.462|0.424|0.484|0.470|0.471|0.538|0.529|0.510|0.616|0.588|0.559|\n|AUC|0.592|0.597|0.595|0.655|0.630|0.605|0.671|0.658|0.658|0.722|0.712|0.708|0.788|0.767|0.753|\n\n\n|Time Series - Overall|F1|AUC|\n|---|---|---|\n|1|0.430 ± 0.002|0.595 ± 0.002|\n|2|0.454 ± 0.028|0.630 ± 0.021|\n|3|0.475 ± 0.006|0.662 ± 0.006|\n|4|0.526 ± 0.014|0.714 ± 0.006|\n|5|0.588 ± 0.029|0.769 ± 0.018|\n\n\n|Text - Overall|F1|AUC|\n|---|---|---|\n|1|0.508 ± 0.073|0.666 ± 0.074|\n|2|0.496 ± 0.061|0.673 ± 0.067|\n|3|0.458 ± 0.045|0.664 ± 0.056|\n\n\n**Q1 Initial time series and text prototypes**\n\nThanks for this question! Yes, we introduce time series and text prototypes as randomly initialized training parameters, which are projected to an initial sample before training, where the specific implementation is provided in the supplementary code."}, {"review_id": "9emgLTu61u", "reviewer": "Reviewer_Vxdr", "summary": "The authors provide a new framework (TimeXL) for text+time series multimodal predictions using a series of prompts and a prototype-based encoder that generates more accurate predictions and contains a higher level of interpretability than previous approaches. The predict-critique-refine cycle applied to the encoder shows improved performance on select multimodal time-series classification tasks.", "questions": "Q1. Channel-based clustering seems fairly effective [link] for long-context time series predictions. I wonder if that might serve as a replacement for the prototype method instead? Have the authors considered other alternatives?\n\nQ2. It seems that the prototypes are found based on the segment-level representations which seems like it may lead to the loss of long-term or irregular sequence information. Is there any way to verify this through longer-sequence benchmarks (even on non-multimodal methods since the prototypes are applied to the time series data as well)?\n\nQ3. From the examples in figure 4, I’m curious what additional information seems to be added in either the original or refined input text. The same segments with matching prototypes in the original text “gradual increase in temperature” can also be found in the refined text, and moreover should be learned by the time-series model in the underlying sequence data anyway. It’s unclear what exactly is contributing to the higher performance. Perhaps investigating some of the classifications that changed to become true positives after introduction of the new framework would shed some light.", "limitations": "L1. The increased performance brought by this method is reliant on the nature of the data in the text modality and its pairing with the hand-engineered prompts and LLM models selected. This seems somewhat brittle, though there are obvious advantages in performance that the authors have shown for some tasks. Improved methods for evaluating the quality of the text data would help practitioners understand when it is feasible or desired to apply a multimodal method like this one vs just using the time series data.\n\nL2. The task generalization is evaluated in a limited fashion here and it may be the case that the classification tasks are particularly well-suited to this kind of fusion and iterative approach.\n\nL3. It is not clear to me if there is an easy way of knowing how much of the current evaluation benchmarks may have leaked into the LLM pre-training data and whether the introduction of these models may be simply providing roundabout (though maybe not direct) access to the ground truth of the task.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S1. Good Performance. Clear performance gains on the Weather and Healthcare datasets do validate the hypothesis that introducing the prototype-encoder and refinement loop improves on previous multimodal methods. \n\nS2. Interpretability through Prototypes. In general the prototype method is a helpful one for understanding similar patterns in data that may be related to a given inference. Other methods have shown the introduction of prototypes can improve performance on other modalities like vision/text, so it makes sense to try it here. It also acts as a helpful gut-check on the model’s reasoning. There is likely further room to develop these ideas in a more robust way.\n\nW1. Interpretability Evals. The claim that “This closed-loop workflow… continuously boosts the framework’s performance and interpretability” is not empirically verified. While performance increases are reported and average +1.9% AUC, the authors claim that TimeXL achieves up to 8.9% improvement in AUC but it seems that the largest increase is 6.6% on the weather task (vs TimeCAP) while the other healthcare baselines seem to be already saturated. It may be useful to create or introduce other multimodal benchmarks that can highlight if there is significant differentiation in the method. For interpretability it seems some predictions are inspected and verified by the authors, which does looks decent, but is not a very robust evaluation strategy. Have the authors considered having subject-matter experts for each task-type evaluate the results and report on whether the interpretations make sense?\n\nW2. Cost. There is no accounting of the latency or cost of either the training or inference method. TimeXL trains the encoder, calls a prediction, reflection and refinement LLM for every optimization step, then re-trains the encoder with the updated text. I imagine this significantly increases both training cost and inference latency compared to the encoder-only baseline. It would be helpful to articulate these costs so the right tradeoff between performance and cost can be evaluated. On many tasks its unclear that the additional interpretability and marginal performance gains would be worth the added complexity.\n\nW3. Task Expansion. Expanding to other tasks is mentioned in the paper but based on the appendix evaluations on regression it seems there is a lot less performance to be gained in these other task areas. There are other multimodal time series datasets the authors can try evaluating on (e.g. TimeMMD) and this might give more insight into the task generalization of the method.\n\nW4. Parameter Selection. Hand-tuned hyper parameters may not generalize well to other tasks. TimeXL’s final prediction is a linear combination of encoder and LLM outputs controlled by a weight, a, selected from validation data. Similar manual tuning applies to three prototype-regularisation coefficients and the number of prototypes injected into the LLM prompt. The paper does not report a sensitivity study, so practitioners may need extensive search to replicate the reported gains in new settings.", "was_revised": true, "final_justification": "I find the approach novel and interesting, however I do think the final results could be strengthened and there are some questions and weaknesses which haven't been fully addressed by the authors. In particular the two remaining areas that make me hesitate to raise my rating:\n1. The claims around explainability which are not well empirically grounded\n2. The concerns raised by reviewer vdBq around potential safety / ethical risks related to concept drift or the introduction of further bias through the LLM approach.", "rebuttal": "We sincerely appreciate the reviewer’s detailed and constructive comments. Our response to these concerns and questions are provided below.\n\n**W1 Clarification of AUC improvement, interpretation results examined by domain expert**\n\nWe first clarify the calculation of claimed AUC improvement. In the weather dataset, our method achieves the best AUC score 0.808, and the second-best AUC score from TimeCAP is 0.742. The percentage of improvement based on TimeCAP is (0.808-0.742)/0.742 = 8.9%. Similarly, the average AUC improvement is 2.2% (average F1 improvement 2.78%). We will differentiate the absolute improvement and relative percentage improvement in our manuscript. \n\nWe appreciate the reviewer’s suggestion for expert evaluation. We would like to kindly provide our justification as follows. Firstly, our framework is primarily task-performance driven, where explanations serve to enhance prediction accuracy. Secondly, we focus on general-domain datasets including weather, finance and influenza surveillance (test-positive and mortality), where we believe the contexts are relatively accessible to general AI practitioners. The results of multi-modal prototypes and case-studies (Figures 3-4, 9-13) are generally relevant and understandable without requiring deep domain expertise. However, we agree with the reviewer that domain experts can provide a more rigorous evaluation of explanation clarity, and will involve them in the future work for specialized domains like clinical decision-making.  \n  \n**W1, W3, L2 Task expansion on other benchmarks**\n\nThanks for suggesting additional tasks and benchmarks. We would also like to clarify that we still achieve noticeable improvement (a 6.62% relative RMSE improvement over the second-best baseline) in the regression setting on the Finance dataset. Overall, our method has been evaluated across three domains, four datasets, and two tasks. Nevertheless, we agree with the reviewer that extending the evaluation is valuable, and we plan to explore this in the future work.\n\n**W2 Computational cost and trade-off**\n\nWe thank the reviewer for this practical concern and address them as follows. For a dataset with 5,000 samples (including 1,000 testing cases) with average length of 400 tokens (~300 words based on OpenAI tokenizer). Following the prompts and strategy detailed in Appendix D (Pages 22-24), we estimate the token cost per iteration over train/validation samples as: Prediction 3.80 M input tokens (output negligible); Reflection 1.66 M input / 0.04 M output; Refinement 4.28 M input / 1.60 M output. It yields ~9.74 M input and ~1.64 M output tokens per iteration. \n\nWe next compute the time cost per iteration. With GPT‑4o (Input latency as TTFT ~0.45 s; output speed 156 tokens/s), prediction takes ~0.5 h, reflection ~0.08 h, and refinement ~3.35 h, totaling ~3.9 h serially. Using Gemini‑2.0‑Flash (TTFT ~0.3 s; 236 tokens/s) reduces this to ~2.6 h. Prediction and refinement can also be parallelized (e.g., 20 concurrent calls cut GPT‑4o’s iteration time to ~12 min). Note that at test time we apply the fixed reflection to refine the test texts and perform prediction, yielding a reasonable inference latency of ~3.5 s/sample for GPT‑4o, and ~2.3 s/sample Gemini‑2.0‑Flash.\n\nWe agree with the reviewer that the trade-off between performance and cost is important in practice. As shown in Figure 5 and discussed in Section 4.5, 1-2 iterations are sufficient to achieve clear performance gains. In practice, the iterations can be controlled by monitoring performance improvements on validation data and stopping early if gains are below a threshold, thereby balancing accuracy with computational cost. Moreover, cost can be further reduced by selective refinement (skipping examples already predicted correctly in prior iterations), making our method more tractable. We will add a detailed discussion in our updated version.\n\n**W4 Parameter selection and generalization** \n\nWe appreciate the reviewer’s comments for parameter selection and respond as follows. Firstly, we would like to emphasize that the weight for output fusion is simple yet effective for augmented prediction. The optimal weight is selected by evaluating the values from 0 to 1 on the validation logits, providing an efficient and training-free solution for generalization. For the sensitivity of case-based examples (with prototypes) fed to LLM, the evaluation is provided in Figure 7 of Appendix B (Pages 18-19), with discussions in lines 742-751, showing the advantage of more relevant case-based explanations. We also provide qualitative ablation for the encoder regarding regularization coefficients in Figure 6 of Appendix B. \n\nTo quantitatively assess their sensitivity, we provide the evaluation results for the multi-modal encoder on the Weather dataset (based on the refined testing texts), where each parameter is selected from [0.1,0.3] with interval 0.05, following Appendix A.2 (Page 17). As shown in the below table, model performance remains relatively stable across different values. For both modalities, moderate values of $\\lambda_1$ and$ \\lambda_2$ tend to yield better performance, suggesting the clustering and evidencing effectively guide meaningful prototype learning. A slight diversity constraint ($\\lambda_3$) also helps to keep a compact interpretation prototype space. We agree with the reviewer that a more detailed sensitivity evaluation is beneficial for new settings, and we will include an expanded analysis in the updated version.\n\n|Time|Metric|0.1|0.15|0.2|0.25|0.3|\n|-|-|-|-|-|-|-|\n|$\\lambda_1$|F1|0.628±0.030|0.627±0.027|0.633±0.029|0.627±0.028|0.625±0.030|\n||AUC|0.733±0.022|0.731±0.026|0.736±0.026|0.731±0.024|0.729±0.023|\n|$\\lambda_2$|F1|0.632±0.028|0.634±0.028|0.627±0.032|0.626±0.028|0.628±0.028|\n||AUC|0.734±0.024|0.736±0.022|0.733±0.028|0.729±0.021|0.733±0.025|\n|$\\lambda_3$|F1|0.631±0.028|0.631±0.027|0.630±0.028|0.624±0.032|0.628±0.029|\n||AUC|0.736±0.022|0.735±0.024|0.733±0.024|0.727±0.026|0.733±0.025|\n\n|Text|Metric|0.1|0.15|0.2|0.25|0.3|\n|-|-|-|-|-|-|-|\n|$\\lambda_1$|F1|0.629±0.027|0.629±0.026|0.631±0.027|0.635±0.027|0.634±0.028|\n||AUC|0.732±0.024|0.736±0.024|0.740±0.026|0.738±0.023|0.735±0.025|\n|$\\lambda_2$|F1|0.632±0.031|0.631±0.026|0.635±0.024|0.629±0.026|0.630±0.028|\n||AUC|0.734±0.027|0.736±0.021|0.740±0.024|0.735±0.021|0.736±0.028|\n|$\\lambda_3$|F1|0.635±0.026|0.633±0.023|0.628±0.025|0.625±0.029|0.627±0.027|\n||AUC|0.740±0.021|0.736±0.025|0.731±0.023|0.729±0.027|0.731±0.026|\n\n**Q1 Channel-based clustering for long-term time series predictions**\n\nThanks for raising channel-based clustering in our discussion. Based on our knowledge, it assigns channel-wise embedding to forecasters and effectively exploits channel similarity via a clustering objective that enhances accuracy. However, its scope differs from our prototype design, thus may not serve as a replacement. Our design focuses on interpretable prediction via case-based reasoning over modalities, based on the explanations identified from training data. While the channel-based clustering provides some insights into channel groups, its granularity is different from our instance-level explanation. We recognize its potential value for multi-modal time series analysis, and will carefully discuss this work and related methods in the related work.\n\n**Q2 Segment-level prototype for long-term and irregular time series predictions**\n\nWe would like to clarify that segment-level modeling remains effective for time series forecasting. The state-of-the-art models like PatchTST and tPatchGNN[1] also rely on patching that essentially operates at the segment level to handle long and irregular time series, respectively. \n\nNext, we sketch how to adapt our method to long-term forecasting. One solution is to increase the convolution window and stride (similar as patching), optionally with other mechanisms like multi-scale modeling or time series decomposition. Alternatively, a prototype layer with learning objectives can be added to the projection layer after patching in PatchTST. In this way, prototypes can capture periodic or salient patterns in long-term time series benchmarks.\n\n[1] Irregular Multivariate Time Series Forecasting: A Transformable Patching Graph Neural Networks Approach.\n\n**Q3 Additional information introduced by TimeXL in the case study**\n\nWe provide a case study in Figure 4 where prediction is calibrated to a rainy outcome by TimeXL. As noted in lines 312-317, the refined texts preserve the statement of stability in original texts while emphasizing more on humidity and wind as more indicative of rain, guided by reflections from training examples, contributing to the calibrated prediction. The refined texts also improve the quality of learned prototypes by filtering out less relevant cues. “Gradual increase in temperature” becomes less relevant as more indicative phrases in the same input yield higher similarity scores, better highlighting upcoming rainy conditions. Nevertheless, we thank the reviewer’s advice and will provide more examples in our updated version.   \n\n**L1 Evaluating text quality to decide if multi-modal method is needed**\n\nWe sincerely thank the reviewer for pointing out this practical challenge. While our method handles noisy or redundant textual inputs through iterative and reflective feedback and supports interpretable predictions, we agree that its effectiveness may be limited when the texts are extremely sparse or uninformative. In such cases, it is important to assess the text quality for model development. Integrating domain-specific prompts and external knowledge retrieval could also help. We will discuss this issue in our updated version.\n\n**L3 Assessing the current benchmarks and LLM pretrain data**\n\nWe thank the reviewer for raising this concern. The potential leakage regarding the LLM pretraining corpus remains a shared challenge in current LLM-based research, and we will carefully assess the risk in the future work."}, {"review_id": "2JN2prFrRM", "reviewer": "Reviewer_vdBq", "summary": "1. A prototype-based encoder producing transparent, case-based rationales.\n\n2. Iterative refinement via LLM-driven reasoning to improve predictions and text quality.\n\n3. Empirical validation showing superior performance over baselines.", "questions": "1. Generalization to Non-Textual Modalities: The framework relies heavily on textual context. How might it perform with other auxiliary modalities (e.g., images, sensor data)? \n\n2. Scalability: The iterative refinement process may incur computational costs. Are there optimizations for large-scale deployment? \n\n3. Ethical Risks: The reflection LLM might amplify biases in textual data. How is bias mitigation handled?", "limitations": "The authors acknowledge limitations (e.g., focus on classification, future extension to regression). \n\nSuggestions:\n\nDiscuss computational trade-offs of the refinement loop.\n\nExpand on ethical safeguards for LLM-generated text (e.g., bias audits).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The methodology is technically sound, with rigorous experiments across diverse datasets (weather, finance, healthcare). The ablation studies validate design choices (e.g., multi-modal fusion, LLM integration).\n\n2. Well-organized with clear figures (e.g., workflow). The appendix supplements key details (prompts, reflection reports).\n\n3. Addresses a critical gap in explainable AI for time series, with potential high-stakes applications (e.g., healthcare, finance). The 8.9% AUC improvement is notable.\n\n4. Novel integration of prototype-based encoders with LLMs for iterative refinement. The closed-loop workflow is innovative.\n\nWeaknesses:\n\n1.  The regression task demonstration (Appendix F) is less detailed compared to classification. Broader regression applications (e.g., long-term forecasting) are noted as future work.\n\n2. Some technical terms (e.g., \"cross-modal relevance fusion\") could be further clarified for non-experts.\n\n3. While the framework is general, domain-specific limitations (e.g., reliance on textual context quality) could be explored further.\n\n4. Similar LLM-augmented approaches exist (e.g., TimeCAP), though TimeXL’s focus on interpretability is distinctive.", "was_revised": true, "final_justification": "Ethical Risks is unresolved. As the authors explain, \"Adapting such evaluations to our setting would require additional effort, such as task-specific annotations.\" \n\nI keep my score.", "rebuttal": "We sincerely appreciate the reviewer’s constructive feedback and address the concerns below.\n \n**W1 Details of regression tasks**\n\nWe thank the reviewer for the helpful feedback. While we acknowledge that our primary emphasis is on classification settings, we would like to clarify that the regression task in Appendix F (Pages 26-29) is methodologically complete and systematically evaluated. Appendix F includes a modified methodology (Figure 24) and prompt design (Figure 25) tailored for regression tasks, along with performance evaluation against state-of-the-art forecasting baselines (Table 6) and regression visualizations (Figure 26). We also provide supporting analyses including an ablation study (Table 7), iterative refinement analysis (Table 8), and a case study (Figure 27). That said, we appreciate the suggestion and agree that broader regression scenarios, such as long-horizon forecasting, deserve further investigation. We will also consider expanding the regression coverage in the main text or supplementing further details in an updated version of the manuscript.\n\n**W2 Clarification of terms**\n\nThanks for the helpful suggestion. As noted in lines 162-163, we consider the cross-modal relevance by computing similarity scores between prototypes and input segments across both modalities, which are then fused using a non-negative weight matrix to obtain prediction probabilities. We will revise the text and add more details to make this term clearer for non-expert readers, and pay closer attention to other terms.\n\n**W3 Exploration of domain-specific limitations such as reliance on textual context quality**\n\nWe sincerely thank the reviewer for pointing this challenge out. Our method is able to handle noisy or redundant textual inputs by using iterative and reflective feedback from training data to improve text quality for more informative prediction. This makes the framework robust to moderate imperfections in the textual context.\n\nHowever, we agree with the reviewer that in domains where textual context is extremely sparse, unstructured or uninformative, the effectiveness of our method could be limited. In such cases, additional strategies from domain-experts such as designing domain-specific prompts or methodology developments such as cross-modal augmentation and external knowledge-base retrieval would be needed to address this challenge. We will view this as an important direction for future work and discuss them in our manuscripts. \n\n**W4 TimeXL’s advantage over existing LLM-augmented approaches**\n\nThanks for acknowledging our contribution in terms of interpretation! We would like to slightly justify that, besides the built-in interpretation mechanism and performance improvement, our approach provides a unique synergy and mutual augmentation between a multi-modal time series model and LLMs through iterative, grounded, and reflective interaction. \n\n**Q1 Generalizing TimeXL to non-textual modalities**\n\nThanks for this question! Our proposed method has the capability to handle other auxiliary modalities for interpretable prediction, where the key idea is to use a prospective modality encoder with a prototype design. For example, to support an image modality, we can use a 2D convolutional encoder to extract spatial features from local regions. Class prototypes can then be defined over the resulting representations, with similarity scores computed via Equation 2 and fused with other modality scores through Equation 3 for outcome prediction. After training, these interpretable prototypes can be identified as representative image regions via Equation 6 (e.g., local abnormal regions of an X-ray in medical imaging). These prototypes enable case-based visual explanations during test inference, when jointly analyzed with other modalities (e.g., lab-test time series, clinical notes, which our framework currently supports).\n\n**Q2 and L1 Scalability and optimizations for large-scale deployment with trade-off**\n\nWe thank the reviewer for pointing this practical question out and provide our discussion as follows. First, we would like to quantitatively discuss the computational cost of the iterative process regarding token usage and time. Following the prompts and strategy detailed in Appendix D (Pages 22-24), each iteration over  4000 training and validation samples (average length of 400 tokens or ~300 words) requires ~9.74 M input tokens and yield ~1.64 M output tokens. With GPT‑4o (with input latency as TTFT ~0.45 s and output speed ~156 tokens/s), it corresponds to ~3.9 hours serially per iteration (prediction takes ~0.5 h, reflection ~0.08 h, and refinement ~3.35 h). As prediction and refinement steps can also be parallelized, we can reduce the iteration time to ~12 min with 20 concurrent calls. Using a more efficient LLM base model Gemini-Flash-2.0 (evaluated in Appendix B, Pages 18-19) with TTFT ~0.3 s and ~236 output tokens/s, the iteration time can be further reduced to ~8 min.\n\nNote that we do not run reflection in the testing/deployment stage, but instead apply the fixed reflection selected during validation to refine the test texts (as noted in lines 245-250), reducing inference cost and latency. Based on our prompt lengths, the refinement and prediction steps take ~3.46 s per sample using GPT‑4o, and ~2.29 s per sample for Gemini-2.0-Flash, which is reasonable for deployment in real-world applications . \n\nBesides using more efficient LLM backbones and parallelized query as aforementioned, there are a few more strategies to optimize the computational cost for large-scale deployment. Firstly, we can control the number of iterations based on the trade-off between performance and cost. As shown in Figure 5 and discussed in Section 4.5, 1-2 iterations are often sufficient to see clear performance improvements. We can monitor performance gains on validation data after over iterations and terminate the loop early if the improvement falls below a predefined threshold. We can also do selective refinement by skipping training examples from the large scale data, which were already predicted accurately in previous iterations. These strategies help make our method more computationally tractable. We will include a detailed discussion in our updated version.\n\n**Q3 and L3 Ethical risk mitigation for LLM-generated texts**\n\nWe appreciate the reviewer’s thoughtful comments regarding the bias in LLM generated texts. As we noted in the checklist, we tried to review and monitor the LLMs responses, using structured prompt templates (provided in Appendix D, Pages 22-24 and supplementary code), and the datasets are from the general domains, which are fairly neutral and not identity-sensitive. Nevertheless, we agree with the reviewer that this is an important concern and we will consider more formal assessments and countermeasures in future work, particularly when applying the framework to sensitive datasets such as clinical data."}]

Author rebuttal:

Meta-review: Following the final round of reviews, the paper received four borderline accept decisions. The reviewers recognized the technical novelty of the proposed approach and noted its strong performance across the evaluated benchmarks. They also emphasized that the explainability feature represents a significant contribution, addressing an important gap in the field of explainable AI for time series analysis. However, the reviewers also raised concerns regarding the computational overhead associated with the method, suggesting this may limit its scalability or applicability to other domains. The rebuttal addressed some of these concerns, but only partially satisfied the reviewers’ concerns.
Decision: Accept (poster)