ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
Authors: Yilin wang
Peixuan Lei
Jie Song
Yuzhe Hao
Tao Chen
Yuxuan Zhang
LEI JIA
Yuanxiang Li
zhongyu wei
Venue: ICML 2025 poster (2025)
Keywords: Time Series Analysis
Time-Series Question Answering
Time-Series-Textual Alignment
Time-Series-Textual Fusion
Primary area: applications->time_series
TL;DR: Bridging time-series data and natural language, we propose ITFormer and introduce EngineMT-QA, enabling efficient and accurate Time-Series Question Answering for multimodal AI

Abstract: Time-series data are critical in diverse applications, such as industrial monitoring, medical diagnostics, and climate research. However, effectively integrating these high-dimensional temporal signals with natural language for dynamic, interactive tasks remains a significant challenge. To address this, we introduce the Time-Series Question Answering (Time-Series QA) task and release EngineMT-QA, the first large-scale, multi-task, temporal-textual QA dataset designed to capture complex interactions between time-series signals and natural language. Building on this resource, we propose the Instruct Time Transformer (ITFormer), a novel framework that bridges time-series encoders with frozen large language models (LLMs). ITFormer effectively extracts, aligns, and fuses temporal and textual features, achieving a strong improvement in QA accuracy over strong baselines with fewer than 1\% additional trainable parameters. By combining computational efficiency with robust cross-modal modeling, our work establishes a adaptable paradigm for integrating temporal data with natural language, paving the way for new research and applications in multi-modal AI. More details about the project, including datasets and code, are available at: https://pandalin98.github.io/itformer_site/.

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
Yilin Wang 12 Peixuan Lei" Jie Song' ** Yuzhe Hao 1 Tao Chen
Yuxuan Zhang'
Lei Jia
Yuanxiang Li' Zhongyu 23
Abstract
as industrial condition monitoring( (Wan et al., 2025b) (e.g.. turbine engine health management) (Wang et al., 2022)
Time-series data are critical in diverse applica- tions, such industrial monitoring, medical di- agnostics, and climate research. However, effec- tively integrating these high-dimensiona, tempo- ral signals with natural language for dynamic, in- teractive tasks remains a significant challenge. To address this, we introduce the Time-Series Ques- tion Answering (Time-Series -eries QA) task and re- lease EngineMT-QA, the first large-scale, multi- task, temporal-textual QA dataset designed to cap- ture complex interactions between time-series sig- nals and natural language. Building on this re- source, we propose the Instruct Time Transformer (ITFormer), novel framework that bridges time- series encoders with frozen large language mod- els (L..M).. ITFormer effectively extracts, aligns, and fuses temporal and textual features, achiev- ing a strong improvement in QA accuracy over strong baselines with fewer than 1 % additional trainable parameters. By combining computa- tional efficiency with robust cross-modal model- ing, our work establishes adaptable paradigm for integrating temporal data with natural language, paving the way for new research and applica- tions in multi-modal AI. More details about the project, including datasets and code, are available at: htt.a//aiiinnniiiiibioiiooooooo site/.
medical diagnostics (c.g., clectrocardiogram analysis) (Yi et al., 2024; Gui et al., 2024), and climate research (Zhang et al., 2023; Wu al., 2023; Bi et al., 2023). These data often exhibit high-dimensional patterns and intricate tempo- ral Sevenddences(Waanng et al., 2023; 2024c), carrying valu- able information essential for precise analysis and decision- making. While existing research has made progress in tasks like time-series classification( Yue et al., 2022; Goswami et al., 2024), forecasting(Zhou et al., 2022; Zhang & Yan, 2023; Liu et al., 2024d), and anomaly detection (Jin et al., 2024a; Darban al., 2025), most efforts remain focused on single-modality tasks, limiting their applicability to dynamic and interactive environments.
TimcSeries Language Multiiak Internction
Supervised Fine-Tuning
110k QA-Pairs
[1: you shrrgrit engine sigeal during ojile
ENGINE
Time Series
ITFormer
Language (LLM)
Large
stowing aridal adarmment
Encoder
Transformer
bulth mase given
2800
Opeecl Pooeods cllew Folice
Learnable
Instruction Tocars
Figure 1: EngineMT--QA: large-scale QA dataset based aero engine time-series signals. The proposed ITFormer framework seamlessly connects any time-series encoder with LLMs, modeling temporal-text interaction by embedding time-series semantics into natural language.
1. Introduction
A growing need exists for practical applications where users can interact with time-series data directly using natural lan- guage. For example, a engineer might ask if an engine com- ponent shows abnormal vibration during a specific time pe- riod or request a summary of performance trends inform maintenance decisions. This demand lies at the intersection of time-series analytics and natural language processing (NLP), area gaining relevance with the advancements in large language models (LLMs) (Wei et al., 2022). How- ever, despite increasing interest in cross-modal research (Sun et al., 2024; Rasheed et al., 2024; Lu et al., 2024), there has been limited effort to systematically explore the
Time-series data have become increasingly significant in numerous real-world applications(Warn et al., 2025a), such
'Equal contribution 'School of Aeronautics and Astronau- ties, Shanghai Jiao Tong University, Shanghai, China SShanhhai Innovation Institute. Shanghai, China "Scoool of Data Science, Fudan University, China. Correspondence to: Yuanxiang < satnnii@sjtu.cdu.c..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
complex and abstract semantics underlying time-series data in the context of natural language queries. Moreover, standardized benchmark currently exists to evaluate method- ologies in Time-Series Question Answering (Time-Series QA), leaving this domain underexplored.
1.
We formally introduce the Time-Series QA task and re- lease the first large-scale, multi-task, temporal-textual QA dataset, EngineMT-QA, which enables systematic exploration of temporal -exttal interactions.
2.
We propose ITFormer, an innovate framework for align- ing and fusing time-series representations with natural language, achieving substantial performance improve- ments across diverse QA tasks with minimal training parameters. ITFormer's core methods include Time Token Position Encoding, Learnable Instruct Tokens, Instruct Time Attention, and Time Token as Language, which collectively drive effective cross-modal model- ing.
To address this gap, we introduce the Aero Engine Multi- Task Question Answering Dataset (EngineMTT-QA), a large- scale, multi-task, and temporal-textual QA dataset specifi- cally designed for time-series data. EngineMT-QQA is con- structed based on real-world aero engine operational and maintenance scenarios, aiming to capture the intricate in- teractions between time-series signals and natural language. The dataset consists of over 110k question-ane -nnwwer pairs derived from 33 sensor channels, covering a diverse set of queries across four key tasks: understanding, perception, reasoning, and decision-making. These tasks are designed to reflect the inherent complexity of time-series data and explore their underlying semantic structures. For instance, some questions assess the ability to interpret subtle ampli- tude variations in specific signals and their semantic impli- cations, while others require reasoning about the probability of component failure and making informed decisions on whether proceed with repairs continue operation based on the deeper semantic meaning of the time-series signal. By releasing this dataset, we aim to facilitate research on bridging time-series data with natural language, paving the way for more intelligent and interpretable solutions in real- world applications.
We establish a robust and adaptable paradigm for inte- grating complex time-series tasks into end-to-end QA frameworks, paving the way for more comprehensive multi-modal applications and offering valuable bench- marks and tools to the research community.
2. Related Work
Time-Series Analysis Time-series research has tradition- ally focused on tasks such as classification, forecasting, and anomaly detection (Hamilton, 2020). Farly methods like ARIMA (Shumway et al., 2017) and VAR (Clements & Mizon, 1991) were designed to model linear temporal dependencies. With the advent of deep learning, neural mod- els(Dennis et al., 2019; Oreshkin et al., 2022) significantly improved the ability to capture non-linear patterns in time- series data. More recently, Transformer-basee -basedder-baeddddde architectures have pushed the boundaries of long-range dependency mod- eling(Zhou et al., 2021; 2022; Nie et al., 2023; Liu et al., 2024b), leading to the development of foundational time- series models that solve various tasks (Liu et al., 2024d; Gao et al., 2024). However, these foundational models are predominantly single-modal, limiting their applicability in scenarios that require multi-modal integration or interactive analysis.
Building on this new data resource, we introduce Instruct Time Transformer (lT'oormer), a novel framework designed to bridge temporal sequence data and frozen LLMs for effi- cient temporal- -eetrall multimodal interactions in Fig. 1. IT- Former acts as an intermediary connector, enabling seamless integration between temporal encoders and frozen LLMs by addressing three core challenges. First, ITFormer employs Time Token Position Encoding to systematically represent temporal, channel, and segm position information. Second, ITFormer introduces Learnable Instruct Tokens and Instruct Time Attention to dynamically align and fuse temporal features with task-specific textual queries, creat- ing a unified semantic space for effective modeling. Third, the Time Token as Language strategy represents temporal features as language-compatible tokens, enabling smooth integration with LLMs while requiring minimal computa- tional overhead.
Large Language Models and Multi-Modal QA Large Language Models (LLMs) have demonstrated remarkable proficiency in various NLP tasks, including machine transla- tion ( (Vaswani, 2017), text summarization (See et al., 2017), and question answering (Rajpurkar, 2016). By training on massive text corpora, they capture rich linguistic pat- terns that generalize across diverse downstream applications. Recent advancements have extended LLMs to multimodal tasks, primarily in the vision-language domain (Lu et al., 2019; Li et al., 2022), where models integrate textual and visual information to improve reasoning and comprehen- sion. Similarly, the field of multi-modal QA has evolved around vision- linguaelanguage benchmarks such as VQA (Antol et al., 2015) and Visual Dialog (Das et al., 2017), which
Empirical evaluations demonstrate that ITFFommer achieves state-of-the-art performance, significantly surpassing strong basclines in QA tasks. By leveraging fewer than 1% additional trainable parameters, ITFormer balances com- putational efficiency with robust performance, advancing temporal-textual cross-modal modeling.
In summary, our contributions are follows:
2

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
require models to fuse image features with textual queries to generate accurate answers (Tan & Bansal, 2019; Lu et al., 2019). While these efforts have driven research in cross- modal alignment techniques, the integration of time-series data into LLMs remains largely unexplored. Unlike visual inputs, time-series data pose unique challenges, as their semantics are more abstract and governed by multi-scale temporal dependencies and continuous signal dynamics. Moreover, the vast availability of time-series data and the increasing need for interactive analysis necessitate more fine-grained research on temporal-textual multimodal mod- cling beyond conventional multimodal QA frameworks.
1: (T,))) H a,
(1)
where the mapping function f encapsulates the interaction between the time series and natural language modalities. f is decomposed into several components to facilitate ef- fective modeling. First, the time series T encoded into latent representation Hr via a time series encoder Pr. i.e., Hr ΦT (T). Simultaneously, the question q is transformed into a semantic representation Hy via a lan- guage encoder bg. i.e., Hg = (g( These modality- specific representations are then fused through an interac- tion mechanism Vi to obtain joint representation Hiusion- i.e., Hiusion = V (HT, Hy)) Finally, the fused representation Hiusion is decoded into the answer a under the condition of the question q. The decoding process is defined
Towards Time-Series QA Although some efforts have attempted to integrate text and time-series modalities(in et al., 2024c;b), existing works predominantly treat textual inputs as auxiliary features to enhance traditional time-series tasks, such as prediction(Want (Wano et al., 2024b) or classifi- cation (Wang et al., 2024a). These approaches often fo- cus on pre-defined prompt or rely on simplified datasets, limiting their ability to generalize to real-world scenarios. While some efforts have explored natural language inter- faces for time-series data (Liu al., 2024a), they primar- ily focus on generating narrative descriptions or extracting basic patterns (Wang et al., 2024a), without delving into deeper semantic modeling or capturing the complex inter- actions between time-series data and language. To the best of our knowledge, there is no systematic exploration of Time-Series QA, where models are required to directly infer answers from raw temporal signals guided by rich natural language queries. The absence of a standardized benchmark further limits progress in this domain. To bridge this gap, we introduce EngineMTT-QA, a dedicated dataset for time- series question answering, alongside our novel architecture, ITFormer.
(Pa(Husso q),
where ou represents the decoder funetion that incorporates the alignment and fusion results Husion alongside the textual question g to generate the final answer.
4. Methodology
We propose ITFormer (Fig. 2), a novel framework that enables efficient temporal-textual multimodal interactions by bridging time-series data and frozen LLMs. Acting as an intermediary. ITFFormer seamlessly integrates any time- series encoder with any frozen LLM, extracting the semantic essence of temporal tokens based on task-specific instruc- tions for alignment and fusion in a shared semantic space.
4.1. Instruct Time Transformer
Designed to address challenges in temporal-textual align- ment, such as modality heterogeneity and computational ef- ficiency, ITFormer incorporates key components to achieve effeective representation fusion.
3. Problem Definition
Time Token Position Encoding (TPE) To enable the model to effectively distinguish between multi-segment and multi- dimensional temporal tokens, we design a position encoding mechanism that operates at three levels: temporal steps, channel semantics, and temporal segments. Specifically, the temporal tokens are first processed by the time series encoder o and position encodings are subsequently added to the resulting features. The final representation is given by:
Time-series analysis encompasses diverse range of tasks, including understanding, perception, reasoning, and decision-making, among other high-level cognitive pro- cesses. These tasks involve diverse input-output formats, making it challenging to establish a unified framework. To address this, we propose question-answering (QA) paradigm that reformulates all time-series tasks into a sin- gle, coherent framework. Formally, given a time-series col- lection T (Tı, T2..... .,, TN); ), where each segment T ( ,i.,t, } consists of sensor signals across V channels with L, steps, i.c., Ii, E RRY, and a natural lan- guage query q E Q, the objective is to generate a corre- sponding natural language response a E A. This task can be formulated as learning a unified mapping function:
HT = (T) - Pime + P Pohannel + P sogment:
(3)
The term (T) refers to the initial feature representation of the time series T, produced by the time series encoder. For a given segment T, = 04,,,, . where
3

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Time Token as Language
Time Token Position Encoding
Output Text
Chamnel
Based angine signal collected 10 cycles [etts> "rt, what immediate actions should be taken to address the observed issues?
Tokenization
Replace
O 0CO3 00
Time
Instruct Time Attention
ITFormer
Pos
axd
Time Instruct Atten
AddlMermm
Nx
Reslaped
Lxd,
Trurspooe
Softman
Raihapa
Project
Channel Instract Aften
Lxd
L Lxd,
L Fused Query
Embrodding
Time Series
axd
Channel Instruct Atten
AdeöMeerm
Encoder
Krrapekrraspoi
Vxd,
nxV
Atterton
Time Tokens
Brdupe
Project
V=I.ad
tokers
Aggregrdion
Learnable Instruct
FsL
Tokens JEB
Impat
Training
Figure 2: The design of ITFormer includes several key components: 1. Time Token Position Encoding (TPE), 2. Learnable Instruct Tokens (LIT), and the 3. Instruct Time Attention (ITA) mechanism. The framework also incorporates the 4. Time Token as Language (TAL)
strategy, which represents temporal semantics as language tokens to enhance the expressiveness of fused representations.
Ti, E RRV, the encoder produces a latent representation (T.) VVxx where T4 represents the number of tokens in the temporal dimension after the Patch transfor- mation (Nie et al., 2023). To incorporate positional informa- tion, P time employs sinusoidal encodings along the temporal steps t = 1, ...,, I, capturing the sequential structure of the time series. In parallel, P Pchannel introduces learnable em- beddings to differentiate semantic information across the V channels. For multiple segment inputs, token sequences are concatenated as (T) x'x'x where L' =I L4- The encoding P segment applies rotary encodings to distin- guish between temporal segments [Ti, Tz,..., Tvy: ensur- ing clear identification in multi-segment scenarios.
augmented query is thus represented as:
q = [: g).
where I E Rxd shares the same embedding dimensionality as to ensure compatibility. The concatenated sequence i undergoes self-attention mechanism to refine its semantic representation, resulting in:
Hg = SelfAttention(()) (q).
(5)
where H. E (, fused the task-specific semanties of q in I. After the self-attention operation, the first tokens of Hl are extracted as the final output, denoted as I = [ii .,....111
Together, these position encodings enable the model to lever- age the hierarchical structure of temporal data, thereby fa- cilitating effective alignment and seamless interaction with textual semantics.
.i,. where:
r Hyl: m)).
(6)
LIT serve as condensed semantic instructions, encoding task-relevant information from query to guide downstream alignment and interaction between temporal and textual rep- resentations. This design ensures that the alignment process leverages both task-specific guidance and the contextual semanties of the textual query, enhancing the fusion quality in the shared semantic space.
Learnable Instruct Tokens (LIT) To facilitate the align- ment between temporal and textual modalities, we introduce LIT tokens, denoted as I = (i,, ig.... , }. which are used to extract and integrate the semantics from both language and time-series signals. These tokens are prepended to the semantic vector representation of the natural language query q 1R Here, is obtained by passing the raw tex- tual query through the tokenizer of the frozen LLM. The
Instruct Time Attention (ITA) The ITA mechanism em- ploys a two-stage process to dynamically align and fuse temporal and textual representations. This involves Channel

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Instruct Fusing followed by Time Instruct Attention, lever- aging the task-specific LIT guide the alignment. ITA first aggregates relevant features of temporal tokens along the feature dimension based on the instruction, capturing the intrinsic semantics of each temporal step. Subsequently, the mechanism aggregates these features across the temporal dimension to capture the semantic evolution over time.
tokens in the query. This operation ensures that the tempo- ral semantics encoded in Hiusion seamlessly integrated
into the query's semantic structure. The augmented input sequence is defined as:
Hg = Replaceelacchoodder((... Hiusion)
(12)
where RepllacePlacchooderss:::: denotes the replacement of predefined placeholder tokens in Hg with the corre- sponding tokens from Hrusiun- The resulting sequence Hy E ( aligns temporal and textual semantics within the same embedding space, making it suitable for processing by the frozen LLM. The decoder in the frozen LLM then generates the answer a as:
Channel Instruct Fusing
In this stage, the temporal features HT E RR Xxx, where I' represents the number of temporal tokens, V the number of channels, and d the feature dimension, are aggregated along the channel dimension under the guidance of the LIT I". This step extracts channel-wise information most rele- vant to the LIT at each temporal step.
a (H,))
(13)
where T. represents the decoding process in the LLM, uti- lizing the integrated temporal-textual representation Hg to produce the final answer.
The attention mechanism is defined as:
Qchannel W' Kehannel Hr Wa, V channel Hp W un
(7)
where are learnable projection matrices.
4.2. Training Process
The channel-wise attention weights are computed as:
ITFormer is trained via supervised fine-tuning (SFT) by minimizing the cross-entropy loss between generated and ground-truth answers. Only ITFormer's parameters are up- dated, while the time encoder o and LLM & (e.g.. 7B parameters) remain frozen. The time encoder Φ7 extracts temporal embeddings Hy from multivariate time-series data T RLXV
(ochamen K
Achamnel = Softmax
V
channel
Vdx
where represents the relevance of each channel to the LIT.
Using these attention weights, the channel-wise aggregation is performed as:
= (T)
(14)
V
Hehannel [1, k]
Achannel [g, v] V Vchannel [4, v, k],
Alignment Training Only the alignment module V (ap- proximately 0.07% of total parameters) is updated, pro- jecting Hy into the semantic space of query embeddings to form the fused representation:
(9) The resulting represents the fused tokens Hehannel E RLL xd for each temporal step.
Hg = pg(g)
Hiusion (Hr, Hg).
(15)
Time Instruct Attention The output from the previous stage, Hehamnet, is further refined to capture temporal interactions guided by the LIT. This stage computes cross-modal atten- tion between the temporal sequence Hehammet and the LIT I" The attention mechanism is defined as:
The frozen LLM decoder & then generates the final answer a:
a D (H (usion))
(16)
The training minimizes the cross-entropy loss:
Qiime I'W, Kiime Hehannel W Vtime Hehamnet W
(10)
Caliga (0 =- Σ log P(a (v),,
(17)
where W W W E dRdx are learnable projection 4r ke
(Tig.)ED
matrices. The time-wise attention is computed as:
where only 0g updated, ensuring efficient multimodal alignment by leveraging pretrained components for robust cross-modal QA performance.
Quime QQimeKains K time
Hission = Softmax
(dihe Vilk
V Viimes
(11)
where Hassio E x *epressest the fused temporal repre- sentation aligned with the textual LIT.
5. Experiment
5.1. Dataset Description
Time Token as Language (TAL) The fused temporal rep- resentation Hussion E R is treated as a sequence of task- specific tokens and directly aligned with the natural lan- guage query Hl E R by replacing specific placeholder
We constructed a scale, complex, multi- task QA dataset, EngineMT-QQ. based on the N- CMAPSS(Arias Chao et al., 2021) aero engine dataset,
5

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
reflecting real-world engine operation and maintenance scenarios. The dataset construction process is detailed in Appendix A. EngineMT-QA comprises 11k QA pairs, designed to cover four key task categories: Understanding, Perception, Reasoning, and Decision-Making. Detailed statistical information can be found in Appendix B.
were trained on the EngineMT--Q dataset, with training conducted on the training subset and evaluation on the test subset. For a fair comparison, existing multimodal models like GPT-4o and Gemini, vision-text models like Instruct- Blip, MCAN-VQA. and CoCa, and time-series-text models like Time-LLM and Time were all adapted to use time- series encoders instead of their original image encoders. Specifically, the PatchTST time-series encoder, which is identical to the one used in ITFFrrmer replaced the image encoder in these vision-text models. This ensures that all models are evaluated under the same conditions, using time- series data in place of images. The training steps and epochs were consistent across all models, ensuring that the perfor- mance differences between ITFormmer and the comparison models are due to the architecture and not variations in training configurations.
The Understanding task requires the model to interpret the relationships among various sensors and derive the semantic implications of changes in sensor signals. The Perception task focuses on uncovering the underlying health state se- mantics of the signals, diagnosing faults, and identifying root causes. The Reasoning task challenges the model to infer degradation trends by analyzing semantic variations across 10 consecutive cycles and to predict both the proba- bility of future failures and the remaining useful life of the engine. Finally, the Decision-Making task involves making end-to-end operational and maintenance decisions, such as determining whether and when to repair potential compo- nents, based on the reasoning capabilities of the model.
The experimental results in Table 1 demonstrate that IT- Former achieves state-of-the-art performance across all four tasks in EngineMT-QA. The largest model, ITFormer- 7B, consistently achieves the best performance on every metric, including Rouge-L. and BLEU for open-ended tasks (Understanding: 58.04/38.23; Decision: 56.62/38.68), as well as Accuracy and F1 for classification tasks (Percep- tion: 65.07/68.36; Reasoning: 88.69/88.69), These results demonstrate that I'FFormer effectively handles both genera- tive and discriminative temporal-textual QA.
Evaluation methodologics are task-specific. For the Under- standing and Decision-Making tasks, the answers are open- ended and evaluated using BLBU(Painneni et al., 2002) and Rouge-L(Lin, 2004) to assess the quality of generated re- sponses. In contrast, the Perception and Reasoning tasks have fixed answers and are evaluated using Accuracy and Fl scores to measure model performance. As shown in Fig. 3, EngineMT-QQ unifies diverse tasks within complex temporal scenarios into a cohesive TimeSeries- QAA frame- work, more QA examples can be found in Appendix C. This dataset provides robust foundation for constructing and evaluating temporal-textual multimodal models capable of effectively handling intricate real-world scenarios.
Performance scales well with model size: ITFormer-3B ranks second across all tasks, while ITFormer-0.5B al- ready surpasses strong baselines such as InstructBlip, CoCa, Time-LLM, and Auto'Time. This confirms the scalability and efficiency of our design, which integrates time-series encoders with frozen LLMs through lightweight temporal- textual alignment modules.
5.2. Main Results
Among baselines, vision-language models like InstructBlip and MCAN-VOR perform reasonably well on generative tasks but exhibit less stability across classification tasks. Time-series-text models like Time-LLM and AutoTTmme un- derperform in Decision-making due to limited generative capacity. Multimodal APIs such as ChatGPT-44 and Gemini achieve low scores across all tasks (e.g., ChatGPT-44 Deci- sion BLEU: 3.30; Reasoning F1: 39.53), highlighting their lack of specialization for structured multivariate temporal signals.
In the main experiments, we use 4-layer, 8-head PatchTST (Nie et al., 2023) as the time-series encoder, with 10-minute (600 points) time-series segments of 33 channel engine signals. The patch length and stride are set to 60, and pretraining runs for 10 epochs. ITFormer is a 2-layer, 8- head model, integrated with the frozen qwen2. 5 (Yang et al., 2024) series LLMss The supervised fine-tuning (SFT) stage lasts 2 epochs, and all training is conducted on 4 NVIDIA H100 GPUs
For evaluation, we compare against multimodal APIs (ChatGPT-40 (OpenAl, 2024) and Gemini (Team et al., 2023), with prompt templates in Appendix D), time-series- text models (Time-LLM (Jin et al., 2024b) and AutoTime (Liu et al., 2024c), adapted to text space), and vision- text models (InstructBlip (Dai et al., 2023), MCAN-VQA (Zhou Yu et al., 2019), and CoCa (Yu et al., 2022), where the image encoder is replaced with the same PatchTST en- coder). In all comparison experiments, ITFommer models
The performance gains of ITFormer can be attributed to its dedicated components-TPp LIT, ITAA, and ThL-which enable hierarchical temporal encoding, modality alignment, and efficient signal-to- longuuu fusion. Ablation results show that TPE and ITA contribute the most individually, while combining all four components leads to the highest overall performance, validating their complementary nature.
Overall, ITFormer offers unified framework for complex
6

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
The degradation trajectory of the system
Temporal Understanding
Task 2: Temporal Perception
Catt Irpar Sinpl Cyck
SORa efeach
(Pormal artaut
Time Series Data
companert
change
dining lsuu operation given engine signal cycle affelencyyrreeee
Oo leng could you exglain whae the change in aititude Indicatos about the englne agnel the ynig for this singl
HPT antictencyyy di HP Flow
b:LPT afficency
ci afficency Fan flow
1
As Well, atttude dropping dwadity What's intereetingl la shat the handling the reduted airfow dorand realiy well, leeping its performance desdy wntttuttttttinng unnecesorry strain on the syttem.
c: Tow
0.2
Timee Toyell
Task 4: Temporal Dedision-Making
Task Temporal Reasoning
Owie
Fysl11
Cyste
bepmyyplttall popeettllleeeeeee 2
Opensional axtriictionn
apus Sturgp Cyds
Cyde
Giieen the time cycles reflect differect health states and performing temporal reasoring predict the health decline trand, what the probubility angine failume?
by arcelving the engine signal across
Uased an angine signal date collected over 10 cycles, what immediate actions shuld be address diserwed
A: Nicessory repairs or replacement of uffected HPT and HPC componentt must be made scon pomsible enaure that the engine continues to operate efficlestly
a:1%
5%%
<:58
30%-50%%
1/909-99%
Figure 3: Overview of the EngineMT-QA dataset. This dataset comprises 11k QA pairs across four tasks: Understanding, Percep- tion. Reasoning, and Decision- Mariii..MMakin.... Each task reflects specific operational requirements of engine maintenance, including sensor relationships, health state interpretation, trend prediction, and maintenance decision-making.
time-series QA, delivering robust performance with minimal parameter tningg and setting strong baseline for temporal- textual reasoning in real-world scenarios.
Artetry
5.3. Adaptive Study
To evaluate the capability of ITFormer in effectively bridg- ing diverse temporal encoders and LLMs, we conducted se- ries of experiments aiming to transform various time-series encoders and LLMs into a unified and robust multimodal framework. Specifically, we examined ITFFormers adapt- ability by integrating multiple LLMs (c.g.. Qwen, LLaMA (Touvron et al., 2023), and GLM (Du et al., 2022)) with different temporal encoders (e.g., PatchTST, Informer, and Crossformer). The pretraining and subsequent SFT settings of the Time-series Encoder remain consistent with those in the main experiment. The results (Fig. 4) demonstrated that I'Foormer consistently maintained stable performance across all combinations, achieving competitive results in terms of both average Accuracy and Rouge-L scores. Fur- thermore, notable trend was observed: as the scale of the LLM increased, the performance of the combined model improved correspondingly. This indicates that larger LLMs may provide ITFormer with a more favorable optimization landscape during the supervised fine-tuning stage, enhanc- ing its overall performance.
Figure 4: ITFFormer maintains robust performance across archi- tectures, with larger LLMs enhancing optimization and improving results.
strate that excessively short tokens fail to capture sufficient multimodal information, whereas overly long tokens result in sparse and less effective feature extraction. The results indicate that an LIT length of 25 tokens strikes an ideal balance, enabling efficient and robust temporal- -extuall mul- timodal integration.
0.05
5.4. Ablation Study
Figure 5: Impact of ITFormer layers and LIT Tength on model performance. A moderate number of ITFormer layers optimizes performance, while very short LIT lengths limit multimodal ex- traction, and overly long lengths lead to sparse information.
We conducted ablation studies to investigate the impact of
ITFormer layer numbers and LIT lengths on model perfor- mance. As shown in Fig. 5, 2-layer ITFormer achieves op- timal performance, balancing computational efficiency and accuracy, while additional layers yield diminishing returns under current data conditions, likely due to optimization challenges. Similarly, experiments on LIT lengths demon-
To evaluate the contribution of each component in ITFormer, we conduct an ablation study on four key modules: the In- struct Time Attention (ITA), Time Token Position Encoding

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
Table Performance table for the four tasks in EngineMT-QA. with second- bestt performances highlighted orange and red, respectively.
Understanding
Perception
Reasoning
Decision
Method
Method Type
Rouge-Lf
BLUE
Accuracyt
FIf
Accuracyt
FI1
Rouge-L1
BLUE
ChatGPT-4o
Multimodal API
15.23
7.69
52.14
46.91
29.32
39.53
9.19
3.30
Gemini
Multimodal API
21.81
6.43
52.11
40.01
22.04
38.42
8.80
2.90
InstructBlip
Vision-Tews
47.31
25.29
53.11
51.61
51.64
51.64
31.21
14.47
MCAN-YQA
Vision-Texh
48.52
26.59
59.79
54,91
54.78
54.78
31.90
14.92
CoCa
Usion-Text
46.21
25.12
52.21
50.11
50.21
50.21
33.92
13.68
Time-LLM
Sine-SSrics-Texx
29.40
18.60
46.05
46.97
47.54
47.54
28.10
11.51
AutoTTime
Time-Serics-Tes
27.24
17.99
43,45
46.47
49.96
49.97
23.55
9.80
ITFormer-0.5B
Proposed Method
49.36
28.05
63.95
54,39
80.18
80.09
31.23
15.26
ITFormer-3B
Proposed Method
54.37
33.44
64.07
58.36
83.30
83.22
44.67
24.37
ITFormer-7B
Proposed Method
58.04
38.23
65.07
68.36
88.69
88.69
56.62
38.68
(TPE), Time Token Language (TAL), and Learnable In- struct Tokens (LIT), shown in Table 2. Starting from the base ITFormer without any auxiliary modules (Row a), we observe gradual performance gains as modules are added individually. Introducing ITA (Row b) yields a slight im- provement in both accuracy and BLEU, indicating its role in guiding token-level alignment. TPE (Row c) brings a more significant boost, especially in BLEU, highlighting its effec- tiveness in modeling multi-scale temporal structure. TAL (Row d) offers minor gains, while LIT (Row e) noticeably improves accuracy, confirming the benefit of token-level lask guidance.
Table Performance of ITFFormer for Component Ablation on Proposed Components
Dense Components
Average
Main
ITA
TPE
TAL
LIT
Acc.
BLUE
(a)
49.82
16.41
(b)
50.94
16.99
54.16
17.92
(d)
V
52.71
13.86
(c)
53.69
15.51
(f)
60.29
21.58
(g)
62.17
22.02
(h)
/
68.21
22.74
When combining ITA and TPE (Row f), both metrics in- crease further, demonstrating their complementary effects in temporal-textual fusion. Adding TAL (Row g) on top of this pairing leads marked increase in accuracy (from 60.29% to 62.17%). suggesting that incorporating learned tempo- ral tokens enhances downstream interpretability. Finally, the full configuration (Row h) with all four components achieves the highest performance (68.21% % accuracy and 22.74 BLEU), validating the overall synergy ITFormer's modular design. These results confirm that each module contributes uniquely temporal-textual modeling, and that their integration yields substantial performance improve- ments. Notably, TPE and ITA contribute the largest individ- ual gains, forming the backbone of ITFormee's 's alignment capability.
replaced the ITA mechanism with traditional attentntto to compare inference efficiency. Second, examined the impact of varying channel numbers V and sequence lengths L. Third, we assessed the LIT design by varying input question lengths Lg.
As shown in Fig. 6, ITA significantly improves inference speed compared to traditional cross-attention, particularly as channel counts and sequence lengths increase (Figs. 6a and 6b). This demonstrates its scalability for processing temporal inputs with long sequences and multiple chan- nels. Moreover, the LIT design (Fig 6.c) achieves notable efficiency gains for long question sequences by reducing computational overhead. These results validate the effec- tiveness of ITA and LIT in accelerating inference while maintaining strong performance.
5.5. Efficiency Study
We conducted three efficiency ablation studies to analyze the computational advantages ITFommer's design. First, we
8

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
the Mibitsc
2:24
across diverse application areas.
2.20
21.0
1.75
6. Conclusion
123
130
1.00
In this work, we introduced EngineMT-QA, a large-scale, multi-task, temporal-textual QA dataset designed to address the challenges of Time-Series QA. By capturing intricate interactions between time-series signals and natural lan- guage, it establishes standardized benchmark for evaluat- ing multimodal models in wall-world scenarios. Building on this resource, we proposed I'FFormer, a novel frame- work that bridges time-series encoders and frozen LLMs for efficient and robust multimodal modeling. Key innova- tions-Time Token Position Encoding, Learnable Instruct Tokens, Instruct Time Attention, and Time Token as Lan- guage-enable ITFormer achieve state-of-the-ar perfor- mance across diverse QA tasks while maintaining computa- tional efficiency.
MecenCC Lengh
(LayyLeagh
Figure 6: Computational Efficiency of FTrmerr (a) and (b) show inference speed across channel counts and sequence lengths, while (c) analyzes input question length's s impact on the LIT mechanism, highlighting efficiency gains.
Table 3:
Performance of ITFFommer and baselines on TemeSerie- sExam (Accuracy). Orange and Red indicate the best and second- best results in each column.
Model
Patter
Anomaly
Noise
Similarity
Causality
GPT-4o (image)
0.82
0.80
0.90
0.87
0.68
GPT-4o (text)
0.81
0.75
0.78
0.80
0.28
Gemini-Pro (image)
0.81
0.69
0.83
0.80
0.48
Gemini Pro (text)
0.82
0.72
0.90
0.80
0.68
Extensive experiments demonstrate ITFormer's 's scalability and effectiveness in integrating time-series signals with nat- ural language, significantly outperforming strong baselines. By establishing a new paradigm for temporal-textual mul- timodal interaction, this work lays the foundation for fu- ture research in Time-Serie QA. Moving forward, we envi- sion advancements in model interpretability, generalization across domains, and adaptation to weal-world irregular time- series patterns. The release of EngineM'T-QA and ITFForme serves as a foundation for the advancement of intelligent, adaptable, and efficient multimodal AI in time-series inter- action environments.
MCAN-VQA
0.81
0.79
0.94
0,90
0.73
ITFoomerr
0.83
0.84
0.95
0,92
0.79
ITFFommer (Tramsferred)
0.86
0.89
0.98
0.94
0.83
5.6. Generalization to Domain-Agnostic Tasks
To assess ITFormer's generalization capability beyond the aero-engine domain, we evaluate it on the aggnoi-a TimeSeriesExam benchmark (Cai et al., 2024), which con- sists of five canonical tasks: Pattern Recognition, Anomaly Detection, Noise Understanding, Similarity Analysis, and Causality Analysis.
Acknowledgements
Direct Generalization Evaluation. To assess the model's robustness, we first fine-tune ITFORMER directly on the TIMESERIESEXAM dataset. Even without pretraining on domain-specific data, ITFORMER achieves strong re- sults, including an accuracy of 0.79 Causality Analysis- outperforming multimodal baselines such as GPT-4o, GEMINI-PRO, and MCAN-VQA. This demonstrates its capacity to generalize temporal reasoning capabilities to novel benchmarks.
This work was partially supported by Grant of National Natural Science Foundation (Grant No. 62371297).
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Transfer Learning. When pre-trained on the large-scale ENGINEMT-ON QA dataset and subsequently fine-tuned on TIMESERIESEEXAM ITFORMER yields further improve- ments, reaching 0.86 in Pattern Recognition and 0.89 in Anomaly Detection. These results suggest that ENGINEMT- QA captures transferable
References
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
The strong performance of ITFormer on an unrelated bench- mark highlights its robust temporal-textuel reasoning capa- bilities. More importantly, it validates EngineMT-QA. as a pretraining resource that benefits time-series understanding
Arias Chao, M., Kulkarni, C., Goebel, K., and Fink, O. Aircraft Engine fun-to-Failure Dataset under Real Flight Conditions for Prognostics and Diagnostics. 6(1):5, 2021. ISSN 2306-5729.

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Bi, K., Xie, L.,, Zhang, H., Chen, X., Gu, X., and Tian, Q- Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970):533-533, 2023.
Jin, M., Koh, H. Y., Wen, Q. Zambon, D., Alippi, C., Webb, G. I., King, I., and Pan, S. A survey on graph neural networks for time series: Forecasting, classification, im- putation, and anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Cai, Y., Zhang. C., Liu, Y., He, J., Wang, Z., Liu, Y., Long, G., Wang. Y., Lin, H., and Zhou, J. Timeseriesexam: A time series understanding exam for large language models. arXiv preprint arXiv:2410.14752, 2024. URL https: / arxiv. org/ abs/ 2410..1. 1..40.147700 14752.
Jin, M., Wang, S., Ma, L.,, Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y., Liang, Y., Li, Y.-F,, Pan, S., et al. Time-llm: Time series forecasting by reprogramming large language models. In International Conference on Learning Repre- sentations, 2024b.
Clements, M. and Mizon, G. E. Empirical analysis of macroeconomic time series: Var and structural models. European Economic Review, 35(4):887-917, 1991.
Jin, M., Zhang, Y., Chen, W., Zhang, K., Liang. Y., Yang, B., Wang, J., Pan, S., and Wen, Q. Position: What can large language models tell us about time series analysis. In FInternational Conference on Machine Learning, 2024c.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint arXin:2305.06500, 2, 2023.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman pre-training for unified vision- languae understanding and generation. In International confer- ence on machine learning, pp. 1288--12900. PMLR, 2022.
Darban, Z. Z., Webb, G. I., Pan, S., Aggarwal, C. C., and Salehi, M. CARLA: Self-supervised contrastive represen- tation learning for time series anomaly detection. Pattern Recognition, 157:1 10874, 2025.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Pro- ceedings of the ACL-04 workshop, volume 8, pp. 74-81, 2004.
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, M., Parikh, D., and Batra, D. Visual dialog. Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 226.335,11 2017.
Liu, H., Xu, S., Zhao, Z.,, Kong, L.., Kamarthi, H., Sasanur, A. B., Sharma, M., Cui, J., Wen, Q., Zhang, C., al. A new multi-domain multimodal dataset for time series analysis. In Advances in Neural Information Processing Systems, 2024a.
Dennis, D., Acar, D. A. E.,, Mandikal, V., Sadasivan, V. S., Saligrama, V., Simhadri, H. V., and Jain, P. Shallow rnn: accurate time-series classification on resource constrained devices. Advances in neural information processing sys- tems, 32, 2019.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. In International Conference on Learning Representations, 2024b.
Du, Z. Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, Glm: General language model pretraining with autoregressive blank infilling and task-specific fine- tuning. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (ACL), 2022.
Liu, Y., Qin, G., Huang, X., Wang, J., and Long, M. Au- totimes: Autoregressive time series forecasters via large language models. In Advances in Neural Information Processing Systems, 2024c.
Gao, S., Koker, T., Queen, O., Hartvigsen, T., Tsiligkaridis, T., and Zimik, M. Units: unified multi-task time se- ries model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., and Long, M. Timer:
Generative pre-trained transformers are large time series models. In International Conference on Machine Learning, 2024d.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski, A. MOMENT: family of open time- series foundation models. In International Conference on Machine Learning, 2024.
Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pre- training task-agnostic visiolinguistic representations for vision-and-languual tasks. Advances in neural informa- tion processing systems, 32, 2019.
Gui, H., Li, X., and Chen, X. Vector quantization pre- training for eeg time series with random projection and phase alignment. In International Conference on Machine Learning, pp. 16731-16755 PMLR, 2024.
Lu, M. Y., Chen, B., Williamson. D. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. A multimodal generative ai copilot for human pathology. Nature, 634(8033):466 473, 2024.
Hamilton, J. D. Time series analysis. Princeton university press, 2020.
10

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Nie, Y., H. Nguyen, N., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Learning Representations. 2023.
Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Chin- tala, S. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Vaswani, A. Attention all you need. Advances in Neural Information Processing Systems, 2017.
OpenAI. Gpt-4 technical report, 2024. URL https:// openai www.///eeeacc///cccocceooooo gpt-4.
Wang, C., Qi, Q. Wang, J., Sun, H., Zhuang, Z., Wu, J., Zhang, L., and Liao, J. Chattime: A unified multimodal time series foundation model bridging numerical and textual data. arXiv preprint arXiv:2412.1 2024a.
Oreshkin, B. N.. Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. International Conference on Learning Representations, 2022.
Wang, X., Feng, M., Qiu, J., Gu, J., and Zhao, J. From news to forecast: Integrating event analysis in Ilm-based time series forecasting with reflection. arXiv preprint arXiv:2409. 17515, 2024b.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Associ- ation Computational Linguistics (ACL), pp. 311-318. ACL, 2002.
Wang, Y., Han, Z., Zhao, Y., Wu, H., Tan, H.-J., Zhang, Y., and Li, Y. Establishment of super sonic inlet flow pattern monitoring system: A workflow. Aerospace Science and Technology, 120: 107297 January 2022. ISSN 1270-9638.
Rajpurkar, P. Squad: 100,000+ questions for machine com- prehension of text. arXiv preprint arXiv: 1Xiv:6606.000 1606.05200 2016.
Wang, Y., Shen, L., Zhang, Y., Li, Y.. Zhang, R., and Yang, Y. Self-supervised health representation decomposition based on contrast learning. Reliability Engineering & System Safety, 239:109455, 2023.
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E.,, Yang, M.-H., and Khan, F. S. GLAMM: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009- 13018, 2024.
Wang. Y., Li, Y., Zhang, Y., Lei, J., Yu, Y., Zhang, T., Yang, Y., and Zhao, H. Incorporating prior knowledge into self- supervised representation learning for long PHM signal. Reliability Engineering & System Safety, 241:109602,, January 2024c. ISSN 09518320.
See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generater networks, arXiv preprint ar arXiv: /704 043604488, 2017.
Wang. Y., Lei, P., Wang, X., Jiang, L., Xuan, L., Cheng, W., Zhao, H., and Li, Y. Leveraging large self-supervised time-series models for transferable diagnosis in cross- aircraft type bleed air system. Advanced Engineering Informatics, 65:103275 2025a. ISSN 1474-0346. doi: https://doi.org/100 a00..000.......22.........
Shumway, R. H., Stoffer, D. S., Shumway, R. H., and Stof- fer, D. S. Arima models. Time series analysis and its applications: with R examples, pp. 75 163, 2017.
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Generative mul- timodal models are in-context Iearners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398-14400 2024.
Wang. Y., Zhao, H., Cheng, W., Zhang, Y., Jia, L., and Li, Y. Data-driven dynamic health index construction for diagnosis and prognosis of engine bleed air system. Aerospace Systems, 8(1):149-161, 2025b.
Tan, H. and Bansal, M. Lxmert: Learning cross-modality en- coder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), 2019.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D.,, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Wu, H., Zhou, H., Long, M., and Wang. J. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence, 5(6):002-6) 611, 2023.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11185 2023.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D.,, Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
K. Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y. Su, Y. Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv preprint 2rX1v:2412.15115. 2024.
Yi, K., Wang, Y., Ren, K., and Li, D. Learning topology- agnostic eeg representations with gcometry-awan model- ing. Advances in Neural Information Processing Systems, 36, 2024.
Yu, J. Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. Transactions on Machine Learn ing Research, 2022. ISSN 2835-8856.
Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., and Xu, B. TS2Vec: Towards universal representation of time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Zhang, Y. and Yan, Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.
Zhang, Y., Long, M., Chen, K., Xing, L., Jin, R., Jordan, M. I., and Wang. J. Skilful nowcasting of extreme pre- cipitation with NowcastNet. Nature, 562-7920 2023.
Zhou, H., Zhang. S., Peng, J., Zhang. S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Zhou, T., Ma, Z., Wen, Q. Wang, X., Sun, L., and Jin, R. FEDformer: Frequency enhanced decomposed trans- former for long-term series forecasting. In International Conference on Machine Learning, 2022.
Zhou Y., Y., Yuhao Cui, D. T., and Tian, Q. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6281-6290, 2019.
12

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
A. Construction Flow for the Dataset.
Engine Multi-Modal Time-Series Dataset
Definition of Basic Flight States
LIOE QA Pairs
Tisks: Temporal Understanding, Perception, Ressoning and Makking
Engineering Experience
ENGINE
Principles to Guide Daraset Generation
Flight Speed: speed:
Create the Inítial Dataset
Flight Time: 1-3h, 5-Th;
Components: FAN.
Analyze Flight Data
Describe Flight Data
Polishing L.M
"Altitude"; "high",
Every component is functioning with remarkable efficiency high altitude and speed.
"Speed": "high",
"Health "healtthstttee:: "health"
"Sensor trend": "steady"
Temporal Perception
Temporal Understanding
Sensor Stater
Perception
Single-Cyele Flight Data
Fault Location
Physical Information
Model
Description
Semantic
Health State
Components Inspection)
Expert Knowledge
Sensors Trends
Health
HPC (Henlth)
Amalyze external factors
Decreasing very rapidly.
Satrt Degeneration
LPC (Health)
Efficiency
Decreasing muderately)
Abnormal Degeneration
FAN [Fultt
Operating
Mointain stability.
Temporal Reasoning Multi-Cycle Flight
Temporal Decision-Makkin
Multi-cylle Monitoring
knowledga
Prior
Engine Maintenance Manual
Maintenance Advice
Predictive Maintenance
XN
Maintenance Plan
Pre-fallure Repair
Petential
Risk Assessment
Degradation Trend
Change Flight Plan
Failure Probability
Degoneration Trajectory
Continue Flight
Semantic Description
Pilot Operating
Procedures
Emergency Landing
Expert Inspection
Books and Reference Manuals
Verification of Knowledge
Lidc-Dhoigg cenmecd aginoe Businos Maiis 2006.
Conforms Physics Model
Right!
Dueittoadd snerat mene digrostoo)) dj
Aurte Jxchon Fincter Mt i Peetctive ratmourze
Fegine Fagine threst decrusdig clasbing stage.
Kasteaza, 2003
Schwanc Sky Rand stanertu runad: Openaion diars. repii, placs swrrd regin/MM] Sdlwsrrr
Wrong!
Figure Dataset Construction Flow:The multi-mode modal time-series dataset for aircraft engines is constructed from NASA's N-MAPSS dataset, integrating raw flight data analysis, LLM-based refinement, and expert validation.
13

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
This appendix details the multi-modal time-series dataset for aircraft engines, covering its generation process and key stages. The framework includes: initial dataset creation, design of question-answering tasks in Fig. 7(including temporal understanding, perception, reasoning, and decision-making and expert validation. The dataset is based on NASA's N-MAPSS dataset, which simulates engine performance under real-world flight conditions. The initial dataset construction involves analyzing raw flight data to represent key parameters (e.g., altitude, engine component status) and refining descriptions using LLMs (e.g., DeepSeek, ChatGPT) to ensure clarity and diversity.
The construction of Q&A pairs in the temporal understanding is guided by physics-based models and expert knowledge. The process involves transforming sensor data trends into Q&A pairs that reflect the engine's internal state. Key steps include: (1) identifying and semantically describing trends, and (2) interpreting their significance using domain expertise. Expert-designed Q&A pairs link sensor trends to internal engine conditions, ensuring alignment with field-specific principles and incorporating operational and diagnostic insights.
In the temporal perception, Q&A pairs focus on identifying abnormal fault patterns in critical engine components (HPC: High-Pressure Compressor, LPC: Low-Pressure Compressor, HPT: ligh-Pressure Turbine, LPT: Low-Pressure Turbine, Fan: Engine Fan) using single-flight data. The N-MAPSS dataset provides ten fault conditions related flow and efficiency, along with engine health labels. Single-cycle sensor trends are analyzed describe signal characteristics and their relationships with engine subsystems. Expert insights link sensor variations to fault patterss, enabling the construction of Q&A pairs that assess component or system health, integrating domain knowledge for accurate fault diagnosis.
For the temporal reasoning, Q&A pairs are built using multi-cycle data to evaluate long-term component health. Gradual wear and performance degradation are identified through the dataset's ' health labels. Q&A pairs in this module focus on constructing probability intervals for future failures and describing system health, combining predictive insights with descriptive evaluations based on multi-cycle analysis.
In the temporal decision-making making module, Q&A pairs are derived from degradation trends to generate maintenance recom- mendations. These are based on engine manuals and operating procedures, enabling predictive scheduling to prevent failures and extend component lifespan. Each module's Q&A construction ensures that data are enriched with domain-specifi knowledge, supporting accurate diagnostics and predictive maintenance.
Finally, the dataset undergoes expert inspection to verify its accuracy and ensure that it conforms to physical models governing engine performance. The data is verified with theoretical models to confirm that the observed trends are physically plausible. Discrepancies are flagged and corrected maintain data integrity, ensuring the dataset remains realistic and reliable.
B. Statistical indicators of the Dataset.
This appendix provide the structure of the dataset developed for this study, which focus on the key indicators of the training and testing datasets. The tasks are designed to evaluate various aspects of temporal tasks, including understanding, perception, reasoning, and docision-making Among these, the understanding and eecisionnmaking tasks are framed as open-ended questions, while the perception and reasoning tasks presented with multiple-choice options ensure precise identification of faulty components and probability of failure.The statistical indicators of the dataset are presented in the Fig.8
The health status of each component in both the training and testing datasets is provided, revealing that the probability distributions of component failures are relatively consistent between the two datasets. Among these, HPT exhibits the lowest failure probability. Additionally, the predicted failure probability intervals for the overall components are presented, with most intervals concentrated between 10% and 70%. In practice, is challenging to make maintenance decisions when the failure probability is below 10%. This indicates that the majority of the data originates from normal operating conditions, which is sufficient to support predictive maintenance strategies.
14

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Statistical indicators of the Dataset
Distribution of Training and Testing Dataset
The Health State of Engine Components the Training Dataset
5580
50300
Training Teading
Healh
5000
23300
Samples
20200
17240
2030
1500
1400
10200
10922
1000
971
1979
1868
1193
CTG3
5200
3184
Undantanding
Precaption
Reasoning
Decislon-ttaking
fan
LPC
HPC Component
HPT
LPT
Tasks
The Health State of Engine onentss in the Testing Dataset
Probability of Failure of Training and Testing Dataset
Health
1710
1950
1500
1281
1533
1402
1480
1917
1250
1012
Samples
1930
ST5
1300
12A
800
D13
820
852
DOS
4M4
63
HPC Component
HPT
1%--0%
18%-30%
30%--88%
50%--78%
70%-100%%
Probability Fallure
Figure& 8: Statistical Indicators of the Dataset: Key statistical characteristics of the dataset, including distributions of sensor readings, fault occurrences, engine health metries. These indicators provide insights into data variability, trends, and coverage, ensuring a comprehensive representation of engine performance and fault conditions.
C. Dataset Example.
The temporal understanding module focuses on analyzing trends sensor data in Fig. 9, such as fuel flow, fan speed, and LPC. Semantic features such as "rapid increase," "moderate increase," and "stable" are used describe sensor trends, thereby enhancing the model's ability to interpret engine operating conditions. For example, "a decrease in temperature the LPT outlet" indicates a steady descent in altitude, with the engine effectively responding to reduced airflow demand, maintaining stable performance without imposing any unnecessary stress on the system. Through these semantic features, the model can gain a deeper understanding of the engine's 's operational status, providing more precise recommendations for maintenance and optimization.
In Temporal Perception, the objective is to assess the engine's 's health state and identify faulty components based on sensor data from single flight cycle. This task consists of three types of questions: the first type involves identifying faulty components, the second type evaluates the cause of engine failure, and the third type focuses on assessing the health status of specific components. For each component in the training dataset, there are more healthy samples compared to defects. This distribution ratio of health and failure effectively ensures that the model can learn the health representation of components from positive samples.
Temporal Reasoning tasks are based on multi-cycle data and are primarily focused on predicting future engine states and assessing associated risks. The problems in reasoning are mainly divided into two categories: the first involves providing an estimated probability of failure through closed options, while the second requires offering semantic description of the engine's health state within that period.
In Temporal Docision-Making tasks, it is designed to make critical decisions based on reasoning results. This part focuses on open-en open-ended questions, where the system generates maintenance suggestions aimed at preventing catastrophic failures and extending the service life of components through degradation trend analysis. The questions in this part include choosing to replace engine components.
15

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
Cyde
Temporal Understanding
Cycl
Cycle
What does the observed change in the Total Temperature at the LPT outlet indicate within the provided engine signal over single cycle?
www. Cycl 10
Temporal Reasoning
Given the time series signal, by erceiving the engine signal across 10 cycles to reflect different health states and performing temporal reasoning to predict the health decline trend, what is the probability range of engine failure?
Well, the altitude is dropping steadily. What's interesting that the engine handling the reduced airflow demand really well, keeping its performance steady without putting any unnecessary strain the system.
1% -00%
B:10%-30%
3:%%-50%
D:50%-70%
E:70%-100%
70%-100%
Temporal Perception
Given the time series signal, by perceiving the engine signal across 10 cycles to reflect different health states and performingtemporal reasoning to predict the health decline trend, what the qualitative condition the engine?
Was the timing issue in the operation of the given engine signal in one cycle caused by efficiency or flow problems?
HPT efficiency Fault Fan efficiency Fault E:LPT flow Fault
B: LPT efficiency Fault HPT flow Fault Fan flow Fault
Good Condition
Moderate Condition
Poor Condition
Condition
E: Extremely Poor Condition
B: Moderate Condition
A: HPT e efficiency Fault
What is the potential health state of the given engine signal in one cycle?
Cycl 1
froherng maa
Cycle 2
A: Operating Normally
Cycle
B: Abnormal Degradation
C: Imminent Failure
Hoduraa www. talpy
Cyde 10
A: Operating Normally
Temporal Decision-Making
In light of the engine signal data collected across 10 cycles, what specific actions should be undertaken to address the observed issues?
What is the health status of HPT of the given engine signal in one cycle?
Health
Fault
Necessary repairs replacement of affected Fan and LPC component must be made as soon as possible to ensure that the engine continues to operate safely and efficiently.
A: Health
Figure 9: Data Example: Example of sensor trend analysis and semantic feature extraction in the dataset. Temporal understanding captures variations in engine parameters (e.g., fuel flow, fan speed) with deseriptors like "rapid increase" and "stable" enhance interpretability. Temporal perception identifies faulty components and assesses engine health hased on single-flight data, while temporal reasoning predicts failure probabilities using multi-cycle trends. Temporal decision-making generates maintenance recommendations based on degradation analysis, ensuring proactive fault prevention and system optimization.
16

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
D. Prompt Templates
To establish a baseline for comparison, we conducted experiments on our dataset using the APIs of two state-of-the-art multi-modal models: ChatGPT-4o and Gemini.
Each cycle in dataset contains time-series signals with 600 time steps and 33 feature dimensions, resulting in total of 600 x 33 19, 800 numerical values per cycle. Directly feeding these raw numerical values as textual input would exceed token limitations, making it impractical for the models to process effectively. To overcome this challenge, we converted each cycle's ' signal data into a visual representation by plotting the time-series signals as curve graphs.
Fig. 10 presents the standardized template used for visualizing input signal plots fed into the multimodal API.
D.I. Prompt for Temporal Understanding
Role Setting
You are an expert aerospace analysis system specializing engine time-series signal interpretation. Your insights connect signal characteristics subsystem behaviors, explaining physical mechanisms.
Core Instructions When provided with single operational eycle's time-series data and subsystem health questions:
1. Signal Identification: Recognize key parameters (altitude, throttle, temperature, pressure).
2. Feature Extraction: Analyze trends (stable/oscillatory, fast/slow) using slope, amplitude, periodicity.
3. Subsystem Mapping: Link signals to components (compressors, combustion chambers, turbines).
4. Mechanism Explanation: Apply engine principles (Bernoulli's equation, thermodynamics) to interpret pressure,
speed, and temperature changes.
Output Output Specifications
Concise & Causal: Use linking words (due to, leading to, triggering).
Engineer-Friendly: Avoid academic jargon.
Probabilistic Language: Use "may," "potentially," or "notably" for long-term effects.
Example Reinforcement
Q: "Hey, could you explain what the change altitude means for the engine signal we're analyzing during this single cycle? [A corresponding image]"
A: "The altitude is dropping really fast. A quick descent like this actually lessens the workload on the engine, but it might also demand rapid adjustments to keep things stable and efficient. This could lead to more wear and tear on the engine's components over time."
Q: "What does the change in altitude represent in the given engine signal during one cycle? [A corresponding image]" A: "The altitude appears be rising rapidly. This rapid ascent could potentially increase the engine's workload, resulting in higher fuel consumption and elevated compressor inlet temperatures. Such conditions might place additional stress on the engine components over time."
Q: "What does the change in Throttle Resolver Angle represent in the given engine signal over one cycle?"[A corresponding image]
A: "Thh Throttle Resolver Angle appears to be increasing steadily. This likely represents a controlled and gradual increase in engine power, which may help maintain a balance between thrust generation and fuel consumption efficiency." Input
[Question] [A corresponding image]
17

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
at
T
P bypaes
P_NRC_ pat__taaii
zaeq speed
P_ Lnut
HPT aPe_ir
fon, stell mergin
mach
T_Hrc_uut
P.
P.burer
Feel flow
fan, flow
Hey, bredt
USC stall mungin
tnrsttle
fow
marqin 1850
speed
HPC
103
200
XD Index
400
500
600
T_LPC_out
P_bypess
PJJIF_DOU__AMNS
coee opeddpeed
P_IPT_out
IPT_D
Thn, on_stell_ms margin
mach
T_MPC_our
P
P.burer
Hlow
HPT
LPC_stal
trrotlle
-PC_stul_
speed
HiC
fow
2no
M
183
200
500
(b)
400
500
600
Index
T.C.C DM
2 tyyesss
PJ DUt Mtale
COIN speed
P. out
1PT. sreetl
lan stal mergn
mach
HPC. out
2
D.
Tusi
Fan.
HPT brea22
UPC. sta marqin
Brotlle
3fannout
P.LFT_OUR
LLC_
out
T_IFT_out
HPC,in
flow, rito,
1.
-05
-1.0
-1.
-2.0
(c)
Figure 10: Input signal template: Example plots for three different cycles (a, b, c).
18

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
D.2. Prompt for Temporal Perception
Role Setting
You intelligent diagnostic system specializing in aircraft engine health monitoring. Your expertise lies in analyzing single-cycle series data to assess subsystem health, detect faults, and identify degradation indicators for preventive maintenance.
Core Instructions
When provided with single operational cycle's time-series data and subsystem health questions:
1. Parameter Evaluation
Identify critical parameters (e.g., HPC pressure ratio, LPT temperature deviation).
Quantify deviations from nominal operational baselines.
2. Fault Detection
Cross-reference parameter combinations against fault matrices (e.g., compressor stall criteria, bearing vibration thresholds).
Detect abnormal signatures: sustained oscillations, non-recoverable pressure drops, thermal overshoots
3. Root Cause Analysis
Map anomalies to failure modes using component fault trees (e.g., HPC degradation -> reduced surge margin).
Output Specifications
Binary responses only (strictly "a" or "b" " in quotes).
No explanatory text unless explicitly requested.
Example Reinforcement
Q: "What is the health status of HPC of the given engine signal in cycle? [A corresponding image]
a: Health b: Fault"
A: "a"
Q: "What is the health status of LPC of the given engine signal in eycle? [A corresponding image] Health b: Fault"
A: "b"
Input
[Question] [A corresponding image]
D.3. Prompt for Temporal Reasoning
Role Setting
You are a multi-cycle aircraft engine health prognosties system specializing in cross-cycle time-series analysis. By integrating historical operational data, you predict remaining useful life (RUL), quantify health degradation rates, and generate clak-class maintenance plans.
Core Instructions
When provided with multi-cycle time-series data and trend prediction queries:
1. Data Aggregation
19

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
Align key parameters across N cycles (e.g., EGT escalation rate, fuel flow drift).
Construct performance degradation trajectory matrices.
2.
Trend Analysis
Detect accelerated degradation inflection points (e.g., sign change in turbine efficiency second derivative).
Calculate RUL confidence intervals (using Weibull/Poisson process models).
3.
Predictive Modeling
Match failure mode signatures.
4. Risk Stratification
Classify condition grades.
Trigger airworthiness alerts.
Output Specifications
Strict five-option responses (quoted "a" to "e").
No explanatory text unless explicitly requested.
Example Reinforcement
"Given the time series signal, by perceiving the engine signal across 10 cycles to reflect different health states and performing temporal inference to predict the health decline trend, what is the qualitative condition of the engine? [Ten corresponding images]
Q:
a: Good Condition b: Moderate Condition c: Poor Condition d: Bad Condition e: Extremely Poor Condition"
A: "d"
Q: "Given the time series signal, by perceiving the engine signal across 10 cycles to reflect different health states and performing temporal inference to predict the health decline trend, what the qualitative condition of the engine? [Ten corresponding images]
a: Good Condition b: Moderate Condition c: Poor Condition d: Bad Condition e: Extremely Poor Condition"
A: "a"
Input
[Question] [Ten corresponding images]
D.4. Prompt for Temporal Decision-Making
Role Setting
You are a multi-cycle engine data-driven decision system that translates health predictions into actionable maintenance strategies and operational adjustments.
Core Instructions
When processing multi-cycle data for operational decisions:
1. Priority Assessment
Flag critical risks (e.g., bearing vibration exceeding limits for 3 consecutive cycles).
Calculate component degradation impact on overall performance.
20

---

ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Seale Multitask Dataset
2. Action Planning
Create three-tier responses: immediate action (within 24h) / scheduled maintenance (7 days) continuous monitoring.
Optimize cost-efficiency (e.g.. prioritize components with >30% failure probability).
3. Resource Allocation
Forecast spare parts demand based on remaining life distribution.
Schedule downtime during off- aff-peak operational periods.
Output Specifications
Use maintenance crew colloquial language.
Include specific action verbs (repllace/isspecdrrsstti)))
Prioritize recommendations with urgent actions first.
Never mention regulatory standards.
Example Reinforcement
"Given the engine signal data collected across 10 cycles, what specific actions should we take to address any potential issues? [Ten corresponding images]"
Q:
A: "It is critical promptly repair or replace any faulty components like HPT and HPC and conduct a comprehensive assessment of the engine's overall health status to ensure optimal performance and safety."
"Hey, what's the plan for dealing with this engine signal we' 'vee got from 10 cycles?" [Ten corresponding images] A: "We need to jump on repairs or swap out any messed-up parts like LPC and Fan right away to avoid any further issues. Time is of the essence here!"
Q:
Input
[Question] [Ten corresponding images]
21


Reviews:
[{"review_id": "8i8uNBpRbM", "reviewer": "Reviewer_JQvF", "summary": "The paper addresses the problem of multimodal time series modelling. The main motivation is to enrich time series with natural language to enrich the time series with textual information. For this reason, a benchmark for answering time series questions is proposed, focusing on real-world aircraft engine operation and maintenance scenarios. In addition, the paper presents an approach to combine the two modalities based on pre-trained and frozen LLMs. For this purpose, a speech and time series encoder is introduced and later combined with a fusion encoder. A decoder finally outputs the response to the input time series and textual description.  The evaluation is standard and shows that the proposed approach leads to promising results compared to the prior work.\n\n## update after rebuttal\n\nAfter reading the reviews and the author's feedback, I am increasing my score to weak acceptance. The rebuttal did a very good job of addressing my concerns as well as the open points of the other reviews.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. **Generalization and Benchmark Scope:**\nThank you for your valuable comments. We conducted additional experiments to evaluate generalization beyond the aero-engine domain:\n1. **ITFormer generalizes across domains**, achieving top performance on the domain-agnostic TimeSeriesExam benchmark.\n\n2. **EngineMT-QA reflects core challenges of temporal-text interaction**, as shown by consistent gains in cross-domain transfer learning.\n\n(Details provided in our full response to Reviewer zRtW.)\n\n# 2. **Ablation studies when outputting numerical values**\nWe thank the reviewer for highlighting this important point. While our original setting unified all outputs in natural language form, we agree that evaluating the model’s ability to express **numerical values directly** is crucial for practical multi-modal time-series applications, where numerical precision is often required for decision-making.\n\nTo address this, we introduced an ablation experiment with a **mixed-output setting**, where the model is required to generate both **structured numerical strings** and natural language in different tasks. Specifically:\n\n- **Task 2 (Perception)**: The model predicts the **health index** of a component (e.g., Component A: `0.87`), followed by a decision of whether the health index exceeds a threshold, resulting in a textual judgment (e.g., \"Fault\" or \"No Fault\").\n- **Task 3 (Reasoning)**: The model predicts the **Remaining Useful Life (RUL)** of a system, and based on the RUL, determines a specific operational range (e.g., `30%-50%` ), followed by an accuracy measure to evaluate if the predicted range is correct.\n\nFor these tasks, the output is structured as numerical strings, such as: `\"Component A: 0.87, RUL: 43\"`. Both the **structure** and **numeric accuracy** are crucial for evaluation.\n\n| ID | Main | ITA | TPE | TAL | LIT | Output Type | Accuracy | BLEU |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| (**h'**) | ✅ | ✅ | ✅ | ✅ | ✅ | Mixed (Text + Num) | **68.12** | **56.74** |\n| (h) | ✅ | ✅ | ✅ | ✅ | ✅ | Text | **73.29** | **60.27** |\n\nAs shown in Row **(h’)** of the updated ablation table, the mixed-output setting results in a slight performance drop compared to the full natural language baseline (Row h):\n\n- **Accuracy**: 73.29 → **68.12**\n- **BLEU**: 60.27 → **56.74**\n\nWe believe this performance difference is due to several reasons:\n\n- **Format sensitivity**: Numerical outputs are evaluated with strict string matching—minor deviations (e.g., `\"0.87\"` vs. `\"0.870\"`) can cause hard penalties in accuracy and BLEU.\n- **Dual-mode decoding complexity**: Generating both text and structured numbers increases the output space and decoding difficulty, requiring the model to switch between narrative and precise formats.\n- **Weaker supervision for numbers**: Unlike text, numerical outputs lack rich contextual anchors and often require implicit regression, making them harder to learn accurately.\n\nMoving forward, we plan to explore further the combination of structured text generation and direct numerical regression, as well as develop a **standardized dual-modality evaluation benchmark** that covers both output types—textual and numerical.\n\n\n\n# 3. **On Model Efficiency and Trainable Parameters:**\n\nThank you for your suggestion. We will update Table 1 in the revised version to include the number of parameters for each model.\nDespite using only 30.94M trainable parameters across all variants, ITFormer consistently outperforms larger models like AutoTime (205M) and InstructBlip (190M), achieving superior accuracy and BLEU scores.\nhis demonstrates that ITFormer achieves an excellent balance between performance and efficiency—detailed comparisons are visualized in https://imgur.com/a/Y53oD5B.\n\nHere is the updated parameter comparison:\n\n| **Model**         | **Total Parameters** | **Trainable Parameters** |\n| ----------------- | -------------------- | ------------------------ |\n| **InstructBlip**   | 7.19B                   | 190M                     |\n| **MCAN-VQA**       | 7.04B                   | 35.2M                    |\n| **CoCa**           | 1.08B                   | 78.74M                   |\n| **Time-LLM**       | 7.09B                   | 86.54M                   |\n| **AutoTime**       | 7.21B                   | 205M                     |\n| **ITFormer-0.5B**  | 0.53B                 | 30.94M                   |\n| **ITFormer-7B**    | 7.03B                   | 30.94M                   |\n| **ITFormer-14B**   | 14.03B                  | 30.94M                   |\n\n\nWe hope this clarifies the efficiency of **ITFormer** in terms of both performance and computational cost.\n# 4. **On Methodological Novelty**\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer ZFEy. Kindly refer to it for further details.\n# 5. **On Constructing More General Time-Series QA Datasets**\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer n3vv", "claims_and_evidence": "The problem has often been approached using ML methods. The proposed methodology is well suited to the problem. Moreover, the evaluation is done on standard datasets.", "methods_and_evaluation_criteria": "The evaluation is sufficient. However, a general QA dataset would be relevant to the paper.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The proposed benchmark is well designed.", "relation_to_broader_scientific_literature": "The new benchmark  is novel compared to the current literature.", "essential_references_not_discussed": "The prior discussion is fine.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to follow.\n\n- The proposed benchmarks are a valid contribution. \n\n- The proposed benchmark is valuable for application specific scenarios.\n\nPaper weaknesses:\n\n- The benchmark focuses on the operation and maintenance of aero engines. It does not explore the generalisation of the questions to a broader framework. It is well thought out and designed, but it doesn't necessarily advance the machine learning community as it is purely application specific. This is a limitation for submission to ICML, but not in general. It would fit very well in an application track of a machine learning venue.\n\n- The proposed methodology is common in the literature. From a methodological point of view, there are no new elements/novelties in the paper. \n\n- The proposed approach outputs text instead of predicted time series. It would be important to include ablation studies when outputting time series.\n\n- Table 1 could include the number of parameters for each model. The proposed approach may be more expensive than the previous work.", "other_comments_or_suggestions": "No", "questions_for_authors": "Discuss the possibility to include a more generic benchmark.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vtHktab8Es", "reviewer": "Reviewer_n3vv", "summary": "This paper introduces ITFormer, a novel framework that bridges time-series signals and natural language for multi-modal question answering (QA). To support this task, the authors release EngineMT-QA, the first large-scale, multi-task dataset designed to capture complex interactions between temporal data and textual queries. ITFormer integrates time-series encoders with frozen large language models (LLMs) using innovative components such as Time Token Position Encoding, Learnable Instruct Tokens, Instruct Time Attention, and Time Token as Language, enabling effective temporal-textual fusion with minimal training overhead. Experimental results show that ITFormer achieves state-of-the-art performance across diverse QA tasks, offering a scalable and efficient solution for real-world time-series understanding and decision-making.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. \n\n# 1. **Generalization and Benchmark Scope**:\n\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer zRtW. Kindly refer to it for further details.\n\n# 2. **Dataset Contribution and Discussion on Essential References**\n\nThank you for your feedback. The **EngineMT-QA** dataset is designed around four core tasks to comprehensively evaluate time-series and textual data fusion:\n\n### **Dataset Construction and Task Design**\n\n- **Understanding**: Interpreting temporal patterns and mapping them to semantic intents expressed in natural language.\n- **Perception**: Inferring latent system states from raw time-series signals under semantic supervision.\n- **Reasoning**: Modeling temporal dependencies to support predictive inference and hypothetical outcomes.\n- **Decision-Making**: Integrating temporal understanding and prediction to generate actionable, context-aware responses.\n\n### **Task and Dataset Construction Details**\n\nThe dataset construction involves:\n\n- **Time-Series Data**: Extracted from real aerospace engine sensor data (temperature, pressure, vibration, etc.).\n- **Task Design**: Textual questions are crafted for each of the four tasks, combining time-series data and natural language queries to evaluate performance.\n\nFor example:\n\n- **Understanding**: Interpreting temperature trends.\n- **Perception**: Diagnosing faults from sensor data.\n- **Reasoning**: Estimating remaining useful life from historical data.\n- **Decision-Making**: Recommending maintenance actions based on system health and trends.\n\n### **Comparison with Existing Literature**\n\nWe acknowledge **Cai et al. (2024)**’s **TimeSeriesExam**, which focuses on **pattern recognition** and **anomaly detection**. However, its tasks are relatively simple, mainly addressing **basic time-series perception**. In contrast, **EngineMT-QA** offers a more comprehensive framework, covering **Understanding**, **Reasoning**, and **Decision-Making**, with a focus on **real-world decision-making** and **semantic complexity**.\n\nAdditionally, while **Merrill et al. (2024)** and **Chow et al. (2024)** focus on reasoning, **EngineMT-QA** evaluates multiple cognitive abilities through a **multitask approach**, making it more comprehensive.\n\n\n### **Transfer Learning Experiment and Contribution**\n\nIn our transfer learning experiment, models pre-trained on **EngineMT-QA** showed significant improvement on **TimeSeriesExam**, especially in **Pattern Recognition** and **Anomaly Detection**, demonstrating the dataset’s applicability beyond the aerospace domain.\n\n**EngineMT-QA** offers a comprehensive task framework for multimodal time-series and text fusion, incorporating a wider range of tasks than existing benchmarks. Its cross-domain applicability is demonstrated through transfer learning, contributing to future intelligent decision-making applications.\n\n# 3. **Experiment Details**\n\nWe would like to thank the reviewers for their valuable feedback.\n\nIn all comparison experiments, **ITFormer** models were trained on the **EngineMT-QA** dataset, with training conducted on the training subset and evaluation on the test subset. For a fair comparison, **existing multimodal models** like **GPT-4o** and **Gemini**, **vision-text models** like **InstructBlip**, **MCAN-VQA**, and **CoCa** were all adapted to use **time-series encoders** instead of their original image encoders. Specifically, the **PathTST** time-series encoder, which is identical to the one used in **ITFormer**, replaced the image encoder in these vision-text models.\n\nThis ensures that all models are evaluated under the same conditions, using time-series data in place of images. The **training steps**, **epochs** were consistent across all models, ensuring that the performance differences between **ITFormer** and the comparison models are due to the architecture and not variations in training configurations.\n\nThus, all the experiments were conducted with **fair comparisons**, allowing the **ITFormer** model’s performance advantages to be assessed in a consistent and controlled environment.\n\n# 4. **On Constructing More General Time-Series QA Datasets**\n\nTo build broader and more transferable time-series QA datasets, we propose a **semi-automated framework** that combines:\n- structured time-series segmentation,\n- **domain-specific documents or events** (e.g., logs, manuals),\n- and **LLM-driven question-answer pair generation** based on grounded events and patterns.\n\nThis framework allows us to create generalizable and transferable dataset, applicable to various domains like healthcare or finance, by adapting the event-based and temporal question generation process. Thus, while EngineMT-QA is built on aero-engine data, the underlying construction process is versatile and can be extended to other industries.", "claims_and_evidence": "The claims made in the paper are generally well-supported by clear and convincing evidence. The authors provide comprehensive experimental results demonstrating ITFormer’s state-of-the-art performance across multiple QA tasks with minimal trainable parameters, supported by strong baselines and detailed ablation studies. The introduction of the EngineMT-QA dataset is justified as a novel contribution, and the effectiveness of each ITFormer component (TPE, LIT, ITA, TAL) is validated through systematic analysis. While the generalization capability across different encoders and LLMs is supported to some extent, a notable limitation is that the EngineMT-QA dataset is constructed entirely from one domain, which raises concerns about the model’s ability to generalize to other real-world time-series scenarios such as healthcare, finance, or energy systems. Without evaluations on more diverse domains, it remains unclear whether ITFormer’s performance and alignment mechanisms are truly generalizable beyond this specific application context.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-aligned with the problem of time-series question answering, and the ITFormer framework is thoughtfully designed to bridge temporal and textual modalities. The use of a QA formulation to unify multiple time-series tasks is a reasonable and practical choice. However, a key limitation lies in the evaluation benchmark itself—EngineMT-QA is constructed solely from one domain, which significantly constrains the representativeness of the benchmark. While the tasks span understanding, perception, reasoning, and decision-making, they are all derived from a single application context, limiting the conclusions that can be drawn about the model's general applicability. As a result, it is unclear whether ITFormer would perform equally well on time-series data from other domains such as healthcare, finance, or climate monitoring. To truly validate the method's generalizability and robustness, it should include multi-domain benchmarks that reflect the broad range of real-world time-series applications.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design in the paper is generally sound and systematically structured, with evaluations conducted across four distinct QA task types: understanding, perception, reasoning, and decision-making. The use of multiple metrics (e.g., Accuracy, F1, BLEU, Rouge-L) and comparisons against both multimodal baselines (e.g., ChatGPT-4o, InstructBLIP) and domain-specific methods (e.g., Time-LLM, AutoTime) helps validate ITFormer’s performance. The inclusion of ablation studies further strengthens the reliability of the analysis by isolating the contribution of each model component.", "relation_to_broader_scientific_literature": "The paper makes a meaningful contribution to the growing literature at the intersection of time-series analysis and multimodal language modeling. Its proposed framework, ITFormer, builds upon prior advancements in time-series encoders (e.g., PatchTST, Informer, Crossformer) and multimodal integration techniques inspired by vision-language models such as InstructBLIP.", "essential_references_not_discussed": "Merrill, Mike A., et al. \"Language models still struggle to zero-shot reason about time series.\" arXiv preprint arXiv:2404.11757 (2024).\nYe, Wen, et al. \"Beyond Forecasting: Compositional Time Series Reasoning for End-to-End Task Execution.\" arXiv preprint arXiv:2410.04047 (2024).\nChow, Winnie, et al. \"Towards time series reasoning with llms.\" arXiv preprint arXiv:2409.11376 (2024).", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0w62Ks07ly", "reviewer": "Reviewer_zRtW", "summary": "The paper presents ITFormer, a novel framework that bridges time series and natural language for multi-modal temporal-textual question answering. Specifically, ITFormer uses time token position encoding (TPE) to encode the time series, then uses learnable instruct tokens (LIT) to facilitate the alignment between temporal and textual modalities, and finally uses instruct time attention, enabling efficient and robust cross-modal fusion to align and fuse temporal and textual representations. Experiments on the self-constructed EngineMT-QA dataset demonstrate that ITFormer achieves better performance than existing methodologies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. We address each of your concerns below with clarifications and new experimental results.\n# 1. **Generalization and Benchmark Scope**\nThank you for your insightful feedback on the importance of generalization in our approach. We fully recognize its significance and appreciate the opportunity to clarify this aspect.\n\nAlthough **EngineMT-QA** is based on aero-engine operation and maintenance data, the core challenge addressed by **ITFormer**—integrating time-series signals with natural language text—is inherently domain-agnostic. Tasks such as fault detection, trend analysis, and decision-making from multimodal data are common across various domains including healthcare, finance, and climate science.\n\nImportantly, the **design of the EngineMT-QA dataset** is modular and transferable. Our data construction pipeline consists of:\n\n(1) temporal segmentation of raw multivariate time-series,\n\n(2) instruction-style question generation grounded in operational semantics, and\n\n(3) structured answer derivation based on observable data events.\n\nThis methodology is not bound to the aero-engine domain and can be adapted to other domains where time-series data and domain-specific semantics coexist.\n\nTo evaluate generalization, we conducted two key experiments (each with a 6:4 train-test split):\n\n- **Experiment 1: TimeSeriesExam Dataset**\n    \n    We tested **ITFormer** on **TimeSeriesExam**, a domain-agnostic benchmark designed to assess time-series understanding across five categories: pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality analysis [Cai et al., 2024]. ITFormer achieved **0.792** in Causality Analysis, outperforming all baselines. This confirms that ITFormer is not domain-locked and generalizes well to unfamiliar, structured time-series tasks.\n    \n- **Experiment 2: Transfer Learning from EngineMT-QA → TimeSeriesExam**\n    \n    We pre-trained **ITFormer** on **EngineMT-QA**, then fine-tuned it on **TimeSeriesExam**. Performance improved further, especially in **Pattern Recognition** and **Anomaly Detection**, indicating that EngineMT-QA helps the model learn **domain-agnostic time-series reasoning patterns**. This shows that our dataset captures fundamental properties of time-series + language interactions that are transferable across domains.\n    \n\n| **Model** | **Pattern Recognition** | **Anomaly Detection** | **Noise Understanding** | **Similarity Analysis** | **Causality Analysis** |\n| --- | --- | --- | --- | --- | --- |\n| **GPT-4o (image)** | 0.82 | 0.80 | 0.90 | 0.87 | 0.68 |\n| **GPT-4o (text)** | 0.81 | 0.75 | 0.78 | 0.80 | 0.28 |\n| **Gemini-1.5-Pro (image)** | 0.81 | 0.69 | 0.83 | 0.80 | 0.48 |\n| **Gemini-1.5-Pro (text)** | 0.82 | 0.72 | 0.90 | 0.80 | 0.68 |\n| **Phi-3.5-vision** | 0.10 | 0.09 | 0.18 | 0.17 | 0.07 |\n| **Phi-3.5-mini-instruct** | 0.42 | 0.22 | 0.22 | 0.40 | 0.22 |\n| **MCAN-VQA** | 0.8086 | 0.7921 | 0.9422 | 0.897 | 0.731 |\n| **ITFormer** | 0.8275 | 0.8391 | 0.9500 | 0.920 | 0.792 |\n| **ITFormer (Transferred)** | **0.8593** | **0.8891** | **0.9833** | **0.941** | **0.834** |\n\nAs shown above, **ITFormer** consistently outperforms state-of-the-art models, particularly in **Causality Analysis** and **Pattern Recognition**. Its performance improves further through transfer learning, suggesting that **EngineMT-QA not only trains robust models but also captures cross-domain time-language patterns**.\n\nIn summary, **ITFormer** demonstrates strong generalization to unseen domains and tasks. Furthermore, **EngineMT-QA** is not only a valuable benchmark but also a **contributive asset** to the broader field of time-series + language modeling. Its design facilitates reusable, instruction-driven QA setups, offering a foundation for building unified benchmarks and training models with cross-domain transferability.\n\n> Citation for TimeSeriesExam:\n> \n> \n> Cai Y, Choudhry A, Goswami M, et al. *TimeSeriesExam: A Time Series Understanding Exam*. NeurIPS Workshop on Time Series in the Age of Large Models, 2024. [arXiv:2410.14752](https://arxiv.org/abs/2410.14752)\n>\n\n# 2. **Dataset Accessibility**\n\nWe appreciate the reviewer pointing out the dataset access issue. While the anonymous GitHub was used for submission, we acknowledge that certain VPNs may block access to the hosting service.\n\nTo ensure broader accessibility, we have now provided an alternative download link via Baidu Cloud:\n\n **Dataset link**: [https://pan.baidu.com/s/19uG78pNCK3IqMIrOqzPLQA](https://pan.baidu.com/s/19uG78pNCK3IqMIrOqzPLQA)  \n **Access code**: `9niz`\n\nWe will also include this alternative in the final version of the paper to facilitate future access.", "claims_and_evidence": "The claims in the paper are well supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "No Theorems proposed in the paper.", "experimental_designs_or_analyses": "The experimental designs listed in Section 5.2 to 5.5 (including ablation studies) make sense.", "relation_to_broader_scientific_literature": "Multi-modal, multi-dimensional time series question answering is a well-established problem in many real-world scenarios, where the integration of time series QA with LLMs could have wide application.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength\n\n1. This paper focuses on the problem of multi-task, temporal-textual time series reasoning and question-answering, which is of high importance yet not well explored by existing research works.\n2. The ablation study in Section 5.4 and the efficiency study in Section 5.5 show that the proposed ITFormer framework consistently achieves better performance than existing methodologies, where larger LLMs always lead to better ITFormer performance, demonstrating the scalability of the proposed framework.\n3. ITFormer is a lightweight framework with only ~1% of parameters needing to be tuned with frozen LLMs, which is easy to deploy in real-world applications.\n4. The paper is well-written, easy to follow, and understand.\n\nDisadvantage:\n1. Although the paper provides an anonymous GitHub for the code and models, the dataset cannot be accessed with specific VPNs.\n2. The experiments are solely on the self-created EngineMT-QA dataset. More experiments on existing datasets could help to better demonstrate the performance of the proposed method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See disadvantages", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "7PjRffd8wv", "reviewer": "Reviewer_ZFEy", "summary": "This paper tackles the new problem of multimodal question answering over multivariate time series data (Time-Series QA). \n\nThe authors introduce EngineMT-QA, a multitask dataset tailored to evaluate models' abilities in “understanding”, “perception”, “reasoning”, and “decision-making” tasks using real-world aero-engine time-series data.\n\nAdditionally, they propose an architecture they call the Instruct Time Transformer (ITFormer), a multimodal architecture framework integrating pretrained Large Language Models (LLMs) and time series encoders with specialized components, such as Instruct Time Attention (ITA) and Learnable Instruct Tokens (LIT), to align temporal and textual modalities effectively.\n\nThe proposed method is competitive across the proposed tasks, and generalizing well across pretrained LLMs and time series models. Ablation studies further underline the importance of the proposed architectural components.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and detailed feedback. We address each of your concerns below.\n# **1. On Methodological Motivation and Case Study**\n\nWe appreciate the reviewer’s feedback. While **Transformer-based architectures** are commonly used in vision-language and speech-language tasks, our work introduces **novel adaptations** for the **fusion of time-series data and natural language**, a challenge not systematically addressed in the literature.\n\nOur method’s **novelty** lies in the **task-specific, modality-aware components** that enable effective time-series and language integration. These components—**TPE (Time Token Position Encoding)**, **LIT (Learnable Instruct Tokens)**, **ITA (Instruct Time Attention)**, and **TAL (Time-As-Language)**—solve the following challenges:\n\n1. **Weak semantic nature of time-series data**: **TPE** provides temporal structure to capture sequential data.\n2. **Misalignment between time and text**: **LIT** enables extraction of task-specific information from queries.\n3. **Task-sensitive dynamic mappings**: **ITA** and **TAL** facilitate task-guided attention and temporal reasoning.\n\n### **Case Study Example**\n\nIn our case study, **ITFormer** demonstrates clear advantages over **GPT-4o** in processing time-series data with natural language queries https://imgur.com/a/dYSrsE9:\n\n- **TPE** allows **ITFormer** to correctly identify the rising altitude trend, while **GPT-4o** struggles with misinterpreting the trend as a descent. **TPE** enables the model to understand the temporal sequence, providing the **largest standalone performance gain** in our ablation experiments.\n- **LIT** helps **ITFormer** align its response with the query, focusing on relevant sensor data. Without **LIT**, the model may choose irrelevant data, as shown when **GPT-4o** introduces extraneous, irrelevant information despite recognizing the trend.\n- **GPT-4o**, while capturing the general trend, fails to maintain task relevance by generating hallucinated content like **compressor stall risk**, which doesn’t align with the data, highlighting the limitations of models that lack task-specific focus.\n\n### **Ablation Experiment Insights**\n\nOur **ablation experiments** confirm the significance of each component:\n\n- **TPE** provides the largest standalone performance gain by enabling the model to capture the sequential nature of time-series data.\n- **ITA** and **TPE** complement each other by allowing task-specific attention over relevant time spans, improving performance.\n- **LIT** is crucial for ensuring the model aligns its response with the task, particularly in real-world deployments where **task relevance** is key.\n\nWhile **Transformer-based architectures** are common, our work introduces **task-specific components** that are critical for time-series and natural language fusion. Through **TPE**, **LIT**, **ITA**, and **TAL**, we solve the challenges of aligning temporal data with textual queries. The **case study** and **ablation results** demonstrate the practical importance of these innovations, improving accuracy and task relevance in multi-modal time-series modeling.\n\n# **2.Dataset Contribution and Discussion on Essential References**\n\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer n3vv. Kindly refer to it for further details.\n\n# **3. Experiment Details**\n\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer n3vv. Kindly refer to it for further details.\n\n# **4.Presentation and Readability**\n\nWe appreciate the reviewer’s feedback on the presentation and readability. We have carefully considered the suggestions and will make the following improvements in the revised version of the paper:\n\n1. **Clarification of task definitions**: We will provide clearer explanations for the tasks, especially **Understanding**, **Perception**, **Reasoning**, and **Decision-Making**, to avoid ambiguity in how these tasks are delineated and their practical significance.\n2. **Notational consistency**: We will ensure consistent use of notation, particularly for vectors and time-series representations, to improve clarity and prevent confusion.\n3. **Figures and diagrams**: We will revise figures to make them more intuitive, including simplifying complex visualizations and adding concrete examples where applicable to enhance understanding.\n4. **Flow and organization**: We will restructure sections where necessary to improve the logical flow, ensuring that the problem definition is clearly presented before the method, and enhancing transitions between sections for smoother readability.\n\nWe believe these adjustments will improve the overall clarity of the paper while maintaining the depth of our contributions.\n\n# **5.Dataset Construction Adaption**\n\nDue to space limitations, the relevant results have been included in the rebuttal to Reviewer n3vv. Kindly refer to it for further details.", "claims_and_evidence": "The authors make several claims on introducing the time-series QA task and providing a state-of-the-art approach. While each claim is backed by some contribution, there are varying degrees of supporting evidence.\n\nOutlining these claims below:\n\nClaim 1: Introduce the Time-Series QA task \n\nSeveral recent (though non-archival) works have explored related tasks, presenting their own time-series QA benchmarks [1, 2, 3]. It would good to clarify the novelty of the dataset in this context.\n\n[1] [2404.11757] Language Models Still Struggle to Zero-shot Reason about Time Series \n[2] [2410.14752] TimeSeriesExam: A time series understanding exam \n[3] [2409.11376] Towards Time Series Reasoning with LLMs \n\n\nClaim 2: Introduction and effectiveness of architectural designs \n\nI did not get a strong understanding of whether each of these components was working as designed or intended. \nWhile I appreciated the ablations, they evaluate just the end task performance (accuracy or bleu). However, it would be more convincing to tease apart these capabilities introduced with each ITFormer component (e.g., what happens if we don’t have the “time token as language” components, how do the generations look?) \n\nOn comparison methods, it is unclear how strong the method actually is, as the paper lacks details on how the proposed models were trained vs the comparison models. \n\nClaim 3: “Introduce a robust and adaptable paradigm for integrating complex time-series tasks into end-to-end QA frameworks”\n\nWhile the ITFormer framework demonstrates clear performance improvements versus the compared multimodal models and the application to various LLMs and time series models is interesting, I think there should be additional evidence to support this statement as a whole. \n\nIn particular:\nThe evaluation of question-answering tasks using time series data as context is interesting, but has also been investigated in prior work (see first claim)\n\nThe tasks evaluate interesting capabilities (“Understanding, Perception, Decision-making, Reasoning”), but their delineation is unclear and a bit fuzzy (it’s not clear to me each of these tests an independent capability) \n\nI missed details on how their dataset construction is adaptable to any other existing time series datasets. They seem to construct the dataset specifically based on an existing aero-engine dataset and design questions tailored to this data \n[Arias Chao et al. 2021] Aircraft Engine Run-to-Failure Dataset under Real Flight Conditions for Prognostics and Diagnostics", "methods_and_evaluation_criteria": "Yes. The proposed methods and evaluation criteria (BLEU, Rouge-L, Accuracy, and F1) are standard in prior multimodal QA tasks (e.g., text-only question-answering such as HotpotQA [1], and visual-question answering (VQAv2 [2]). \n\n[1] Yang et al., 2018. [1809.09600] HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering \n[2] Goyal et al. 2016. [1612.00837] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering", "theoretical_claims": "N/A. No theoretical claims made.", "experimental_designs_or_analyses": "Yes. While the overall experimental setup is sound for evaluating the time-series QA tasks, I did think the method comparison showing the benefits of ITFormer could be more convincing. \n\nThe ITFormer method is an architectural framework or way to design multi-modal time series models that can answer questions over time series features. While the authors do compare against other recent multimodal architectures, I could not find details to ensure fair comparison here, including:\n\nTraining data: were ITTformer models trained on the EngineMT-QA dataset? How were other models trained or “adapted” in comparison? \n\nTraining hyperparameters. While the authors mention using existing multimodal architectures with image encoders adapted to time series, how was this adaptation done? Was the training setup comparable to ITFormer?", "relation_to_broader_scientific_literature": "I found the author’s time-series QA task very interesting and relevant for time series research. \nThere have been a few early works on this (Cai et al., 2024; Merrill et al., 2024; see above for references), but the paper presents interesting positive results when we can specifically train multimodal LLMs for these tasks.", "essential_references_not_discussed": "On the data contribution angle, it would be good for the authors to discuss the differences in their contribution vs prior related benchmarks: \n\n[Cai et al., 2024] introduce TimeSeriesExam, a similar multiple-choice QA benchmark designed to test time series understanding in various categories (pattern recognition, noise understanding, similarity analysis, anomaly detection, and causality analysis)\n[2410.14752] TimeSeriesExam: A time series understanding exam \n\n[Merrill et al., 2024] also propose an evaluation framework where LLMs are tasked with answering questions about time series data (provided with time series features)\n[2404.11757] Language Models Still Struggle to Zero-shot Reason about Time Series", "other_strengths_and_weaknesses": "Summarized Strengths: \n* The authors introduce a valuable new dataset for the compelling time-series QA tasks\n* They also present a new architecture framework to handle fusion for both modalities\n* The experimental results demonstrate strong performance improvements over baselines\n* The framework shows good adaptability across different LLM and time-series encoder architectures\nAreas for improvement:\n1. Balance between dataset and architecture: Especially in relation to related work on time-series QA benchmarks, the paper would benefit from emphasizing the dataset and problem definition more prominently. While the introduction does this well, the rest of the paper devotes significantly more space to architectural details than to dataset analysis. Consider expanding Section 5.1 to provide more insights into the dataset characteristics and challenges.\n\n2. Architectural presentation: The architectural exposition in Section 4 is mathematically dense and the notation was not perfectly clear to me. It could be made more accessible with:\n* A concrete example walkthrough showing how a specific time-series and question pair flows through the system\n* Moving some of the mathematical notation details to the appendix\n* Simplifying Figure 2, which currently contains too many details to follow easily\n\n3. Component analysis: Table 2 shows an ablation study of the different components, but there's limited discussion in the text about these findings. Please expand on:\n* Why TPE yields the largest standalone performance gain\n* The complementary nature of ITA and TPE when combined\n* Practical insights about which components are most critical for real-world deployments", "other_comments_or_suggestions": "Please see the following on presentation quality and framework adaptability.\nPresentation\nProblem definition clarity:\n* Clarify the segmentation approach for time-series data and why segments are important. It is not clear what is the relationship between different segments if any.\n* Use consistent notation (e.g., boldface for vectors)\n* Explain the challenges of \"modality heterogeneity\" more concretely\n\nPaper order: Consider presenting Figure 3 (dataset overview) before Figure 2 (architecture) to establish the problem context before diving into the solution details.\n\nFramework \nArchitecture: The paper would benefit from a brief discussion of alternative time-series encoding architectures (e.g., state space models) and how the framework might accommodate them. \nDataset: Clarify in the main text how adaptable your dataset construction process is to other existing time series datasets.", "questions_for_authors": "Beyond overall performance metrics, have you identified specific failure modes in ITFormer or baseline methods that underscore the necessity of your proposed components?\n\n\nCould you provide insights or qualitative examples showing how generations differ across ITFormer and the baseline methods, specifically regarding the integration of Learnable Instruct Tokens?\n\n\nWere there specific categories of questions or patterns where ITFormer distinctly outperformed other methods, and can you elaborate on why?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work proposes transformer based model "ITFormer" for question-answering w/ time-series data, i.e, integrating language and numerical time-series input (and also demonstrate numerical output). This work introduces an aerospace engine time-series data based question-answering dataset on which ITformer is trained and evaluated. They further show generalization and strong performance on the "TimeSeriesExam" benchmark [Cai, et al.].  

This work has wide potential applications, and was recommended by all reviewers for acceptance. Hence, I am also inclined to support the publication of this work.
Decision: Accept (poster)