Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data

Authors: Paul Quinlan, Qingguo Li, Xiaodan Zhu
Primary area: foundation or frontier models, including LLMs
Keywords: Time-Series, Natural Language Processing, Multi-Modal

Abstract:
Time-series analysis is critical in many industries such as healthcare, finance and energy sectors, where understanding time-series trends alongside contextual information is essential for informed decision making. However, current time-series models are limited in their ability to perform reasoning that involves both time-series data and textual information. In this work we address this gap by introducing Chat-TS, a large language model (LLM)  designed specifically for reasoning over time-series and textual data. Unlike traditional time-series models Chat-TS integrates time-series tokens into the LLM vocabulary, enhancing its reasoning ability over both text and time-series modalities without compromising its core natural language capabilities.
To support the development and validation of Chat-TS we contribute three new datasets: the TS Instruct Training Dataset which pairs diverse time-series data with relevant text instructions and responses for instruction tuning, the TS Instruct question and answer (QA) benchmark, a set of nearly 4000 multiple-choice questions designed to evaluate multi-modal reasoning and the TS Instruct Qualitative Benchmark which provides a smaller subset of QA, math and decision making questions for LLM evaluation. Our training strategy preserves the inherent reasoning capabilities of the LLM while augmenting it with time-series reasoning capabilities. Evaluation results show that Chat-TS achieves state-of-the-art performance in multi-modal reasoning tasks, maintaining strong natural language proficiency while advancing time-series reasoning. All models, datasets, and code will be made publicly available [link].

Decision: reject
Reviewer scores: 6, 1, 3, 6

Reviews and discussion:
<thread>
creation date:
1730633267670

thread_type:
official_review

<thread_object>
creator: reviewer_zldd

<content>
summary:
This paper introduces a novel approach to leverage multi-modal large language models (LLMs) for time-series analysis. It focuses on maintaining the LLM's reasoning capabilities and utilizing its extensive knowledge base to enhance time-series data reasoning. Specifically, the authors develop Chat-TS, a new foundation model capable of explicitly preserving the general knowledge and reasoning abilities of LLMs, as well as a training strategy that integrates time-series tokens into the LLM vocabulary, enhancing the model's reasoning ability over both text and time-series modalities without compromising its core natural language capabilities. Besides, the authors curate a new time-series instruction tuning dataset (TS Instruct) and a new testbed for benchmarking multi-modal reasoning in time-series analysis (TS Instruct QA).

strengths:
1. The paper provides a detailed quantitative analysis that assesses the quality of model responses, going beyond simple accuracy metrics.

2. The proposed discrete tokenizer for multivariate time-series data requires no training while maintaining a very high reconstruction quality (in terms of MSE on the validation set).

3. The proposed Chat-TS models show an average improvement in reasoning capabilities of ~15% on the new TS Instruct QA benchmark compared to existing methods.

weaknesses:
While the TS Instruct dataset and the TS Instruct QA benchmark are comprehensive, they primarily consist of synthetic data. Real-world time-series data can be more complex and noisy, which might affect the model's performance in practical applications.

questions:
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678689492

thread_type:
official_review

<thread_object>
creator: reviewer_dxnt

<content>
summary:
This paper introduces a method for leveraging LLM in time-series reasoning and question-answering (QA) tasks. The authors propose a discrete tokenizer that transforms continuous time-series data into sequences of discrete bins, facilitating better integration of time-series data with LLMs. The paper also introduces three new datasets: (1) a time-series instruction tuning dataset; (2) a time-series QA dataset designed to assess multimodal reasoning capabilities in time-series tasks; and (3) a TS-Instruct Quantitative benchmark for evaluating performance in QA, mathematical reasoning, and decision-making beyond accuracy alone. The authors' best-performing model demonstrates strong results relative to a baseline model, though some experiment outcomes are mixed.

strengths:
1. The discrete tokenizer proposed in this paper offers a straightforward yet effective approach to transforming continuous time-series data into a form compatible with LLMs. This innovative approach could potentially inspire future work on bridging time-series analysis with language models, expanding the versatility of LLMs in temporal domains.

2. The authors present a comprehensive synthetic datasets including benchmarks specifically tailored for time-series tasks, which includes three new resources: 1) TS Instruct Training Dataset for instruction tuning on time-series data, 2) TS Instruct QA Benchmark for evaluating multimodal reasoning in time-series contexts, and 3) TS Instruct Quantitative Benchmark for a detailed assessment across QA, mathematical reasoning, and decision-making questions.

3. Performance on Text-Only Tasks: Despite the focus on time-series tasks, the model also maintains strong performance on traditional text-only tasks, such as those in the MMLU benchmark. This versatility indicates that the proposed methods do not compromise the model's capabilities in established LLM domains, making the approach both flexible and robust.

weaknesses:
- Generalization: It is unclear whether the synthetic dataset can generalize effectively across diverse types of time-series data or to sequences longer than those encountered during training. Experiments regarding this concern is highly recommended to be added.

- Dataset Alignment Process: The alignment between time-series data and natural language relies solely on GPT-4, without any additional quality control to validate this alignment process. This raises concerns about potential inconsistencies or inaccuracies in the dataset.

- Soundness and Baselines:
    - Reporting results from common multimodal baseline models would provide readers with a clearer understanding of the current state-of-the-art performance for these tasks. This is particularly relevant, as the base model shows very strong performance in the detailed quantitative evaluation presented in Table 5.
    - Given that GPT-4o is heavily involved in the dataset creation, a direct performance comparison with GPT-4o is missing.
    - Mixed Experimental Results: The experimental results are somewhat mixed (as detailed in Question), which makes it difficult to conclusively assess the model's strengths and limitations.

questions:
1. Why does the base model show strong performance on the TS Instruct QA benchmark but perform less well on TS Math Analysis and Decision Making tasks?

2. Since you describe this work as the development of a new model family, do you expect that the improvements from your method would still hold with larger models? A scaling experiment with a larger model would be beneficial, if feasible.

3. Typo in Line 23: "qualitative" should be "quantitative."

4. I found it difficult to follow the process for using GPT-4 to create the dataset. Does this process is prompting GPT4o for generating responses based on a tailored system prompt, along with the image of the time series, metadata such as length, and number of channels? If so, is this essentially a form of distillation using GPT-4 to produce the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697945500

thread_type:
official_review

<thread_object>
creator: reviewer_xwsl

<content>
summary:
This paper introduces Chat-TS, a large language model (LLM) designed for multi-modal reasoning over time-series and natural language data. It addresses the gap in existing time-series models' ability to integrate textual information for decision-making. The authors contribute three new datasets for training and evaluating LLMs on time-series reasoning tasks: the TS Instruct Training Dataset, the TS Instruct QA Benchmark, and the TS Instruct Qualitative Benchmark. They also propose a training strategy that enhances the LLM's reasoning capabilities without compromising its natural language proficiency. Evaluation results demonstrate Chat-TS's state-of-the-art performance in multi-modal reasoning while maintaining strong language skills. Although the paper showcases promising results, the experimental section could benefit from more rigorous analysis and a deeper exploration of the model's limitations.

strengths:
The paper identifies a relatively underexplored area in the field of time-series analysis by focusing on the integration of natural language processing with time-series data, suggesting that further research is needed to fully leverage the potential of multi-modal reasoning in this context.

weaknesses:
It appears that the manuscript has some formatting issues, particularly with the margins not conforming to ICLR's template specifications, and it could benefit from a more in-depth analysis of the model's limitations, potential biases, and practical applications, as well as from more rigorous experimental validation against current methodologies.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721222533

thread_type:
official_review

<thread_object>
creator: reviewer_fvka

<content>
summary:
This paper combines time-series data with text to train a chatbot with numerical data understanding abilities. This enables directly interact with raw data without modality conversion. To deal with the task, the authors propose to apply vision-language models to generate synthesized instruction tuning datasets. Experiments show time-series data pretraining does not affect the original text-based evaluations.

strengths:
- This paper constructs a novel time-series+text dataset for multi-modal instruction tuning. The construction method is interesting and there includes four kinds of conversations.
- The time-series pretraining does not affect text-based benchmark results and may improve the reasoning (BBH).

weaknesses:
Please feel free to discuss and leave comments.

- There lacks of dataset quality check. Since the instruction tuning dataset is generated by GPT-4o-mini with images, it may introduce errors. Although the authors have filtered out bad cases (as introduced in the appendix), it is not sure whether hallucinations do not exist. Maybe I missed some details, please feel free to have a discussion.
- Limited modality interleaving: since the trained model could only generated text (as the dataset constructed to do so), it limits the application to time-series data understanding, and ignores the time-series forecasting. Maybe a more fine-grained instruction dataset should be constructed with multi-modal responses.
- Lack of baselines: Maybe visual language models could be considered as baselines to take images and instructions as input and output their options. This could be a reference to tell readers why numerical data pretraining is useful for relevant time-series data understanding.

questions:
- For Table 3: Why TS training could improve the BBH and GPQA results while harming the MMLU-Pro results? Is it just performance fluctuation or there is an inner difference between these benchmarks?
- Since the dataset is generated by visual QA, I’m wondering if multi-modal (vision + text) pretraining & SFT could also handle the downstream benchmark evaluations.
- Open question: what affects the time-series data pretraining? e.g., context length, the vision ability, the reasoning ability, etc.
- Typo:
    - Line 049: LLava → LLaVA
    - Line 057: it’s effectiveness → its effectiveness
    - Line 327: LLama3.1 → LLaMA 3.1
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230554091

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
We appreciate the reviewers' feedback and the opportunity to address their concerns. In response to the comments, we have made significant additions and clarifications. We are currently working in integrating these changes into our submission and are aiming to have them integrated into the paper before the rebuttal deadline.

### 1. **Dataset Quality Check**

To address concerns about the dataset quality:

- We have manually conducted an evaluation on 100 random samples from the dataset to measure the quality, correctness, and relevance. Across these samples, we recorded **92** correct samples. The remaining **8** either had multiple correct answers or the correct answer was unclear and open to interpretation. This shows that overall our synthetically generated datsets is of reasonably high quality.
- We also recalculated performance metrics using this subset. For this subset we took the opportunity to **manually re-write** many of the samples to improve the diversity and accuracy for each sample in the datset. We will refer to this data subset as **TS Instruct QA Gold**. These results reinforce the robustness of the dataset and its minimal impact on model performance due to potential noise.

### 2. **Baseline Additions**

Several new baselines have been added to evaluate the models more comprehensively:

- **Llama-3.2-11B Vision-Instruct**
- **Phi-3.5 Vision-Instruct**
- **Phi-3-medium-4k-instruct**
- **gemma-2-2b-it**
- **gemma-2-9b-it**
- **Ministral-8B-Instruct-2410**
- **GPT-4o-mini**
- **GPT-4o**

These baselines enable a more detailed comparison between models and demonstrate the advantages of using numerical data pretraining for time-series understanding. We also included a direct performance comparison against GPT-4o and GPT-4o-mini, which was heavily involved in dataset creation.

The full results and subsequent analysis is provided below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Chat-TS, a novel approach to integrating time-series data with large language models through a discrete tokenization method and synthetic instruction datasets. The reviewers acknowledge some notable strengths, including the innovative approach to transforming continuous time-series data into discrete tokens, the creation of comprehensive synthetic datasets for time-series reasoning, and the maintenance of performance on text-based tasks (Reviewer FVKA, dxnt). The authors propose three key datasets: a time-series instruction tuning dataset, a time-series QA benchmark, and a quantitative evaluation benchmark (Reviewers xWsL, dxnt).

However, significant weaknesses substantially undermine the paper's contributions. Reviewers consistently raised concerns about the methodological soundness, including the lack of rigorous dataset quality control and potential hallucinations introduced by GPT-4o during dataset generation (Reviewer FVKA). The generalizability of the synthetic datasets is questionable, with limited exploration of performance across diverse time-series types and sequence lengths (Reviewer dxnt). Additionally, the experimental results are mixed, with inconsistent performance across different benchmarks, and the paper lacks comprehensive baseline comparisons and a direct performance comparison with GPT-4o (Reviewers xWsL, dxnt). The presentation was also critiqued for formatting issues and clarity of the dataset creation process. Consequently, the reviewers unanimously recommend rejection, with ratings ranging from "strong reject" to "marginally above the acceptance threshold."

Owing to the violation of the formatting instructions, the paper should be rejected.