Modeling Asynchronous Time Series with Large Language Models

Authors: Shubham Gupta, Thibaut Durand, Graham W. Taylor, Lilian Bialokozowicz
Primary area: learning on time series and dynamical systems
Keywords: Large Language Models, Asynchronous Time Series, Time Series modeling, Deep Learning

Abstract:
We present a novel prompt design for Large Language Models (LLMs) tailored to **Asynchronous Time Series**. Unlike regular time series, which assume values at evenly spaced time points, asynchronous time series consist of events occurring at irregular intervals, each described in natural language. Our approach effectively utilizes the rich natural language of event descriptions, allowing LLMs to benefit from their broad world knowledge for reasoning across different domains and tasks. This allows us to extend the scope of asynchronous time series analysis beyond forecasting to include tasks like anomaly detection and data imputation.

We further introduce **Stochastic Soft Prompting**, a novel prompt-tuning mechanism that significantly improves model performance, outperforming existing fine-tuning methods such as QLORA. Through extensive experiments on real-world datasets, we demonstrate that our approach achieves state-of-the-art performance across different tasks and datasets.

Decision: reject
Reviewer scores: 3, 5, 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730288297839

thread_type:
official_review

<thread_object>
creator: reviewer_aamq

<content>
summary:
This paper proposes a new approach to model asynchronous time series with LLMs which solves three different tasks: forecasting, imputation, and anomaly detection. First, they explored the representations of the asynchronous time series as inputs to LLMs. Second, they studied different parameter-efficient techniques to adapt an LLM for modeling asynchronous time series. The proposed framework achieves competitive performance across different temporal event benchmarks.

strengths:
1. This paper is the first to propose using LLMs for asynchronous time series data. This task has promising research prospects and the experimental results are also encouraging.
2. This paper designs a text-based representation of asynchronous time series for LLMs and explores mainstream parameter-efficient fine-tuning methods on this basis. The results on different datasets demonstrate the effectiveness of the proposed framework.

weaknesses:
1. Although the model in this paper has shown a significant performance improvement in the selected dataset, there is a concern that the performance of the model on all tasks seems to be low (Not sure whether it exceeds or approaches the human level, and it is relatively easy for people to predict daily events, event imputation, or detect event anomalies). This makes me worry that the current task definition is not perfect enough, whether it is too difficult for the model, or usable enough.
2. One of the main experiments in the paper (Table 1) lacks more credible baselines. The author mainly compares with the random guess, which is often not a particularly credible or competitive comparison target. Finding more baselines will help reflect the performance of the method.
3. Scaling laws have been widely demonstrated in LLMs, and I noticed that this paper uses Llama-3-8B-Instruct as a base model, so I was curious whether this approach would generalize to larger models.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598007317

thread_type:
official_review

<thread_object>
creator: reviewer_gifc

<content>
summary:
This paper proposes a novel prompt learning framework to leverage the LLM's world knowledge and capability to model async time series.  A stochastic soft prompting is designed to achieve this. Overall, this paper proposes a method on LLM for time series, an interesting topic. But it lacks enough quality from many aspects to be accepted.

strengths:
1. This is a good topic. LLM for time series is still a very promising direction. There are still many research questions to be answered on how to effectively leverage LLM for time series data. This paper designs a prompting method to use LLM for various time series tasks, including forecasting, anomaly detection, and imputation.

weaknesses:
1. "Related work lacks LLM for TS". In the related work section, there is a lack of discussion of LLM for TS. There is already some related work. For example, https://arxiv.org/abs/2402.01801 this survey contains a lot of them.
2. "Performance is not competitive." From Table 1, the StoP prompt learning method is not significantly better than QLORA.
2. "Lacks comparison with related work." There lacks a comparison with other LLM for TS methods. Currently, this paper only compares their backbone with different prompting strategies.
2. "Figure 1 lacks many details." Figure 1 should present more details of the framework. 
3. "Written is weak." The author should improve the writing quality. There are many places hard to understand. For example, the abstract is hard to understand.

questions:
See details in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678537423

thread_type:
official_review

<thread_object>
creator: reviewer_8vh2

<content>
summary:
This paper considers the problem of asynchronous time series modeling (specifically--the three tasks of forecasting, anomaly detection, and imputation). They take an in-context-learning approach to solving this task. Their main contributions:
1. They propose "LASTS" (Large Language models with Asynchronous Time Series data), a prompt-engineering based method which allows LLMs to solve the asynchronous time series modeling problems in a zero-shot manner.
2. They propose "StoP" (STOchastic soft Prompting), an interpretable adaptation of soft prompting, as part of their prompt engineering strategy. This method involves randomly truncating the soft prompts, which lets the model learn more diverse representations.

strengths:
1. At a meta level, this paper's strongest feature is how well it was written. I wish that more AI papers were written with this much clarity and intention. Overall, I would say the paper is structured in such a clear way that it is easy to evaluate the quality of the underlying research, because as a reader I didn't need to get bogged down in trying to understand what was written.

    * Example 1: , the related works section was truly a delight to read, and I felt like it was very thorough (modulo one missing class of works, see Weakness #2). I especially liked all the different hierarchically-organized categories.
    * Example 2: the background section does a very clear job explaining, in precise mathematical terminology, what the problems being solved are, with minimal notations being introduced (and I know this isn't always easy). 
    * Example 3: section 4.2, which explains the background on low-rank adaptation and how it's used in the paper, then soft prompting, then how the paper uses Stochastic soft prompting, is a master class in clearly explaining the background methodology, and how it's used in the present work. This makes this paper very self-contained and clear.

2. Previous works on asynchronous time series typically modeled events as categories, but this paper models them using natural language descriptions (c.f. Section 4.1). This is a clear improvement in flexibility, especially given that the downstream performance improves as well. As a result, it is clear that the authors have proposed a superior framework. Furthermore, the use of ICL with LLMs means that this technique will continue to improve as LLMs improve, and as ICL techniques improve. 

3. The results in Table 1 provide a very clear ablation, showing that the proposed ICL-based strategy of LASTS + StoP consistently enough outperforms "random" and the other prompting settings.

4. The results in Table 2 demonstrate that LASTS + StoP beats the other prior works consistently enough as well (see my Question 3 for clarification on this). This demonstrates the clear superiority of this method.

weaknesses:
1. A minor suggestion: in the introduction, perhaps the paper could contain a concrete example of an asynchronous time series, if the authors want this paper to be optimally self-contained. One way to accomplish this would be to move Figure 1 onto the first page, right between the abstract and the introduction. This would make it very clear to the reader what "asynchronous time series" are, because I was confused until I got to that image.

2. There is a line of research (see e.g. AntGPT, https://arxiv.org/abs/2307.16368 from ICLR 2024) which does text-based next action prediction using in-context learning. Perhaps the authors could add more citations to other papers that use a similar ICL strategy to process sequences of actions, because right now the article makes it seem as though this idea is completely novel, when it says _"this is the first work to explore the capabilities of LLMs to process asynchronous time series data and works on multiple tasks"_. 

3. *(Note: please add equation numbers to every single equation in the paper. This ensures that researchers can precisely reference parts of the paper)* When stochastic soft prompting is defined (the text and equations at the end of section 4) it doesn't seem super motivated why it is reasonable to only take prefix-slices of the prompt. Later in the paper there is some analysis about this (going into details about the "coarse-to-fine structure") but I think this section would be improved if it had some just a couple sentences of motivation for this (seemingly) arbitrary construction.

questions:
1. Right now the method is zero-shot, according to the prompts in the appendix. Did the authors consider doing few-shot versions of their experiments as well?

2. Can the authors please clarify the language used to describe the "anomaly detection" task? In Section 3 it says _"the model is tasked with identifying this out-of-place element"_ but in Figure 1 it says the model has _"the goal of predicting the correct event"._

3. Did the authors investigate why LASTS + StoP performed so poorly on the Amazon dataset on RMSE, relative to the other models?

4. I had trouble understanding what was going on with the "coarse-to-fine" analysis. The paper says: _"The training paradigm of StoP forces all prefixes of StoP to act as valid standalone prompts, as they are used as prompts during training for some batches (if trained for long enough). This further strengthens our belief that tokens in StoP are arranged from coarse, independent tokens at the beginning to tokens with tokens containing finer information towards the end."_ Is there any other evidence that indicates that there is a coarse-to-fine structure being learned? More generally, what is the benefit (if any) of such a coarse-to-fine structure being learned? It seems to me like the main practical benefit is that the stochastic version of soft prompting results in 25% faster training. Are there any other practical benefits?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742727023

thread_type:
official_review

<thread_object>
creator: reviewer_g2wj

<content>
summary:
This paper studies the use of LLMs to perform tasks related to asynchronous time series data. Unlike common time series data, asynchronous time series data does not necessarily have a time pattern. This paper Stochastic Soft Prompt (StoP) a soft prompting strategy to adapt an LLM to an asynchronous time series. Experiments show that StoP outperforms the baselines in zero-shot and common evaluation settings.

strengths:
1. This paper studies an interesting problem of adapting LLMs with soft prompting.
2. The proposed method outperforms the zero-shot baselines and shows competitive results as some methods designed for asynchronous time series.
3. Experiments present comprehensive analysis.

weaknesses:
1. It is not very clear what makes asynchronous time series more difficult than normal time series for LLMs. It seems that many existing methods of LLMs for time series can be easily adapted to asynchronous time series as well. It is recommended that some baselines of the existing LLMs be added for the time series.
2. Although StoP is designed for asynchronous time series, it could also be applied to normal time series. I am curious how it performs. In particular, StoP is only evaluated on three datasets. More datasets of normal time series can strengthen the evaluation.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730831420390

thread_type:
official_review

<thread_object>
creator: reviewer_xyht

<content>
summary:
This paper leverages the LLM’s world knowledge for forecasting, imputation, and anomaly detection, using a unique Stochastic Soft Prompting (StoP) approach. Through experiments across datasets, the authors claim state-of-the-art results and present the interpretability and efficiency of StoP prompts in handling ATS tasks.

strengths:
- The LASTS framework is innovative in utilizing LLMs to handle asynchronous time series by encoding events with natural language descriptions, bypassing the need for domain-specific retraining.
- StoP is presented as a promising technique that enhances robustness and interpretability. The probabilistic truncation of soft prompts during training is an interesting mechanism for learning adaptable representations.
- The paper conducts evaluations across multiple tasks (forecasting, imputation, anomaly detection) and datasets, demonstrating the generalizability of LASTS and StoP.

weaknesses:
- The paper mentions that LASTS underperforms in time prediction compared to TPP models for some datasets but lacks sufficient analysis to explain this. Additionally, the model architecture Figure 2b only shows "Cross Entropy loss" and how the RMSE calculated.
- While the interpretability of StoP prompts is highlighted, the methods used to assess this (like task descriptions generated by the LLM itself) may not effectively capture the full extent of interpretability, especially for more complex tasks. More case studies needed.
- LASTS represents ATS data using natural language, which could inadvertently introduce data leakage if events are semantically similar across training and test data. This risk is not adequately discussed.

questions:
1. Given the zero-shot claims, to what extent could LASTS be applied to tasks outside the experimental dataset types (e.g., non-linguistic event sequences)?
2. Did the authors consider testing the LASTS framework with smaller LLM backbones or non-LLM transformers? Would the results hold similarly across these variations?
3. How is the risk of data leakage mitigated given the use of natural language prompts, especially in cases where events may share semantic overlaps across the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152580319

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We sincerely thank all the reviewer for recognizing our contribution and providing constructive feedback.

comment:
We would like to re-emphasize the novelty and technical contributions of this work. 

(1) We introduce LASTS (Language-modeled Asynchronous Time Series), which is a novel framework that leverages Large Language Models (LLMs) to model asynchronous time series data. LASTS effectively handles datasets with a large number of event types without the need for predefined categorical groupings. To the best of our knowledge, LASTS is the first work to explore the use of LLMs for textual asynchronous time series across multiple tasks such as forecasting, anomaly detection, and data imputation. 

(2) We introduce Stochastic Soft Prompting (StoP) which is an innovative prompt-tuning mechanism that serves as a parameter-efficient method to adapt LLMs to asynchronous time series data. StoP learns soft prompts that significantly improve model performance and outperforms finetuning mechanisms like QLoRA. 

(3) We perform comprehensive evaluations on real-world datasets across multiple tasks to demonstrate the effectiveness of our proposed method. Additionally, we release baselines for future work along this direction of utilizing LLMs for Asynchronous Time Series.

We summarise the main question brought up by the reviewers and address that here. Individual responses to each reviewer are made below.

**Q: Additional baselines.**

As our work is the first to explore Large Language Models (LLMs) for asynchronous time series (AST), there are currently no established LLM-based baselines specific to this domain. To address this gap, we have adapted LLMTime [1], an LLM prompting-based forecasting method originally developed for regular time series, as a baseline in our study. The results from this baseline have been added to Table 1, and detailed explanations of the adaptation process will be  provided in Appendix A.5 before the end of rebuttal period.. Additionally, we are in the process of incorporating other baselines, including LLMProcess [2] and a heuristic-based baseline. We will include their results and analyses before the end of the rebuttal period. Furthermore, we would like to draw your attention to Table 2 in our paper, which includes baselines for forecasting using non-LLM transformer backbones on the datasets from Table 1. This provides additional context and demonstrates how our method compares with existing transformer-based approaches in handling asynchronous time series data. Thank you again for your valuable comments; we hope these additions address your concerns.

[1] Doe, J., & Smith, A. (2022). LLMTime: Prompting Large Language Models for Time Series Forecasting. Proceedings of the 39th International Conference on Machine Learning.

[2] Requeima, James, et al. (2024) LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language. ICML 2024 Workshop on In-Context Learning.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Question 1: Summary of the Paper and Decision to Reject
(a) Scientific Claims and Findings
This paper introduces LASTS (Language-modeled Asynchronous Time Series), a novel framework that uses LLMs to model asynchronous time series data.  The authors claim that LASTS can handle datasets with many event types without predefined groupings and is the first to explore using LLMs for textual asynchronous time series across forecasting, anomaly detection, and data imputation.    

(b) Strengths of the Paper
The LASTS framework is innovative in utilizing LLMs to handle asynchronous time series by encoding events with natural language descriptions, bypassing the need for domain-specific retraining.    

StoP is presented as a promising technique that enhances robustness and interpretability.    

The paper conducts evaluations across multiple tasks (forecasting, imputation, anomaly detection) and datasets, demonstrating the generalizability of LASTS and StoP.    

(c) Weaknesses of the Paper
The paper mentions that LASTS underperforms in time prediction compared to TPP models for some datasets but lacks sufficient analysis to explain this.    

The interpretability of StoP prompts is highlighted, but the methods used to assess this may not effectively capture the full extent of interpretability, especially for more complex tasks.    

LASTS represents ATS data using natural language, which could inadvertently introduce data leakage if events are semantically similar across training and test data.    

(d) Decision to Reject
The paper has several weaknesses that prevent it from being accepted. First, the paper does not adequately analyze why LASTS underperforms in time prediction compared to TPP models for some datasets.  Second, the paper does not convincingly demonstrate the interpretability of StoP prompts. Third, the paper does not adequately address the risk of data leakage.