In-context Time Series Predictor

Authors: Jiecheng Lu, Yan Sun, Shihao Yang
Primary area: learning on time series and dynamical systems
Keywords: Time Series Forecasting, In-context Learning, Transformer

Abstract:
Recent Transformer-based large language models (LLMs) demonstrate in-context learning ability to perform various functions based solely on the provided context, without updating model parameters. To fully utilize the in-context capabilities in time series forecasting (TSF) problems, unlike previous Transformer-based or LLM-based time series forecasting methods, we reformulate "time series forecasting tasks" as input tokens by constructing a series of (lookback, future) pairs within the tokens. This method aligns more closely with the inherent in-context mechanisms and is more parameter-efficient without the need of using pre-trained LLM parameters. Furthermore, it addresses issues such as overfitting in existing Transformer-based TSF models, consistently achieving better performance across full-data, few-shot, and zero-shot settings compared to previous architectures.

Decision: accept (poster)
Reviewer scores: 3, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730252570982

thread_type:
official_review

<thread_object>
creator: reviewer_uzy9

<content>
summary:
The authors propose a new in-context learning framework for time series forecasting by using "time series forecasting tasks" as input tokens. They show that this new framework outperforms the more traditional Temporal-wise Transformers and Series-wise Transformers across full-data, few-shot, and zero-shot settings.

strengths:
In-context learning for time series forecasting is an important problem with wide applications. As far as I can tell, using "time series forecasting tasks" as input tokens is a new and interesting idea, and the authors demonstrate that it indeed has utility using systematic benchmarks. The approach has a "category theory" flavor to it, and the differences from earlier frameworks are well explained.

weaknesses:
In terms of presentation, I think there is room to make it more accessible to readers not directly working in the same field. In particular, more detailed explanation and illustration of the "using time series forecasting tasks as input tokens" idea and how it is implemented could be helpful.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730455877143

thread_type:
official_review

<thread_object>
creator: reviewer_h6bm

<content>
summary:
This paper present In-context Time Series Predictor (ICTSP), a new framework of doing time series forecasting. It formulates the time series prediction task into the form of in-context learning, such that in the input data is split into a few forecasting examples to guide the model complete the target prediction with the adjacent input. Experimental results demonstrate the method’s effectiveness.

strengths:
1. **Novel Framework**: The paper proposes an innovative approach to time series forecasting by adopting in-context learning, which could inspire further research and development in this field.
2. **Comprehensive Experiments**: The authors present extensive experimental results, including ablation studies, which provide a robust foundation for evaluating ICTSP’s effectiveness.
3. **Clarity of Presentation**: The paper is well-written and clearly presented, facilitating a good understanding of the approach and its contributions.

weaknesses:
1. **Comparative Fairness**: The baselines use an input length of 512, while ICTSP utilizes an input length of 1440. This discrepancy could impact the fairness of the comparisons.
2. **In-context Example Selection**: ICTSP frames forecasting as an in-context learning task, but it uses adjacent historical data as context examples. This approach raises questions about whether the model truly learns from in-context examples or simply encodes the input tokens in a different manner.
3. **Model Reduction Analysis**: Although the paper highlights the adaptive model reduction feature of ICTSP, there is limited visualization or analysis to support this claim.

questions:
1. Could you provide results with ICTSP’s input length ($L_I$) set to 512 for a more equitable comparison with other baselines in Table 1 and 6?
2. Could you conduct an ablation study on in-context examples, possibly using randomly selected examples (without look-ahead bias) instead of adjacent history to clarify the influence of this design choice?
3. Could you provide more detailed analysis or visualizations related to the Adaptive Model Reduction to support its impact on performance?
4. How does ICTSP handle cross-channel dependencies, especially in cases with specific inter-channel relationships in the data (like traffic)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730564826586

thread_type:
official_review

<thread_object>
creator: reviewer_nx33

<content>
summary:
This paper presents the In-context Time Series Predictor(ICTSP), which utilizes the in-context capabilities for time series forecasting tasks. It proposes an innovative way to construct the tokens with context examples, showing promising results across multiple experimental settings.

strengths:
1. The writing is clear and fluent.
2. This paper's motivation is relatively novel. It proposes utilizing the in-context learning capabilities of large language models (LLMs) for time series forecasting tasks. 
3. The experiments are comprehensive, encompassing various experimental settings and datasets.

weaknesses:
1. The formulation of TSF transformers is not sufficient. As far as I am concerned, there are some methods [1,2] that utilize patching embedding to form the input tokens. These methods cannot be simply categorized as Temporal-wise Transformer or Series-wise Transformer in Section 2.2

2. The baselines results for baselines are collected from source papers. However, this paper applies different input time series length from these papers, which may lead to an unfair comparison.

3. The details of Computational Costs and Table.5 are not provided. 


[1] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations.

[2] Jin, Ming, et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." The Twelfth International Conference on Learning Representations.

questions:
Based on the listed weakness,
1. Can you add more discussions for these methods based on patching embedding from an ICL perspective?
2. For the Full-data TSF setting, can you rerun the baselines under the same experimental settings?  Alternatively, you could articulate the rationale behind your experimental settings, which would make the comparisons more convincing.
3. Can you provide more details for Computational Costs? What are the number of layers and hidden dimensions of different models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576232140

thread_type:
official_review

<thread_object>
creator: reviewer_hkwk

<content>
summary:
The paper introduces a model named "In-context Time Series Predictor" (ICTSP), designed specifically for time series forecasting (TSF). Unlike traditional Transformer-based models, ICTSP adopts a novel input token structure, resulting in improved performance.

strengths:
ICTSP model transforms time series forecasting tasks into input tokens, aligning closely with the inherent mechanisms of the Transformer model and efficiently leveraging its contextual learning capabilities.

weaknesses:
1. The innovation appears limited. While the paper extensively discusses the approach from an in-context learning perspective, the methodology itself seems simplistic and more akin to a heuristic trick due to excessive textual explanation without solid theoretical support.

2. The paper does not address whether this model could be applied to other time series tasks, such as classification, interpolation, or anomaly detection. The utility of the model appears confined to the forecasting tasks, necessitating stronger performance to justify its specialized use.

3. The significant improvements in zero-shot tasks over full-data tasks lack detailed explanations. The paper does not sufficiently clarify why the ICTSP demonstrates a pronounced advantage in few-shot and zero-shot learning scenarios than in full-data tasks

questions:
The provided link to the code repository is inaccessible, which hinders the reproducibility of the results. Could you please update or fix the link to the source code?
</content>
</thread_object>

</thread>
<thread>
creation date:
1748289843036

thread_type:
public_comment

<thread_object>
creator: ~jiecheng_lu1

<content>
comment:
We apologize for forgetting to replace the temporary anonymous code repository link in the main text with the permanent one. The correct GitHub repository for this paper is: [https://github.com/LJC-FVNR/In-context-Time-Series-Predictor](https://github.com/LJC-FVNR/In-context-Time-Series-Predictor).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an in-context time-series predictor, a unified formulation integrating temporal-wise and series-wise Transformers for time-series forecasting. This general formulation encompasses several existing models as special cases, thereby offering insights into their limitations. The experiments demonstrate superior performance across full-data, few-shot, and zero-shot settings.

One limitation of this work, raised by Reviewer H6bM, is that it does not sufficiently isolate the contribution of in-context learning capability and longer lookback window ot its strong performance. AC think this is acceptable, because, unlike in NLP, context samples from distinct time series often differ significantly in distribution, potentially hindering in-context learning. While Reviewer hkWk also holds a similar concern, he/she arguments that a Transformer trained from scratch, rather than pretrained on large-scale data, should not exhibit ICL capability. This stance contradicts a growing consensus in the community, and the authors provide extensive literature support for their position. 

Given that in-context learning remains a particularly important but underexplored topic in time-series forecasting, and considering this paper’s systematic literature discussion and solid experimental results, I recommend acceptance.