ICLR 2025

Published as a conference paper at ICLR 2025
IN-CONTEXT TIME SERIES PREDICTOR
Jiecheng Lu, Yan Sun, Shihao Yang
Georgia Institute of Technology
{jlu414,yansun}@gatech.edu, shihao.yang@isye.gatech.edu
ABSTRACT
Recent Transformer-based large language models (LLMs) demonstrate in-context
learning ability to perform various functions based solely on the provided context,
without updating model parameters. To fully utilize the in-context capabilities
in time series forecasting (TSF) problems, unlike previous Transformer-based or
LLM-based time series forecasting methods, we reformulate "time series fore-
casting tasks" as input tokens by constructing a series of (lookback, future) pairs
within the tokens. This method aligns more closely with the inherent in-context
mechanisms, and is more parameter-efficient without the need of using pre-trained
LLM parameters. Furthermore, it addresses issues such as overfitting in existing
Transformer-based TSF models, consistently achieving better performance across
full-data, few-shot, and zero-shot settings compared to previous architectures 1.
1
INTRODUCTION
Transformer-based large language models (LLMs) have significantly impacted various research
and application areas (Brown et al., 2020). Their inherent in-context learning (ICL) capabilities
highlighted by previous studies (Müller et al., 2022; Min et al., 2022; Wei et al., 2023; Xie et al.,
2022) allow them to adapt and generalize from context examples provided in input prompts without
any parameter updates. This enables LLMs to effectively handle few-shot and zero-shot tasks. Recent
research on ICL (Zhang et al., 2023; Garg et al., 2022; Akyürek et al., 2023; Li et al., 2023; Dai et al.,
2023; Bai et al., 2023) has shown that Transformers can adaptively learn to perform various functions
including linear predictors, shallow MLPs, gradient descent, algorithm selection, etc., based on a
series of (input, label) pairs as input tokens, as shown in Figure 1 (a).
Timestep
Input
Labe...