In-context Time Series Predictor

## Abstract

Abstract Recent Transformer-based large language models (LLMs) demonstrate in-context learning ability to perform various functions based solely on the provided context, without updating model parameters. To fully utilize the in-context capabilities in time series forecasting (TSF) problems, unlike previous Transformer-based or LLM-based time series forecasting methods, we reformulate "time series forecasting tasks" as input tokens by constructing a series of (lookback, future) pairs within the tokens. This method aligns more closely with the inherent in-context mechanisms, and is more parameter-efficient without the need of using pre-trained LLM parameters. Furthermore, it addresses issues such as overfitting in existing Transformer-based TSF models, consistently achieving better performance across full-data, few-shot, and zero-shot settings compared to previous architectures.

## 1 Introduction

Transformer-based large language models (LLMs) have significantly impacted various research and application areas []. Their inherent in-context learning (ICL) capabilities highlighted by previous studies [, , ] allow them to adapt and generalize from context examples provided in input prompts without any parameter updates. This enables LLMs to effectively handle few-shot and zero-shot tasks. Recent research on ICL [, , , , , ] has shown that Transformers can adaptively learn to perform various functions including linear predictors, shallow MLPs, gradient descent, algorithm selection, etc., based on a series of (input, label) pairs as input tokens, as shown in Figure 1 (a)

Figure: Figure 1: Overview of in-context TSF learning in our setup.
Refer to caption: x1.png

Time series forecasting (TSF) is critical in fields like epidemiology, finance, and transportation [, , , ], predicting future values from historical data. Temporal-wise Transformers, which build input tokens from series values at each timestep, have been widely researched in TSF [, , , ]. Some studies have identified several issues with such Transformers, like timestep mixing and permutation invariance [], leading to overfitting and underperformance on real-world datasets compared to simpler models like linear predictors. Previously proposed solutions include channel independence [], random channel dropout [], and using Series-wise Transformers [, ], which consider each series as a token. Yet, their underlying mechanisms are not well explained. Additionally, the fixed structure of input series of these existing Transformers restricts their adaptability in few-shot and zero-shot learning scenarios for multivariate TSF.

Moreover, recent research have expanded the application of Transformer-based LLMs to TSF [], achieving improvements in few-shot and zero-shot generalization. They use the methods like prompt engineering [], fine-tuning and embedding inversion [, ] to integrate time series context into prompts, improving forecasting accuracy for new data. However, these methods are designed to adapt LLMs to TSF tasks rather than directly addressing the core aspects of TSF problems. This adaptation leads to inefficient use of LLM parameters and substantially increased computational costs.

In this study, we apply the most fundamental ICL settings in TSF problems to construct the In-context Time Series Predictor (ICTSP). This structure allows us to leverage the in-context generalization capabilities of Transformers efficiently without relying on large-scale LLM parameters. In ICTSP, we generate a sequence of (lookback, future) pairs, representing "time series forecasting tasks," from the original time series data, which are used as input tokens for the Transformer, as shown 1 (b). This setup enables the Transformer to adaptively learn the most effective predictor for the target tasks based on the ground truth forecasting examples as context. Additionally, the ICTSP effectively resolves aforementioned longstanding issues in previous TSF Trasnformers, significantly enhancing the performance in few-shot and zero-shot learning scenarios in multivariate TSF settings.

The main contributions of this paper are summarized as follows:

a) We innovatively use forecasting tasks — instead of traditional timestep values or single series — as input tokens to construct the ICTSP structure. By utilizing ground truth (lookback, future) pairs as context examples, we fundamentally and efficiently leverage ICL abilities for TSF tasks. ICTSP outperforms previous methods across full-data, few-shot, and zero-shot scenarios, positioning it as a potential solution for building universal large TSF models.

b) From an ICL perspective, we explain that issues in existing TSF Transformers, like timestep mixing, permutation invariance, and channel structure restriction, are caused by inappropriate token formulation. We show how previous solutions have partially addressed these issues and how ICTSP effectively solves them without the drawbacks of previous approaches.

c) We show that the ICTSP structure encompasses several simpler models as special cases, allowing for a sequential adaptive reduction in complexity: i) predictors learned from context examples through ICL, ii) a series-wise Transformer without context examples [], and iii) univariate MLPs or linear predictors []. This connection ensures stable performance across different complexities of time series dataset and prevents the significant overfitting that has previously hindered Transformers from outperforming simpler models consistently in real-world datasets.

## 2 Method

### 2.1 Preliminaries

Time Series Forecasting  Let $X\in\mathbb{R}^{C\times L}$ represent a multivariate time series, where $L$ is the total length and $C$ is the number of channels of input series. $X$ is split into historical input $X_{I}\in\mathbb{R}^{C\times L_{I}}$ and future data $X_{P}\in\mathbb{R}^{C\times L_{P}}$, with $L=L_{I}+L_{P}$. Here, $L_{I}$ and $L_{P}$ represent the lengths of the historical and forecasting periods, respectively. The value at the $t$-th timestep in the $j$-th channel is $X^{(t)}_{j}$, where $t\in\{1,\ldots,L\}$ and $j\in\{1,\ldots,C\}$. The objective is to develop the best predictor $f:\mathbb{R}^{C\times L_{I}}\rightarrow\mathbb{R}^{C\times L_{P}}$ that maps historical inputs to future outputs, yielding $\widehat{X}_{P}=f(X_{I})$.

Transformer Architecture  In this study, we employ a Transformer architecture with pre-normalization settings, processing $D$-dimensional input tokens $\{\mathbf{z}_{i}\}_{i=1}^{N}$ from the input matrix $\mathbf{Z}^{D\times N}=[\mathbf{z}_{1},\dots,\mathbf{z}_{N}]\in\mathbb{R}^{D
\times N}$. Each token $\mathbf{z}_{i}$ passes through $K$ layers of the Transformer. Each layer begins with layer normalization $\mathtt{LN}(\cdot)$, followed by self-attention $\mathtt{Attn}(\cdot)$ and a feed-forward network $\mathtt{FFN}(\cdot)$. The output $\mathbf{Z}_{k}$ for each Transformer layer, $\mathtt{TF}_{k}$, is computed as:

$$ (1) $$

with $\mathbf{Z}_{0}=\mathbf{Z}$ and the first addition of $\mathbf{Z}_{k-1}$ providing a residual shortcut directly from input to output. The Transformer’s final output is $\mathtt{TF}(\mathbf{Z}_{0})=\mathbf{Z}_{K}$.

In-context Learning  ICL involves each datapoint $(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathbb{R}^{a}\times\mathbb{R}^{b}$ from a dataset $\{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{N}$. Unlike traditional supervised learning that learns direct mappings from $\mathbf{x}_{i}$ to $\mathbf{y}_{i}$, ICL predicts $\mathbf{y}_{i}$ using both historical observations $\{(\mathbf{x}_{j},\mathbf{y}_{j})\}_{j<i}$ and the current input $\mathbf{x}_{i}$. The input format for the $i$-th datapoint in the ICL settings is structured as:

$$ (2) $$

where $\mathbf{p}_{\cdot}$ are positional embeddings and $\mathbf{o}$ is a placeholder embedding for the target, typically zero-filled. Note that in some ICL settings, $\mathbf{x}_{\cdot}$ and $\mathbf{y}_{\cdot}$ may be divided into separate tokens, which is proved interchangeable with our expression above [, ]. The positional embeddings $\mathbf{p}_{\cdot}$ are added to the tokens and shown concatenated here just for illustration. Finally, the predicted output for the $i$-th data point, $\widehat{\mathbf{y}}_{i}$, can be extracted from the corresponding token in the transformer’s output $\mathbf{\hat{Z}}=\mathtt{TF}(\mathbf{Z})$. For simplicity, we have omitted the input and output linear projections for matrices $\mathbf{Z}$ and $\mathbf{\hat{Z}}$ in this description. In practice, these projections map the tokens to a $d$-dimensional latent space. The computation is formulated as $\mathbf{\hat{Z}}=\mathbf{W}_{\text{out}}\mathtt{TF}(\mathbf{W}_{\text{in}}
\mathbf{Z}+\mathbf{b}_{\text{in}})+\mathbf{b}_{\text{out}}$, where $\mathbf{W}_{\text{in}}\in\mathbb{R}^{d\times(a+b)}$ and $\mathbf{W}_{\text{out}}\in\mathbb{R}^{(a+b)\times d}$ are the linear layer weights, and $\mathbf{b}_{\text{in}}$ and $\mathbf{b}_{\text{out}}$ are the biases. The ICL settings allow the model to use $\{\mathbf{x}_{1},\mathbf{y}_{1},\cdots,\mathbf{x}_{i-1},\mathbf{y}_{i-1},
\mathbf{x}_{i}\}$ to determine the best predictor $f_{i}^{*}:\mathbf{x}_{i}\rightarrow\mathbf{y}_{i}$.

### 2.2 Formulation of TSF Transformers

In this section, we present the existing Temporal-wise and Series-wise Transformers for TSF from an ICL perspective. Building on this, we introduce the In-context Time Series Predictor (ICTSP). In §2.3, we discuss the issues within existing structures and the advantages of ICTSP for TSF problems.

Figure: Figure 2: Architecture and characteristic comparison among the three main TSF Transformer structures. Please note that, for simple illustration, the ICTSP part present a special case, where the sampling steps equal to $L_{b}+L_{P}$, creating non-overlapping context forecasting examples.
Refer to caption: x2.png

Temporal-wise Transformer  This type of method, as shown in Figure 2 (a), typically constructs input tokens based on the values of multiple time series channels at timesteps or within intervals along the temporal dimension. The format of these input tokens can be represented as:

$$ (3) $$

where $\mathbf{O}$ and $\{o_{1},\cdots,o_{C}\}$ represent zero-padding over the forecasting horizon. Again, the added positional embeddings $\mathbf{p}_{\cdot}$ are shown concatenated here just for illustration. Each token represents the time series channel structure at a specific timestep. Therefore, from the ICL perspective, the Transformer doesn’t learn the necessary prediction from $X_{I}$ to $X_{P}$ for TSF. Instead, it learns the mapping $f^{*}_{t}:\mathbf{p}_{t}\rightarrow[X^{(t)}_{1},\cdots,X^{(t)}_{C}]^{\top}$, which tries to find the underlying dynamic of the channel structure determined by the position from the context and applies it to the outputs. If true dependencies between the input series are lacking, which frequently occurs in real-world data, its focus on channel structure over temporal dependencies leads to significant overfitting. In §2.3, we will explain how existing methods like channel independence can partially address overfitting while introducing new issues. In the last part of §3, we will use examples to show that Temporal-wise Transformers perform well with strong channel dependencies but overfit when these dependencies are weak. Previous models using this architecture include LogTrans[], Informer [], Autoformer [], etc. Note that while some of these models use an encoder-decoder structure, their token construction based on multi-channel values along the temporal dimension shares the same issues we highlight.

##### Series-wise Transformer

Some recent methods [, ] transpose the input format above so each token represents an individual series, forming the Series-wise Transformer. This format, shown in Figure 2 (b), can be expressed as follows:

$$ (4) $$

However, iTransformer [] introduced this formulation believing it enhances modeling of inter-series dependencies compared to the Temporal-wise Transformer, which contrasts with our ICL analysis. Each token in the Series-wise Transformer represents an entire input series, capturing relationships between timesteps within that series. For series $j$, the Transformer learns a predictor $f^{*}_{j}:X_{j}^{(1:L_{I})}\rightarrow X_{j}^{(L_{I}+1:L_{I}+L_{P})}$, which also adjusts its own parameters to the inputs $\{X^{(1:L_{I})}_{i}\}_{i\neq j}$ from other series. Essentially, it acts more like a univariate predictor capable of adapting its parameters based on the context series. We show in the last part of §3 that the Series-wise Transformer is better suited for real-world datasets with weak multivariate relationships than those with strong inter-series dependencies, supporting our view from ICL. Also from this perspective, the Series-wise Transformer lacks ground truth context examples for the predictor, meaning it doesn’t fully utilize the Transformer’s ICL abilities. This limits its generalization and few-shot learning capabilities.

In-context Time Series Predictor  To fully exploit the Transformer’s ICL capabilities, we treat forecasting tasks, instead of timestep values or input series, as tokens. For each input series $j$, we use a lookback length $L_{b}<L_{I}$ to construct context examples based on input data $X^{(1:L_{I})}_{j}$. We perform stepwise sampling on input, obtaining $N=L_{I}-L_{b}-L_{P}$ ground truth training samples with their lookback and future parts span of length $L_{b}+L_{P}$. By combining each series’ output target token with the context examples to form each series’ input $\mathbf{H}_{j}$, we concatenate these into the multivariate input token matrix $\mathbf{Z}$ for the ICTSP as follows:

$$ $\begin{gathered}\mathbf{Z}^{(L_{b}+L_{P})\times(N+1)C}=[\mathbf{z}_{1},\ldots, \mathbf{z}_{(N+1)C}]=[\mathbf{H}_{1},\ldots,\mathbf{H}_{C}]\\ \mathbf{H}_{j}^{(L_{b}+L_{P})\times(N+1)}=\begin{bmatrix}\mathbf{p}_{j,1}&X_{j }^{(1)}&\ldots&X_{j}^{(L_{b})}&X_{j}^{(L_{b}+1)}&\ldots&X_{j}^{(L_{b}+L_{P})} \\ \mathbf{p}_{j,2}&X_{j}^{(2)}&\ldots&X_{j}^{(L_{b}+1)}&X_{j}^{(L_{b}+2)}&\ldots &X_{j}^{(L_{b}+L_{P}+1)}\\ \vdots&\vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\ \mathbf{p}_{j,N}&X_{j}^{(L_{I}-L_{P}-L_{b})}&\ldots&X_{j}^{(L_{I}-L_{P})}&X_{j }^{(L_{I}-L_{P}+1)}&\ldots&X_{j}^{(L_{I})}\\ \mathbf{p}_{j,\text{target}}&X_{j}^{(L_{I}-L_{b})}&\ldots&X_{j}^{(L_{I})}&o_{L _{I}+1}&\ldots&o_{L_{I}+L_{P}}\\ \end{bmatrix}^{\top}\end{gathered}$ (5) $$

where $\mathbf{p}_{j,\cdot}$ represents the positional embedding for each token on the $j$-th series illustrated as concatenation. Therefore, $\mathbf{Z}$ contains $(N+1)C$ tokens, with $NC$ context tokens. We can increase the step size in stepwise sampling to reduce computational costs, retaining one sample for every $m$ steps among $N$ samples per series. Figure 2 (c) shows a special case with no overlap between samples for simplicity, achievable when $m=L_{b}+L_{P}$. ICTSP flexibly forms ground truth forecasting examples to learn the predictor $f_{j}^{*}:X_{j}^{(t+1:t+L_{b})}\rightarrow X_{j}^{(t+L_{b}+1:t+L_{b}+L_{P})}$. Since ICTSP does not restrict the input channel structure (with $\mathbf{p}_{j,\cdot}$ serving only to distinguish) and can construct context examples during inference, it generalizes well across different multivariate datasets, exhibiting robust zero-shot learning capabilities.

Adaptive Model Reduction of ICTSP  Existing complex Temporal-wise Transformers often struggle to outperform simple baselines, like linear predictors, in noisy datasets lacking clear temporal-wise and channel-wise dependencies. In contrast, ICTSP adapts to different dataset complexities and reduces to simpler models when necessary. Specifically:
i) When temporal-wise dependencies are weak, ICTSP can adaptively select simpler functions for TSF, such as linear predictors and shallow MLPs, as achievable by ICL.
ii) When series patterns change greatly over time where the impact of context examples is weak, ICTSP can reduce to a Series-wise Transformer focusing only on local lookback of target tokens without context tokens.
iii) When local lookback from other series have minimal contribution, ICTSP degrades to a simple univariate MLP, in which case the inter-token information in attention term from Eq.(1) disappears, and token $j$ at layer $k$ becomes $\mathbf{z}_{j,k}=\mathbf{z}_{j,k-1}+\mathtt{LN}(\mathtt{FFN}_{j,k}(\mathbf{z}_
{j,k-1}))$. If $\mathtt{FFN}$ also becomes ineffective, ICTSP processes target tokens only with input and output linear projections outside the Transformer. At this stage, the model reduces to a linear predictor, effective for highly noisy time series data. ICTSP’s design provides appropriate shortcuts to enhance adaptability. We further explain in §A.4.1 how inappropriate shortcuts in the model reduction of the Temporal-wise Transformer limit its ability to learn dynamic patterns.

Sampling and Token Retrieval  We can reduce computational costs of ICTSP by sampling one context token every $m$ steps from the total $N$ tokens for each series. For very large token amount, to avoid losing information with large sampling steps, we can use token retrieval (TR) to further reduce token size. We employ a simple method where each token $\mathbf{z}_{i}$ is reduced to a $\delta$-dimensional latent vector $\tilde{\mathbf{z}}_{i}$ via a linear layer. We calculate the cosine similarity between the $\tilde{\mathbf{z}}_{i}$ pair of each context token and target token, averaging the results for each context token. We rank the context tokens by similarity, select the top $q\%$ as context examples, and merge the remaining tokens into $r$ tokens by grouped weighted averaging, resulting in $\lfloor q(L_{I}-L_{b}-L_{P})/m\rfloor C+r$ context tokens. TR reduces computational costs, preserves enough information, and helps the model infer which examples are more important with token positions. More details of the ICTSP structure can be found in §A.3.

### 2.3 Solving Key Issues of TSF Transformers

Figure: Figure 3: Previous solutions of Temporal-wise Transformers’ overfitting issue. From the ICL perspective, they are actually introducing more learnable temporal dependencies within token formulation.
Refer to caption: x3.png

Timestep Mixing and Overfitting  From the ICL perspective, the Temporal-wise Transformer’s token formulation overemphasizes channel relationships within specific timesteps. This leads to outputs overfitted to the non-existent underlying channel structures of context examples in real-world datasets with weak inter-series relationships, as shown in Figure 3 (a). Previous studies attribute this overfitting to the mixing of channel values at timesteps in the attention mechanism, proposing various solutions. The following analysis shows these partially effective solutions are actually trying to enhance the focus on temporal dependencies in the token formulation.

PatchTST [] suggested that univariate Temporal-wise Transformers, which inputs each channel independently (Figure 3 (b)), can address overfitting. This approach breaks the series-wise relationship in the tokens, transforming the ICL predictor into $f^{*}_{j,t}:\mathbf{p}_{t}\rightarrow X^{(t)}_{j}$, which predicts values solely from positions. This shift enhances the model’s focus on temporal dependencies while ignoring series-wise dependencies, making the formulation more suitable for real-world TSF tasks. Notably, the prompt structure used by recent LLMs for TSF, "… timestep: value …", also benefits from this approach.

ARM [] introduced random dropping (Figure 3 (c)), which mitigates overfitting by randomly removing series from both input and output during training. This randomly breaks some relationships between channels within the tokens. With only positional information consistently preserved, the model focus more on the temporal generation process and retains only the most likely channel dependencies, reducing overfitting to non-existent channel relationships. However, the degree of channel dropout requires further tuning based on different dataset characteristics.

Methods such as LogTrans [] and ARM [] use strategies similar to Figure 3 (d) to build local representations for tokens across time intervals instead of single timesteps. This introduces temporal relationships within the token. For TSF data with weak channel dependencies, tokens naturally focus on intra-series temporal relationships (solid lines in Figure 3 (d)) rather than the weak channel-wise relationships (dashed lines in Figure 3 (d)) as in the original token format. This approach alleviates overfitting but requires careful tuning of the local window selection.

These previous solutions emphasize temporal dependencies more in tokens but do not address the fundamental formulation issue. The Series-wise Transformer solve the issue by treating input series as tokens, helping each token focus on temporal dependencies but lacks contextual examples and ICL capabilities. In contrast, the ICTSP treats forecasting tasks as tokens, providing context forecasting examples to help the model learn temporal predictors and potential series dependencies through interactions between the target token and the historical context of all the series.

Permutation Invariance   Studies like DLinear [] and iTransformer [] suggest that the poor performance of Temporal-wise Transformers is partly due to the inherent permutation invariance issue in Transformer, meaning swapping the positions of input tokens does not significantly affect the output. The Temporal-wise Transformer treats context tokens more as representations of the underlying channel structure. Thus, when a dataset lacks a temporally varying underlying channel structure, $\mathbf{p}_{t}$ in $f^{*}_{t}:\mathbf{p}_{t}\rightarrow[X^{(t)}_{1},\cdots,X^{(t)}_{C}]^{\top}$ becomes ineffective, and the output tends to converge to the context’s average. This is consistent with DLinear observations [] where shuffling of input token minimally impact the output. Conversely, in the ICTSP, each context token represents a forecasting training example, so swapping their positions does not harm the forecasting. This makes the ICTSP better suited to the Transformer’s characteristics for TSF compared to previous structures.

Channel Structure Restriction  In the Temporal-wise Transformer, the number of input series and their relative positions within the tokens must be fixed. This limits the model’s ability to adapt to datasets with varying series numbers. Channel independence [] may simplify the Temporal-wise Transformer to a univariate model, enhancing generalization but losing the ability to model inter-series dependencies. The Series-wise Transformer does not restrict the number of input series structurally but learns the characteristics of $C$ series during training through $\mathbf{p}_{j}$, making it hard to transfer embeddings to other datasets. Conversely, the ICTSP does not restrict the number of series and includes in-context training examples within the input, enabling easy transfer to input data of any size. Here, positional embeddings can be trained to only distinguish between samples without describing the characteristics of the series. The flexibility in input channel structure provides zero-shot multivariate forecasting ability, which previous multivariate models could not achieve.

## 3 Experiments

We conduct comprehensive experiments under full-data, few-shot, and zero-shot settings using widely-used TSF datasets (details in §A.2), including ETTs [], Traffic, Electricity (ECL), and Weather. We use $K=3$ $\mathtt{TF}$ layers with $d=128$ and 8 heads. We set $L_{I}=1440$, $L_{b}=512$, and $L_{P}\in\{96,192,336,720\}$, performing 4 experiments for each dataset. We use sampling step $m=8$ and the token retrieval method with $q=10\%$, $r=30$ in main experiments. See §A.3.2 for more hyper-parameter and implementation details. We use the same experimental environment as [, ]. Extensive efforts are made to ensure fair comparisons across all experiments (detailed in §A.3.3).

Baselines We compare our method with previous SOTA models in the categories of LLM for TSF (Time-LLM [], GPT4TS [], LLMTime []), Temporal-wise Transformers (PatchTST [], FEDformer [], Autoformer [], Informer []), Series-wise Transformer (iTransformer []), CNN (TimesNet []), and simple methods (DLinear [], Last-value Repeat). Baseline results are sourced from [, ] when applicable and rerun for the missing experiments. We report the average test set MSE for each dataset in the main paper and provide the full raw results in §A.1. The average rank and the number of times each model is ranked as the best are also summarized in the results.

Full-data TSF Results We train the ICTSP from scratch on TSF datasets, with the results shown in Table 1. ICTSP consistently shows superior performance compared to previous methods in all categories. The ICL capability, activated by the context examples, allows it to excel on larger datasets with stable patterns like ECL and Traffic. Its adaptive model reduction ability, originated from its token and structure design, ensures strong performance on small and noisy datasets like ETTs.

**Table 1: Full-data TSF results. Averaged test set MSE on each dataset is reported. The best and second-best results are in bold and underlined, respectively. See Table 6 for the original results.**
| Models | ICTSP | Time-LLM | GPT4TS | iTransformer | PatchTST | FEDformer | Autoformer | Informer | DLinear | Repeat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ETTh1 | 0.404 | 0.408 | 0.428 | 0.454 | 0.413 | 0.440 | 0.496 | 1.040 | 0.423 | 1.321 |
| ETTh2 | 0.328 | 0.334 | 0.355 | 0.375 | 0.330 | 0.437 | 0.450 | 4.431 | 0.431 | 0.536 |
| ETTm1 | 0.342 | 0.329 | 0.352 | 0.374 | 0.351 | 0.448 | 0.588 | 0.961 | 0.357 | 1.269 |
| ETTm2 | 0.247 | 0.251 | 0.267 | 0.269 | 0.255 | 0.305 | 0.327 | 1.410 | 0.267 | 0.385 |
| Weather | 0.218 | 0.226 | 0.237 | 0.249 | 0.226 | 0.309 | 0.338 | 0.634 | 0.249 | 0.353 |
| ECL | 0.154 | 0.159 | 0.167 | 0.173 | 0.162 | 0.214 | 0.227 | 0.311 | 0.166 | 1.612 |
| Traffic | 0.386 | 0.388 | 0.414 | 0.420 | 0.391 | 0.610 | 0.628 | 0.764 | 0.434 | 2.770 |
| AvgRank | 1.36 | 1.93 | 4.29 | 5.64 | 2.79 | 6.86 | 7.89 | 9.43 | 5.00 | 9.50 |
| #Rank1 | 18 | 8 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 0 |

Few-shot Learning The few-shot learning ability of ICTSP is tested by training it on only the first 10% or 5% of the training data. ICTSP can flexibly build context samples based on the input data to perform in-context fitting without parameter updates, which compensates for the lack of training samples closer to the test set. Thus, it stably outperforms other models in both the 10% and 5% settings, as shown in Table 2. Note that we ensured ICTSP only uses datapoints perceivable by other baselines to build the context examples, ensuring a fair comparison, as detailed in A.3.3.

**Table 2: Few-shot learning results on 10% and 5% training data. The best and second best results are in bold and underlined, respectively. See Table 7 and 8 for the original results.**
| Models | ICTSP | Time-LLM | GPT4TS | PatchTST | FEDformer | Autoformer | Informer | TimesNet | DLinear | Repeat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ETTh1 (10%) | 0.525 | 0.555 | 0.590 | 0.633 | 0.639 | 0.702 | 1.199 | 0.869 | 0.691 | 1.321 |
| ETTh2 (10%) | 0.369 | 0.371 | 0.397 | 0.415 | 0.466 | 0.488 | 3.872 | 0.479 | 0.605 | 0.536 |
| ETTm1 (10%) | 0.397 | 0.404 | 0.464 | 0.501 | 0.722 | 0.802 | 1.192 | 0.677 | 0.411 | 1.269 |
| ETTm2 (10%) | 0.275 | 0.277 | 0.293 | 0.296 | 0.463 | 1.342 | 3.370 | 0.320 | 0.316 | 0.385 |
| Weather (10%) | 0.237 | 0.234 | 0.238 | 0.242 | 0.284 | 0.300 | 0.597 | 0.279 | 0.241 | 0.353 |
| ECL (10%) | 0.174 | 0.175 | 0.176 | 0.180 | 0.346 | 0.431 | 1.195 | 0.323 | 0.180 | 1.612 |
| Traffic (10%) | 0.428 | 0.429 | 0.440 | 0.430 | 0.663 | 0.749 | 1.534 | 0.951 | 0.447 | 2.770 |
| AvgRank (10%) | 1.57 | 1.89 | 3.21 | 4.07 | 6.39 | 7.46 | 9.43 | 6.68 | 4.93 | 9.21 |
| #Rank1 (10%) | 15 | 10 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| ETTh1 (5%) | 0.540 | 0.627 | 0.682 | 0.695 | 0.659 | 0.722 | 1.225 | 0.926 | 0.750 | 1.314 |
| ETTh2 (5%) | 0.368 | 0.382 | 0.401 | 0.439 | 0.441 | 0.470 | 3.923 | 0.464 | 0.828 | 0.519 |
| ETTm1 (5%) | 0.397 | 0.425 | 0.472 | 0.527 | 0.731 | 0.796 | 1.163 | 0.717 | 0.401 | 1.269 |
| ETTm2 (5%) | 0.285 | 0.274 | 0.308 | 0.315 | 0.381 | 0.389 | 3.659 | 0.345 | 0.399 | 0.385 |
| Weather (5%) | 0.256 | 0.261 | 0.264 | 0.270 | 0.310 | 0.311 | 0.584 | 0.298 | 0.264 | 0.353 |
| ECL (5%) | 0.175 | 0.177 | 0.179 | 0.181 | 0.267 | 0.346 | 1.281 | 0.402 | 0.177 | 1.612 |
| Traffic (5%) | 0.418 | 0.423 | 0.434 | 0.418 | 0.677 | 0.833 | 1.591 | 0.867 | 0.451 | 2.757 |
| AvgRank (5%) | 1.56 | 2.36 | 3.48 | 3.96 | 6.04 | 7.08 | 9.44 | 6.80 | 4.84 | 9.24 |
| #Rank1 (5%) | 12 | 7 | 1 | 2 | 0 | 0 | 0 | 0 | 4 | 0 |

Zero-shot Learning We also test the zero-shot transfer learning ability of ICTSP by training on one ETT dataset and testing on another ETT dataset. ICTSP surpasses all previous methods by a large margin in the zero-shot settings, as shown in Table 3. The new patterns in unseen datasets can be prompted with the forecasting task examples constructed in the tokens of ICTSP, which perfectly fits the scenario of zero-shot transfer learning.

**Table 3: Zero-shot transfer learning results. The best and second best results are in bold and underlined, respectively. See Table 9 for the original results.**
| Models | ICTSP | Time-LLM | LLMTime | GPT4TS | PatchTST | Autoformer | TimesNet | DLinear | Repeat |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ETTh1→ETTh2 | 0.337 | 0.352 | 0.992 | 0.407 | 0.381 | 0.582 | 0.421 | 0.494 | 0.536 |
| ETTh2→ETTh1 | 0.441 | 0.479 | 1.961 | 0.758 | 0.565 | 0.758 | 0.866 | 0.703 | 1.321 |
| ETTm1→ETTh2 | 0.356 | 0.381 | 0.992 | 0.433 | 0.439 | 0.470 | 0.454 | 0.464 | 0.536 |
| ETTm1→ETTm2 | 0.255 | 0.265 | 1.867 | 0.314 | 0.297 | 0.469 | 0.322 | 0.335 | 0.385 |
| ETTm2→ETTh2 | 0.352 | 0.354 | 0.992 | 0.435 | 0.409 | 0.423 | 0.435 | 0.456 | 0.536 |
| ETTm2→ETTm1 | 0.408 | 0.414 | 1.933 | 0.769 | 0.568 | 0.755 | 0.769 | 0.649 | 1.269 |
| AvgRank | 1.13 | 1.88 | 8.67 | 4.54 | 3.29 | 6.21 | 5.75 | 5.33 | 7.88 |
| #Rank1 | 21 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

Ablation Studies ICTSP has a concise model structure, with forecasting tasks as input tokens, enabling efficient transfer of ICL abilities to TSF. We conducted experiments on a) the inclusion of context examples, b) token retrieval (TR), and c) different sampling steps $m$, as shown in Table 4. Without context example tokens, ICTSP reduces to a Series-wise Transformer, lacking ground truth forecasting references, which significantly reduces performance. Token retrieval is used to lower computational costs while preserving model performance. As observed, using TR in the full model does not significantly reduce performance compared to the original model without TR. The sampling step $m$ controls the density of context samples that ICTSP can see. We aim to choose the largest $m$ that does not lose information to reduce computational cost. Selecting $m=8$ does not lose much performance compared to using full samples with $m=1$. while increasing $m$ to 256 degrades performance to the level close to the scenarios without context examples.

**Table 4: Results of the ablation studies of the ICTSP structure with $L_{P}\in\{96,192,336,720\}$.**
| Models | Full ($m=8$) | w/o Context | w/o TR | $m=1$ | $m=64$ | $m=256$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| Weather (96) | 0.139 | 0.194 | 0.150 | 0.202 | 0.140 | 0.200 | 0.140 | 0.196 | 0.140 | 0.194 | 0.142 | 0.195 |
| Weather (192) | 0.186 | 0.236 | 0.195 | 0.241 | 0.188 | 0.240 | 0.185 | 0.233 | 0.185 | 0.234 | 0.187 | 0.239 |
| Weather (336) | 0.239 | 0.279 | 0.245 | 0.280 | 0.241 | 0.287 | 0.240 | 0.284 | 0.239 | 0.281 | 0.240 | 0.283 |
| Weather (720) | 0.306 | 0.320 | 0.309 | 0.324 | 0.307 | 0.328 | 0.307 | 0.321 | 0.310 | 0.325 | 0.312 | 0.327 |
| Weather (Avg) | 0.218 | 0.257 | 0.225 | 0.262 | 0.219 | 0.264 | 0.218 | 0.259 | 0.219 | 0.259 | 0.220 | 0.261 |
| ETTm2 (96) | 0.159 | 0.248 | 0.162 | 0.249 | 0.159 | 0.248 | 0.160 | 0.249 | 0.162 | 0.249 | 0.162 | 0.250 |
| ETTm2 (192) | 0.212 | 0.289 | 0.215 | 0.295 | 0.211 | 0.287 | 0.209 | 0.289 | 0.213 | 0.287 | 0.214 | 0.288 |
| ETTm2 (336) | 0.268 | 0.326 | 0.273 | 0.329 | 0.268 | 0.330 | 0.270 | 0.327 | 0.269 | 0.327 | 0.273 | 0.327 |
| ETTm2 (720) | 0.347 | 0.382 | 0.352 | 0.387 | 0.345 | 0.383 | 0.345 | 0.381 | 0.348 | 0.384 | 0.350 | 0.386 |
| ETTm2 (Avg) | 0.247 | 0.311 | 0.251 | 0.315 | 0.246 | 0.312 | 0.246 | 0.312 | 0.248 | 0.312 | 0.250 | 0.313 |

Computational Costs In Table 5, we compare the computational costs between different settings of $m$ and token retrieval (TR), as well as between ICTSP and other TSF Transformers. The total number of context tokens is $\lfloor q(L_{I}-L_{b}-L_{P})/m\rfloor C+r$. Using $m=8$ and token retrieval with $q=10\%$ reduces the first part of tokens to $1.25\%$ of the original amount without losing much information, as shown above. These settings effectively reduce computational costs compared to the full context token scenario, making ICTSP more efficient than most of the previous TSF Transformers. Note that in our setting with a fixed input length $L_{I}$, longer forecasting horizon $L_{P}$ result in fewer context samples and may have less computational costs than shorter $L_{P}$. We do not include the computational costs of LLMs for TSF here, as they usually have thousands of times more parameters, making a direct comparison unnecessary.

**Table 5: Comparison of computational costs. The data format of ETTh2 is utilized to construct model inputs. The hyper-parameters for every model are set according to their default configurations.**
| Models | ICTSP ($m=8$, w/ TR) | ICTSP ($m=8$, w/o TR) | ICTSP ($m=1$, w/ TR) | ICTSP ($m=1$, w/o TR) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | FLOPs | Params | FLOPs | Params | FLOPs | Params | FLOPs | Params |
| $L_{P}=96$ | 1.51G | 7.35M | 11.2G | 7.34M | 9.58G | 7.35M | 88.0G | 7.34M |
| $L_{P}=192$ | 1.40G | 7.43M | 9.96G | 7.41M | 9.61G | 7.43M | 78.2G | 7.41M |
| $L_{P}=336$ | 1.20G | 7.54M | 8.09G | 7.52M | 8.14G | 7.54M | 63.2G | 7.52M |
| $L_{P}=720$ | 605M | 7.83M | 3.02G | 7.82M | 3.29G | 7.83M | 22.6G | 7.82M |
| Models | Autoformer | Informer | PatchTST | iTransformer |  |  |  |  |
| Metric | FLOPs | Params | FLOPs | Params | FLOPs | Params | FLOPs | Params |
| $L_{P}=96$ | 10.9G | 10.5M | 9.41G | 11.3M | 12.6G | 10.7M | 148M | 6.72M |
| $L_{P}=192$ | 11.6G | 10.5M | 10.1G | 11.3M | 12.7G | 15.2M | 149M | 6.77M |
| $L_{P}=336$ | 12.7G | 10.5M | 11.2G | 11.3M | 12.8G | 21.8M | 151M | 6.85M |
| $L_{P}=720$ | 15.5G | 10.5M | 14.0G | 11.3M | 13.2G | 39.5M | 155M | 7.04M |

Analysis of TSF Transformer Characteristics From the ICL perspective, the Temporal-wise Transformer is better for learning inter-series relationships, while the Series-wise Transformer better at intra-series relationships. Here, we use a synthesized "Multi" dataset from [] with strong inter-series dependencies, generated by shifting an random walk process. Fig. 4 (a) and (b) show that the Temporal-wise Transformer performs well on this dataset, while the Series-wise Transformer fails to learn the shifting dependencies. ICTSP, with its context tokens featuring a shifting structure, excels in learning these time-interleaved dependencies. Conversely, the Temporal-wise Transformer struggles with real-world datasets with weak dependencies comparing to Series-wise Transformer, such as the ETTm2 datasets in Fig. 4 (c) and (d). ICTSP handles these datasets well, demonstrating significantly better performance. Note that we build these baselines in this analysis with the same structure shown in Fig. 2, using $L_{I}=512$, $L_{P}=192$, $K=3$, $d=128$ for the 3 models; and $L_{b}=256$ for ICTSP. See §A.4.2 for more discussion about the inter-series learning abilities of the TSF Transformers.

Figure: (a) Multi (Test Loss)
Refer to caption: x4.png

Attention Visualization We visualize the attention maps of the three $\mathtt{TF}$ layers in ICTSP using the same forecasting data points from the Multi and ETTm2 datasets as above. We disable token retrieval to observe the interaction within and between the tokens from each time series. In Fig. 5 (a), it can be observed that the shifted series 2 and 3 fetch information from the corresponding inter-series context tokens. In Fig. 5 (b), ICTSP focuses on the first half of the context tokens, aligning with Fig. 4 (d), where the future shape resembles the earlier part of the input rather than the later part.

Figure: Figure 5: Visualization of averaged attention maps of the 3 $\mathtt{TF}$ layers of ICTSP on Multi and ETTm2.
Refer to caption: extracted/5613212/analysis/AttentionVisualization.png

## 4 Related Works

Extensive research on TSF has been conducted in both traditional statistics [, , ] and deep learning [, , ]. Recent studies have seen a surge in Transformer-based TSF models [, , , , , ]. However, a study [] indicated that these complex models might not outperform linear predictors on real-world datasets, leading to subsequent research [, , ] proposing possible solutions. Additionally, due to the significant achievements of LLMs in various tasks [], several LLM-based TSF models [, , ] have been introduced. These models also use Transformer architectures but rely on pre-trained parameters and fine-tuning to achieve superior few-shot and zero-shot capabilities []. While studies [, , , , ] have highlighted that these few-shot abilities in LLMs stem from ICL, there is a lack of research on how to efficiently implement ICL in TSF.

## 5 Conclusion and Limitation

In this study, we propose the In-context Time Series Predictor (ICTSP) to leverage in-context learning capabilities for time series forecasting. By treating "forecasting tasks" as tokens, we build context forecasting examples into the input to activate the ICL abilities. ICTSP addresses issues in TSF Transformers such as timestep mixing and overfitting with its effective token formulation, achieving SOTA performance across full-data, few-shot, and zero-shot experiments. For limitation, due to the lack of large collective TSF datasets, we have not tested ICTSP’s scaling ability with more layers on larger datasets. We also have not explored the possibility of training a single global model that can handle different lookback and future length, as allowed by the ICTSP structure.

## Appendix A Appendix

### A.1 Additional Experimental Results

In this section, we provide the detailed raw experimental results presented in the main paper. Table 6 presents the detailed results in the full-data setting. Tables 7 and 8 demonstrate the results of few-shot learning. Table 9 shows the results of zero-shot learning.

**Table 6: Full-data results with forecasting horizons $L_{P}\in\{96,192,336,720\}$. The test set MSE and MAE results are reported. The best and second best results are in bold and underlined, respectively.**
| Models | ICTSP | Time-LLM | GPT4TS | iTransformer | PatchTST | FedFormer | Autoformer | Informer | TimesNet | DLinear | Repeat |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 (96) | 0.366 | 0.393 | 0.362 | 0.392 | 0.376 | 0.397 | 0.396 | 0.428 | 0.370 | 0.399 | 0.376 | 0.419 | 0.449 | 0.459 | 0.865 | 0.713 | 0.384 | 0.402 | 0.375 | 0.399 | 1.295 | 0.713 |
| ETTh1 (192) | 0.399 | 0.412 | 0.398 | 0.418 | 0.416 | 0.418 | 0.431 | 0.451 | 0.413 | 0.421 | 0.420 | 0.448 | 0.500 | 0.482 | 1.008 | 0.792 | 0.436 | 0.429 | 0.405 | 0.416 | 1.325 | 0.733 |
| ETTh1 (336) | 0.426 | 0.427 | 0.430 | 0.427 | 0.442 | 0.433 | 0.459 | 0.470 | 0.422 | 0.436 | 0.459 | 0.465 | 0.521 | 0.496 | 1.107 | 0.809 | 0.491 | 0.469 | 0.439 | 0.443 | 1.323 | 0.744 |
| ETTh1 (720) | 0.424 | 0.446 | 0.442 | 0.457 | 0.477 | 0.456 | 0.528 | 0.523 | 0.447 | 0.466 | 0.506 | 0.507 | 0.514 | 0.512 | 1.181 | 0.865 | 0.521 | 0.500 | 0.472 | 0.490 | 1.339 | 0.756 |
| ETTh1 (Avg) | 0.404 | 0.420 | 0.408 | 0.424 | 0.428 | 0.426 | 0.454 | 0.468 | 0.413 | 0.431 | 0.440 | 0.460 | 0.496 | 0.487 | 1.040 | 0.795 | 0.458 | 0.450 | 0.423 | 0.437 | 1.321 | 0.737 |
| ETTh2 (96) | 0.265 | 0.335 | 0.268 | 0.328 | 0.285 | 0.342 | 0.299 | 0.358 | 0.274 | 0.336 | 0.358 | 0.397 | 0.346 | 0.388 | 3.755 | 1.525 | 0.340 | 0.374 | 0.289 | 0.353 | 0.432 | 0.422 |
| ETTh2 (192) | 0.326 | 0.374 | 0.329 | 0.375 | 0.354 | 0.389 | 0.365 | 0.399 | 0.339 | 0.379 | 0.429 | 0.439 | 0.456 | 0.452 | 5.602 | 1.931 | 0.402 | 0.414 | 0.383 | 0.418 | 0.534 | 0.473 |
| ETTh2 (336) | 0.349 | 0.396 | 0.368 | 0.409 | 0.373 | 0.407 | 0.407 | 0.429 | 0.329 | 0.380 | 0.496 | 0.487 | 0.482 | 0.486 | 4.721 | 1.835 | 0.452 | 0.452 | 0.448 | 0.465 | 0.591 | 0.508 |
| ETTh2 (720) | 0.370 | 0.397 | 0.372 | 0.420 | 0.406 | 0.441 | 0.427 | 0.454 | 0.379 | 0.422 | 0.463 | 0.474 | 0.515 | 0.511 | 3.647 | 1.625 | 0.462 | 0.468 | 0.605 | 0.551 | 0.588 | 0.517 |
| ETTh2 (Avg) | 0.328 | 0.376 | 0.334 | 0.383 | 0.355 | 0.395 | 0.375 | 0.410 | 0.330 | 0.379 | 0.437 | 0.449 | 0.450 | 0.459 | 4.431 | 1.729 | 0.414 | 0.427 | 0.431 | 0.447 | 0.536 | 0.480 |
| ETTm1 (96) | 0.280 | 0.336 | 0.272 | 0.334 | 0.292 | 0.346 | 0.325 | 0.376 | 0.290 | 0.342 | 0.379 | 0.419 | 0.505 | 0.475 | 0.672 | 0.571 | 0.338 | 0.375 | 0.299 | 0.343 | 1.214 | 0.665 |
| ETTm1 (192) | 0.318 | 0.361 | 0.310 | 0.358 | 0.332 | 0.372 | 0.356 | 0.391 | 0.332 | 0.369 | 0.426 | 0.441 | 0.553 | 0.496 | 0.795 | 0.669 | 0.374 | 0.387 | 0.335 | 0.365 | 1.261 | 0.690 |
| ETTm1 (336) | 0.355 | 0.383 | 0.352 | 0.384 | 0.366 | 0.394 | 0.382 | 0.405 | 0.366 | 0.392 | 0.445 | 0.459 | 0.621 | 0.537 | 1.212 | 0.871 | 0.410 | 0.411 | 0.369 | 0.386 | 1.283 | 0.707 |
| ETTm1 (720) | 0.416 | 0.415 | 0.383 | 0.411 | 0.417 | 0.421 | 0.432 | 0.434 | 0.416 | 0.420 | 0.543 | 0.490 | 0.671 | 0.561 | 1.166 | 0.823 | 0.478 | 0.450 | 0.425 | 0.421 | 1.319 | 0.729 |
| ETTm1 (Avg) | 0.342 | 0.374 | 0.329 | 0.372 | 0.352 | 0.383 | 0.374 | 0.402 | 0.351 | 0.381 | 0.448 | 0.452 | 0.588 | 0.517 | 0.961 | 0.734 | 0.400 | 0.406 | 0.357 | 0.379 | 1.269 | 0.698 |
| ETTm2 (96) | 0.159 | 0.248 | 0.161 | 0.253 | 0.173 | 0.262 | 0.187 | 0.281 | 0.165 | 0.255 | 0.203 | 0.287 | 0.255 | 0.339 | 0.365 | 0.453 | 0.187 | 0.267 | 0.167 | 0.269 | 0.266 | 0.328 |
| ETTm2 (192) | 0.212 | 0.289 | 0.219 | 0.293 | 0.229 | 0.301 | 0.238 | 0.316 | 0.220 | 0.292 | 0.269 | 0.328 | 0.281 | 0.340 | 0.533 | 0.563 | 0.249 | 0.309 | 0.224 | 0.303 | 0.340 | 0.371 |
| ETTm2 (336) | 0.268 | 0.326 | 0.271 | 0.329 | 0.286 | 0.341 | 0.285 | 0.344 | 0.274 | 0.329 | 0.325 | 0.366 | 0.339 | 0.372 | 1.363 | 0.887 | 0.321 | 0.351 | 0.281 | 0.342 | 0.412 | 0.410 |
| ETTm2 (720) | 0.347 | 0.382 | 0.352 | 0.379 | 0.378 | 0.401 | 0.365 | 0.393 | 0.362 | 0.385 | 0.421 | 0.415 | 0.433 | 0.432 | 3.379 | 1.338 | 0.408 | 0.403 | 0.397 | 0.421 | 0.521 | 0.465 |
| ETTm2 (Avg) | 0.247 | 0.311 | 0.251 | 0.314 | 0.267 | 0.326 | 0.269 | 0.334 | 0.255 | 0.315 | 0.305 | 0.349 | 0.327 | 0.371 | 1.410 | 0.810 | 0.291 | 0.333 | 0.267 | 0.334 | 0.385 | 0.394 |
| Weather (96) | 0.139 | 0.194 | 0.147 | 0.201 | 0.162 | 0.212 | 0.183 | 0.239 | 0.149 | 0.198 | 0.217 | 0.296 | 0.266 | 0.336 | 0.300 | 0.384 | 0.172 | 0.220 | 0.176 | 0.237 | 0.259 | 0.254 |
| Weather (192) | 0.186 | 0.236 | 0.189 | 0.234 | 0.204 | 0.248 | 0.225 | 0.273 | 0.194 | 0.241 | 0.276 | 0.336 | 0.307 | 0.367 | 0.598 | 0.544 | 0.219 | 0.261 | 0.220 | 0.282 | 0.309 | 0.292 |
| Weather (336) | 0.239 | 0.279 | 0.262 | 0.279 | 0.254 | 0.286 | 0.266 | 0.302 | 0.245 | 0.282 | 0.339 | 0.380 | 0.359 | 0.395 | 0.578 | 0.523 | 0.280 | 0.306 | 0.265 | 0.319 | 0.377 | 0.338 |
| Weather (720) | 0.306 | 0.320 | 0.304 | 0.316 | 0.326 | 0.337 | 0.322 | 0.341 | 0.314 | 0.334 | 0.403 | 0.428 | 0.419 | 0.428 | 1.059 | 0.741 | 0.365 | 0.359 | 0.333 | 0.362 | 0.465 | 0.394 |
| Weather (Avg) | 0.218 | 0.257 | 0.226 | 0.258 | 0.237 | 0.271 | 0.249 | 0.289 | 0.226 | 0.264 | 0.309 | 0.360 | 0.338 | 0.382 | 0.634 | 0.548 | 0.259 | 0.287 | 0.249 | 0.300 | 0.353 | 0.320 |
| ECL (96) | 0.127 | 0.221 | 0.131 | 0.224 | 0.139 | 0.238 | 0.145 | 0.249 | 0.129 | 0.222 | 0.193 | 0.308 | 0.201 | 0.317 | 0.274 | 0.368 | 0.168 | 0.272 | 0.140 | 0.237 | 1.588 | 0.946 |
| ECL (192) | 0.148 | 0.241 | 0.152 | 0.241 | 0.153 | 0.251 | 0.166 | 0.269 | 0.157 | 0.240 | 0.201 | 0.315 | 0.222 | 0.334 | 0.296 | 0.386 | 0.184 | 0.289 | 0.153 | 0.249 | 1.595 | 0.950 |
| ECL (336) | 0.158 | 0.257 | 0.160 | 0.248 | 0.169 | 0.266 | 0.176 | 0.270 | 0.163 | 0.259 | 0.214 | 0.329 | 0.231 | 0.338 | 0.300 | 0.394 | 0.198 | 0.300 | 0.169 | 0.267 | 1.617 | 0.961 |
| ECL (720) | 0.181 | 0.283 | 0.192 | 0.298 | 0.206 | 0.297 | 0.206 | 0.283 | 0.197 | 0.290 | 0.246 | 0.355 | 0.254 | 0.361 | 0.373 | 0.439 | 0.220 | 0.320 | 0.203 | 0.301 | 1.647 | 0.975 |
| ECL (Avg) | 0.154 | 0.251 | 0.159 | 0.253 | 0.167 | 0.263 | 0.173 | 0.268 | 0.162 | 0.253 | 0.214 | 0.327 | 0.227 | 0.338 | 0.311 | 0.397 | 0.193 | 0.295 | 0.166 | 0.264 | 1.612 | 0.958 |
| Traffic (96) | 0.359 | 0.251 | 0.362 | 0.248 | 0.388 | 0.282 | 0.381 | 0.266 | 0.360 | 0.249 | 0.587 | 0.366 | 0.613 | 0.388 | 0.719 | 0.391 | 0.593 | 0.321 | 0.410 | 0.282 | 2.723 | 1.079 |
| Traffic (192) | 0.372 | 0.256 | 0.374 | 0.247 | 0.407 | 0.290 | 0.410 | 0.278 | 0.379 | 0.256 | 0.604 | 0.373 | 0.616 | 0.382 | 0.696 | 0.379 | 0.617 | 0.336 | 0.423 | 0.287 | 2.756 | 1.087 |
| Traffic (336) | 0.382 | 0.273 | 0.385 | 0.271 | 0.412 | 0.294 | 0.431 | 0.281 | 0.392 | 0.264 | 0.621 | 0.383 | 0.622 | 0.337 | 0.777 | 0.420 | 0.629 | 0.336 | 0.436 | 0.296 | 2.791 | 1.095 |
| Traffic (720) | 0.432 | 0.291 | 0.430 | 0.288 | 0.450 | 0.312 | 0.458 | 0.299 | 0.432 | 0.286 | 0.626 | 0.382 | 0.660 | 0.408 | 0.864 | 0.472 | 0.640 | 0.350 | 0.466 | 0.315 | 2.811 | 1.097 |
| Traffic (Avg) | 0.386 | 0.268 | 0.388 | 0.264 | 0.414 | 0.295 | 0.420 | 0.281 | 0.391 | 0.264 | 0.610 | 0.376 | 0.628 | 0.379 | 0.764 | 0.416 | 0.620 | 0.336 | 0.434 | 0.295 | 2.770 | 1.090 |

**Table 7: Few-shot learning results on 10% training data with forecasting horizons $L_{P}\in\{96,192,336,720\}$. The best and second best results are in bold and underlined, respectively.**
| Models | ICTSP | Time-LLM | GPT4TS | PatchTST | FedFormer | Autoformer | Informer | TimesNet | DLinear | Repeat |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 (96) | 0.411 | 0.428 | 0.448 | 0.460 | 0.458 | 0.456 | 0.516 | 0.485 | 0.512 | 0.499 | 0.613 | 0.552 | 1.179 | 0.792 | 0.861 | 0.628 | 0.492 | 0.495 | 1.295 | 0.713 |
| ETTh1 (192) | 0.465 | 0.462 | 0.484 | 0.483 | 0.570 | 0.516 | 0.598 | 0.524 | 0.624 | 0.555 | 0.722 | 0.598 | 1.199 | 0.806 | 0.797 | 0.593 | 0.565 | 0.538 | 1.325 | 0.733 |
| ETTh1 (336) | 0.512 | 0.498 | 0.589 | 0.540 | 0.608 | 0.535 | 0.657 | 0.550 | 0.691 | 0.574 | 0.750 | 0.619 | 1.202 | 0.811 | 0.941 | 0.648 | 0.721 | 0.622 | 1.323 | 0.744 |
| ETTh1 (720) | 0.710 | 0.612 | 0.700 | 0.604 | 0.725 | 0.591 | 0.762 | 0.610 | 0.728 | 0.614 | 0.721 | 0.616 | 1.217 | 0.825 | 0.877 | 0.641 | 0.986 | 0.743 | 1.339 | 0.756 |
| ETTh1 (Avg) | 0.525 | 0.500 | 0.555 | 0.522 | 0.590 | 0.525 | 0.633 | 0.542 | 0.639 | 0.561 | 0.702 | 0.596 | 1.199 | 0.809 | 0.869 | 0.628 | 0.691 | 0.600 | 1.321 | 0.737 |
| ETTh2 (96) | 0.280 | 0.335 | 0.275 | 0.326 | 0.331 | 0.374 | 0.353 | 0.389 | 0.382 | 0.416 | 0.413 | 0.451 | 3.837 | 1.508 | 0.378 | 0.409 | 0.357 | 0.411 | 0.432 | 0.422 |
| ETTh2 (192) | 0.366 | 0.385 | 0.374 | 0.373 | 0.402 | 0.411 | 0.403 | 0.414 | 0.478 | 0.474 | 0.474 | 0.477 | 3.856 | 1.513 | 0.490 | 0.467 | 0.569 | 0.519 | 0.534 | 0.473 |
| ETTh2 (336) | 0.401 | 0.433 | 0.406 | 0.429 | 0.406 | 0.433 | 0.426 | 0.441 | 0.504 | 0.501 | 0.547 | 0.543 | 3.952 | 1.526 | 0.537 | 0.494 | 0.671 | 0.572 | 0.591 | 0.508 |
| ETTh2 (720) | 0.430 | 0.451 | 0.427 | 0.449 | 0.449 | 0.464 | 0.477 | 0.480 | 0.499 | 0.509 | 0.516 | 0.523 | 3.842 | 1.503 | 0.510 | 0.491 | 0.824 | 0.648 | 0.588 | 0.517 |
| ETTh2 (Avg) | 0.369 | 0.401 | 0.371 | 0.394 | 0.397 | 0.421 | 0.415 | 0.431 | 0.466 | 0.475 | 0.488 | 0.499 | 3.872 | 1.513 | 0.479 | 0.465 | 0.605 | 0.538 | 0.536 | 0.480 |
| ETTm1 (96) | 0.341 | 0.376 | 0.346 | 0.388 | 0.390 | 0.404 | 0.410 | 0.419 | 0.578 | 0.518 | 0.774 | 0.614 | 1.162 | 0.785 | 0.583 | 0.501 | 0.352 | 0.392 | 1.214 | 0.665 |
| ETTm1 (192) | 0.367 | 0.390 | 0.373 | 0.416 | 0.429 | 0.423 | 0.437 | 0.434 | 0.617 | 0.546 | 0.754 | 0.592 | 1.172 | 0.793 | 0.630 | 0.528 | 0.382 | 0.412 | 1.261 | 0.690 |
| ETTm1 (336) | 0.405 | 0.415 | 0.413 | 0.426 | 0.469 | 0.439 | 0.476 | 0.454 | 0.998 | 0.775 | 0.869 | 0.677 | 1.227 | 0.908 | 0.725 | 0.568 | 0.419 | 0.434 | 1.283 | 0.707 |
| ETTm1 (720) | 0.473 | 0.458 | 0.485 | 0.476 | 0.569 | 0.498 | 0.681 | 0.556 | 0.693 | 0.579 | 0.810 | 0.630 | 1.207 | 0.797 | 0.769 | 0.549 | 0.490 | 0.477 | 1.319 | 0.729 |
| ETTm1 (Avg) | 0.397 | 0.410 | 0.404 | 0.427 | 0.464 | 0.441 | 0.501 | 0.466 | 0.722 | 0.605 | 0.802 | 0.628 | 1.192 | 0.821 | 0.677 | 0.537 | 0.411 | 0.429 | 1.269 | 0.698 |
| ETTm2 (96) | 0.176 | 0.258 | 0.177 | 0.261 | 0.188 | 0.269 | 0.191 | 0.274 | 0.291 | 0.399 | 0.352 | 0.454 | 3.203 | 1.407 | 0.212 | 0.285 | 0.213 | 0.303 | 0.266 | 0.328 |
| ETTm2 (192) | 0.239 | 0.307 | 0.241 | 0.314 | 0.251 | 0.309 | 0.252 | 0.317 | 0.307 | 0.379 | 0.694 | 0.691 | 3.112 | 1.387 | 0.270 | 0.323 | 0.278 | 0.345 | 0.340 | 0.371 |
| ETTm2 (336) | 0.288 | 0.336 | 0.274 | 0.327 | 0.307 | 0.346 | 0.306 | 0.353 | 0.543 | 0.559 | 2.408 | 1.407 | 3.255 | 1.421 | 0.323 | 0.353 | 0.338 | 0.385 | 0.412 | 0.410 |
| ETTm2 (720) | 0.395 | 0.391 | 0.417 | 0.390 | 0.426 | 0.417 | 0.433 | 0.427 | 0.712 | 0.614 | 1.913 | 1.166 | 3.909 | 1.543 | 0.474 | 0.449 | 0.436 | 0.440 | 0.521 | 0.465 |
| ETTm2 (Avg) | 0.275 | 0.323 | 0.277 | 0.323 | 0.293 | 0.335 | 0.296 | 0.343 | 0.463 | 0.488 | 1.342 | 0.930 | 3.370 | 1.440 | 0.320 | 0.353 | 0.316 | 0.368 | 0.385 | 0.394 |
| Weather (96) | 0.164 | 0.214 | 0.161 | 0.210 | 0.163 | 0.215 | 0.165 | 0.215 | 0.188 | 0.253 | 0.221 | 0.297 | 0.374 | 0.401 | 0.184 | 0.230 | 0.171 | 0.224 | 0.259 | 0.254 |
| Weather (192) | 0.209 | 0.252 | 0.204 | 0.248 | 0.210 | 0.254 | 0.210 | 0.257 | 0.250 | 0.304 | 0.270 | 0.322 | 0.552 | 0.478 | 0.245 | 0.283 | 0.215 | 0.263 | 0.309 | 0.292 |
| Weather (336) | 0.259 | 0.294 | 0.261 | 0.302 | 0.256 | 0.292 | 0.259 | 0.297 | 0.312 | 0.346 | 0.320 | 0.351 | 0.724 | 0.541 | 0.305 | 0.321 | 0.258 | 0.299 | 0.377 | 0.338 |
| Weather (720) | 0.315 | 0.333 | 0.309 | 0.332 | 0.321 | 0.339 | 0.332 | 0.346 | 0.387 | 0.393 | 0.390 | 0.396 | 0.739 | 0.558 | 0.381 | 0.371 | 0.320 | 0.346 | 0.465 | 0.394 |
| Weather (Avg) | 0.237 | 0.273 | 0.234 | 0.273 | 0.238 | 0.275 | 0.242 | 0.279 | 0.284 | 0.324 | 0.300 | 0.342 | 0.597 | 0.495 | 0.279 | 0.301 | 0.241 | 0.283 | 0.353 | 0.320 |
| ECL (96) | 0.138 | 0.236 | 0.139 | 0.241 | 0.139 | 0.237 | 0.140 | 0.238 | 0.231 | 0.323 | 0.261 | 0.348 | 1.259 | 0.919 | 0.299 | 0.373 | 0.150 | 0.253 | 1.588 | 0.946 |
| ECL (192) | 0.153 | 0.252 | 0.151 | 0.248 | 0.156 | 0.252 | 0.160 | 0.255 | 0.261 | 0.356 | 0.338 | 0.406 | 1.160 | 0.873 | 0.305 | 0.379 | 0.164 | 0.264 | 1.595 | 0.950 |
| ECL (336) | 0.174 | 0.272 | 0.169 | 0.270 | 0.175 | 0.270 | 0.180 | 0.276 | 0.360 | 0.445 | 0.410 | 0.474 | 1.157 | 0.872 | 0.319 | 0.391 | 0.181 | 0.282 | 1.617 | 0.961 |
| ECL (720) | 0.229 | 0.318 | 0.240 | 0.322 | 0.233 | 0.317 | 0.241 | 0.323 | 0.530 | 0.585 | 0.715 | 0.685 | 1.203 | 0.898 | 0.369 | 0.426 | 0.223 | 0.321 | 1.647 | 0.975 |
| ECL (Avg) | 0.174 | 0.270 | 0.175 | 0.270 | 0.176 | 0.269 | 0.180 | 0.273 | 0.346 | 0.427 | 0.431 | 0.478 | 1.195 | 0.891 | 0.323 | 0.392 | 0.180 | 0.280 | 1.612 | 0.958 |
| Traffic (96) | 0.416 | 0.298 | 0.418 | 0.291 | 0.414 | 0.297 | 0.403 | 0.289 | 0.639 | 0.400 | 0.672 | 0.405 | 1.557 | 0.821 | 0.719 | 0.416 | 0.419 | 0.298 | 2.723 | 1.079 |
| Traffic (192) | 0.413 | 0.306 | 0.414 | 0.296 | 0.426 | 0.301 | 0.415 | 0.296 | 0.637 | 0.416 | 0.727 | 0.424 | 1.454 | 0.765 | 0.748 | 0.428 | 0.434 | 0.305 | 2.756 | 1.087 |
| Traffic (336) | 0.422 | 0.307 | 0.421 | 0.311 | 0.434 | 0.303 | 0.426 | 0.304 | 0.655 | 0.427 | 0.749 | 0.454 | 1.521 | 0.812 | 0.853 | 0.471 | 0.449 | 0.313 | 2.791 | 1.095 |
| Traffic (720) | 0.460 | 0.330 | 0.462 | 0.327 | 0.487 | 0.337 | 0.474 | 0.331 | 0.722 | 0.456 | 0.847 | 0.499 | 1.605 | 0.846 | 1.485 | 0.825 | 0.484 | 0.336 | 2.811 | 1.097 |
| Traffic (Avg) | 0.428 | 0.310 | 0.429 | 0.306 | 0.440 | 0.310 | 0.430 | 0.305 | 0.663 | 0.425 | 0.749 | 0.446 | 1.534 | 0.811 | 0.951 | 0.535 | 0.447 | 0.313 | 2.770 | 1.090 |

**Table 8: Few-shot learning results on 5% training data with forecasting horizons $L_{P}\in\{96,192,336,720\}$. The best and second best results are in bold and underlined, respectively. "-" means the lacking of data to build the training set in the 5% scenario.**
| Models | ICTSP | Time-LLM | GPT4TS | PatchTST | FedFormer | Autoformer | Informer | TimesNet | DLinear | Repeat |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| ETTh1 (96) | 0.499 | 0.468 | 0.483 | 0.464 | 0.543 | 0.506 | 0.557 | 0.519 | 0.593 | 0.529 | 0.681 | 0.570 | 1.225 | 0.812 | 0.547 | 0.503 | 0.892 | 0.625 | 1.295 | 0.713 |
| ETTh1 (192) | 0.550 | 0.497 | 0.629 | 0.540 | 0.748 | 0.580 | 0.711 | 0.570 | 0.652 | 0.563 | 0.725 | 0.602 | 1.249 | 0.828 | 0.720 | 0.604 | 0.940 | 0.665 | 1.325 | 0.733 |
| ETTh1 (336) | 0.572 | 0.500 | 0.768 | 0.626 | 0.754 | 0.595 | 0.816 | 0.619 | 0.731 | 0.594 | 0.761 | 0.624 | 1.202 | 0.811 | 0.984 | 0.727 | 0.945 | 0.653 | 1.323 | 0.744 |
| ETTh1 (720) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| ETTh1 (Avg) | 0.540 | 0.488 | 0.627 | 0.543 | 0.682 | 0.560 | 0.695 | 0.569 | 0.659 | 0.562 | 0.722 | 0.599 | 1.225 | 0.817 | 0.750 | 0.611 | 0.926 | 0.648 | 1.314 | 0.730 |
| ETTh2 (96) | 0.304 | 0.360 | 0.336 | 0.397 | 0.376 | 0.421 | 0.401 | 0.421 | 0.390 | 0.424 | 0.428 | 0.468 | 3.837 | 1.508 | 0.442 | 0.456 | 0.409 | 0.420 | 0.432 | 0.422 |
| ETTh2 (192) | 0.381 | 0.408 | 0.406 | 0.425 | 0.418 | 0.441 | 0.452 | 0.455 | 0.457 | 0.465 | 0.496 | 0.504 | 3.975 | 1.933 | 0.617 | 0.542 | 0.483 | 0.464 | 0.534 | 0.473 |
| ETTh2 (336) | 0.419 | 0.430 | 0.405 | 0.432 | 0.408 | 0.439 | 0.464 | 0.469 | 0.477 | 0.483 | 0.486 | 0.496 | 3.956 | 1.520 | 1.424 | 0.849 | 0.499 | 0.479 | 0.591 | 0.508 |
| ETTh2 (720) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| ETTh2 (Avg) | 0.368 | 0.399 | 0.382 | 0.418 | 0.401 | 0.434 | 0.439 | 0.448 | 0.441 | 0.457 | 0.470 | 0.489 | 3.923 | 1.654 | 0.828 | 0.616 | 0.464 | 0.454 | 0.519 | 0.468 |
| ETTm1 (96) | 0.331 | 0.372 | 0.316 | 0.377 | 0.386 | 0.405 | 0.399 | 0.414 | 0.628 | 0.544 | 0.726 | 0.578 | 1.130 | 0.775 | 0.332 | 0.374 | 0.606 | 0.518 | 1.214 | 0.665 |
| ETTm1 (192) | 0.363 | 0.376 | 0.450 | 0.464 | 0.440 | 0.438 | 0.441 | 0.436 | 0.666 | 0.566 | 0.750 | 0.591 | 1.150 | 0.788 | 0.358 | 0.390 | 0.681 | 0.539 | 1.261 | 0.690 |
| ETTm1 (336) | 0.413 | 0.418 | 0.450 | 0.424 | 0.485 | 0.459 | 0.499 | 0.467 | 0.807 | 0.628 | 0.851 | 0.659 | 1.198 | 0.809 | 0.402 | 0.416 | 0.786 | 0.597 | 1.283 | 0.707 |
| ETTm1 (720) | 0.481 | 0.472 | 0.483 | 0.471 | 0.577 | 0.499 | 0.767 | 0.587 | 0.822 | 0.633 | 0.857 | 0.655 | 1.175 | 0.794 | 0.511 | 0.489 | 0.796 | 0.593 | 1.319 | 0.729 |
| ETTm1 (Avg) | 0.397 | 0.410 | 0.425 | 0.434 | 0.472 | 0.450 | 0.527 | 0.476 | 0.731 | 0.593 | 0.796 | 0.621 | 1.163 | 0.792 | 0.401 | 0.417 | 0.717 | 0.562 | 1.269 | 0.698 |
| ETTm2 (96) | 0.181 | 0.265 | 0.174 | 0.261 | 0.199 | 0.280 | 0.206 | 0.288 | 0.229 | 0.320 | 0.232 | 0.322 | 3.599 | 1.478 | 0.236 | 0.326 | 0.220 | 0.299 | 0.266 | 0.328 |
| ETTm2 (192) | 0.244 | 0.310 | 0.215 | 0.287 | 0.256 | 0.316 | 0.264 | 0.324 | 0.394 | 0.361 | 0.291 | 0.357 | 3.578 | 1.475 | 0.306 | 0.373 | 0.311 | 0.361 | 0.340 | 0.371 |
| ETTm2 (336) | 0.301 | 0.378 | 0.273 | 0.330 | 0.318 | 0.353 | 0.334 | 0.367 | 0.378 | 0.427 | 0.478 | 0.517 | 3.561 | 1.473 | 0.380 | 0.423 | 0.338 | 0.366 | 0.412 | 0.410 |
| ETTm2 (720) | 0.412 | 0.410 | 0.433 | 0.412 | 0.460 | 0.436 | 0.454 | 0.432 | 0.523 | 0.510 | 0.553 | 0.538 | 3.896 | 1.533 | 0.674 | 0.583 | 0.509 | 0.465 | 0.521 | 0.465 |
| ETTm2 (Avg) | 0.285 | 0.341 | 0.274 | 0.323 | 0.308 | 0.346 | 0.315 | 0.353 | 0.381 | 0.405 | 0.389 | 0.434 | 3.659 | 1.490 | 0.399 | 0.426 | 0.345 | 0.373 | 0.385 | 0.394 |
| Weather (96) | 0.170 | 0.225 | 0.172 | 0.263 | 0.175 | 0.230 | 0.171 | 0.224 | 0.229 | 0.309 | 0.227 | 0.299 | 0.497 | 0.497 | 0.184 | 0.242 | 0.207 | 0.253 | 0.259 | 0.254 |
| Weather (192) | 0.219 | 0.267 | 0.224 | 0.271 | 0.227 | 0.276 | 0.230 | 0.277 | 0.265 | 0.317 | 0.278 | 0.333 | 0.620 | 0.545 | 0.228 | 0.283 | 0.272 | 0.307 | 0.309 | 0.292 |
| Weather (336) | 0.278 | 0.313 | 0.282 | 0.321 | 0.286 | 0.322 | 0.294 | 0.326 | 0.353 | 0.392 | 0.351 | 0.393 | 0.649 | 0.547 | 0.279 | 0.322 | 0.313 | 0.328 | 0.377 | 0.338 |
| Weather (720) | 0.358 | 0.371 | 0.366 | 0.381 | 0.366 | 0.379 | 0.384 | 0.387 | 0.391 | 0.394 | 0.387 | 0.389 | 0.570 | 0.522 | 0.364 | 0.388 | 0.400 | 0.385 | 0.465 | 0.394 |
| Weather (Avg) | 0.256 | 0.294 | 0.261 | 0.309 | 0.264 | 0.302 | 0.270 | 0.304 | 0.310 | 0.353 | 0.311 | 0.354 | 0.584 | 0.528 | 0.264 | 0.309 | 0.298 | 0.318 | 0.353 | 0.320 |
| ECL (96) | 0.145 | 0.240 | 0.147 | 0.242 | 0.143 | 0.241 | 0.145 | 0.244 | 0.235 | 0.322 | 0.297 | 0.367 | 1.265 | 0.919 | 0.150 | 0.251 | 0.315 | 0.389 | 1.588 | 0.946 |
| ECL (192) | 0.158 | 0.246 | 0.158 | 0.241 | 0.159 | 0.255 | 0.163 | 0.260 | 0.247 | 0.341 | 0.308 | 0.375 | 1.298 | 0.939 | 0.163 | 0.263 | 0.318 | 0.396 | 1.595 | 0.950 |
| ECL (336) | 0.177 | 0.273 | 0.178 | 0.277 | 0.179 | 0.274 | 0.183 | 0.281 | 0.267 | 0.356 | 0.354 | 0.411 | 1.302 | 0.942 | 0.175 | 0.278 | 0.340 | 0.415 | 1.617 | 0.961 |
| ECL (720) | 0.221 | 0.315 | 0.224 | 0.312 | 0.233 | 0.323 | 0.233 | 0.323 | 0.318 | 0.394 | 0.426 | 0.466 | 1.259 | 0.919 | 0.219 | 0.311 | 0.635 | 0.613 | 1.647 | 0.975 |
| ECL (Avg) | 0.175 | 0.269 | 0.177 | 0.268 | 0.179 | 0.273 | 0.181 | 0.277 | 0.267 | 0.353 | 0.346 | 0.405 | 1.281 | 0.930 | 0.177 | 0.276 | 0.402 | 0.453 | 1.612 | 0.958 |
| Traffic (96) | 0.406 | 0.293 | 0.414 | 0.291 | 0.419 | 0.298 | 0.404 | 0.286 | 0.670 | 0.421 | 0.795 | 0.481 | 1.557 | 0.821 | 0.427 | 0.304 | 0.854 | 0.492 | 2.723 | 1.079 |
| Traffic (192) | 0.415 | 0.295 | 0.419 | 0.291 | 0.434 | 0.305 | 0.412 | 0.294 | 0.653 | 0.405 | 0.837 | 0.503 | 1.596 | 0.834 | 0.447 | 0.315 | 0.894 | 0.517 | 2.756 | 1.087 |
| Traffic (336) | 0.432 | 0.311 | 0.437 | 0.314 | 0.449 | 0.313 | 0.439 | 0.310 | 0.707 | 0.445 | 0.867 | 0.523 | 1.621 | 0.841 | 0.478 | 0.333 | 0.853 | 0.471 | 2.791 | 1.095 |
| Traffic (720) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| Traffic (Avg) | 0.418 | 0.300 | 0.423 | 0.299 | 0.434 | 0.305 | 0.418 | 0.297 | 0.677 | 0.424 | 0.833 | 0.502 | 1.591 | 0.832 | 0.451 | 0.317 | 0.867 | 0.493 | 2.757 | 1.087 |

**Table 9: Zero-shot tranfer learning results between ETT datasets with forecasting horizons $L_{P}\in\{96,192,336,720\}$. The best and second best results are in bold and underlined, respectively.**
| Models | ICTSP | Time-LLM | LLMTime | GPT4TS | PatchTST | Autoformer | TimesNet | DLinear | Repeat |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| h1→h2 (96) | 0.272 | 0.335 | 0.279 | 0.337 | 0.510 | 0.576 | 0.335 | 0.374 | 0.304 | 0.350 | 0.469 | 0.486 | 0.358 | 0.387 | 0.347 | 0.400 | 0.432 | 0.422 |
| h1→h2 (192) | 0.330 | 0.373 | 0.351 | 0.374 | 0.523 | 0.586 | 0.412 | 0.417 | 0.386 | 0.400 | 0.634 | 0.567 | 0.427 | 0.429 | 0.447 | 0.460 | 0.534 | 0.473 |
| h1→h2 (336) | 0.359 | 0.402 | 0.388 | 0.415 | 0.640 | 0.637 | 0.441 | 0.444 | 0.414 | 0.428 | 0.655 | 0.588 | 0.449 | 0.451 | 0.515 | 0.505 | 0.591 | 0.508 |
| h1→h2 (720) | 0.387 | 0.431 | 0.391 | 0.420 | 2.296 | 1.034 | 0.438 | 0.452 | 0.419 | 0.443 | 0.570 | 0.549 | 0.448 | 0.458 | 0.665 | 0.589 | 0.588 | 0.517 |
| h1→h2 (Avg) | 0.337 | 0.385 | 0.352 | 0.387 | 0.992 | 0.708 | 0.407 | 0.422 | 0.381 | 0.405 | 0.582 | 0.548 | 0.421 | 0.431 | 0.494 | 0.489 | 0.536 | 0.480 |
| h2→h1 (96) | 0.411 | 0.426 | 0.450 | 0.452 | 1.130 | 0.777 | 0.732 | 0.577 | 0.485 | 0.465 | 0.693 | 0.569 | 0.848 | 0.601 | 0.689 | 0.555 | 1.295 | 0.713 |
| h2→h1 (192) | 0.435 | 0.447 | 0.465 | 0.461 | 1.242 | 0.820 | 0.758 | 0.559 | 0.565 | 0.509 | 0.760 | 0.601 | 0.860 | 0.610 | 0.707 | 0.568 | 1.325 | 0.733 |
| h2→h1 (336) | 0.448 | 0.461 | 0.501 | 0.482 | 1.328 | 0.864 | 0.759 | 0.578 | 0.581 | 0.515 | 0.781 | 0.619 | 0.867 | 0.626 | 0.710 | 0.577 | 1.323 | 0.744 |
| h2→h1 (720) | 0.469 | 0.481 | 0.501 | 0.502 | 4.145 | 1.461 | 0.781 | 0.597 | 0.628 | 0.561 | 0.796 | 0.644 | 0.887 | 0.648 | 0.704 | 0.596 | 1.339 | 0.756 |
| h2→h1 (Avg) | 0.441 | 0.454 | 0.479 | 0.474 | 1.961 | 0.981 | 0.758 | 0.578 | 0.565 | 0.513 | 0.758 | 0.608 | 0.866 | 0.621 | 0.703 | 0.574 | 1.321 | 0.737 |
| m1→h2 (96) | 0.299 | 0.359 | 0.321 | 0.369 | 0.510 | 0.576 | 0.353 | 0.392 | 0.354 | 0.385 | 0.435 | 0.470 | 0.377 | 0.407 | 0.365 | 0.415 | 0.432 | 0.422 |
| m1→h2 (192) | 0.361 | 0.394 | 0.389 | 0.410 | 0.523 | 0.586 | 0.443 | 0.437 | 0.447 | 0.434 | 0.495 | 0.489 | 0.471 | 0.453 | 0.454 | 0.462 | 0.534 | 0.473 |
| m1→h2 (336) | 0.376 | 0.412 | 0.408 | 0.433 | 0.640 | 0.637 | 0.469 | 0.461 | 0.481 | 0.463 | 0.470 | 0.472 | 0.472 | 0.484 | 0.496 | 0.494 | 0.591 | 0.508 |
| m1→h2 (720) | 0.388 | 0.431 | 0.406 | 0.436 | 2.296 | 1.034 | 0.466 | 0.468 | 0.474 | 0.471 | 0.480 | 0.485 | 0.495 | 0.482 | 0.541 | 0.529 | 0.588 | 0.517 |
| m1→h2 (Avg) | 0.356 | 0.399 | 0.381 | 0.412 | 0.992 | 0.708 | 0.433 | 0.440 | 0.439 | 0.438 | 0.470 | 0.479 | 0.454 | 0.457 | 0.464 | 0.475 | 0.536 | 0.480 |
| m1→m2 (96) | 0.168 | 0.255 | 0.169 | 0.257 | 0.646 | 0.563 | 0.217 | 0.294 | 0.195 | 0.271 | 0.385 | 0.457 | 0.222 | 0.295 | 0.221 | 0.314 | 0.266 | 0.328 |
| m1→m2 (192) | 0.223 | 0.293 | 0.227 | 0.318 | 0.934 | 0.654 | 0.277 | 0.327 | 0.258 | 0.311 | 0.433 | 0.469 | 0.288 | 0.337 | 0.286 | 0.359 | 0.340 | 0.371 |
| m1→m2 (336) | 0.273 | 0.329 | 0.290 | 0.338 | 1.157 | 0.728 | 0.331 | 0.360 | 0.317 | 0.348 | 0.476 | 0.477 | 0.341 | 0.367 | 0.357 | 0.406 | 0.412 | 0.410 |
| m1→m2 (720) | 0.355 | 0.384 | 0.375 | 0.367 | 4.730 | 1.531 | 0.429 | 0.413 | 0.416 | 0.404 | 0.582 | 0.535 | 0.436 | 0.418 | 0.476 | 0.476 | 0.521 | 0.465 |
| m1→m2 (Avg) | 0.255 | 0.315 | 0.265 | 0.320 | 1.867 | 0.869 | 0.314 | 0.349 | 0.297 | 0.334 | 0.469 | 0.485 | 0.322 | 0.354 | 0.335 | 0.389 | 0.385 | 0.394 |
| m2→h2 (96) | 0.288 | 0.351 | 0.298 | 0.356 | 0.510 | 0.576 | 0.360 | 0.401 | 0.327 | 0.367 | 0.353 | 0.393 | 0.360 | 0.401 | 0.333 | 0.391 | 0.432 | 0.422 |
| m2→h2 (192) | 0.352 | 0.395 | 0.359 | 0.397 | 0.523 | 0.586 | 0.434 | 0.437 | 0.411 | 0.418 | 0.432 | 0.437 | 0.434 | 0.437 | 0.441 | 0.456 | 0.534 | 0.473 |
| m2→h2 (336) | 0.371 | 0.415 | 0.367 | 0.412 | 0.640 | 0.637 | 0.460 | 0.459 | 0.439 | 0.447 | 0.452 | 0.459 | 0.460 | 0.459 | 0.505 | 0.503 | 0.591 | 0.508 |
| m2→h2 (720) | 0.398 | 0.445 | 0.393 | 0.434 | 2.296 | 1.034 | 0.485 | 0.477 | 0.459 | 0.470 | 0.453 | 0.467 | 0.485 | 0.477 | 0.543 | 0.534 | 0.588 | 0.517 |
| m2→h2 (Avg) | 0.352 | 0.402 | 0.354 | 0.400 | 0.992 | 0.708 | 0.435 | 0.444 | 0.409 | 0.426 | 0.423 | 0.439 | 0.435 | 0.444 | 0.456 | 0.471 | 0.536 | 0.480 |
| m2→m1 (96) | 0.371 | 0.395 | 0.359 | 0.397 | 1.179 | 0.781 | 0.747 | 0.558 | 0.491 | 0.437 | 0.735 | 0.576 | 0.747 | 0.558 | 0.570 | 0.490 | 1.214 | 0.665 |
| m2→m1 (192) | 0.387 | 0.411 | 0.390 | 0.420 | 1.327 | 0.846 | 0.781 | 0.560 | 0.530 | 0.470 | 0.753 | 0.586 | 0.781 | 0.560 | 0.590 | 0.506 | 1.261 | 0.690 |
| m2→m1 (336) | 0.416 | 0.427 | 0.421 | 0.445 | 1.478 | 0.902 | 0.778 | 0.578 | 0.565 | 0.497 | 0.750 | 0.593 | 0.778 | 0.578 | 0.706 | 0.567 | 1.283 | 0.707 |
| m2→m1 (720) | 0.456 | 0.444 | 0.487 | 0.488 | 3.749 | 1.408 | 0.769 | 0.573 | 0.686 | 0.565 | 0.782 | 0.609 | 0.769 | 0.573 | 0.731 | 0.584 | 1.319 | 0.729 |
| m2→m1 (Avg) | 0.408 | 0.419 | 0.414 | 0.438 | 1.933 | 0.984 | 0.769 | 0.567 | 0.568 | 0.492 | 0.755 | 0.591 | 0.769 | 0.567 | 0.649 | 0.537 | 1.269 | 0.698 |

### A.2 Datasets

Our main TSF experiments are conducted based on commonly used time series forecasting datasets, detailed as follows:

ETT Datasets(^1^11[https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset)) []: This dataset includes load and oil temperature data from electricity transformers, recorded at 15-minute intervals from July 2016 to July 2018. It comprises four subsets: ETTm1, ETTm2, ETTh1, and ETTh2, representing two transformers (identified as 1 and 2) and two time resolutions (15 minutes and 1 hour). Each subset contains seven features related to oil and load of the transformers.

Electricity Dataset(^2^22[https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014)): This dataset covers hourly electricity usage data of 321 consumers from 2012 to 2014 and is frequently used in energy consumption forecasting and analysis.

Traffic Dataset(^3^33[http://pems.dot.ca.gov/](http://pems.dot.ca.gov/)): Sourced from freeway sensors in the San Francisco Bay area, this dataset provides hourly road occupancy data from 2015 to 2016, making it a key resource for traffic flow studies.

Weather Dataset(^4^44[https://www.bgc-jena.mpg.de/wetter/](https://www.bgc-jena.mpg.de/wetter/)): This dataset captures 21 weather variables, such as temperature and humidity, recorded every 10 minutes throughout 2020, supporting detailed meteorological studies.

Multi Dataset [] is generated from a master random walk series. The first series is the master series, while the second to fifth series are created by shifting the master series backward by 96, 192, 336, and 720 steps, respectively. The last three series are combinations of these first five series.

### A.3 Implementation Details

#### A.3.1 More Details of Model Implementation

Token Construction Figure 6 provides another illustration of how to construct the input tokens in ICTSP. In the implementation, we first concatenate the output embedding to the input series. Then we cut the last $L_{b}+L_{P}$ part as the target tokens. We perform stepwise sampling from the end part to the initial part of $X_{I}$. If the remaining initial part of $X_{I}$ cannot constitute a token with the required length of $L_{b}+L_{P}$, we drop these initial steps in $X_{I}$.

Figure: Figure 6: Another illustration of the process of sampling the context examples from the input time series in ICTSP.
Refer to caption: x8.png

Trainable Embeddings As illustrated in Equation 5, we use two types of trainable positional embeddings in the token construction. One embedding represents the series source of a token, and the other represents the position of the token within the entire input. All the positional embeddings have the same $d$ dimension and are added to the latent tokens.

Randomized Training During training, we can introduce some random shifting $r\in\{0,\cdots,m-1\}$ to the sampling start point to expose the model to varied context examples. To prevent the positional embedding from learning information with specific time series channel, we randomly shuffle the order of series in the input to make the series show up in different position each training step. This can slightly enhance the performance in the few-shot and zero-shot tasks and for small datasets. Since ICTSP does not restrict the number of series in each input, we can randomly pick subsets of input series to build the tokens during training. This random selection slightly improves performance in few-shot learning with small datasets that have weak dependencies, while not significantly affecting performance on larger datasets with stable dependencies.

Token Retrieval We can reduce the computational costs of ICTSP by sampling one context token every $m$ steps from the total $N$ tokens for each series. For very large token amounts, to avoid losing information with large sampling steps, we use token retrieval (TR) to further reduce token size. Below, we provide a detailed description of the TR process.

Let $\mathbf{z}_{i}$ be the $i$-th token in the context. Each token $\mathbf{z}_{i}$ is reduced to a $\delta$-dimensional latent vector $\tilde{\mathbf{z}}_{i}$ via a linear layer:

$$ $\tilde{\mathbf{z}}_{i}=\mathbf{W}\mathbf{z}_{i}+\mathbf{b}$ (6) $$

where $\mathbf{W}\in\mathbb{R}^{\delta\times d}$ and $\mathbf{b}\in\mathbb{R}^{\delta}$ are the weights and biases of the linear layer. We calculate the cosine similarity between the $\tilde{\mathbf{z}}_{i}$ of each context token and each target token $\tilde{\mathbf{z}}_{c}$:

$$ $\mathtt{cosine_similarity}(\tilde{\mathbf{z}}_{i},\tilde{\mathbf{z}}_{c})= \frac{\tilde{\mathbf{z}}_{i}\cdot\tilde{\mathbf{z}}_{c}}{\|\tilde{\mathbf{z}}_ {i}\|\|\tilde{\mathbf{z}}_{c}\|}$ (7) $$

Next, we average the cosine similarity results for each context token:

$$ $\mathtt{average_similarity}(\mathbf{z}_{i})=\frac{1}{C}\sum_{c=1}^{C}\mathtt{ cosine_similarity}(\tilde{\mathbf{z}}_{i},\tilde{\mathbf{z}}_{c})$ (8) $$

where $C$ is the number of target tokens. We then rank the context tokens by their average similarity and select the top $q\%$ as context examples.

The remaining tokens are merged into $r$ tokens by grouped weighted averaging. Let $k=\lfloor qN\rfloor$ be the number of selected tokens, and let $p=N-k$ be the number of remaining tokens. The $r$ merged tokens are obtained as follows:

$$ $\mathbf{z}_{i}^{\text{merged}}=\sum_{j\in\mathcal{G}_{i}}\alpha_{j}\mathbf{z}_ {j}$ (9) $$

where $\mathcal{G}_{i}$ is the set of tokens in group $i$, determined by the ranked order, and $\alpha_{j}$ are the weights obtained from a softmax over the cosine similarities:

$$ $\alpha_{j}=\frac{e^{\mathtt{cosine_similarity}(\tilde{\mathbf{z}}_{j},\tilde{ \mathbf{z}}_{j^{\prime}})}}{\sum_{k\in\mathcal{G}_{i}}e^{\mathtt{cosine_ similarity}(\tilde{\mathbf{z}}_{k},\tilde{\mathbf{z}}_{j^{\prime}})}}$ (10) $$

The final set of context tokens includes the selected $k$ tokens and the $r$ merged tokens:

$$ $\mathbf{Z}^{\text{final}}=\{\mathbf{z}_{1},\ldots,\mathbf{z}_{k},\mathbf{z}_{1 }^{\text{merged}},\ldots,\mathbf{z}_{r}^{\text{merged}}\}$ (11) $$

This results in $\lfloor q(L_{I}-L_{b}-L_{P})/m\rfloor C+r$ context tokens.

Token Rationalization and Linear Layer Warm-up  ICTSP excels at handling datasets with varied characteristics and performing transfer learning across different datasets. The robustness of ICTSP, when facing small and noisy datasets, is further enhanced with following training strategies. For each context or target token $\mathbf{z}_{i}$, we can subtract the last value in its lookback from all values to align forecasting tasks across all tokens. This reduces the impact of mean shifting within or across series, serving as a naive repeat method baseline. Additionally, as mentioned earlier, the direct shortcut in Equation 1 helps ICTSP achieve a simple linear baseline in weak dependency scenarios. To prevent the attention and FFN terms from learning noise patterns too fast, we can initially train only the shortcut: for the first $W$ steps, remove the transformer layers and train only the input and output projections on the target tokens. After this warm-up, reintroduce the transformer blocks with context examples, gradually increasing the model’s complexity to adapt to small and noisy data. Figure 7 shows three cases: i) not using linear layer warm-up ($W=0$), ii) using linear layer warm-up ($W=5000$), and iii) only using the linear layer ($W=\infty$). We trained the ICTSP with these settings on ECL ($L_{P}=192$) with reduced input ($L_{I}=512,L_{b}=256$). The model with initialized weights for the input and output projection shortcut outperforms the models without warm-up and with only the linear layer as a baseline comparison. The linear layer warm-up stage structures the embedding space into a forecasting task, guiding the model to build tokens that focus on temporal forecasting power. The effectiveness of this warm-up stage is due to the token design and the shortcuts in the model structure.

Figure: Figure 7: The effects of using linear layer warm-up to initialize the embedding space as forecasting tasks with the input and output projection shortcut. Using a warm-up step $W=5000$ helps the model structure the embedding space as forecasting tasks before fitting the entire transformer structure, gaining performance margin and preventing overfitting. These three model are trained on ECL ($L_{P}=192$) with reduced ICTSP settings ($L_{I}=512,L_{b}=256$).
Refer to caption: x9.png

#### A.3.2 Hyperparameter Settings and Training Details

Model Training Setup The ICTSP model is trained using the Adam optimizer and MSE loss in Pytorch, with a
learning rate of 0.0005 each dataset. We test the model every 200 training steps with a early-stopping patience being $30\times 200$
steps. The first 1000 steps are for learning rate warm-up, followed by a linear decay of learning rate. We set the random seed as 2024. Our models are trained on single Nvidia RTX 4090 GPU with a batch size equals to 32 for most of the datasets. We decrease the batch size to 16 and 8 for the larger dataset ECL and Traffic, respectively. In the full-data experiment setting, we split each dataset with 70% training set, 10% validation, set and 20% test set. We fit a standardization scaler on the training set and apply it to the whole dataset. This setting is applied in consistent with previous studies like [, , , ].

Hyper-parameters of ICTSP For the hyper-parameters of ICTSP, we use $L_{I}=1440$ and $L_{b}=512$ for all the experiments. We use the pre-normalization Transformer with $K=3$ layers, $d=128$ latent dimension, and 8 heads. The hidden dimension of $\mathtt{FFN}$ is set to 4 times of latent dimension $d$ and the dropout rate of Transformer layers is set to 0.5. We use $m=8$ as the sampling step and $q=10\%,r=30$ for the token retrieval module.

Randomness of Training We test the randomness of training the ICTSP by alternating the random seed among {2022, 2023, 2024, 2025, 2026} and conducting multiple training runs on the same dataset settings. The error bars of the results with the five random seeds on the Weather and ECL datasets are shown in Figure 8, alongside the results of the two best previous baselines, Time-LLM [] and PatchTST []. The results show that ICTSP can stably surpass previous methods, even with the worst results among the five random seeds, indicating the robustness of ICTSP in performance improvement.

Figure: Figure 8: Randomness in Training ICTSP. The error bars for training ICTSP on the Weather and ECL datasets with random seeds {2022, 2023, 2024, 2025, 2026} are illustrated, compared with the previous best two baselines, Time-LLM and PatchTST. The results show that ICTSP can stably surpass previous baselines even in the worst training cases, demonstrating the robustness of the ICTSP structure.
Refer to caption: x10.png

#### A.3.3 Discussion of Fair Comparison

Our experimental environment aligns with the codes used by previous studies, including [, , , , , ]. Our few-shot and zero-shot learning setups follow [, ].

In the full-data experiment, for the input length $L_{I}$ setting, previous studies like [, ] suggest that $L_{I}$ should be considered a hyper-parameter that needs tuning for each dataset. Therefore, we chose to directly source their results where applicable, to avoid underestimating their performance when rerunning with a fixed $L_{I}$.

In the few-shot experiment, careful consideration of $L_{I}$ is very crucial. Since [, ] use the first 10% or 5% of the training set for few-shot training data, this represents a scenario where the model’s forecasting time is far from the training data collection. Thus, we should avoid leaking too much recent data to the model, especially for ICTSP. Because ICTSP can use historical input as in-context training samples, increasing $L_{I}$ too much might leak data that should be invisible to the model, defeating the purpose of the few-shot experiment.

Given the difference between $L_{I}=512$ used by [, ] and $L_{I}=1440,L_{b}=512$ for ICTSP, we chose to mask the part of the test set input data not visible to the $L_{I}=512$ setting in [, ] with zero-filling for ICTSP. This ensures that no unseen data for [, ] is used by ICTSP. As a result, there are no context examples for ICTSP (which reduces it to a Series-wise Transformer) at the very beginning of the test set input data, and the number of context examples gradually increases in the subsequent test set input data points. Thus, the few-shot learning results in Tables 2, 7, and 8 can be regarded as results from a weakened version of ICTSP, yet still achieving SOTA performance.

### A.4 Additional Analysis of TSF Transformers

#### A.4.1 Direct Shortcut In Temporal-wise Transformers

Revisiting the expression of Temporal-wise Transformer in Equation 3, it is clear that if context tokens become ineffective, the remaining MLPs or linear predictors are essentially guessing channel values and structures based solely on target positional embeddings. Past studies often found such Transformers to produce output resembling a flat line of input means rather than learning dynamic patterns, which could be explained by a failure of context and ineffective learning from positional embeddings.

#### A.4.2 More Discussion of Inter-series Modeling Ability

From the ICL perspective, the Series-wise Transformer primarily learns temporal dependencies rather than inter-series dependencies. This is evident in the "Multi" dataset with simple shifting inter-series dependencies, as shown in Figure 4 (b). This may indicate a lack of representation ability in the series token construction. Transformers can easily transfer information within the same token vector channel between different tokens via multi-head attention. When building the token example pool for ICTSP, a shifting process inherently exposes inter-series time-interleaved dependencies directly in the token representations, making it easy to capture these dependencies.

However, if these dependencies can be directly modeled into a single token dimension rather than between tokens, the Series-wise Transformer with enlarged tokens may still acquire inter-series learning ability alongside its inherent temporal learning ability. In a scenario with a very large latent dimension for the input token, where the token vector encompasses every shifting possibility of the input series, multi-head attention can find the alignment of possible inter-series time-interleaved dependencies between tokens by exploring this high dimension and fixing these alignments into the learned token embedding. This analysis suggests that increasing the token dimension of the Series-wise Transformer may enhance its inter-series modeling capability on specific datasets. Nonetheless, since these inter-series dependencies are learned and fixed in the token representation space, this capability cannot be transferred to other datasets in few-shot or zero-shot scenarios, unlike ICTSP.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The provided link to the code repository is inaccessible, which hinders the reproducibility of the results. Could you please update or fix the link to the source code?
soundness: 2
strengths: ICTSP model transforms time series forecasting tasks into input tokens, aligning closely with the inherent mechanisms of the Transformer model and efficiently leveraging its contextual learning capabilities.
summary: The paper introduces a model named "In-context Time Series Predictor" (ICTSP), designed specifically for time series forecasting (TSF). Unlike traditional Transformer-based models, ICTSP adopts a novel input token structure, resulting in improved performance.
weaknesses: 1. The innovation appears limited. While the paper extensively discusses the approach from an in-context learning perspective, the methodology itself seems simplistic and more akin to a heuristic trick due to excessive textual explanation without solid theoretical support.

2. The paper does not address whether this model could be applied to other time series tasks, such as classification, interpolation, or anomaly detection. The utility of the model appears confined to the forecasting tasks, necessitating stronger performance to justify its specialized use.

3. The significant improvements in zero-shot tasks over full-data tasks lack detailed explanations. The paper does not sufficiently clarify why the ICTSP demonstrates a pronounced advantage in few-shot and zero-shot learning scenarios than in full-data tasks

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Based on the listed weakness,
1. Can you add more discussions for these methods based on patching embedding from an ICL perspective?
2. For the Full-data TSF setting, can you rerun the baselines under the same experimental settings?  Alternatively, you could articulate the rationale behind your experimental settings, which would make the comparisons more convincing.
3. Can you provide more details for Computational Costs? What are the number of layers and hidden dimensions of different models?
soundness: 2
strengths: 1. The writing is clear and fluent.
2. This paper's motivation is relatively novel. It proposes utilizing the in-context learning capabilities of large language models (LLMs) for time series forecasting tasks. 
3. The experiments are comprehensive, encompassing various experimental settings and datasets.
summary: This paper presents the In-context Time Series Predictor(ICTSP), which utilizes the in-context capabilities for time series forecasting tasks. It proposes an innovative way to construct the tokens with context examples, showing promising results across multiple experimental settings.
weaknesses: 1. The formulation of TSF transformers is not sufficient. As far as I am concerned, there are some methods [1,2] that utilize patching embedding to form the input tokens. These methods cannot be simply categorized as Temporal-wise Transformer or Series-wise Transformer in Section 2.2

2. The baselines results for baselines are collected from source papers. However, this paper applies different input time series length from these papers, which may lead to an unfair comparison.

3. The details of Computational Costs and Table.5 are not provided. 


[1] Nie, Yuqi, et al. "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." The Eleventh International Conference on Learning Representations.

[2] Jin, Ming, et al. "Time-LLM: Time Series Forecasting by Reprogramming Large Language Models." The Twelfth International Conference on Learning Representations.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Could you provide results with ICTSP’s input length ($L_I$) set to 512 for a more equitable comparison with other baselines in Table 1 and 6?
2. Could you conduct an ablation study on in-context examples, possibly using randomly selected examples (without look-ahead bias) instead of adjacent history to clarify the influence of this design choice?
3. Could you provide more detailed analysis or visualizations related to the Adaptive Model Reduction to support its impact on performance?
4. How does ICTSP handle cross-channel dependencies, especially in cases with specific inter-channel relationships in the data (like traffic)?
soundness: 3
strengths: 1. **Novel Framework**: The paper proposes an innovative approach to time series forecasting by adopting in-context learning, which could inspire further research and development in this field.
2. **Comprehensive Experiments**: The authors present extensive experimental results, including ablation studies, which provide a robust foundation for evaluating ICTSP’s effectiveness.
3. **Clarity of Presentation**: The paper is well-written and clearly presented, facilitating a good understanding of the approach and its contributions.
summary: This paper present In-context Time Series Predictor (ICTSP), a new framework of doing time series forecasting. It formulates the time series prediction task into the form of in-context learning, such that in the input data is split into a few forecasting examples to guide the model complete the target prediction with the adjacent input. Experimental results demonstrate the method’s effectiveness.
weaknesses: 1. **Comparative Fairness**: The baselines use an input length of 512, while ICTSP utilizes an input length of 1440. This discrepancy could impact the fairness of the comparisons.
2. **In-context Example Selection**: ICTSP frames forecasting as an in-context learning task, but it uses adjacent historical data as context examples. This approach raises questions about whether the model truly learns from in-context examples or simply encodes the input tokens in a different manner.
3. **Model Reduction Analysis**: Although the paper highlights the adaptive model reduction feature of ICTSP, there is limited visualization or analysis to support this claim.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: N/A
soundness: 3
strengths: In-context learning for time series forecasting is an important problem with wide applications. As far as I can tell, using "time series forecasting tasks" as input tokens is a new and interesting idea, and the authors demonstrate that it indeed has utility using systematic benchmarks. The approach has a "category theory" flavor to it, and the differences from earlier frameworks are well explained.
summary: The authors propose a new in-context learning framework for time series forecasting by using "time series forecasting tasks" as input tokens. They show that this new framework outperforms the more traditional Temporal-wise Transformers and Series-wise Transformers across full-data, few-shot, and zero-shot settings.
weaknesses: In terms of presentation, I think there is room to make it more accessible to readers not directly working in the same field. In particular, more detailed explanation and illustration of the "using time series forecasting tasks as input tokens" idea and how it is implemented could be helpful.

Meta-review: This paper proposes an in-context time-series predictor, a unified formulation integrating temporal-wise and series-wise Transformers for time-series forecasting. This general formulation encompasses several existing models as special cases, thereby offering insights into their limitations. The experiments demonstrate superior performance across full-data, few-shot, and zero-shot settings.

One limitation of this work, raised by Reviewer H6bM, is that it does not sufficiently isolate the contribution of in-context learning capability and longer lookback window ot its strong performance. AC think this is acceptable, because, unlike in NLP, context samples from distinct time series often differ significantly in distribution, potentially hindering in-context learning. While Reviewer hkWk also holds a similar concern, he/she arguments that a Transformer trained from scratch, rather than pretrained on large-scale data, should not exhibit ICL capability. This stance contradicts a growing consensus in the community, and the authors provide extensive literature support for their position. 

Given that in-context learning remains a particularly important but underexplored topic in time-series forecasting, and considering this paper’s systematic literature discussion and solid experimental results, I recommend acceptance.

Decision: Accept (Poster)