In-context Time Series Predictor
Authors: Jiecheng Lu
Yan Sun
Shihao Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series Forecasting
In-context Learning
Transformer
Primary area: learning on time series and dynamical systems

Abstract: Recent Transformer-based large language models (LLMs) demonstrate in-context learning ability to perform various functions based solely on the provided context, without updating model parameters. To fully utilize the in-context capabilities in time series forecasting (TSF) problems, unlike previous Transformer-based or LLM-based time series forecasting methods, we reformulate "time series forecasting tasks" as input tokens by constructing a series of (lookback, future) pairs within the tokens. This method aligns more closely with the inherent in-context mechanisms and is more parameter-efficient without the need of using pre-trained LLM parameters. Furthermore, it addresses issues such as overfitting in existing Transformer-based TSF models, consistently achieving better performance across full-data, few-shot, and zero-shot settings compared to previous architectures.

Published as a conference paper at ICLR 2025
IN-CONTEXT TIME SERIES PREDICTOR
Jiecheng Lu, Yan Sun, Shihao Yang Georgia Institute of Technology (ilu414, 4, yansun (arrssunnnggeeeeee egatech..die shihao. s langgisye.gatecce
ABSTRACT
Recent Transformer-based large language models (LLMS) demonstrate in-context learning ability to perform various functions based solely on the provided context, without updating model parameters. To fully utilize the in-context capabilities in time series forecasting (TSF) problems, unlike previous Transformer-base or LLM-based time series forecasting methods, we reformulate "time series fore- casting tasks" as input tokens by constructing a series of (lookback, future) pairs within the tokens. This method aligns more closely with the inherent in-context mechanisms, and is more parameter-efficiet without the need of using pre-trained LLM parameters. Furthermore, it addresses issues such overfitting in existing Transformer- based TSF models, consistently achieving better performance across full-data, few-shot, and zero-shot settings compared to previous architectures !
INTRODUCTION
Transformer-based large language models (LLMs) have significantly impacted various research and application areas (Brown et al., 2020). Their inherent in-context learning (ICL) capabilities
highlighted by previous studies (Müller et al. 2022 Min et al., 2022 Wei al. 2023 Xie al.) 2022) allow them to adapt and generalize from context examples provided in input prompts without any parameter updates. This enables L.MMs to effectively handle few-shot and zero-shot tasks. Recent research on ICI. (Zhang et al.) 2023 Garg et al. 2022 Akyürek et al. 20233 Li et al. [0233] Dai et al., 2023 Bai et al. 2023) has shown that Transformers can adaptively learn to perform various functions including linear predictors, shallow MLPs, gradient descent, algorithm selection, etc., based series of (input, label) pairs as input tokens, as shown in Figure (a).
Time series forecasting (TSF) is criti- cal in fields like epidemiology, finance, and traffic (Kaastra & Boyd] 1996 Lana let al. 2018: Yang et al. [2015] Ma et al. 2022). predicting future values from his- torical data. Temporal-wist Transform- ers, which build input tokens from se- ries values at each timestep. have been widely researched in TSF (Li al.) 2019]Zhou et al. 2021 Wu et al. [021]] Nie al. 2022). Some studies have
Conteet Examople
Target
Contoot Ferscasing Exaopl Tokone
Tarae
Yang:
Input
Lookback
X
Label
1/1
(4) In-content Learning
In-contect
Serize Predictor
Figure Overview of in-context TSF learning in our setup.
identified several issues with such Transformers, like timestep mixing and permutation invariance
(Zeng et al. 2023). Icading to overfitting and undcrperformance on real-world datasets compared to
simpler models like linear predictors. Previously proposed solutions include channel independence
(Nie et al.
2022), random channel dropout (Lu et al.) 2024), and using Beries-wise Transformers
(Liu al.
2024a Wang al. 2024). which consider each series token. Yet, their underlying
mechanisms are not well explained. Additionally, the fixed structure of input series of these existing Transformers restricts their adaptability in few-shot and zero-shot learning for multivariate TSF.
Moreover, recent research have expanded the application of Trransformer-basedd LLMs to TSF (Zhang al. ) achieving improvements in few-shot and zero-shot generalization. They the methods like prompt engineering (Gruver et al. 2023), fine-tuning and embedding inversion (hou et al. 2023
Code implementation available http://aaayymnnsssaoonsssccceees

---

Published as a conference paper at ICLR 2025
Jin et al. 2024) to integrate time series context into prompts, improving forecasting accuracy for new data. However, these methods are designed to adapt LLMs to TSF tasks rather than directly addressing the core aspects of TSF problems. This adaptation leads to inefficient use of LLM parameters and substantially increased computational costs.
In this study, we apply the most fundamental ICL settings in TSF problems construct the In-context Time Series Predictor (ICTSP). This structure allows us to leverage the in-context generalization capabilities of Transformers efficiently without relying on large-scale LLM parameters. In ICTSP, we generate a sequence of (lookback, future) pairs, representing "time series forecasting tasks," from the original time series data, which are used as input tokens for the Transformer, as shown (b). This setup enables the Transformer to adaptively learn the most effective predictor for the target tasks based on the ground truth forecasting examples as context. Additionally, the ICTSP effectively resolves aforementioned longstanding issues in previous TSF Trasnformers, significantly enhancing the performance few-shot and zero-shot learning scenarios in multivariate TSF settings.
The main contributions of this paper are summarized as follows:
a) We innovatively use forecasting tasks - instead of traditional timestep values or single series as input tokens to construct the ICTSP structure. By utilizing ground truth (lookback, future) pairs as context examples, we fundamentally and efficiently leverage ICI. abilities for TSF tasks. ICTSP outperforms previous methods across full-data, few-shot, and zero-shot scenarios, positioning it as potential solution for building universal large TSF models.
b) From an ICL perspective, explain that issues in existing TSF Transformers, like timestep mixing, permutation invariance, and channel structure restriction, are caused by inappropriate token formulation. We show how previous solutions have partially addressed these issues and how ICTSP effectively solves them without the drawbacks of previous approaches.
c) We show that the ICTSP structure encompasses several simpler models as special cases, allowing for sequential adaptive reduction in complexity: i) predictors learned from context examples through ICL, ii) a series-wise Transformer without context examples (Liu al. 2024a), and iii) univariate MLPs or linear predictors ZZngg et al., (023). This connection ensures stable performance across different complexities of time series dataset and prevents the significant overfitting that has previously hindered Transformers from outperforming simpler models consistently in real-world datasets.
2
METHOD
2.1
PRELIMINARIES
Time Series Forecasting Let X E RCxL represent multivariate time series, where L is the total length and C is the number of channels of input series. X split into historical input X, E C' and future data Xp E RRp,,, with L = L, + Lp. Here, L, and Lp represent the lengths of the historical and forecasting periods, respectively. The value at the t-th timestep in the j-th channel is x where t E fl....,L L} and j E (I. The objective is to develop the best predictor f: CC + RCXLe that maps historical inputs to future outputs, yielding X = f(x))
Transformer Architecture In this study, we employ Transformer architecture with pre- normalization settings, processing D-dimensional input tokens { from the input matrix DDxN [ai..... 2N] E RDN,, Each token z, passes through K layers of the Transformer. Each layer begins with layer normalization LN(-), followed by self-attention Attn(.) () and a feed-forward network FF.(.). The output Zy for each Transformer layer, TF is computed as:
Z = TF. TF (Z+-1) = Z + Attn (LN(Zz,))) LN (FFN, (Z.-1 + Attn, (LN(Z)-1))))
(I)
with Z, Z and the first addition of Zę-1 providing residual shorteut directly from input to output. The Transformer's 's final output is TF())) = Zk.
In-context Learning ICL involves each datapoint (x,, ys) Ra R² from dataset ((x,, ys) ) 1= Unlike traditional supervised learning that learns direct mappings from x; to y., ICL predicts y; using both historical observations ((x;, and the current input x;. The input format for the
2

---

Published as a conference paper at ICLR 2025
Category
Temporal-wise Transformer
(b) Series-wise Transformer
In-context Time Series Predictor
Ox Lp
CxLp
Cx
Forecusting Resulte
Tram
Output Tokons and Projection
x
dx
Transformer Layers
Transformer Layers
Transformer Layers
d
C-3
Contost
4x
Step 1. Projection
Projction
SE
CX
Cx
Ox Oxx
(0 dxo 2.
(0x S) T dxx00
0-8
2. Cp
J. Lp)
Shape
(0xx hp)
Idontfficaton
Takens
Takana
foed-eeedd Channal Structure:
Non-fized Charnel Structare:
Non food Channel Birecture:
With Contest Exsmples:
Contact Esamples:
Contect Exemples:
Characteristice
Nultvvrrttts Generniization Abilt:: X
Muttvaritte Generaltzztio Abllity:
Mattivariate Generalization Ability
LogTrane, Informer, Autoformer,
Examples
FedFormer,
Figure 2: Architecture and characteristic comparison among the three main TSF Transformer struc- tures. Please note that, for simple illustration, the ICTSP part present a special case, where the sampling steps equal to Lb + Lp,, creating non-overlapping context forecasting examples.
i-th datapoint in the ICL settings is structured as:
P1
P2
P
p
Z (a+b)xi= 121....,24
X1
X2
X;-1
X;
(2)
Y1
Y2
Vi-1
where p. are positional embeddings and is placeholder embedding for the target, typically zero- filled. Note that in some ICL settings, x. and y. may be divided into separate tokens, which is proved interchangeable with our expression above (Bai et al., 2023 Guo al. 2024). The positional embeddings are added to the tokens and shown concatenated here just for illustration. Finally, the predicted output for the i-th data point, yc. can be extracted from the corresponding token in transformer's output 2 = TZ( For simplicity, we have omitted the input and output linear projections for matrices Z and Z this description. In practice, these projections map the tokens to a d-dimensional latent space. The computation is formulated as 2 = W our TF W Z + b + bouts where Win in E (x++6) and W... E R(++)) xd are the linear layer weights, and b and b are the biases. The ICL settings allow the model use (x,, 1; 1 x;} determine the best predictor J x; yi.
2.2 FORMULATION OF TSF TRANSFORMERS
In this section, present the existing Temporal-wise and Series-wise Transformers for TSF from ICL perspective. Building on this, we introduce the In-context Time Series Predictor (ICTSP). In diseuss the issues within existing structures and the advantages of ICTSP for TSF problems.
Temporal-wise Transformer This type of method, as shown Figure 2 input tokens based the values of multiple time series channels at timesteps within intervals
(a), typically constructs
3

---

Published as a conference paper at ICLR 2025
along the temporal dimension. The format of these input tokens be represented as:
Pi x
PL
Pl
x x(e)
P1s., X1 0
01
91
zexl = (71...., .,,,,,
(3)
X (L)))
oC
where O and {or,-.. , represent zero-padding over the forecasting horizon. Again, the added positional embeddings p. are shown concatenated here just for illustration. Each token represents the time series channel structure Transformer doesn't 't learn the necessary prediction from X, to Xp for TSF. Instead, it learns the
specific timestep. Therefore, from the ICL perspective, the
XL which tries to find the underlying dynamic of the channel structure determined by the position from the context and applies it to the outputs. If true dependencies between the input series are lacking, which frequently occurs in real-world data, its focus channel structure over temporal dependencies leads to significant overfitting, In $2.3] we will explain how existing methods like channel independence can partially address overfitting while introducing new issues. In the last part of $ we will use examples to show that l'emporalwwise Transformers perform well with strong channel dependencies but overfit when these dependencies are weak. Previous models using this architecture include Lranrans(Li et al.] 2019), Informer (Zhou et al. (021), Autoformer (Wu al. 2021 etc. Note that while some of these models use an encoder-decoder structure, their token construction based on multi-channel values along the temporal dimension shares the same issues we highlight.
mapping f P: [x[
Series-wise Transformer Some recent methods (Liu al. 2024a Wang 2024 transpose the input format above so each token represents an individual series, forming the Series- wise Transformer. This format, shown in Figure (b), can be expressed as follows:
P
p
(
01,
or.
L [Z)...., acl =
X
=
(4)
0
However, iTransformer (Liu et al. 2024a) introduced this formulation believing enhances modeling of inter-series dependencies compared to the Temporal-wise Transformer, which contrasts with our ICL. analysis. Each token the Series-wise Transformer represents an entire input series, capturing relationships between timesteps within that series. For series j. the Transformer learns a predictor fj (((1:t.)) + x (flo which also adjusts its own parameters to the inputs (xx(1:11)) }:45 from other series. Essentially, acts more like a univariate predictor capable of adapting its parameters based on the context series. It is capable of modeling temporally-aligned inter-series dependencies, but it is difficult to model shifted dependencies across timesteps. Similar to supervised learning, ICL treats each input within token independent features. This means that values across different timesteps are considered distinct features. The algorithm or model generated by ICL excels building relationships within input features and build relationship on the same feature (i.e. same timestep) across different data points (different series here) by performing linear combinations. However, is less inclined establish relationships between different features across different data points. In traditional supervised learning settings, different features from different data points are usually unrelated, so no relationship built, but this contradicts the requirements of multivariate TSF settings when using this series-as-daad crriss-s-datapoint setting. We show in the last part of 33 the Series-wist Transformer is better suited for real-world datasets with weak multivariate relationships than those with strong shifted inter-series dependencies, supporting ICL. Also from this perspective, the Series-wise Transformer lacks ground truth context examples for the predictor, meaning it doesn't 't fully utilize the Transformer's ICL abilities. This limits its generalization and few-shot learning capabilities.
imeestpss.Smiiarr
view from
In-context Time Series Predictor
To fully exploit the Transformer's ICL capabilities, we treat forecasting tasks, instead of timestep values or input series, as tokens. For each input series j. we use lookback length Lb L1 construct context examples based on input data x(!l We perform stepwise sampling on input, obtaining N = L, - L,, Lp ground truth training samples with their lookback and future parts span of length L,, Lpp By combining each series" output target token
4

---

Published as a conference paper at ICLR 2025
with the context examples to form each series' imput Hj, concatenate these into the multivariate input token matrix Z for the ICTSP as follows:
Z (
[Z
3N+1jo
|H.....,HH.
Hell
x(1)
x x(ba)
x ((hot1)
x x(lsbte)
Pa
x
x x
x) x
x) x(lotlot))
P322
H (((otErx((++)
x x(br-bp-bo) bo)
x(li-be)
x )
x((r-bet))
x( the
PANN
x(11)-1n)
x((())
Patarger
01.11
(5)
where Pj., represents the positional embedding for each token on the j-th series illustrated as concatenation. Therefore, Z contains (N +]C tokens, with NC context tokens. We can increase the step size in stepwise sampling to reduce computational costs, retaining one sample for every y steps among N samples per series. F shows a special case with no overlap between samples for simplicity, achievable when m Ta + Lpp ICTSP flexibly forms ground truth forecasting examples
to learn the predictor fj x : (((+++++++a)) + X ( ( +it+++++tsstt++t Since ICTSP does not restrict the input channel structure (with Pj. serving only to distinguish) and can construct context examples during inference, it generalizes well across different multivariate datasets, exhibiting robust zero-shot learning capabilities.
daptivv Model Reduction of ICTSP Existing complex Temporal-wise Transformers often struggle to outperform simple baselines, like linear predictors, in noisy datasets lacking clear temporal-wist and channel-wise dependencies. In contrast, ICTSP adapts to different dataset complexities and reduces simpler models when necessary. Specifically: i) When temporal dependencies are weak, ICTSP can adaptively select simpler functions for TSF, such as linear predictors and shallow MLPs, achievable by ICL. ii) When series patterns change greatl over time where the impact of context examples is weak, ICTSP can reduce to a Series-wise Transformer focusing only on local lookback of target tokens without context tokens. iii) When local lookback from other series have minimal contribution. ICTSP degrades to a simple univariate MLP, in which case the inter- token information in attention term from Eqqequation| [ disappears, and token j at layer becomes 2j,6 = A + LN(FFW (2,,,,))))))) If FFN also becomes ineffective, ICTSP processes target tokens only with input and output linear projections outside the Transformer. At this stage, the model reduces to linear predictor, effective for highly noisy time series data. ICTSP's design provides appropriate shortcuts to enhance adaptability. We further explain in $A.4.1 how inappropriate shortcuts in the model reduction of the Temporal wise Transformer limit its ability to learn dynamic patterns.
Sampling and Token Retrieval We can reduce computational costs of ICTSP by sampling one context token every steps from the total N tokens for each series. For very large token amount, to avoid losing information with large sampling steps, we can use token retrieval (TR) to further reduce token size. We employ simple method where each token z; is reduced to a ö-dimensional latent vector 2; via a linear layer. We calculate the cosine similarity between the 2; pair of each context token and target token, averaging the results for each context token. We rank the context tokens by similarity, select the top 9% as context examples, and merge the remaining tokens into r tokens by grouped weighted averaging, resulting in [g(L, L3- Lp) //mmm + r context tokens. TR reduces computational costs, preserves enough information. and helps the model infer which examples are more important with token positions. More details of the ICTSP structure can be found AA.3
2.3 SOLVING KEY ISSUES OF TSF TRANSFORMERS
Timestep Mixing and Overfitting From the ICL perspective, the Temporal-wise Transformer's token formulation overemphasizes channel relationships within specific timesteps. This leads to outputs overfitted to the non-existent underlying channel structures of context examples in real-world datasets with weak inter-series relationships, as shown in Figure (a). Previous studies attribute this overfitting the mixing of channel values at timesteps in the attention mechanism, proposing various solutions. The following analysis shows these partially effective solutions are actually trying to enhance the foeus temporal dependencies in the token formulation.
5

---

Published as a conference paper at ICLR 2025
(i)
Vanilia Structure
(b) Channel Independence
(d] Random Dropping
(6) Local Temporal Representation
Orntedt Foumples
Targel
Contrel Puanples
Tegget
Cunind Froeples
Tagel
Conteet Fuamplee
Teggo
Ps
Pry Platt
Pluy pPain
Pii Plaet
Pc
20
X
X
Ag
X
PS Punt
X
X
Torpuook
vibe
Tran ceaaaaommrr
Figure 3: Previous solutions of Temporal-wise Transformers' overfitting issue. From the ICL perspec- tive, they are actually introducing more learnable temporal dependencies within token formulation.
PatchTST (Nie al. 2022) suggested that univariate Temporal-wise Transformers, which inputs each channel independently (Figure 3 (b)), can address overfitting. This approach breaks the series-wise relationship in the tokens, transforming the ICL predictor into fã. Pt - x which predicts values solely from positions. This shift enhances the model's 's focus on temporal dependencies while ignoring series-wise dependencies, making the formulation more suitable for real-world TSF tasks. Notably, the prompt structure used by recent LLMs for TSF, timestep: value also benefits from this approach. Existing methods like ForecastPFN (Dooley al., 2024) use similar structures.
ARM Lu et al. 2024) introduced random dropping (Figure 3 (c)), which mitigates overfitting by randomly removing series from both input and output during training. This randomly breaks some relationships between channels within the tokens. With only positional information consistently preserved, the model focus more on the temporal generation process and retains only the most likely channel dependencies, reducing overfitting to non-existent channel relationships. However, the degree of channel dropout requires further tuning based on different dataset characteristics.
Methods such as LogTraans (Li et al. 2019) and ARM (Lu et al. 2024 use strategies similar to Figure 3 (d) to build local representations for tokens across time intervals instead of single timesteps. This introduces temporal relationships within the token. For TSF data with weak channel dependencies, tokens naturally focus on intra-series temporal relationships (solid lines in Figure 3 (d)) rather than the weak channel-wise relationships (dashed lines in Figure 3 (d)) in the original token format. This approach alleviates overfitting but requires careful tuning of the local window selection. Recent methods like Liu al. (2024b) tend to combine method (c) and (d) to further improve the generalization ability.
These previous solutions emphasize temporal dependencies more in tokens but do not address the fundamental formulation issue. The Series-wise Transformer solve the issue by treating input series as tokens, helping each token focus on temporal dependencies but lacks contextual examples and ICL capabilities. In contrast, the ICTSP treats forecasting tasks as tokens, providing context forecasting examples to help the model learn temporal predictors and potential series dependencies through interactions between the target token and the historical context of all the series.
Permutation Invariance Studies like DLinear (Zng et al. (023) and iTransformer (Liu al. 2024a) suggest that the poor performance of Temporal-wise Transformers is partly due to the inherent permutation invariance issue in Transformer, meaning swapping the positions of input tokens does not significantly affect the output. The Temporal-we wiise Transformer treats context tokens more as representations of the underlying channel structure. Thus, when a dataset lacks a temporally varying underlying channel structure, in fi P. + [X[") ..... X(O) becomes ineffective, and the output tends to converge to the context's average. This is consistent with DLinear observations (Zeng et al.. 2023) where shuffling of input token minimally impact the output. Conversely, in the ICTSP, each context token represents forecasting training example, so swapping their positions does not harm the forecasting. This makes the ICTSP better suited to the Transformer's characteristics for TSF compared to previous structures.
Channel Structure Restriction In the Temporal- Transformer, the number of input series and their relative positions within the tokens must be fixed. This limits the model's abilíty to adapt to datasets with varying series numbers. Channel independence (Nie et al. 2022) may simplify the

---

Published as a conference paper at ICLR 2025
Temporal-wis wise Transformer univariate model, enhancing generalization but losing the ability to model inter-series dependencies. The Series-wit wwie Transformer does not restrict the number of input series structurally but learns the characteristies of C' series during training through Pj. making hard to transfer embeddings to other datasets. Conversely, the ICTSP does not restrict the number of series and includes in-context training examples within the input, enabling easy transfer input data of any size. Here, positional embeddings can be trained to only distinguish between samples without describing the characteristics of the series. The flexibility input channel structure provides zero-shot multivariate forecasting ability, which previous multivariate models could not achieve.
3 EXPERIMENTS
We conduct comprehensive experiments under full-data, few-shot, and zero-shot settings using widely- used TSF datasets (details in (A.2). including ETTs Zhou et al. 202T), Traffic, Electricity (ECL), and Weather. We use K = 3 TF layers with d = 128 and heads. We set Lı = 1440, L, = 512, and Lp {96, 192, 336, 720}. performing 4 experiments for each dataset. We use sampling step m = and the token retrieval method with g= 10%, r 30 in main experiments. See $A.3.2 for more hyper- pareeep and implementation details. We use the same experimental environment as (Zhou al. 2023 Jin et al. (2024). Extensive efforts are made to ensure fair comparisons across all experiments (detailed in A.3.3).
Baselines We compare our method with previous SOTA models in the categories of LLM for TSF (Time-LLM (Jin et al. 2024), GPT4TS (Zhou et 2023) LLMTime (Gruver al. 2023)). Temporal-
wise Transformers (PatchTST (Nie et al. 2022). FEDformer (Zhou et al. (0022). Autoformer (Wu)
al. 202T), Informer (Zoo
202T)). Series-wise Transformer (iTransformer (Liu et al. 2024a
CNN (TimesNet (Wu et al. 2022)) ), and simple methods (DLinear (Zeng et al., 2023). Last-value
Repeat). Baseline results are sourced from (Zhou et al.) 2023 Jin et 20.44 when applicable and
rerun for the missing experiments. We report the average test set MSE for each dataset in the main paper and provide the full raw results $ The average rank and the number of times each model is ranked as the best are also summarized in the results.
Full-data TSF Results We train the ICTSP from scratch on TSF datasets, with the results shown in Table D ICTSP consistently shows superior performance compared to previous methods in all categories. The ICL capability, activated by the context examples, allows it to excel on larger datasets with stable patterns like ECL and Traffic. Its adaptive model reduction ability, originated from its token and structure design, ensures strong performance on small and noisy datasets like ETTs.
Table 1: Full-data TSF results. Averaged test set MSE on each dataset is reported. The best and second-best results are in bold and underlined, respectively. See Table the original results.
Models
ICTSP
Time-LLM
GPT4TS
iTransformer
PachTST
FEDiormer
Autoformer
Informer
DLincar
Repeat
ETTb1
0.404
0.408
0.428
0.454
0.413
0.440
0.496
1.040
0.423
1.321
ETTB2
0.328
0.3.g
0.351
D.375
0.330
0.437
0.450
4,431
01.41
0.536
ETTml
0.342
0.329
0.352
0.374
0.351
0.448
0.588
0.961
01358
1. 1,269
ETTm2
0.247
0.251
0.267
0.269
0.255
0.305
0.327
1.410)
0.267
0.385
Weather
0.218
0.226
0.237
D.349
0.236
0.309
0.338
0.644
(1.44)
0.355
ECL
0.154
0.159
0.167
0,173
0.162
0.214
0.227
0,311
0.166
1.612
Traffic
0.386
0.353
0.114
0.120
0.391
0.610
0.628
0.761
0.134
2.770
AvgRank
1,93
4.29
2.79
6.86
7,89
5,00
9.50
18
3
Few-shot Learning The sho-shot learning ability of ICTSP is tested by training it on only the first 10% 5% of the training data. ICTSP can flexibly build context samples based on the input data to perform in-context fitting without parameter updates, which compensates for the lack of training samples closer to the test set. Thus, stably outperforms other models both the 10% and 5% settings, as shown in Table 2 Note that we ensured ICTSP only uses datapoints perceivable by other baselines to build the context examples, ensuring a fair comparison, as detailed in A.3.3
Zero-shot Learning We also test the zero-shot transfer learning ability of ICTSP by training one ETT dataset and testing on another ETT dataset. ICTSP surpasses all previous methods by a large margin in the zero-shot settings, as shown in Table The patterns in unseen datasets can be prompted with the forecasting task examples constructed in the tokens of ICTSP, which perfectly fits the scenario of zero-shot transfer learning.
7

---

Published as a conference paper at ICLR 2025
Table 2: Few-shot learning results on 10% and 5% training data. The best and second best results
in bold and underlined, respectively. See Table 7 and 8 for the original results.
Medell
ICTSP
Time-II.M
(PPT4TS
ParchTST
PRinformer
Auteffomer
Informer
TimesNer
Dlaneee
Repest
0.555
0,839
ET
0.397
0.414
0.501
0.722
0.677
0.411
1.250
ETTm2
0.275
0.277
0.293
0.29%
0.463
0.316
0.385
Weather (10%))
0.257
0.254
0.238
0.242
0.284
0.300
0.597
0.279
0.241
0.353
(CL(101))
0.174
0.175
0.176
0.180
0.346
0.431
0.180
1.612
Traffic (1016)
0,428
0.429
0,440
0.430
0.003
0.749
0
0.447
2,770
(10%)
1,57
1,89
6.39
9.43
4,93
#RankI (10%6)
15
10
ETThi (596)
0.540
0.627
0.682
0.695
0.659
0.722
0.750
1.314
ETTh2
0.365
0.582
0.401
0.439
0.641
0.470
3,923
0.828
0.519
ETTml (5%))
0L.37
0.425
0.472
0.537
0.751
0.796
1.163
0.717
0.481
1,269
ETTT22 (556)
0.285
0.2774
0,308
0.315
0.581
0.389
3,689
00:345
0.399
0.385
Weather (5%)
0.255
0.261
0.264
0.271
0.510
0.311
0.584
00.298
0.264
0,353
ECL (596)
0.175
0.177
0.179
0.181
0.267
0.346
1.281
014412
0.177
1.612
Tratho (356)
0.418
0.425
0.454
0.418
0,677
0.835
1,591
0,867
D.451
2.757
AvgKank (556)
1.56
2.30
3,48
3.96
6.04
7.08
9.44
6,80
4.84
9.24
#Ranki (5%))
12
2
2
0
4
0
Table 3: Zero-shot transfer learning results. The best and second best results in bold and underlined, respectively. See Table for the original results.
Models
ICTSP
Time-I11
I.MeTime
GPT4TS
PatchTST
Autoformer
TimesNer
iinar
Repear
ETTh
D:582
0.494
ETTh2-ETTh!
0.441
0.479
1.911
0.758
0.565
0.758
0.866
0.703
1.321
ETTml -ETT52
0.356
0.381
0.992
0.433
0.439
0.470
0.454
0,464
0.536
KTTm) -b1Tm2
0.255
0.265
1.667
0,514
0.297
0.469
0.322
0.335
0:385
ETTm2- -EIT22
0.352
0.354
0.992
0.135
0.409
0.123
0.435
0.156
0.536
ETTm2 .ETTml
0.408
0.44
1.933
0.769
0.568
0.755
0.769
0.649
1,269
AvgRank
1.13
1.88
8.67
4.54
3.29
6.21
5,75
5.33
7,88
4Rank1
3
Ablation Studies ICTSP has concise model structure, with forecasting tasks as input tokens, enabling efficient transfer of ICL abilities to TSF. We conducted experiments on a) the inclusion of context examples, b) token retrieval (TR), and c) different sampling steps m, shown in Table Without context example tokens, ICTSP reduces to a Series-wise Transformer, lacking ground truth forecasting references, which significantly reduces performance. Token retrieval is used to lower computational costs while preserving model performance. As observed, using TR in the full model does significantly reduce performance compared to the original model without TR. The sampling step m controls the density of context samples that ICTSP can see. We aim to choose the largest m that does not lose information to reduce computational cost. Selecting m 8 does lose much performance compared to using full samples with m = 1. while increasing m to 256 degrades performance to the level close to the scenarios without context examples.
Table 4: Results of the ablation studies of the ICTSP structure with Lp E (96, 192, 336, 720)
Full
Metrie
MAF
MAE
MSE
Weather (96)
0.139
0.194
0.150
0.202
0.140
0.200
0.140
0.195
0.130
0.194
0.142
0,195
Weather (192)
0.186
0.236
0.195
0.211
0.158
0.240
0.185
0,233
0.185
0.238
0.187
0.239
Weather (336)
0.239
0.279
0.245
0.200
11241
0.267
0.240
0.284
0.239
0.281
0.241)
0.283
Weather (720)
0.306
0.320
0.309
0.334
0.307
0.338
0.307
0.321
0.300
0.325
0.312
0 0.327
Weather
0.218
0.257
0.225
0.262
0.219
0.254
0.259
0.19
0.259
0.261
0.248
(192)
0.212
0.211
0.257
0213
0.288
(336)
0.268
0.326
0.268
0.330
0.927
0.269
0.327
ETTm2 (720)
0.347
0.382
0.352
0.387
0.345
0.383
0.345
0.381
0.345
0.384
0.350
0.386
ETTm2 (Avg)
0.247
0.311
0.251
0.315
0.346
0.312
0..446
0.312
0.248
0.312
0.250
0.313
Computational Costs Table 5 we compare the computational costs between different settings of m and token retrieval (TR), as well as between ICTSP and other TSF Transformers. The total number of context tokens Lq( L, - La - Lp)/m] //mmC + r. Using m = and token retrieval with = 10% reduces the first part of tokens to 1.25% of the original amount without losing much information, as shown above. These settings effectively reduce computational costs compared to the full context token scenario, making ICTSP more efficient than most of the previous TSF Transformers. Note that in our setting with a fixed input length L,. longer forecasting horizon Lp result in fewer context samples and may have less computational costs than shorter Lpp. We do not include the computational
8

---

Published as a conference paper at ICLR 2025
costs of LLMs for TSF here, they usually have thousands of times more parameters, making direct comparison unnecessary.
Table 5: Comparison of computational costs. The data format of ETTh2 is utilized to construct model for every model are set according to their default configurations.
inputs. The ICTSP (m 8, w/ TR)
Models
ICTSP (m 8, w/o TR)
ICTSP (m 1. w/ TR)
ICTSP (m w/o TR)
Metrie
HLOPs
Parms
HLOPS
Params
FLOP
Params
FLOPS
Params
Le
1,516
11.2G
9.58G
7.35M
8800
7.34M
Le 192
7.43M
Le 330
8.09
7.54M
7.52M
Lp 720
3.02G
7.83M
7.82M
Models
Autoformer
Informer
PatcTST
iTannfoomme
Metric
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Paramss
Lp
10.90
10.5M
9.41G
11.3M
12.60
10.7M
148M
6.72M
LP 192
11.63
10.5M
10.1G
11.3M
12.70
2M
149M
6.77M
LP 336
12.76
10.5M
11.20
11.3M
12.80
1M
151M
6.85M
Lp 720
15.50
10.5M
14.0G
11.3M
13,2G
39.5M
155M
7.04M
Analysis of TSF Transformer Characteristics From the ICL perspective, the Temporal-wise Transformer is better for learning inter-series relationships, while the Series-wise Transformer better at intra-series relationships. Here, we use synthesized "Multi" dataset from (Lu et al., 2024) with strong inter-series dependencies, generated by shifting random walk process. Fig. 4 (a) and (b) show that the Temporal-wist Transformer performs well on this dataset, while the Series Transformer fails to learn the shifting dependencies. ICTSP, with its context tokens featuring shifting structure, excels in learning these time-interleaved dependencies. Conversely, the Temporal-wise Transformer struggles with real-world datasets with weak dependencies comparing Series-wise Transformer, such as the ETTm2 datasets in Fig. (c) and (d). ICTSP handles these datasets well, demonstrating significantly better performance. Note that build these baselines in this analysis
with the same structure shown in Fig. 2 using L = 512, Lp 192, K = 3, d = 128 for the models; and L2 256 for ICTSP. See SA4.22 for more discussion about the inter-series learning abilities of the TSF Transformers.
0.35
11
66
100
(a) Multi (Test Loss)
(b) Multi (Forecasting)
(c) ETTm2 (Test Loss) (d) ETTm2 (Forecasting)
Figure 4: Comparison of the 3 architectures. The first 3 series of Multi and ETTm2 are visualized.
Attention Visualization We visualize the attention maps of the three TF layers in ICTSP using the same forecasting data points from the Multi and ETTm2 datasets as above. We disable token retrieval to observe the interaction within and between the tokens from each time series. In Fig. (a)), (a), it can be observed that the shifted series 2 and 3 fetch information from the corresponding inter-series context tokens. In Fig. (b), ICTSP focuses the first half of the context tokens, aligning with Fig. (d), where the future shape resembles the earlier part of the input rather than the later part.
4 RELATED WORKS
Extensive research on TSF has been conducted both traditional statistics (Box et al., 1974 Holt
2ims 1980) and deep learning (Hochreiter & Schmidhuber 1997 Rangapuram al. 2018
Salinas et al. 2020 Xing et al. 2024h. Recent studies have seen a surge in Transformer based TSF
models (Wen al. 2022 Li et al. 2019; Zhou et al., 2021: Wu et al. 2021 Zhou et al.) 2022
Zhang & Yan 2023). However, study (Zeng et al. 2023) indicated that these complex models
might not outperform linear predictors real-world datasets, leading subsequent research (Nie

---

Published as a conference paper at ICLR 2025
Visualization Avereged Attention Maps Layer (wio Taken Retrievel, Dataset: Multi)
Layer "54
Layer
Layer 3
152
153
B5
155
157
131
TS1
152
151
T54
155
156
157
130
151
122
153
154
150
15e
737
150
60
0035
112
0 2000
0.055
724
724
724
135
0.000
Visual caationationn Aver Attention Maps for Each Layer ICTSP (wwo loken Retrieval, Delaset: ETTm2)
Layer
761
157
TO
TO
TC3
0.003
113
722
0000
0.009
Figure 5: Visualization of averaged attention maps of the 3 layers of ICTSP on Multi and ETTm2.
2024 proposing possible solutions. Additionally, due the significant achievements of LLMs in various tasks (Brown et al.) (020), several LLM-based TSF models (Gruver al. 2023 Zhou et al.
al. 2022 Liu et al. 2024a Lu et al.
2023 Jin al.] (2024) have been introduced. These models also use Transformer architectures but rely on pre-trained parameters and fine-tuning to achieve superior few-shot and zero-sho shot capabilities (Zhang al., 2024). While studies (Min al. 2022 Wei et al.] 2023) Xie et al.] 2022: Zhang et 2003 Garg et al. (2022) have highlighted that these few-shot abilities in LLMs stem from ICL, there lack of research on how efficiently implement ICL in TSF.
5 CONCLUSION AND LIMITATION
In this study, we propose the In-context Time Series Predictor (ICTSP) to leverage in-context learning capabilities for time series forecasting. By treating "forecasting tasks" tokens, build context forecasting examples into the input to activate the ICL abilities. ICTSP addresses issues TSF Transformers such as timestep mixing and overfitting with its effective token formulation, achieving SOTA performance across full-data, few-shot, and zero-shot experiments. For limitation, due to the lack of large collective TSF datasets, we have tested ICTSP's scaling ability with more layers on larger datasets. We also have not explored the possibility of training a single global model that can handle different lookback and future length, as allowed by the ICTSP structure.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International
Conference Learning Representations, 2023. URL TRLhttps:: /ttp:://pperrvvee wwt. net/forum?
id- (d-0g0X44ByN4T
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In seventt Conference Neural Information Processing Systems, 2023. URL btttps://openeeeeeee htt.//www.e...o.eeeeeeeeeeee
id=1 1MSgUugg
10

---

Published as a conference paper at ICLR 2025
George EP Box, Gwilym M Jenkins, and John F MacGregor. Some recent advances in forecasting and control. Journal of the Royal Statistical Society: Series C (Applied Statistics). 23(2):158-113 1897 1974.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
URL
net / forum? id= fzHR AA8U
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha V Naidu, and Colin White. Forecastpfn: Synthetically-traine trained zero-shot forecasting. Advances Neural Information Process- ing Systems, 36, 2024.
Shivam Garg, Dimitris Tsipras, Percy $ Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Nate Gruver, Mare Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? case study on learning with repre- sentations. In The Twelfth International Conference on Learning Representations, 2024. URL
iew. net forumm wtw.rmmmmwdwwwwwwaaaa....
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1997.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang. Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2024.
net/ wtteee//oorummlddnnnsScce
lebeling Kaastra and Milton Boyd. Designing neural network for forecasting financial and economic time series. Neurocomputing, 10(3):215-236, 1996.
Ibai Lana, Javier Del Ser, Manuel Velez, and Eleni Vlahogianni. Road traffic forecasting: Recent advances and new challenges. IEEE Intelligent Transportation Systems Magazine, 10(2):93-109, 2018
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. Advances in neural information processing systems, 32, 2019.
Yingcong Li, Muhammed Emrullah lidiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference
Machine Learning, pp. 19565-19594. PMLR, 2023.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers effective for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024a. URL httpps://opereviiiiie het:/oozumiiosJePP
Yong Liu, Guo Qin. Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Autore- gressive time series forecasters via large language models. arXiv preprint 2xXi::4402.02370 2024b.
11

---

Published as a conference paper at ICLR 2025
Jiecheng Lu, Xu Han, and Shihao Yang. ARM: Refining multivariate forecasting with adaptive temporal-contextual learning. In The Tweifth International Conference on Learning Representa- tions, 2024. URL
Simin Ma, Yan Sun, and Shihao Yang. Using internet search data to forecast covid-19 trends: A sys- tematic review, Analytics, 1(2):210-227, 2022. ISSN 2813-2203. doi: 10.3390/aallttess000000
URL https://wwwt mwppi com www.co...m11cee wtt.:///222222/12/22222222
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint urXiv:2202.1285 2022.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022. URL
htw.:eee/foooo fort/forum? Taeee//oorummmGKKKggXXcC
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022.
Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7796-7805, 2018. URL httppss::/prcceeeeioooooo neurips. wwt.scccppeee ptcc//aper/20110 10t001//hhhhi abw.aadddddd.e..eaaaaaaa........
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1181-1191, 2020.
Christopher A. Sims. Macroeconomics and reality. Econometrica, 48()) 1980. ISSN 00129682, 14680262. URL/http://ww// jstor. wtg.sttabee/99100111...
Yuxuan Wang, Haixu Wu, Jiaxiang Dong. Yong Liu, Yunzhong Qiu, Haoran Zhang, Jianmin Wang, and Mingsheng Long. Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprini arXiv:2402.19072 2024.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu,
Da Huang, Denny Zhou, al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303.088. 033846 2023.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07/25, 2022.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Mare' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 22419-22430, 2021. URL http:tttts/ppopeediiiiiii neurips. cc http//aper/222////gg/ ww Bcc044022219eeccccccaaaaaaaaaaa2
html
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations, 2022.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL
12

---

Published as a conference paper at ICLR 2025
Qianwen Xing, Chang Yu, Sining Huang, Qi Zheng, Xingyu Mu, and Mengying Sun. Enhanced credit score prediction using ensemble deep learning model. arXiv preprint arXiv:2410.00256, 2024.
Shihao Yang, Mauricio Santillana, and S. C. Kou. Accurate estimation of influenza epidemics using google search data via argo. Proceedings of the National Academy of Sciences, 112(47): 14478, 2015. doi: 10.1073/pnas. 1515333311.11133111...1 URLhttps: /ww. www.//wwwppppnppp/..c..aa.../..s org/doi/
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. In Ro-soMe:Robuusteess of Few-shot and Zero-shol Learning in Large Foundation Models, 2023, URL / net/ formm? Rdd
Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh Gupta, and Jingbo Shang. Large language models for time series: A survey. arXiv preprint 2rXiv:2402.01801, 2024.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 11106-11115,11111111111111111111 AAAI Press, 2021. URLlttts:://ojs aaai org/ index.iexx.///// wwnpdeewpppppp//////////pppppppp phpp AARI/ article/view/1132.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 27268-27228 PMLR, 2022. URL https://proceecing: .wr.iisssm.......m.. mlr. www.sppes/eeeoc... v22/ zhou22g....oo html
Tian Zhou, Peisong Niu, Xue Wang. Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In seveth- Conference Neural Information Processing Systems, 2023. URL https:tt///// htttepp:///peerrvviee wtt.w/wwww.ewwtte///// d=gMS VVVZmm
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide the detailed raw experimental results presented in the main paper. Table presents the detailed results in the full-data setting. Tables 7 annd demonstrate the results of few-shot learning. Table 9 shows the results of zero-shot learning.
Table 6: Full-data results with forecasting horizons Lp E (96, 192, 336, 7203. The test set MSE and MAE results are reported. The best and second best results are in bold and underlined, respectively.
Repoor
Mone
3420
34AB
KIAE
3452
SEAB
MISE
MAE
5452
MAE
M556
MAA
M4A
KEAE.
34502
MAA
3852
KIAE
SEAE
0:713
0.515
0.99
ETEM (U82)
0.373
03.433
1,36
0415
0.415
0.418
0431
0.51
0.413
0.431
0,439
0.485
0.590
0.483
1.000
0.792
0.235
0.425
0.405
1.325
0.333
EITM (3961
01:30
0.110
8.477
11:00
0.431
B.422
11115
B-065
11.31
81 198
186
00009
01:41
0.110
0.815
BITH (hrg))
0.494
0.438
0.466
0,424
01:28
0.426
0.454
0.805
0.413
0.431
0.440
0:900
0,386
0.48
1.40
0.755
0.458
0.450
0.433
0.437
1.301
0.337
ETTE
3.200
1.525
0.29
0.35
(17
0.356
0.374
0.50
0.75
11351
0.785
0.
0.9M
0.334
01.18
0.679
B-7M4
1.455
01 153
1.600
1941
0.933
01111
0.58
6/11
14531
0.173
ETT22 (396)
0.323
(977)
01313
0.401
0429
0.388
0.4%%
0.492
0.485
1,835
0,432
6,63
0.991
0.328
0.37%
0.534
0.36
0.355
0.365
0.355
0.410
0.332
0.31
0.437
0:009
0.490
0.455
4.431
1.259
0,414
0.427
0.431
0.477
0.53
0.460
ETTed
0.912
0.355
0.475
0.25
0.343
HITh (195)
0.3
0.310
6.00
01313
0.333
0156
0.991
0.113
0.388
0.77
D:01
0.553
0 0.505
0.795
D1606
0.3
01353
8.516
0.96
1.301
0.500
ETTal (200)
US54
0.365
0.582
0,41
0.521
0.53
0.711
0400
0.411
0.600
0,56
1.29
0404
0.63
0.130
196
0.33
(lgg
0.32
0.314
0,39
6.32
0.352
0.363
9.574
0.412
0.351
0.361
0.445
0/622
0.585
0.513
0.961
0.754
0.430
0.416
0.557
0.379
1.269
0.646
IL199
0.316
0.51
0.763
0.751
0.365
13315
01355
0.314
1.96
0/151
0157
01360
0.39
0.33
RTTe22 (192)
6222
0.389
0316
025
1622
0.350
0:36
0336
(.22)
02322
0.200
0.281
0.340
0.355
5366
0223
0.205
0224
0.30
(134)
0.311
6320
01296
0.35
0.35
0.339
0.312
0987
0.311
0.35
0.211
0,34
0410
STT13 (730
1L307
1.59
(lgg)
0.24
0.331
0251
5314
14307
0.269
0.354
0.255
11315
0.46
11:27
0.31
1.410
01221
0.333
0.267
0.554
14365
0334
Worder (4)
0.139
0117
0:11
0.313
0.108
0.317
D.7%
0.364
0.316
0:385
0173
01330
0.317
0.511
(192)
0.586
0.115
6.34
1L314
0.248
0:23
01255
0.214
0144
0.7%
0,307
0.361
0.506
0.54
0219
0.262
0220
0.201
1.30
0.50
Westbrr (20)
0.35
0.125
9.364
0,36
0.333
0.32
0.341
0311
0334
0,410
0-425
0449
0.42
0,741
0.355
0.355
0.335
0,362
0.465
0.394
Wogg
0.204
0.357
0.26
0.288
14337
0.211
0.249
0.30
0.225
11:24
0.46
0.501
11:35
0.312
0.508
0.299
0.317
0.344
0.341
14353
01300
DXT (WC
0.127
0,321
0111
0.3
013
01318
3115
0:30
01333
1.99
D:V8
01:11
0111
0.751
0.96
01:58
01337
5.100
0.31
1.544
0516
BOL (12))
0.548
0.541
0.122
6241
1L333
0.233
2.166
(.29)
1337
11346
0.201
0311
0.222
0.334
0.26
0.154
0.185
0.135
6,20
1.333
0.556
DCL (120)
0,591
0.333
0.102
0256
0,335
0.299
0.266
0,26
0.997
0.59
0.346
03355
0.254
0.363
0,499
0.230
0.135
0.233
0.301
1.547
0.515
0.354
0.351
0.119
0.255
14507
0.133
0.265
0.362
11223
0.114
11:27
0.318
0.311
0.777
11143
0.309
0.166
0.34
1,13
0.599
Vtile 195)
0,399
0.351
0362
6.34
1.355
0.382
0.361
0,26
0.362
0.34
0.51
0,309
01513
0.355
0.719
5.91
0,313
0.311
0/400
6.20
2:723
1,595
Tartk (197)
0.372
6337
0410
01778
1776
0101
04414
11
1966
0.179
0617
0.13
0.80
3.55
1061
Valle (738)
0.422
0,30
0.312
9.45
0.412
0.296
0.26
0.92
0.590
0.405
0472
064)
0.350
0.466
0.315
2811
1,000
Tualle (Agg)
0.38
0,55
6.364
0.400
0.61
0.34
0,10
0.76
0.533
0416
0.30
0.05
1.00
A.2 DATASETS
Our main TSF experiments are conducted based on commonly used time series forecasting datasets,
detailed as follows:
ETT Datasets (Zhou et al., 2021): : This dataset includes load and oil temperature data from
electricity transformers, recorded at 15-minute intervals from July 2016 to July 2018. It comprises four subsets: ETTm1, ETTm2, 1 and ETTh2, representing two transformers (identified as 1 and 2) and two time resolutions (15 minutes and hour). Each subset contains seven features related to oil and load of the transformers.
Electricity D This dataset covers hourly electricity usage data of 321 consumers from 2012 to 2014 and is frequently used in energy consumption forecasting and analysis.
Traffic D Sourced from freeway sensors in the San Francisco Bay area, this dataset provides hourly road occupancy data from 2015 to 2016, making it key resource for traffic flow studies.
Weather 5 This dataset captures 21 weather variables, such as temperature and humidity, recorded every 10 minutes throughout 2020, supporting detailed meteorological studies.
Multi Dataset (Lu et al.| 2024) is generated from a master random walk series. The first series is the master series, while the second to fifth series are created by shifting the master series backward by 96, 192, 336, and 720 steps, respectively. The last three series are combinations of these first five series.
http://pems.dot.co
https://www.bg: joos-jjena.m. mpw.a.mp...oo wwt.mmpp..ddeete..e de/vetter/
14

---

Published as a conference paper at ICLR 2025
Table 7: Few-shot learning results on 10% training data with forecasting horizons T.p
{96, 192, 336, 7203. The best and second best results are in bold and underlined, respectively.
Medels
Time
L1.36
CPT415
PSCCTTT
Autoforsee
Informes
VinscaNe:
DLiness
Mesie
MSE
MAE
MIE
MAN
MSS
MAA
MSE
MAE
MIE
MAE
MSE
MAA
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
ETTEI (96)
0.411
8:428
1,455
0.400
0.455
0,555
0.516
0.485
0.512
0,429
0.433
0.532
3.379
0.752
0.561
0.628
0.492
0.455
1,255
0.713
ETTEI (192)
0.454
0.43
0,588
0.634
0.598
1.159
0.505
ETT5 (7200
0,210
0.612
1330
0.91
0.725
0
0.622
0610
0.725
0.614
0.721
0.615
1.217
0.825
0.641
0.985
0.343
1.339
0.756
RTTH (Arg)
0.525
0,500
0.555
0.52
11 1.990
0.575
0.511
0,542
0.69
01:661
0.703
0.586
1.109
0.500
01808
0.675
0.681
0.600
1.331
0.757
ETT32 1961
0.115
0.375
0.336
0.353
0.389
0.382
0.413
0.45
3.337
1.508
0.378
0.357
0441
0412
0.455
110
0.429
0.416
0.411
0.501
01.00
0.543
3.553
1.576
0.084
0.61
0.573
0.591
ETTE2 (720)
0:430
0.41
0427
0.449
0.44
0.454
0447
0.40
0.499
0,509
0.515
0.52
3.942
1.503
0.510
0.491
0.244
0.648
0.588
0.517
ETT2 [Avg]
0.369
0.401
0.371
0.5M4
0.357
0:421
0,415
3.451
0.466
01475
0.485
0.499
3,812
1.513
0.471
0.465
0.005
0,538
0.536
01450
ETTel (192)
0.366
0.38
1.214
0.300
0.47
0454
0,592
1.172
0.76
0.382
0412
ETiml (156)
0,465
8.415
111
0.42
0.469
0.439
0476
345
0.996
0.399
0.871
1.237
0506
0.725
0.565
0.41
0.11
1.385
6.300
ETTml (120)
0.475
8.4%
03589
0.005
0.511
0.556
0.693
01570
03810
0.630
1.307
0.997
0.790
0.540
0.890
0.41
1.319
0.720
RTTWl (Aog]
0.197
0.410
0.004
0.422
0.464
0.441
0.901
5,466
0.722
01501
0.901
0.675
1.192
0.921
0.677
0.517
0.411
0/479
1.769
0.08
0.274
0.352
3,205
0.285
0.213
0.333
Elim22 (192)
0.239
6.367
0241
0334
0.251
0.309
6,252
0.177
0.307
0.579
0.694
0.691
3,112
1.367
0.270
0.523
0.255
0.345
0.340
0.371
ETTm2 (356)
0.745
0.176
R374
0.30
0316
0.366
0.785
0.53
0.500
2.815
1.457
3.355
1.431
0.325
01351
0.338
6.355
0413
0,110
BTTH2 (720)
0.795
0.91
0.676
0:417
0.435
3.477
0.712
0.514
1911
1.156
3,009
1.563
0.624
0
0.435
0.440
0.531
0.165
ETTo2 (hog)
0,313
0.07
0.33
0,335
0.256
0.345
01448
1,343
0,930
3,770
1.440
0.300
0.353
0.315
0.355
0.765
Woaher (96)
0.154
0.115
616
0.30
0.161
0.315
0.165
0:115
0.168
0.251
0.221
00:40
0.316
0-401
0.164
0.230
01121
0.754
0.359
0:354
Weathe (192)
0,299
0,5
0.00
0.38
0.10
0.254
6210
0.257
0.250
0.364
0.275
0,312
0,552
0.7%
0.345
0.28
0.115
0,253
0,30
0,292
Woather (336)
0.259
0.294
026
0.0
0.356
0.293
0.259
0,297
0,312
0.546
0.300
0.35
0.724
0.54
0.305
0.20
0.255
0.59
0.377
0.358
Wiseber (730)
0.315
0.30
0.30
0.301
0.319
0.312
5.346
0.387
0.393
0.346
0.739
0.558
0.381
0.371
0.335
0:34
0.465
(U99
Weater
6.242
0,342
0.597
ECL (96)
0.136
0.236
0.150
0.241
0.139
0.231
0,540
0.288
0.231
0.523
0.261
0.346
1,259
0819
0.299
0.573
0.150
0,253
1.588
0.946
ECL (192)
0.153
0,252
0.151
0.348
0.156
02511
0.580
R155
0.261
0.356
0.338
0.466
1.150
0.333
0.345
0.579
0.164
0:254
1,595
01550
EXTI 1350)
0.1
0315
0.00
0.221
0.77
8.76
0.140
9.776
0.300
01445
0.11
0.474
1:157
0.772
0.318
01391
0.811
0.441
1ATT
0.961
1770)
0750
0.341
0.775
0.685
1,705
0.213
(Avg)
9,231
0.175
0.70
0,380
0,315
0.346
0.475
1.155
0,250
1,612
Thathe (961
0:416
0.758
1.118
0.291
0.414
0.201
8.465
8.389
0.69
01:00
0.673
0.804
1.157
0.531
0.719
11416
03419
0.788
1.135
1.079
Tuflle (192)
0.433
0.306
0.36
0.426
0.30
CA15
0.366
0.657
0.416
0.727
0.434
1.454
0.765
0.768
0.428
0.44
0.325
2.756
1,087
Tialle
0:423
0.307
0431
0.11
6.436
0.30
0.655
0.454
1.531
0812
0.71
0.449
0.313
2.791
1.685
0.316
Tralie (20g)
0,428
0,310
0.29
0,306
0.440
0.30
6.40
0.366
0.66
0.425
0.749
0.446
1,534
0.811
0.951
0.535
0.447
0.313
2,770
1,000
Table 8: Few-shot learning results on 5% training data with forecasting horizons Lp E {96, 192, 336, 720]. The best and second best results are in bold and underlined, respectively. means the lacking of data to build the training set in the 5% scenario.
11
PaNATTT
Autiferner
DLineer
Repeat
Metric
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MBE
MAB
MSS
MAE
MSE
MAE
MISE:
MAE
MSE
MAE
MSB
MAE
R (96)
0.400
0.818
1.633
0.964
545
0 9-5
0.557
0.119
0.591
8 1.000
0.681
0.530
1.336
0.512
(1547
0,892
0465
1.704
(XT13
(193)
B,497
0.659
0.500
0.711
3.570
0.003
0.600
1:349
1.828
11720)
0.910
0.665
1.128
0.634
1 (Arg)
0.540
0.458
D167
0.583
01007
0.500
0.805
0,560
0.609
01:67
0.733
0.599
1.335
0.817
01 0.770
0.611
0.956
0.45
1.316
0.79)
ETT2 (96)
0.384
0.341
0.356
0.507
0.576
0-211
0.401
5,421
0,500
11424
0
0.466
3.837
1.506
0.442
11 01455
0.400
0.420
0/430
0.422
D.455
3.975
3.056
ETTE2 (720)
ETTE2 (Arg)
0.368
0.309
0.12
0.415
0.401
0.434
0.419
3.448
0.641
01457
0.420
0.489
3.925
1.644
01555
Bm66
0.464
C:454
0.519
ETTal 196)
0.331
0.316
0.372
0.386
0.405
6,399
04114
0.63
0.544
0.725
0.575
1.130
0.775
0.324
0.605
0.518
1.214
ETTml (196)
0.111
9:115
0.4%0
03384
0.140
3.667
0,00
0.538
0.841
0.640
1.158
11509
11416
0.65
0.977
1.385
ETTal (720)
0.453
0.47
0453
0.47L
0.517
0.499
0757
0.587
0,52
0.633
0.55
0.655
1.175
0.783
0.511
0.489
0.75
0.993
1.319
0.729
BTTel (Ang)
0.397
8.418
0.455
0.55
0.423
0.450
0.517
3.476
0.751
0.:93
1.
076011
5.165
0.790
0.:01
0.417
0.213
0.563
1.369
BTT
(192)
0.344
0.30
0.15
0.57
0.256
0.54
0.54
0.9
0.361
0.351
3.578
1.475
0.375
0.11
0.351
0.3m0
ETTs2 (336)
0.301
637%
0.73
0.30
0.31
0,353
0334
0367
037
0.421
0.47%
0.517
3.561
1.45
0.590
0.23
0335
C336
0412
0.400
ETTm2 (720)
0.412
0.400
0453
0.422
0.460
0.45
C.44
0452
0.525
0.50
0.553
0.536
3.556
1.588
0.644
0.53
0.509
C465
0.521
01465
ETTm2 (Aog)
0.245
0.341
0.374
0.223
0.308
0.445
6,15
0355
0.381
0.405
0.389
0.434
3,659
1.400
0.390
0.425
0.345
0.373
0.165
0.994
Weater (96)
0.215
0172
0.3
0151
0.124
0.20
0.227
0.239
0.497
0.407
0.207
0.253
0.259
Wosher (193)
0.219
8:387
0721
0.21
0.518
6310
0337
03361
01317
0.224
0.511
0.680
0.565
0.288
0.381
0.233
0.377
0.11
(1392
Wochher (536)
0.27%
0.113
0311
0.764
7.76
0353
01:90
0.341
0.321
0.649
0.547
1.220
11593
0.513
6.735
0.117
01188
Wiceher (730)
0.35%
0,371
0.0
0.77
0.344
0.391
0.391
0.357
0.349
0.510
0.522
11364
0.388
0.405
0.315
0.465
(.19
(Arg)
0.710
3.106
0.354
0.114
0.318
0.153
BCT, (96)
0.165
6.340
11147
0362
DM1
0.15
0.44
0.255
0.297
0.367
1.365
0.119
0.351
0.315
0.359
1.588
0/866
ECT, (193)
0.155
0.216
018
0341
0155
0.55
6163
3,300
0.347
[1341
0.315
0.375
1.256
0.530
0.163
01363
0.315
0.366
1.555
0.950
ECT, (350)
0.177
0.75
1.77
0.77
0.775
0.74
0.145
5.281
0.367
01356
0.354
0.411
1.300
D1442
1.175
01778
0.343
0415
1.617
0.561
ECL (7220)
0.221
0.315
0.224
0.512
0.323
0.23
0.335
0.318
0.394
0.425
0.456
1.259
0.919
0.219
0.311
0.635
0.613
1.647
0.955
9,369
0.151
0,405
0.402
0.453
Trafic (96)
0.406
0.265
0414
0.201
0.419
0.295
8.404
6.286
0,670
0.421
0.705
0.481
1.557
0.621
0.427
0.364
0.854
0.492
2.725
1,600
Tillle (192)
0,415
0.255
1400
0.29
0.434
0.305
9413
0.633
0.485
0.337
0.503
1.596
0.834
0.447
0315
0.294
0337
1.736
1/67
(5W)
0,432
9.111
0.344
0.44
0.313
0419
1310
0.100
0.445
0
0.811
1/611
0.978
0.511
0.83
0/011
3.391
1.091
Thafhe
0.118
0. 300
0.200
0.434
0. 305
8.41%
8.297
0.677
0.831
0.900
1.991
04832
0.317
08667
0
2.757
15

---

Published as a conference paper at ICLR 2025
Table 9: Zero-shot tranfer learning results between ETT datasets with forecasting horizons L.p
(96, 192, 336, 720}. The best and second best results are in bold and underlined, respectively.
Models
ICTSP
Time
LLMTTiee
GPT4TS
PuchTS
Autoformer
TimesNet
DLineur
Repeat
MS5
MAE
MAB
MSS
a1-h2
0.219
0.576
0.547
(316)
0.414
(230)
0.419
0.665
0.517
B1- (Ang)
0.352
0.708
0.422
0
0.582
0.548
0.431
0.494
0.536
0.480
B2-hh
0.411
0.425
0.40
1.130
0.777
0.455
0.003
0.509
0.601
1.295
0.713
3:2 -h (192)
0.435
1.242
0,820
0.565
0.760
0.601
0.610
0.707
1.325
0.753
b:2 -hi (336)
0,864
0.575
0.619
0.626
0.710
1.323
0.744
b2 -hi (730)
0.469
4.145
1,461
0.751
0.597
0.628
0.7%6
0.644
0.648
0.704
0.596
1.339
0.756
b2 -bi (Ang)
0.499
1.963
0
0.578
0.565
0.758
0.608
0.621
0,703
1.321
0.757
ma -22 (96)
0.299
0.389
0.321
0.510
0.576
0.353
0.392
0.354
0.435
0.470
0.407
0.365
0.432
0.422
mI-hd (190)
0.561
0.364
0.39
0410
0.521
0.586
0.463
0.477
0.447
0.444
0.485
0.489
0.421
0.45
0.454
0.462
0534
0.473
m1 m1-ad (336)
0.776
0.412
0.40k
0218
0.640
0.637
0.450
0.461
0.451
0.461
0.420
0.42
0.422
0.484
0.496
0.494
0.591
0.508
ml--12 (730)
0,388
0.431
0.406
0.456
2.296
1.034
0.466
0.468
0.474
0.471
0.480
0.485
0.495
0.482
0.541
0,529
0.588
0.517
ml h11-hh (Ang)
0.356
0.309
0.381
0.42
0.997
0,708
0.433
0,440
0.419
0.458
0.470
0.479
0.454
0.457
0.464
0.475
0.5%
0.480
m1 11mmm
6.168
0.258
0.169
0.49
0.646
0.563
0.217
0.294
0.185
0.271
0.585
0.457
0.222
0.285
0.221
0.314
0.256
0.328
m1 1--m22 (190)
0.223
0.283
6337
8318
0.934
0.654
0.277
0.327
0.258
0.111
0.435
0.450
0.288
0,317
0.786
0.350
(1:40)
0.371
ml ---m (336)
0.273
0.329
0390
0338
1.57
0.728
0.351
0.360
0.317
0.345
0.476
0.47
0.341
0.967
0355
0.406
(1412
0.40
--22 (750)
0.355
0.384
6375
0366
4,350
1,531
0.429
0.413
0.416
0.404
0,542
0.535
0.436
0.418
0.476
0.476
0.521
0.465
m11 -m2 (Arg)
0.255
0.315
0.365
0320
1.567
0.509
0.314
0,349
0.297
0.334
0.469
0.455
0.322
0.354
1.335
0.389
0.355
0.394
2a2 -12 (96)
0.288
0.351
0.295
0356
0,510
0.576
0.350
0.401
0.327
0.367
0,553
0.393
0.360
0.401
11333
0.391
0.432
0.422
ma2 122 (152)
0.352
0.395
0359
0.97
0.523
0,586
0.434
0.437
0.411
0.418
0.432
0.437
0.434
0.437
11441
0.455
0.534
0.473
m22 -82 (3355
0.571
0.115
0.67
0412
0.580
0.637
0.450
0.459
0.439
0.44)
0.452
0.459
0,460
0.659
11.00
0.503
0.591
0.508
m=2 -2 (7305
0, 0.398
0.453
0,305
0.454
2.296
1,004
0.485
0.477
0.459
0,470
0.455
0.457
0.485
0.677
01.43
0.534
0.588
0,517
m2- h2 (Aeg)
0,352
0.80
0.354
0,300
0,592
0,708
0.435
0.448
0.409
0.425
0.425
0.439
0,435
0.648
0.456
0.471
0.536
0,490
m2-mm (96)
0.571
0.385
0.389
0 1397
1.179
0.781
0.747
0.558
0.691
0.437
0.735
0.576
0.747
0.558
0.570
0-490
1.214
0.665
n2 m2-mm (100)
0.87
0:11
0390
0.50
1.327
0.846
0.781
0.560
0.530
0.470
0.735
0.566
0.781
0.580
1.590
0.506
1.261
0.690
=2- mmm
1336)
8.46
0-427
0411
0.455
1.476
0.902
0.778
0.576
0.565
0.497
0.750
0.533
0.778
0.538
0.706
0.567
1.253
0.707
a2 mml
(720)
0.456
0.444
0.48
0.458
3.749
1,408
0.769
0.573
0.686
0.563
0.782
0.609
0.769
0.573
0.731
0.584
1.319
0.729
2x2 -mml (Aog)
0.408
0,419
0.414
0.438
1.933
0.084
0.759
0.567
0.588
0.492
0.755
0.591
0.769
0.567
0.649
0.537
1.259
0.598
A.3
IMPLEMENTATION DETAILS
A.3.1 MORE DETAILS OF MODEL IMPLEMENTATION
Token Construction F provides another illustration of how to construct the input tokens in ICTSP. In the implementation, we first concatenate the output embedding to the input series. Then we cut the last L, + Lp part as the target tokens. We perform stepwise sampling from the end part to the initial part of X₁- If the remaining initial part of X, cannot constitute a token with the required length of Lg, + Lp, we drop these initial steps in X1.
Contest
-p-1
by Ap
Output
Imput
Embediing
Up
1.g
Timestap
Targel
Contect
Torgat
Figure 6: Another illustration of the process of sampling the context examples from the input time series in ICTSP.
Trainable Embeddings As illustrated in Equation 5 we use two types of trainable positional embeddings in the token construction. One embedding represents the series source of token, and the other represents the position of the token within the entire input. All the positional embeddings have the same d dimension and are added to the latent tokens.
Randomized Training During training. we can introduce some random shifting r E fo.... m 1} to the sampling start point to expose the model to varied context examples. To prevent the positional embedding from learning information with specific time series channel, we randomly shuffle the order of series in the input to make the series show up in different position each training step. This
16

---

Published as a conference paper at ICLR 2025
can slightly enhance the performance in the few-shot and zero-shot tasks and for small datasets. Since ICTSP does not restrict the number of series in each input, we can randomly pick subsets of input series to build the tokens during training. This random selection slightly improves performance in few-shot learning with small datasets that have weak dependencies, while not significantly affecting performance on larger datasets with stable dependencies.
Token Retrieval We can reduce the computational costs of ICTSP by sampling one context token every m steps from the total N tokens for each series. For very large token amounts, to avoid losing information with large sampling steps, we use token retrieval (TR) to further reduce token size. Below, we provide a detailed description of the TR process.
Let z be the i-th token in the context. Each token z; is reduced to a 6-dimensional latent vector ž; via a linear layer:
2 Wz, +
where W E Roxd and b E R are the weights and biases of the linear layer. We calculate the cosine similarity between the 2; of each context token and each target token Z,:
2; 2
cosine aecine_similarityiee Ze)
2, Zel
Next, we average the cosine similarity results for each context token:
average average_ similarity(s))) 1 Σ cosine_siiie sittiesssmmmlratyy.....
(8)
where C' is the number of target tokens. We then rank the context tokens by their average similarity and select the top 9% as context examples.
The remaining tokens are merged into r tokens by grouped weighted averaging. Let k = [gN]] be the number of selected tokens, and let p = N k be the number of remaining tokens. The merged tokens are obtained as follows:
merged
Σ azzz
z
jEG:
where G. is the set of tokens in group i, determined by the ranked order, and ag are the weights obtained from a softmax over the cosine similarities: e "esssee_eiillarttylk,......
aj Exeg, e coeine sisilaeeeaillarittyBb....
(10)
The final set of context tokens includes the selected tokens and the r merged tokens:
final (2), Zk: zi
merged
....., x merged)
(11)
This results in [g(L, L. Lp)/ m/ C + context tokens.
Token Rationalization and Linear Layer Warm-up ICTSP excels at handling datasets with varied characteristics and performing transfer learning across different datasets. The robustness of ICTSP, when facing small and noisy datasets, is further enhanced with following training strategies. For each context target token 2,, we can subtract the last value in its lookback from all values to align forecasting tasks across all tokens. This reduces the impact of mean shifting within or across series, serving naive repeat method baseline. Additionally, as mentioned earlier, the direct shorteut in Equation helps ICTSP achieve simple linear baseline in weak dependency scenarios. To prevent the attention and FFN terms from learning noise patterns too fast, we can initially train only the shortcut: for the first W steps, remove the transformer layers and train only the input and output projections on the target tokens. After this warm-up, reintroduce the transformer blocks with context examples, gradually increasing the model's complexity to adapt to small and noisy data. Figure 7 shows three cases: i) not using linear layer warm-up (W 0), ii) using linear layer warm-up (W = 5000), and iii) only using the linear layer (W co). We trained the ICTSP with these settings on ECL (Lp = 192) with reduced input (L, = 512, La = 256). The model with initialized weights for the input and output projection shorteut outperforms the models without warm-up and with only the linear layer as a baseline comparison. The linear layer warm-up stage structures the embedding space into forecasting task, guiding the model to build tokens that focus on temporal forecasting power. The effectiveness of this warm-up up stage is due to the token design and the shortcuts in the model structure.
17

---

Published as a conference paper at ICLR 2025
Model
0.28
Test Set MSE Loss
W 5000
0.26
inf
0.24
0.22
0.20
0.18
20 30
50 60 70
80
Training Step
Figure 7: The effects of using linear layer warm-up to initialize the embedding space as forecasting tasks with the input and output projection shortcut. Using a warm-up step W 5000 helps the model structure the embedding space as forecasting tasks before fitting the entire transformer structure, gaining performance margin and preventing overfitting. These three model are trained on ECL (Lp 192) with reduced ICTSP settings (L, 512, Lb 256).
A.3.2 HYPERPARAMETER SETTINGS AND TRAINING DETAILS
Model Training Setup The ICTSP model is trained using the Adam optimizer and MSE loss in Pytorch, with learning rate of 0.0005 each dataset. We test the model every 200 training steps with a early-stopping patience being 30 x 200 steps. The first 1000 steps are for learning rate warm-up, followed by linear decay of learning rate. We the random seed 2024. Our models trained on single Nvidia RTX 4090 GPU with a batch size equals to 32 for most of the datasets. We decrease the batch size to 16 and 8 for the larger dataset ECL and Traffic, respectively. In the full-data experiment setting, we split each dataset with 70% training set, 10% validation, set and 20% test set. We fit standardization scaler on the training set and apply it to the whole dataset. This setting
applied in consistent with previous studies like (Wu et al. 2021 2022, Nie et al. 2022 Jin et al. 2024).
Hyper-parameters of ICTSP For the hyper-parameters of ICTSP, we use L, = 1440 and L. 512 for all the experiments. We use the pre-normalization Transformer with latent dimension, and heads. The hidden dimension of FFN is set to 4 times of latent dimension and the dropout rate of Transformer layers set to 0.5. We use m = as the sampling step and q 10%, 30 for the token retrieval module.
layers, d = 128
Randomness of Training We test the randomness of training the ICTSP by alternating the random seed among (2022, 2023, 2024, 2025, 2026) and conducting multiple training on the same dataset settings. The error bars of the results with the five random seeds on the Weather and ECL datasets shown in Figure & alongside the results of the two best previous baselines. Time-LLM (Jin et al. (024) and PSt (Nic et al. 2022). The results show that ICTSP can stably surpass previous methods, even with the worst results among the five random seeds, indicating the robustness of ICTSP in performance improvement.
A.3.3 DISCUSSION OF FAIR COMPARISON
Our experimental environment aligns with the codes used by previous studies, including (Wu et al. 2021 Zeng et al. 2023 Wu et al. 2022 Nie et al. 2022 Zhou et al. 2003 Jin et al. 2024). Our few-shot and zero-shot learning setups follow (Zhou et al. 2023 Jin al. (024).
In the full- data experiment, for the input length L, setting, previous studies like (ie et al. 2022
Zhou et al. 2023) suggest that L, should be considered dataset. Therefore, we chose to directly source their results where applicable, to avoid underestimating their performance when rerunning with fixed I,,.
parpeerr that needs tuning for each
18

---

Published as a conference paper at ICLR 2025
Time-LUM PatchTST
---
0.275
0.250
Text MSE
0.200
0.175
0.150
Weather Weather (192)
(336)
Weather (720) (96)
(192)
ECL (336)
ECL (720)
Experiment
Figure 8: Randomness in Training ICTSP. The error bars for training ICTSP on the Weather and ECL datasets with random seeds (2022, 2023, 2024, 2025, 2026] are illustrated, compared with the previous best two baselines, Time-LLM and PStchTST. The results show that ICTSP can stably surpass previous baselines even in the worst training cases, demonstrating the robustness of the ICTSP structure.
In the few-shot experiment, careful consideration of L, is very crucial. Since (Zhou et al. 2023 Jin et al. 2024) use the first 10% or 5% of the training set for few-shot training data, this represents a scenario where the model's forecasting time is far from the training data collection. Thus, should avoid leaking too much recent data to the model, especially for ICTSP. Because ICTSP can use historical input in-context training samples, increasing L, too much might leak data that should be invisible to the model, defeating the purpose of the few-shot experiment.
Given the difference between L 512 used by (Zhou et al., 20023 Jin et al. 2024) and L1 1440, La = 512 for ICTSP, we chose to mask the part of the test set input data not visible to the L 512 setting (Zhou
2023 Jin al. 2024) with zero-filling for ICTSP. This ensures that
2024) is used by ICTSP. As a result, there are no Series-wiss Transformer) at the very beginning of the test set input data, and the number of context examples gradually increases in the subsequent test set input data points. Thus, the few-shot learning results Tables results from weakened version of ICTSP, yet still achieving SOTA performance.
no unseen data for (Zhou et al. 2023 Jin et al.
context examples for ICTSP (which reduces
2 and cean regarded as
A.4 ADDITIONAL ANALYSIS OF TSF TRANSFORMERS
A.4.1 DIRECT SHORTCUT IN TEMPORAL- WISE TRANSFORMERS
Revisiting the expression of Temporal-wise Transformer in Equation 3 it is clear that if context tokens become ineffective, the remaining MLPs or linear predictors are essentially guessing channel values and structures based solely on target positional embeddings. Past studies often found such Transformers to produce output resembling flat line of input means rather than learning dynamic patterns, which could be explained by failure of context and ineffective learning from positional embeddings.
A.4.2 MORE DISCUSSION OF INTER-SERIES MODELING ABILITY
From the ICL perspective, the Series-wise Transformer primarily learns temporal dependencies rather than inter-series dependencies. This is evident in the "Multi" dataset with simple shifting inter-series dependencies, shown in Figure 4 (b). This may indicate lack of representation ability in the series token construction. Transformers can easily transfer information within the same token vector channel between different tokens via multi-heat attention. When building the token example pool for ICTSP, a shifting process inherently exposes inter-series time- iime-inerleavee dependencies directly in the token representations, making easy to capture these dependencies.
19

---

Published as a conference paper at ICLR 2025
However, if these dependencies can be directly modeled into single token dimension rather than between tokens, the Series-wing -wie Transformer with enlarged tokens may still acquire inter-series learning ability alongside its inherent temporal learning ability. In a scenario with very large latent dimension for the input token, where the token vector encompasses every shifting possibility of the input series, multi-head attention dependencies between tokens by exploring this high dimension and fixing these alignments into the learned token embedding. This analysis suggests that increasing the token dimension of the Series- wise Transformer may enhance its inter-series modeling capability
find the alignment of possible inter-series time-interleaved
specific datasets. Nonetheless, since these inter-series dependencies are learned and fixed in the token representation space, this capability cannot be transferred to other datasets in few-shot or zero-shot scenarios, unlike ICTSP.
A.5 COMPARISON SUMMARY DIFFERENT TRANSFORMER -ASSED TSF STRUCTURES
Table 10: Summary of the characteristics of Transformer-based TSF structures
Methods
Temporal-wise Transformer
Temporal-wise Transformer (with Channel Independence)
Temporal-wise Transformer (with Patching)
Series-wise former
Truns-
ICTSP
Tokenization
Timestep embed- ding
Scalar embedding
Patch embedding
Series embedding
Embedding of forecast- ing tasks
ICL Pair
9) (Timestep index, Multi-channel val- ues)
(Timestep index, Single-channel value)
(Timestep index, Multi-channel val- ues in a temporul interval)
(Iuput series, Output series)
(Input series, Output se- ries)
Token Repre- sentation for cies
Single-step inter- series dependen-
Scalar value single timestep
Temporal and series-wise relationships local interval
Temporal wise tionships within sin- gle series input
Temporal relation- ships in forecasting task
Attention Learn
Temporal-wise dynamics of se- ries dependencies
Temporal-wisede de- pendencies
Temporal-wise relationships between local intervals
Aligned inter-series relationships
Forecasting task predic- tors, considering both aligned and unaligned inter series relationships in shifted context exam- ples
Suited String con- sistent inter-series dependencies
Temporal se- ries relationships changing tempo- ral dependencies
String
Real- worl-w with strong temporal-
changing temporal relationships, aligned unaligned series relation- ships: transfering to dutasets with different channel structures
can automati- cally balanced by this structure
wise and aligned series-wise relation- ships (e.g traffic)
Suited For
Not
Weak inter-series dependencies in datasets
inter- series dependen- cics
Strong
Few-shot/zero- shot cases with different scries relationship chur- acteristics
Unaligned series- wise relationships, such as shifting effects between series
worldwrldd
Underlying inter- Examples series effects vary scribing
ICL Context
Temporal relation- ships within context for se- ries
Temporal-wise relationships between local intervals
No context examples
Historical ground truth forecasting tasks refer- ences
with the time indices in the lookback (likely nonexis- tent or weak)
Chanmel
Fixed
Fixed
Fixed
Flexible requires additional training adapt to datasets with new channel structure
Flexible; additional train- ing necessary since context examples exist
Structure
Problem
Only series re- lationships in tokenization: tionships overfitting to series-wise effects from the input context
Unable to model series series-wise rela-
Fixed channel structure in tok- enization prevents generalization datasets with dif- ferent structures
Restricted specific dataset channel strue- tures; unable to learn unaligned effects
20

---

Published as a conference paper at ICLR 2025
A.6 ADDITIONAL ABLATION STUDY: ICTSP WITH LIMITED (MASKED) LOOKBACK INFORMATION
To fully align with the L, = 512 used in the baseline models, we relax the design of ICTSP, which originally required all context tokens to be ground truth for forecasting tasks. We allow some forecasting tasks to contain incomplete information. In this case, we still use L, 1440, but mask all information from the first 1440- 512 = 928 timesteps in the input, ensuring that only the last 5 512-step true information remains in the input, For the masked portion of the first 928 steps, we fill it with the mean value of each series token from the last 512 steps.
To maintain the presence of some true forecasting task tokens for shorter Lp, we shorten Ls to 336. All other hyperparameter settings remain unchanged. We denote this structure as ICTSP (Limited). We repeated the experiments on the ETTm2 and Weather datasets, including full data, few-shot 10% (FS 10%), few-shot 5% (FS05%), and zero-shot (ZS) experiments on the ETT datasets. The results are shown in the Table A.6
Table 11: Performance comparison of ICTSP (Limited) with six baselines on ETTm2 and Weather datasets across full data, few-shot (FS10% and FS5%), and zero-shot (ZS) settings.
Methods
ICTSP
ICTSP (Limited)
Time-LLM
GPT4TS
PatchTST
TimesNet
DLLineur
Metric
MSE
MAE
MSE
MAE
MSE
MAK
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Full ETTm2 (96)
0.159
0.248
0.160
0.241
0.161
0.253
0.173
0.262
0.165
0.255
0.187
0.267
0.167
0.269
Full HTTm2 (192)
0.212
0.289
0,216
0,290
0.219
0.293
0.229
0.301
012200
0.292
0.249
0.300
0.224
0.303
Full ETTm2 (336)
0.268
0.268
0.326
0.286
0.341
0.329
0.321
0.281
0.342
Full BTTm2 (720)
0.347
0.382
0.354
0.383
0.352
0.379
0.378
0.401
0.362
0.385
0.408
0.403
0.397
0.41
Full ETTm2 (Avg)
0.247
0.311
0,250
0.310
0.251
0.314
0.267
0.326
0.255
0.315
0.291
0.333
0.267
0.334
Pull Weather
0.139
0.194
0.142
0.198
0.147
0.201
0.162
0.212
0.149
0.198
0.172
0.220
0.176
0.257
Full Weather (192)
0.186
0.236
0.187
0.237
0.189
0.234
0.204
0.248
0.194
0.241
0.219
0.261
0.220
0.282
Full Weather
0.239
0.279
0,242
0,281
0.262
0.279
0.254
0.286
0.245
0,282
0,280
0.306
0,265
0.319
Full Weather (720)
0.306
0.320
0.308
0.322
0,304
0.316
0.326
0.337
0.314
0.334
0.365
0.359
0.333
0.362
Full Weather (Avg)
0.218
0.257
0.220
0,260
0,226
0.258
0.237
0.271
0.226
0.264
0.259
0.287
0.249
0.300
ES10% ETTm2 (96)
0.176
0.258
0.176
0.263
0.177
0.261
0.188
0.269
0.191
0.274
0,212
0.285
0,213
0.303
PS10% ETTm2 (192)
01239
0.307
0,2388
0.308
0.241
0.314
0.251
0.309
01255
0.317
0.270
0.323
0.278
0.345
PS10% ETTm2 (336)
0.288
0.286
0.334
0,274
0.327
0.307
0.346
0.306
0.353
0.325
0.353
0.338
0.385
HS10S 6TTm2 (720)
0.395
0.391
0. 0.399
0,405
0.417
0.390
0.426
0.417
0.433
0.4477
0.474
0.449
0.436
0.440
FS10% ETTm2 (Arg)
0.275
0.323
0,274
0.327
0,277
0.323
0.293
0.335
0.296
0.343
0,320
0.353
0.316
0.368
PS10%
Weather (96)
0.164
0.214
0.165
0.212
0,161
0.210
0.163
0.215
0.215
0.184
0.171
0.224
FS10
Weather (192)
0.209
0.252
0,208
0.253
0.204
0.248
0,210
0.254
0.210
0.257
0.245
0.288
0.215
0.263
FS10%
Weather (336)
0.259
0.294
0,261
0.295
0,261
0.302
0.256
0.292
0.259
0.297
0.305
0.321
0.258
0.299
FS10%
Weather (720)
0.315
0.319
0.316
0,509
0.321
0.339
0.346
0.381
0.371
0.320
0.346
TS10%
Weather (Avg)
0.237
0.273
0.238
0.274
0,234
0.273
0.238
0.275
0.242
0.279
0.279
0.301
0.241
0.283
FS05%
ETTm2 (96)
0,181
0.265
0.179
0.264
0,174
0.261
0.199
0,280
0.206
0.288
0.236
0.326
0.220
0.299
FS05% ETTm2 (192)
0.264
0.310
0,241
0.312
0,215
0.287
0.256
0.316
0.324
0.306
0.311
0.361
FS05%
ETTm2 (336)
0.301
0.378
0.299
0.341
0.273
0.330
0.318
0.353
0.334
0.367
0.380
0.423
0.338
0.366
H505%
KTTm2 (720)
0.412
0.410
0.405
0.408
0.433
0.412
D.460
0.436
01454
0.432
0.674
0,583
0.509
0.465
FS05%
ETTm2 (Arg)
0.285
0.341
0.331
0.274
0,308
0.315
0.353
0.399
0.426
0.345
0.373
FS05%
Weather (96)
0.170
0.225
0.170
0.223
0.172
0.263
0.175
0.230
0.171
0.224
0.184
0.242
0.207
0.253
FS05%
Weather (192)
0.219
0.267
0,225
0.269
0,224
0.271
0.227
0.276
0.230
0.277
0.228
0,283
0.272
0.307
FS05%
Weather (336)
0.278
0.313
0.277
0.310
0.282
0.321
0,286
0.322
0.326
0.279
0.313
0.328
FS058
Weather (720)
0.358
0.371
0.155
0.368
0.366
0.381
0.366
0.379
0.384
0.387
0.364
0.388
0,400
0.385
FS05%
Weather (Avg)
0.256
0.294
0,256
0.293
0,261
0.309
0.264
0.302
0.270
0.304
0.264
0,309
0,298
0.318
ml-am2 (96)
0.168
0.255
0.261
0,169
D:217
0.294
0.221
0.314
ml-m2 (192)
0.223
0.293
0,223
0.295
0.227
0.318
0.277
0.327
0.258
0.311
0.288
0.337
0.286
0.359
ml -2 (336)
0.273
0.329
0.276
0.335
0.290
0.338
0.331
0,360
0.317
0.348
0.341
0.367
0.357
0.406
(720)
0.355
0.384
0.362
0.381
0.429
0.413
01416
0,404
0.436
0.476
0.476
ml ml-m2 (Axg)
0.255
0.315
0,258
0.318
0.265
0.320
0.314
0.349
0.297
0.334
0.322
0.354
0.335
0.389
b1 >b2 (96)
0.272
0.338
0.274
0.336
0.279
0.337
0.335
0.374
0.304
0.350
0.358
0.387
0.347
0.400
b1-+h2 (192)
0.530
0.375
0.412
0.400
0.447
0.460
h1 h1-5h2 (336)
0.359
0.402
0.361
0.406
0.388
0,415
D.441
0.444
0,414
0.428
0.449
0.451
0.515
0.505
b1-+12 (720)
0.387
0.431
0.38%
0.437
0.391
0.420
0.438
0.452
0.419
0.443
0.448
0.458
0.665
0.589
bl- (Avg)
0.537
0.385
0.339
0.389
0.352
0.387
0.407
0.422
0.381
0.405
0.421
0.431
0.494
0.489
From the average ranking, it can be observed that even with masking applied to information prior to the 512-step window, the limited ICTSP still significantly outperforms all baselines. In the full data and zero-shot experiments, limited ICTSP performs slightly weaker than the original ICTSP but remains superior to all previous baselines. However, in the few-shot training experiments, limited ICTSP surpasses the original ICTSP to some extent. It is important to note that this improvement is due to an additional constraint in the original ICTSP, which, despite using L1 1440. imposes zero-fill firo-filiing restrictions outside of L, = 512 to ensure fair comparison with baselines using L, = 512 (see Section A.3.3 Discussion of fair comparison for details on few-shot experiments).
21

---

Published as a conference paper at ICLR 2025
In the full data and zero-shot experiments, all data can be used with any L, without causing data leakage. However, in the few-shot scenario, due to the skipping nature of data sampling, a longer L accesses more lookback data (in val/test set), creating an unfair advantage over models with shorter LT- To eliminate this effect, in the original method, the input to the original ICTSP for few-shot experiments was subjected data before the start of the validation and test sets, avoiding any unfair data usage. This means that in the first few datapoints, the original ICTSP could only see 512 timesteps of valid data, gradually expanding to L1 = 1440 as valid data became available.
mero-masking, ensuring that it could access at most 512 timesteps of
However, since the model applies a last-value demeaning operation to the forecasting task tokens, the initial zero-filling leads to meaningless negative last values being filled in those positions. This effectively suppresses the model's performance. In ICTSP (Limited), we address this issue by using mean-value filling, ensuring that the filled values in the forecasting task tokens are more reasonable. This adjustment allows ICTSP (Limited) to outperform the original ICTSP in few-shot experiments.
To clarify this, we replaced the zero-filling used for fair comparison in original ICTSP with mean- filling and present the results below. The results show that ICTSP with mean-filling for fair comparison significantly outperforms all models, including Time-LLM, in all experiments. This demonstrates that ICTSP's few-shot capability is inherently stronger than the zero- arrlifiling results presented in the original paper when proper mean- mean-filling strategies are used to handle unknown values in forecasting tasks. This result further enhances the significance of ICTSP. The results can be found in Table A.6
mean-filling Table 12: Results of using mean- filling strategy for the 512-step data retaining of fair comparison in the few-shot experiments.
Methaas
ICTSP ZZer-Giling)
ICTSP (Mean (illnn)
ICTSP (Limited)
Time-LIM
GPT4TS
PuichTST
TinesNet
DLincar
Motric
MSE
MAR
MSE
MAR
MSR
MAR
MSR
MAF
MSE
MAE
MSE
MAF
MSE
MAF
MSE
MAE
PsiD RM 1969
0.763
0.171
01888
0.769
0.181
0.334
0212
0.285
0211
0.90%
ETT (192)
0.308
0.241
0.314
0251
0.309
0.52
0,317
0.523
0.278
0.345
0.333
0.594
0.403
0.426
0:433
0.444
0.466
0.440
61Tm2 (Agg)
0.327
0.320
0.316
0.368
ES10% Weatter (96)
0.164
0.214
0.164
0.214
0.165
0.212
0.161
0.210
0.165
0,215
0.165
0.215
0.154
0.230
0.171
0.224
FS (cathe((133)
0.209
0.353
0.304
0.250
0.388
0.253
0.304
0.348
0210
0.354
0200
0.357
02455
0.285
02135
0.263
FS10s Wcathe (338)
0.259
0.264
0.259
0.293
0.36
0.205
0.261
0.302
0.556
0.202
0.25
0.207
0.355
0.521
0.558
0.259
FS10 (720
0.315
0.331
0,310
0.551
0.119
0.330
0.309
0.331
0.211
0.39
0.332
03346
0.311
0.371
0.30
0.546
FS10s Waather (Ava)
0.25
0.274
0.234
0.38
0.242
0.279
0.79
0.501
0.241
0.283
FS05 ETTm2
0.179
0.174
0.190
0.280
0.206
0.285
0.23
0.506
0.220
0.299
1806% 61Tm2 (192)
0.241
0.312
0.55
0.316
0.264
0.311
0
190% 617m2 (136)
0
0,378
0.299
0.341
0.273
0,599
0318
0,55
0.314
0.56
0.300
0.625
0.338
0.386
FS054 ETTa2 (220)
0,412
0,410
0,391
0.405
0.405
0.408
0.433
0.412
D.60
0.436
0.454
0.432
0.674
0.585
0.500
0.465
F908 KTTm2 (Aeg)
0.205
0.141
0.273
0.129
0.781
0.351
0.224
0.323
0.908
0,566
0.315
0.53
0.99
0.476
03345
0.395
1806% Weather (96)
0.170
0.225
0.167
0.215
0.170
0.223
0.178
0.263
0.175
0.230
0.171
0.24
0.184
0.242
0.257
0.255
1806 Weather (193)
0.219
0:260
0.177
0.261
0.221
0.264
0.214
0.271
0227
0.226
0220
0.77
0238
0.283
0272
0.997
F505% Weather
0.77
0.310
0.222
0.321
0.36
0.323
0.254
0.336
0.79
0.523
0.13
0.328
F520% Weather
0.295
0.54
0.298
0.318
A.7 ADDITIONAL ABLATION STUDY: ADAPTIVE MODEL REDUCTION AND RANDOM SELECTION OF CONTEXT EXAMPLES
Regarding the Adaptive Model Reduction of ICTSP, because we use a pre-LayerNorm Transformer as described in Eq. 1, it inherently maintains direct residual shortcut from the input token to the output token.
a) With small-scale initialization, if the Transformer layer does not introduce any residual effects, the model can be approximated as having only a linear projection from input to output. This is similar to single linear layer NLinear model (as we apply last-value demeaning for tokens). b) If all attention layers in ICTSP are bypassed during training, the model can be viewed as pure univariate MLP predictor, which many recent TSF structures consider the core temporal predictor [5-7]. Note that neither of above scenarios utilizes context tokens. c) If the information in context examples provides no benefit for predicting the target token, the context information is entirely bypassed, reducing the model to a Series-wise Transformer (like iTransformer). In these cases, the target token can be directly interpreted as the series embedding since the future portion of the target token is not trained in-context. d) When context forecasting task tokens are helpful in determining the in-context predictor for target tokens, ICTSP adjusts its prediction for the target token based on the context forecasting tasks. Here, context examples can be viewed as training samples in a supervised learning setup. Even when only a subset of context examples is used, ICTSP can still perform in-context predictor updates based on the provided few samples, achieving better performance than models that
sample
22

---

Published as a conference paper at ICLR 2025
completely disregard context examples. However, its performance will not match that of the full ICTSP, which uses more in-context training samples.
Based on the above discussion, designed the following progressive experiments to compare the performance differences resulting from using only a subset of the model structure. The experiments conducted on the Weather and ETTm2 datasets, as shown in Table A.7
A. We remove all Transformer blocks from ICTSP, In this case, the model only includes the input and output linear layers for the target token, effectively reducing an NLinear model.
B. We retain only the MLP feedforward layers within the Transformer blocks of ICTSP. The model then be considered univariate MLPP predictor.
C. We retain the Transformer blocks in ICTSP but drop all context tokens. The model then reduces to Series-wise Transformer (iTransformer).
D. We retain the full ICTSP structure but randomly keep only 25% of the context tokens in the context examples. The model sees randomly incomplete in-context forecasting task samples.
E. We increase the random retention rate of context tokens to 50%.
F. We further increase the random retention rate of context tokens to 75%.
G. We use all context tokens, representing the full ICTSP model.
Table 13: Ablation study results of the adaptive model reduction and random selection of context examples
Methods
A.
B.
C.
D.
E.
F.
G.
Metric
MSE
MAE
MSE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather (96)
0.168
0,216
0.148
0.200
0.145
0.205
0.146
0,203
0.140
0.199
0.140
0.198
0.139
0.194
Weather (192)
0.213
0,259
0.191
0.247
0.193
0.253
0.190
0.248
0.188
0.245
0.185
0.238
0.186
0.236
Weather (336)
0.259
0,294
0.245
0.295
0.249
0.302
0.244
0.295
0.241
0.286
0.240
0.283
0.239
0,279
Weather (720)
0,325
0.344
0.319
0.349
0.314
0.343
0.311
0.341
0.312
0,339
0.309
0.335
0.306
0.320
Weather (Avg)
0,241
0,278
0,226
0.273
0.225
0.276
0.223
0.272
0.220
0.267
0.219
0.264
0.218
0.257
ETTm2 (96)
0.163
0.253
0.163
0.255
0.166
0.258
0.162
0.255
0.161
0.253
0.161
0.251
0.159
0.248
ETTm2 (192)
0.219
0,290
0.219
0.298
0.220
0.301
0.216
0.297
0,214
0.295
0.215
0.294
0.212
0,289
ETTm2 (336)
0.274
0.328
0.272
0.331
0.271
0.331
0.267
0.329
0.268
0.330
0.268
0.328
0.268
0.326
ETTm2 (720)
0.361
0.385
0.354
0.387
0.361
0.386
0.358
0.386
0.355
0.388
0.349
0.385
0.347
0.382
ETTm2 (Avg)
0.254
0.314
0.252
0.318
0.254
0.319
0.251
0.317
0.250
0.317
0.248
0.315
0.247
0.311
It can be observed that when the model transitions from A. NLinear to B. Univariate ML., there is sigmificant performance improvement. This is because the MLP introduces more weights, enabling it more effectively handle the temporal relationships within tokenization.
When the model transitions from B. Univariate MLP to C. Series-wise Transformer, the performance improvement is less significant. This indicates that on weakly correlated real-world datasets, the primary improvement of the Series-wisg Transformer compared to a Temporal-wise Transformer comes from the use of series tokenization for better handling of temporal relationship, rather than using the attention layers to capture aligned series-win wise relationships. The MIP structure in B. alone sufficient for this architecture to achieve performance comparable to the Series-wise Transformer.
As the model expands from C. to D., E., F., and G., the number of visible context examples gradually increases. It can be seen that using only 25%-50% randomly selected context tokens sufficient for ICTSP to significantly outperform the Series-wise Transformer without any context examples. This demonstrates that the attention layers in the model effectively perform the task of in-context predictor determination. Moreover, increasing the proportion of context tokens to 75% yields further performance improvements, with the model's performance closely approaching that of the full ICTSP utilizing all context tokens.
A.8 EXPLAINING THE EFFECTIVE CAPTURING OF SERIES- BERIES-WISE RELATIONSHIP OF ICTSP THROUGH A FEATURE SPACE PERSPECTIVE
We propose a feature space perspective to explain why ICTSP effectively models both temporal-wise and series-wise relationships.
23

---

Published as a conference paper at ICLR 2025
From the ICL perspective, the attention layers in Transformers construct algorithms (e.g.. linear regression, shallow MLPs) between the input (z, y) pairs. These layers can be seen as dynamically solving supervised learning problem based the imput datapoint tokens.
Temporal-wise Transformers can be viewed as solving regression problem where the input is temporal indices (z). and the output multi-channe values (y).
Series-wise Transformers and ICTSP, on the other hand, can be seen as solving a regression problem where the input a time series segment (z), and the output is another time series segment (y).
In the default assumptions of regression problems, it is relatively straightforward to model relationships between input features within the same datapoint and relationships between the same input features across different datapoints (e.g., through linear combinations). However, it is much harder model relationships between different input features across different datapoints.
For Series-wise Transformers and ICTSP, we want the input time series segment (r) to have shifting lag effects between its features (similar to the classic Box-Jenking method). However, without explicitly indicating this, the default assumptions of regression problems make it easier for ICL model:
Relationships between features within the same datapoint (temporal dependencies). Relationships between the same features across different datapoints cemporal-aligner inter-series dependencies).
However, it is much harder for ICL to recognize that relationships may exist between different features across different datapoints (unaligned inter-series dependencies). This is a key limitation of Series-wise Transformers when modeling cross-channel dependencies.
ICTSP addresses this limitation by explicitly providing historical forecasting tasks as context example tokens. These historical forecasting tasks inherently have a shifted structure, which directly prompts the model to consider the necessity of unaligned inter-series relationships.
As a result, ICTSP achieves:
The ability model temporal relationships through its tokenization approach. The ability model aligned inter-series dependencies through its output token formu- lation.
The ability to model unaligned inter-series dependencies through the provision of shifted context forecasting examples.
The underlying cause of these challenges lies in the shifting time-dependent structure unique to time series data. In other supervised learning problems, we typically do not consider shifting certain features to analyze their relationships with others. Therefore, in ICL problems where each datapoint treated as a token, ICTSP outperforms Series-wise Transformers by effectively teaching the model to learn these shifting dependencies.
A.9
ADDITIONAL RESULTS OF ABLATION STUDY: ICTSP WITH LIMITED (MASKED) LOOKBACK INFORMATION
A.10 ADDITIONAL RESULTS OF ABLATION STUDY: ADAPTIVE MODEL REDUCTION AND RANDOM SELECTION OF CONTEXT EXAMPLES
Building on the experiments A. G. discussed earlier, we add the following three experiments, H. J. Here, we consider 4096 historical timesteps preceding each forecasting timestep. We sample our context examples from the first 25% of this historical time interval. In other words, the starting points of our context forecasting examples may range from step I to step 1024. Since the forecasting task length LG + Lp ranges from 512 + 96 = 608 to 512 + 720 = 1232 the ending points of the context forecasting examples range from step 609 step 2256.
Because the target token requires a lookback window of 512 timesteps, any context portion entirely unrelated to the target token lies before step 3584. In this setting:
For Lp = 720, the region from step 2256 to step 3584, a span of 1328 steps, is completely skipped and does not contribute forecasting.
24

---

Published as a conference paper at ICLR 2025
Table 14: Full results of performance comparison of ICTSP (Limited) with six baselines full data
setting.
Methods
ICTSP
ICTSP (Limited)
Time-LLM
GPT4IS
PatchIST
FedFFormer
TimesNet
DLineur
Metric
MSR
MAF
MSE
MAF
MSR
MAR
MSE
MAE
MSR
MAE
MSR
MAE
MSR
MAE
MSK
ETThl (96)
0.366
0.393
01366
0.393
0.362
0.392
0.376
0.397
0.370
0.309
0.376
0.419
0.384
0.402
0.375
0.390
ETTbl (192)
0.399
0.412
0.400
0.414
0.398
0.118
0.116
0.418
0.413
0.421
0.420
0.645
0.436
0.429
0.405
0.416
TT (356)
0.426
0.427
0.428
0,429
0.430
0.427
0.442
0.433
0.422
0.436
0.459
0.465
0.491
0.669
0.439
0.443
ETThL (720)
0.424
0.446
0.428
0.451
0.442
0.456
0.447
0.466
0.506
0.507
0.521
0.500
0.490
ETThl (Avg)
0.404
0.420
0.405
0.422
0.408
0.424
0.428
0.426
0.413
0.431
0.440
0.460
0.458
0.450
0.437
ET (96)
0.265
0.335
0.265
0.335
0.268
0.328
0.285
0.342
0.274
0.336
0.358
0.397
0.340
0.374
0.289
0.353
ET (192)
0.326
0.374
0.327
0.375
0.520
0.375
0.354
0.389
0.339
0.579
0.49
0.459
0.402
0.414
0.383
0.488
ET (336)
0.349
0.596
0,352
0.399
0.368
0.409
0.373
0.407
0.329
(1390
0.4%6
0.487
0.452
0.452
0.448
0.415
ET (720)
0.370
0.397
0.371
0.402
0.372
0.420
0.406
0.441
0,379
0.422
0.463
0.474
0.462
0.55]
ETT22 (Avg)
0.376
0.329
0.378
0.330
0.379
0.437
0.449
0.414
FTTal (96)
0.336
0.283
0.540
0.272
0,346
0.200
0.342
0.379
0.419
0.338
0.399
ETTml (192)
0.318
0.361
0.324
0.365
0.310
0.358
0.332
0.372
0.332
0.369
0.426
0.64]
0.374
0.387
0.335
0.365
6T (336)
0.355
0.383
0.355
0.385
0.352
0.384
0.366
0.394
0.366
0.392
0.445
0,459
0.410
0.411
0.369
0.386
FT (720)
0.416
0.415
01415
0.418
0.381
0441
0.17
0.41
0.416
0.430
0.543
0.400
0.478
0.450
01425
01421
ETTml (Avg)
0.342
0.574
0.344
0.377
0.329
0.392
0.352
0.383
0.351
0.381
0.418
0.452
0.400
0.406
0.357
0.579
ETTm2
0.268
0.160
0.241
0.161
0.255
0.287
0.187
ETTh22 (192)
0.212
0.289
0.216
0.290
0.219
0,293
0,301
0.220
0.292
0.309
0.328
0.249
0,503
ETTm22 (336)
0.268
0.326
0.268
0.326
0.271
0.329
0.286
0.341
0.274
0.329
0.325
0.366
0.321
0.351
0.281
0.342
ETTmm2 (720)
0.347
0.382
0.351
0.383
0,352
0.379
0.178
0.401
0.362
0.385
0.421
0.415
0.408
0.403
0.397
0.421
ETTm22 (Ang)
0.247
0.311
0.250
0,310
0.251
0.314
0.267
0.326
0.255
0.315
0.305
0.349
0.291
0.333
0.267
0.334
Weather (96)
0.139
0.194
0.142
0.198
0.147
0.149
0.198
0.217
0.296
0.237
Weather (192)
0.236
0.187
0.237
0.189
0,194
0.241
0.266
0.336
0219
0.282
Wanher (356)
0.239
0.279
0.342
0.281
0.762
0.279
0.254
0.286
0.245
0.282
0.339
0.380
0.306
0.265
0,319
Weather (720)
0.306
0.320
0.308
0.322
0. 1.904
0.316
0.326
0.357
0.314
0.334
0.403
0.428
0.365
0.339
0.353
0.362
Wouther (Avg)
0.218
0.257
0.220
0,260
0.226
0.258
0.237
0.271
0.226
0.264
0.309
0.360
0.259
0.287
0.249
0,300
ECL (96)
0.127
0.221
0.129
0.221
0.131
0.221
0.139
0,238
0,129
0.222
0.193
0.308
0.168
0,272
0.140
0.237
ECL
0.241
0.243
0.152
0.340
0.201
0.315
ECL (336)
0.257
0.158
0.259
0.160
0,163
0.259
0214
0.329
0.198
0.267
ECL (720)
0.181
0.283
0.187
0.286
0,192
0.298
0.206
0.297
0.197
0.290
0.216
0.355
0.220
0.320
0.203
0.30]
ECL (Avg)
0.154
0.251
0.156
0,252
0.159
0.251
0.167
0.263
0.162
0.255
0.214
0.327
0.193
0.295
0.166
0.264
Traffic (96)
0.359
0.251
01368
0.252
0.362
0.248
0.388
0.282
0.300
(.249
0.587
0.366
0.593
0.321
0.410
0.282
Trafic (192)
0.372
0.256
0.371
0,251
0.374
0,290
0,379
0.256
0.604
0.373
0.617
0.336
0.423
0.287
Trafli (336)
0.362
0..733
0.383
0.275
0.284
0.392
0.364
0.621
0.383
0.629
0.296
Traffic (720)
0.291
0.422
0.295
0.430
0.450
0332
0,422
0.256
0.626
0.382
0.640
0,350
0,466
0.315
Traffic (Avg)
0.386
0.268
0.389
0.268
0.388
0.264
0.414
0.995
0.391
0.264
0.610
0.376
0.620
0.336
0.434
0.295
For Lp- = 96, the region from step 609 to step 3584, a span of 2975 steps, is completely skipped and does not contribute to forecasting.
This pattern goes accordingly for other values of Lp.
Since ICL can be viewed as dynamic supervised learning performed during inference, the above setup can be considered few-shot context example ICL scenario. It resembles the few-shot training setup described in the original paper, where datapoints from a certain time period are skipped.
We conduct the experiments as follows:
H. From the range of 1 to 1024 for context example starting points, we randomly select 10% of the steps as starting points to construct forecasting task examples. This is referred to as using 10% Distant Context Examples.
I. From the range of to 1024 for context example starting points, we randomly select 25% of the steps as starting points to construct forecasting task examples. This is referred to as using 25% Distant Context Examples.
J. From the range of 1 to 1024 for context example starting points, we randomly select 75% of the steps as starting points to construct forecasting task examples. This is referred to as using 75% Distant Context Examples.
can be observed that even when introducing relatively distant context examples, they still provide better performance compared to the Series-wise Transformer, which does not use any context tokens. Additionally, when only a small portion of context examples is introduced, the performance improvement compared to having context is relatively modest. However, the number of context examples increases, their contribution to performance also grows. This aligns with previous observations in ICI. literature regarding the impact of the number of context examples used.
25

---

Published as a conference paper at ICLR 2025
Table 15: Ablation study results of the adaptive model reduction and the effects of distant context example.
Weather (96)
0.168
0.148
0.145
0.145
0.145
0.142
0.139
Weather
0.213
0.191
0.193
0.192
0.191
0.189
0.186
Weather (336)
0.259
0.245
0.249
0.246
0.243
0.242
0.239
Weather (Avg)
0.241
0.226
0.225
0.225
0.223
0.221
0.218
ETTm2 (96)
0.163
0.163
0.165
0.164
0.164
0.161
0.159
FTTm2 (192)
0.219
0.219
0.220
0.218
0.319
0.215
0.212
ETTm2 (336)
0.274
0.272
(720)
ETIm2 (Avr))
0.254
0.252
0.254
0.252
0.251
0.249
0.247
A.11 ABLATION STUDY: UNDERSTANDING THE SPECIFIC CONTRIBUTIONS OF CONTEXT EXAMPLES
We designed the following experiments to demonstrate how introducing forecasting tasks as prompt tokens specifically contributes to performance. These experiments aim to validate that ICTSP relies on context examples to work via ICL. Particularly in the zero-shot setting, where no related training samples' information is in model parameters, ICTSP's superior performance is primarily driven by its ability to update the forecasting predictor using context examples. Different types of context forecasting examples provide specific improvements to the corresponding forecasting task targets.
We use ETTm2 as the target dataset and train ICTSP under three different training set configurations: ETTm2, ETTm1, and combined set of (ETThI, ETTh2, ETTml). For these configurations, conduct the corresponding full data training and zero-shot transfer learning. During evaluation, we independently generate context examples from series context part of ICTSP for test set MSE evaluation. We use Lp 96, with all other settings matching those in the main experiments.
in ETTm2 and feed them the
We construct the following variants:
w/o Context: No context examples are used from any series.
2. Context (Xy): Only the context examples corresponding to series X, are used.
Context (Full): The full ICTSP using all context examples.
For each variant, we report the MSE (Xj) corresponding to each series j in the test set forecasting results. Below are the results for the three experimental settings.
A.11.1 ETTM2 >> ETTM2 (FULL DATA)
From the results below, we can observe that using the context examples from specific series X, improves the model's performance on the corresponding MSE (X,). However, we cannot fully attribute this improvement to the benefits of ICL. In the full data setting, the model's parameters already contain information about the relationships between series within the dataset. As result, the model may still leverage the information from context examples through these fixed relationships encoded in the weights. We can that introducing X, context not only improves the performance on X, but also significantly enhances the performance of other series that are highly correlated with X,. Therefore, we cannot really distinguish whether the improvement arises from fixed series relationships in the parameters or from ICL.
Table 16: Ablation study results of understanding the specific contributions of context examples. The models are trained on ETTm2 and tested on ETT'2..
Makel
Couent
Coazan (Nd0
Tew 11M
GPT-TS
PachiST
Traedeer
M52 0.1195
13.234
0.152
E.195
11134)
3.209
0L196
01.322
01.355
18:30
11:83
0.407
11:11
11.13
1100
0111
11:00)
3.155
0.111
11.36
01317
345 (2:3
0.578
(1.72
0.80
(070)
0.278
3
0283
0/66
0.53
26

---

Published as a conference paper at ICLR 2025
A.11.2 ETTMl ETTM2 (ZERO-SHOT)
We conducted zero-shot experiments where ICTSP trained on ETTml and evaluated on ETTm2. In this scenario, the model parameters contain no fixed effects from the target dataset. Without using context examples, the model relies solely on its fixed weights learned from the source dataset to perform temporal forecasting inference for each target token, resulting in significantly worse MSE of 0.208 compared to other results.
We can observe the performance imporvements when introducing context tokens from Xj corre- sponding series j. Since the model parameters lack prior knowledge the channel structure of the target dataset, the improvement is most pronounced for the corresponding X₃ and diminish for other series. This task-specific improvement originates from ICL, where the model uses a task's question-answert prompt to enhance performance on the specific task.
Through these experiments, we effectively disentangled the influence learned channel structure from the effectiveness of ICL. This demonstrates the true reason behind ICTSP's strong performance on zero-shot tasks.
Table 17: Ablation study results of understanding the specific contributions of context examples. The models are trained on E and tested on ETTm2.
Methel wa Eornet
Creaut Vi)
Caaver
Curver
Creverki (Kc)
(3)) [ Coant (Ndo
Tee-11
GPI-TS
PainTT
ToweNe
M38 1X27
18:20
11334
8.303
0,301
11334
3.304
0310
01213
0.275
MS8-1.3 (3,
0.191
08131
8:115
B/46
0.13
3.113
0453
0/12
0134
358
0,385
0.835
8.00
3.85
0022
338
0.303
0345
6.93
0221
A.11.3 ETTHLEETTH2ETTTL ETTM2 (ZERO-SHOT)
We further trained ICTSP on combined dataset (ETTh1, ETTh2, ETTml) containing more data- points. Compared to the previous experiment, we observed two key improvements:
The model's s ability use different context prompt tokens achieve corresponding performance improvements was further enhanced, resulting in better overall performance than in the previous experiment.
2. The task-specific performance improvement effect became more pronounced for the series corresponding to the provided context.
These results highlight the dominant role of ICTSP's ICL ability in achieving strong performance on zero-shot tasks.
Table 18: Ablation study results of understanding the specific contributions of context examples. The models trained ETTml and tested ETTm2.
Mathod
Context
Context (Xa)
Contest (Xx)
Context (Xa)
Contest (X4)
Consest (X,)
Contest (Xx)
Context (X;)
Context (Pull)
MSE
MSE(X2)
0.225
0.207
0.206
0.202
0.204
0.198
0.198
MSF: (Na)
0.116
0.101
0..007
0.099
0.105
0.100
0.101
0.102
0.098
MSE (Xa)
0.442
0.432
0.413
0.439
0.400
0.43
0.488
0.414
0.392
MSE
0.126
0.110
0.112
0.109
0.110
0.107
0.112
0.108
0.107
(SE(X.)
(X,)
MSE (All)
0.189
0.174
0.174
0.175
0.172
0.174
0.173
0.170
0.167
27


Reviews:
[{"review_id": "A50l5DtP9X", "reviewer": "Reviewer_hkWk", "summary": "The paper introduces a model named \"In-context Time Series Predictor\" (ICTSP), designed specifically for time series forecasting (TSF). Unlike traditional Transformer-based models, ICTSP adopts a novel input token structure, resulting in improved performance.", "questions": "The provided link to the code repository is inaccessible, which hinders the reproducibility of the results. Could you please update or fix the link to the source code?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "ICTSP model transforms time series forecasting tasks into input tokens, aligning closely with the inherent mechanisms of the Transformer model and efficiently leveraging its contextual learning capabilities.", "weaknesses": "1. The innovation appears limited. While the paper extensively discusses the approach from an in-context learning perspective, the methodology itself seems simplistic and more akin to a heuristic trick due to excessive textual explanation without solid theoretical support.\n\n2. The paper does not address whether this model could be applied to other time series tasks, such as classification, interpolation, or anomaly detection. The utility of the model appears confined to the forecasting tasks, necessitating stronger performance to justify its specialized use.\n\n3. The significant improvements in zero-shot tasks over full-data tasks lack detailed explanations. The paper does not sufficiently clarify why the ICTSP demonstrates a pronounced advantage in few-shot and zero-shot learning scenarios than in full-data tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xcQh0pHM8j", "reviewer": "Reviewer_NX33", "summary": "This paper presents the In-context Time Series Predictor(ICTSP), which utilizes the in-context capabilities for time series forecasting tasks. It proposes an innovative way to construct the tokens with context examples, showing promising results across multiple experimental settings.", "questions": "Based on the listed weakness,\n1. Can you add more discussions for these methods based on patching embedding from an ICL perspective?\n2. For the Full-data TSF setting, can you rerun the baselines under the same experimental settings?  Alternatively, you could articulate the rationale behind your experimental settings, which would make the comparisons more convincing.\n3. Can you provide more details for Computational Costs? What are the number of layers and hidden dimensions of different models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and fluent.\n2. This paper's motivation is relatively novel. It proposes utilizing the in-context learning capabilities of large language models (LLMs) for time series forecasting tasks. \n3. The experiments are comprehensive, encompassing various experimental settings and datasets.", "weaknesses": "1. The formulation of TSF transformers is not sufficient. As far as I am concerned, there are some methods [1,2] that utilize patching embedding to form the input tokens. These methods cannot be simply categorized as Temporal-wise Transformer or Series-wise Transformer in Section 2.2\n\n2. The baselines results for baselines are collected from source papers. However, this paper applies different input time series length from these papers, which may lead to an unfair comparison.\n\n3. The details of Computational Costs and Table.5 are not provided. \n\n\n[1] Nie, Yuqi, et al. \"A Time Series is Worth 64 Words: Long-term Forecasting with Transformers.\" The Eleventh International Conference on Learning Representations.\n\n[2] Jin, Ming, et al. \"Time-LLM: Time Series Forecasting by Reprogramming Large Language Models.\" The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9hIc2w8Z4D", "reviewer": "Reviewer_H6bM", "summary": "This paper present In-context Time Series Predictor (ICTSP), a new framework of doing time series forecasting. It formulates the time series prediction task into the form of in-context learning, such that in the input data is split into a few forecasting examples to guide the model complete the target prediction with the adjacent input. Experimental results demonstrate the method’s effectiveness.", "questions": "1. Could you provide results with ICTSP’s input length ($L_I$) set to 512 for a more equitable comparison with other baselines in Table 1 and 6?\n2. Could you conduct an ablation study on in-context examples, possibly using randomly selected examples (without look-ahead bias) instead of adjacent history to clarify the influence of this design choice?\n3. Could you provide more detailed analysis or visualizations related to the Adaptive Model Reduction to support its impact on performance?\n4. How does ICTSP handle cross-channel dependencies, especially in cases with specific inter-channel relationships in the data (like traffic)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. **Novel Framework**: The paper proposes an innovative approach to time series forecasting by adopting in-context learning, which could inspire further research and development in this field.\n2. **Comprehensive Experiments**: The authors present extensive experimental results, including ablation studies, which provide a robust foundation for evaluating ICTSP’s effectiveness.\n3. **Clarity of Presentation**: The paper is well-written and clearly presented, facilitating a good understanding of the approach and its contributions.", "weaknesses": "1. **Comparative Fairness**: The baselines use an input length of 512, while ICTSP utilizes an input length of 1440. This discrepancy could impact the fairness of the comparisons.\n2. **In-context Example Selection**: ICTSP frames forecasting as an in-context learning task, but it uses adjacent historical data as context examples. This approach raises questions about whether the model truly learns from in-context examples or simply encodes the input tokens in a different manner.\n3. **Model Reduction Analysis**: Although the paper highlights the adaptive model reduction feature of ICTSP, there is limited visualization or analysis to support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGyD8Lk20F", "reviewer": "Reviewer_uZY9", "summary": "The authors propose a new in-context learning framework for time series forecasting by using \"time series forecasting tasks\" as input tokens. They show that this new framework outperforms the more traditional Temporal-wise Transformers and Series-wise Transformers across full-data, few-shot, and zero-shot settings.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "In-context learning for time series forecasting is an important problem with wide applications. As far as I can tell, using \"time series forecasting tasks\" as input tokens is a new and interesting idea, and the authors demonstrate that it indeed has utility using systematic benchmarks. The approach has a \"category theory\" flavor to it, and the differences from earlier frameworks are well explained.", "weaknesses": "In terms of presentation, I think there is room to make it more accessible to readers not directly working in the same field. In particular, more detailed explanation and illustration of the \"using time series forecasting tasks as input tokens\" idea and how it is implemented could be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)