In-Context Fine-Tuning for Time-Series Foundation Models
Authors: Matthew Faw
Rajat Sen
Yichen Zhou
Abhimanyu Das
Venue: ICML 2025 poster (2025)
Keywords: Time-series forecasting
Foundation models
In-context learning
Primary area: deep_learning->sequential_models_time_series

Abstract: Motivated by the recent success of time-series foundation models for zero-shot forecasting, we present a methodology for _in-context fine-tuning_ of a time-series foundation model. In particular, we design a pretrained foundation model that can be prompted (at inference time) with multiple time-series examples, in order to forecast a target time-series into the future. Our foundation model is specifically trained to utilize examples from multiple related time-series in its context window (in addition to the history of the target time-series) to help it adapt to the specific distribution of the target domain at inference time.  We show that such a foundation model that uses in-context examples at inference time can obtain much better performance on popular forecasting benchmarks compared to supervised deep learning methods, statistical models, and other time series foundation models.  Interestingly, our in-context fine-tuning approach even matches the performance of a foundation model that is explicitly fine-tuned on the target domain.

In-Context Fine-Tuning for Time-Series Foundation Models
Matthew Faw! Rajat Sen? 2
Yichen Zhou? 2 Abhimanyu Das?
Abstract
hered to the traditional supervised machine learning frame- work of having to train forecasting model on task-specific training data, before being able to perform forecasting for that task. On the other hand, in Natural Language Process- ing (NLP), Large Language Models (LL.M) (Radford et al., 2019 Brown et al., 2020) have shown the promise of foun- dation models: a single pretrained model can perform well and adapt to tasks like translation, code generation during inference time in zero-shot or few-shot manner.
Motivated by the recent success of time-series foundation models for zero-shot forecasting, we present methodology for in-context fine-tuning of time-series foundation model. In particu- lar, we design a pretrained foundation model that can be prompted (at inference time) with multiple time-series examples, in order to forecast a target time-series into the future. Our foundation model is specifically trained to utilize examples from multiple related time-series in its context window (in addition to the history of the target time-series) to help it adapt to the specific distribution of the target domain at inference time. We show that such a foundation model that uses in in-context ex- amples at inference time can obtain much better performance on popular forecasting benchmarks compared to supervised deep learning methods, statistical models, and other time series founda- tion models. Interestingly, our in-context fine- tuning approach even matches the performance of a foundation model that is explicitly fine-tuned on the target domain.
Motivated by the success in NLP, there has been significant work in recent years on time-series foundation models for forecasting, ranging from re-purposing LLMs directly for forecasting (Gruver et al., 2023) to fine-tuning pretrained LLMs on time-series data (Zhou al., 2023; Chang et al., 2023) to pretraining time-series foundation models from scratch (Das et al., 2024; Goswami et al., 2024; Woo et al., 2024; Ansari et al., 2024; Garza & Srrgenthaler-Canseee 2023). The last approach, in particular, has been shown to obtain strong zero-shot accuracy, rivaling the best super- vised models trained specifically for the target datasets.
Several of these papers (Zhou al., 2023; Ansari al., 2024; Goswami et al., 2024) have shown an opportunity for further accuracy improvement by fine-tuning of their pretrained models on target datasets. However, this breaks the zero-shot paradigm that precisely makes these time- series foundation models so appealing to practitioners who do not want to build training pipelines. This raises a natural question: Can we recover the benefits of fine-tuning a time- series foundation model by providing examples from largel dataset at inference time?
1. Introduction
Time-series data is ubiquitous in several domains such retail, finance, manufacturing, healthcare, and the natural sciences. In many of these domains, time-series forecasting, i.e, predicting time-series into the future, is a critical prob- lem - for example, in applications like retail forecasting, cli- mate and weather predictions, and traffic forecasting. In the last decade, deep learning approaches (Salinas et al., 2020; Oreshkin et al., 2020; Sen et al., 2019) have become popular in forecasting, often outperforming statistical approaches like ARIMA (Box & Jenkins, 1968). However, until re- cently, deep learning approaches for forecasting have ad-
At the same time, the first generation of time-series foun- dation models lack some of the desirable features of LLMs with respect to in-context learning: the zero-shot perfor- mance of an LLM can be greatly improved at inference time by using its context window for prompting techniques such as few-shot (Brown al., 2020), chain-of-thouuht (Wei et al., 2022b) or instruction tuning (Wei et al., 2022a). These papers have shown emergent in-context learning abilities for LLMs. In particular. if we prompt them with related exam- ples and demonstrations then ask a specialized question, the model is able to reason similarly for the question at hand.
Georgia Institute of Technology. Part of this work was done while the author was Student Researcher and Visiting Researcher Google Research. *Google Research. Correspondence to: Matthew Faw <ffaw3 gatech..ctcc..
Proceedings of the nrnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
In this work, we study methodology to enable similar in- context ability for time-series foundation models i.e. being

---

In-Context Fine-Tuning for Time-Series Foundation Models
able prompt the model with time-series examples from the target domain, and recover the benefits of domain-specific fine-tuning. We refer to this as in-context fine-tuning."
Took
historical event happexed.
Edeatiify the
Porecant fucice hourly traftic on Highhay So. 1.
0: The Trunch havolution
Biggheyy No.
Incoottot
oxappis
31 1759 1779
1816
modeen Olympic Gowns
Troffie Peliern
Righmay Ny
Pattarn
We train foundation model that lets us forecast a time- series by providing in its context window not just the his- torical values of the time-series, but also examples from other related time-series that could help the model adapt, at inference time, to the distribution of the target time-series. For example, consider a highway traffic prediction system that stores hourly data from the last week, in order to predict the future hourly traffic for a particular highway. Consider a time-series foundation model that has not seen data in pretraining that captures the temporal patterns in this traffic data. Then, simply prompting the model with the previous week's traffic time-series for that highway might not be enough to obtain accurate zero-shot performance. However, adding to the prompt historical traffic data from other high- ways and weeks, might help the model better adapt to the traffic data distribution.
A:
6: Leunch of the Vogigge
Highway No.
Traffic Pattern
Queskor
Q: Fooof of rermtt Sast Thooren
Hikey
I11 Blgghwyy No. 1 Traffie
M
Figure I. Analogous to few-shot prompting of foundation LLM (left), we train time-series foundation model to support few-shot prompting with an arbitrary number of related in-context time- series examples (right). The dashed box encloses the full context window/prompt.
context fine-tuney model is 6.8% better than the base model we start from, while also being 5% better than the next best baseline. More importantly, (and perhaps surprisingly), it achieves similar performance to that obtained by explicitly fine-tuning the base model on the training split of every dataset in the benchmark.
The main contributions of our paper are as follows:
2. Related Work
(i) We introduce the study of in-context fine-tuning for time- series foundation models, and propose the use of prompts that not only include the usual history of the target time- series for forecasting, but also include related time-series examples in-context.
As mentioned previously, there has been a spurt of re- cent work on time-series foundation models for forecasting. These approaches can be broadly divided into three cate- gories. (i) Prompting LLMs like GPT-4 directly predict the future of numerical series encoded as text. This was investigated in LLMTime (Gruver et al., 2023); despite the initial promise subsequent works have shown that such approaches can be lacking in accuracy (Woo et al., 2024; Das et al., 2024). (ii) Fine-tuning pretrained LLMs like GPT2 on time-series data with adapter layers (Zhou et al., 2023; Chang et al., 2023). These approaches have mostly been shown work well in the tattssetttodattsee transfer learning setting (rather than in the zero-shot setting), and they are also disadvantaged from having to use excessively large models due to their LLM backbones. (iii) Pretraining transformer based models from scratch on huge volumes of time-series data, which seems to be the most promising ap- proach towards times-series foundation models (Das et al., 2024; Garza & Senpeetthale-CCaseeee 2023; Ansari et al., 2024; Woo et al., 2024; Goswami et al., 2024). Indeed, some of these models have shown superior zero-shot accu- racy when compared to supervised deep forecasters even on datasets that are outside of their pretraining set.
We propose a methodology for training such a model by starting from a base time-series foundation model and con- tinue pretraining it with in-context examples. Our training is decoder-only (Liu et al., 2018) and can adapt varying his- tory and horizon lengths (up to certain maximum history) and to a varying number of related time-series examples in the context window (again up to a certain maximum number of examples). The resulting model can then learn to borrow patterns from these related examples to perform better on the target forecasting task.
(ii) We empirically evaluate the benefits of in-context fine- tuning using our foundation model, and show that in-context fine-tuning can lead to better zero-shot performance on pop- ular forecasting benchmarks as compared to supervised deep learning methods, statistical models as well as other foun- dation models. In particular, on a well known forecasting benchmark, comprised of 23 datasets not included in the pretaining of our foundation models, we show that our in-
'Terminoloyy: In the LLM domain, this notion is also called "few-shot learning (Brown et al., 2020). "few-shot prompting" (Ye & Durrett, 2022), or "in-context tuning" (Chen et al., 2022). Also, borrowing from I.I literature, we will refer to the generic abil- ity of pretrained foundation models to learn from information in their context window at inference time as "in-context learning". Additionally, we will refer to pretrained models that do not need gradient-updates via explicit training or tuning for an unseen target datasct "zero-shot".
Some of the above papers, e.g., (Ansari et al., 2024; Goswami al., 2024), have additionally shown that their pretrained models' performance fine-tuning the model on examples from the target dataset. While this supervised fine-tuning results in improved per- task accuracy, this practice breaks the zero-shot paradigm in terms of requiring extra training on the target dataset.
be further improved by
In the NLP domain, a defining property of foundation
2

---

In-Context Fine-Tuning for Time-Series Foundation Models
LLM is its ability be further adapted to domain-specific tasks through either fine-tuning or prompting. In particular, LLMs have been shown to perform in-context learning a variety of downstream NLP tasks by prompting them with a natural language instruction (Radford et al., 2019) and a few demonstrations or examples of the task. This phenomenon also referred to as few-shol learning (Brown et al.. 2020). Subsequent works (Min et al., 2022a; Chen et al., 2022) have proposed fine-tuning pretrained L.LM to obtain better performance on few-shot learning prompts. Other papers (Min et al., 2022b; Wei et al., 2023) have empirically investigated how few-shot learning works in LLMs. More recently, Shi et al. (2024) explored a similar idea for in-context pretraining, where they pretrain an LLM on sequences of related documents. This in-context learning ability is widely recognized as being associated with the stacked transformers used in the LLMs, and their theoretical properties are studied in more precise sense (Garg et al., 2022; Von Oswald et al., 2023; Ahn et al., 2024) for simpler function classes such as linear regression.
true sid histooyy and separatoo
Foooo 00000
voooo
Vic.. Vorte--
Figure 2. An example prediction task. The three black dashed lines (separators) separate the three in-context examples {yi.7, ]aE and the history yI.r.- The goal is to predict the horizon y 1144::444 of the history y1:L.
examples of the form {y 1:7, y 1:Ty; I: represent the past time-points of other related time-series (with possibly varying lengths T,. ,T...,,TT_.)) Tn-1). In the case of our motivating example of highway traffic prediction, Yi:z is week of hourly traffic data that highway, and [y 1 Vi.T, ... ........-ii... (1 Yi.r, highways. We plot an example prediction task with three in-context examples in Figure 2.
v } that can
Despite the commonality between time-series foundation models and LLMs, it not obvious how (or even if) the phe- nomenon of few-shot learning for NLP tasks carry over to the time-series setting. There is clear definition of few-shot learning in terms of a time-series foundation model. In fact, prior pretrained time-series foundation mod- els like (Ansari et al., 2024; Das et al., 2024; Garza & Bergenthaler-Canseee 2023) do not provide a clear way for prompting with anything other than the past values of time- series in the context window. The MOIRAI model (Woo et al., 2024) supports the functionality of any-variate fore- casting, which allows the model to take as input arbitrary number of variates (up to a certain maximum). While one could provide in-context examples as additional variates, their approach is mostly aimed at multi-variate datasets. Nevertheless, we also compare with this model in Table
are traffic data on n 1 nearby
Therefore, the enhanced forecasting problem is aimed at training model f.
(I) S: ( SI Vu:t) - SL+IE+H (2)
(2)
(n-1)
As before, our time-series foundation model should be able to handle different values of L and H. Additionally it should be able to support any number of in-context ex- amples (n - 1) ranging from zero to a maximum value. With some abuse of notation, let us index the target task's forecasting history and horizon as the n-th example i.e. (n)
yI., = YI.L+H, where T2 = L + H. Therefore, our decoder-only model will work with examples of the form (1)
3. Problem Definition
Wi v ".... y which are drawn from related time-series. Henceforth, we will refer to (yirr, } the context (synonymous with prompt) supplied to the model.
Time-series foundation models aim to build general pur- pose forecaster that can take in a past history of a target forecasting task, YI.z = {v/1 3/2- - }r where we look back T. time-steps and map them to a forecast FL+1IL+ for horizon length of H. The aim is to have ÿ VL+1:L+H as close as possible to the unseen future 1:+1+4 fr according to some well defined error metric. Such model can be thought of as function,
(i)
4. Model Architecture
Motivated by the strong zero-shot performance achieved by stacked transformer models in decoder-only mode for time- series forecasting, we propose to adapt a base TimesFM model (Das et al., 2024) to leverage the additional infor- mation available via in-context examples. In particular, we first pretrain TimesFM in its original fashion to obtain a base checkpoint TimesFM (base). We then modify the model architecture and continue pretraining from TimesFM (base) using training data with in-context examples (we call
g VI.L IL+LLL
(1)
which is capable for handling different values of T. and H. In this work, we aim to further enhance the abilities of such models by enriching their context. In addition to the target task's history YI:L, we add up to n in-context
3

---

In-Conter, Fine- Fine-Tuning for Time-Series Foundation Models
Curput MLP Layer
Stacked Transfo srers
Rependac layera
retack
S4
SA
Geuneted
SA
Alord to greaaus pazzies
Token
Token
Sepureet Takan
Town
Embedcing L Layar
saparator
peparator
rywk Wey
outpptt wwwpt_eeto___aaee 1ea-128
Figure 3. TimesFM-ICF employs the decoder-only architecture for time-series prediction with in-context examples.
this phase continued pretraining) to obtain a new pretrained foundation model limesFM-ICH.
Figure 4a) together, then the combination of these trends may appear to the model as an entirely different time-series (e.g,, a triangle wave, Figure 4b). Therefore, we choose to insert a common learnable separator token after each in- context example. We visually depict these separators the dashed lines in Figure 4a. When feeding examples to the decoder, we sequentially pass each tokenized patch of each sere-series example to the model, followed by the separator token at the end of an example; depicted in Figure 3.
Adapting their model architecture to make use of the in- context examples is somewhat delicate, and requires modifi- cations the original model. depiction of our proposed model architecture is given in Figure 3. As in their model, our model partitions each example into non-overlapping input patches, and uses a shared input residual block (a one-hidden layer perceptron with skip connection, see Das et al. (2023)), to embed each patch as a token before feeding the tokens into the stacked transformers fashion. The output embeddings mapped to the next output patches via another shared output residual block.
decoder-only
To teach the model to use the new in-context examples, we adapt the original TimesFM architecture to better handle (1) the in-context example separators, (2) the cross-cxand cxss-cx attention, and (3) the positional encoding (by applying no positional encoding when pretraining TimesFM (base), see still able
(a) Multiple linear trends.
(b) A triangular wave.
Figure 4. Concatenating in-context examples together without sep- arators can confuse the model: multiple linear trends look like triangular wave concatenated naively.
Appendix A.3). Despite these changes, to leverage the TimesFM (base) checkpoint, which pretrained for forecasting given just the history of the target time-series. We describe the key details of our architecture design below.
4.2. Cross-Example Attention
In order to allow our model distinguish between different in-context examples, we allow the transformer to attend (causally) all previous patches including the separator to- kens. Note that, if the model did not attend to the separator tokens, then we could never hope to distinguish between the two scenarios from Figure 4a and Figure 4b. By attending to the previous separator tokens, the model can potentially distinguish how many in-context examples have been pro-
4.1. Separators for In-Context Examples
Our context window contains in-context examples from different time-series. Hence the model needs to be able to separate these, since naive concatenation can confuse the model. Consider the example in Figure If naively concatenate multiple in-context examples (e.g., linear trends,

---

In-Context Fine-Tuning for Time-Series Foundation Models
cessed so far.
Then, the output of the stacked transformer layer for token (i) can be described as:
Although at the input to the stacked transformer use common separator token to separate the examples, the out- put tokens corresponding to the positions of these separator tokens can play a much more nuanced role as we proceed through the subsequent transformer layers. As the output tokens corresponding to these separator tokens causally at- tend to all previous tokens, after several transformer layers these tokens can, for instance, potentially summarize the information in all the patches corresponding to their exam- ple and/or convey the separation boundaries of the different in-context examples to the model.
= Stacked
((L) 1:J, 1:J.-1
((-11
We emphasize the output 0f for token 1 defined above depends on (i) all previous (unmasked) tokens t ( 1' and j' [[T2/p]], (ii) the i- separator tokens [+/++1+1 o for i' < i, and (iii) the tokens 7(1)
Ligj for the current example.
Finally, we feed the outputs o ( for each example i with patch j from the stacked transformer through a residual block to obtain the predicted time-series:
4.3. Overall Model
Since our model builds upon the TimesFM architecture (Das et al., 2024), we introduce a similar notation style for ease of exposition. The model processes in-context examples in the following fashion. Starting with an example input {y 1.TT......> yier, J. each example Vi:;; is partitioned into input patches of length p:
y pi+lipt = OutpurResidualllayrr("""
This corresponds to the model's prediction of the next h steps (output patch length) of pp ps+1:pt+
(i)
We use the same loss function as original TimesFM model.
Vj E [[T]/>]] and i E [m]. 8/00 y 1(i-1)-1:pi
(6)
5. Continued Pretraining Data
As mentioned before, we start with TimesFM (base) which was pretrained on diverse corpus of about 400B time- points (see Table 3 in Appendix A.2 and Das et al. (2024) for more details on the datasets). We then continue pretrain- ing it on training data containing in-context examples.
As in (Das et al., 2024), our model takes an additional padding mask i to ensure that it makes good predic- tions on time-series which are not a multiple of the patch length p. Analogously to the partitioning of the example inputs, we partition the padding masks as:
(+)
m ( m 16-1)+1p Vj E [[%]/"]] and e [n].
5.1. Context Generation
We convert individual datasets to generate contexts with in-context examples that the model sees during the contin- ued pretraining. Recall that the original TimesFM model is trained up to a maximum history length of Iymaz = 512. During the training of TimesFM (base) a time-series of length T Lmaa h is loaded for back propagation where h = 128 is the output patch length. Therefore, choose T as the maximum length of our in-context examples. For any time-series in a particular dataset, we use win- dowing with a shift of 1 to generate examples of length T i.c. for a time-series y1.M the possibles examples are (yu:r, WVirryy:T+11 V2:4+1, ... YM-T+I:M): ). For time-series that are less than T in length, we generate padded examples de- tailed in Appendix A.2. Now these examples are packed in groups of n to form the context. We consider two kinds of grouping:
Given these patches and masks, we feed each patch y/ through a common MLP embedding layer to obtain tokens:
InputResiduallaaer(()) (1) O (1 m.m) ()))
We will slightly abuse notation by denoting the separator (2)
token o as [rr//]++ o,, and let the mask for the separator
[+/+++1 = 0 (i.e., the separator tokens are never masked). Write J, [T//p] + 1, the total number of patches (3)
token m (8)
belonging to the example y1.7:
After tokenizing the input patches, we feed the tokens, to- gether with learnable separator token or, autoregressively to the stacked transformer layers in decoder-only mode. We take m (4) to be the last entry of mn! (0), and denote the sequence of token/mask pairs corresponding to example as
Times-series level: For a long time-series, we can split the original time-series into shorter time-series examples, each of length T, then select n of those shorter examples to form the context (8) Im for the original time-series.
2 = ((t()) (1) m1 (1))
( (..m) m))) Vj E [JA].
(1) Intuitively, (+) indicates whether or not patch y is masked from the right. We attend only to patches which are not padded from the right, and have at least unpadded values (see Ap- pendix A.2)
ri;
Dataset level: For each dataset, we can group any n seg- ments of length T from any of the time-series in that dataset,
5

---

In-Context Fine-Tuning for Time-Series Foundation Models
to form a context. For instance, a of n segments from any of the time-series from the Electricity dataset could be grouped to form a context {-
length of 512 unless otherwise specified.
12
6.1. The Fine- Sine-Tuning-per--attaae Baseline
Both time-series level and dataset level groupings guarantee that the grouped examples have similar patterns to borrow from each other.
In all our experiments, we also compare with an ex- tremely strong baseline TimesFM-FT which is the TimesFM (base) model fine-tuned the training split of each dataset and then evaluated the corresponding test split. Our main goal is to study whether TimesFM-ICF can match or surpass the performance of this baseline but in a zero-shot manner (without any gradient updates during inference). In both the benchmarks in Sections 6.2 and 6.3, we perform two kinds of fine-tuning (i) Full: update all the weights of the models (ii) Linear Probe (L.P): update just the input and output MIP layers. Then we report the numbers from the best of the two. The details of fine-tuning are provided in Appendix A.4.
5.2. Dataset Mixture
We choose all datasets in Table 3 (the dataset list used to train TimesFM (base)) other than the four Wiki datasets to generate in-context examples for continued training. The Wiki datasets contain millions of time-series that correspond to a wide variety of articles, which need not be related to each other.
For the remaining datasets, we set the number of examples in each context as n 50 and generate contexts from both time-series level and dataset level grouping. Note that if all the time-series in a dataset have a total of N examples, then generating all (x) such contexts is intractable. Therefore, we randomly generate 20 such groups of n examples as our training contexts.
6.2. Out-of-Domail Forecasting
Prtraiied
TimesFWee
TimesFMMICF
0.636
K-HT5
0.824
ChronesITS (Large)
0:627
K-RFATS
TimesFM (Base)
Chronos-15 (Base)
Following the original TimesFM paper, the training data
Model
Chronoe-15 (5mal3
0.55
loader samples 90% real data and 10% synthetic, with the
Ehronss-1a (Min)
DeepAR
0355
real data mixture providing equal weights to the groups: hourly + sub-hourly, daily, weekly, and monthly datasets. Moreover, we provide equal weights to the two kinds of examples i.e., time-series level and dataset level.
Chrones GPT2
0.50
AutrThets
DLincar
GFT4TS
E:903
6:347
LLMTme-
0.500
Warehet
0.22
Seasonal kalve
5.3. In-Context Example Selection
0.
0.2
D-4
0.6
2.8
3.0
Scaled MASE (GM)
For choosing the 50 examples to be added to the context, we adopt a very simple strategy of using 5 examples from the immediate history of the time-series and the remain- ing examples chosen at random from the history of other time-series in the same datasets. In Section A.9, we experi- ment with some other simple example selection strategies. While we leave more detailed investigation to future work, our results show that even naive approaches like random selection and selecting examples from the immediate his- tory are sufficient to obtain accuracy gains with in-context fine-tuning.
Figure 5, Geometric mean of scaled MASE on the OOD Bench- mark. This benchmark is essentially the zero-shot benchmark used in (Ansari et al., 2024), modified slightly to guarantee a zero-shot evaluation of TimesFM-ICF. Our in-context fine-tuning approach improves the performance TimesFM (base) over all other benchmark models, and achieves the same performance as that of TimesFM-FT. the model which separately fine-tunes TimesFM (base) on the training split of each task before making predictions.
The Chronos zero-shot benchmark is collection of 27 datasets of different training and prediction lengths that cov- ers granularities ranging from minutes to years and domains including finance, demand forecasting, weather, and traffic. We report our results on the 23 datasets from this benchmark that were not used during training of TimesFM-ICC (or the base model). We will refer to this benchmark as the OOD Benchmark and provide more details in Appendix A.2.
6. Experimental Results
Similar to prior works, we report our results on the Chronos zero-shot benchmarks from Ansari et al. (2024), as well as rolling-window evaluation of the ETT datasets (Zhou et al., 2021). No data from these datasets (not even the training splits) was used in the training of our base model TimesFM (base), our in-context fine-tuned model TimesFM-ICH Since TiMFFFM-ICF uses examples of length 512, we report the numbers of TimesFM (base) with a maximum history
The datasets contain time-series with vastly different scales, so we cannot aggregate the raw metrics. Therefore, fol- lowing Ansari et al. (2024), we calculate the MASE for all methods and normalize them by the MASE achieved by a seasonal naive baseline that just repeats the last period's
6

---

In-Context Fine-Tuning for Time-Series Foundation Models
values in the history for the whole horizon. Then we report the Geometric Mean of these scaled MASE values across all datasets. Note that when dealing with normalized met- rics it is better to report the Geometric Mean (Fleming & Wallace, 1986). We borrow the official numbers for base- lines from (Ansari et al., 2024) whenever possible. More details about the benchmark, error metrics and baselines are provided in Appendix A.
datasets in these benchmarks in our pretraining datasets (like Electricity and Traffic). Therefore, for the interest of zero-shot evaluation we use the 4 Electricity Transformer Temperature (ETT) datasets, specifically ETTh1, ETTh2 (hourly) and ETTm1, ETTm2 (15 min).
We conduct the same evaluation as in the long se- quence forecasting evaluation (Woo et al., 2024) on these datasets, focusing on the task of predicting horizon lengths 96, 192, 336, and 720. We provide rolling validation num- bers for the test time-period which consists the last 1/5- th of the time-points. This is standard for these bench- marks (Nie et al.. 2023), where the datasets are split into vaiitttaiittionntees in the ratio 7:1:2.
We include around 20 baselines including task specific (trained on the training set of each dataset and then evaluated) deep learning models such as DeepAR (Sali- nas et al., 2020), N-BEATS (Oreshkin et al., 2020) and WaveNet (van den Oord et al., 2016), statistical local (per time-series) models like as Exponential Smoothing(FTTS) and ARIMA and other leading pretrained foundation models like Chronos- T5 (Ansari et al., 2024), and LLMTime (Gru- ver et al., 2023), more in Appendix A.2. Note that we are unable to compare with MOIRAI (Woo et al., 2024) be- cause its pretraining data has more than 80% overlap with this benchmark. We extensively compare with MOIRAI on long horizon tasks in Section 6.3.
In addition to the evaluations on these datasets from (Woo et al., 2024, Table 6), we evaluate our TimesFM-ICF against TimesFM (base), and the TimesFM-FT model discussed in Section 6.1.
We present the MAE loss for each dataset, averaged over the four horizon lengths 96, 192, 336, 720, in Table 1. For a detailed breakdown of the MAE losses, see Table 9. Note that since the MAE is computed on scaled datasets in this benchmark (Zhou al., 2021), we can directly report the arithmetic mean across datasets. We see that TimesFM- ICF rivals or outperforms TimesFM-FT which was fine- tuned explicitly on the target dataset's 's distribution. More- over, TimesFM-ICF outperforms or equals the performance of all other baselines.
TimesFM-ICF can handle maximum of 50 examples in its context. Whenever the time-series is long enough we include 5 within series examples (including the history of the time-series to be forecasted on) within the context and the rest of the examples are gathered randomly from other time-series in the same dataset (we ablate on a few other ways of choosing examples in Section A.9). Since there is randomness during inference, we average over 10 runs and report the standard error bar.
6.4. Ablation
6.4.1. NUMBER OF IN CONTEXT EXAMPLES
Results. We summarize our results in Figure 5. We can observe that unsurprisingly the per dataset fine-tuned TimesFM-FT is the strongest performer as it starts from a strong zero-shot model and is further finetuned to adapt to the dataset at hand. However, it can be seen that TimesFM- ICF can match the performance of TimesFM-FT completely out of the box at inference time, when supplied with in- context examples. Due to this ability TimesFM-ICF im- proves over TimesFM (base) by 6.8%. Moreover, TM- ICF is better than the next best baseline, PatchTST by 5%.
An important trade-off between speed and accuracy can be achieved by the number of in-context examples. We perform an ablation study of the same using the short context datasets in the OOD Benchmark i.e., datasets where we can only get one example per time-series (the one whose future are predicting) and the rest of the examples are generated randomly from all the other time-series across the dataset. We perform the ablation over these datasets for two reasons (i) this removes the complication of the example selection strategy (studied separately later) (ii) short context datasets are where we know for sure that the positive effects in- context fine-tuning and not from just having a longer context (again studied separately in the next section). The short context datasets are listed in Table 5.
Timing-wise, TimesFM-ICF directly utilizes in-context ex- amples and TimesFM-FT needs to be fine-tuned per dataset. Although TimesFM-ICF requires more time perform each forecast, it overall completes the OOD Benchmark 16x faster than TimesFM-FT does (25 minutes vs 418 minutes, see Appendix A.7).
coming from few-shot examples
We plot the Scaled MASE (GM) vs number of in-context examples in Figure The same figure also shows the total inference time?. The experiments are repeated 5 times and
6.3. Long Horizon Forecasting on ETT
A group of long horizon datasets have been commonly used for benchmarking (mainly) transformer based deep learning algorithms starting from (Zhou et al., 2021). Some of the
'The inference numbers are reported on TPUv5e with 8 tensor cores.
7

---

In-Context Fine-Tuning for Time-Series Foundation Models
Table 1. MAE of TimesFM-ICF against other supervised and zero-shot methods on ETT Rolling Window, averaged over forecast horizons (96, 192, 336, 7203. See Table 9 for a detailed breakdown. We bold the numbers which are the best in every row, and including the ones that are within standard error of the best.
Fow-shot
Zero-hot
Taok-specific
TimesMM-ICT
TimesfM (Base)
Moirai (Small)
Meirai (Base)
Meirai (Large)
Times
iTransfomee
TimesNet
PatchTST
Crossformer
SCINet
FtDMormer
Tastsctt
6TTh
0.405
0417
0.424
0.438
0.469
0.407
0:447
0.450
0.454
0.522
0.452
0.647
0,460
E7T12
0.578
0.396
0.579
0.382
0.377
0.381
0/407
0.497
0.417
0.683
0.515
0,723
0.44)
ETTml
0.378
0.391
0.430
0.388
0.389
0.371
0.406
0.495
0,407
0.452
0
0.329
0.341
0.321
0.320
0.306
0.332
0.332
0.326
0.610
0,401
0.537
0.549
Scaled
100
This shows that our technique of in-context fine-tuning can be more effective than training a longer history model, espe- cially when there is mix of short-history and long- mig-hhitorry time-series. This is because, for in-context fine-tuning, many short time-series can be packed as in-context exam- ples inside the context, while for the case of usual long history training such time-series will just be padded and most of the context is wasted. As shown in the detailed results in Table 6, the long history model performs at par or very slightly better on longer datasets like ERCOT, but degrades on shorter datasets like CIF and Toursim.
Inference Time
80
Scated HASE (0N)
0.80
40
3.29
3.78
Examptes
Figure 6. Scaled MASE (GM) vs number of in-context examples over the short context datasets in the OOD Benchmark. We also plot the total inference time for all the datasets as we vary the number examples. All numbers are averaged over runs with the corresponding one standard error.
7. Conclusions
In this paper, we introduce methodology for in-context fine-tuning of a time-series foundation model for forecast- ing. In particular, we start with a base foundation model and adapt it to be able to effectively utilize, at inference time, not just the history of the target time-series for forecasting, but also in-context examples from related time-series. Our results show that in-context fine-tuning can lead to signifi- cantly better zero-shot performance on popular forecasting benchmarks compared to the base foundation model and state-of-the-art supervised models. Furthermore, it even matches the performance of a version of the base foundation model that explicitly fine-tuned on the target domain.
the standard error bars are reported. We can see that the error decreases monotonically as the number of in-context examples are increased, At the same time, the total inference time increases, signifying a trade-off.
6.4.2. LONGER HISTORY
In this section, we compare the performance of TimesFM- ICF with a modified version of TimesFM (base) pretrained and evaluated with a longer history L = 2048 which we will refer to as TimesFM (LH). We provide the aggregate scaled MASE on the OOD Benchmark in Table 2. We restrict TimesFM-ICF to at most in-context examples (of length 512 each), so it is fair comparison.
While we have chosen a specific base time-series foundation model (TimesFM) for our in-context fine-tuning approach, it would be an interesting direction of future work to study these adaptations for other base foundation models. It would also be interesting to study better forms of relative positional encodings specifically designed for handling in-context ex- amples and length generalization.
Dataset
TimesFM-ICF
TimesPM (LH)
TimesFM (base)
OOD Benchmark
0.777
0.811
0.834
Impact Statement
Table Comparison with longer context TimesFM (LH) model, which has maximum history of 2048. We report the Scaled MASE (GM) on the OOD Benchmark.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
We can see that TimesFM (LH) yields modest 2.4% im- provement over TimesFM (base) (maximum history of 512) while TimesFM-ICF yields a 6.8% improvement.
PPeetrainin performed in manner similar to the latest version of the TimesFM Hugging Face repo
8

---

In-Context Fine-Tuning for Time-Series Foundation Models
References
Fleming, P. J. and Wallace, J. How not lie with statis- ties: the correct way to summarize benchmark results. Communications of the ACM, 29(3):218-221, 1986.
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Trans- formers learn to implement preconditioned gradient de- scent for in-context learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What transformers learn in-context? case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Ansari, A. F., Stella, L., Turkmen, A. C., Zhang, X., Mer- cado, P., Shen, H., Shchur, O.. Rangapuram, S. S.,
Arango, S. P., Kapoor, S., Zschiegner, J., Maddix, D. C.,
Garza, A. and Mergenthaler-Canssco, M. Timegpt-1. arXiv preprint arXiv:23 2023.
Wang, H., Mahoney, M. W., Torkkola, K., Wilson, A. G., Sohlke-Schneider M., and Wang, B. Chronos: Learning the language of time series. Transactions on Machine Learning Research, 2024.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski, A. MOMENT: A family of open time-series foundation models. In Proceedings of the 41st Inter- national Conference on Machine Learning, pp. 161 15- 16152, 2024.
Assimakopoulos, V. and Nikolopoulos, K. The theta model: decomposition approach to forecasting. International journal of forecasting, 16(4):521 -5300, 2000.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, volume 36, pp. 19622-19635, 2023.
Box, G. and Jenkins, G. M. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):11109, 1968.
Haviv, A., Ram, O., Press, O., Izsak, P., and Levy, O. Trans- former language models without positional encodings still learn positional information. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pp. 1382 1390, 2022.
Brown, B. et al. Language models are few-shot learn- ers. Advances Neural Information Processing Systems, 2020.
Challu, C. Olivares, K. G., Oreshkin, B. N., Garza, F., Mergenthaler, M., and Dubrawski, A. NHITS: Neural Hierarchical Interpolation for Time Series forecasting. In The Association for the Advancement of Artificial In- telligence Conference 2023 (AAAI 2023), 2023. URL https: h/tp://aaxvvvvi///eeeeeeia wwg. 22012220122 1120001.111 12886.
Hyndman, R. and Koehler, A. B. Another look mea- sures of forecast accuracy. International journal of fore- casting, 22(4):679- -878 2006.
Jiang, J., Han, C., Jiang W., Zhao, W. X., and Wang. J. Libeity: A unified library towards efficient and compre- hensive urban spatial-temporal prediction. arXiv preprint arXiv:2304.1 1344:33304.14333,, 2023.
Chang, C., Peng, W.-C., and Chen, T.-F. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained 1lms. arXiv preprint arXiv:2308. 08469, 2023.
Koenker, R. and Hallock, K. F. Quantile regression. Journal of economic perspectives, 15(4):143-155 2001.
Chen, Y., Zhong, R., Zha, S., Karypis, G., and He, H. Meta- learning via language model in-context tuning. Associa- tion for Computational Linguistics, 2022.
Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang. Y., Kumar, S., and Bho- janapalli, S. Functional interpolation for relative positions improves long context transformers. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Das, A., Kong. W., Leach, A., Mathur, S. K., Sen, R., and Yu, R. Long-term forecasting with TiDE: Time- series dense encoder. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreview. net / formm? d=ppCC3a@BBB..
Lim, B. and Zohren, S. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, (77(2199)) 2021.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder-only foundation model for time-series forecasting. Interna- lional conference on machine learning, 2024.
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L.., and Shazeer, N. Generating wikipedia by sum- marizing long sequences. In International Conference on Learning Representations, 2018
Dooley, S., Khurana, G. S., Mohapatra, C., Naidu, S. V., and White, C. Forecastpfn: Synthetically-traiee trained zero-shot forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. Association for Computational Linguistics, 2022a.

---

In-Context Fine-Tuning for Time-Series Foundation Models
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zetrr I.. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022b.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, A time series is worth 64 words: Long-term forecasting with transformers. International conference on learning representations, 2023.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174. PMLR, 2023.
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference Learning Representations, 2020.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. ICLR, 2022a.
Petropoulos, and Svetunkov, I. A simple combination of univariate models. International journal of forecasting, 36(1):110-115, 2020.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 2022b.
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv preprint vrXiv:2303.00 038466 2023.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Biloš, M., Ghonia, H., Hassen, N., Schneider, A., et Lag-Ilama: Towards foundation models for time series forecasting. In RO- FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Woo, G., Liu, C., Kumar, A., Xiong. C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers. International conference mu- chine learning, 2024.
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, ((43):181-11999 2020.
Ye, and Durrett, G. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in neural information processing systems, 35:30378-30392. 2022.
Salvador, S. and Chan, P. Toward accurate dynamic time warping in linear time and space. Intell. Data Anal., 11 (5):561-580, October 2007.
Zeng. A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121-111 11121111228 11128, 2023.
Sen, R., Yu, H.-F, and Dhillon, I. S. Think globally, act locally: deep neural network approach to high- dimensional time series forecasting. Advances neural information processing systems, 32, 2019.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, 2021.
Serra, J. and Arcos, J. L. An empirical evaluation of simi- larity measures for time series classification. Knowledge- Based Systems, 67:305-31 -344, 2014.
Zhou, T., Niu, Sun, L., Jin, R., et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322 43355, 2023.
Shi, W., Min, S., Lomeli, M., Zhou, C.,, Li, M., Lin, X. V., Smith, N. A., Zettlemoyer, L., Yih, W.-t,, and Lewis, M. In-context pretraining: Language modeling beyond document boundaries. The Twelfth International Con- ference on Learning Representations, 2024.
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner. N., Senior, A., and Kavukcuoglu, K. Wavenet: generative model for raw audio. In Proc. SSW 2016, pp. 125-125, 2016.
10

---

In-Context Fine-Tuning for Time-Series Foundation Models
A. Appendix
A.1. Illustrative Examples
illustrate visually in Figure 7 how in-context examples can help disambiguate the prediction tasks, by plotting the actual forecasts from TimesFM-ICF with and without the in-context examples. In the left two figures, the history not sufficiently informative for the model to make accurate prediction. By providing in-context examples together with this short history (see the right figures), however, the model is able make accurate forecast.
3-
Tue
prod
hilbory end of horizon separator
2-
2-
monn 100000
booood
0
-2
(a) In-context examples help the history disambiguate between increasing trend and oscillating seasonality.
true
pred
end of history
50-
end of horizon
40
40
30
30
20
(b) In-context examples help the history disambiguate between an increasing linear trend and triangular wave.
Figure 7. Two illustrative examples on how in-context examples can help disambiguate the prediction tasks, that likely patterns based solely on the history can get proved or disproved by the patterns from the in-context examples.
In Figure 8, we plot the forecasts of TimesFM-ICF on the 5 time-series in the Monash Australian Electricity Demand dataset, operating our model three modes: 0 in-context examples, 20 (random) in-context examples, and 50 in-context examples (5 of which are within-series examples). These three configurations have increasingly better MASE scores on this dataset (with MASE values 1, -9, and .8, respectively). The predictions visually appear to improve with the MASE values.
A.2. Baselines on the OOD Benchmark
For the OOD Benchmark which derived from the zero-shot Chronos benchmark, we borrow benchmark evaluation numbers from Table 10 (Ansari al., 2024) for our evaluations of the Chronos models, as well as that LLMTime (Gruver et al., 2023), ForecastPFN (Dooley al., 2024), Lag-Llama (Rasul al., 2023), PatchTST (Nie al., 2023). DeepAR (Salinas al., 2020), WaveNet (van den Oord et al., 2016), TFT (Lim & Zohren, 2021) DLinear (Zeng et al., 2023), N-HiTS (Challu et al., 2023), N-BEATS (Oreshkin et al., 2020), GPT4TS (Zhou et al., 2023), SCUM (Petropoulos & Svetunkov, 2020) AutoETS, AutoARIMA, Auto'Theta, Naïve, and Seasonal Naïve (Assimakopoulos & Nikolopoulos, 2000).
We omit the Moirai (Woo et al., 2024) evaluations from this benchmark, since this model's 's training data has more than 80% overlap with the benchmark.

---

In- In-Context Fine- Fine-Tuning for Time-Series Foundation Models
contest hogten
misernann shamptay
aitroontes excmplcs
Inccontec exsmples
7000
3200
0000
5500
0300
4000
3300
6500
6000
3:00
5000
4000
3200
7700
1500
1400
1000
350
MM
505
M
250
Figure 8, Visualization of TimesFM-ICF predictions on the Monash Australian Electricity dataset
12

---

In-Context Fine-Tuning for Time-Series Foundation Models
We conduct the remaining evaluations ourselves, using the datasets available at this URL from the Chronos authors. We detail the datasets used in our evaluation in Table 4. To ensure evaluations are zero-shot, omit the M4 Quarterly, M4 Yearly', Traffic,, and Weather" datasets from the Chronos zero-shot benchmark.
We give detailed breakdown in Appendix A.6.
A.3. Details of Pretraining TimesFM (base)
We start from the model architecture in Das et al. (2024) then create TimesFM (base) with 16 attention heads, 50 layers, an input patch length of 32 and output patch length of 128. The model dimension is set to 1280. We use the learning rate schedule in (Vaswani et al., 2017) with peak learning rate of 5e - 4. The hidden dims of both the residual block and the FFN in the transformer layers are set as the same as model dimensions. We keep layer norm in transformer layers but not in the residual blocks. The pretraining datasets are detailed in Table 3.
The only difference between the model in Das al. (2024) and our base model is that we use No Positional Encodings (NoPE) instead of the absolute positional encoding (Vaswani et al., 2017).
Based on the findings in Haviv et al. (2022), we create the pretrained TimesFM (base) checkpoint with NoPE, in contrast to the absolute positional encodings (Vaswani et al., 2017) used in the original TimesFM model. We note that we can achieve the same accuracy reported in the original TimesFM paper without using any positional encodings. Indeed it has been hypothesized in Haviv et al. (2022) that the presence of causal attention itself can encode positional information when there are more than one stacked transformer layers.
The advantages of NoPE for our continued pretraining two fold: (i) NoPE models usually have better length generalization, which is particularly important here since we increase the prompt length by adding in-context examples to the context. (ii) If we use the original absolute positional encodings used in (Das et al.. 2024), the meaning of these positional encodings in the base model would be different from their meaning during the continued pretraining with in-context examples. This could cause problems for the continued pretraining phase.
Empirically, NoPE leads to no loss in accuracy on validation during training, and works on par with other positional encodings that generalize length, e.g., FIRE (Li et al., 2024), Figure 9.
A.4. Details of TimesF Fine-Tuning Per Dataset
On both the OOD Benchmark and the Long Horizon ETT, we also compare with TimesFM-FT which fine-tunes on the train split for every dataset and the forecasting on the corresponding test split. For all our fine-tuning runs, 16 and (1) up to 10k iterations for the OOD Benchmark and (2) up 100k for the Long Horizon ETT. We use a maximum learning rate of 1e-3, with 500-step linear warm-up and exponential decay. Note that this means that the fine-tuned model will see many more training examples than the in-context examples given to our model. For the fine-tuning runs, we use the same decoder only loss function that was used in the original pretraining of TimesFM (base). We do two kinds of fine-tuning:
batch size of
Full: All weights in the model are updated during fine-tuning.
Linear Probing (LP): We hold the transformer weights fixed and only update the parameters in the input and output residual blocks.
A.5. Additional Details of TimesFM-ICF
We continue to train TimesFM-ICF model from TimesFM (base). Therefore, most of the parameters in the model remain the same. Here, are the key training details that are unique to TimesFM-ICF:
Separator Token: We have a trainable separator token that is also updated during the continued pretraining. The token is nothing but a learnt embedding whose dimension is equal to the model dimension i.c. 1280 in our case.
http://mpppiiiiot//////ae-.. -
13

---

In-Context Fine-Tuning for Time-Series Foundation Models
Number of Examples: We use maximum of 50 in-context examples for each context during training.
Padding: In short datasets like M4 yearly and quarterly, each time-series might have number of time-points much less than T = 640. Sometimes the number of time-points are even less than our input patch length p 32. For such cases, whole time-series can fit into one of the examples and they are preprocessed in the following manner:
If the length of the time-series I is less than p. we left pad with k padding time-points such that p < +++ 2p. This is because we want the decoder only model to predict something meaningful for the second patch after seeing the first patch and if not, is penalized by the loss on the second patch. If the > p. do not need to perform this left padding.
- Lastly, we right pad such that the length of the total padded example is T' = 640.
Note that the last patch in such examples would be padded from the right, i.e., they will have real time-series values for the first few points and padding for the rest. We make sure that such incomplete from the right patches are not attended by subsequent tokens belonging to examples coming after.
The continued pretraining datasets are detailed in Table 3.
A.6. OOD Benchmark Detailed Results
We give a detailed breakdown of the zero-shot evaluations on the datasets from Table (displayed in Figure 5) in Table 6 with additional baselines as mentioned in Appendix A.2. We report the mean absolute scaled error (MASE) (Hyndman & Koehler, 2006), which, for given time-series Y1:L+H = ..l....,V,,..................... ,LL+H with context and horizon and seasonality parameter S, together with a predicted time-series YL+I-L+ (ÚL+11-..3 (L+H), is defined as the mean absolute error for the forecast normalized by the seasonal naïve forecast error the context:
(ASE( L-S
L+H E Li=lt1 De-
Vel
21=1 Be
V
Each evaluation of TimesFMMIIC is averaged over 10 random seeds, where the randomness corresponds the random selection of in-context examples used to make predictions. The reported MASE numbers are averaged over the five evaluations of the dataset. The confidence intervals in Table 6 correspond to one standard deviation of the 5 evaluations, averaged over the dataset.
Since TimesFM (base) supports probabilistic forecasting, in Table 7, we additionally report the weighted quantile loss (WQL), which averages the quantile loss QL,, of (Koenker & Hallock, 2001) over quantiles {0.1, 0.2. 0.9). The quantile loss is defined as follows: given quantile a E (0, 1) and predicted quantiles (()) = (12411..... .44 for 1i+H
time-series y with context L and horizon H, the quantile loss at level a of an observation is:
[ali=- a (
if yf > 4 (a) otherwise
Qu, (ge v1)
(1 (1-a) (1 y1)
The weighted quantile loss over a dataset D, where each y; E D has a predicted quantile q is given by:
2 2i 2++
QLa
WQL,
ΣiED E1+4 Mite
Then, the weighted quantile loss is the average of WQL, over a E (0.1,0.2.....0.333
Note that, in addition to the detailed results in Tables 6 and 7, additionally report the geometric mean of the MASE on each dataset, normalized by the MASE of the seasonal naïve baseline. This geometric mean is then averaged over the 5 evaluation runs. We report both the geometric mean of the 23 datasets from the Chronos zero-shot benchmark for which we are zero-shot (see Table 4), i.e., that were not used during training of TimesFM-ICF or the base model TimesFM (base), as "Geometric Mean (ZS)". We also report the the geometric mean over all 27 datasets from the Chronos zero-shot benchmark as "Geometric Mean (All)*.
14

---

In-Context Fine-Tuning for Time-Series Foundation Models
Table List of datasets included in pretraining TimesFM (base). All datasets except the Wiki datasets also repurposed for continued pretraining LimesMM-ICF with in-context examples. The datasets labeled LOTSA are obtained from the LOTSA collection (Woo et al.. 2024). For continued pretraining. the Wiki dataset can be clustered into groups of related articles, and time-series each group could be deemed to form separate dataset. But we leave such preprocessing of the Wiki dataset for future work and leave these datasets out of our continued pretraining.
Dataset
Granularity
# Time series
Synthetic
3,000,000
Electricity
Hourly Hourly
321
Traffic
862
Weather (Zhou et al., 2021)
10 Min
42
Favorita Sales
Daily
1 111,840
LibCity (Jiang et al., 2023)
15 Min
6,159
M4 hourly
Hourly
414
M4 daily
Daily
4,227
M4 monthly
Monthly
48,000
M4 quarterly
Quarterly
24,000
M4 yearly
Yearly
22,739
Wiki hourly
Hourly
5,608,693
Wiki daily
Daily
68,448,200
Wiki weekly
Weekly
66,579,850
Wiki monthly
Monthly
63,151,306
Trends hourly
Hourly
22,435
Trends daily
Daily
22,435
Trends weekly
Weekly
22,435
Trends monthly
Monthly
22,435
LOTSA Azure VM Traces
5 Min
7,179,194
LOTSA Residential LoadPower
Min
6,861,408
LOTSA Borg Cluster Data
5 Min
6,651,848
LOTSA Residential PyPower
Min
5,890,848
LOTSA QTraffic
15 Min
4,334,208
LOTSA London Smart Meters
30 Min
2,681,312
LOTSA Taxi
30 Min
2,175,488
LOTSA Solar Power
4 Sec
115,584
LOTSA Wind Power
4 Sec
115,584
LOTSA Kdd2022
10 Min
77184
LOTSA Largest
5 Min
91,005,824
LOTSA Era5
Hourly
728,078,400
LOTSA Buildings
Hourly
22,937,60
LOTSA Cmip6
Daily
892,769,47 ,422
LOTSA China Air Quality
Hourly
593,532
LOTSA Beijing Air Quality
Hourly
76,032
LOTSA Subseasonal
Daily
165,504
LOTSA Kaggle WebTraffic Weekly
Weekly
4,642,016
LOTSA CdC Flu
Weekly
9,472
LOTSA Godaddy
Monthly
200,640
A.7. Detailed Timing on the OOD Benchmark
We provide the comparison between TimesFM-ICF and TimcsFM-FT regarding their benchmark timing and accuracy in Table For a fair comparison we calculate the time here assuming that fine-tuning and inference tasks on all datasets are completed in a one-by by-one sequential manner. Fine-tuning on each dataset requires more time than applying in-context examples while does not always guarantee performance over TimesFM-ICF.
15

---

In- Fine-Tuning for Time-Series Foundation Models
Table Details of the OOD Benchmark: We report our results the datasets from the Chronos zero-shot benchmark that used during training of TimesFM-ICE or the base model TimesFM (base).
Dataset
Category
Frequency
Num TS
MinLen
AvgLen
MaxLen
Horizon
Australian Electricity
energy
30min
5
230736
23
232272
48
Car Parts
retail
IM
2674
51
51
51
12
CIF 2016
banking
IM
72
28
98
120
12
Covid Deaths
healthcare
1D
266
212
212
212
30
Dominick
retail
ID
100014
201
296
399
ERCOT Load
energy
IH
8
154854
154854
154854
24
ETT (15 Min., Last Window)
energy
15min
14
69680
69680
69680
24
ETT (Hourly, Last Window)
energy
1H
14
17420
17420
17420
24
Exchange Rate
finance
IB
8
7588
7588
7588
30
FRED-MD
economics
IM
107
728
728
728
12
Hospital
healthcare
1M
767
84
84
84
12
M1 (Monthly)
various
1M
617
48
90
150
18
M1 (Quarterly)
various
3M
203
18
48
114
8
MI (Yearly)
various
IY
181
15
24
58
6
M3 (Monthly)
various
IM
1428
66
117
144
18
M3 (Quarterly)
various
3M
756
24
48
72
8
M3 (Yearly)
various
1Y
645
20
28
47
6
M5
retail
ID
30490
124
1562
1969
28
NNS (Daily)
finance
ID
111
791
791
791
56
NN5 (Weekly)
finance
IW
113
113
113
8
Tourism (Monthly)
various
1M
366
298
333
24
Tourism (Quarterly)
various
IQ
427
30
99
130
8
Tourism (Yearly)
various
IY
518
11
24
47
4
All timing is reported TPUv5e with 4 tensor cores.
A.8. ETT Rolling Window
We provide the detailed results of the mean absolute error (MAE) of TimesFM-ICE against other supervised and zero-shot methods on ETT Rolling Window in Table 9. The MAE of prediction F1+1:44 of a time-series VI.L+ is:
MAE(SLHLLHH = 1 1VL+LL+H YL-1L+H111
Note that, in addition to the baselines Table 1, we additionally evaluate against the Moirai (Large) model operating in multivariate forecasting mode. We refer to Moirai operating in this mode Moirai-MV. We provide Moirai-MV with total of 50 multivariate features 21 (the maximum possible number for these datasets) from the time-series' history, and 29 features from randomly selected time series. We select this for several reasons: (i) by selecting the largest possible context history, we provide the model with the largest possible number of relevant features, (ii) according to (Woo al., 2024), Moirai is trained to accommodate at least 50 multivariate features, and (iii) our model is trained to accommodate maximum of 50 in-context examples. We choose each multivariate feature to have the same length (512) since both TimesFM-ICF and Moirai-MV is trained to accommodate such example lengths. Note that we provide exactly the same features/in-conter examples to Moirai-MV and to TimesFM-ICF for ease of comparison. Moirai-MV's performance degrades on these datasets with the additional multivariate features provided.
We remark, however, that the performance degradation of the Moirai-MV model may be explained subtle distinction between in-context examples and multivariate features. All multivariate features used when training Moirai are assumed to be temporally aligned. By contrast, the in-context examples provided during our evaluations are from different time- windows. Thus, by providing in-context examples as multivariate features for Moirai-MV, we are evaluating the model outside of its intended use. Nonetheless, we include this result in the appendix (with this caveat) to demonstrate that naïvely concatenating related time-series as multivariate features may not work out-of-the-box.
16

---

In-Context Fine-Tuning for Time-Series Foundation Models
Table Part of the OOD Benchmark which only has for in-context example per time-series. The used for the ablation study in Section 6.4.1. We exclude Dominick evaluation takes significant amount of time on that dataset especially when averaged over multiple rums.
Dataset
Category
Frequency
Num TS
MinLen
Avglen
Maxl.en
Horizon
Car Parts
retail
1M
2674
51
12
CIF 2016
banking
98
120
12
Covid Deaths
healtheare
ID
266
212
212
212
30
FRED-MD
economics
1M
107
728
728
728
12
Hospital
healtheare
1M
767
84
84
84
12
MI (Monthly)
various
1M
617
48
90
150
18
MI (Quarterly)
various
3M
203
18
48
114
8
MI (Yearly)
various
181
15
24
58
6
(Monthly)
various
1M
1428
66
117
144
18
M3 (Quarterly)
various
3M
756
24
48
8
M3 (Yearly)
various
IY
645
20
28
47
6
M5
retail
ID
30490
124
1562
1969
28
NN5 (Daily)
finance
ID
111
791
791
791
56
NN5 (Weekly)
finance
IW
111
113
113
113
8
Tourism (Monthly)
various
1M
366
91
298
333
24
Tourism (Qaarterly)
various
IQ
427
30
99
130
8
Tourism (Yearly)
various
IY
518
24
Table OOD Benchmark (MASE)
840
136
146
1831
180
130
280
136
135
190
110
618
2A0
200
110
500
A.9. Selecting In-Context Examples
practical consideration for in-context fine-tuning is the question of how to choose the (up to 50) related in-context examples put in the context window (at both training and inference time). While our in-context fine-tuning methodology compatible with any algorithm for in-context example selection, in this paper for each time-series forecasting target combination of a) few consecutive examples chosen from the immediate history of the time-series to be forecasted, and b) many examples chosen random (across timestamp and time-series) from the past history of other time-series in the same dataset. We also tried a more sophisticated approach that uses dynamic time warping (DTW) (Serra & Arcos, 2014; Salvador & Chan, 2007) to select the top 20% most "similar" time series to the time-series of interest, and restricting the random examples from within those time-series. In Figure 10 we perform lightweight ablation study to understand the effects of these approaches on the performance in-context fine-tuning on OOD Benchmark, and observe very minor differences in performance among the example selection approaches. Choosing examples from the immediate history, and the remaining 45 examples random seemed outperform the other three approaches (and is indeed the approach we use for TimesFM-ICF in the OOD Benchmark results).
a very simple strategy of choosing
As simple test how our simple choice of in-context example selection could be improved, we performed a grid-search over various splits of in-series and random in-context examples as follows: First, we constructed a validation dataset from the training portion of a subset of the Monash datasets (specifically: weather, traffic, australian electricity, ercot, ETTm, and ETTh). We chose these datasets because they contained many training examples long enough construct up to 20
17

---

In-Context Fine-Tuning for Time-Series Foundation Models
Table OOD Benchmark (WQL)
Wockst
205
641
641
19c
641
1.00
641
1.00
6.10
641
640
5031
110
210
694
601
830
130
130
630
160
180
180
600
686
530
530
130
610
430
130
612
610
640
6.50
4411
611
630
440
130
002
100
190 12E
100
370
100
030
135
100
109
122
205
170
Table 8. MASE and Timing of TimesFM-ICF and TimesFM-FT the OOD Benchmark.
Model
Total Fine-Tune Time
Total Inference Time
MASE
TimesFM-ICF
Not needed
25 min
0.777
TimesFM-FT (FULL)
84 min
28 sec
0.789
167 min
0.794
250 min
0.779
334 min
0.776
418 min
0.776
TimesFM-FT (LP)
29 min
28 see
0.807
57 min
0.805
85 min
0.802
113 min
0.802
141 min
0.799
in-series examples. We measured the validation MASE error of TimesFM-ICF with the number of in-series examples varying from 0-20, and the total number of in-context examples (including randomly selected examples) varying from 1-50. The resulting heatmap is show in Figure 11. The configuration with smallest validation MASE was 11 in-series examples and 34 total examples. The geometric mean MASE ratio (averaged over runs with different random examples selected) was 0.780 + .003 (so within standard error of the MASE value we report in Figure 5).
While we leave to future work a more detailed investigation of how best to chose relevant examples to add to the context, the results in this paper show that even simple approaches like random selection and selecting examples from the immediate history are sufficient to obtain accuracy gains with in-context fine-tuning.
18

---

In- In-Context Fine- Tine-nnnig for Time-Series Foundation Models
Validation error
APE (alecoltee posilional ercooding)
NoPE (no positional enood ng)
43
(funciional posiiional encoding)
3.9
2004
400k
600k
800k
1.2M
1.4M
1.6M
1.5M
2000200
Steps
Figure Validation errors during training time suggest that (1) NoPE better APE, and (2) NoPE performs on par with other positional encodings that generalize length.
Example Selection Strategy
examples
0.784
in-series examples 45 "similar" examples
0.781
10 in-series examples 40 random chunks
0.779
in-series examples 45 random examples
10.777
Scaled MASE (GM)
Figure 10. Scaled MASE (GM) for various in-context example selection strategies for the OOD benchmark: 1) 50 random examples, 2) 45 Random examples and 5 examples from the immediate past history 3) 45 examples chosen at random from similar time-series (according DTW distance) and 5 examples from the immediate past history 40 Random examples and 10 examples from the immediate past history. The error hars are one standard deviation of the evaluations averaged over 10 random seeds.
19

---

In- In-Context Fine-Tuning for Time-Series Foundation Models
Table MAB TimesFM-ICF against other supervised and zero-shot methods ETT Rolling Window
Tew-shet
Zero-chet
Taskopacili:
Mota-M
Maini
FitesFM
iTandoune
TinesNet
PuEiIST
Crosfernur
Hirroo
96.
8:174
1:40
0.386
0-432
1,412
6.788
0.176
0:405
0.455
0-119
01648
0:400
11.90
0441
8440
0142
0.49
0419
B.429
0424
04001
04466
0429
0445
0.44
0.42
0.681
0.48
ETT1]
336
B414
0452
0.24
0,45
0.40
0474
0416
0.45
0469
0466
0.58
0-39
0.60)
0.65
9430
0448
0.500
0488
0.011
0.56
0.007
15
0131
031
0.111
01327
0.10
0.125
1130
3.31
03115
0251
0.50
193
3.11
11300
0.384
0.371
0.778
0.677
0.773
0.000
0.414
0.000
0.66
0.-76
0.688
0.47
ETTa2
336
0,393
8417
0417
0393
04401
0.393
0485
BA42
0.541
0433
0.231
0.541
0.744
0.477
720
0422
0.507
0446
0446
0446
0.21
04444
0.445
04657
0446
0,703
0.657
0.538
0.444
09g
95
3731
0342
0391
6333
0.765
0.315
13357
02235
0.373
10535
0341
192
0.364
0.558
0.376
0.422
0,379
6,300
0.358
0.391
0.357
0.355
0.51
0.585
0.400
0.441
BTTm)
336
0.357
0.42
0.415
0.394
63355
0.381
0.430
3.41
0410
0,515
0.41
0.885
0.459
720
9450
0444
0431
0419
0447
04419
01499
0.40
0439
0,59
0.43
0.530
0.400
D.318
0.991
0.399
0.488
0450
0.995
0.407
0.481
6.29
0250
0,283
0.306
0.318
6.900
0.700
0.330
0.483
0.560
0.545
0.756
D/222
0.345
0335
0.314
0.388
0.311
0344
0.543
0.022
0.59)
0.366
720
0.385
0.599
0.444
0440
0377
0.366
0.379
0.407
0.409
0400
1.042
0.533
0.735
0.415
19g
5.107
0.31
6.10
0.32
0.155
0.7
0.000
0.401
0.532
4.0
6.0
10.0
12.0
14.0
0.77
Goometiic Mean Validation MASE
20.0
0.76
22.0
0.75
28.0
30.0
35.3
38.0
0.72
44.3
4b.3
0.71
20
60
50
7.0
11.0
15.0
32.0
18
19.0
Nom Series Exemples
Figure Heatmap in-context example configurations. The configuration with smallest validation loss has 11 in-series examples and 22 randomly-selected examples.
20


Reviews:
[{"review_id": "LCuIInU5Zt", "reviewer": "Reviewer_BzCE", "summary": "The authors propose a framework to obtain pretrained models for time series forecasting that are capable of doing in-context learning. The authors approach is verified on top of TimesFM (a decoder only pretrained model for time series forecasting) accompanied with extensive evaluations. The authors show that the proposed approach not only outperforms data-specific models and other pretrained models in zero-shot settings, but that it even outperforms fine-tuning variants of the proposed model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns. If our response sufficiently addresses your concerns, we hope that you will consider raising your score accordingly.\n\n\n\n> I would suggest the authors to update the bibliography entries.\n\n\nThank you for pointing this out. We will update the bibliography accordingly in the final version of our paper.\n\n\n> why in the proposed paper the base model is trained on even more datasets (LOTSA datasets) than it was in the original TimesFM paper? I wonder if the results would still hold if the same pretraining methodology was followed as in the original TimesFM paper.\n\n\nWhile our study focuses on the improvement introduced by the in-context fine-tuning continued training over a base model, we intend to start from an up-to-date base model. Therefore we use the same datasets that are specified in the latest version (v2) of the TimesFM huggingface repo (https://huggingface.co/google/timesfm-2.0-500m-pytorch), which does include LOTSA. \n\n\n> In the original TimesFM paper one of the mentioned limitations is that covariates can not be included in the pretrained model. I wonder if the proposed approach in this paper can accomodate for this. Perhaps in-context-samples can be taken from covariates that evolve on time (dynamic features). This would be a clear-cut for practitioners to adopt this model in the industry, as covariates often come as de-facto request.\n\n\nAs our model is trained only with in-context examples coming from the same or a similar time-series that shares the same / a similar distribution as the target time series, we expect that providing other covariates as in-context examples likely would not yield good performance out-of-the-box. Adapting our model to allow for additional covariates is an interesting direction for future work.\n\n\n> I wonder if the provided model can generate probabilistic forecasts and what the corresponding evaluations in terms of mean weighted quantile loss would be.\n\n\nThank you for your suggestion. We have evaluated the wQL from our proposed model along with some of the other baseline methods, and the results show that our model performs significantly better on wQL compared to the baselines. The full results are uploaded separately in Table 2 here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf, and we will add these results to the next version of the paper.\n\n\n> Missing visualization.\n\n\nThanks for pointing out our lack of clarification here. In the current paper, Fig. 7 is a limited visual example of how the ICF model behaves differently from a base model. We’ve created additional visualization on the australian_electricity dataset (which has 5 time series) to demonstrate how the forecast changes with a large number of in-context examples. Refer to Figure 2 in the following link: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In this figure, we plot the predictions of TimesFM-ICF operating in three modes: 0 in-context examples, 20 (random) in-context examples, and 50 in-context examples (5 of which are within-series examples). These three configurations have increasingly better MASE scores on this dataset (with MASE values ~1, ~.9, and ~.8, respectively). The predictions visually appear to improve with the MASE values. \n\n> Report on 27 datasets instead of 23\n\nWe have added evaluations on the missing four datasets (m4 quarterly, m4 yearly, weather, and traffic) in Table 1 (MASE) and Table 2 (WQL) here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In these tables, we additionally report the geometric means of the scores over the 23 (zero-shot) datasets originally reported in our paper (row “Geometric Mean (ZS)”) and over all 27 datasets (“Geometric Mean (All)”). The results of Figure 5 continue to hold when adding the additional 4 datasets (which recall are not zero-shot for our model).", "claims_and_evidence": "The main claim of the paper is that the proposed approach is a direct extension of NLP pretrained models to the field of time series forecasting. Whereas in NLP in-context-learning has shown to be a relevant property, it remains as an open question in time series if this can be extended. The authors claim that the proposed approach provides a positive answer to this question. \n\nPerhaps one of the most relevant points of the paper is that the extension is relatively easy. It is based on adding extra tokens that represent a split between consecutive in-context-samples, so that the model is able to identify that different entities are provided in the context, which can be used to improve the quality of the generated forecast.\n\nFurther, the authors provide extensive empirical evidence on the relevance and methodology to identify in-context-samples. They show that indeed the more samples are provided, the better the performance, albeit at the expense of larger inference time.\n\nThe paper claims are well sustained with extensive empirical evidence. I have a couple of questions about the methodology, but I believe this can be addressed somehow by the authors, and this do not hinder the interesting contribution done in the paper.", "methods_and_evaluation_criteria": "**Evaluation**. The authors provide evaluations in terms of MASE, which assume that the considered models provide only point forecasts. Several of the models here considered are able to provide probabilistic forecasts, and at least in the original TimesFM paper it was claimed that it would be possible to generate probabilistic forecasts as well. Moreover, since the authors main reference for evaluation setup is Ansari et al, which introduces Chronos (a probabilsitic pretrained model), I wonder if the provided model can generate probabilistic forecasts and what the corresponding evaluations in terms of mean weighted quantile loss would be.\n\n**Datasets**. The authors provide extensive evaluations. Although the authors take as reference the experimental set up of Ansari et al, the authors decided to evaluate in 23 of 27 of the datasets. This decision seems to have been made because the excluded datasets where used for pretraining the proposed model. This makes it unclear if the main results of Figure 5 hold if these datasets were included in the zero-shot evaluation.\n\n**Qualitative Evidence**. The paper is missing visualizations of the generated forecasts. It would be great to see if there are any visual clear changes of the generated forecasts when using a larger number of in-context-samples.", "theoretical_claims": "There are not theoretical claims.", "experimental_designs_or_analyses": "The experimental design is methodologically sound. The authors provide interesting studies on the effect of how the in-context samples can be chosen (time-series level and Dataset level), and the amount of samples to be chosen (the larger the better at the expense of larger inference time), and competitive performance against full-fine tuning of the model. In general the authors did emphasis in that no data-leakage happens when doing out-of-domain evaluations.", "relation_to_broader_scientific_literature": "The paper is well relevant in the field of time series forecasting, and it is comparing with state of the art models.", "essential_references_not_discussed": "The authors do a fair job in comparing with other state of the art models.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "I would suggest the authors to update the bibliography entries. Several papers are already accepted at top conferences, and still the authors cite them with their Arxiv entries. Examples of this are:\n\n- Ansari, A. F., Stella, L., Turkmen, C., Zhang, X., Mercado,\nP., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P.,\nKapoor, S., et al. Chronos: Learning the language of time\nseries. arXiv preprint arXiv:2403.07815, 2024.\n\n- Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and\nDubrawski, A. Moment: A family of open time-series\nfoundation models. arXiv preprint arXiv:2402.03885, 2024.\n\n- Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large\nlanguage models are zero-shot time series forecasters.\narXiv preprint arXiv:2310.07820, 2023.\n\n- Haviv, A., Ram, O., Press, O., Izsak, P., and Levy, O.\nTransformer language models without positional encodings still learn positional information. arXiv preprint\narXiv:2203.16634, 2022.\n\n- Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S.,\nZaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bhojanapalli, S. Functional interpolation for relative positions improves long context transformers. arXiv preprint\narXiv:2310.04418, 2023.\n\n- Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating\nwikipedia by summarizing long sequences. arXiv preprint\narXiv:1801.10198, 2018.\n\n- Zhou, T., Niu, P., Wang, X., Sun, L., and Jin, R. One fits\nall: Power general time series analysis by pretrained lm.\narXiv preprint arXiv:2302.11939, 2023.", "questions_for_authors": "1. why in the proposed paper the base model is trained on even more datasets (LOTSA datasets) than it was in the original TimesFM paper? I wonder if the results would still hold if the same pretraining methodology was followed as in the original TimesFM paper.\n2. if the authors are pretraining the base model, why did they decide to exclude certain datasets that could be used for zero-shot evaluations as in Ansari et al? Reading the supplementary material (Section A.5 -- lines 678-681), for some of these cases maybe the context length would have been problematic, but it seems that even more standard datasets like traffic and weather were excluded from the zero-shot evaluation. Whereas I believe that the contribution of the authors is strong enough to not be hindered by these decisions, these issues open questions rather than clarity.\n3. In the original TimesFM paper one of the mentioned limitations is that covariates can not be included in the pretrained model. I wonder if the proposed approach in this paper can accomodate for this. Perhaps in-context-samples can be taken from covariates that evolve on time (dynamic features). This would be a clear-cut for practitioners to adopt this model in the industry, as covariates often come as de-facto request.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ywvZcFZZvH", "reviewer": "Reviewer_pqjj", "summary": "The paper proposes a \"fine-tuning strategy via in-context learning\" for pre-trained time series forecasting models. Essentially, the approach is similar to few-shot learning in LLM as multiple time series are added to the context in addition to the forecasted context.\nThe authors modify an existing architecture and introduce a separation token to utilize multiple series in one context.\nExperiments suggest that the approach is effective and improves the forecasting performance of the pre-trained model.\n\n## update after rebtuall\nwe thank the reviewer for clarification. i think the paper is valuable and i keep my score at accept", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns. \n\n> Only the subset of zero-shot benchmark is utilized. While this is a good idea to preserve the \"zero-shot\" setting, it would be advisable to additionally report results for the full benchmark and report which datasets are non zero-shot as this would make comparison to other papers/models easier.\n\nThank you for the suggestion. We have added evaluations on the missing four datasets (m4 quarterly, m4 yearly, weather, and traffic) in Table 1 (MASE) and Table 2 (WQL) here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In these tables, we additionally report the geometric means of the scores over the 23 (zero-shot) datasets originally reported in our paper (row “Geometric Mean (ZS)”) and over all 27 datasets (“Geometric Mean (All)”).\n\n> As the evaluation includes randomness of the in-context samples, the authors should provide the variation over the individual datasets of the benchmark similar to Figure 9 for the \"overall result\".\n\nWe have also added confidence intervals to Tables 1 and 2 in the link above (https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf). Note that all of our figures already include error bars. It was not clear enough in Figure 5 and we will fix it.\n\n> Context length increases linearly with an increasing amount of context samples, therefore, computation complexity for the transformer increases quadratically\n\nWe agree with the reviewer’s comment here. This quadratical attention computational complexity does not directly translate to a quadratical inference time likely due to (1) the latency of the feedforward layers and (2) optimization of transformer implementation on modern accelerators. \n\n> Why did you decide to call the benchmark OOD benchmark? Typically, OOD is more often referred to situations which one has to detect (OOD detection) as the models do not perform reliable on it. In my understanding, the idea of pre-trained models is that the models should generalize over time series data, including the evaluated zero-shot data. Hence, in most related literature, this evaluation is more often referred to as \"zero-shot\" benchmark.\n\nThanks for pointing this out. We have abused the notion of OOD as out of the pretraining data distribution. We did not call it “zero-shot benchmark” because such a name was mentioned in [1] and we wanted to be clear that our benchmark is its subset that’s zero-shot for TimesFM-ICF.  We will clarify and update the name to zero-shot benchmark.  \n\n[1] Ansari, Abdul Fatir, et al. \"Chronos: Learning the Language of Time Series.\" Transactions on Machine Learning Research.", "claims_and_evidence": "The main claim that the approach can improve the performance of pre-trained time series models is well supported by the empirical results.", "methods_and_evaluation_criteria": "The method is well motivated by the success of in-context learning in LLM.\nA comprehensive benchmark and standard metrics are utilized for evaluation and, therefore, make sense.", "theoretical_claims": "There are no theoretical claims or proofs", "experimental_designs_or_analyses": "The experiments in section 6 seem appropriate and sound.\nThe selected benchmark data is especially a good choice as it is more comprehensive than a lot of other work.\nThe ablation experiments are reasonable and give important insights to justify the approach (comparison to long context model) as well as the specific in-context sample selection procedure.\nRegarding the \"in-context sample selection procedure\", I would suggest to further add the performance variation over the runs over the individual datasets (similar to Figure 9 for the overall results).", "relation_to_broader_scientific_literature": "The approach is embedded in the literature of pre trained time series models, more specifically in pretraine time series forecastin models.\nThe contribution can improve the performance of such models in general, as it does not necessarily depend on a certain architecture. Further, it might extend the potential use case scenarios for these models\nFurther, it is related to in-context learning in general, which became especially popular for LLM.", "essential_references_not_discussed": "I am not aware of any essential references that are missing.", "other_strengths_and_weaknesses": "Strength:\n- The approach is not dependent on the specific architecture but could be applied to other pre-trained time series architectures.\n- Approach reaches fine-tuning performance\n- Evaluation on a comprehensive benchmark\n\nWeakness:\n- Only the subset of zero-shot benchmark is utilized. While this is a good idea to preserve the \"zero-shot\" setting, it would be advisable to additionally report results for the full benchmark and report which datasets are non zero-shot as this would make comparison to other papers/models easier.\n- As the evaluation includes randomness of the in-context samples, the authors should provide the variation over the individual datasets of the benchmark similar to Figure 9 for the \"overall result\".\n- Context length increases linearly with an increasing amount of context samples, therefore, computation complexity for the transformer increases quadratically", "other_comments_or_suggestions": "On Page 2 / line 259 the authors refer to A.2. although the authors probably want to refer to A.5", "questions_for_authors": "- Why did you decide to call the benchmark OOD benchmark? Typically, OOD is more often referred to situations which one has to detect (OOD detection) as the models do not perform reliable on it. In my understanding, the idea of pre-trained models is that the models should generalize over time series data, including the evaluated zero-shot data. Hence, in most related literature, this evaluation is more often referred to as \"zero-shot\" benchmark.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0YnPoHXjb1", "reviewer": "Reviewer_U55B", "summary": "This paper proposes a novel in-context finetuning strategy for a specific Time Series Foundation Model (TSFM). By continual pretraining a TSFM (TimesFM in the paper) with in-context examples, the updated model is able to be prompted with related past time series examples at inference time, enhancing forecasting accuracy without requiring additional training pipelines. The main contribution is to adapt the idea of \"in-context learning\" in NLP domain to TSFM. Experimental results validate the effectiveness of the proposed approach across multiple forecasting tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further questions. If our response addresses your concerns, we hope you consider raising your score accordingly.\n\n> Section 6.3: Moirai\n\nThank you for clarifying this point. Our interpretation from the Moirai paper in Section 3.2.2 Task Distribution was that the training datasets were augmented by “constructing multivariate time series from sub-datasets with univariate time series, by randomly concatenating them”. Based on this, it seemed that Moirai could support past related examples as multivariate features. We do not want to misrepresent the capabilities of the Moirai model in our paper, so we will contact the Moirai authors for clarifications, and correct this as the reviewer suggests.\n\n\n> Section 6.4.2: TimesFM (base) \n\nWe apologize for the lack of clarity - we indeed pretrained TimesFM-LH with a longer history length of 2048 (in a manner similar to the latest version of the TimesFM repo (https://huggingface.co/google/timesfm-2.0-500m-pytorch). We didn't just use TimesFM (base) directly with a 2K context length at inference time to get the TimesFM-LH results. We will make this more clear in Section 6.4.2 \n \n> ICF appears to be highly inefficient in terms of inference time [...]\n\nSince the total context length for TimesFM-ICF is 50 times larger than TimesFM, the 50 times inference speed slowdown is not unreasonable. However, TimesFM-ICF is still a zero-shot model, the addition continual pretraining phase for training TimesFM-ICF is a one-time cost that is independent of the target dataset – there is no cost to be paid to adapt the model for a new domain or dataset (unlike fine-tuning costs, which has to be paid to adapt TimesFM for every new dataset). More importantly, for practitioners, the ability to use the ICF model out of the box, and avoid having to build a fine-tuning pipeline to customize the model for their use-case is a very significant cost and resource advantage that we think can more than offset the 50x inference speed disadvantage (note that even after this 50x inference slowdown, TimesFM ICF can still perform inference for a single example in 43 ms on average - totalling 25 minutes on approximately 140k testing time series on TPUv5e with 4 cores -  which is often sufficient for most practical forecasting use-cases)\n> Suggestions [Sec 6.2]\n\nThank you for the suggestions. We will update the final version of our paper accordingly.\n\n> other TSFMs beyond TimesFM?\n\nWe believe that our methodology is generalizable to other TSFMs. This is an interesting direction for future research.\n\n> maximum history length limited to 512? \n\nThe maximum history length constraint in TimesFM comes because of the pretraining configuration - the model has been pretrained on examples up to the maximum history length and might not generalize well to lengths beyond what it has been pretrained on. Increasing the maximum history length during pretraining will result in longer training times and compute resources\n\n> maximum of 50 examples in context?\n\nThe choice of 50 examples for TimesFM-ICF comes from its continual pretraining setup where the model is trained with up to 25K (512*50) context window. Increasing the context window in continual pretraining would have caused longer training time and compute resources, which we would not have been able to afford. At inference time the model may generalize to beyond 50 examples - there is no mechanism within the model casting a limit of 50 indeed. For simplicity and conciseness of the paper, we also use 50 examples in our empirical study for consistency with the continual pretraining setup. Verifying the generalization is a separate, empirically heavy task that we plan to study in the future. \n\n> I believe Moirai is trained to accommodate a maximum of 128 variables, not 50\n\nSorry for the wording issue. We did not intend to imply that it accommodates at most 50, just that it accommodates 50. We will clarify our wording in the final version.\n\n> What is the fundamental difference between in-context fine-tuning and using exogenous variables?\n\nWe view in-context fine-tuning and exogenous variables as two separate concepts: The former supplements the main forecasting history with time series generated from the same or a similar distribution, in which sense it is in-context “fine-tuning” to fit this distribution. On the other hand, exogenous variables can be of any distribution, and their usefulness mainly comes from their correlation with the main forecast task. In this regard, in-context fine-tuning is a more strict practice.\n\n> Why the length of in-context examples can be different?\n\nAt both training and inference time we apply paddings to short in-context examples to bring them to length L+h while at the same time apply proper attention masks. Please see Section 5.1., Context Generation.", "claims_and_evidence": "Most of the claims are reasonable. However, some may not be clearly supported—please refer to *Experimental Designs or Analyses* and *Other Strengths and Weaknesses* sections for details.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "Nothing to discuss as no proofs for theoretical claims are included in the paper.", "experimental_designs_or_analyses": "Most experimental designs and analyses make sense, except the following ones. \n\n* *Section 6.3*: The design of using Moirai for in-context examples appears problematic. The key issue with this experiment is that **past related examples cannot be used as multivariate features in Moirai**. In Moirai’s framework, multivariate features correspond to **different variables within the same example**, sharing the same temporal range and aligned time ID. However, treating past related examples as additional multivariate features introduces a mismatch, as they belong to entirely different temporal periods. This misalignment causes Moirai’s time ID to convey misleading information, leading to the extraction of meaningless temporal relationships via attention. Consequently, this likely explains why Moirai-MV’s performance degrades when additional multivariate features are introduced in your experiments. In summary, the comparison with Moirai-MV in this manner is confusing and potentially misleading.\n\n* *Section 6.4.2*:  TimesFM (base) has a maximum history length of 512 and relies on positional encoding. Thus, I don't think directly inferencing with a longer history ($L=2048$) would significantly improve performance without additional training. Notably, TimesFM-ICL undergoes a continual pretraining phase. A fair comparison would involve continually pretraining TimesFM (base) with a longer history length ($L=2048$) before evaluating its performance against TimesFM-ICL. In my view, the current evaluation may potentially undervalue the impact of a longer history in TimesFM.", "relation_to_broader_scientific_literature": "The key contribution of this work relates to in-context learning or few-shot learning in the LLM domain. It aims to enable TSFM to leverage a few prompted examples at inference time, enhancing forecasting performance without additional training. This work represents a pioneering effort in exploring this direction.", "essential_references_not_discussed": "Related works are properly cited and discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The exploration of in-context learning for TSFM is pioneering and important.\n2. The paper is generally well-written and clearly structured. \n3. Extensive experiments are conducted, providing a comprehensive analysis of the proposed method.\n\n\nWeakness:\n1. In-context fine-tuning appears to be highly inefficient in terms of inference time. According to Table 7, the total inference time of TimesFM-ICF is 25 minutes—almost 50 times slower than TimesFM without ICF. The authors argue that fine-tuning is inefficient by taking training time into account; however, there are two key issues with this claim:\n    * In-context fine-tuning also requires an additional continual pretraining phase, yet the authors do not account for this computational cost.\n    * When compared to TimesFM (zero-shot), whose total inference time should be similar to TimesFM-FT, TimesFM-ICF offers no efficiency advantage at all.\n\n    This raises a critical question: is the forecasting improvement provided by in-context fine-tuning worth such a significant trade-off in inference speed?", "other_comments_or_suggestions": "* Suggestions: The paragraph **In-Context Example Selection** in Section 6.2 should be moved to an earlier section if it is a fundamental component of the framework. Placing it within the experimental results section appears confusing and incoherent:  it is unclear whether this in-context example selection strategy is applied during both continual pretraining and testing or only during inference in this specific experiment.\n* Typos: In the title of Figure 1, it should be _left/right_ instead of _top/bottom_.", "questions_for_authors": "1. Is the proposed in-context fine-tuning method generalizable to other TSFMs beyond TimesFM?\n2. Why is the maximum history length in TimesFM limited to 512? What constraints impose this limit—positional encoding, architectural consideration, or just pretraining configurations?\n3. Why can TimesFM-ICF handle only a maximum of 50 examples in its context? Is this limitation due to the continual pretraining process being designed specifically for 50 prompted examples, or are there other underlying constraints?\n4. I believe Moirai is trained to accommodate a maximum of 128 variables, not 50 as stated in Line 353. Could you clarify this discrepancy?\n5. What is the fundamental difference between in-context fine-tuning and using exogenous variables? Can the ICF method be interpreted as treating past examples as exogenous variables?\n6. Why the length of in-context examples can be different? Aren't they in the identical length of $L+h$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rIVF5jPwKP", "reviewer": "Reviewer_pyrU", "summary": "The paper extends a foundational forecasting models so that it can be conditioned on addition time-series information. Using in-context learning, the values of other time-series and also the value of the time-series to be predicted is added to the model input. Then the model is trained with the initial objective with a modified architecture to handle this additional information. Experiments are conducted on real-world datasets where the method is shown to reach a similar accuracy as fine-tuned models. Compared to those the trade-off is different since no time is spent on the fine-tuning but a larger time is spent at inference.\n\n## edit after rebuttal\n\nI raised my score as the authors addressed my main concerns. Using a validation procedure is I think much cleaner and the heatmap is an interesting addition to understand the benefit of condititioning on more in/out series examples.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns. If our response sufficiently addresses your concerns, we hope that you will consider updating your score accordingly.\n\n> Given that the cost of evaluating performance should be reasonable, it would be worth to explore a grid of options for the number of in-series and out-series example and plot a heatmap of the performance. For this, I believe it is a must to consider a set of datasets that are not used for the final selection (or at least to restrict the collections and be explicit about this) to not risk overfitting.\n\nThank you for this feedback. Based on your suggestion, we constructed a validation dataset from the training portion of a subset of the Monash datasets (specifically: weather, traffic, australian electricity, ercot, ETTm, and ETTh). We chose these datasets because they contained many training examples long enough to construct up to 20 in-series examples. We measured the validation MASE error of TimesFM-ICF with the number of in-series examples varying from 0-20, and the total number of in-context examples (including randomly selected examples) varying from 1-50. The resulting heatmap is attached in Figure 1 in this link: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. The configuration with smallest validation MASE was 11 in-series examples and 34 total examples. The geometric mean MASE ratio (averaged over 5 runs with different random examples selected) was 0.780+/.003 (so within a standard error of the MASE value we report in Figure 5).\n\n> Given that your results depend on a random sampling, analysing the noise coming from this decision would be important, it could be done with the analysis I discussed above by reporting confidence interval on mean performance estimates. In addition, reporting the confidence interval on the aggregated results on Fig 5 and others across datasets would be good to convey the uncertainty on the scores.\n\nPlease see updated OOD tables in https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf which now include confidence intervals for TimesFM-ICF. Each entry in the tables is an average over 5 runs of random in-context example selection (using our original in-series configuration). Note that our Figures already include confidence intervals. \n\n> instead of a separator, is there a reason why you did not consider a simple positional encoding? (eg a categorical embedding mapping the time-series index concatenated to your input, of-course using a fix index for the input time-series)\n\nUsing a time series level positional encoding is definitely a valid option. One advantage of our proposal is that the pretrained model weights could potentially generalize to more than 50 in-context examples thanks to our current choice of not using any form of positional encoding. This is an open question we will empirically explore in the future.\n\n> why are you making a causal mask in your architecture for the conditioning out-series? I would have imagined a mask that allows only the current in-series to look at other out-series but not to let an out-series looks at previous ones as it is less relevant for a out-series to look at the previous ones (also reinforce the random choice of the time-series order when sampling)\n\nAs we train the model in decoder-only manner, during training time there is no notion of a single “in-series”: for the forward pass on a training context (see Section 5.1. Context Generation), every series (example) is its own “in-series” and all its preceding series are its “out-series”, therefore we have to apply a full context level causal attention so that decoder-only works. At inference time we hence choose to keep the attention between out-series, which is for consistency with model training. This is identical to how language models commonly handle in-context examples (few-shot prompts).\n\nIt is a good point that our design breaks the symmetry among out-series. Empirically we tried to minimize its effect by training with randomizing the order of in-context examples when there is no causal leakage (see Section 5.1. Context Generation, grouping, “Dataset level:”). It’s also an interesting question that what happens if we remove this attention between out-series at inference time - we will do it as a future study. \n\n\n> Fig 9: how are you computed the confidence interval? have you considered ensembling the predictions obtained when sampling different history?\n\nIn Fig 9, the uncertainty of the reported metric comes from the random selection of in-context examples, and the confidence interval is computed based on 10 runs of the same setup with different random seeds. \n\nIt’s definitely possible to ensemble the predictions based on repeated sampling of different in-context examples. The practitioner can make this choice at the cost of increased latency.", "claims_and_evidence": "Yes, the method performance is validated on out-of-sample datasets. The accuracy is shown to match fine-tuned foundational models.", "methods_and_evaluation_criteria": "Yes, a large collection of real-world datasets is considered, standard metrics are reported, and several are used to account for the randomness of the method.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, the author mostly reused benchmarks that have been peer-reviewed and are standard. They also clearly indicates which method performance are in-sample (eg trained on the datasets) and which are not.", "relation_to_broader_scientific_literature": "Related work is well cited.", "essential_references_not_discussed": "Your work reminded me of Tactis (https://proceedings.mlr.press/v162/drouin22a/drouin22a.pdf), in particular feeding all the time-series dimensions as token to a transformer, it may be worth including it in your related work section.", "other_strengths_and_weaknesses": "Strength:\n- impactful application\n- sound method that could be applied to other foundational models\n\nWeakness:\n- some ablation may be missing\n- lack of analysis in the variance and behavior on the conditioning of additional data\n- the method makes a trade-off between fine-tuning time and inference time, therefore it is mostly applicable when fine-tuning time is problematic which reduces the scope\n\n\n-- edit post rebuttal\nThe authors performed ablations and now use a clear validation protocol.", "other_comments_or_suggestions": "* l326 Exponential Smoothing (ETS) (missing space)\n* toursim", "questions_for_authors": "**Analysis and model selection on the Conditioning of extra data.** My biggest complaint on the paper is that there are very few experiments on the conditionning on random data, except for Fig 6. (and Fig 9.) in the appendix and that the process done to select the hyperparameters is not ideal.\n\nThe paper is making key design decisions such as selecting some number of in-series and out-series example which appears to be based on the OOD dataset but since you are performing model selection, you should rather use another set of validation datasets to make this selection as you would be overfitting your model selection otherwise.\n\nIn addition, I found that the analyses on this aspect lacking, you are reporting the performance on only 4 setup (except in Fig 6 where the number of in-series is varied). Given that the cost of evaluating performance should be reasonable, it would be worth to explore a grid of options for the number of in-series and out-series example and plot a heatmap of the performance. For this, I believe it is a must to consider a set of datasets that are not used for the final selection (or at least to restrict the collections and be explicit about this) to not risk overfitting.\n\n**No variance reported.** Given that your results depend on a random sampling, analysing the noise coming from this decision would be important, it could be done with the analysis I discussed above by reporting confidence interval on mean performance estimates. In addition, reporting the confidence interval on the aggregated results on Fig 5 and others across datasets would be good to convey the uncertainty on the scores.\n\nThose are my biggest concern on the paper, if those points are adressed, I would be keen to revise my score. The other points bellow are not as crucial.\n\n**Additional points**.\n\n- instead of a separator, is there a reason why you did not consider a simple positional encoding? (eg a categorical embedding mapping the time-series index concatenated to your input, of-course using a fix index for the input time-series)\n- why are you making a causal mask in your architecture for the conditioning out-series? I would have imagined a mask that allows only the current in-series to look at other out-series but not to let an out-series looks at previous ones as it is less relevant for a out-series to look at the previous ones (also reinforce the random choice of the time-series order when sampling)\n- Fig 9: how are you computed the confidence interval? have you considered ensembling the predictions obtained when sampling different history?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a methodology for in-context fine-tuning of a time-series foundation model, allowing the model to exploit multiple support series at inference without further training. Tested on multiple public datasets, it yields consistent zero‑shot improvements. The authors also demonstrate the performance of the methodology through various ablation studies on context size and selection, plus extra validation provided during the rebuttal stage. Overall, all reviewers agree on the idea’s novelty, simplicity, and generality. Therefore, I recommend acceptance of this paper.
Decision: Accept (poster)