AT4TS : Autotune for Time Series Foundation Models
Authors: Shivani Tomar
Seshu Tirupathi
Radu Marinescu
Elizabeth M. Daly
Ivana Dusparic
Venue: Accepted by TMLR ()

Abstract: Foundation models have been successfully adapted to the task of time series forecasting due to their ability to capture long-range dependencies, as demonstrated in the field of Natural Language Processing (NLP). However, effectiveness of applying these pre-trained time series foundation models (TSFMs) in the target domain is limited due to the need for hyperparameter optimization to match the characteristics of the target domain. To address this limitation, we propose a novel algorithm AT4TS: Autotune for Time Series Foundation Models that aims to efficiently automate the process of selective fine-tuning of pre-trained TSFMs for a given target domain. Our approach helps remove the tedious task of accurately configuring the tunable hyperparameters required to selectively update parameters to enhance predictive performance on unseen out-of-domain target datasets. AT4TS has been validated through diverse pre-trained models like Chronos and Tiny Time Mixers (TTM), fine-tuning strategies like Low Rank Adaptation (LoRA) and custom fine-tuning and state-of-the-art hyperparameter optimization (HPO) methods. Extensive experimental results on real-world benchmark datasets demonstrate that AT4TS efficiently identifies the optimal configuration of tunable hyperparameters for autotuning TSFMs. We show improvements as high as 20.55% and 45.34% for one of the out-of-domain datasets compared to zero-shot pre-trained models for Chronos and TTM respectively.

Published in Transactions on Machine Learning Research (09/2025)
AT4TS: Autotune for Time Series Foundation Models
Shivani Tomar
shivani.tomar Wibm.
Trinity College Dublin IBM Research, Ireland
Seshu Tirupathi IBM Research, Ireland
seshutir ibm.iiiieemmmmmmmmmmmmmmm. com
Radu Marinescu IBM Research, Ireland
mudummarinesccusie..mm
Elizabeth M. Daly 1BM Reseorch, Ireland
elizabeth. brrabett.dllllee daly www.imm.e...m.mmm com
Ivana Dusparic Trinity College Dublin
duspori@tod.ie
Reviewed on OpenReview: https: // openreview. net/ forum? id- U64 vyLnaMX
Abstract
Foundation models have been successfully adapted the task of time series forecasting due their ability to capture long-range dependencies, demonstrated in the field of Natural Language Processing (NIP)) However, effectiveness of applying these pre-trained time series foundation models (TSFMss in the target domain is limited due to the need for hyperparameter optimization to match the characteristics of the target domain. To address this limitation, we propose novel algorithm AT4TS: Autotune for Time Series Foundation Models that aims to efficiently automate the process selective fino-tuning of pre-trained TSFMs for a given larget domain. Our approach helps remove the ledious task of accu- rately configuring the tunable hyperparameters required to selectively update parameters to unscen out-of-domain target datasets. AT4TS has been
enhance predictive performance validated through diverse pre-trained models like Chronos and Tiny Time Mixers (TTM), fine-tuning strategies like Low Rank Adaptation (ToRA) and custom fine-tuning and state- of-the-art hyperparameter optimization (HPO) methods. Extensive experimental results on real-world benchmark datasets demonstrate that AT4TS efficiently identifies the optimal configuration of tunable hyperparameters for autoluning TSFMs. We show improvements as high as 20.55% and 45.34% for one of the out-of-domain datasets compared to zero-shot pre-trained models for Chronos and TTM respectively.
1 Introduction
Time series forecasting has always been critical for decision-making across various domains including retail, smart grids, healtheare, finance, weather, traffic control among others (Peterson) 2017) (Hernandez et al, 2014). The remarkable success of LLMs in broad domains like Computer Vision (CV) and Natural Language
Processing (NLP) (Vaswani et al. 2017) (Wen et al. 2022 has prompted rescarchers to adapt these models to the task of time series forecasting [Wu el al.] 2022 Garza & Mergent CarrgenthalerCCCae halethhCrer--eeeo 2023 Gruver et. al.,
2024) given their ability to capture long range dependencies present in time series data. These models are pre-trained on vast amounts of data spanning multitude of domains leveraging the general purpose representations learnt the process. However, excel on the domain specific downstream task it becomes important to fine-tune these models datasets from the target domain (Wen et al. 2022 Such adaptation

---

Published in Transactions on Machine Learning Research (09/2025)
is generally achieved via fine-tuning, which involves updating all the parameters of the pre-trained model (Jin et al. 2023 Bommasani et al. 2021 Lv et al. 2023 Given the large number of parameters that are originally trained, full fine-tuning becomes an operational challenge if we wish to adapt these models to the target domain.
Recent studies have shown that the computational burden and inefficiencies associated with full fine-tuning can be mitigated by adapting only small percentage of the parameters in addition to the pre-trained model for each task which greatly enhances the operational efficiency of these models (Hu et al. 2021). This helps in learning the specific characteristics of the target-domain while also retaining the broad understanding of general time series data acquired during the pre-training phase. These techniques are termed as Parameter- Efficient TaneeTuning (PEFT). One of the popular PEFT techniques, LoRA (Low Rank Adaplation) (Hu) et al.) 2021) trains only selective dense layers in a neural network while keeping the pre-trained weights frozen. This approach has been widely applied to various domains such as medical imaging, video text gencration and speech synthesis (Balne al. 2024) to name a few. However, its application time series is under explored.
AutoML, which involves automating the process of composing and parameterizing ML algorithms to maxi- mize a specific metric such as model accuracy on given dataset has been widely used to improve the accuracy of traditional machine learning and deep learning models (He et al. 2021 Leveraging this knowledge, auto- tuning of pre-trained foundation models for a given target domain can potentially lead to improvement in the accuracy of these models compared to traditional fine-tuning with fixed hyperparameters of the fue-tuning algorithms.
To this end, propose novel algorithm called AT4TS Autotune for Time Series Foundation Models, which achieves efficient fine-tuning of pre-trained TSFMs through the integration of parameter-effiient fine- tuning techniques along with hyperparameter optimization of the fine-tuners to improve the performance of time series foundation models in the target domain. We illustrate our algorithm using two different categories of TSFMs: Chronos, a Transformer-based architecture and TTM, a lightweight non Transformer- based model. In particular, to achieve an efficient implementation of autotuning, we adopt the classical Limited Discrepancy Search (LDS) algorithm introduced by (Harvey & Ginsberg) 1995) to optimize the hyperparameter sclection process. This algorithm is essentially depth-first scarch strategy that identifies new set of solutions by ileralively increasing the number of discrepancy values where the discrepancy refers to the number of variables in the current configuration that differ from their values in the initial configuration. We further establish the robustness of AT4TS by using it with other hyperparameter optimization meth namely, Hyperopt (Bergstra et al.) 2015 and Bayesian Optimization and Hyperband (BOHB) (Falkner et al. 2018). The novel contributions of this paper include:
Automated fine-tuning exploring tunable hyperparameter configurations for fine-tuners of pre- trained time series foundation models to find the optimal hyperparameters for the target domain. To the best of our knowledge, this is the first paper to explore the potential of autotuning Ms.
Generalized applicability of AT4TS algorithm across disparate pro-trained models (transformer based and non-transformer based), fine-tuning strategies (LoRA and custom fine-tuning) and HPO meth- ods (LDS, Hyperopt and BOHB).
Extensive tests across a suite of out-of-domain real-world benchmark datasets to compare the per- formance of autotuned TSFMs, traditional fine-tuning strategies, and zero-shot pre-trained models.
2
Related Work
With the recent success of foundation models in the domains of NLP and CV, there has been growing rescarch for leveraging their polential for time series analysis particularly for long-term forecasting, To this end, TSFMs aim to accomplish the zero-shot generalization capabilities across a diverse spectrum of downstream tasks, thereby, minimizing the need for explicit task-specific model development. Furthermore, TSFMs hold the promise of attaining superior performance through domain specific fino-tuning catering to a wide range of applications. Authors in (Liang al. 2024), propose comprehensive taxonomy offering a thorough
2

---

Published in Transactions on Machine Learning Research (09/2025)
understanding of the rapidly evolving landscape of TSFMs. Based their methodological categorization, most of the popular TSFMs are: Transformer-basedl non Transformer-based and diffusion-based models. We briefly discuss the first two categories of TSFMs in this section as we illustrate our approach using one model from each category.
Transformer-based TSFMs: In the first category, authors in Zhou et al. 2021 Ww et al. 2021
(Zhou et al., 2022] employ transformer-based architectures to capture long-terin dependencies in time series data. They further handle the challenges of quadratic time complexity and high memory usage through self-attention and auto-correlation mechanisms. PatchTST (Nie et al. 2022) on the other hand, introduced two key components segmentation of time series into patches serving as input tokens to the transformer along with channel-independence overcoming the challenge of high memory usage of attention maps. Yet another pioneering work in this area is Lag-Llama (Rasul et al. 2023) inspired by the LLaMA (Touvron et al. 2023) LLM which utilizes a simple decoder-only transformer architecture for time series forecasting by using lagged features as covariates. Another line of work, LLMTime (Gruver et al., 2024), uses time series as strings with careful pre-processing specific to the given LLMs' tokenizer. Likewise, Chronos (Ansari et al., 2024) employs the encoder-decoder transformer architecture from the T5 (Rafiel et al. 2020 LLM family requiring minimal modifications i.e, tokenization through scaling and quantization.
Non Transformer-based TSFMs: Non- Transfooforr based models use diverse architectures to efficiently process temporal patterns. One groundbreaking contribution in this category is a relatively smaller model (1-5 million parameters) called Tiny Time Mixers (TTM) (Ekambaram et al. 2024). Build upon TSMixer architecture, it uses light-weight MLPMixer blocks interleaved with simple gated attention outperforming many foundation models with significantly larger model parameter sizes.
Parameter Efficient Fine Tuning (PEFT): In order to make TSFMs more accessible and adaptable to lacilitate their wider application to real-world downstream lasks, they can be directly deployed to leverage the general temporal knowledge acquired in the pre-training phase. However, in operational deployments, domain specific target data is used to fine-tune these models to further enhance their zero-shot performance. Most commonly, PEFT techniques have been proposed in NLP and CV for fine-tuning subset of parameters in various downstream tasks. One of them is LoRA (Hu et al. 2021 which adds trainable low-rank matrices into transformer layers to approximate the weight updates while keeping the existing weights frozen. The application of LoRA in the domain of fine-tuning TSFMs remains largely unexplored due to the rapidly evolving nature of this field and hence, motivates this work. The authors in (Gupta et al., 2024) investigate the impact of LoRA basod fino-tuning across popular TSFMs which relates to our work, however, they do not allempt autotuning TSFMs using different parameter efficient fine-tuning techniques which makes our contribution novel.
Hyperparameter Optimization (HPO): Hyperparameter optimization (HPO) can potentially act an important component in searching for optimal hyperparametens for selectively fine-tuning the pre-trained models. Bayesian optimization (BO) has been the state-of-tho-art method for optimizing hyperparamoters for neural networks [Eggensperger el al., 2013) (Feurer et al.) 2015). It relies on probabilistic modeling of the objective function given the observed data points. Other methods include Hyperband (Li et al., 2022) which is a bandit based method that adaptively allocates different budgets b and utilizing successive halving to find the best out of a set of randomly sampled configurations. BOHB leverages the benefits of both BO and HB to achieve faster convergence to the most optimal configurations. Hyperopt is based on bayesian optimization that uses hyperparameter optimization for ML algorithms.
(TPE) (Bergstra et al. 2011) algorithm for carrying out
Autotuning methods in other domains: Autotuning has not been explored in the context of time series forecasting using TSFMs. However, the importance of obtaining the best hyperparameters for fine- tuning has been studied extensively across other domains. The authors in (Liu & Wang) 2021) discuss the impact of automated HPO methods on finctuning pro-trained language models. They systematically compare automated HPO methods with grid search and analyse the failure scenarios in HPO for fine-tuning. Another work (Triess et al.] 2023) explores HPO using blackbox optimization algorithms such NOMAD and
3

---

Published in Transactions on Machine Learning Research (09/2025)
hyperparameter contig space
oop RAY
Founcation (Pre-Trinned on Source Domair)
'dropout': du "Learning_rate" 'batch_size": :: bu
'drapout': d., reaminn_ratee' Th satch_size" by
generates
Hyperparamater config
Hyperperameter config 2
Hyperperameter config 3
Hyperparameter contigg
Auto- Lan-Tuaed model
Training
Time dataset (Torget Domain)
Valicittion
Trail
Tral2
Triak3
Triel
Evaluation
Ranking error metric
Test de
Triel
Trial
Test performance
Figure 1: Architecture diagram showing the workflow of our algorithm, AT4TS using Hyperparameter Optimization and Ray Tune for parallelization of the tune trials.
demonstrate improved performance of fine-tuned LLMs downstream tasks. (Li et al. 2020 suggest best practices to set hyperparameters for fine-tuning of models in image domain tasks. They further demonstrate that key hyperparameters such as effective learning rate, momentum and weight decay are dependent on the similarity between the source and target domains.
3 Methodology
In this sontion, we first define the problem stalement and then present a detailed description of the AT4TS design approach we adopted using selective fine-tuning: LoRA and custom fine-tuning coupled with LDS the search algorithm.
Problem Statement Given a multivariate time series E Rxxx of length n, number of iitannels/variables and context length al where 1 < < n, the forecasting task is to predict future values Y Rd where d' denotes the number of channels being forecasted and h is the forecast. horizon.
To solve the forecasting task described above while also demonstrating the contribution of this work through their application, we employ two TSFMs: Chronos (Ansari et al. 2024). transformer-basee model which handles univariate datasets and TTM (Fkambaram et al. 2024). a non-transformer- -based model built for mutivariate data setting. These models have been pre-trained on a large collection publicly available time series datasets from varied domains. However, since these models are trained on broad spectrum time series data, their performance on specific task which in our case the unseen target cltasett may not always be optimal.
We implement AT4TS using Chronos with PEFT. LoRA Hu ct al. 2021 preva- lent PEFT technique helps in integrating domain specific knowledge the pre-trained model by ine-tuning minimal number of weights. This is achieved in storage and compute-efficien manner by constraining the
Chronos with PEFT:
4

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1 AT4TS algorithm using selective fine-tuning and LDS
Input: Pre-trained Model M, target dataset X split into Xtrains Xval and Xtest, search space
(Yi,..., Vx), SEARCH with maximum discrepancy 5
Output: Optimally luned hyperparametens Yopt and Autotuned Model My
1: Define hyperparameter search space Y= (Y...... Yn)
2: Execute SEARCH: Initialize yo to default hyperparameters and let y + yo
3: for all f = =1.,... 1,...,, do SEARCH(y", Y, 1)
end for return y"; MM*
procedure SEARCH(y, Y, 8, i)
8:
if 0 = 0 or i >> Y then Yo My + SCORE(y, Xtrin: Xyal, M)
9:
10:
return Yopt: My.pc
11: elsc
12:
for all values y D(Y do y then if yl1] t SEARCH(y, SEARCCHy.Y,++110) 1,,+
13:
14:
15:
else
16:
y' y: y(1+ + 2 = SEARCH(' 1, end if return
17:
18:
19:
end for 21: end if
20:
22: end procedure
23: procedure SCORE(y, Xarain X M)
24: M + TrainModelly". Xtrain)
25: score +- EvaluateModdel(M M, x
26: if score best score then
27:
Myopa M
28: Yop y
29: best score +- score
30: end if
31: return Yopt, My
32:
end procedure
update (AW) to the pre-trained weight matrix wo E Bdik by representing with a low-ranke decomposition,
Wo AW Wo BA
where E Rdxr, A E Rrxk and the rank r S min of A and are updated. We adapt the weight matrices corresponding to the self-attention module and the leed-forward layer modules of the transformer architecture.
k). During fine-tuning, Wo is frozen, while the weights
TTM with custom fino-tuning: In the case of TTM, we adopt the fine-tuning technique as described in (Ekambaram el al. (0224) wherein the backbone of the model is frozen while only updating the weights of the TTM forecast head. This is because of the following three reasons: (1) this is the fine-tuning strategy used by TTM developers and therefore, makes it credible comparison; (2) the size of these models much smaller and so LoRA is not efficient as shown by preliminary experiments we conducted; and (3) we want to demonstrate that AT4TS can be applied across different. fine-tuning strategies.

---

Published in Transactions on Machine Learning Research (09/2025)
AT4TS using parameter efficient fine-tuning and LDS: We design a novel algorithm to perform automated fine-tuning of time series foundation models described above using LoRA and custom fine-tuning methods in conjunction with LDS. This fine-tuning is achieved by using a distributed Ray-based framework Figure shows the architecture diagram of our approach wherein we take a pre-trained model and specify a minimal subset of weights to be trained to adapt. the model on the target dataset. The hyperporameeer config space block corresponds to the hyperparameter search space which is defined as the domain of values that are explored and evaluated during the process of hyperparameter tuning. For instance, LoRA hyperparameters include learning rate, batch size, rank, scaling factor etc as shown in Table 2 The hyperparameter config blocks to n correspond to trials which are executed concurrently in distributed cluster. Fach trial represents a model fine-tuned and evaluated on the train and validation split of the target dataset respectively. The most promising hyperparameter configuration is obtained by ranking the trials based on the chosen error or performance metric on the validation split. In the end, the fine-tur tuned model corresponding to the best- found hyperparameter configuration (also called the autotuned model) is evaluated on the test split of the larget dataset.
Aggorithmooutlinn the steps involved in AT4TS. The algorithm starts with initialising the hyperparameter search space. We use Limited Discrepancy Search or LDS (Harvey & Ginsberg, 1995) to traverse this space effectively starting from an initial configuration of default hyperparameters It should be noted that AT4TS provides the flexibility of using any other search algorithm and is not limited by LDS. Specifically, LDS takes as input a vector of variables Y = [X,,....,Y.) Y,) corresponding to the hyperparameters together with their domains of values D (D(Y), .... D(Y,)) representing the hyperparameter search space to be explored and the maximum discrepancy value 8 which limits the number of allowed variable-value assignment changes from the initial solution yd = (yf,.... ..., -.. vn) where vf E D(Y)) is a value in variable's Y, domain and outputs the next solution y" based on the discrepancy value. Notice that LDS requires the variables to have finite domains of values, and therefore any continuous hyperparameter needs to be discretized. In addition, LDS is required to look for a reasonably good solution which is typically given by the default hyperparameter configuration here, hence, we set y" to the default LoRA configuration for Chronos and default fine-tuning parameters in case of TTM. We begin with a discrepancy value 0 of 1 and conduct an iterative search that allows to change the values of at most 0 hyperparameters in the initial solution yo. We then increment until 0 exceeds the maximum discrepancy value. Function SEARCH defined in line 7 performs the actual exploration of the hyperparameter search space limited by discrepancy 0. This assists in incrementally searching around the default configuration compared to random exploration. For each configuration y' returned by LDS, we fine-tune the model in line 24 and evaluate it on the validation split to compute the error metric as outlined in the SCORE function in the algorithm. At the end, we get the best configuration y* corresponding to the lowest error and the autotuned model M line 6 which is then evaluated on the held out test split.
For illustration, Figure 2 shows the search space explored by LDS with discrepancy value 1 (denoted by LDS(1)) for 3 dummy variables [A, B, CJ with domain values {a1, a2), b2) and (21, respectively. In this case, LDD((1) starts from the initial assignment of (01,b,,) } which corresponds to the leftmost blue Icaf node and traverses the search space in depth-first manner visiting only the blue Icaf nodes.
The algorithm described above is then implemented in a distributed manner using Ray Tune (Liaw et al., 2018) which provides an open source framework for distributed model training and selection. Each configu- ration returned by LDS corresponds to the trials exccutod concurrently in a cluster. We use the default Ray tune scheduler which is first-in-firstoout (FIFO) passing through the trial configurations without performing any early stopping.
4 Experimental Setup
In this section, we present the datasets used in the fine-tuning experiments along with the implementation details of the proposed AT4TS framework.
Datasets For our experiments, we use 10 univariate datasets from the Monash Time Series Forecasting Repository (Godahewa et al. 2021 for autotuning Chronos models. These datasets are part of the Bench-
Inttps:/ /ww.ray.oo////
6

---

Published in Transactions on Machine Learning Research (09/2025)
A
a2
a1
B
b
22
b1
b2
C
1
C2
C1
C2
C1
C2
C1
C2
Figure 2: Example of the search space traversed by LDS with maximum discrepancy value
Table 1: Univariate Dalasets used for the Chronos experiments.
Num. Series
Series Length
Prediction Length (H)
Dataset
Domain
Freq.
Traffic Weather
Transport
1H
862
17544
24
Nature
ID
3010
14296*
30
ETT (Hourly)
Energy
1H
14
17420
24
ERCOT Load
Energy
IH
8
154854
24
Australian
Energy
30min
5
231052*
48
Electricity
Exchange Rate
Finance
1B
8
7588
30
FRED-MD
Economics
1M
107
728
12
NN5 (Daily)
Finance
1D
111
791
56
M5 ETT (15 min.)
Retail
1D
30490
1562*
28
Energy
15min
14
69680
24
mark datasets in (Anari et al. 2024h used for zero-shot evaluation. Table 1 provides the details of the datasets used for the experiments. Each dataset is a collection of series where each series split into train, validation and test instances. The * in the series length represents average length of the time series for 3 datascts, Weather, Australian Electricity and M5 which consists of variable length time series. The number of instances in validation and test label depend on the prediction horizon which specific for each dataset. We use the GluonTS library to split the datasets using the same strategy used in (Ansari et al., 2024). For each datasot, the train split includes all the data points from the beginning of the time scries up until the last two prediction horizon lengths, which are held out for validation and testing respectively. We use these datasets as they have not been used in the pre-training phase of the Chronos T5 models. Similarly, Table lists the datasets used for TTM in our experiments. All the datasets except MORE were used in Ekambaram
2024) for zero-shot shot evaluation with no overlap with the pre-training datasets. These multivariate benchmark datasets which do not contain any exogenous variables. To further vali- date the efficacy of our approach in unseen target domain,
real-world wind energy dataset called M data containing exogenous channels. This dataset uses wind park data for 18 months from 11 wind turbines (WT2 to WT11) in wind park. The dataset consists of SCADA data from the sensors on the wind turbines combined with weather data. In our experiments, we only utilize a sample of this dataset with 9500 points. Since TTM can capture cross-channel correlations and exogenous variables, MORE data is an ideal fit for the autotuning application scenario targeted by our approach.
httpe://ghhhbbccmm/MORRRFFU

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Chronos: LoRa Hyperparamater Search Space.
Parameter Name
Range of values
alpha
{4, 8, 16, 32, 64] {0.0, 0.05, 0.1) {2, 4, 16, 32}
dropout
rank
bias
("none', "all', "lora _olaa" {0.0001, 0.001, 0.013 {4. 8, 16) {1, 4, 8}
learning rate
batch_sst size
grad
accumulation steps
Table 3: Multivariate Datasets used for the TTM experiments.
Scries Length
Total Channels
Target variables
Exogenous
Dataset
Freq.
variables
MORE
1H 9500
13
1
12
ETTh1
IH 17420
ETTh2
1H 17420
7
7
N/A
ETTml 15min 69680
ETTm2 15min 69680
Weather 10min 52696
21
21
Electricity IH
26304
321
321
Models We used Chronos T5 models as the Transformer-based architecture for demonstrating our ap- proach, AT4TS. These models, trained from scratch on time series data, are widely adopted open-source transformers that have achieved state-of-the-ars (SOTA) performance time series analysis. Chronos T5 models have been pre-trained and released in sizes ranging from Tiny (16M), Mini (20M), Small (46M), Base (200M) and Large (710M) with number of model parameters brackets. In our experiments, we focus univariate time series forecasting as Chronos models use the lightweight version of the models i.e. Mini in order to utilize minimal computational resources for demonstrating the applicability of our approach. We also used TTM which is significantly smaller (1-5M) pre-trained model for effective zero/fene few-shot multivariate forecasting. TTM is composed of TSMixer archi- tecture, based on ML.P blocks and has 4 key components namely, TTM backbone, TTM decoder, Forecast head and the optional exogenous mixer known for capturing cross channel relationships and exogenous vari- ables. They have been released as variants: TTM-Base (1M), TTM-Enhanced (4M) and TTM-Advanced (5M) trained with context length, 512, 1024 and 1536 respectively. We specifically used the wight TTM-Base in our experiments. Our experiments validate that AT4TS is model agnostic and can be flexibly adapled to use both transformer and non-transformer based TSFMs.
pre-trained for the univariate setting. We
Table 4: TTM: Custom Fine- Fine-Tuning Hyperparamater Search Space.
Parameter Name Range of values
head dropout
(0.1, 0.4, 0.7]
batch_ size
{8, 16, 32, 64) (0.001, 0.0001}
learning rate
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Mean MASE scores obtained using Chronos T5 mini model in Zero Shot, Default LoRA Fine-Tuning and our approach, AT4TS with LDS.
Dataset
Zero Shot
Default LoRA Fine Tuning
AT4TS(LDS)
Traffic
0.853 (=0.0012)
0.776 (#0.0015)
0.747 (#0.0157)
Weather
0.839 (#0.0032)
0.848 (#0.0039)
0.822 (#0.0043)
ETT (Hourly)
0.795 (=0.0111)
0.830 (+0.0179)
0.797 (+0.0235)
ERCOT Load
0.582 (=0.0107)
0.566 (+0.0341)
0.566 (+0.0341)
Australian Electricity
0.965 (#0.0406)
1.151 (#0.0300)
0.832 (#0.0827)
Exchange Rate
2.054 (+0.1561)
1.871 (+0.0527)
1.632 (+-0. 1756)
FRED-MD
0.473 (=0.0105)
0.508 (+000000)
0.511 (+0.0092)
NN5 (Daily)
0.648 (#0.0059)
0.605 (+0.0012)
0.620 (+0.0138)
M5 ETT (15 min.)
0.942 (=0.0004)
0.926 (#0.0003)
0.925 (=0.0005)
0.709 (=0.0269)
0.712 (+0.0172)
0.727 (+0.0378)
Table 6: Mean MASE scores obtained with Chronos T5 mini model using different HPO techniques in AT4TS and average rank (lower is better).
AT4TS
Dataset
BOHB
HyperOpt
LDS
Traffic
0.768 (=0.0041)
0.779 (#0.0495)
0.747 (#0.0157)
Weather
0.818 (=0.0037)
0.824 (+0.0028)
0.822 (#0.0043)
ETT (Hourly)
0.798 (1.00231)
0.794 (#000036)
0.797 (+00.233)
ERCOT Load
0.617 (+0.0301)
0.602 (+0.0456)
0.566 (+0.0341)
Australian Electricity
0.738 (#0.0699)
0.829 (#0.0257)
0.832 (#0.0827)
Exchange Rate
2.262 (=0.0568)
2.143 (#0.5007)
1.632
(#011756)
FRED-MD
0.540 (+0.0560)
0.554 (+00.288))
0.511
(+0.0076)
NN5 (Daily)
0.601 (=0.0116)
0.597 (#0.0090)
0.620
(#0.0138)
M5 ETT (15 min.)
0.923 (+0.0005)
0.922
(10.0011)
0.925
(+000005)
0.694 (+092))
0.694 (#0.0121)
0.727
(#0.0378)
Avg. Rank
2.0
1.9
1.9
Implementation Details AT4TS has been implemented using Ray Tune (Liaw et al. 2018 and Trans- formers librarics. supports both Transformer and non-Trans non-Transformer TSFMs as well as parameter efficient fine-Luning (backed by the PEFT Library). The number and range of tunable hyperparameters are set basod on the fine-tuning strategy being used. Tables and show the search space for LoRA hyperparam- eters and custom fine-tuning hyperparameters used autotuning Chronos and TTM models respectively. We execute multiple trials selected using LDS for each dataset to find the best hyperparameter configuration and output the autotuned model corresponding to it. We limit the number of trials to 10 in case of Chronos and 15 for TTM to demonstrate the robustness of our approach in a resource-onntraiine environment. We used the maximum discrepancy value based on the number of hyperparameters to be tuned. In our case, we set this to for Chronos and for TTM. lower value of maximum discrepancy involves a more focused search while higher values allow more broader exploration of the potential hyperparamcter search space. To ensure comprehensive evaluation across different fine-tuning settings in the case of Chronos, we use mean absolute scaled error (MASE) as the evaluation metric. Since the model produces probabilistic forecasts, the forecasted value for each datapoint is caleulated the median (0.5-quantile) of 20 samples, which is then used to calculate the metrics similar to (Ansari et al.) 2024). The MASE scores are averaged across 5 runs. For TTM, we use mean squared error (MSE) as the standard error metric and report average MSE scores across five forecast lengths, FLs E (24, 48, 60, 96, 192} similar to [Eammbaran et al. 2024). The
http:s//hwugghggaeeco/////s/ssss https///muggggghhre.o/////ssss/
9

---

Published in Transactions on Machine Learning Research (09/2025)
hyperparameters for default fine-tuning of TTM for same values as used in (Fkambaram et al., 2024p. For MORE dataset, we set the default values as follows: learning rate=0.001, head dropout=0.4 and batch size=64. The experiments were performed on a multi-node cluster environment using a combination of CPUs and A100 GPUs. In order to compare the performance of LDS search algorithm with other slate-of-the-arl HPO methods, we run AT4TS by plugging in Hyperopt and BOHB as the search algorithm. The detailed analysis of the results obtained are discussed in the next section.
CeSTTTTmwwwaeereeeettteeee are configured to the
Table 7: Mean MSE scores obtained using TTM-Base model in Zero Shot, Default Fine- Tunn-Tuning and our approach, AT4TS.
Dataset
Zero Shot
Default Fine Tuning
AT4TS
MORE
0.247
(9919
0.213 (+0.1044)
0.135
(+0.0122)
ETTh1
0.348
(=0.0281)
0.348 (#0.0303)
0.343 (+0.0329)
ETTh2
0.245
(=0.0614)
0.246 (+0.0639)
0.241 (+0.0635)
ETTml
0.286
(=0.0534)
0.279 (+0.0513)
0.259 (+00.045)
ETTm2
0.152
(=0.0471)
0.150 (+0.0464)
0.144 (+0.0436)
Weather
0.133
(0.03533
0.133 (+0.0359)
0.130 (+0.0347)
Electricity
0.145
(=0.0305)
0.137 (#:00292)
0.120 (#0.0179)
Table 8: Mean MSE scores obtained with TTM using different HPO techniques in AT4TS and average rank (lower is better).
AT4TS
Dataset
BOHB
HyperOpt
LDS
MORE
0.136 (#0.0129)
0.137 (#0.0144)
0.135
(#0.0122)
ETTh1
0.343 (+0003299)
0.343 (+0.0334)
0.343
(+0.0329)
ETTh2
0.246
(+0.0681)
0.244 (=0.0663)
0.241
(+0.0635)
ETTml
0.260
(#000044)
0.259
(#0.0445)
0.259
(+0.0445)
ETTm2
0.144
(#0.0434)
0.144
(#0.0436)
0.144
(+..0436)
Weather
0.135
(+000365))
0.130
(+0.0348)
0.130
(+.03347)
Electricity
0.120
(+0.0179)
0.120
(#0.0178)
0.120
(+0.0179)
Avg. Rank
1.71
1.42
1.00
5 Results
We present comparative results of our approach for Chronos in Table which shows the performance of mini T5 Chronos model variant in zero-shot setting along with different fine-tuning settings. We report MASE averaged over 5 runs for both zero-shot and default fine-tuning setting. For AT4TS, we run 10 trials and report MASE corresponding to the best LoRA configuration, which is also averaged over 5 runs. We observe that the performance of our approach is better than zero-shot and default LoRA fine-tuning for out of datascts as highlighted in the last column of Table This can be attributed to the superior performance of LoRa when coupled with automated hyperparameter optimization.
We observe that, for target domain datasets such as Exchange Rate and Australian Electricity which do not share any similarity with the pro-training source datascts, our approach outperforms with significant margin. To support our argument, we measure dataset similarity between pre-training and target datasets using Maximum Mean Discrepancy (Wang et al.) 2021) along with Principal Component Analysis (PCA) important to note that dataset similarity not the only criterion that influences the zero-shot and fine-tuned performances of the pre- trained time series foundation models. Based on the findings in (Fkambaram et al. 2024) Section 4.7
(Abdi & Williams/ 2010) (Sce details in Appendix 7.1). However,
10

---

Published in Transactions on Machine Learning Research (09/2025)
Chronos: AT4TS Zero zero-Shot
Zero
25
goometric
4.3%%
geometric 13.52%
MASE improvement (N)
15
79%
12.4%%
MSE Improvement
30
275%
1,80%
20
24%
4.2%%
2.54%
10
344%
-5
5.0%%
0.03%
140%
143%
Traffic
Weather
(Hourly)
ERCOT Load
Australian Electricity
Exchange Rate
FRED MD
NN5 (Daily)
M5
ETT (15 min
MORE
ETh1
ETT2
ET7m1
ETTmZ
Weather
Bectniit
Dataset
Dataset
Figure 3: % improvement using geometric mean achieved by AT4IS over zero-shot across datasets for Chronos and TTM.
Horah uotell Electiitty
Madt
and Oornos
1/
w
30000
2007
20000
20008
2000
1455
250
505
Figure 4: Performance comparison for Zero-shot and Autotuned models for Monash Australian Electricity and Exchange Rate datasets.
(Ablation studies) and (Ansari al. 2024 section 5.6 (TSMixup Augmentations), dataset coverage and resolution diversity have also shown to play pivotal role in improving the zero-shot performance of these pre-trained models on unseen (out-ol-domain) datasets.
Our results also validate that LoRA achieves superior performance while significantly reducing the number of trained parameters for most of the out-of-domain target datasets. We compare our method against the zero- shot performance using relative error scores. In Figure 3 left. plot visualizes the improvement achieved by AT4TS comparison to the zero-shot model for each datasct. The relative error scores are then aggregated across all datasets using the geometric mean similar to (Ansari al., 2024). Therefore, our method achieves overall relative improvement of 5.26% over the zero-shot model performances across all datasets. This is denoted by the red dotted line in the plot. We can see that the autotuned model outperforms the zero- shot model for most datasets. Moreover, it exhibits particularly strong performance on datasets such as exchange rate with a significant MASE improvement of 20.55%. This can be partially altributed to its relative dissimilarity with the pretraining datasets (see the Appendix Section 8.1).
Figure illustrates the predictive performance of the various Chronos models on the target domain datasets Monash Australian Electricity and Exchange Rate respectively. Here, compare the forecasts obtained
11

---

Published in Transactions on Machine Learning Research (09/202))
1500
AT415
1000
Ground Tnuth
500
2022-08-23
2022-06-25
2022-06-27
2022-06-29
2022-07-01
2022-07-03
2022-07-05
2022-07-07
2022-07-09
Zero-Shot
2000
AT4TS
Ground Truth
0
2022-05-13
2022-05-15
2022-05-17
2022-05-21
Zero- Shor-Shot
1500
AT4TS
Ground Truth
2022 06-25
2022-06-27
2022-06-29
2022-07-01
2022-07-03
2022-07-05
2022-07-07
2022
2022-07-11
Figure 5: Performance comparison for Zero-shot TTM and autotuned TTM using AT4TS for MORE data.
Avg MSE Score Comparison Across Different HPO Methods for AT4TS using TTM
100 Mathod
Hypcropt
0.25
MSE Score
0.20
0.30
0.00
MORE
BTT2
mmmm
Weather
Bectricky
Figure 6: Avg MSE score comparison across different HPO Methods for AT4TS using TTM.
on the test split by our autotuned model with zero-shot models, clearly highlighting the improved prediction accuracy obtained using our approach. Tn summary, our findings demonstrate that AT4TS can efficiently autotune time series foundation models enhancing their downstream performance in the target domain.
Table 0 further explores the effectiveness of LDS in comparison to other state-of-the-art HPO techniques, namely BOHB and Hyperopt. We observe that the average rank for HyperOpt and LDS is equal. However, in comparison to BOHB, LDS is marginally better with a lower average rank. This demonstrates that LDS performs on par with the stale-of-the-arl HPO lechniques. It is noteworthy that we only run 10 trials in our experiments for Chronos to demonstrate the robustness of AT4TS in a resource constraint environment and achieve strong performance in comparison to zero-shot and default LoRA fine tuning. However, in the presence of more computational and time resources, AT4TS can easily scale well explore much larger hyperparameter search space.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Next, we present the results obtained on using TTM-Base model with AT4TS, zero-shot and default fine- tuning approaches for 7 datasets. For default fine-tuning setting, we fix the parameters such as batch size and head dropout based on the target dataset similar to (Ekambaram et al. 2024j. The most striking results are observed with the MORE datasct. As shown in Table 7 we observe that the autotuned model using our approach, AT4TS, outperforms both zero-shot and default. fine-tuning approaches for all datasets. The predictive performance of autotuned model using AT4TS better than zero-shot by 45.34% for MORE data as shown in the right plot of Figure 3 The same is illustrated in Figure 5 where we observe significant improvement in the prediction accuracy. Our approach also demonstrates superior performance compared to default fine-tuning, achieving average MSE of 0.135 with 5.48% overall improvement across forecast lengths and up to 13% improvement for longer forecast horizons. We highlight that AT4TS successfully explores the best. hyperparameter configurations leading to improved fine-tuned performance on out-of- domain target datasets overcoming the drawbacks associated with traditional fine-tuning techniques with fixed hyperparameters.
the comparison of LDS to other state-of-the-arl HPO techniques, namely ВOHB and Hyperopt. In general, we observe that HPO methods tend to consistently improve the per- formance of the fine-tuned model, compared to default fine-tuning configuration. Furthermore, we also calculate the average rank for the 3 HPO methods across all datasets to clearly highlight that LDS is on par with state-of-the-art HPO techniques such as Hyperopt and BOHB, performing slightly better in some cases. The findings are particularly interesting as we can observe from the bar plot in Figure F that all HPO methods converge to the lowest MSE corresponding to the best hyperparameter configuration except for datasets, MORE and ETTh2 where LDS is marginally better than other HPO methods. This behaviour be attributed to the following reasons, first, TTM-Basc is much smaller (1M) model in comparison to Chronos-mini (16M) model which implies fewer parameters to fine-tune. Second, in the custom fine-tuning approach for TTM, the model backbone is frozen and only the model head is updated. And, Third, we run 15 trials tuning a much smaller hyperparameter search space in comparison to Chronos which results in faster convergence to the optimal configuration.
Similar to Chronos, T
6 Conclusion and Future Work
Time series foundation models efficiently capture long-range patterns and dependencies, improving the model's ability to predict complex temporal relationships. However, their successful application to specific downstream tasks needs adaptation to the target domain datasets which can be achieved via fine-tuning, In this work, we propose AT4TS, Autotune for time series foundation models using parameter efficient fine-tuning methods along with LDS as the search strategy. We also compare LDS with other SOTA HPO methods namely BOHB and Hyperopt and show its strong competence. Our approach outperforms tradi- tional fine-tuning strategies specifically for out-of-domain datasets not seen during the model pre-training phase, In addition, we show that AT4TS can be easily applied to both transformer and non transformer- based architoctures making it highly scalable yielding competitive results for out-of-domain datascts. In the future, we aim to extend AT4TS for more complex TSFMs and hybrid fine-tuning approaches as we witness continued research in the rapidly evolving TSFM space.
Acknowledgments
This work has been partially supported by the 6G-XCEL project (grant agreement 101139194), funded by the EU Horizon Europe program.
References
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinory reviews: compu- tational statistics, 2(4):433-455 2010.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmon, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Olek- sandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint or arXiv:2403. 03..0781, 2024.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. Parameter efficient fine tuning: A comprehensive analysis across applications. arXiv preprint orXin:2404. 13506, 2024.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-paramett opti-
mization. Advances in neural information processing systems, 24, 2011.
James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox. Hyperopt: a python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8(1):014008, 2015.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Anloine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXie:2108.0 008.07508 2021.
Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, Kevin Leyton-Brown, et al. Towards an empirical foundation for assessing bayesian optimization of hyperpa- rameters. In NIPS workshop on Bayesian Optimization in Theory and Practice, volume 10, pp. 1 5, 2013.
Vijay Ekambaram, Arindam Jati, Nam H Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. Tims: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. aix preprint arXiv:2401.03960 039555, 2024.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International conference machine learning, pp. 1437-1446. PMLR, 2018.
Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
Azul Garza and Max Mergenthaler-Canse Canseccc.. Timegpt-1, arXiv preprint arXiv:2310.09582 2023.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob J Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. arXiv preprint arXto:2105.066 06643, 2021.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Wilson. Large language models are zero-shot time series forecasters. Aduances in Neural Information Processing Syslems, 36, 2024.
Divij Gupta, Amubhav Bhatti, Suraj Parmar, Chen Dan, Yuwei Liu, Bingjie Shen, and San Lee. Low-rank adaptation of time series foundational models for out-of-domain modality forecasting. In Proceedings of the 26th International Conference on Multimodal Interaction, pp. 182-386,4 2024.
William D Harvey and Matthew L Ginsberg, Limited discrepancy search. In IJCAI (1), pp. 1995.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-are Snowledge-basee systems, 212:106622 2021.
Luis Hernandez, Carlos Baladron, Javier M Aguiar, Belén Carro, Antonio J Sanchez-Esguevillee Jaime Llorct, and Joaquim Massana. A survey on electric power domand forccasting: future trends in smart grids, microgrids and smart buildings. IEEE Communications Surveys & Tutorials, 16(3):1460-1145, 2014.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.100 10196, 2023.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. ai preprint. 1xX.:2200..17700 2020.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: Bandit- based configuration evaluation for hyperparameter optimization. In International Conference on Learning Representations, 2022.
Yuxuan Liang, Haomin Wen, Yuqi Nic, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6555-6565, 2024.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint 0rXim:1807.05116 2018,
Xueqing Liu and Chi Wang. An empirical study on hyperparameter optimization for fine-tuning pre-trained language models. arXiv preprint Xiv: 202::0006099044 2021.
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint urXiv:2306.0975 2023.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. Xiv preprint arXin:2211.14 1.14730, 2022.
Martin Peterson. An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, edition, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140))11-67, 2020.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, al. Lag-Ilama: Towards foundation models for time series forccasting. arXiv preprint arXim:2310.0827, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amir Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprinl Xix:22300. 13971, 2023.
Christophe Tribes, Sacha Benarroch-Lelong, Peng Lu, and Ivan Kobyzev. Hyperparameter optimization for large language model instruction-tuning, arXiv preprint arXin:2312.000 009499 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you neod. CoRR, abs/1706.03762, 2017. URL http:
//arxiv. wtt.:///g.aa/./....../. 03762
Wei Wang. Baopu Li, Shuhui Yang, Jing Sun, Zhengming Ding, Junyang Chen, Xiao Dong, Zhihui Wang, and Haojie Li. A unified joint maximum mean discrepancy for domain adaptation. arXiv:2101.0999 2021.
ar preprint
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. Xiv preprint ar arXiv:2202.0r 07125, 2022.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with aulo-correlation for long-term series forecasting. Advances in neural information processing systems, 34: 22419 22430, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time scries analysis. arXiv preprint 2rXiu:2210.02186. 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning. pp. 27268 27286. PMLR, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2025)
7 Appendix
7.1 Dataset Similarity Experiments
In this section, we add the details of the additional experiments performed to measure the similarity between pre-training datasets against the target datasets used in evaluating our approach. We use Maximum Mean Discrepancy(MM)) (Wang et al. 2021 to measure the marginal distribution disparity between the pre- training and evaluation datascts. MMD is a non-parametric distance mcasure between the source and target domains computed by converting the data into a Reproducing Kernel Hilbert Space (RKHS), using the below equation,
NED 1 N
N Σ 9=1
MMDDP((s)), [XxD() Bio(XTD))ll
1
p( Σ p=1
NSD
where P(XsD) and P(XTD) are the marginal data distributions of the source and target domains, respec- tively, NSD and N are the number of data samples in the source and target domains, ip is the mapping function from the original feature space to the RKHS, and H is the RHKS space. The value of MMD starts from O, indicating that the two domains are completely identical. As the values increase, this indicates the increase in dissimilarity between datasets.
Due to the diverse sizes of pre-training datasets (ranging from 12*1320 to 225250 of these foundation models, we first perform Principal Component A Analysis (PCA) (Abi & Williams, 2010 to reduce the dimensionality and transform the feature space to principal components which is then used to calculate the MMD values. For datasets where the time series length is longer than 10000, we randomly sample 10000 points lo perform PCA. For all other datasels, we use the complete dataset for PCA. Table 9 and [0 list the dataset sizes considered for performing PCA for both pre-training and target datasets. The datasets were normalized to have zero mean and unit variance prior performing PCA. Based on experimentation with different component values, select 5 principal components for all datasets.
Table 9: Dalaset size used for PCA of pre-training datasets.
Dataset size for PCA (Time Series len of series)
Pre-training Datasets
Mexico City Bikes
100000494
Solar (5 min)
5006
Solar (Hourly)
5760
Taxi (Hourly)
734*2428
Wind Farms (Daily)
354"337
Wind Farms (Hourly)
8514"337
Table 10: Dataset size used for PCA of target datasets.
Dataset size for PCA
Target Datasets
(Time Series length of series)
Monash Traffic
10000 862
(TT(H)uuly)
10000*14
ERCOT Load
10000*8
Australian Electricity
10000*5
Exchange Rate
7588*8
FRED-MD
107
NN5
791*111
ETT((5 min)
10000*14
M5
1562+30490
Weather
1000
17

---

Published in Transactions on Machine Learning Research (09/2025)
Table 11: MMD values obtained on comparing the Chronos target datasets with a subset of pre-training
datasets.
Subset of Pre-training Datasets (Chronos)
Avg
Target Dataset
City
Solor min)
Solar (Hourly)
(Hourly)
Farms (Daily)
Wind (Hourly)
Monash Traffic
0.068671
0.393231
0.304424
0.080626
0.083929
0.080088
0.165
Weather
0.034445
0336337
0. 267396
0,013908
0.066638
0.047634
0.1244
(Horlyy
0,181774
0.821962
0. 73733122
0.491731
0:4373188
0,421104
0.5638
ERCOT Load
0.658115
1,00162
0.91165
0.068
0.615951
0.383116
0.7399
Austrilia Electricity
L058788
0. 1.968819
0.6608822
0.645624
Exchange Rate
0.594269
0.936563
0. 846584
0.603339
0.547911
0.523279
0.6753
FRED-MD
0.096056
0.459704
0. 0.300734
12033
0.1 1.118786
0.092687
0.2107
NNG
0.146068
0,516722
0.4207748
08142433
0. 0.163181
0.126783
0.2000
0,038101
0.3417983
0. 0.259012
0.014831
0.059337
0.050811
0.1282
EIT min)
0,450568
0.793353
0.. 117033383
0,460112
0.407911
0.382236
Table 11 shows the MMD values obtained by comparing the pre-training datasets with the target datasets.
We compare each dataset used validate AT4TS to subset of pre-training datasets the Chronos models.
We consider a subset of pre-training datasets from the Chronos paper (Ansari et al. 2024) mentioned in Table 3 Appendix section B. We calculate MMD values per dataset and then provide average across all the pre-training datasels. We observe that MMD values support our argument for both Australian Electricity and Exchange Rate datasets which show high MMD values of 0.798 and 0.675 respectively indicating rela- tively high dissimilarity with the pre-training datasets compared with other target domain datasets. Hence, Autotune leads to significantly better results for these datasets given they are both out-of-domain as well as showing dissimilarity with the pre-training datasets.
However, we hypothesize that dataset similarity is not the only criteria that influences the zero-shot and fine-tuned performances of the pre-trained time series foundation models as discussed in (Ekambaram et al,, 2024) Section 4.7 (Ablation studies) and (Ansari et al. 2024) section 5.6 (TSMixup Augmentations) where the dataset coverage and resolution diversity have shown to improve the zero-shot performance of these pre-trained models on unseen datasets. These findings point towards other factors beyond distance-based similarity that affect the final performance these models, As we can see, MMD values only partially support the argument as expected and should not be considered the only deciding factor in assessing the performance improvements achieved auto-tuning these models.
18


Reviews:
[{"review_id": "ujTtRDjwcF", "reviewer": "Reviewer_uxTe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper studies an interesting and relevant question - should we fine-tune TSFMs, and if so, how it should be done? The findings that 1/ fine-tuning consistently improves the forecast accuracy and 2/ the specific configuration used during fine-tuning can noticeably affect the results are relevant for the research community.\n- The experiments are designed in a way that clearly answers the research questions posed in the paper. The effectiveness of the proposed approach is demonstrated across different model architectures (encoder-decoder transformer / tsmixer), fine-tuning strategies (LoRA / output layer fine-tuning), evaluation metrics (MASE / WQL) and dataset types. \n\n**Weaknesses**\nI have not identified major weaknesses related to the submission, but there are a few claims and methodological choices that could be adjusted to further strengthen the work. Please see \"Requested Changes\" below.\n\n**Minor comments**\n- Consider using the `\\citep` command for citations that should be put in parenthesis.\n- A potentially relevant work that also investigates fine-tuning strategies for TSFMs including Chronos: Arango et al., \"ChronosX: Adapting Pretrained Time Series Models with Exogenous Variables\", AISTATS 2025", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Time series foundation models (TSFMs) are able to produce accurate forecasts in zero-shot manner. However, it has been repeatedly been shown that fine-tuning these models on the target task often leads to even better performance. The specific configuration for fine-tuning, such as choice of the learning rate or batch size, can significantly affect the final performance. This paper address this problem, and introduces an automated hyperparameter selection procedure for fine-tuning of TSFMs. The proposed approach is based on the Limited Discrepancy Search methods. Empirical evaluation demonstrates the effectiveness of this approach, as it often leads to better forecast accuracy compared to zero-shot forecasting or fine-tuning with the default configuration. The results are validated using two TSFM architectures (Chronos-mini and TTM), various datasets and evaluation metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The ablation study comparing LDS with other existing HPO strategies (Tables 6 and 8) does not convincingly show that the LDS strategy consistently outperforms the other HPO methods. The errors bars for the methods overlap, and no method emerges as universally superior to other. A more appropriate conclusion based on these results could be that HPO methods tend to consistently improve the performance of the fine-tuned model, compared to fine-tuning with the default configuration.\n- Using arithmetic mean when aggregating the relative improvements over the zero-shot model in Section 5 / Figure 3 is a questionable choice since we are effectively dealing with rations, and the relative improvement (1 - error_method / error_baseline) can take values in (-\\infty, 1). A more appropriate choice would be the geometric mean of the relative errors error_method / error_baseline.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "unMA5GXWy3", "reviewer": "Reviewer_CEoN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weakness**\n- In general, it is a little bit difficult to understand the contributions of this paper. This paper propose an end-to-end time-series forecasting model fine-tuning with hyper-parameter optimization which is more like combining the fine-tuning techniques with hyper-parameter optimization on time-series forecasting domains.\n- Also, the results are quite trivial. Zero-shot < fine-tuning with default hyper-parameters < fine-tuning with hyper-parameter optimization. This is quite trivial on not only the time-series forecasting, but also any predictive model cases.\n- In general, to make the customized time-series model, researchers and developers usually follow the similar procedure: (i) fine-tuning with LoRA or other PEFT, (ii) hyper-parameter optimization. It is unclear how the proposed framework is different (and novel) from the standard procedure.\n- Minor: Citation format (and style) in the entire paper needs to be modified (please use parenthetical citation)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors proposed an end-to-end fine-tuning method for time-series forecasting framework with hyper-parameter optimization.\n- The authors provided various experiments with diverse combinations of time-series foundation models as well as hyper-parameter optimization methods.\n- The results show the following conclusion in terms of performance: fine-tuning with hyper-parameters optimization > fine-tuning > zero-shot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weakness part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SMeA6SJUQT", "reviewer": "Reviewer_y4UW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* Interesting real-world problem\n* The choice of the methods to be benchmarked is sound\n\n\n**Weaknesses:**\n\n* The paper's novelty is limited and its conclusions are expected (using standard hyper-parameter selection helps even at the fine-tuning stage for time series foundation models)\n* Some experimental details could be improved\n* Tables 6 and 8 show that there is no clear winner among the hyper-parameter selection methods", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for tuning the hyper-parameters of the fine-tuning procedure for time series foundation models on target data. The novelty of the work does not lie in the method itself, but rather in the combination of standard hyper-parameter tuning strategies with standard fine-tuning methods (such as LoRA or forecasting head fine-tuning) applied to standard time series forecasting tasks using standard architectures. The main contribution is therefore the integration and evaluation of these existing components together.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "First, this paper discusses autotuning strategies for fine-tuning time series foundation models. It would hence be beneficial to provide a review of the state of the art for autotuning fine-tuning methods in fields other than time series forecasting.\n\nSecond, as discussed lightly in the experiment section, fine-tuning is likely to behave very differently depending on whether the target domain (or a very similar task) was present in the training set of the model or not.\nWhether the target domain was present in the training data is already discussed, but it should be clearly highlighted in the result tables (eg. have separate sections for in-domain and out-of-domain settings).\nThen, for domains that are supposed to be \"close to\" some of the training domains, this claim should be backed by some measure of closeness.\n\nFinally, I have two minor remarks:\n\n* Table 8 should have bold to indicate winners and ex-aequos\n* You state that you use two different metrics for the evaluation: why not stick to the same evaluation metric for all your experiments?\n    * Also you call MASE the Mean Absolute Squared Error, I guess you meant Mean Absolute Scaled Error", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)