Multi-Scale Finetuning for Encoder-based Time Series Foundation Models
Authors: Zhongzheng Qiao
Chenghao Liu
Yiming Zhang
Ming Jin
Quang Pham
Qingsong Wen
Ponnuthurai Nagaratnam Suganthan
Xudong Jiang
Savitha Ramasamy
Venue: NeurIPS 2025 poster ()
Keywords: Time Series Foundation Model
Time Series Forecasting
Multiscale modeling
Finetuning
Primary area: deep_learning

Abstract: Time series foundation models (TSFMs) demonstrate impressive zero-shot performance for time series forecasting. However, an important yet underexplored challenge is how to effectively finetune TSFMs on specific downstream tasks. While naive finetuning can yield performance gains, we argue that it falls short of fully leveraging TSFMs' capabilities, often resulting in overfitting and suboptimal performance. Given the diverse temporal patterns across sampling scales and the inherent multi-scale forecasting capabilities of TSFMs, we adopt a causal perspective to analyze finetuning process, through which we highlight the critical importance of explicitly modeling multiple scales and reveal the shortcomings of naive approaches. Focusing on encoder-based TSFMs, we propose Multiscale finetuning (MSFT), a simple yet general framework that explicitly integrates multi-scale modeling into the finetuning process. Experimental results on three different backbones (Moirai, Moment and Units) demonstrate that TSFMs finetuned with MSFT not only outperform naive and typical parameter efficient finetuning methods but also surpass state-of-the-art deep learning methods. Codes are available at https://github.com/zqiao11/MSFT.

Multi-Scale Finetuning for Encoder-based Time Series Foundation Models
Zhongzheng Qiao!:2 1,2,3
Chenghao Liu'
Yiming Zhang'
Ming Jin
Quang Pham 4
Qingsong Wen
P.N.Suantthan 7
Xudong Jiang'
Savitha Ramasamy 2,3
Abstract
Time series foundation models (TSFMs) demonstrate impressive zero-shot per- formance for time series forecasting. However, an important yet underexplored challenge is how to effectively finetune TSFMs on specific downstream tasks. While naive finctuning can yield performance gains, argue that it falls short of fully leveraging TSFMs' capabilities, often resulting in overfitting and subop- timal performance. Given the diverse temporal patterns across sampling scales and the inherent multi-scale forecasting capabilities of TSFMs, we adopt a causal perspective analyze finetuning process, through which we highlight the critical importance of explicitly modeling multiple scales and reveal the shortcomings naive approaches. Focusing encoder-based TSFMs, we propose MultiScale FineTuning (MSFT), a simple yet general framework that explicitly integrates multi-scale modeling into the finetuning process. Experimental results three different backbones (MOTRAI, MOMENT and UNITS) demonstrate that TSFMs finetuned with MSFT only outperform naive and typical parameter efficient finetuning methods but also surpass statc-of-the-an deep learning methods. Codes are available at https: / /tttpp:://ittann httpchwww.cco////////1////
Introduction
Time series foundation models (TSFMs) have emerged as a transformative direction within the time series forecasting (TSF) community 2 [3
By pretraining on extensive time series datasets, these models possess universal knowledge, enabling them to achieve impressive zero-shot performance on various forecasting tasks. Despite significant advancements in TSFM research, current studies predominantly focus on model pretraining and zero-shot evaluation, while paying limited attention to the critical challenge effectively finetuning these universal models for specific downstream tasks. In contrast, finetuning pretrained models has become the standard pipeline for real-world applications in domains such as natural language processing (NLP) and computer vision (CV). Research in these fields has revealed key challenges in finetuning foundation models, including preserving pretrained knowledge [24], avoiding overfitting [15], and ensuring efficient adaptation
[13 48
Existing finetuning strategies for TSFMs often rely naive approaches, such as full finetuning or linear probing 12 [.]. While these methods may offer performance gains, we argue that naive finetuning is suboptimal for TSFMs as it fails account for the intrinsic multi-scale properties of both time series data and TSFMs. As data modality generated from continuous real-world processes, time series are inherently entangled and can be decomposed across multiple scales [23] [8] A time series can exhibit distinct temporal patterns at different sampling scales. For instance, as shown in
Nanyang Technological University. Pnsttitute for Infocomm Research, STAR. 4 'Salesforee Al Research. 'Griffith University. "Squirrel Learning. "Qatar University. Mail to: Zhongzheng Qiao <qiac0020@e, ntu. edu. ag>, Chenghao Liu <chenghao liu@ealesforce. com>.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prediction with Moinai on Turkey Power Dataswl
8
Data-ievel
Scale (bwby)
Multi-scale property
ange
191
Multi-ronle Prenuining
Singlesccule Finettning
Model-level
Livl
TSFM
TSRM
X: Context Horizon S: Scale
Ninusaty Cotaeet
M: TSFM Activated Knowledge
Daramars
(a)
(b)
Figure 1: (a) Multi-scale property in time series foundation model (TSFM) finetuning. Finetuning TSFMs on the original scale may overlook potential temporal patterns in time series and underutilize their multi-scale forecasting capabilities learned during pretraining. (b) Causal graph for forecasting TSFMs. Nodes denote the abstract data variables and directed edges denote the causality, i.e. cause + effect. Scale S acts as a confounder, influencing both input context series X and model's activated knowledge M (shown in red).
Figure [ (a), energy consumption measured at the hour level shows micro- scopic local usage patterns, whereas daily records suppress these finer details, highlighting macro-scopic consumption trends instead. This multi-scale nature poses additional challenges, naive finetuning tends to overfit the model to patterns at the original scale, overlooking the latent dynamics that prevail at coarser scales. From a modeling perspective, TSFMs pretrained on extensive, multi-scale datasets are inherently equipped with robust multi-scale forecasting capabilities. However, naive finetuning fails to harness this potential, it restricts learning the original scale. Consequently, it underutilizes the pretrained knowledge of TSFMs, capturing only partial temporal patterns. Such failure not only limits the generalizability of TSFMs across scales but also leads to suboptimal downstream performance.
To address the aforementioned challenge, we begin by analyzing the finetuning process of TSFMs through a causal lens. The relationship among key variables is shown in Figure (b). Specifically, the objective of finetuning is to adapt the model P(Y|X) to capture temporal patterns and better predict the horizon Y given the context X. However, the presence of scale S as confounder introduces spurious correlations between context X and the knowledge M activated within TSFM, causing the model to rely correlations that lack causal grounding. Directly forecast with P(Y)X) would mistakenly associate non non-causal but positively correlated context X to horizon Y. To overcome this, propose using the interventional distribution P(Y [do(X)) which isolates the true causal effect of X on Y by blocking the influence of the confounder S. We will elaborate on how this is achieved through backdoor adjustment 27 in Section 3
This causal perspective highlights the need for explicitly modeling multiple scales during TSFM fine- tuning. However, integrating multi-scale modeling in this context remains underexplored and presents several non-trivial thallenges-despinn its success in standard time series forecasting modeling First, most TSFMs tokenize time series through patching [25], resulting in tokens at different scales exhibiting varying resolutions and temporal dynamics. This discrepancy complicates the finetuning of the unified input projection and attention weights. Second, applying attention across multi-scale tokens can introduce spurious dependencies due to misaligned time indices, making diffieult to capture true temporal relationships. Thus, the attention mechanism must account for or bypass index-related biases. Finally, since the model produces separate predictions at each scale, effectively aggregating these multi--cale outputs is essential for accurate and robust forecasting.
To close the gap, we propose a novel basede--based TSFM finetuning framework using multi-scale modeling, namely MSFT. Our contributions are summarized as follows:
Building on causal insights, we identify the limitations of naive finetuning for TSFMs and propose a multi-scale modeling approach for TSFM finetuning, To the best of our knowledge, this is the first work to introduce multi-scale modeling into TSFMs.
2. We propose MSFT, a simple yet effective finetuning framework for encoder-based TSFMs. MSFT begins by downsampling time series into multiple scales and independently tokenizing each scale at its own resolution. Scale-specific modules are applied to the input projection and attention layers to activate scale-specific knowledge. Decoupled dependency modeling is then performed on the concatenated multi-scale sequence, enabling the model to capture both within-scale (via

---

in- attention) and cross scale (via scale aggregator) dependencies. Finally, a learnable weighting strategy is employed aggregate the multi-scale prediction results.
3. Our extensive evaluation on various datasets for Long Sequence Forecasting [44] and Probabilistic Forecasting [4] demonstrates that MSFT not only significantly improves the fintuning results of TSFMs but also surpasses other state-of-the-art models trained from scratch.
2
Preliminaries
Problem Formulation. We first define the TSF task, in which the model predicts horizon window given a context window. Let C denote the context length and H the horizon length. Context window X E IR CxD and horizon window Y E IR Rx are consecutively extracted from the same time series X1. ( X2,.... XT). where D is the feature dimension each time step. The sample time step / is denoted (X,, Y.). where X, (Xr-C.... X and Y, (Xe..... H++H- Given model parameterized by 0 and training dataset Dirain {X,, Y, Yo model parameter 8 to achieve minimum error on the testing set Diest {(X,, Y))
f the objective is to learn the
Multi-Scale Generation. In multi-scale modeling (see Appendix A.2 for the detailed definition of this concept), the standard approach for generating multi-scale sequences is based on average pooling Given a training sample (X, Y), both context and horizon windows are downsampled into multiple temporal scales using non overraapag average pooling. Specifically, downsampling factor is commonly set to 2, resulting in a set of scales defined by 2....... 2 where K is the number of downsampled scales. Let S denote the set of multi-scale time series as S (So..... Sk), where S, (X), Y corresponds to the i-th scale series, formed by concatenating the downsampled context X E x and downsampled horizon Y " Here, C, = [ and H; Note that So represents the input series at the original scale.
B4 42
#
2
Encoder-based TSFM. We outline the architectural framework of existing encoder-based TSFMs 43 12 11 from high-level perspective. These models adopt an encoder-only Trans- former 39 architecture and segment univariate time series into a sequence of patch tokens (25). While multivariate extensions are supported in some models 43 M. we focus on the univariate case for illustration (D = 1), without loss of generality. The pretraining is conducted by masked reconstruction [9]. Given a time series (X, Y), the series is segmented into non-overlap patch tokens of size P. resulting in sequence of patches æ E R Nx where N [ + [ 1. The goal is forecast the predictive horizon by Y = fo(()) where fo is transformer with the block number and model dimension d. Specifically, Equation represents the procedure of calculating Y = [a(()):
0 InProject(æ); h' AttnBlock 11), I L...,, Y OutPPrject(h'")
(1)
Let h R Nxd represent the token embeddings produced by layer I. The input projection InProject embeds patch tokens into input embeddings h. Each AttnBlock consists of a multi-head self- attention layer, followed by feed-forward network (FFN) and normalization layers. The output
projection OutProject maps the output embeddings h l. to the prediction Y, either directly [1] or indirectly by first producing distributional parameters from which Y is sampled [43] We summarize the architectural features and training losses of each model in Appendix B.3
B.3
3
Multi-Scale Finetuning of TSEM
3.1 Multi-Scale Effect on TSFM: A Causal View
As we discussed in Section T both time series data and TSFMs exhibit multi-scale properties. We take scale into account during TSFM finetuning and construct a Structural Causal Model (SCM) (28) as illustrated Figure (b). The nodes denote the abstract data variables, and the directed edges denote the causality, i.c., cause + effect. Denoting input context window data as X, scale as S, and prediction horizon window data Y, we discuss the rationale for each link follows:
X + S. Given an observed recording of the context period, the input context series X is directly influenced by the scale S. Although corresponding to the same temporal range, X exhibits different temporal patterns and resolutions at different sampling rates.
S + M + X. We denote M as the activated knowledge within the pretrained TSFM's 's knowledge space, conditioned on input context. S + M indicates that the scale of data activates the correspond-
3

---

Overall Framework
Challenges
...
Output Projection
Transforner Encoder
Soulc
M
Mogtised cobollitg
Trea
Pooting
Concatenation
Scale
Buiee Soles
X
Inpur Projection
TewelD
Down-sample
Scale
Scale-variant Token
Resolution
Misaligned Croo-scale Attention
(3)
Figure 2: (a): The intervened Structural Causal Models (SCM) and overall MultiScale Tuning (MSFT) framework, which directly model P(Y|do (b): Challenges in directly applying the framework. Left: Downsampling and patching process for constructing multi-scale sequences. Patch tokens at different scales have varying resolution and schematics. Right: Directly applying self- attention over multi-scale embeddings leads to biased cross-scale attention due to misaligned time id.
ing scale-specific knowledge in the TSFM. Meanwhile, + M reflects that the TSFM activates context-specific knowledge with the input data X.
X X Y < M. This link represents that the model utilizes the activated knowledge M to generate predictions Y based on the lookback context data X.
is evident that scale is a confounder that induces spurious correlations between input context series (via S + X) and activated knowledge of TSFM (via S + M). The former captures the multi-scale properties of time series, while the latter corresponds to the multi-scale capabilities of TSFM. Scale S ultimately affects the forecasting of the prediction horizon via the backdoor path X S M - Y. Naive finetuned forecaster for P(Y ()) overlooks the impact of this backdoor path, learning forecasting only at the original scale. This oversight would mistakenly associate non-causal but positively correlated input context to forecast horizon in the original scale, resulting in problematic forecasting. Further diseussion can be found in Appendix
3.2 Causal Intervention via Backdoor Adjustment
Given this, we propose using P(Y]do(X)) X)) the new finetuned forecaster, which eliminates the confounding effect S and captures the true causal relationship from X to Y. As the "physical" intervention is impossible, by (1) blocking the link S + X and (2) S. As illustrated in Figure 2 (a, left), we have:
apply the backdoor adjustment (27] to "virtually" realize P(Y|do
P(Y)do(X))
Σ P(Y|X, M g(X, s))
where g is a function activate scale-specific knowledge of input. Grounded in this causal formula- tion, we design the MultiScale FineTuning (MSFT) framework to instantiate the intervention based forecasting process shown in Equation [ As shown in the right panel of Figure 2 a). the framework stratifies the confounder S by down- sampling the original time series into multiple scales. Each scale captures distinct statistical properties of the series and corresponds
specific value S.
Specifically, multi-scale series S (So..... Ss} } generated through the process described in Section 2 Each scale series S; is segmented into scale-specific patch tokens æ; E R.XP,, where N; the number of patches for scale i. The scale-specific input embeddings are computed by h InProject
Following the design of masked encoder (9], the embeddings falling within the forecast horizon are replaced with the learnable [mask] embedding. The input embeddings from all scales are concatenated into multi-scale sequence, ho Concat (h.. hf................. .h.....,. h&)). which is then passed to the Transformer for processing.
3.3 Challenges
Although the framework of Figure a can be directly applied without initiation of M g(X, s). we argue that it leaves following challenges unaddressed. First, the token schematics and intra-scale dependencies vary significantly across scales. As shown in the left part of Figure (b)), patch tokens at different scales exhibit distinct resolution and temporal schematics. When directly finetuning the

---

Input/Output Projections
Self-attention Blocks
Decoupled Dependency Modeling
Ouiget Enbeeding
10C rest Sonde Agzregaive
Mi & Noro
Impur Projection
Foeward
Ciarse
& Neem
Srute
Crom-icalee Augrepptor
Uppesmplc & Mixing
In-ecallo
Osipue Projetioo
Atication
Frozen Pretrained Parameteers
Decoupled Dependancy Modding
Muttipicaaion
Scaloopezfic New Trainable Parumatera
Eralwoddigg
Softnos
Figure 3: Complete design of MSFT based on the overall framework in Figure 2(a). ! Linear adapters are attached to the frozen input projection learn scale-variant input embeddings. @ Self-attention layers incorperate scale-specific Lora and decoupled dependency modeling. @ In-scale attention employs in-scale masking, ensuring tokens attend only to others within the same scale. ! Cross-scale aggregators progressively fuse tokens across scales in two directions, ensuring correct temporal alignment between tokens. @ Output projection generates separate predictions for each scale, which are then mixed by up- pamsling and learned weights.
input projection layer over all scales, each scale inherently tends to learn its own specific intra-token patterns, which can lead to interference across scales and suboptimal performance. Moreover, the resolution discrepancy induces ccale-inequivalent inter-token correlation, requiring the attention mechanism to capture scale-specific dynamics rather than assuming uniform interaction patterns.
Second, standard self-attention introduces misleading cross-scale dependencies due to mismatched time (position) indices. Since time indices are independently generated within each scale, tokens with the same index at different scales (shaded in gray in Figure b)) correspond to different temporal ranges. When self-attention is directly applied over over the concatenated multi-scale embedding sequence, attention scores across scales become biased: tokens attend more to others with the same time index, regardless of actual temporal relevance (see the right part of Figure (()).. This leads attention to capture spurious temporal correlations and attend to semantically irrelevant tokens.
Finally, the model generates distinct predictions each scale, and effectively mixing multi-scale predictions remains a non-trivial challenge. Although cross-scale information is partially fused through attention, prior studies [42] have shown that explicitly combining multi-scale predictions improves forecasting performance. However, naively averaging predictions across scales fails to account for their semantic and temporal heterogeneity, potentially leading suboptimal results.
4 Methodology
To address the aforementioned challenges, we propose MSFT to realize the high-level framework in Figure a) as an effective multiscale finetuning strategy. Specifically, to activate scale-specific knowledge, we freeze the pretrained parameters and introduce scale-specific, parameter-efficient modules into the ① input projection and @ attention layers. To eliminate the cross-scale attention bias and correctly capture temporal correlations, we propose a decoupled token dependency modeling mechanism: @ in-scale self-attention captures within-scale dependencies, while @ scale aggregators explicitly fuse information across scales, ensuring correct temporal alignment between tokens. Finally, apply multi-scale mixing to the i output projection, combining scale-specific predictions with learned weights. Figure iillustrates our MSFT method, with the detailed pseudo- code provided in Appendix B.I
Scale-specific Knowledge Activation. To address the problem of scale-variant token resolution, instead of directly finetuning the unified input projection layer across all scales, we freeze the pre- trained input projection and introduce a scale-specific adapter for each scale, implemented as a linear layer Linear;. Now, the input embeddings of scale is computed as he Linear, Aanear,,InProjjji (InProject(z:)).
5

---

Conditioned on the pretrained embeddings, these adapters independently learn specific representations variant resolutions, effectively avoiding interference across scales.
Similarly, to enhance the attention mechanism's 's ability to capture scale-variant dynamics, we incorpo- rate independent LoRA [33 modules for each scale. Specifically; we freeze the pretrained attention weight matrices, and the FFN block, and introduce a set of LoRA modules for each scale. Since both input embeddings and attention weights reflect scale-activated TSFM knowledge, this design serves as the implementation of g in Equation 2 enabling the activation of scale-specific knowledge M.
Decoupled Token Dependency Modeling. To ensure attention blocks capture the multi-scale embedding sequence's correct dependencies, we decouple the token dependency modeling into two parts: within-scale and across dependencies. Specifically, for tokens within the same scale, if they share the same pooolution-dpendeeeee they can be directly learned via self-attention. Thus, we only apply an in-scale attention mask Min ensure that each token attends only to tokens from the same scale.
To aggregate the knowledge between tokens from different scales, add cross-scale aggrega- tor after the attention operation. The aggregator consists of two branches, namely course-to-fine andfmne-to-course, where temporal-aligned token-level information fusion is iteratively conducted between consecutive scales two directions. First, since tokens at different scales correspond to varying resolutions, it is necessary to map embeddings to shared space before fusion. To this end, following [29] 30], we adopt a linear mapping Øi,, to project token embeddings from scale i to the embedding space of scale j in each layer I, where the mapped embeddings are defined as h = - = w h! b
Based on this mapping, token embeddings from one scale are projected to the adjacent scale and then fused according to their temporal alignment. We define the cross-scale token-wise fusion for the coarse-to-fine (C2F) and finett-ccoars (F2C) branches as follows:
C2F: h = h111 + Repeal (h), F2C: h1 = h441 AvgPool(h (11), for {O...., 1}
for i (K,..., 1}
(3)
(4)
where Repeat()) duplicates each coarse-scale token in h| along the sequence dimension to match the finer-scale resolution, based on their temporal correspondence. Conversely, AvgPool(-) aggregates groups of fine-scale tokens in h/,441 by averaging them according the downsampling factor, thereby aligning them to the coarser-scale resolution. Finally, the outputs from the two branches are combined by averaging their updated token embeddings. This decoupled two-stage design enables the model to capture temporal dependencies within each scale while effectively fusing complementary information across scales, leading to improved multi-scale temporal understanding.
Multi-scale Mixing. In the output projection, each scale independently predicts a forecasting horizon Y, based on its scale-specific tokens h from the final layer embedding h The training objective formulated as weighted summation of the scale-wise forecasting losses Lpedd (e.g., MSE or NLL). Since different scales may exhibit varying forecasting abilities and contribute differently to the final performance, we assign a learnable weight w; to each scale, corresponding the prior P(s) in Equation 2 The weights are obtained by applying a softmax function over a set of learnable parameters during training: fpeed Zi=0
K+1 W,Fpred During inference, we upsample the forecasting
results from each new scale to the original temporal resolution. The final prediction Y is computed as the weighted sum of the upsampled forecasts, using the same learned weights wy, This weighted mixing strategy can be seen as ensembling [26). which helps mitigate overfitting on the original scale, Additional implementation details for different TSFM architectures are provided in Appendix B.3
5
Related Work
5.1
Time Series Foundation Model
We focus our discussion solely on transformer-based TSFMs for TSF. Such TSFMs can be broadly categorized according to the backbone architecture. Encoder-only models like Moirai (3), Mo- ment [12 and UniTS [|] use masked reconstruction for pretraining. Decoder-only models, such

---

as TimesFM [8) Lag-Llama 32,, Timer 200 and Time-MoE B31 are pretrained by next-token prediction in an auto-regressive manner. Chronos (2) an encoder-decoder model. quantizes scaled time series values into discrete tokens and adopts the training objective originally developed for NIP. Despite the advancement of the field, existing TSFM research predominantly emphasizes pretraining and zero-shot performance. Although some studies 2 12 8 20) mention naive finetuning methods, these attempts are limited compared to the efforts devoted to pretraining and zero-shot evaluation. We include more detailed discussion in Appendix A
5.2 Multi-scale modeling in time series forecasting
Multi-scale modeling has garnered growing attention in the TSF community. Existing works mostly involves down-sampling, where coarser scales are derived from the original series using pooling or convolution. Models are then designed to capture multi-scale characteristics from these different views. Pyraformer [18] constructs a pyramidal graph of different scales and employs a pyramid attention mechanism extract multi-resolution representations. MICN [40] processes different scales separately through multiple branches with distinct convolution kernels and subsequently merges the outputs. Inspired by hierarchical forecasting, Scaleformer (34 and GPHT 21 iteratively refine the outputs from coarser to finer scales. TimeMixer [4] and TimeMixer++ 41 decompose each scale into seasonal and trend components, then integrate these components across multiple scales.
6 Experiments
We evaluate our proposed finetuning method, MSFT, on two prevalent TSF tasks: long sequence forecasting (LSF) and probabilistic forecasting (PF). For LSF, experiment with three TSFMs: MOIRAI, MOMENT and UNITS. For PF, we focus solely on MOIRAI, as it is the only model capable of probabilistic forecasting. Our evaluation includes comparisons with both deep saaming-based methods and other finetuning approaches applied to TSFMs. Detailed model configurations and experimental setups are provided in the Appendix
Table 1: Long sequence forecasting results, which are averaged across prediction lengths (96, 192, 336, 720). Each TSFM shows its zero-shot performance (highlighted in gray and results with different finetuning methods. The best finetuning results for each TSFM are highlighted in bold, while the global best results across all models are highlighted in red.
ETTml
ETTm2
ETThi
ETTh2
Electricity
Weather
Method
MSR
MSF
MSF
MSR
MSF
MAR
DI i..ieer
2023
0.403
0.419
0.350
0.401
0.456
0.452
0.589
0.515
0.212
0.365
0.265
0.317
PaccTTST
0.400
0.469
0.387
0.216
0.259
Thansformer
X024n
0.407
0.410
0.288
0.382
0.454
0.448
0.383
0.407
0.178
0.270
0.258
0.278
TimeMixer
2124
0.381
0.395
0.275
0.323
0.447
0.440
0.364
0.395
0.182
0.272
0.240
0.271
SimpleTM
2025
0.381
0.396
0.275
0.322
0.422
0.438
0.353
0.391
0.166
0.260
0.243
0.271
MOIRAlsaull
0.448
0.409
0.300
0.341
0.416
0.428
0.355
0,381
0.233
0.320
0.268
0.279
Full finetuning
0.382
0.316
0.415
0.352
0.378
0.193
0.228
Linear probing
0.38%
0.392
0.295
0.337
0.414
0.427
0.354
0.380
0,212
0.299
0.237
0.260
Prompt toning
0.384
0.391
0,292
0.334
0.414
0.418
0.354
0.381
0.217
0.304
0.235
0.258
LLRRA
0.370
0.383
0.272
0.314
0.414
0.427
0.354
0.380
0.192
0.279
0.225
0.252
oRA
0.381
0.386
0,273
0.319
0.414
0.427
0.354
0,380
0.191
0.279
0.226
0.252
MSFT
0.377
0.301
0.412
0.349
0.187
0.216
MOIRAlBe
0.381
0
0.281
0.326
0.412
0.424
0.356
0.388
0,188
0.274
0.246
0.265
Full finetuning
0.371
0.357
0.173
0.232
Linear probing
0.388
0.387
0.277
0.319
0.409
0.424
0.356
0.387
0.182
0.269
0.229
0.253
Prompt tuning
0.378
0.386
0,280
0.325
0.412
0.423
0.360
0,387
0.183
0,271
0.230
0.255
LoRA
0.371
0.308
0.409
0.358
0.384
0.173
0.230
0.258
AdaLoRA
0.359
0.371
0.258
0.307
0.410
0.423
0.356
0.384
0.173
0.264
0.236
0.260
MSFT
0.332
0.369
0.247
0.315
0.407
0.432
0.352
0.383
0.169
0.260
0.213
0.245
MONENT
Full finenaning
0.352
0.380
0,260
0.320
0.425
0.440
0.347
0.394
0.224
0.311
0.336
0.310
Linear probing
0.355
0.381
0.261
0.321
0.429
0.441
0.347
0.395
0.226
0.313
0.338
0.312
Prompt tining
0.356
0
0.261
0.320
0.427
0.440
0.348
0.395
0.226
0.312
0.336
0.310
LORRA
0.381
0.260
0.425
0.347
0.225
0.335
+ AdaLoRA
0.355
0.381
0.259
0.319
0.426
0.440
0.347
0.394
0.224
0.311
0.336
0.311
MSFT
0.344
0.377
0.255
0.316
0.422
0.436
0.345
0.392
0.221
0.309
0.332
0.3W7
UNITS
0.713
0.553
0.321
0.355
0.527
0.491
0,406
0.418
0.432
0.488
0.291
0.313
Full finetuming
0.395
0.405
0.297
0.338
0.442
0.435
0.386
0.409
0.190
0.283
0.257
0.283
Linear probing
0.409
0.445
0.437
0.392
0,412
0.200
0.291
0.274
Prompi tuning
0.431
0.430
0.299
0.341
0.438
0.433
0.386
0.405
0.191
0.287
0.247
0.276
LoRA
0.393
0.405
0.296
0.338
0.437
0.434
0.384
0.407
0.188
0.282
0.250
0.279
MSFT
0.390
0.403
0.288
0.334
0.434
0.430
0.380
0.405
0.184
0.279
0.242
0.273
7

---

Table 2: Probabilistic forecasting results. The best finetuning results for each TSFM are highlighted in bold, while the global best results are highlighted in red. See Table 15 for full results.
Electricity
Solar
Weather
Istanbul Traffic
Turkey Power
Mathod
CRPS
MSIS
CRPS
MSIS
CRPS
MSIS
CRPS
MSIS
CRPS
MSIS
DeepAR 2020
0.065
6.893
0.431
11.181
0.132
21.651
0.108
4.094
0.066
13.520
TFT 2022
0,050
6.278
0.446
8.057
0,043
7.791
0.110
4.057
0.039
7.943
PatelTTST 2023
0.052
5.744
0.518
8.447
0.059
7.759
0.112
3.813
0.054
8.978
TiDE (2223
(203
0.048
5.672
0.420
13.754
0.054
8.095
0.110
4.752
0.046
8.579
MOIRAIsmol
0.072
7.999
0.471
8.425
0.049
5.236
0.173
5.937
0.048
7.127
+ Full finetuning
0.055
6.009
0.395
6.947
0.039
4.477
0.151
6.735
0.040
6.887
Linear probing
0.062
6.438
0,369
5.865
0.049
4,785
0.154
4.645
0.047
6.912
+ Prompt tuming
0.066
6.595
0.421
6.936
0.050
4,901
0.154
4.733
0.045
7.042
+ LoRA
0.064
6.753
0.372
6.582
0.039
4.386
0.154
4.753
0.042
7.051
AAaaLoRA
0.064
6.892
0.366
8.015
0.040
4.496
0.152
4.670
0.041
7.127
+ -SFFT
0.047
5.327
0.353
7.706
0.036
4.178
0.141
4.447
0.038
6,810
MOIRAlnse:
0.055
6.172
0.419
7,011
0,041
5.136
0.116
4,461
0.040
6.766
Full finetuning
0.049
5.414
0.188
4.292
0.038
5.282
0.120
7.272
0.036
6.712
+ Linear probing
0.055
5,951
0.379
5.645
0.039
4,544
0.104
3.736
0.042
7,259
Prompt taning
0.054
6.024
0.412
6.885
0.040
5.274
0.105
3.987
0.040
6.698
+ LoRA
0.051
5.651
0.382
6,745
0.037
4,904
0.113
4.752
0.036
6.744
+ AdaLoRA
0.054
5.937
0.383
8.825
0.038
4.802
0.110
3.895
0.037
6.762
MSFT
0.046
5.199
0.142
3,464
0,035
4,603
0.098
3.685
0.034
6.419
6.1 Long Sequence Forecasting
subset of the widely-used long sequence forecasting benchmark [45) This subset is identical to the one used in Moirai [43 for LSF experiments and is not included in the pretraining data of TSFMs. Each dataset involves predictions at four different lengths, with the model is finetuned separately for each prediction length. We evaluate the performance using Mean Squared Error (MSE) and Mean Absolute Error (MAE).
Setup. We conduct experiments
As shown in Table 1 MSFT consistently enhances the forecasting performance of TSFMs. Across all models, MSFT outperforms other finetuning methods that use only the original scale, consistently delivering the best finetuned results. For MOIR A and MOIRAlBase- MSFT further improves their forecasting accuracy over their solid zero-shot performance, achieving competitive results across all datasets, with 10 out of 12 metrics showing the best performance. Notably, MSFT substantially improves MOIRAT'S AT's finetuned performance on minutely-level datasets. Compared to full finetuning, it achieves 6.8% lower MSE in ETTml, 6.3% lower MSE in ETTm2 and 6.7% lower MSE in Weather. In contrast, the improvement brought by MSFT on hourly datasets are relatively smaller compared to minute-level datasets. This discrepancy can be explained by the richer multi-scale patterns present in minute-level data, which MSFT can effectively leverage. For MOMENT, the improvements brought by MSFT are generally less pronounced compared to MOIRAI and UNITS. This can be attributed its pretraining with fixed context lengths, which limits their ability to extract information from new scales of varying lengths. Despite these differences, MSFT exhibit superior finetuned performance across diverse models and datasets, demonstrating its generalizability.
Results.
6.2 Probabilistic Forecasting
Setup. We evaluate on six datasets spanning various domains, using the rolling evaluation setup described Moirai 43) The test set comprises the final time steps, segmented into multiple non-overlapping evaluation windows. The length of the prediction window and the number of rolling evaluations are tailored for each dataset based on its frequency (see Table 5 details), For performance evaluation, we report the Continuous Ranked Probability Score (CRPS) and Mean Scaled Interval Score (MSIS) metrics.
Results. Experimental results in Table demonstrate that MSFT consistently delivers superior performance across all datasets. Building upon the strong zero-shot performance, MOIRAlBae achieves the best results for nearly all the datasets. MSFT provides consistent improvements over other finetuning methods, achieving an additional 24.4 CPRS relative reduction in Solar and 18.3 % CPRS relative reduction in Istanbul Traffic compared to full finetuning. A similar trend is also observed in the small model, demonstrating that our multi-scale modeling method can effectively enhance the fine-tuned performance of probabilistic forecasting.
8

---

Table 3: Ablation study on three LSF datasets using MOIRAISmall
InPreeject
Aftendion
In-scale Mack
X-scale Aggre.
Mixing
ETTml
ETTm2
Weather
Arg Diti
Scale
Shared
Scale
Shared
C2R
F2C
Avg.
Weigtted
MSE
MAE
MSE
M6N
MSE
MAB
MSE
MAE
0.219
0.340
0.304
0.318
0.374
0.308
0.224
0.361
0.306
0.222
0.254
0.233
0.255
0.363
0.220
0.253
0.357
0.218
0.251
0,360
0.220
0.253
0.359
0.379
0.269
0.313
0.226
0.252
0.011
0.252
MSPT
0.354
0.378
0,301
0.216
0.248
6.3
Model Analysis
To fully understand MSFT, conduct model analysis using the MOIRAISmall model on three LSF datasets, selected for its strong zero-shot performance and relatively low training cost. Due to page limits, we present the analysis of down-sampling approaches, down-sampling factors, detailed attention analysis, and visualizations in the Appendix [ We also discuss the potential application of MSFT to decoder-based structures and its limitation Appendix E
Ablation Study. ensure statistical robustness, we report mean results over three runs Table 3 with standard deviations provided in Table [ Ablations ! to ① examine the effectiveness of scale-specific knowledge activation. For both input projection and attention, either freezing (O, @ or finetuning shared weights (2. ()) yields inferior performance to using scale-specifie modules, with freezing causing larger performance drops. Among the two, attention has a greater impact than input projection, highlighting its critical role in capturing temporal dependencies.
Ablations ① & evaluate the effect of each component in decoupled dependency modeling. In ②) we remove cross-scale aggregators and only retain in-scale attention masking. Without cross-scale modeling, the performance suffers a significant decline. In ④ and @. we ablate the coarse-to-fine and fine-to-coarse branches, respectively. Both cases lead to performance drops, with the coarse-to-fine branch showing a stronger impact. In @, we completely remove decoupled dependency modeling, capturing dependency directly via attention the concatenated multi-scale sequence. This approach leads to misaligned cross-scale scole interactions and further degrades performance.
Finally, we assess the impact of multi-scale mixing. In O we disable prediction mixing, only using the original scale for prediction. ( we aggregate the multi-scale predicitions by averaging. Both approaches result in lower performance compared to our full model.
Effect of Number of New Scales. As shown in Figure 4 increasing the number of new scales K initially reduces errors. However, beyond a certain point, performance plateaus or declines, likely due to overly coarse predictions with few tokens disrupting multi-scale modeling. Our results indicate that setting K to 2 or 3 achieves the best balance.
Attention Analysis. Figure 5lshows the attention score heatmaps of three attention strategies. (a), direct attention (Ablation @) exhibits spurious temporal dependencies, with attention scores biased toward tokens sharing the same time indices. In (b), we align time indices during attention, ensuring that cross-scale tokens corresponding to the same temporal region share identical time indices. While this approach produces "correct" attention patterns, is limited to RoPE and performs worse than our method (see Appendix D for details). In (c), our in-scale masking strategy eliminates misleading cross-scale attention, focusing on accurate within-seale dependency modeling.
025
AGE
5.30
2024
6136
06.3
Scatee
(a) ETTml
(b) ETTm2
(c) Weather
(a) Naive
(b) Aligned
(c) Ours
Figure 4: LSF accuracy w.r.t. number of scales
Figure 5: Attention heatmaps of various methods

---

7 Conclusion
We introduce MSFT, a multi-scale finetuning strategy for encoder-based TSFMs. From a causal view, we highlight the limitations of naive finetuning and propose to use multi-scale modeling backdoor adjustment to mitigate the confounding effect of scale. By using concatenated multi-scale sequence input, applying simple scale-specifie model modifications, and employing decoupled dependency modeling, our method effectively aggregates performance. Our experiments show that MSFT not only significantly enhances the performance of the original foundation models but also surpasses other state-of-the-ard models trained from scratch.
scalt information and improves the forecasting
Acknowledgments and Disclosure of Funding
This research is part of the programme DesCartes and is supported by the National Research Founda- tion, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. Zhongzheng Qiao is affiliated with Energy Research Institute @ NTU, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore.
References
[1] Alexander Alexandrov, Konstantinos Benidis, Michael Bohike-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner TAfarkmen, and Yuyang Wang. 2020. GluonTS: Probabilistic and Neural Time Series Modeling Python. Journal of Machine Learning Research 21, 116 (2020), 1-6. http:// jmmr org/ wwwpr//pp//////a/e pttppers/21/19---1/ h/1////111111/// 820 .tmml
[2] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Syndar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Sohlke-SSchnidde and Yuyang Wang. 2024. Chronos: Learning the Language of Time Series. Transactions on Machine Learning Research (2024).
[3] Abdelhakim Benechehab, Vasilii Feofanov, Giuseppe Paolo, Albert Thomas, Maurizio Filippone, and Balázs Kégl. 2025. AdaPTS: Adapting Univariate Foundation Models to Probabilistic Multivariate Time Series Forecasting. arXiv:2502.10233 [stat.ML] actps://arxiv org/ abs/2502. 10235
[4] Hui Chen, Viet Luong, Lopamudra Mukherjee, and Vikas Singh. 2025. SimpleTM: A Simple Baseline for Multivariate Time Series Forecasting. In The Thirleenth International Conference on Learning Representations.
[5] Peng Chen, Yingying ZHANG, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. 2024. Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.
[6] Abhimanyu Das, Matthew Faw, Rajat Sen, and Yichen Zhou. 2024. In-context Fine-tuning for Time-series Foundation Models.
[7] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen, and Rose Yu. 2023. Long-term Forecasting with TiDE: Time-series Dense Encoder. Transactions on Machine Learning Research (2023).
[8] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2024. A decoder-only foundation model for time-series forecasting. In Forty-firs International Conference on Machine Learning.
[9] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
10

---

[10] Vijay Ekambaram, Arindam Jati, Nam H Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M Gifford, and Jayant Kalagnanam. 2024. Tims: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. arXiv preprint arXiv:2401.03395 8 (2024), 22.
[11] Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridiss and Marinka Zitnik. 2024. UniTS: A Unified Multi-Task Time Series Model. In The Thirly-eighth Annual Conference on Neural Information Processing Systems.
[12] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: A Family of Open Time-series Foundation Models. In International Confer- ence on Machine Learning.
[13] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.
[14] Tacsung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. 2022. Reversible Instance Normalization for Accurate Time-Serice Forecasting against Distribution Shift. In International Conference on Learning Representations.
[15] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. 2022. Fine-Tuming can Distort Pretrained Features and Underperform Out-of Cil-of-Distribution. In International Conference on Learning Representations.
[16] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long- and Short-Tere Temporal Patterns with Deep Neural Networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). 95-104.
[17] Bryan Lim, Sercan Anı, Nicolas Loeff, and Tomas Pfister. 2021. Temporal fusion transform- ers for interpretable multi-horizon time series forecasting. International Journal of Forecasting
37, 4 (2021), 1748-1764.
[18] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. 2022. Pyraformer: compCempyy Pyramidal Attention for Long-Range Time Scries Modeling and Forecasting. In International Conference on Learning Representations.
[19] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.
[20] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024. Timer: Generative Pre-trained Transformers Are Large Time Series Models. In Forty-first International Conference on Machine Learning.
[21] Zhiding Liu, Jiqian Yang, Mingyue Cheng, Yucong Luo, and Zhi Li. 2024. Generative Pretrained Hierarchical Transformer for Time Series Forecasting. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '24). 2003-2013.
[22] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-a thr Parameter-Efficien muniiinin methods.
https://github com hhwgg//ggggggg/effftt/t
[23] Michael C Mozer. 1991. Induction of Multiscale Temporal Structure. In Advances in Neural Information Processing Systems, Moody, S. Hanson, and R.P. Lippmmann (Eds.), Vol. Morgan- Kaugaann.
[24] Jishnu Mukhoti, Yarin Gal, Philip Torr, and Puneet K. Dokania. 2024. Fine-tuning can cripple your foundation model: preserving features may be the solution. Transactions on Machine Learning Research (2024). Featured Certification.
[25] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference Learning Representations.
11

---

[26] Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. N-BEATS:
Neural basis expansion analysis for interpretable time series forecasting. In International Con- ference on Learning Representations. htttps://openrevieex.u wttp://weeettttomommdeeeeeeeeeee
[27] Judea Pearl. 2014. Interpretation and identification of causal mediation. Psychological methods 19, 4 (2014), 459.
[28]
Judea Pearl. 2016. Causal inference in statistics: a primer. John Wiley & Sons.
[29]
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. 2018. 8. FiLM: visual reasoning with a general conditioning layer /AAAA/AAAAAAAAAAAA AAAI Press, 10 pages.
[30] Quang Pham, Chenghao Liu, and Steven Hoi. 2021. Dualnet: Continual learning, fast and slow. Advances Neural Information Processing Systems 34 (2021), 16131-16144.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1-67.
[32] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. 2023. Lag- Llama: Towards Foundation Models for Time Series Forecasting. In
of
Few-shot and Zero-shot Learning in Large Foundation Models.
[33] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36, 3 (2020). 1181-1191.
[34] Mohammad Amin Shabani, Amir H. Abdi, Lili Meng, and Tristan Sylvain. 2023. Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting. In The Eleventh International Conference on Learning Representations.
[35] Xiaoming Shi, Shiyu Wang. Yuqi Nie, Dianqi Li. Zhou Ye, Qingsong Wen, and Ming Jin. 2025. Time-MoE: Billion-Scall Time Series Foundation Models with Mixture of Experts. In The Thirteenth International Conference on Learning Representations.
[36] Peter Spirtes and Clark Glymour. 1991. An algorithm for fast recovery of sparse causal graphs. Social science computer review 9, (1991), 62-72.
[37] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Ro- Former: Enhanced transformer with Rotary Position Embedding. Neurocomput. 568, C (March 2024), 12 pages.
[38] Artur Trindade. 2015. Illerrictty.oaaDiiaaaas221111111 12014. UCI Machine Learning Repository. DOI: http:///co.org/1022444//23
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett (Eds.), Vol. 30.
[40] Huiqiang Wang, Jian Peng, Feihu Huang. Jince Wang, Junhui Chen, and Yifei Xiao. 2023. MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting. In The Eleventh International Conference on Learning Representations.
[41] Shiyu Wang. Jiawei LI, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Ju Shengtong. Zhixuan Chu, and Ming Jin. 2025. TimeMixer+:: A General Time Series Pattern Machine for Universal Predictive Analysis. In The Thirteenth International Conference on Learning Representations.
[42] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and JUN ZHOU. 2024. TimeMixer: Decomposable Multiscale Mixing for Time Series Fore- casting. In The Twelfth International Conference Learning Representations.
12

---

[43] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified Training of Universal Time Series Forecasting Transformers. In Forty-first International Conference on Machine Learning.
[44] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2023. TimesNet: Temporal 2D- Variation Modeling for General Time Series Analysis. In The Eleventh International Conference Learning Representations.
[45] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposi- tion transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems 34 (2021), 22419-22430.
[46] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023, Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 11121-11128.
[47] Jiawen Zhang, Shun Zheng, Xumeng Wen, Xiaofang Zhou, Jiang Bian, and Jia Li. 2024. ElaSTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[48] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, In The Eleventh International Conference on Learning Representations.
[49] Yitian Zhang, Liheng Ma, Soumyasundar Pal, Yingxue Zhang, and Mark Coates, 2024, Multi- resolution Time-Series Transformer for Long-term Forecasting. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. PMLR, 4222-4230.
[50] Yujia Zheng, Biwei Huang, Wei Chen, Joseph Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, and Kun Zhang. 2024. Causal-learn: Causal discovery in python. Journal of Machine Learning Research 25, 60 (2024), 1-8.
[51] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence casting, In The Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Virtual Conference, Vol. 35. AAAI Press, 11 11106-11115.
Fore-
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, which show that we focus on the multi-scale finetuning for encoder-based time series foundation models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Appendix details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results that need theorems or proofs.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental results can be reproduced. To prove the reproducibility, we provide our codes in the provided anonymous Github link.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and data will be accessed in the provided anonymous Github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setting/details can be found in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Since our models are based on foundation models and given the large number of experiments, do not have sufficient computational resources conduct three five runs for reporting error bars. However, based on our experience, we note that the finctuning results are quite stable, without large deviation between several runs using different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the information on the computer resource in Appendix B.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttts://neuriis... httpp//pg...p.bblttccttccccclaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on techinical problem of the finetuning of Time Series Foundation Models for time series forecasting. There no societal impacts on potential malicious unintended uses.
17

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data and models sourced from publicly available popular datasets or released pretrained time series models, which do not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the code package, models and datasets. The version and url of models/datasets are stated in the Appendix.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the balaaets.ppperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets provided in the anonymized URL are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only use LLM for writing, editing or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Detailed Related Works
A.1 Time Series Foundation Model
In this section, we further discuss related works involved TSFM finetuning. Some TSFMs conduct basic experiments in their original papers to show the effect of naive finetuning. For instance, Moment 12] applies linear probing and reports the corresponding results in their main experiments. Chronos
2 and Timer (20] apply full finetuning, with improvements over zero-shot perofrmance on their
respective benchmarks. TimesFM
tunes the input and output residual blocks ETT datasets.
Notably, some recent studies explore specialized finetuning methods for TSFMs. UniTS [TT] introduces prompt tuning strategy, and [6] employs in-context tuning TimesFM. However, both methods are tailored to their own model architectures and rely on specialized pretraining designs, which limits their generalizability and plug-and-play applicability. [3] inserts adapters on Moment to adapt this univaraite TSFM for multivariate probabilistic forecasting. However, this approach is restricted to this specific application, rather than serving general finetuning method applicable to various models or forecasting tasks. This gap underscores the need for more general, effective, and modular finetuning strategies for TSFMs.
A.2 Multi-scale modeling
The term multi-scale modeling has been used inconsistently in prior time series works. In our paper, specifically refers to constructing multiple downsampled versions of the same time series and jointly leveraging them through cross-scale aggregation during prediction. This enables the model to integrate temporal information from both coarse and fine resolutions within a single forward pass.
Several related terminologies exist in the literature. TTM [0] introduces multi-resolution pretraining by downsampling high-frequency datasets into frequecyy versions, which serves form of data augmentation during pretraining. However, each time series sample is still processed in single scale/resolution during its prediction process. Thus, TTM does not perform multi-scale modeling as defined in our work. Another similar strategy is multi-patch-siize modeling, which applies specifically to patch-based TSTs. Here, multiple patch sizes are used segment the time series into tokens in different resolutions. Pathformer [5] applies layer-by-layet routing to select patch sizes and aggregate the outputs from multiple scales. MTST [49 proposes multi-branch architecture for modeling diverse temporal patterns at different resolutions. ELasTST [47] leverages a shared transformer backbone with tunable RoPE for multi-scale patch assembly. Moirai [43) adopts multiple
patch-size projection layers, yet it does not downsample inputs into multiple temporal scales. In summary, this line of methods requires the model to compatible with multiple patch sizes, which is not feasible for most TSFMs. Under this clarified definition, our work is the first to introduce multi-scale modeling in TSFM finetuning.
B Implementation Details
B.1 Pseudo-code of MSFT
For clarity, we provide the Pytorch-like pseudo codes of MSFT in Algorithm T and Algorithm 2 illustrating the overall training pipeline and the MSFT attention block described in Section 4 Importantly, the method does not construct a new model from scratch, but enhances the pretrained TSFM through additional plug-in modules.
B.2 Dataset details
For long sequence forecasting (LSF), we conduct experiments on six well-established datasets, including the ETT datasets (ETTh1, ETTh2, ETTml, ETTm2) [51) Weather (45), and Electricity 44). We note that these datasets are not included in the pretraining datasets of the TSFMs we evaluated. The key properties of these LSF datasets detailed in Table 4
21

---

Algorithm Overall Training Pipeline for MSFT: PyTorch-like Pseudo-code class MSFTPipelineemn.. (on.uulls)).
__init__. (self, pretrained, _odel,, K, s=2, P-16): self.K # number of scales dounsample factor
self.s
self.P P # patch size
# frozen input projection, scale-specific linear layers self.in in.o pretrained,modet itt.eeddddddddded.......o.... in_proj self linears [Linear_i() for in range(K+1)]
# encoder layers with NSFT blocks self. encoder un. Modulelist(([
SstaBlock_watthMMPFT (block,,, K-K, 8=8)
block pretrained model. eacoder,_ayerr
])
frozen output projection self. colf.out_proo
def forward(self,, x, Y): Step 1: Multi-Scale Generation
S []] for in rangf(eelf.Kt))
X_i AvgPool (x, sandow_size=zef.oco pre-pad if needed
Y_1 AvgPool(Y, sindow_eize-self.eee. post-pad if needed s.append((X_1, Y_1))
# Step 2: Patching & Profection
[
for 1, (X_1, Y_1) in enmmerate (8))
x_i Batchhnn(((__3 Y_1), bitch_siz=-geff.P
h_1 self 1ii.ears (self. inelffin_projeee (x_i)) # frozen InProject
h_1 (haking(h_1)
prediction
(h...))
h Concat scale, index
GetScaleIndex (H_0)
# Step 3: Multi-Scale Attention Encoding
for in range L): encoder [1] (h, scale_index)
# Step 4: Output & Loss
HLL Split (h, scale_ index) recover [h_0",, ..., h_K"L]
losses, preds
[] []
for 1, C, Y_1) in enumerate
Y_hat_i = welf. oelf.oot_pprofff (H_L(i))
L_1 Loss(Y_1, Y_hat_i)
losses aaaaes.append(v....
L_i) weighted by learnable w_i
# upsample prediction back to original scale Y_hat_up (psample(Y_hats scale=aelf. areds.append(<x.. Y_hat_up)
L_total = sum (lossee)))
Y_hat sum sumeeeds)
L_total, Y_hat
Following Moirai 4
we use 5 out-of--istribuiie datasets for probabilistic forecasting: Electricity
[38, Solar-Power 16 Jena Weather, Istanbul Traffid and Turkey Power Detailed descriptions of these datasets are provided in Table
B.3 Encoder-based TSFMs
We describe the architectural details and training objectives of each encoder-based TSFM used in our experiments. Table summarizze the fundamental details of the models based on their origin setup.
turkey
22

---

Algorithm 2 Multi-Scale Attention Block (AttBlock_withhMsTT]]) PyTorch-like Pseudo-code class AttnBlock_ wtw.aht.ssPeeeerrrraeeeeeeeeeeee __inát___ (self, s=2):
super init_.
self self.s dounsample factor
number of new scales
# LoRA adapters for each scale self. M_Q (LoRAsself...... for in range (K+1)]
self M_K [LoRA(se1f __K) for in range (K+1)] self.u_V [LoRA(self
range(K+1)]
# aross-scale projectiong
Self.F2CMa. [Linear() for in range [X)] fine -> coarse self. C2FMap [Linear() for range (K)] coarse fine
def (srl, h_in, scale_index):
# Step 1: Split input into scale-sise representation H_i [h_in[.... idx, :] for idx in scale _cale_index]
# Step 2: Scale-apacific Attention with LoRA q, for in range (welf..mell.-4
[], D,
Q.append (W_Q[4] (H_in[i]))
a.append (M_X(]] in(i]))
(N_ (H_in(1])) V Concat(Q), Concat (K), Concat (V)
# Step 3: In-scale masked attention RaaleDDotProdccttttetttiiii.C X, mask-M_in)
# Step 4: Cross-scale Aggregation
Haatta (h_attn[.... idx, for idx scale_index]
(a) Coarse-to-Fine (C2F)
H_c2f = H_attn, copy
for in (self, 0, -1);
H_c2f[i-1] www.ccFFpppii111 (H_attn[il)
h.proj self
(h.proj fapeat_f
# (b) Pine-to-Coarse (F2C)
H_f2c = "Lattn.copy"
in range(Belf.
h_proj self FeMMap (H]atta[i])
H_f2e[i+1] +- AvgPool (h_proj, pool_ pool_eiz==eel..
# (c) Merge outputs from two branches
Houut
for in (self
H_out. appun..ppend (H_c2f(1) H_f2c(a)))
# Step 5: Re-concatenate b_out Concat(H_out)
# N_o Norm & FeedForward omitted for brevity return h_out
Moirai Moirai 43 one of the pioneering TSFMs for universal time series forecasting based on masked encoder architecture. It segments dingee-dmmessooal time series (a variate) into patch tokens and can extended to multivariate setup by flattening multiple variate into single sequence. Moirai employs multi patch size projection layers in both input and output projections, allowing it effectively handle data with varying frequencies. In the attention blocks, it encodes the temporal position of tokens using Rotary Positional Encoding (RoPE) [37], and encodes simple variate correlation by using binary attention biases to indicate whether two tokens belong to the same variate or not. The model produces distribution parameters for mixture distribution over the predictive horizon. The training objective is to minimize the negative log-likell likellooooo (NLL). During inference, predictions of horizon obtained by sampling from the predictive distribution. Point forecasts be derived by taking the median from the samples. our experiments, we use the univariate mode of Moirai, encoding different scales using distinct variate indices.
is suite of open-source foundation models designed for versatile time- series analysis tasks. Moment follows channel independence assumption and leverages T5
Moment Moment 12
23

---

Table 4: Summary of datasets used in the long sequence forecasting evaluation.
Task
Dataset
Variate
Dataset Size
Predict Length
Frequency
Information
ETTh1
7
17420
196, 192, 336, 720]
Hourly
Temperature
ET
7
17420
336, 720]
Hourly
Temperature
Long Sequence
ETTml
7
69680
[96, 192, 336, 720]
15 min
Temperature
Forecasting
ETTm2
7
69680
[96, 192, 336, 720)
15 min
Temporature
Electricity/
321
26304
[96, 336, 720]
Hourly
Electricity
Weather 21
52696
[96, 192, 336, 720]
10 min
Weather
Table 5: Summary of datasets used in the probabilistic forecasting evaluation setting.
Task
Dataset
Variate
Dalaset Size
Predict Length
Rolling Evaluation
Frequency
Information
Electricity
321
26304
H
Energy
Probahilistic
Solar
137
8760
24
H
Fairg
Forecasting
Weather
21
52696
144
7
10T
Climane
Istanbul Traffic|
3
14244
24
7
H
Traneport
Turkey Power
18
26304
Eiergyy
encoder architecture enhanced with sinusoidal positional encoding to effectively capture temporal dependencies within time series. Distinctively, during the forecasting fine-tuning phase, MOMENT utilizes the entire context series as input to directly get prediction results, diverging from traditional masked reconstruction methods commonly employed in pretraining. The model's forecasting head comprises a flatten operation followed by linear layer, and it is trained using the MSE loss function. Due to the computational resource constraints associated with finetuning and the large scale of the models, we employ Moment (Small) for our experiments.
UNITS UNITS originally designed for multi-tasks learning with specific task prompts. The transformer encoder is composed of multiple UNITS Blocks and ultimately processed through the GEN Tower to generate the final predictions. Specifically, within each UNTTS Block, the data sequentially passes through Time Self-attention, Variable Self-attention, and Dynamic FFN. Each of these modules followed by Gate Module, which enhances the model's generalization capability in multi- task learning by dynamically scaling the feature vectors. Time Self-attention and Variable Self-attention compute attention scores along the time and variable dimensions, respectively, while the Dynamic FFN dynamically adjusts the shape of the weight matrix through bilinear interpolation to match the lengths of the input and output. The GEN Tower is designed to accommodate varying input lengths for different tasks and to ultimately generate the output sequence. The model applies learnable additive position encoding. For forecasting task, the training objective is MSE loss.
B.4 Finetuning baselines
Full Finetune and Linear Probe Full finetuning involves updating all parameters of the pretrained model. We observe that using a small learning rate is crucial for stability and performance. In contrast, linear probing only updates the output head while keeping the backbone frozen; a larger learning rate is generally more effective in this case.
LoRA and AdaLoRA LoRA [13] introduces trainable decoopposoiiioon matrices into the attention layers, enabling parameter-efficien finetuning by injecting updates into low-rank subspace. AdaLoRA (48] extends LoRA by dynamically allocating the rank during training based on parameter importance, improving adaptation under a parameter budget. For Moirai and Moment, we directly adopt the PEFT library [22] for both LoRA and AdaLoRA. We apply LoRA and AdaLoRA to the query, key, and value projection layers. addition to the LoRA modules, we also allow the output prediction head to be trainable. The LoRA configuration follows standard settings with rank r = 16 and scaling factor a = 32. For AdaLoRA, we use the default configuration provided by the PEFT library. Since the original attention implementation in the UNITS codebase uses large shared weight
24

---

Table 6: Summary of encoder-based time series foundation models.
Feature
Moirai
Moment
UNITS
Citation
Woo el al. 2024
Guswarmi et al. 2024 I5 Encoder
Gao el al. 2024 Modifred Encoder 3.4M
Architecture
Naive Encoder
Params
Small: 14M Bare: 91M
40M
Open Souree
Point Forecasting, Probabilistic Forecasting
Point Forecasting, Classification. Anomaly detection, Imputation
Point Forecasting, Classification, Anomaly detection, Imputation
Evaluation Tasks
Layer
Small: 6, Base: 12
8 512 8 512 Sinusoidal
3
dronddd
Small: 384, Base: 768
128
Patch
[8, 16, 32, 64, 128]
Context Length
1000-5000
96
Position Embedding
RoPE 33
Leurnable Additive
for query, key, and value, applying LoRA or AdaLoRA from PEFT is not feasible. Therefore, implement a custom LoRA for it and do not conduct AdaLoRA experiments on UNITS.
Prompt Finetuning For Moirai and Moment, we implement prompt fine-tuning by introducing trainable soft prompt embeddings, which are prepended to the input tokens in the embedding space. We avoid inserting them into the patch token space, as doing so can interfere with the statistical computation of RevIN [4] and offers less expressive capacity compared to the high-dimensional embedding space. During inference, we discard the prompt embeddings from the encoder output and only the time series embeddings final representation for prediction. Similarly, only the output head and the prompt embeddings are finetuned, while all other parameters remain frozen. Prompt length is set to 2 by default. For UNITS, directly use its original prompt tuning implementation.
B.5
Metric details
For long sequence forecasting, we follow the standard protocols use mean square error (MSE) and mean absolute error for evaluation. For probabilistic forecasting, we include Continuous Ranked Probability Scoremean (CRPS), Mean Scaled Interval Score (MSIS), absolute percentage error (MAPE), symmetric mean absolute percentage error (sMAPE), mean absolute scaled error (MASE), normalized deviation (ND), and normalized root mean squared error (NRMSE) as metries. The definitions and calculations of probabilistic forecasting metrics are follows. Note that the notations used here are independent of those in the main text.
Continuous Ranked Probability Score truth Y, the CRPS is defined as:
Given predicted distribution with c.d.f. and ground
CRPS /' 2A (F Y)da
(g Y) = (a-1ap 1- 4).
where Ag is the a-quuanile loss, also known as the pinball loss at quantile level a.
In practice, the CRPS is intractable or computationally expensive to compute, and we also want compute normalized metric, thus we compute normalized discrete approximation, the mean weighted sum quantile loss, defined as the average of K quantiles:
1
CRPS ~
w [a[
K
wQL(a]= = E, Aa Y,)
Σ.
where Y, the ground truth at at time step i and ÿr (a) the predicted a-quantile at time step We
take K 9, aq 0.1, a2 ..,,..., ..., ag 0.9 in practice.
25

---

Mean Scaled Interval Score The MSIS is metric to evaluate uncertainty around point forecasts. Given an upper bound prediction, U. and lower bound prediction Li, the MSIS defined as:
MSIS
UU -
m E Ei=m+1 Y; -Y-mi
2 (L, (.) + (U (T,
where a = 0.05 the significance level for a 95% prediction interval, over a forecast horizon of length H, and m is the seasonal factor.
The sMAPE is a accuracy measure based on percentage errors, treating over- and under- peddi-predsstions symmetrically, commonly used in forecasting.
symmetric Mean Absolute Percentage Error
200 H
Y
Y, Y
SMAPE
Mean Absolute Scaled Error the in-sample mean absolute error of naive forecast, ensuring interpretability and comparability.
The MASE is a metric for forecasting accuracy, scaling errors by
H Σ E H-
Y. Y, 2j=0+1 Y Y
MASE
H
where s is the periodicity of the data. Y. Y E H are the ground truth and prediction results of the future with time pints and dimensions. Y, means the i-th future time point.
Normalized Deviation The ND measures prediction accuracy by standardizing deviations between predicted and actual values, aiding model evaluation and optimization.
H
Y;
ND
x 100%,
H
Y.
Normalized Root Mean Squared Error The NRMSE quantifies prediction error, enables model comparison, aids optimization, and provides interpretable results time series forecasting.
H H Σ (Y: (Y.)).)
NRMSE max Y) min (Y)
B.6 Experiment Details
Dataset Construction Unlike pretraining Moirai, where samples are randomly cropped from time series of varying lengths, we create the training, validation, and test datasets by cropping time series windows with fixed sequence lengths. Given the context and prediction lengths, samples are segmented using sliding window, where the window size is C' + H. The train-val-test split follows the default LSF setup. Data are normalized for LSF but not for PF.
Training Setup Since there is no official fine-tuning implementation for Moirai, we configure the training setup as follows. We use the AdamW optimizer with weight decay=0. 31 0.9, and 32 = 0.98 for optimization. Specifically, unlike pretraining, which uses a learning rate of 1e-3, we find that finetuning requires a much smaller learning rate. Based on validation performance, we select learning rate of either 5c-6 or 5e-7 for finetuning our models. The batch size is set to 512 by default for experiments using MOIR MOIRAlsmall- and reduced 256 on MOIRAIBue if GPU memory reaches its limit. We adopt constant learning rate scheduling, and early stopping is employed to monitor training. The context lengths are used directly from the values in the original Moirai models, which are tuned from a range of [1000, 2000, 3000, 4000, 5000]. The patch sizes are also taken from their provided values, which are selected based on data frequency. Since samples have the same sequence length, sequence packing is not used during training. For Moment and UNITS, we directly follow their provided their original finetuning configurations for experiments, with the learning rate selected from 5e-5, 5c-6, or 5c-7.
26

---

Evaluation Setup For Moirai, the evaluation is based on the GluonTS Library [.. Predictions are sampled 100 times from the learned predicitive distributions, and evaluation metrics are computed over those samples. For Moment and UNITS, the LSF metrics are directly computed based on the output predicted series.
Computational environment Our experiments conducted on a server equipped with AMD OPYC77733 CPU (64 cores, 128 threads) and four NVIDIA A40 GPUs, each with 40 GB of memory.
C Causal Analysis
C.1 Causal Modeling Motivation
Here we elaborate the motivation of our causal modeling in Figure 2 A time series can be viewed as discretized sequence of sampled observations derived from an underlying continuous process, Under this perspective, the observed input window X corresponds to a discrete observation of the latent process during context, period. The variable arises from two factors: the latent continuous process (unobserved) and the scale parameter S, which governs the sampling resolution. The scale S determines how densely the latent process is sampled, thereby shaping both the temporal granularity and the length of the observed sequence X. While our formulation omits the latent process / for tractability, the edge S + X in our causal graph reflects this observation mechanism. Importantly, scale influences the form of the observed input series but not the latent process itself.
Formally, we treat both and X as random variables. The scale S is a discrete variable that selects the resolution level for downsampling the input context. It takes values from finite index set = (s0, $i...., sK); }. where each sy corresponds to a specific downsampling factor or temporal resolution. The observed input X is then random variable whose sequence length depends on the selected scale S. Thus, X E X, where X U.es RR,, and I.2 denotes the input length corresponding to scale s. This formalization clarifies the role of scale in shaping observed input time series, consistent with the causal edge S + X in our proposed graph.
C.2 Empirical Validation of the Causal Graph
To empirically validate the proposed causal graph, we perform causal structure learning and partial correlation analysis on the ETTml dataset across multiple scales. Specifically, we extract context windows from the training split, downsample each window into three additional resolutions, and feed each scaled input into the pretrained Moirai model to obtain corresponding embeddings M. For each sample at each scale, we form triplet (S, X, M), where S is the scale index, X is the autocorrelation (ACF) computed on the input, and M is the &g norm of the mean embedding. These triplets enable graph-based causal discovery.
We first apply the PC algorithm (36 50 with Fisher's Z-test (a 0.01). The learned graph includes directed edges S + X and S + M, supporting assumption that scale causally influences both the input signal and the model representation. To further test whether acts as confounder between X and M, we compare their raw correlation with the partial correlation conditioned on S. The Pearson correlation between and M is 70.732; conditioning on reduces the partial correlation to -0.481 (p < 0.001). This reduction indicates that scale partially explains the dependency between X and M, consistent with its role as a confounder in our causal formulation. Together, these complementary analyses guantitatively support the causal assumption proposed in Section 3], namely that the scale variable influences both the observed input X and the model knowledge M..
D More Experimental Results
D.1 Further Model Analysis
Effect of Down-Sampling Methods While average pooling is the most commonly used method for generating down-sampled scales, we also investigate two alternative wown-sampling techniques to assess their impact. Specifically, we consider max pooling, which selects the maximum value within each down-sample kernel, and the first-step method, which directly selects the first time step of each kernel. We replace the original average pooling operation in MSFT with these alternatives
27

---

and evaluate their performance using On the ETTml and ETTm2 datasets. As shown in Figure 6al the results demonstrate that average pooling consistently outperforms the other two approaches, serving as the most effective method for multi-scale generation.
Aug hoot FTTm3
Mac Pool
Fest Skop FFTm?
0.26
0.25
g6.375
MS2
0.50
NSE
3..00
H5E
0.365
MAE
NAE
20.23
0.26
0.22
0.21
Downsomple
Downsample
MAE
NAE
(b) ETTml
(c) ETTm2
(d) Weather
(a) LSF accuracy for three
down-
sampling methods.
(c) LSF accuracy w.r.t down-sampling rate (with only I new scale).
Figure 6: Overview of LSF accuracy comparison.
Effect of Down-Sampling Rate We investigate the effect of down-sampplnn rate by using only one new scale (K = 1) and comparing the results across different wown-sampling factors (2, 4, 6, 8). As shown in Figure Giel the results reveal that the impact of down-sampling rate varies significantly across datasets. For E and Weather, the choice of cown-sampling factor is relatively less important, with no single down-sample factor is significantly better than the others. In contrast, ETTm2 exhibits a clear pattern: down-sampling factors of 4 and 8 obviously yield better performance, indicating that the periodic patterns in ETTm2 are better captured with these specific factors. These results demonstrate that the effect of downn-ampling is dataset-dependent Furthermore, they indicate that the performance improvement from using multiple scales not merely due to adding one particularly important scale but rather results from aggregating information across multiple scales.
Attention with Aligned Time Indices In this section, we provide a detailed discussion of the attention misalignment problem and explore another potential solution. As illustrated in Figure I (b) and Figure 5 (a), the problem of directly applying self-attention over the concatenated multi-scale sequence is that cross-scale dependencies are biased to the tokens with the same time ID. However, as tokens in different scales represent various resolution, their time indices do not represent the same temporal location information. The tokens in different scales with the same time id do not correspond the same temporal range (b)) left part). Therefore, this time ID-induced bias causes attention to learn misleading temporal correlations.
To address this problem, test another method based on time id alignment during attention operation. As illustrated in Figure 7 when performing attention between two scales, we map the time ID of tokens in the finer scale the other coarser scale before ROPE, ensuring that finer-scale tokens from the same temporal range share the same time ID as the corresponding coarse-scale token. Consequently, the resulting attention heatmap Figure eliminates the cross-scale bias caused by time ID, leading to more reasonable temporal correlations between cross-scale tokens.
fincio
ETTml
ETTm2
Weather
MSE
MAE
MSE
MAE
MSE
MAE
Seate
Scute
Naive
0.359
0.380
0.253
0.303
0.219
0.252
Aligned
0.356
0.378
0.250
0.302
0.222
0.254
Ours
0.353
0.377
0.250
0.301
0.216
0.248
Table 7: LSF results for MOIRAIsmal using three different attention strategies in MSFT
Figure 7: Map the token indices of finer scale to the coarser scale during cross-scale attention
Table ? presents the results of the three methods corresponding to the attention patterns in Figure 5 Compared to the Naive method, the time ID alignment approach improves performance on ETTml and ETTm2 but shows a performance decline on the Weather dataset. In contrast, our decoupled strategy consistently outperforms both methods. Apart from its inconsistent performance, another limitation of this time ID alignment method is that it is only applicable to Moirai, which employs RoPE for time IDD encoding. RoPE allows direct modification of token time IDs during attention, making this adjustment feasible. In contrast, for models using additive position encoding-whie
28

---

positional information is directly added to each token's 's imput embedding it is impossible to alter the time ID within the attention blocks. Our method, however, does not rely on modifying time IDs during attention. Instead, it achieves cross-scale alignment through aggregators, making it universally applicable to any model architecture.
Computation Efficiency We compare the memory footprint and training speed of our our methods with the following models: PatchTST 25 iTTraformer 19], TimesNet [44], and Scaleformer [34 For Scaleformer, we follow their original implementation and test on two backbones Autoformer and Informer [S]] referred to as Scaleformer-A A and Scaleformer-I respectively. For MSFT, we test its performance on MOTR small and MOIRAL A Base. referred to as MSFT-S and MSFT-B. To ensure fair comparison, we use a consistent batch size of 32 and context length of 512 across all models, with prediction length set to 96. To eliminate external interference, the experiments in this section exclusively conducted another server equipped with 12 vCPU Intel(R) Xeon(R) Platinum 8352V CPU 2.10GHz and a single RTX 3080 GPU with 20GB memory.
The results on ETTml and Weather datasets are shown in Table g The comparison shows that fine-tuning with MSFT on Moirai does not demand more computational resources than other models. Its GPU memory usage is lower than that of alternative methods. In terms of training speed, MSFT achieves moderate level among the compared methods. However, it significantly outperform Scaleformer, which is also a multi-scale modeling approach.
Table Quantitative comparison of computation efficiency across different methods.
Metric
Dataset
PatchTST
TimesNet
ITTrasformee
Scaleformer-A
Scaleformer-1
MSFT-S
MSFT-B
ETTml
63.92
334.67
27.97
307.36
180.09
103.53
185.35 33.38
Training Speed (ms/iter)
Weather
127.67
103.20
28,90
315.44
184.05
110.80
163.93
GPU Memory (MB)
ETTml
4198
2786
1952
11130
5104
808
1916
Weather
6866
2592
2110
11138
5106
808
1702
We also compare the computational and parameter efficiency of MSFT with several representative fine-tuning strategies, including full finetuning, linear probing, LoRA, and AdaLoRA. Results MOIRAlSmal with the Weather LSF task are summarized Table 9
Table 9: Quantitative comparison of computation efficiency across finetuning methods.
Method
Params (M)
GPU Mem (MB)
Train Speed (it/s)
Test Speed (it/s)
MSE/ / MAE
Full finetuning
13.8
2996
104.5
0.228 0.254
Linear probing
3,0
762
19.7
113.6
0,237 0,260
LoRA
3.4
2760
11.1
115.9
0.225 / 0.252
AdaLoRA
3.4
2756
10.0
102.4
0.226 / 0.252
MSFT (ours)
4.4
5616
4.6
102.5
0.216 0.248
As shown in Table MSFT uses fewer trainable parameters than full finetuning and achieves the best forecasting accuracy (lowest MSE/MAE). The cost higher GPU memory usage and slower training due the expanded token length introduced by multi-scale inputs. Nevertheless, its test-time speed remains comparable, offering a reasonable efffciency-peffommance performance trade-off compared to other fine-tuning methods.
More Ablation Study To assess robustness, conducted multiple runs (three seeds) for the ex- periments in Table Results in Table IT show that the standard deviations are generally insignificant across settings.
To further validate the role of in-scale masking. we conduct additional ablations where cross-scale scale aggregators are retained even without masking. Results in Table that this setting yields worse performance, as attention without in-scale masking learns misaligned cross-scale dependencies, and subsequent aggregation amplifies these inconsistencies. This confirms that cross-scale aggregators must operate jointly with in-seale masking to effectively fuse temporal correlations, while their standalone use significantly compromises performance.
29

---

Table 10: Ablation studies on in-scale masking, with mean + standard deviation over 3 runs.
Config
ETTml
ETTm2
Weather
MSE
MAE
MSE
MAE
MSE
MAE
0,360 + 0.003
0.380
0.002
0.253 + 0.000
0.303 + 0.000
0.220 + 0.000
0.253 + 0.001
++ C2F
0.364 + 0.004
0.383
0.003
0.256 + 0.002
0.307 + 0.002
0.224 + 0.000
0.256 + 0.000
+ F2C
0.365 + 0.003
0.384
= 0.002
0.253 0.000
0.303 0.001
0.225 = 0.001
0.255 = 0.001
3 Both
0.365 = 0.003
0.385
0.003
0.255
0.002
0.305 + 0.001
0.226 = 0.001
0.256 = 0.001
MSFT
0.354 0.003
0.378
0.002
0.250 0.000
0.301 + 0.000
0.216 + 0.000
0.248 + 0.000
Table 11: Ablation study three LSF datasets using MOIRAASmalI- Mean + standard deviation over
3 runs are reported. MSFT consistently outperforms all ablation variants.
Config
ETTml
ETTm2
Weather
MSE
MAE
MSE
MAE
MSE
MAE
!
0.362 + 0.003
0.380 = 0.002
0.253 + 0.001
0.305
0.000
0.219 = 0.000
0.252 + 0,000
0.360 = 0.002
0.379 = 0.002
0.252 = 0.000
0.304
0.000
0.218 = 0.000
0.249 + 0,000
0.374 = 0.004
0.385 = 0.003
0.256 - 0.001
0.308
0.001
0.224 = 0.002
0.256 + 0.001
4
0.361 0.002
0.382 0.002
0.254
0.306
0.000
0.222 0.001
0.001
3
0.371 0.003
0.384 0.002
0.256
0.000
0.307
0.001
0.223 0.001
0.255 0.001
R
0.363 0.003
0.382
0.002
0.254
0.000
0,304
0.000
0.220 + 0.000
0.252 + 0.000
7
0.357 0.002
0.379
0.001
0.252
0.000
0.304
0.000
0.218 + 0.000
0,251 + 0.000
8
0.360 0.003
0.380
0.002
0,253 0.000
0.303
0.000
0.220 + 0.001
0,253 + 0.001
0.359 0.003
0.379
0.003
0.269
0.003
0.313
0.002
0.226 - 0.003
0,252 1 0.002
10
0,384 + 0.007
0.388 = 0.004
0.255 0.003
0.311
0.002
0.219 - 0.001
0,252 1 0.000
MSFT
0.354 + 0.003
0.378 = 0.002
0.250 0.000
0.301 + 0.000
0.216 1 0.000
0.248 1 0.000
D.2
Evaluation of knowledge forgetting
To further investigate the potential issue of knowledge forgetting, we conduct a simple zero-shot transfer experiment following the setup of [4]] Specifically, we finetune the model on source dataset and directly evaluate it on an unseen target dataset B, denoted as + B. Table [2 reports MSE averaged over four prediction lengths for MOTRAAsmall- The results reveal no consistent pattern across transfer settings. some cases (e.g., ETTml > ETTm2), fine-tuning leads to improved zero- shot generalization, while in others (c.g., ETTm2 + ETTml) the performance degrades, suggesting that finetuning overrides certain pretrained knowledge. When comparing full finetuning and MSFT, neither method consistently outperforms the other, indicating that both approaches are not explicitly designed to mitigate catastrophic forgetting. Moreover, the multi-scale knowledge learned by MSFT from dataset A may not always generalize to dataset B if their temporal structures differ significantly.
Table 12: Cross-dataset transfer performance (MSE) on MOIRAASmall- Each entry reports the per- formance when the model is finetuned on dataset A and evaluated on an unseen dataset B (denoted B). The results are averaged over four prediction lengths. Best values are highlighted in bold.
Transfer
Zero-shot
Full FT
MSFT
ETTml +
ETTm2
0.300
0.293
0.288
ETTm2 ETTml
0.448
0.454
0.470
ETTml + ETTh1
0.416
0.410
0.420
ETTm2 ETTT1
0.416
0.415
0.414
ETTml + ETTh2
0.355
0.350
0.363
ETTm2 > ETTh2
0.355
0.359
0.350
From another perspective, MSFT is inherently more conservative in overwriting pretrained repre- sentations, due to its plug-in design. During finetuning, only lightweight adapters, normalization layers, and the output head are updated, while the majority of pretrained weights remain frozen. This design principle is consistent with common strategies in continual learning, where task-specific modules are introduced to reduce forgetting [2], Furthermore, if users wish to preserve the zero-shot
30

---

performance on unseen datasets after finetuning, one can simply deactivate or remove the MSFT modules, effectively reverting the model to its original pretrained TSFM with minimal performance change.
D.3 Full results
We report the full LSF results on four different prediction lengths, with MSE are shown in Table 13 and MAE are shown in Table [4 Results of deep learning-based baselines are obtained from Liu
et al. [9 and Chen et al. 4.
Table 13: Full MSE results of long sequence forecasting experiments.
ETimll
KTTm2
kribi
ETT
Ekancas
Waiter
192
306
733
136
133
355
123
206
120
136
731
3.55
137
6:445
01.00
1.39
1.454
0.775
0:41
8.355
8.422
0.41
6.683
1.00
11251
0.195
0199)
0.396
077
0.25
0:344
8:55
0.:30
01:81
0.38)
13434
0.77
0.27
0.354
0391
6339
BAZ9
11.8
11411
11.35
11066
4.22
8.355
0.88
0:31
8:39
Finl
0.30
0.305
1.50
0.425
0.774
3.36
2.31
2.365
0.82
BAS
1.99%
01134
0.315
0.342
115
0.91
B311
Liwaryminng
0.307
00.05
08900
0.45
BA1
BAS
0,80
0130
0.21
09200
0:34
B315
0.79
8.554
0360
01776
01.01
0,574
0.4%
0.441
0.254
BAIS
BAS
0.40
0.99
0.345
0.13
0.155
8:34
5355
0,29
0.53
9.371
9.80
DAB
6A5
0,76
0.72
0.995
1250
06.03
0.996
3.311
0.436
0.391
0.34
8.311
BA6
BANK
1.24
11113
11.30
0.39
0.22
0.355
9.381
9:36
0.53
0,890
0.3%
0.411
0.358
0.34
0.39
6354
BAD
1023
01.00
0.000
0.56
0.221
0.30
0.155
8245
B334
0.111
0,54
0.37
0414
0.377
0.30
0.39
0347
6315
0411
0168
0.000
0.77
0.210
13131
0.155
0,384
0.317
8.35
0.312
8.396
8367
8.355
0.39
0352
04439
0.386
0030
0.779
0.40
0.130
0.159
0.576
0.345
0.344
0.354
0329
6356
MONEST
3,365
0,12
2.37
BAI
0,99
0,38
0,550
0.29
6399
BA25
I1544
0,35
0.229
0.30
0.247
0:31
B442
Phogsraniyy
0.396
0413
0.76
0.29
0329
038
BAI
0.39
0.29
0.72
0.99
0.35
0.29
0,302
0.33
0.20
0,35
0.225
0.442
8.55
(44)
0.802
0.38
0.34
0416
0.71
0.30
0.21
0360
0385
B434
6,25
FLTS
0.38
0.563
0.50
0.996
0.71
0.995
0.300
0.34
0.611
6441
0.236
6.39
0.318
0,771
3.97
0.72
0.55
3355
1356
8435
0390
BASS
ILST
0-21
0,42
0162
0.77
0.39
0.26
175
0111
8:31
1331
0.300
0.79
3.99
0336
0.164
0.99
0356
8444
6382
B433
0306
0/04
0,46
01.9
0179
0.90
0331
199
0312
8.29
536
0.356
8.36
8.6
9.61
8.79
836
8313
8.85
635
6/63
0
0.00
03355
0.116
0.43
0.54
0.69
0.86
0.37
8.104
0:34
8:31
6342
Table 14: Full MAE results of long sequence forecasting experiments.
Miticc:
ETibal
KTTxa
KTT12
Flemicas
Weather
190
336
730
96
302
335
133
96
162
335
735
36
102
336
725
15
19C
336
130
96
302
336
T33
0,372
0,300
0.413
0.455
0.32
6.427
6:122
0.00
B.32
DJ29
D315
0,37
0,541
0,57
0.32
0.255
0.311
0.33
0.35
8.335
02311
0.43
0J66
0483
0,46
0337
0,508
0.391
3,40
0.499
8.39
6467
BA36
0J60
0,63
0.30
0.353
0.99
8114
8.36
B349
0387
0.411
0.56
8340
8.36
B400
B431
BJ2S
0482
0341
0
6414
0.44
0.47
0.36
0.77
0310
8.39
0.5)
0.37
0.300
8.39
6341
0,34
0.772
3.30
0.49
0.31
2.32
8.329
8.90
B400
BA5S
(40)
0331
0.6G
0406
0342
0.35
0.99
0.19
8199
0336
8.71
B317
Preesrasing
0.35
0.300
0.43
0.43
0.35
0.30
0.345
8.409
0462
B433
8435
(.61
0311
0.94
0,80
0366
0.6
0.37
0.31
0.39
6341
8.%
6317
0331
0.311
3.39
0.435
0355
0.34
0.33
0.353
6389
8-33
D435
0334
0313
0.39
0.06
0364
0332
0385
0.31
6119
0335
8371
6315
0.41
0.21
330
0.39
0:388
B437
0429
0380
0,66
0,42
0.33
0337
3.39
0.32
B338
0.334
0.361
3.30
0.5
0337
0.377
3.344
0.45
0.35
0335
6415
0371
0401
4.67
036
035
3.301
0335
6317
01306
0354
0.33
01377
0.30%
0.17
877
0.37
B436
0.73
0-67
0:31
0334
0.33
0.7
0.35
0.56
8.568
D/24
0367
0.34
510
8.300
8.77%
8.991
8366
8.885
9.30
16311
0:91
03382
0.306
0.361
8.361
8:01
MINEET
03355
0.370
3385
3.1
03305
0.330
8.33
6.35
B466
B445
D/60
8355
0331
0/09
0:35
0.76
0.56
0
0.5
0.3%
0.360
6156
09.31
07700
8116
0.360
0.74
135
8440
8-05
B-34
D/84
0-09
0.78
0.76
0.300
0.314
0.31
0.79
0.385
8.338
6161
MYV
18385
0.566
8.385
8.112
8.39
8.346
8.334
8:301
8.484
TUB1
0.274
18.90
8.111
0.99
0.35
0.366
8:334
6.41
01511
D:T8
D:85
0.580
0.76
11.01
111
6-114
6-68
6,23
0:56
0-31
0-251
6114
0.11
0.311
11:91
01:11
0.5
B427
1U86
0.000
0.286
0777
8.38
0350
0.00
8.8
948
1199
8111
MARE
1003
1099
0.384
1136
6389
In addition, for PF, apart from the two metrics we listed in the main text, we demonstrate the results
of four additional PF evaluatiin metrics in Table 15 The baseline results are obtained from Woo et al.
D.4 Forecast Visualizations
We visualize the forecasting predictions of MSFT using MOTRATsmal on ETTml and ETTm2, with the models finetuned on the predict-96 setup. In addition to the point forecast, which is the median of the samples, the 0.5 and 0.9 quantiles are also plotted for illustration. Only part of the context series is included in the plots.
31

---

Table 15: Full results for probabilistic forecasting experiments.
Mackal
Elvennity
ND
11320
10
Weather
N0
NO
116.5
Tucko Prenc
NO
PHEMSE
01115
1.835
(138)
0/214
1.30
[333
03.211
01.7%
1131
0.061
01:26
(1:3.0
01613
1:30
01531
0.031
1.355
0181
11
0.60
0333
01995
0.38
1.901
[127
018455
1.40
01900
0:344
03:972
0.340
01:057
0:133
11855
11/90)
0.416
1334
0.001
01156
0.02
0.35
0.31
0.51
01533
1862
0.822
0.396
0.214
(8.38)
0.635
0.185
0.399
0.15
0124
0351
0/8
0940
1.40
1.43
18834
1223
00800
0.31
0
01223
01.300
0.990
0.324
01264
0.390
2048
0181
01:00
0134
0.39
0330
0941
1.304
13355
0..00
1800
08883
0.533
0.383
0.271
08:23
03330
0.390
0.364
0.397
0.000
0.166
6123
0.31
0079
6460
1.301
1.300
08:93
0063
6617
0.33
0.845
0.327
0.807
0.300
0.302
0333
8357
0.555
0.331
NFSFF
0099
0.889
0,21
1.50
13303
08.875
0056
00905
0.41
01345
0119
01.315
03335
112
0.350
0,277
0236
1151
0,53
0.90
0110
3.16
0196
0.31
0.53
01.39
0.54
0.05
0:11
0.865
2160
04381
0133
0.18
0.33
0.77
03.15
1.44
0.11
0.000
0.13
0.99
0983
140
1385
0.56
1883
0813
0.684
0.36
0.38
8313
0.33
0170
0.37
0366
0383
0.30
0.168
0.79
0888
056
1400
186
0.59
1810
6613
8.656
0.34
0/117
0.31
0360
0.3
0.9
0.70
0351
0.00
target
target
target
prod
proz
pred
prest
prect 0.5
pred: 0.5
predi
target
target
target
pred
prec
pred
predi:
pradi 0.5
prod: 0.5
pred:
preci 0.9
pred: 0.9
Figure 8: Visualization ETTml (predict-96)
-22
-1.8
tanget
tarqet Fred
tar get pred pred: D.5
-1.
pred
prech 0.5
prid: 0.5
prodi 0.9
predi 0.9
pred: 0.9
M
-1.4
4/
-1.8
predi 0.9
predi 0.9
pred: 0.9
Figure 9: Visualization on ETTm2 (predict-96)
E Limitation and Future Work
As indicated in experiments. MSFT consistently delivers outstanding finetuning results on encoder-based TSFMs, validating the effectiveness of incorporation of multi-scale modeling into TSFM finetuning. However, a natural question that one may be curious about is how to apply our multi-scale finctuning method to TSFMs with other structures, such as decoder-based models.
Here, we first clarify why we focus solely on encoder- TSFMs in this paper. First, encoder-bar based models are more flexible to prediction length, making them more efficient finetune on standard LSF datasets. In contrast, decoder-based models, due to their auto-regressive nature, are significantly slower when finetuning and predicting on long time series. Although some decoder-based models provide finetuning examples, they are often applied to limited datasets without following the standard LSF pipeline. For example, TimesFM are only finetuned on subset of ETTm dataset for predict- 96 setup. This limitation hinders comprehensive comparisons between our methods and existing LSF baselines or other fine-tuning approaches. Secondly, decoder-based models inherently employ causal masking in their attention mechanisms, which imposes specific dependency structure. As
32

---

pioneering study, we choose to use encoder-based models without such constraints, providing greater flexibility and generality.
Despite the aforementioned challenges, we provide a potential direction for applying MSFT to decoder-based models. Due to their auto-regressive nature, the causal attention mechanism in decoder-only models can only attend to preceding tokens in the sequence, rather than all tokens simultaneously. Therefore, the creation of multi-scale embedding sequence needs take the order of scales into account. Similar to Scalerformer [34, we arrange the scales in a coarse-to-fine order and sequentially using coarse information to refine the fine-graines predictions subsequent levels. First, we concatenate the multi-scale input embeddings as hu Concat(h* - hg), ensuring the scales are in a coarse-tor coatse-to-fine to-fine order. Then, for the attention, we keep using the in-scale masking on the original causal masking. ensuring that the tokens can only attend to the previous tokens from the same scale. Regarding cross- scase-scale aggregators, the original dual-branch design cannot be directly applied due to the auto-regressive nature. Instead, we adopt single coarse-to-fine branch to fuse the token-level information. The multi-scale mixing remains unchanged, enabling the aggregation of predictions across different scales. We leave the further exploration of this direction as a future work
Another potential limitation is that multi-scale modeling increases the number of input tokens due to the introduction of new scales, Given the transformer's O(N²) complexity with respect to input sequence length, this inevitably increases the computational cost. On the other hand, finetuning with new scales can exceed the upper bound of fine-tuning performance achieved on single scale. Consequently, a trade-off exists between computational cost and performance. Another future direction is to further investigate this trade-off and develop more efficient strategy to achieve an optimal balance.
33


Reviews:
[{"review_id": "GgYmq6lfgY", "reviewer": "Reviewer_7Ko1", "summary": "This paper focuses on the task of fine-tuning time series foundation model. To address the issues of overfitting and suboptimal performance of naive fine-tuning methods, the paper develops a diversified fine-tuning approach based on different sampling scales. Through extensive experiments, the paper emphasizes the importance of explicitly modeling multiple scales and highlight the shortcomings of simpler methods.", "questions": "\tIt is recommended to provide an experiment to demonstrate how the new method better addresses the overfitting issue, such as by showing the trend of the loss curves.\n\tIt is suggested to add a set of zero-shot experiments to compare the predictive ability of the model before and after fine-tuning on unseen datasets.", "limitations": "Some", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\tThe paper addresses a highly relevant topic on effectively fine-tuning time series foundation models, aligning well with practical needs and significance.\n\tThe paper is well-completed, using comprehensive main experiments, analysis, and ablation studies to clearly demonstrate the rationale and utility of each component.\n\tThe writing is logically structured and clearly articulated.\n\tThe paper takes a novel approach by introducing in-scale and cross-scale plugins to address issues like time-index confusion in multi-scale joint modeling, which is an interesting concept.\nWeakness:\n\tThe paper mentions that naive fine-tuning strategies can lead to overfitting; however, it does not provide comparative experiments to support this claim.\n\tAnother critical aspect of fine-tuning tasks is preventing knowledge forgetting, and this work appears to lack analysis on this issue.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and encouraging comments. We are glad that the reviewer finds the topic of fine-tuning time series foundation models (TSFMs) to be timely and practically significant. We also appreciate the recognition of our efforts in presenting a well-completed manuscript with comprehensive experiments, as well as the acknowledgment of the clarity in our writing and the logical structure of the paper. We are especially encouraged that the reviewer considers our proposed method to be novel, well-designed, and reasonable.\n\nWe also sincerely appreciate the valuable questions and suggestions raised. We now provide detailed responses to the two specific weaknesses (W) and questions (Q) raised by the reviewer.\n\n**W1&Q1: Request for empirical validation of overfitting and how MSFT mitigates it**\n\nWe thank the reviewer for the valuable suggestion.  To further illustrate the overfitting phenomenon, we use Moirai-Small to compare Full Finetuning, Linear Probing (LP), LoRA, and MSFT on two datasets: ETTm1 and Weather. Since figures are not allowed in the rebuttal, we present key training NLL loss and validation MSE loss values in a table format at three critical checkpoints:\n* **Epoch1**: after training for one epoch\n* **Epoch***: the epoch when early stopping is triggered\n* **Last**: three epochs after epoch* (i.e., the patience of early stopping)\n\nIn our context, overfitting is reflected not only by the upward trend of validation loss after a certain point, but also by the model's tendency to overfit to a single scale, resulting in suboptimal validation performance. Therefore, we analyze both the loss trend after Epoch* and the validation loss value at Epoch*. The results are summarized below:\n\n\n|                | ETTm1 (Epoch1) | ETTm1 (Epoch*) | ETTm1 (Last) | │ | WTH (Epoch1) | WTH (Epoch*) | WTH (Last) |\n|:--------------:|:------------:|:------------:|:----------:|:-:|:-------------:|:------------:|:----------:|\n| Full           |   0.557      |    0.438     |   0.412    | │ |   -0.644      |   -0.758      |  -1.080    |\n| LP             |   0.524      |    0.497     |   0.471    | │ |   -0.366      |   -0.445      |  -0.446    |\n| Lora           |   0.630      |    0.448     |   0.438    | │ |   -0.373      |   -0.851      |  -0.873    |\n| MSFT           |   0.614      |    0.467     |   0.458    | │ |   -0.144      |   -0.615      |  -0.633    |\n\n*Table 1. Train NLL loss on ETTm1 and Weather (WTH) at different checkpoints*\n\n\n\n|  | ETTm1 (Epoch1) | ETTm1 (Epoch*) | ETTm1 (Last) | │ | WTH (Epoch1) | WTH (Epoch*) | WTH (Last) |\n|:------------:|:------------:|:------------:|:----------:|:-:|:-----------:|:----------:|:--------:|\n| Full         |   0.4000     |    0.385     |   0.391    | │ |   0.398     |   0.394     |  0.408   |\n| LP           |   0.444      |    0.408     |   0.408    | │ |   0.421     |   0.402     |  0.403   |\n| Lora         |   0.431      |    0.391     |   0.393    | │ |   0.414     |   0.394     |  0.396   |\n| MSFT         |   0.428      |    0.384     |   0.384    | │ |   0.398     |   0.366     |  0.366   |\n\n*Table 2. Validation MSE loss on ETTm1 and Weather (WTH) at different checkpoints*\n\nWe observe the following patterns:\n\n* **Training loss**: All methods show a consistent decrease in training loss throughout training. Note that the training losses are negative log-likelihood (NLL) values based on predictive distributions, and can therefore be negative. Also, since MSFT uses a weighted sum of losses on multiple scales, its training loss is not directly comparable to single-scale baselines.\n\n* **Validation loss**: We monitor the validation MSE loss for clearer comparison across methods, as it aligns with the evaluation metric used on the test set.\n  \n  - Among the single-scale baselines, Full Finetuning achieves the lowest validation loss at Epoch*, but the loss quickly increases afterward, forming a clear V-shaped curve, indicating that full finetune is prone to overfitting.\n\n  - Compared to Full Finetuning, Linear Probing and LoRA exhibit more stable loss after Epoch*, with only mild increases. However, due to limited trainable parameters, LP shows noticeably higher loss than both Full Finetuning and LoRA. The performance of LoRA is generally comparable to Full Finetuning in terms of loss magnitude.\n\n  - In contrast, MSFT not only maintains stable loss after Epoch*, but also achieves lower validation loss than the single-scale Full Finetuning, indicating that MSFT mitigates overfitting to a single scale and achieves better performance by using potential temporal resolutions.\n\nWe will **include the full loss curves in the revised manuscript** for better visualization.\n\n**W2&Q2: Request for evaluation of knowledge forgetting via zero-shot transfer experiments**\n\nWe thank the reviewer for raising this interesting question. We agree that our current manuscript does not include experiments specifically targeting knowledge forgetting.\n\nFirst, we would like to clarify that the primary focus of this work is on fine-tuning for a single downstream task, aiming to fully activate the pretrained knowledge of a TSFM on a specific dataset.  Given this focus, the continual learning paradigm was not considered necessary in our main experimental design.\n\nHowever, we agree that preventing knowledge forgetting is an intriguing direction. To explore the adaptation ability of MSFT, we follow the setup of TimeMixer++ [1] and conduct a simple cross-dataset evaluation using Moirai-small. We finetune the model on dataset A and evaluate it on the unseen dataset B, denoted as A → B. The MSE results averaged over four prediction lengths are summarized below:\n\n\n\n|    | Zero-shot | Full FT | MSFT |\n|:------------|:---------:|:-------:|:----:|\n| ETTm1 → ETTm2     | 0.300     | 0.293   | **0.288** |\n| ETTm2 → ETTm1     | **0.448** | 0.454   | 0.470     |\n| ETTm1 → ETTh1     | 0.416     | **0.410** | 0.420   |\n| ETTm2 → ETTh1     | 0.416     | 0.415   | **0.414** |\n| ETTm1 → ETTh2     | 0.355     | **0.350** | 0.363   |\n| ETTm2 → ETTh2     | 0.355     | 0.359   | **0.350** |\n\n*Table 3. Transfer performance (MSE) across datasets. Best values are in bold.*\n\nThe results in the table indicate that there is no consistent pattern across all transfer settings. In some cases, knowledge forgetting occurs. The performance of the model on the unseen dataset B after fine-tuning on dataset A is worse than its original zero-shot performance, suggesting that fine-tuning overrides the pretrained knowledge. In other cases, we observe positive knowledge transfer. The model fine-tuned on dataset A achieves better zero-shot performance on dataset B compared to the original pretrained model, indicating that certain knowledge acquired from dataset A generalizes and improves forecasting on dataset B. When comparing Full FT and MSFT, there is no clear winner overall. Both methods do not explicitly address catastrophic forgetting. Moreover, the multi-scale knowledge learned by MSFT from dataset A may not generalize well to dataset B, especially if the temporal structures differ significantly.\n\nFrom another perspective, **MSFT is fundamentally effective in preserving pretrained knowledge, due to its plug-in design**. MSFT introduces lightweight adapters to learn new tasks, while **keeping the vast majority of pretrained weights frozen** during fine-tuning. Among the pretrained components, only the normalization layers and the output projection head are updated. This design aligns with a common strategy in continual learning that adopts task-specific architectural modules to avoid catastrophic forgetting [2].\n\nTherefore, if users wish to retain the zero-shot performance or the pretrained knowledge on unseen datasets after fine-tuning, they can simply **deactivate or remove all MSFT-related modules**, including multi-scale modeling and aggregation mechanisms. In this case, the model reverts to its original single-scale version, with performance closely resembling that of the pretrained TSFM without fine-tuning.\n\n[1] TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysis, ICLR 2025\n\n[2] A Comprehensive Survey of Continual Learning: Theory, Method and Application, TPAMI 2024"}, {"review_id": "5u6YAzBi33", "reviewer": "Reviewer_oFpM", "summary": "The paper addresses the problem of fine-tuning a foundation\ntime series model to a specific dataset, esp. for models that \nare based on attention between patches. The authors \npropose \n-  i) to use adapter layers after the input encoding,\n- ii) LoRA modules in the attention layers, and\n- iii) let the foundation model process patches at different\n    downsampled resolutions, alternating self attention\n    only between patches at different times, but the same resolution, \n\tand between patches at different resolutions, but the same time.\n\nIn experiments on three different time series foundation models\nand the usual forecasting tasks, the authors show that their\nmethod consistently outperforms other fine-tuning methods\nsuch as full fine-tuning, fine-tuning only the last layer, LoRA etc.", "questions": "- q1. Do you rely on a multi-resolution / patch-length property of \n  the underlying foundation models? Or do you wrap a multi-resolution\n  model around each foundation model, basically feeding it with\n  patches in different resolutions?\n\n- q2. Could you also report standard deviations from different runs?\n  Which results are significant?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strong points:\n- s1. interesting, timely aspect: fine-tuning time series foundation\n  models.\n- s2. well engineered, plausible solution: adapter layers, LoRA modules\n- s3. novel aspect: feed multi-resolution patches to the time series\n  foundation models.\n- s4. ample experiments with very good results, improving the state\n  of the art.\n\nweak points:\n- w1. the formal description of the method is vague.\n- w2. the causal modeling motivation is unclear.\n- w3. experiments do not report standard deviations.\n\n\nad w1. the formal description of the method is vague.\n- the method is mostly explained in text and with a big diagram (fig. 3).\n  a concise formalization is missing. esp. for the multi-resolution\n  aspect I am not sure I understood exactly what the authors\n  propose.\n- exact pseudocode, e.g., in the appendix, might help a lot.\n\nad w2. the causal modeling motivation is unclear.\n- I do not understand the causal modeling motivation. for example,\n  what does \"X <- S [...] the input context series X is directly influenced\n  by the scale S\"?  If X and S denote random variables here, what \n  are their domains?\n\nad w3. experiments do not report standard deviations.\n- It is not clear how many runs the authors have conduted.\n- As they do not report standard deviations between runs,\n  it is not clear which of the differences might be significant.\n  \n\na few typos and grammar issues:\n- line 196 \"share the same resolution --- dependencies\"\n- line 201 \"andfine-to-coarse\"", "was_revised": true, "final_justification": "--- added after the rebuttal\n- the authors have answered my three issues well.\n- it would be good if the material of their answers could be added to the paper, or at\n  least its appendix.\n- I stay with my anyway positive rating.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and positive evaluation of our work. We are glad that the reviewer finds the topic of fine-tuning TSFMs both timely and practically significant. We also appreciate the recognition of our paper as well-completed, with comprehensive experiments and clear, well-structured writing. We are especially encouraged that the reviewer considers our method to be novel, well-designed, and reasonable, and acknowledges the strength of our empirical results.\n\nWe are especially thankful for the valuable comments and questions raised. Below, we provide detailed responses to the weaknesses (W) and questions (Q) raised by the reviewer.\n\n**W1: The formal description of the method is vague. Adding pseudocode would improve clarity.**\n\nWe appreciate the valuable suggestions from the reviewers. We agree adding such a pseudocode can greatly improve the clarity. While we experimented with LaTeX-style pseudocode in the OpenReview system, we found it cumbersome and quite difficult to read. Therefore, as a temporary solution, we decided to show our pseudocodes in a Python-like version to convey the core ideas more clearly. \n\n* Overall Training Pipeline for MSFT:\n\n```\n\"\"\"\nInput / Hyper-parameters:\n    X, Y                    # input context and prediction windows\n    K                       # number of scales\n    s = 2                   # downsample factor\n    P                       # patch size\n    L                       # number of MSFT layers\n\"\"\"\n\n# Step 1: Multi-Scale Generation\nfor i = 0 to K:\n    X_i = AvgPool(X, window_size = s^i)       # pre-pad X if len(X) % P ≠ 0\n    Y_i = AvgPool(Y, window_size = s^i)       # post-pad Y if len(Y) % P ≠ 0\n    S_i = (X_i, Y_i)\n\n# Step 2: Patching and Scale-specific Input Projection\nH_0 = []\nfor i = 0 to K:\n    x_i = Patching(S_i, patch_size = P)\n    h_i = Linear_i(InProject(x_i))            # frozen InProject\n    h_i = Masking(h_i)                        # mask prediction tokens\n    H_0.append(h_i)\n\nh_0 = Concat(H_0)                              # concatenate all scales\nscale_index = GetScaleIndex(H_0)              # index range for each scale in h_0\n\n# Step 3: Multi-Scale Attention Encoding\nh_l = h_0\nfor l = 1 to L:\n    h_l = AttnBlock_with_MSFT(h_l, scale_index)\n\n# Step 4: Output Projection and Loss Computation\nH_L = Split(h_l, scale_index)                 # recover [h_0^L, ..., h_K^L]\nlosses = []\nfor i = 0 to K:\n    Y_hat_i = OutProject(H_L[i])\n    L_i = Loss(Y_i, Y_hat_i)\n    losses.append(w_i * L_i)                  # weighted by learnable w_i\n\nL_total = sum(losses)\n```\n\n* Multi-Scale Attention Block (AttnBlock_with_MSFT):\n\n```\n\"\"\"\nInput:\n    h_in                                  # concatenated multi-scale token embeddings\n    scale_index = [idx_0, ..., idx_K]     # index ranges for each scale\n    s                                     # downsampling factor (e.g., s = 2)\n\nInitialization / Parameters:\n    F2CMap = [Linear_0→1, ..., Linear_{K-1}→K]   # fine-to-coarse projections\n    C2FMap = [Linear_1→0, ..., Linear_K→{K-1}]   # coarse-to-fine projections\n\"\"\"\n\n# Step 1: Split input into scale-wise representations\nH_in = [h_in[..., idx, :] for idx in scale_index]     # H_in = [h_0, h_1, ..., h_K]\n\n# Step 2: Scale-specific Attention with LoRA\nQ, K, V = [], [], []\nfor i = 0 to K:\n    Q_i = W_Q_i(H_in[i])\n    K_i = W_K_i(H_in[i])\n    V_i = W_V_i(H_in[i])\n    Q.append(Q_i)\n    K.append(K_i)\n    V.append(V_i)\n\n# Step 3: In-scale masked attention\nH_attn = ScaledDotProductAttention(Q, K, V, attn_mask = M_in)  \n\n# Step 4: Cross-scale Aggregation\n\n# (a) Coarse-to-Fine (C2F); Eq 3\nH_c2f = H_attn.copy()\nfor i = K to 1:\n    h_proj = C2FMap[i-1](H_attn[i])\n    H_c2f[i-1] = H_c2f[i-1] + Repeat(h_proj, repeat_factor = s)\n\n# (b) Fine-to-Coarse (F2C);Eq 4\nH_f2c = H_attn.copy()\nfor i = 0 to K - 1:\n    h_proj = F2CMap[i](H_attn[i])\n    H_f2c[i+1] = H_f2c[i+1] + AvgPool(h_proj, pool_size = s)\n\n# (c) Merge embeddings from two branches\nH_out = []\nfor i = 0 to K:\n    H_out.append((H_f2c[i] + H_c2f[i]) / 2)\n\n# Step 5: Re-concatenate and project\nh_out = Concat(H_out)\nh_out = W_out(h_out)\n\n# (Add & Norm → FeedForward → Add & Norm omitted for brevity)\n\nReturn: h_out\n```\n\nWe will include a more formal, academic-style algorithm description in the revised manuscript.\n\n**W2:  The causal modeling motivation is unclear. Why Input context series X is directly influenced by the scale S? Are they random variables? If yes, what are their domains?**\n\nWe thank the reviewer for the insightful question. We take this opportunity to clarify our causal modeling motivation and discuss our understanding of time series from a data generation perspective.\n\nFundamentally, a time series is a discretized sequence of sampled observations derived from an underlying continuous process. Under this view, the input context window $X$ is effectively a discrete observation (in the form of time series) from the true process that occurred during the context period. The variable $X$ arises from two factors: the latent continuous process $I$ (which we do not observe), and the scale parameter $S$, which governs the sampling resolution. $S$ determines how densely the underlying process is sampled, thus shaping both the temporal granularity and the length of the observed sequence $X$. While our manuscript omits the latent variable $I$ for tractability, the edge $S \\rightarrow X$ in our causal graph is intended to capture this observation mechanism. Note that scale influences the form of the observed input time series, though not the underlying process itself.\n\nBuilding on the above discussion, we answer the following questions. Yes, both $S$ and $X$ are treated as random variables. We define the scale variable $S$ as a discrete random variable that selects a resolution level to sample the input (in practice, it is downsampling the original given context time series). $S$ takes values in a finite index set, i.e., $S \\in \\mathcal{S} =$ { $s_0, s_1, \\dots, s_K $ }, where each scale level $s_k$ corresponds to a specific downsampling factor or temporal resolution. \n\nWe define $X$ as a random variable that represents the observed input time series segment corresponding to the context period, whose length depends on the scale variable $S$. Formally, we have $X \\in \\mathcal{X}$, where $\\mathcal{X} = \\bigcup_{s \\in \\mathcal{S}} \\mathbb{R}^{L_s}$. Here, $L_s$ is the input sequence length corresponding to scale $s \\in \\mathcal{S}$.\n\nWe also analyze the causal model in the response of Q1, Reviewer 1tu4. If the reviewer has remaining concerns, we would be happy to further discuss.\n\n**W3 & Q2: Experiments do not report standard deviations.**\n\nWe appreciate the reviewer for raising this point. During the rebuttal period, we have started running multiple seeds for key configurations. As an initial step, we report below the multi-run results of Full FT and MSFT on Moirai-small. For each experiment, we run 3 times with different random seeds and report the average MAE and MSE over four prediction lengths. Most results align with our original results and the standard deviations are generally insignificant across settings. \n\n| Method   | ETTm1          | ETTm2          | ETTh1          | ETTh2          | Electricity     | Weather         |\n|----------|----------------|----------------|----------------|----------------|------------------|------------------|\n| Full FT  | 0.369 ± 0.004  | 0.274 ± 0.004 | 0.416 ± 0.002  | 0.352 ± 0.002  | 0.194 ± 0.004    | 0.229 ± 0.005    |\n| MSFT     | 0.354 ± 0.003  | 0.250 ± 0.000  | 0.412 ± 0.001  | 0.349 ± 0.001  | 0.187 ± 0.001    | 0.216 ± 0.000    |\n\n*Table 1. Multi-run MSE results*\n\n| Method   | ETTm1          | ETTm2          | ETTh1          | ETTh2          | Electricity     | Weather         |\n|----------|----------------|----------------|----------------|----------------|------------------|------------------|\n| Full FT  | 0.383 ± 0.003  | 0.317 ± 0.002  | 0.429 ± 0.001  | 0.379 ± 0.002  | 0.279 ± 0.001    | 0.255 ± 0.003   |\n| MSFT     | 0.378 ± 0.002  | 0.301 ± 0.000  | 0.426 ± 0.001  | 0.375 ± 0.001  | 0.275 ± 0.001    | 0.248 ± 0.000    |\n\n*Table 2. Multi-run MAE results*\n\nAs we acknowledged in the checklist, we did not perform multiple runs in the main submission due to computational constraints, similar to Time-MoE and Moment. For example, our LSF evaluation involves 4 backbones × 5 fine-tuning methods × 6 datasets × 4 prediction lengths, totaling 480 experiments. Given this scale, running multiple seeds was not feasible during the rebuttal period. We plan to continue running additional runs for other important settings and will include the full results with standard deviations in a follow-up discussion and future version of the paper.\n\n**Q1: Do you rely on a multi-resolution / patch-length property of the TSFM? Or do you wrap a multi-resolution model around each TSFM?**\n\nWe do **not** rely on a multi-resolution or multi-patch-length property of the underlying TSFM. This is because not all TSFMs inherently possess such architectural designs. For instance, while Moirai adopts multiple patch sizes, other models like Moment and UniTS do not. Our goal is to build a general framework that can be applied across different encoder-based TSFMs, regardless of their specific architectures.\n\nTherefore, your second interpretation is more accurate:  our MSFT Framework effectively serves as a wrapper that transforms any TSFM into a multi-scale version by feeding it inputs at multiple temporal resolutions. Specifically, the patch size is fixed, but patches from different scales correspond to different temporal resolutions (see Figure 2b). In summary, what we rely on is not specific architectural properties; instead, we reply on the general pretrained knowledge that the TSFM has acquired, which we hypothesize contains rich multiscale temporal information.\n\n**Typos and grammar issues**\n\nThank you very much for pointing these out. We have corrected the typos in the manuscripts."}, {"review_id": "wTujJCtZRI", "reviewer": "Reviewer_pUUh", "summary": "The paper identifies a key limitation in the naive fine-tuning of Time Series Foundation Models (TSFMs), specifically their tendency to overfit to a single scale of the data. To address this, the authors propose the MultiScale Fine-Tuning (MSFT) framework, designed for encoder-only TSFMs such as MOIRAI, MOMENT, and UNITS. MSFT enhances performance by fine-tuning on multi-scale inputs. In addition to the core framework, the method incorporates multi-scale mixing, LoRA, and in-scale attention mechanisms into the modified TSFMs. Experimental results demonstrate substantial improvements across benchmarks in both long-sequence forecasting and probabilistic forecasting tasks.", "questions": "1. In lines 66–67, the authors claim to be the first to introduce multi-scale modeling in TSFMs. This appears to be inaccurate—see [1] and the Multi-Patch Size Projection layers in MOIRAI.\n\n2. The authors state in lines 155-157: “When directly finetuning the input projection layer over all scales, each scale inherently tends to learn its own specific intra-token patterns, which can lead to interference across scales and suboptimal performance.\" Wouldn’t this issue also affect the output projection layer? If not, please clarify why.\n\n3. Authors propose in-scale masking to address two challenges:\n\n        a. Lines 153-154: “First, the token schematics and intra-scale dependencies vary significantly across scales.”\n\n        b. Lines 160-164 “Second, standard self-attention introduces misleading cross-scale dependencies due to mismatched time (position) indices.”\n\n\t\nboth claims are supported only by a single illustrative example. Can the authors provide empirical validation or theoretical justification?\n\n4. Could the authors include extra ablation results in Table 3 showing the impact of removing the in-scale masking component?\n\n5. In Table 3, it would be helpful to visually mark the best and second-best results per column (e.g., boldface, underline) for easier comparison.", "limitations": "- The proposed method is limited to encoder-only TSFMs.\n\n- The framework introduces additional computational overhead during both training and inference.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper provides a clear identification of the limitations associated with naive finetuning of TSFMs.\n\n- The authors conduct an extensive evaluation, reporting strong empirical performance of their proposed framework.\n\n**Weaknesses:**\n- Several claims in the paper lack corresponding empirical support or ablation studies (see questions section for specifics).\n\n- The paper does not report standard deviations in Table 3, making it difficult to assess whether the results are statistically significant or consistent.\n\n- Section 5.2 omits citations to relevant recent work such as [1] and [2], which discuss multi-scale modeling in time series.\n\n- The comparison table is missing key baseline models discussed in Section 5.1, such as [3] and[4].\n\n[1] Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wesley M. Gifford, Chandra Reddy, Jayant Kalagnanam: “Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series.” NeurIPS 2024\n\n[2] Shengsheng Lin, Weiwei Lin, Xinyi Hu, Wentai Wu, Ruichao Mo, Haocheng Zhong: “CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patterns.” NeurIPS 2024\n\n[3] Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin: “Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts.” ICLR 2025\n\n[4] Ansari, Abdul Fatir and Stella, Lorenzo and Turkmen, Caner and Zhang, Xiyuan, and Mercado, Pedro and Shen, Huibin and Shchur, Oleksandr and Rangapuram, Syama Syndar and Pineda Arango, Sebastian and Kapoor, Shubham and Zschiegner, Jasper and Maddix, Danielle C. and Mahoney, Michael W. and Torkkola, Kari and Gordon Wilson, Andrew and Bohlke-Schneider, Michael and Wang, Yuyang: “Chronos: Learning the Language of Time Series.” Trans. Mach. Learn. Res. 2024\n\n[5] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, Jun Zhou: “TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting.” ICLR 2024", "was_revised": true, "final_justification": "The authors have addressed most of my concerns.", "rebuttal": "We sincerely thank the reviewer for the time and effort dedicated to reviewing our paper. Below, we provide detailed responses to the weaknesses (W) and questions (Q) raised by the reviewer.\n\n**W1-Q1:” The authors claim to be the first to introduce multi-scale modeling in TSFMs. This appears to be inaccurate—see TTM and MOIRAI.”**\n\nWe sincerely thank the reviewer for raising this important point. Although multi-scale modeling is a widely used forecasting strategy, we acknowledge that it lacks a universally agreed-upon definition in the time series literature. Furthermore, we recognize that our manuscript lacks a clear and explicit definition of multi-scale modeling, which may have led to ambiguity or misunderstanding when compared with similar concepts such as multi-resolution or multi-patch-size. We appreciate the opportunity to clarify this.\n\nIn our context, multi-scale modeling refers to **constructing multiple downsampled versions of a time series instance and jointly leveraging their representations through aggregation during the prediction process**. All works discussed in Section 5.2 follow this paradigm. Below, we clarify the subtle differences between our definition of multi-scale modeling and the related works mentioned by the reviewer:\n\n*TTM* introduces multi-resolution pretraining by downsampling high-frequency datasets into lower-frequency versions, which serves as a form of data augmentation during pretraining. Despite using a learnable resolution prefix and adaptive patching, each time series sample is still processed in a single scale/resolution during its prediction process. Thus, TTM does not perform multi-scale modeling as defined in our work.\n\n*Moirai* uses multiple patch size projection layers for input and output projection. However, it does not downsample the time series to multiple scales/resolutions. Each input series still uses only one patch size during prediction, and all tokens belong to a single temporal scale. While multi-patch-size designs can potentially be used for multi-scale modeling, Moirai itself does not do so. We discuss related works in Appendix A.2.\n\nTherefore, we believe our statement was not overstated. However, we fully understand the reviewer’s concern and are happy to revise the claim to a more precise version: **“the first to introduce multi-scale modeling in TSFM finetuning”**.  We will also revise Section 2 to provide a clearer and more explicit definition of multi-scale modeling.\n\n**W1-Q2: Wouldn’t the multi-scale issue affect the output projection layer? If not, please clarify why.**\n\nWe thank the reviewer for this insightful question. In fact, we also experimented with adapting the output head in Moirai into a scale-specific version. However, this led to no performance improvement and significantly increased the parameter count: the output head has a large number of parameters ( ~3M params in Moirai_small has a total of 13.8M), and its mixture distribution nature makes it difficult to apply lightweight adaptations such as adapters.\n\nWe posit that this difference arises because the input projection layer lies at the bottom of the model and directly operates on the raw time series, making it more sensitive to the input scale. As it is responsible for learning token-level representations, it plays a crucial role in shaping how subsequent attention and feedforward layers extract dependencies. In contrast, the output head sits at the top of the model and already captures relatively generalizable prediction patterns. As such, further specialization across scales at this stage may have limited effect.\n\n**W1-Q3: Request for empirical validation or theoretical justification regarding following claims**\n\n*(1) \"Token schematics vary significantly across scales\"*\n\nThe intuitive explanation is that tokens across scales correspond to different temporal ranges, leading to differences in their schematics. To quantitatively validate this, we compute the mean cosine similarity of token embeddings between different datasets and scales. Specifically, we sample training sequences from two datasets, ETTm1 (D1) and ETTm2 (D2). Using Moirai’s input projection, we extract embeddings from training sequences of ETTm1 (D1) and ETTm2 (D2) at the original scale (S0) and a downsampled scale (S1, by 2). We compute the mean cosine similarity between the token embeddings from different dataset-scale pairs\n\n| **Pair**              | **Mean Cosine Similarity** |\n|:---------------------:|:--------------------------:|\n| D1 S0 vs. D1 S1 |           0.0974           |\n| D1 S0 vs. D2 S0 |           0.1291           |\n| D2 S0 vs. D2 S1 |           0.1084           |\n| D1 S1 vs. D2 S1 |           0.1447           |\n\nThese results show that token embeddings from different scales within the same dataset are less similar than embeddings from the same scale across datasets. This suggests that scale differences even introduce more variation in token schematics than dataset differences, supporting our claim that token semantics vary significantly across scales.\n\n*(2) \"Intra-scale dependencies vary significantly across scales\"*\n\nThe intuitive explanation is that timesteps and tokens across scales correspond to different resolutions, leading to distinct temporal dependencies or dynamics at each scale. To quantitatively support this claim, we compute ACF-based features of the series across different data segments and scales, capturing their temporal dependencies. Specifically, we divide the target series of ETTm1 into two equal-length segments, denoted as D1 and D2. We then compute the ACF and first-order differenced ACF at the original scale (S0) and a downsampled scale (S1). Finally, we calculate the differences in ACF values between various pairs to measure the variation in temporal structures.\n\n| **Pair**             | **Δ ACF** | **Δ Diff1_ACF** |\n|:---------------------|:--------:|:---------------:|\n| D0 S0 vs. D1  S0     | 0.080    | 0.004           |\n| D0 S1 vs. D1  S1     | 0.100    | 0.017           |\n| D0 S0 vs. D0 S1      | 0.390    | 0.044           |\n| D1 S0 vs. D1 S1      | 0.410    | 0.031           |\n\nThese results show that scale changes introduce more significant variations in temporal dependencies than changes across time segments, supporting our claim that intra-scale dependencies vary significantly across scales.\n\n\n(3) “Standard self-attention introduces misleading cross-scale dependencies due to mismatched time indices.”\n\nThank you for the attention to this important point. In fact, we have provided empirical evidence in the manuscript to support this claim. Specifically, both Figure 2 and Section 6.3 visualize attention maps showing that, without in-scale masking, tokens from one scale attend to tokens at the same time index from another scale, even though they represent different actual time range. This results in spurious cross-scale attention that misaligns with true temporal correlations.\n\nWe believe these attention maps serve as strong empirical evidence to support our claim. However, if the reviewer finds this insufficient, we would greatly appreciate further suggestions on this. \n\n**W1-Q4: Could the authors include extra ablation results in Table 3 showing the impact of removing the in-scale masking component?**\n\nWe thank the reviewer for the suggestion. We would like to clarify that the impact of removing the in-scale masking component is already included in our current manuscript (Row #8 in Table 3). If the reviewer had a different variation or experimental setting in mind, we are happy to further clarify or provide additional results as needed.\n\n**W2: The paper does not report standard deviations in Table 3**\n\nWe appreciate the reviewer for pointing this out. As mentioned in our response to Reviewer oFpM (W3 & Q2), we have started running multiple seeds during the rebuttal stage. Preliminary results with standard deviation for several key settings in Table 1 have already been included in the rebuttal, and we will continue expanding them. Due to the large number of experiments and limited computational resources, we were unable to complete all configurations before the rebuttal deadline. We plan to update the results with standard deviations in the follow-up discussion stage, including Table 3 as noted by the reviewer.\n\n\n**W3: Section 5.2 omits citations to relevant recent work (TTM & CycleNet), which discuss multi-scale modeling in time series.**\n\nWe thank the reviewer for pointing out these important works. Upon revisiting them carefully, we believe this concern is closely related to the previous question W1-Q1 regarding the definition of multi-scale modeling. We have explained why TTM is not regarded as multi-scale modeling in W1-Q1. For CycleNet, it directly learns a globally shared cycle and forecasts by modeling the residuals after removing them, which does not involve multi-scale modeling.\n\nWe acknowledge that both papers represent valuable and timely contributions. We will cite them and add a paragraph in Related Work to explicitly clarify and differentiate these related notions.\n\n\n\n**W4: The comparison table is missing key baseline models (Time-MoE, Chronos)**\n\nWe thank the reviewer for raising this point. While the scope of our work is focused on encoder-based TSFMs, we appreciate your valuable feedback. To clarify your suggestion, are you recommending that we (a) include these models as additional baselines in our table (e.g., comparing their zero-shot performance), or (b) apply our MSFT framework to these backbones for a more comprehensive comparison? We would be happy to further investigate this issue in the following discussion stage.\n\n**Q5: Suggestion on Table 3.**\n\nWe appreciate the reviewer's suggestion. We have updated Table 3 to highlight the best and second-best results per column using boldface and underlining, respectively."}, {"review_id": "1d3KRhYS9f", "reviewer": "Reviewer_TU3Z", "summary": "This paper proposes MSFT, a novel multi-scale finetuning method for encoder-based Time Series Foundation Models (TSFMs). The authors argue that naive fine-tuning approaches overlook the multi-scale nature of both time series data and TSFMs, leading to suboptimal performance and potential overfitting. Through a causal analysis, they identify scale as a confounding factor between input data and model predictions, and propose MSFT framework, which generates multi-scale inputs via downsampling to block the confounding effect. To address the challenge of scale-varying token resolutions, MSFT employs scale-specific adapters, introduces a decoupled token dependency modeling mechanism, and aggregates predictions across scales using learnable weights. Experimental results on diverse datasets demonstrate that MSFT improves the performance of TSFMs compared with naive fine-tuning and other parameter-efficient fine-tuning methods.", "questions": "1. Why is the performance gain of MSFT marginal on the ETTh datasets compared to the ETTm datasets?\n\n2. How is the up-sampling process conducted during multi-scale prediction mixing?\n\n3. Are the models channel-independent?\n\n4. How should the number of scales and downsampling factors be selected to balance performance and computational cost? Would the method benefit by incorporating the prior knowledge of data frequency? For example, would down-sample the minute-level data to hourly data improve efficiency without sacrificing important patterns?\n\n5. Is the proposed method also applicable to TSFMs that do not use patch-based tokenization, such as Chronos [R1] and Time-MOE [R2]?\n\n6. Recent advances in time series foundation models primarily fall into two categories: LLM-based approaches [R3] and models trained from scratch [R4]. This paper adopts the latter strategy—fine-tuning models such as Moirai and UniTS. Could the authors comment on the feasibility and potential advantages or limitations of fine-tuning LLM-based time series models and pre-trained from scratch models? While additional experiments are not necessary, a brief discussion contrasting the two paradigms would be valuable for readers.\n\n[R1]  Chronos: Learning the language of time series. TMLR, 2024.\n\n[R2] Time-moe: Billion-scale time series foundation models with mixture of experts. ICLR 2025\n\n[R3] A survey of time series foundation models: Generalizing time series representation with large language model. arXiv, 2024.\n\n[R4] A survey on time-series pre-trained models. TKDE, 2024.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The experiments are comprehensive. The paper evaluates multiple encoder-based TSFMs on diverse datasets. In addition, it includes various single-scaled baselines for comparison. The experimental results demonstrate the effectiveness of their proposed method.\n\n2. The paper is easy to follow, with a coherent and well-organized  logical structure. Additionally, the research problem of adapting TSFMs to downstream tasks is both timely and essential. Also, multi-scale modeling of TSFMs represents a promising and unexplored direction. \n\n3. The proposed method is simple and easy to understand. Components are well-motivated and aligned with specific challenges. Decoupling within-scale and inter-scale dependencies is novel and reasonable.\n\nWeaknesses:\n\n1. The performance of the proposed method shows less performance improvement on ETTh datasets.\n\n2. Although the authors acknowledge this in the Limitation section, the proposed method requires downsampling the input time series into multiple resolutions and concatenating them, which increases the input sequence length and consequently raises the model's computational cost.\n\n3. Some details in Figure 3 are not clearly explained. For example, the tokens in the cross-scale aggregator are shown in different colors, but the meaning of these colors is not clarified in the text.", "was_revised": true, "final_justification": "Thank you for the authors' response, which resolved most of my concerns. I hold a positive view of this work and have decided to keep my overall rating as Accept, while slightly increasing the clarity score.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We appreciate the recognition of our comprehensive experimental design, including the evaluation across diverse datasets, multiple encoder-based TSFMs, and strong single-scale baselines. We are also grateful for the acknowledgment of the clarity and organization of our paper, and for identifying the problem of adapting TSFMs to downstream tasks as both timely and important. We are encouraged by the reviewer’s appreciation of the simplicity and motivation behind our method, as well as the novelty of decoupling within-scale and inter-scale dependencies.\n\nWe also greatly appreciate the reviewer’s thoughtful questions and constructive suggestions. Below, we provide detailed responses to the weaknesses (W) and questions (Q) raised by the reviewer.\n\n**W1 & Q1: Performance gain of MSFT is marginal on the ETTh datasets compared to the ETTm datasets, why?**\n\nThank you for the question. We believe there are two main reasons behind this. First, datasets with different original temporal resolutions inherently differ in their capacity to benefit from multiscale modeling. For minute-level data (e.g., ETTm), downsampling by factors like 2/4/8 yields hour-level sequences that still retain clear temporal structures and meaningful patterns. In contrast, for hour-level data (e.g., ETTh), such downsampling offers much less room to generate additional scales with strong periodicity or semantic interpretability.\n\nSecond, ETTh datasets are significantly smaller than the ETTm datasets. Since TSFMs typically require long input contexts for effective modeling, the limited data in ETTh makes the models more prone to overfitting, which can reduce the observable benefit from MSFT.\n\n**W2 & Q4: A limitation is the computational overhead from multi-scale input. Is there any guidance on selecting scales and downsampling factor to balance the trade-off?**\n\nThank you for this practical and insightful question. Incorporating prior knowledge about the raw data frequency into the choice of downsampling factors is indeed a natural and intuitive idea. While most multiscale modeling works (and our main paper) follow the common practice of using powers-of-two downsampling, we also experimented with frequency-informed downsampling strategies, as detailed in “Effect of Down-Sampling Rate”, Appendix C.1. Our results show that frequency-aware downsampling does not consistently bring advantages. It led to noticeable improvement only on ETTm2 (ds_factor = 4, i.e., 15T to 1H), while showing little to no benefit on ETTm1 (ds_factor = 4) and Weather (ds_factor = 6, i.e., 10T to 1H). \n\nWe recommend to use 2–3 scales with power-of-two downsampling factors. If GPU memory is limited and a slight drop in performance is acceptable, we suggest using a single scale and treating the downsampling factor as a tunable hyperparameter, including the frequency-informed factor as a candidate value.\n\n**W3: Some details in Figure 3 are not clearly explained, e.g. the meaning of the token's colors is not clarified in the text.** \n\nWe thank the reviewer for pointing this out. We will improve Figure 3 and provide additional textual descriptions for the cross-scale aggregator. The color gradient of tokens is intended to illustrate the aggregation across different scales, and we will clarify this in the caption and main text. Additionally, we will include the pseudocode of the aggregation algorithm to help readers better understand the implementation details.\n\n\n**Q2: How is the up-sampling process conducted during multi-scale prediction mixing?**\n\nWe experimented with both linear interpolation and repeat-based upsampling, and observed minimal differences in performance on the validation sets. By default, we adopt repeat-based upsampling for its simplicity, but users are free to choose linear interpolation if preferred.\n\n**Q3: Are the models channel-independent?**\n\nThe answer depends on the design of the underlying TSFM. Moment is channel-independent, while UniTS adopts variate-wise attention and thus is multivariate. For Moirai, whether to use multivariate modeling is a configurable option. In fact, many datasets in its official implementation adopt the channel-independent setting to achieve the best performance. In our work, we follow the same channel-independent setting for Moirai, including in the zero-shot scenario. This setting is also compatible with our multi-scale implementation, which currently treats different scales as distinct variates. We clarify the channel-independence property of each model in Appendix B.2.\n\n**Q5: Is the proposed method also applicable to TSFMs that do not use patch-based tokenization, such as Chronos and Time-MOE?**\n\nWe thank the reviewer for this insightful question. Yes, the proposed MSFT framework can be extended to time series foundation models (TSFMs) that do not rely on patch-based tokenization. Here we focus specifically on the tokenization scheme, and discuss how MSFT can be adapted accordingly.\n\nFor example, Time-MoE adopts a point-wise tokenization strategy, which can be treated as using a patch size of 1. Under this view, our multi-scale pipeline can be directly applied without any modification regarding the tokenization process.  For Chronos, the tokenization strategy involves quantizing real-valued time series into discrete tokens by scaling and binning. Each time step is represented by a discrete token, without applying any input projection layers. To adapt MSFT to this setting, we can remove the input projection component used in the original MSFT design and directly apply the same quantization-based tokenization to time series at multiple scales.\n\nSince the scope of this paper focuses on encoder-based TSFMs, and both Chronos and Time-MoE adopt different architectural designs, we did not include experiments on these models. Nonetheless, we discuss potential strategies for extending MSFT to non-encoder architectures in the Limitation and Future Work section. In general, our method is not restricted to patch-based tokenization; it can also be adapted to alternative tokenization schemes.\n\n**Q6: Could the authors comment on the feasibility and potential advantages or limitations of fine-tuning LLM-based time series models and pre-trained from scratch models?**\n\nWe thank the reviewer for this valuable open-ended question. Indeed, current advances in time series foundation models (TSFMs) largely fall into two paradigms: \n\nLLM-based TSFMs (e.g., GPT4TS [1], LLM4TS [2], Time-LLM [3]) typically adopt architectures or pretrained checkpoints from large language models. A central focus of these works lies in the embedding strategy, i.e., how to effectively transform time series into embeddings that are compatible with LLM input spaces. One potential advantage of this paradigm is its capacity to naturally incorporate rich contextual or meta-information of the downstream task, which can be helpful in fine-tuning. However, a key challenge is that LLMs are not inherently tailored to capture inductive biases commonly found in time series, such as autocorrelation, periodicity, or hierarchical temporal dependencies. While recent efforts have tried to adapt embeddings and fine-tuning strategies to better suit LLM architectures, it remains difficult to fully preserve the temporal continuity and fine-grained dynamics intrinsic to time series data. \n\nIn contrast, pretrained-from-scratch TSFMs (e.g., Moirai, UniTS, TimesFM) are designed and optimized specifically for time series data. These models are pretrained on large-scale univariate or multivariate time series datasets. The main focus is to capture universal temporal patterns or variate-wise correlation by leveraging architectural inductive biases like patching, variate-wise attention, or domain prefix, along with forecasting-aligned pretraining objectives. One clear advantage of this paradigm is that it aligns well with the unique structure of time series data, which helps the model learn fine-grained temporal dynamics more effectively and generalize better across datasets with different sampling rates or sequence lengths. These models also tend to be more lightweight and efficient compared to LLM-based approaches, making them easier to train and deploy in practical settings. On the other hand, they are typically designed for pure time series input, which makes it harder to naturally incorporate external context or metadata, such as text descriptions or user intent. \n\nConsidering the respective strengths and limitations of both paradigms, our proposed MSFT framework selects pre-trained-from-scratch models to leverage their general pretrained knowledge on temporal patterns, especially across multiple scales. We would like to investigate applying multi-scale modeling to LLM-based TSFMs as a future direction.\n\n[1] One Fits All: Power General Time Series Analysis by Pretrained LM, NeuriPS 2023\n\n[2] LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecast, ACM TIST 2025\n\n[3] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models, ICLR 2024"}, {"review_id": "oRvixMtI85", "reviewer": "Reviewer_1tu4", "summary": "This paper addresses the limitation that current fine-tuning methods for Time Series Foundation Models (TSFMs) often fail to fully exploit the multi-scale characteristics inherent to TSFMs. To this end, the authors propose **MultiScale FineTuning (MSFT)**, a method that captures multi-scale representations of downstream datasets via downsampling. MSFT adopts LoRA to fine-tune the attention computation modules, and explicitly models both intra-scale and cross-scale dependencies. For cross-scale interactions, the method integrates both coarse-to-fine and fine-to-coarse pathways. The final prediction is obtained through a weighted fusion across different scales. The proposed MSFT achieves state-of-the-art fine-tuning performance across multiple TSFMs. Additionally, the authors provide a causal perspective to explain the potential impact of scale on the final prediction outcomes and present corresponding high-level formalizations.", "questions": "1. In the causal model proposed in Section 3, the paper lacks empirical validation of the causal graph or model. The current explanation appears more intuitive than analytical. It would be valuable if the authors could design experiments to demonstrate that the scale variable (S) indeed acts as a confounder between the input (X) and the model knowledge (M).\n2. In line 190, the paper states: “introduce a set of LoRA modules for each scale.” However, at this stage, the inputs from multiple scales are concatenated. It is unclear how LoRA modules are introduced separately for each scale under this setup. The authors are encouraged to provide more implementation details.\n3. It would be helpful if the authors could provide a comparison of MSFT with other fine-tuning methods in terms of parameter count and computational cost across different TSFMs.\n4. It is suggested to split Table 3 into three separate tables to present the ablation results more clearly. The current table presents conditions in a complex and unintuitive manner.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The overall writing and logical flow of the paper are well-structured, with clear explanations of the proposed method and experiments.\n2. The paper introduces a novel fine-tuning approach for TSFMs that explicitly accounts for the multi-scale nature of time series data.\n3. The proposed MSFT method achieves superior fine-tuning performance across multiple TSFMs.\n4. The authors provide a theoretical interpretation of MSFT from a causal perspective using causal graphs.\n\nWeaknesses:\n\n1. The current fine-tuning method, MSFT, is designed specifically for encoder-based models, without offering strategies for other types of TSFMs.\n2. The paper does not compare the computational or parameter efficiency of MSFT with other fine-tuning methods.", "was_revised": true, "final_justification": "Most of my concerns have been addressed.", "rebuttal": "We thank the reviewer for the thoughtful and detailed review. We are glad that the reviewer found the writing and logical flow of our paper to be clear and well-structured, and that the proposed method and experiments were easy to follow. We also appreciate the recognition of our novel fine-tuning approach that explicitly leverages the multi-scale nature of time series data. We are particularly encouraged by the reviewer’s acknowledgment of MSFT’s strong empirical performance across multiple TSFMs and the value of our theoretical interpretation from a causal perspective using causal graphs.\n\nWe also sincerely thank the reviewer for raising thoughtful questions and suggestions. Below, we provide detailed responses to the weaknesses (W) and questions (Q) raised by the reviewer.\n\n**W1: The current fine-tuning method, MSFT, is designed specifically for encoder-based models, without offering strategies for other types of TSFMs.**\n\nWe thank the reviewer for pointing this out. We acknowledge that the current framework design and experimental validation of MSFT are based on encoder-based TSFMs. We choose encoder-based models because they are more efficient to fine-tune than architectures with decoder, making them more suitable for multi-epoch fine-tuning in the popular forecasting benchmark (e.g. LSF datasets). We clearly stated the scope in the title and the paper.  However, the core idea of MSFT is general and can potentially be extended to other types of TSFMs with appropriate adaptations based on their tokenization schemes or attention mechanisms. We have discussed some preliminary strategies in the “Limitation and Future Work” section, and we consider applying MSFT to non-encoder architectures as an important direction for future research.\n\n**W2 & Q3: Request to compare the computational or parameter efficiency of MSFT with other fine-tuning methods.**\n\nThank you for the valuable comment. We did not include comparisons across other finetuning methods in the manuscripts because our design explicitly introduces multi-scale modeling, which naturally leads to increased input token length and thus extra memory and computational overhead. Therefore, we instead compared with popular deep-learning-based methods and a few baselines that also incorporate multi-scale structure for a more meaningful evaluation.\n\nHowever, we fully agree with the reviewer that comparing different finetuning strategies on top of TSFMs is also valuable. As such, we conducted additional experiments below, comparing MSFT with full finetuning, linear probing, LoRA, and AdaLoRA, all under the same settings. \n\n\nMethod          | Params (M) | GPU Mem (MB) | Train Speed (it/s) | Test Speed (it/s) |  MSE   |  MAE\n----------------|------------|---------------|---------------------|--------------------|--------|-------\nFull finetuning |    13.8    |     2996      |        9.4         |       104.5        | 0.228  | 0.254\nLinear probing  |     3.0    |      762      |        19.7         |       113.6        | 0.237  | 0.260\nLoRA            |     3.4    |     2760      |         11.1         |       115.9        | 0.225  | 0.252\nAdaLoRA         |     3.4    |     2756      |        10.0         |       102.4        | 0.226  | 0.252\nMSFT (ours)     |     4.4    |     5616      |         4.6         |       102.5        | 0.216  | 0.248\n\n*Table 1: Efficiency analysis of Moirai-Small on Weather LSF task (Prediction length = 96, Batch size = 512)*\n\nAs shown, although MSFT uses fewer trainable parameters, it does consume more GPU memory and incurs slower training speed due to the expanded token length from multi-scale inputs. However, test-time speed remains comparable, and importantly, forecasting performance improves (lowest MSE/MAE), highlighting the benefit of our design.\nTo summarize, as we have also stated in the Limitation section, multi-scale modeling inherently introduces additional computational overhead due to increased input length and processing complexity. But compared to many deep learning baselines, MSFT still offers a reasonable efficiency–performance trade-off, making it practical and effective for real-world use.\nWe hope this clarifies our rationale and we’re happy to add these results into the appendix if helpful.\n\n**Q1: In the causal model proposed in Section 3, the paper lacks empirical validation of the causal graph or model. The current explanation appears more intuitive than analytical. It would be valuable if the authors could design experiments to demonstrate that the scale variable ($S$) indeed acts as a confounder between the input ($X$) and the model knowledge ($M$).**\n\nWe thank the reviewer for this insightful comment and valuable suggestion. To empirically validate the proposed causal graph in Section 3, we conduct causal structure learning and partial correlation analysis on ETTm1 dataset across four different scales.\n\nSpecifically, we extract context windows from the training split of ETTm1, and downsample each window into three additional resolutions to construct multiscale views. Each scaled input is then passed into the Moirai model to obtain the corresponding input embeddings, which reflect part of the model’s activated knowledge $M$. Since both the scaled input context and the input embeddings vary in length across scales, we construct fixed-length scalar proxy features to enable graph-based causal analysis. For each sample at each scale, we form a (S, X, M) triplet, where S is the scale index (0 to 3), X is the ACF computed on the sample series and M is the L2 norm of the mean input embedding.\n\nWe first apply the PC algorithm [1, 2] with Fisher’s Z test (α = 0.01) to the 3-variable dataset, with the goal of identifying whether directed causal edges exist from scale to the input signal and model representation. The inferred structure includes the directed edges scale → acf_sum and scale → emb_l2, supporting the assumption that scale causally influences both the input and the model representation.\n\nTo further assess whether scale acts as a confounder between $X$ and $M$, we compare their raw correlation with the partial correlation conditioned on scale. The Pearson correlation between acf_sum and emb_l2 is −0.732. However, after conditioning on scale, the partial correlation reduces to −0.481 (p < 0.001). This reduction in correlation magnitude suggests that scale partially explains the statistical dependency between input and model representation, which is consistent with its role as a confounder in the proposed causal graph.\n\nTogether, these two complementary analyses quantitatively support the role of scale as a confounder between input and model representation, consistent with the causal assumptions proposed in Section 3.\n\n[1] An Algorithm for Fast Recovery of Sparse Causal Graphs, 1991\n\n[2] Causal-learn: Causal discovery in python, JMLR 2024\n\n**Q2: How LoRA modules are introduced separately for each scale on the concatenated multi-scale embeddings? The authors are encouraged to provide more implementation details.**\n\nThank you for the detailed question. Although inputs from multiple scales are concatenated before being fed into the attention layers, we keep track of the token length corresponding to each scale. During the attention computation, we slice the concatenated embeddings to extract the token groups associated with each scale and apply the corresponding LoRA modules accordingly. This allows us to introduce scale-specific LoRA modules even under the concatenated input setup. We also provide a pseudocode in the response to Reviewer oFpM, which the reviewer may refer to for further implementation details.\n\n\n\n**Q4: It is suggested to split Table 3 into three separate tables to present the ablation results more clearly.**\n\nThank you for the helpful feedback. Due to page limitations, we kept Table 3 in a compact format for the submission version. We will split it into three separate tables in the future version to improve clarity."}]

Author rebuttal:

Meta-review: The paper argues that current fine-tuning techniques for time series (TS) foundation models are suboptimal because they fail to exploit the inherent multi-scale nature of the data and the ability of foundation models to capture dynamics at multiple temporal resolutions. The authors propose a causal analysis of fine-tuning for TS forecasting and identify scale as a confounding factor that may induce spurious correlations in the trained model. To address this, they generate multi-scale inputs through downsampling. To effectively integrate information across scales, they introduce scale-specific adapters and decouple token dependencies by employing distinct attention mechanisms for within-scale and cross-scale information. Experiments are conducted on both deterministic and probabilistic forecasting tasks.


The reviewers agree on the relevance of the problem, the novelty of the multi-scale integration approach for fine-tuning pretrained foundation models, and the breadth of the experimental evaluation supporting the method. Their initial reviews, however, noted weaknesses, including an imprecise technical description, limited support for the causal analysis, and insufficient experimental evidence for certain claims. In their rebuttal, the authors provided extensive clarifications and additions, including new experiments (comparisons with baselines, ablations, and analyses), pseudo-code, and further explanations. The reviewers consider that most of their concerns have been addressed and hold a positive view of the paper. They encourage the authors to carefully incorporate the additional material into the main text and the appendix.
Decision: Accept (poster)