Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?
Authors: Xu Zhang
Peng Wang
Wei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Time series analysis
large models
fine-tuning
Primary area: learning on time series and dynamical systems
TL;DR: We find that large models may suffer from performance limitations during fine-tuning due to overfitting in pre-training. We propose to smooth the loss landscape then fine-tuning to improve the fine-tuning performance.

Abstract: Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models. However, prior studies indicate that pre-trained LTSMs may exhibit a poorly conditioned non-convex loss landscape, leading to limited trainability. As a result, direct fine-tuning tends to cause overfitting and suboptimal performance, sometimes even worse than training from scratch, substantially diminishing the benefits of pre-training.
To overcome this limitation, we propose Smoothed Full Fine-tuning (SFF), a novel fine-tuning technology. Specifically, we construct an auxiliary LTSM via random initialization to obtain a smoother loss landscape, and then linearly interpolate its weights with those of the pre-trained model to smooth the original landscape. This process improves trainability while preserving pre-trained knowledge, thereby enabling more effective downstream fine-tuning. From an optimization perspective, SFF perturbs sharp minima without significantly harming flat regions, facilitating escape from poor local basins toward smoother and more generalizable solutions. Extensive experiments on benchmark datasets demonstrate consistent improvements across eight representative LTSMs, including Timer, TimesFM, MOMENT, UniTS, MOIRAI, Chronos, TTMs, and Sundial, on diverse downstream tasks. The code is available at the link: \url{https://github.com/Meteor-Stars/SFF}.

Published as a conference paper at ICLR 2026
LOST IN THE NON-CONVEX LOSS LANDSCAPE: How TO FINE-TUNE THE LARGE TIME SERIES MODEL?
Xu Zhang, Peng Wang, Wei Wang Shanghai Key Laboratory of Data Science College of Computer Science and Artificial Intelligence Fudan University, Shanghai 200082, China Xuzhang22@m. fudan. edu. (pengwang5, weiwangl | ) fudan. edu. cn
ABSTRACT
Recently, large time series models (LTSMs)) have gained increasing attention due to their similarities to large language models, including flexible context length, scala- bility, and task generality, outperforming advanced task-specific models. However, prior studies indicate that pre-trained LTSMs may exhibit poorly conditioned non- convex loss landscape, leading to limited trainability. As a result, direct fine-tuning tends to cause overfitting and suboptimal performance, sometimes even worse than training from scratch, substantially diminishing the benefits of pre-traie vre-training. To overcome this limitation, we propose Smoothed Full Fine-tuning (SFF), a novel fine-tuning technology. Specifically,
construct auxiliary LTSSM via random initialization to obtain a smoother loss landscape, and then linearly interpolate its weights with those of the pre-trained model to smooth the original landscape. This process improves trainability while preserving pre-trained knowledge, thereby enabling more effective downstream fine-tuning. From an optimization perspective, SFF perturbs sharp minima without significantly harming flat regions, facilitating escape from poor local basins toward smoother and more generalizable solutions. Extensive experiments benchmark datasets demonstrate consistent improve- ments across eight representative LTSMs, including Timer, TimesFM, MOMENT, UnITS, MOIRAL, Chronos, TTMs, and Sundial, on diverse downstream tasks. The code is available at the link: https:: / wig.ttttub...... www.hh...oooooeooccc wtw.//teo///-e//e/ Starsrsstttrarrrr
INTRODUCTION
Time series (TS) are ubiquitous in the real world, such as financial trading (Zhang al. 2025a 2024 Sen & Mehtab 2022), industry sensor monitoring (Zhang et al. (2025)), and traffic systems (Gao et al. 2024a)). TS analysis can help inform better decisions by revealing underlying trends and behaviors. Recently, large models developed via generative pre-training transformers (GPT) have demonstrated advanced capabilities, including flexible context length, strong cross-domain generalization, versatility across diverse tasks, and scalability with increasing model size and pre-training data. Inspired by this success, large time series models (LTSMs). c.g., Timer (Liu et al. 2024). TimesFM (Das et al. 22024). and MOMENT (Goswami al.) 2024), have been proposed transfer these advantages to time series analysis, achieving improved performance forecasting (Box et al. 2015), interpolation (Friedman 1962), and anomaly detection (Ren al.) 2019). Pre-trained large-scale time series datasets, LTSMs effectively capture universal temporal patterns, such as trends, amplitudes, frequencies, and phases (Goswami et al. 2024), thereby substantially benefiting downstream tasks.
However, recent theoretical studies suggest that large-scale training may drive models toward sharp minima (Keskar et al.) 2016) in highly non-convex loss landscape (Li et al. 2018). leading to optimization challenges during fine-tuning. Empirically. Figure visualizes the loss landscape of a pre-trained LTSM on downstream datasets, revealing pronounced local protrusions (highlighted by the black contour), consistent with theoretical findings. Such steep and non-convex (non-smooth) landscapes indicate poor trainability and may trap models in suboptimal local minima (e.g.. orange
"Corresponding author

---

Published as a conference paper at ICLR 2026
loca
lacal
Original pre-trained LTSM
(b) Random Initialization
Utilizing smouth
(d) Placing (h) and semi- transparent (a) in the same coordinate system
(e) Placing (b) and semi- transparint (c) in the same coordinate system
Figure 1: Loss landscape comparisons based the LTSM Timer and exchange rate dataset. Smoother is better. The unsmooth loss landscape shows a non-convex structure and indicates poorer trainabil- ity (Li al. 2018 e.g., Figure [a). More cases are shown in Appendix Figure 7 and Figure X
arrows in Figure ! a)), thereby exacerbating overfitting and degrading generalization (Li et al. 2018). Furthermore, we observe that direct full fine-tuning consistently achieves the lowest training loss but even higher test loss than training from scratch, as shown in Appendix Figure further confirming severe overfitting and supporting our analysis.
Intuitively, the non-convex loss landscape may arise from overfitting during pre-training, such as learn- ing data-specific patterns rather than generalizable features or converging sharp minima (Keskar et al. 2016). Consequently, existing fine-tuning strategies, including Full Fine-tuning (FF), Linear Probing (LP), and Linear Probing then Full Fine-tuning (LPFF) (Kumar et al.), often fail to achieve satisfactory performance, they cannot effectively smooth the non-convex landscape. This limitation hinders the pre- pre-trained trained LTSM from converging to better local minima during fine-tuma tine-tuning (c.g., the dark blue region in Figure [a)).
To mitigate the adverse impact of such a "non-convex loss landscape" on downstream fine-tuning, we empirically observe that randomly initialized LTSMs generally exhibit much smoother landscapes, shown in Figure ()). This observation naturally raises the question: Can we leverage the smooth landscape in Figure (b)) to smooth and reshape the loss landscape the pre-trained model in Figure Day) (). thereby enhancing its trainability while preserving its pre-trained knowledge?
Based on these insights, we propose Smoothed Full fine-tuning (SFF) better exploit the pre-traue pre-trained knowledge of LTSMs for improved fine-tuning performance. The method consists of two key steps. First, we construct an auxiliary LTSM via random initialization. Unlike the pre-trained model, this auxiliary one exhibits smoother and more convex loss landscape (Figure Tb)), leading to better trainability. However, lacks pre-traine preitrained knowledge, as shown Figure dd) where its minimum loss is significantly higher than that of the pre-trained model, explaining the latter's strong zero-shot performance. Second, we smooth the pre-trained model's loss landscape by linearly interpolating its weights with those of the auxiliary LTSM. The resulting smoothed LTSM effectively preserves the pre- rre-trained knowledge while inheriting the improved trainability of the auxiliary model. As shown in Figure e)), ee), its minimum loss remains substantially lower than that of the randomly initialized model, confirming effective knowledge preservation. Empirical evidence provided in Section 5.4
Overall, smoothing alleviates severe protrusions in the pre-trained loss landscape (Figure [c)), yielding more convex structure and facilitating optimization. As a result, gradient descent is more likely converge better local optima (indicated by the dark blue arrows), thereby enhancing fine-tuning stability and effectiveness. The proposed technology requires only linear interpolation of model parameters prior fine-tuning, without introducing additional memory or computational overhead. From an optimization perspective, SFF perturbs sharp minima while largely preserving flat regions, enabling escape from suboptimal basins toward smoother and more generalizable solutions. Further theoretical analysis is provided in Section 3.1
In summary, our contributions are as follows:
We uncover key observation that pretrained LTSMs may suffer from overfitting during pre- training, leading to poorly conditioned non-conver loss landscape and reduced trainability, which substantially limits their fine-tuning performance on downstream tasks.
We are the first to propose the Smoothed Full Fine-tuning (SFF). SFF smooths the loss landscape via linear interpolation between pretrained model (with rich knowledge but poor trainability) and randomly
basee fine-tuning technology. termed
2

---

Published as a conference paper at ICLR 2026
initialized one (with limited knowledge but good trainability), yielding smoothed model that preserves pretrained knowledge while achieving improved trainability, We further pro- vide theoretical insights into its effectiveness from optimization perspective (Section 3.1
Notably, SFF introduces no additional memory or computational overhead.
Extensive experiments on time series forecasting (TSF) and anomaly detection tasks demon- strate that SFF consistently outperforms popular fine-tuning strategies, including Full Fine- tuning (FF), Linear Probing (LP), Linear Probing then Full Fine-tuning (LPFF), and other popular optimization techniques. SFF improves the performance of eight representative LTSMs across diverse architectures (encoder-only, decoder-only, encoder sendder-eecder, and MIPP-only) and model scales (3.8GB to 3MB).
2
RELATED WORKS
Popular fine-tuning approaches include full fine-tuning, linear probing, and linear probing followed by full fine-tuning (LP-FF) (Kumar al.). Due limited space, we place related work on fine-tuning. optimization strategies, and time series foundation models in Appendix A.2
Difference from existing weight interpolation methods. To the best of our knowledge, weight interpolation has not been explored for fine-tuning from the perspective of loss landscape theory. Although weight averaging and interpolation (Vlaar Frankle, 2022) have been studied in model merging (Wortsman et al. 2022) and continual learning (Kozal et al. (202)) they do not address the core challenge in LTSMs. Our method differs fundamentally the following aspects:
(1) Different goals. Existing methods (Wortsman et al., 2022 Kozal et al. 2024) mainly focus on model ensembling, i.e., interpolating among multiple well-trained models to improve generalization or alleviate catastrophic forgetting. In contrast, we exploit interpolation to smooth the loss landscape of a single pretrained model using a randomly initialized model, thereby improving its trainability for fine-tuning.
(2) Different pipelines. Prior works typically apply the interpolated model directly to downstream tasks without further training. In contrast, perform additional fine-tuning after interpolation, enabling the smoothed loss landscape to yield better optimization and performance.
(3) New theoretical analysis. We formalize the contrast between the smooth loss landscape of randomly initialized models and the steep, irregular landscape of pretrained ones, and provide theoretical analysis explaining why interpolation can effectively exploit this difference enhance fine-tuning.
SMOOTHING THE LOSS LANDSCAPE OF THE PRE-TRAINED LTSM FOR FINE-TUNING
We propose the smoothed fine-tuning technology to enhance the fine-tuning performance of various pretrained LTSMs, as illustrated in Figure 2
3.1 MOTIVATION AND THEORETICAL ANALYSIS
In this section, explain why Smoothed Full Fine-tuning (SFF) effective from the perspective of deep learning optimization theory. Specifically, the loss landscape can be divided into flat and sharp regions Li et al. (2018). Flat regions imply better generalization, as the model is more tolerant to weight perturbations within these regions Keskar et al. 2) Hochreiter & Schmidhuber (1997); Foret et al. (2020). This motivates SFF to perform linear interpolation between randomly initialized weights and pretrained weights to obtain smoother loss landscape and improved fine- tuning performance. Such interpolation does not significantly alter the weights located in flat regions due to their high tolerance perturbations. For weights in sharp regions, SFF behaves similarly to introducing momentum (analogous to the momentum mechanism in the Adam Kingma & Ba (2014) optimizer) via random weight interpolation, enabling them to escape sharp regions and move toward smoother basins, thereby facilitating more stable and effective fine-tuning.
Next, we provide more rigorous theoretical derivation to prove the effectiveness of SFF.
3

---

Published as a conference paper at ICLR 2026
Lasze (LM)
Model ridlomily inttialzed from uniform cistribution
targeted
M
Elcanatr
Finetuning on target dataset using smootlied modkl
Model pretrained from series dataset
(b) Good earves struture of d'aucillaty model's loss handscape (with poor pre-rnined knowledge but good trainabiliny)
Auciliary model weights
Pretrained moddl weights
Smoothing land scape pretrained midde through interpolation of model weigtts utiliinng poetrained knowlelge achievin trsindsilitty
Good of smoothed model's loss landscape (with both good pre- triined knewledge and good trainabilitt))
(a) Poer convex stnicnare of pretrained model's hadscape good pre- trsinol knowlolge but poor (usissbilitt)
Figure 2: Smoothed Full Fine-tuning (SFF). By linearly interpolating the pretrained and randomly initialized LTSMs, we obtain smoothed model that preserves pretrained knowledge while exhibiting a smoother and more trainable loss landscape, thereby enabling more stable and effective fine one-tuning.
3.1.1 INFLUENCE OF INTERPOLATION ON SHARP AND FLAT LOSS LANDSCAPE
Given an MSE loss function ((O). where è denotes the model parameters, the minimizer @* can correspond to either sharp or flat minimum. Inspired by (Keskar al.. 2016), we characterize the sharpness and flatness of the loss landscape using the Hessian matrix. Specifically, by analyzing V²L(e>) (H Rddd where d denotes the
the maximum eigenvalue Amax(-) (-) of the Hessian parameter dimension), we define sharp and flat minima as follows:
Theorem (Sharp minima). The Hessian (²)( exhibits large eigenvalues (i.e., Amar ( (()))) > where > O is a threshold), indicating steep loss landscape in which small parameter perturbations can cause substantial increases loss.
Theorem 2 (Flat minima). The Hessian V²L(e") has small eigenvalues (i.e., Amaae (())))) T where T 0), indicating a flat loss landscape in which the loss is robust to parameter perturbations.
Proof details for the above theorems are provided Appendix Section A.1 Next, we analyze the impact of interpolation the loss landscape in both sharp and flat regions.
Smoothing (perturbing) sharp minima. The SFF interpolation defines the smoothed parameters as θ3 af; + (1 a 22, where O denotes the pretrained LTSM, which is more likely to lie in sharp minimum (Figure Ta)) due to large-scale pretraining (Keskar et al. 2016). In contrast, O (randomly initialized) lies in flat region, illustrated in Figure (b.. In the next section, we provide theoretical analysis showing that commonly used initialization schemes, including Kaiming and Xavier, inherently produce flat loss landscapes. After pre-training the LTSM, the sharp minimum O is characterized by large Hessian eigenvalue, i.c., Amax (=((())) > т. In contrast, the randomly initialized point O lies in flat region, yielding Amax (V2 L(O))) ST.
Under a local quadratic approximation of the loss function along the interpolation path, the Hessian Og can be approximated as a convex combination of the Hessians at Oj and Og:
A LL(( = a (())) + - (2)
Consequently, the maximum eigenvalue satisfies:
Amax (V²((e)) S aa (V²C(01)) + (1-a) ( (V2(()2)
(2)
Since Aemux. 2 (22(02)) & Amax (V2 ((i() (flat vs. sharp), follows that (V² < Amax. 2 ((O))) for a E (0. 1). This indicates that interpolation effectively reduces local sharpness, enabling parameters sharp regions to escape unfavorable basins and move toward smoother ones, thereby improving fine-tuning. These provide theoretical support for the smoothing effect of SFF.
Preservation Flat Regions. After pre-training the LTSM, flat minimum points i interpolation preserves their flatness. Specifically, both θ and O₂ reside in flat regions of the loss landscape, which can be formulated as:
Amax (V² S Amax (Ve ST
4

---

Published as a conference paper at ICLR 2026
Similarly, the Hessian at the interpolated point O₂ =afi + (1 a) satisfies V2(((s)) ~ a (Q)( + a) (2 Consequently, it follows that:
Amax (VL X amax (V2) + (1 - a1-aa)aua Amxx (V2)(02)) - aT + (1-a)== =
(4)
Hence. 3 remains within a flat region. As noted in (Foret et al. 2020), this implies that interpolation does not compromise the pre-existing flat minima Gj.
Overall, from a rigorous mathematical perspective, interpolation (or perturbs) smooths sharp minima while preserving the integrity of originally flat regions, improving fine-tuning performance.
3.1.2 PARAMETER INITIALIZATION AND THE SMOOTHNESS THE CORRESPONDING Loss LANDSCAPE
Visualizations indicate that initialization methods such as Kaiming (He et al. 2015 and Xavier (Glorot Bengio 2010) yield smooth loss landscape. Motivated by this, we adopt them as auxiliary initialization schemes for LTSM to facilitate smoothed full finetuning. In this section, we further analyze this phenomenon from a theoretical perspective.
Prior work (Fort& & Scherlis 2019) quantifies loss landscape smoothness under Kaiming and Xavier initializations via the ratio of the Hessian trace Tr(H) to its Frobenius norm [H|].. demonstrating TH(H) > 1, both theoretically and experimentally. That is, mainstream initialization indeed produces smooth, flat, and more easily optimizable loss landscape. We provide a further theoretical analysis of this conclusion. Specifically, exploiting the symmetry of H, we expand the formula as:
that THI
Tr(H)) H
Tr(H) VELL h2
Tr(H)
Σ VE
>
(5)
VTT(HTI)
This implies that the sum of eigenvalues far exceeds the square root of the sum of squared eigenvalues, suggesting that most eigenvalues are positive and relatively evenly distributed, rather than dominated by extreme outliers. In other words, most cases satisfy X X(V2L((*)) S with 0.
According to the definitions in Theorem 1 and Theorem 2, this indicates that the loss surface exhibits smooth, valley-like geometry dominated by positive curvature. Consequently, most random descent directions remain stable and low-curvature, facilitating more stable and efficient optimization.
|| S 1, it indicates a roughly balanced mix of positive and negative eigenvalues A, resulting in steep and irregular loss landscape. Consequently, convergence slows, and the optimizer is more prone to suboptimal local minima. Therefore, Kaiming or Xavier initialization provides a stable and smooth loss landscape, and we adopt them for randomly initializing the auxiliary LTSM.
In contrast, if (T(H)
3.2 SMOOTHING THE LOSS LANDSCAPE THEN FINE-TUNING
We define the training set D {(11, Yi)...., (Xx, YN)}. where each time series Xx [21, æ2,...... 37l E RRXx has length t across e time variables. The pre-trained LSTM is divided into backbone G(X, Φ2) and a linear head W Whedl E Rdxh, where t denotes the backbone parameters, d and h are the output dimensions of the backbone and head.
Similarly, the parameters of the auxiliary LSTM are defined O₂ 0 Wheaa)-- Given coefficient o, the new model weights O₂ = ( W Whents] are obtained via linear interpolation between the auxiliary LSTM (92) and the pre-trained LSTM (O1). Accordingly, the formulations for full fine-tuning, linear probing, and the loss function are given in Eq. 6 Eq. 7 and Eq. g respectively.
f(X, θ2) G(X, ab + (1- 2 (a W head1 + (1 a W Wnead2)
(6)
f(x, 0₂) = G(V a1 + (1-a) a. Pruzen (a W beadt + (1 aaW W Wbea22)
(7)
arg a0 +1
min
Σ C (F((X, + O2))), Y.)
(8)
(X.Y))D
where a is the interpolation coefficient controlling the proportion of pretrained knowledge retained, with larger preserving more of it. L denotes the Mean Squared Error (MSH) loss. Y; depends
5

---

Published as a conference paper at ICLR 2026
the downstream task, e.g., for imputation, Y, X, represents the masked values; for forecasting, Y, denotes the consecutive future values of X;
Smoothing the loss landscape be implemented in just few lines of PyTorch and the example code is provided in Appendix Algorithm
EXPERIMENTAL SETTINGS
LTSM Baselines. We use the eight encoder-only (Moirai (Woo et al.
popular LTSMs as baselines with diverse architectures, including
2024a) and MOMENT (Goosamii et al. 2024)). decoder-only (Liu et al. 2024). and TimesFM (Das et al., (0244)) encoder- 2024) and UniTs (Gao et al. 2024b)), and light-weight MLP 2024)). They also include models of different sizes, ranging from larger TimesFM (3.8GB) to smaller TTMs (3MB). We verify that our method can enhance their performance. We download these pre-trained models from the official links for experiments, e.g., with model sizes being 85 851MB, 2.6GB, and 3,8GB for Timer, MOMENT, and TimesFM, respectively.
(Sundial (Liu et al., 2025), Timer
decoder (Chronos (Ansari et al. model (TTMs (Ekambaram et al.
Fine-tuning baselines. addition to comparing with typical fine-tuning baselines, e-g., full fine- tuning (FF), linear probing (LP), and linear probing first and then full fine-tuning (LP-FF). We also incorporated various optimization strategies employed during model training. such as label smoothing. SAM (Foret et al. 2020). SWA (Izmailov et al. 20188) Mixout (Lee et al.), and L2-SP (Xuhong et al., 2018). Comparison results are shown in the Appendix Table due to limited space. More details about datasets, evaluations, and implementation details are shown in Appendix A.4
5 EXPERIMENTAL RESULTS
We conduct extensive experiments to validate the effectiveness of the proposed smoothed fine-tuning tunintning including TSF datasets and 250 anomaly detection datasets, also involving 8 popular LTSMs. To ensure the effectiveness of our method is not random occurrence, we have conducted experiments with multiple random seeds under varying available data proportions. Our method also achieves improvements on the imputation task, as shown in Appendix Figure 10 due to limited space.
We also evaluate different initialization schemes and random seeds SFF in Appendix Section A.6.1 Tables [1 and 12 The results show that mainstream initializations (e.g.. Kaiming, Xavier) consistently yield stable gains, and SFF shows no noticeable sensitivity to the initialization seed.
Table 1: MSE results of fine-tuning the LTSM Timer for TSF with prediction length 96 under different data proportions. SFF, FF, and TFS denote smoothed full Fine-tuning.J full fine-tuning, and training from scratch, respectively. Full standard deviations and additional results under %%%00% data proportions, as well as MAE results, are reported in Appendix Tables
16
19 and 20
Data propartion
25%
50%
75%
100%
Methods
SFF
FF
TES
SFF
FF
TES
SFF
TFS
SFF
FF
TFS
Exchange
0.0805
0,0865
0,1441
0.0802
0,0891
0.114
0,0802
0,0914
0,1026
0.08
0.091
0.0981
Standard deviation
4..5e
1.9c44
4..0e-
5..c.4
3..3e-3
4.9.e
+ 1.2e-3
1.6e--
4.8.e
4..6e
1.3e.4
1..2e-3
ETThl
0.3506
0.355
0,3788
0.3494
0.3573
0.367
0,3493
0.358
0.3593
0.3547
0,3709
0.36
ETTh2
0.271
0.2866
0.2891
0.273
0.2905
0.2775
0.2772
0.3032
0.2796
0.2737
0.3047
0.2777
ETTml
0.298
0.3049
0.333
0.2955
0.3069
0.3189
0.2956
0.3092
0.3116
0.2954
0.3128
0.3093
ETTm2
0.1594
0.1707
0.1741
0.1605
0.1718
0.1627
0.1623
0.1838
0.1651
0.16
0.1784
0.1644
Weather
0.144
0.1472
0.1627
0.1441
0.1523
0.1538
0.1466
0,1666
0.1559
0.1443
0,1612
0.1526
Electricity
0.1303
0.1344
0.1365
0.1301
0.1347
0.1327
0.13
0.1367
0.1326
0.1304
0.1344
0.1324
Traffic
0,3582
0.3688
0.3497
0.3586
0.3552
0.3478
0.361
0.3606
0.3551
0.3599
0.3609
5.1 FINE-TUNING TIMER FOR MULTIVARIATE TIME SERIES FORECASTING (TSF) AND ANOMALY DETECTION
Experiments show that training from scratch (TFS) requires substantially more data to achieve strong accuracy, whereas pretrained L.TTSs can reach even surpass TFS's full-data performance using

---

Published as a conference paper at ICLR 2026
Fine-tuning (FF)
Smrotthing loss lendscapee then Fine-turing (SFF)
0.080
0.38
0.30
0.35
0.085
0.28
0.30
Assilabe data proport thi Exchangs)
Ansiace dasa propetion 1% cmho
Aesilatl data propetton ETho
Anaiaae tra propertion 1%, (TTmd)
0.16
0.135
0.36
0.15
0.35
10 10
10
10:20
10 20 "fot
Electrccyl
Figure 3: MSE of different fine-tuning strategies Timer with prediction length 96, across multiple datasets (test sets) under varying data proportions (1%-..0%)
only 10%%25% of the data. However, pretrained LSTMs may suffer from poor trainability caused by overfitting, leading to steep and unsmoothed loss landscapes that hinder effective fine-tuning, As result, full fine-tuning (FF) may struggle to improve and can even degrade as the data scale increases (. In contrast, Smoothed Full Fine-tuning (SFF) consistently surpasses FF, exhibiting steadily improving performance as more data becomes available. Across nine public datasets, SFF reduces MSE by an average of 3% and up to 6.5% compared to FF (Table I.. These results demonstrate that SFF effectively smooths the loss landscape, enhances trainability, and better exploits pretrained knowledge without introducing additional memory or computational overhead.
Similarly, in the anomaly detection task, we observe that TFS generally outperforms FF. When MSE is used as the anomaly score, higher predicted MSE values anomalous segments indicate better detection performance. As shown in Ta TFS yields a higher average MSE on anomalous segments than FF. In contrast, SFF achieves significantly higher MSE scores than both FF and TFS, demonstrating its clear advantage in anomaly detection.
Table 2: Anomaly detection results with Timer. Predicted MSE on anomalous segments is reported, with higher values indicating better performance. Results over 250 datasets and four seeds summarized in six groups, with full results and standard deviations in Appendix Tables and 23
Group I datasets)
Group dalasets)
Group
datasets)
SFF
FF
TFS
SFF
FF
TFS
SEF
FF
TES
MSE
Wins
MSE
Wins
MSE
Wins
MSE
Wins
MSB
Wins
MSE
Wins
MSE
Wins
MSE
Wins
| MSE Wins
0.136
31.3
0.072
4.3
0.073
5.3
0.206
32.0
0.098
5.3
0.089
3.7
0,209
30.3
0.104 5.0
| 0.112
5.7
11 1e-e22
+1.
1.3e2
+1.9
1..6222
+0.47
7.7e.3
0.82
[4.0e-3
0.94 1.0e-2
0.47
2.9e-22
1.7
1.0e22 0.0 [2.1e-2
1.7
Group 4 (41 datavets
Group 5 (41 datasets)
Group (45 datasets)
SEF
FF
TFS
SEF
FF Wins|
TFS
SFF
FF
TES
MSE
Wins
MSE
Wins
MSE
Wins
MSE
Wins
MSE
MSE
Wins
MSE
Wins
MSE
Wins|
MSE
Wins
33.7
0.084
3.3
4.0
0.163
29.0
0.078 3.3
0.09
0.157
34.3
0.085
5.0
0.09
5.7
+2.9e-2 #0.47
19.9-3
10.47
18.ee3
40.82
3.4e-2
2.2 66.2e-3
0.47
[2.4e-2 1.9
|6.7e-3
1.2
998e-3
2.4
3.6e-3 2.4
Table 3: Fine-tuning MSE of applying SFF to TimesFM and MOMENT with prediction length 96. Full standard deviations and MAE results are reported in Appendix Tables and 25
Data proportion
25% (TimesFM)
100% (TimesFFM
25% (MOMENT)
100% (MOMENT)
Methods
SPF
FF
TES
SFF
FF
TFS
SFF
FFF
TFS
SFF
FF
TFS
Exchange
0,1139
0,1276
0,1209
0,1149
0,1452
0,1199
0,1502
0.2648
0.1564
0.1064
0,1448
0.1091
Standard deviation
2.0e-3
4.2e-3
2.9c4
6.3e-4
1.7e:2
2.3e-3
2.4e-3
4.6c-4
3.6e
5.8e-4
1.4e-4
2.6e-4
ETThl
0.3955
0.4382
0.4638
0.406
0,5101
0.4358
0.4287
0.4454
0.454
0.3757
0.3951
0.387
ETTh2
0.3232
0.3384
0.3325
0.3198
0.3483
0.347
0.3199
0.3328
0.3326
0.2818
0.2936
0.2979
ETTml
0.3429
0.4001
0.3903
0.3756
0.3926
0.3457
0.3587
0.3538
0.3139
0.3148
0.3272
ETTm2
0.1983
0.2061
0.2091
0,2026
0.2122
0.225
0.1793
0.192
0. 0.1846
0.1692
0.172
0.1736
Weather
0.0865
0.0885
0.1995
0.082
1844
0.1902
0.1673
0. 0.1682
0.169
0.1548
0.1558
0.161
7

---

Published as a conference paper at ICLR 2026
5.2 APPLYING SMOOTHED FULL FINE-TUNING (SFF) FOR OTHER LTSMS
As shown in Table 3 for TimesFM and MOMENT, FF performs worse than TFS in several cases, indicating that pretrained LTSMs may indeed suffer from overfitting. In contrast, SFF consistently outperforms both FF and TFS. Compared with FF, SFF achieves average improvements of 11.45% for TimesFM and 8.31% for MOMENT across different available data proportions.
Moreover, as reported in Table 4 experiments on additional LTSMs, including UniTS, MOIRAI, Chronos, TTMs, and Sundial, further demonstrate that SFF consistently surpasses FF, confirming the effectiveness of the interpolation- based smoothing technology. This is because interpolation perturbs sharp minima without significantly affecting inherently flat regions, thereby facilitating the escape from sharp basins toward smoother and better optima.
Overall, our method generalizes well across diverse LTSM architectures, including encoder-only, decoder-only, encoder-decoder, and MLP-only models, demonstrating universality and robustness.
Table 4: MSE of fine-tuning additional LTSMs for TSF with prediction length 96. *_" indicates unavailable preprocessed datasets (Chronos) or out-of-memory errors (Sundial). Results for prediction length 720 are reported in Appendix Table 13
UnITS-SFF UniTS FF SOI MOIRAL |F| Chronos Chronos FF TT/
TTMs FF (Sundial SFF Sundial FF
ETTh1
0.656
0.678
0.448
0.501
0.773
0.799
0.367
0.371
0.368
0.372
ETTh2
0.364
0.374
0.32
0.321
0.275
0.278
0.293
0.31
ETTml
0.355
0.365
0.308
0.348
0.687
0.724
0.309
0.308
0.419
0.428
ETTm2
0.179
0.184
0.181
0.184
0.17
0.182
Weather
0.171
0.198
0.166
0.173
1.137
1.276
0.157
0.159
0.179
0.186
Elect.
0.309
0.476
0.221
0.226
0.861
0.885
0.149
0.151
Traffic
0.877
1.195
0.476
0.497
0.831
0.834
0,453
0.462
Table 5: MSE comparison of SFF with linear probing (LP), and linear probing followed by full fine-tuning (LPFF) (Kumar et al.) on TSF with Timer and prediction length 96. Average performance is computed over each group of three data proportions. Full standard deviations and MAE results are reported in Tables 26 and 27 Detailed MSE, MAE, and standard deviations for individual proportions are provided in Appendix Tables 28
29
30
31
32
and and33
Data proportion
Avg. on 19, 2%, 3%
Avg. on 4%.
5%,
10%
Avg.
15%, 20%, 25%
AAg. on 50%, 75%.
100%
Methods
SEF
LP
LPFF
SEF
LP
LPFF
SFF
LP
LPFF
SEF
LP
LPFF
Exchange
0.0856
0.5943
0.4001
0.0848
0.5906
0.4186
0,0816
0.563
0.1743
0,0812
0,474
0.0962
Standard deviation
4.4e-4
7.3e-3
3.9e-3
3.7e.4
7.2e-3
6.7e-3
6.1e.4
6.6e-3
7.4e
8.3e-4
4.7e.3
1.9e-3
ETThl
0.3722
0.8806
0.7171
0.3641
0.8594
0.6367
0.3523
0.7955
0.4127
0,3829
0.6356
0.3731
ETTh2
0.28
0.4427
0.4026
0.4375
0.3707
0.2768
0.4234
0.3113
0.2758
0.3849
0.3001
ETTml
0.3448
1.046
0.7038
0.3162
1.0043
0,4772
0.301
0.8975
0.3245
0.2976
0.6608
0.3124
ETTm2
0.1723
0.3555
0.3024
0.1663
0,3499
0,2559
0.1623
0.3297
0.1847
0.1616
0.2771
0.1804
Weather
0.1515
0.324
0.2478
0.146
0,3082
0.1741
0.1441
0.2699
0.1481
0.1483
0.2013
0.1565
Electricity
0.1346
0.6069
0.181
0.132
0.3242
0.1398
0.1305
0.2023
0.132
0.1301
0.1561
0.1335
Traffic
0.3678
0.9577
0.4081
0.3562
0.5999
0.3638
0.3494
0.4529
0.3572
0.3516
0.4079
0.3575
5.3 COMPARE SEF WITH OTHER FINE-TUNING STRATEGIES
We compare SEF with popular fine-tuning strategies, including linear probing (L.P) and linear probing followed by full fine-tuning (LPFF) (Kumar et al.). As reported in Table SFF consistently outperforms both LP and LPFF across multiple data proportions, achieving average MSE reductions ranging from 7.17% to 41.57% relative to the best competitor, LPFF. These results demonstrate that SFF is a highly effective teeituning strategy, as it first smooths sharp regions of the loss landscape and subsequently enables improved trainability and more effective fine-tuning.
Due to limited space, comparisons with additional baselines, including label smoothing, SAM, SWA, Mixout, and L.2-SP. provided in Appendix Table TO) SFF consistently outperforms them.
8

---

Published as a conference paper at ICLR 2026
5.4 DISCUSSION OF RETAINED PRETRAINING KNOWLEDGE AFTER SMOOTHING THE LOSS LANDSCAPE
Impact of smoothing the loss landscape on convergence speed. We record the test loss of different fine-tuning strategies at each epoch, as shown in Figure 4 Pretrained models capture general knowledge and can rapidly extract universal features from time series, thus requiring only a few fine-tuning steps to converge. As illustrated in Figure 4 both SFF and FF converge within the first epoch (highlighted by the red rectangles), indicating that SFF effectively preserves pretrained knowledge after smoothing the loss landscape via model interpolation. Moreover, SFF converges to lower MSE than FF, demonstrating that the smoothing process further enhances model trainability, which aligns well with our design motivation.
Training From Scratch (TFS)
Full Fine-tuning (FF)
Smoothing loss landscape then Full Fine-tuning (SFF)
MSE
MSE
0.40
MSE
0.16
MSE
0.14
0.38
0.15
10
4
Epochs (Exchange)
Epochs
(ETTh))
Epochs (Weather)
Epochs (Electricity)
Figure 4: Test MSE over epochs for different fine-tuning methods on Timer in TSF with prediction
length 96 across multiple datasets.
Impact of smoothing the loss landscape on zero-shot forecasting, Since pretrained knowledge is well preserved after smoothing, we further investigate its impact on zero-shot prediction. As shown in Table 6 the smoothed LTSM consistently improves zero-shot accuracy on seven datasets, achieving average gains of 6.13% for Timer and 75% for TimesFM. These results, averaged over multiple random seeds, indicate that smoothing guides the pretrained model toward a better local optimum. Furthermore, Figure 5 shows that an interpolation coefficient of R 0.85 yields the best zero-shot performance, striking favorable balance between sufficient smoothing and effective preservation of pretrained knowledge, In addition, as reported in Table smoothing also generally improves zero-shot accuracy on additional LTSMs, including MOIRAI, Chronos, TTMs, and Sundial, demonstrating the strong generalizability our method.
Table Zero-shot forecasting results with prediction length 96 before and after smoothing the loss landscape. They both outperform the "Re-initialize" version. MAE results are reported in Table 21
ETThl
ETTh2
EfTml
ETTm2
Weather Timer
Electricity Timer
Traffic
Timer
+Smooth
Timer
+Smooth
Timer
+Smooth
Timer
+Smooth
+Smooth
+Smonth
Timer +Smooth
MSF
0.454
0.399
0.316
0.289
0.816
0.794
0.225
0.209
0.19
0.179
0,210
0.203
0.479
0.463
Std.
+0
11.30-3
=0
-2.20-3
+0
-1-20-2
=0
1166--
=0
19.0c-4
=0
26.20-4
=0
66.6c-4
TimesFM +Smooth TimestM 4mmooth TimesH Smoooh Tim HM +Smooth FM +Smooth [im +Smooth (im FM Smooth
MSE
0.782
0.741
1.865
0,382
1,359
0.993
1.375
0.256
0.397
0.227
0.94
0.886
1.665
1,521
Std.
+0
77.9c-3
20
448
4:0
4224e-2
+0
43.8e-3
=0
---3e-4
+0
3366--
10
+ +.3ee2
Timar
Timer Re iniistie
Timer
1.5
Times YM
Timeafi
0.6
Re Re-reialin
0.3
Re visste
100 1009000
0.75
An toviee
15mpott
15mocth
15moot
1.-
Stropohh
Wntaliee (Sriooth
Smoooth
0.4
0.5
US5 0.5 8/5 9as 9.9
0.5 ues 0.75 pits 0.5
555 8865 0.75
Das C.ts
Ss 1.66 0.15 Das Eas Conffddent (TTT)2))
5.55 0166 0.15 Dus Contidant (TTTm13))
5. dass is u.ss. DS5
Coufficant a (ETM12I
Coutident a (ETTm12)
Coeft (Masther)
Corfident a (Weather)
Figure 5: Impact of interpolation coefficient a on zero-shot forecasting with prediction length 96. The "Re-initialize" model performs the worst, which is reasonable.
5.5 HYPERPARAMETER SENSITIVITY ANALYSIS
Influence of the interpolation coefficient o. As shown in Figure 6 although the interpolation coefficient a in SFF influences performance, SFF consistently achieves lower MSE than standard full fine-tuning (red dashed line) across wide range of values on the Timer model. This further confirms that smoothing enhances trainability and enables pretrained LTSMs to exploit their pretrained knowledge better, thereby improving fine-tuning accuracy. Similarly, the zero-shot results in Figure 5 indicate that a broad range of a values also benefits Timer and TimesFM, further demonstrating the effectiveness and robustness of SFF.

---

Published as a conference paper at ICLR 2026
Table 7: Zero-shot forecasting MSE of additional LTSMs with prediction length 96. Results for UniTS is excluded as it targets few-shot learning.
length 720 are reported in Appendix Table 14
MOIRAI
+Smooth
Chrenos
+Smooth
TTMs
+Smooth
Sundial
+Smooth
ETTh1
0.419
0.405
0.816
0.779
0.364
0.362
0.394
0.385
ETIh2
0.305
0.295
0.277
0.275
0.306
0.303
ETTml
0.557
0.552
0.697
0.685
0.322
0.315
0,365
0.362
ETTm2
0.227
0.219
0.171
0.172
0.2
0.191
Weather
0.192
1.259
0.158
0.158
0.175
0.173
Elect.
0.21
0.197
0.823
0.822
0.166
0.167
Traflic
0.555
0.544
0,854
0.836
0.514
0.516
0.00
FF
0.33
0.35
SFF
0240
SFF
0.55
SFF
3132
0.300
0131
0.20
0.1 02 5- 0. D. 0.
0. 07 0.
0.
32 01 04 D5 0.6. 0.7 a.
01 01 01 04 05 0..
Idteppation conthilent (ETTh32)
Interpotation coeficeent (ITTm12)
Inteccation contident (Weether
Interpoletion coeftec
Figure 6: Fine-tuning Timer for TSF with prediction length 96 (100% training data) under different interpolation coefficients a the test set. ETTh12 denotes the average MSE of ETThl and ETTh2.
Table 8: MSE of fine-tuning pre-trained UniTS by adding different proportions of randomly initialized parameters. The percentages indicate the ratio of parameters subject to weight perturbation.
ETIbl
ETTh2
ETiml
ETTm2
Weather
Electricity
Trallic
Propertion (17.91%) 96
0.678
0.373
0.359
0,181
0.192
0.474
1.194
Proportion (568899)-96
0.665
0.356
0.464
Proportion (53.73%)-96
0.656
0.364
0,36
0,181
0.183
0.462
1.138
Proportion (100%))-9
0.662
0.367
0.355
0.179
0.171
0.309
0.877
Propartion (17.91%) 720
0.735
0.431
0.626
0.419
0.339
0.492
1,305
Proportion (25582%5)775 --20
0,466
Proportion (53.73%)-720
0,704
0.431
0.595
0,418
0.334
0.431
1,238
Proportion (1005))-720
0.707
0.436
0.496
0,419
0.324
0.355
1,01
Influence of the interpolation proportion of model parameters. Notably, SFF operates at the parameter level rather than the layer level (e.g., LayerNorm or FFN), without favoring specific layers. The key factor is therefore the proportion of interpolated parameters. To verify this, start from the first UniTS block and progressively increase the proportion of parameters undergoing weight interpolation, evaluating fine-tuning performance across datasets. As shown in Table larger datasets benefit from higher interpolation ratios (MSE: 100% < 53.73% < 35.82% < 17.91 % for Electricity and Traffic), whereas smaller datasets achieve better performance with lower ratios (MSE: 53.73% 100% for ETThl and ETTh2). This is intuitive, as larger datasets support broader parameter updates that explore more non-convex regions and facilitate escape from sharp minima, while excessive interpolation on smaller datasets may lead to under-training and degraded performance.
6 CONCLUSION
In this work, we identify key challenge: pretrained LSTMs may exhibit poor trainability during fine-tuning due to steep and unsmoothed loss landscapes, which limit the benefits of pretraining. We address this with novel technology that first smooths the loss landscape without additional memory or computational overhead, followed by downstream fine-tuning, thereby improving trainability while preserving pretrained knowledge and achieving consistently better performance. Theoretically, SFF perturbs sharp minima without affecting inherently flat regions, enabling sharp minima escape from unfavorable basins. Extensive experiments eight public datasets using eight pretrained L.TTss with diverse architectures and model sizes demonstrate the robustness of our method across different seeds and data regimes. We believe these insights may also generalize broader pretrained-model fine-tuning scenarios.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint
2024.
George EP Box, Gwilym M Jenkins, Gregory Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308. (rX.::2388.0869, 2023.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In Forty-first International Conference on Machine Learning, pp. 10148-10167. PMLR, 2024.
Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wesley M Gifford, Chandra Reddy, and Jayant Kalagnanam. Tiny time mixers (ttms): Fast pre-trained models for enhanced zero/few-she shot forecasting of multivariate time series. Advances in Neural Information Processing Systems, 37:74147-74181, 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412. 2020..
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural
network loss landscapes. In Proceedings of the aaai conference artificial intelligence, volume 33, pp. 3574-3581, 2019.
Milton Friedman.
The interpolation of time series by related series. Journal of the American Statistical Association, 57(300):729-757, 1962.
Max Gao, Ricky Mok, Esteban Carisimo, Eric Li, Shubham Kulkarni, and kc claffy. Darksim: A similarity-basee time-series analytic framework for darknet traffic. In Proceedings of the 2024 ACM on Internet Measurement Conference, pp. 241-258 2024a.
Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and
Marinka Zitnik. Units: Building unified time series model. arXiv, 2024b. URL https
7/arxiv. wwtppot///poooooooo.oooooo... /240222433.. 00131. pdf
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: a family of open time-series foundation models. In Proceedings of the 41st International Conference Machine Learning, pp. 16115-161 2024.
Trevor Hastie. The elements of statistical learning: data mining, inference, and prediction, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034,003,222 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Izmailov, AG Wilson, D Podoprikhin, Vetrov, and Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 376-885, 2018.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. In The Twelfth International Conference Learning Representations.

---

Published as a conference paper at ICLR 2026
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-bute batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609 (60.006699. 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jędrzej Kozal, Jan Wasilewski, Bartosz. Krawezyk, and Michat Wožniak. Continual learning with weight interpolation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4187-4195, 2024.
Ananya Kumar. Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform aut-of-disttibution. International Conference on Learning Representations.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. International Conference on Learning Representations.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: generative pre-trained transformers are large time series models. In Proceedings of the 41st International Conference Machine Learning, pp. 32369-322222 32399,, 2024.
Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang. Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Sundial: A family of highly capable time series foundation models. arXiv preprint arXiv:2502.00876 2025.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. net forrmmormmm id=Jbdc0vToco!
URL /tRL/tttss://ppeneee// http://pppeeeveeeeee
Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. Time-series anomaly detection service microsoft. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 3009-3017, 2019.
Jaydip Sen and Sidra Mehtab. Machine learning in the analysis and forecasting of financial time series. Cambridge Scholars Publishing, 2022.
Tiffany J Vlaar and Jonathan Frankle. What linear interpolation of neural network loss landscapes tell us? In International Conference on Machine Learning, pp. 22325-22341. PMLR, 2022.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. 2024a.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In International Conference on Machine Learning, pp. 53140-33164 PMLR, 2024b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppss. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference machine learning, pp. 23965-23998, PMLR, 2022.
Renjie Wu and Eamonn J Keogh. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE transactions on knowledge and data engineering, 35(3): 2421-2429, 2021.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. International conference on machine learning, pp. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2026
Xu Zhang, Zhengang Huang, Yunzhi Wu, Xun Lu, Erpeng Qi. Yunkai Chen. Zhongya Xue, Peng Wang, and Wei Wang. Self-adaptive scale handling for forecasting time series with scale hetero- geneity. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7485-7489. IEEE, 2024.
Xu Zhang, Zhengang Huang, Yunzhi Wu, Xun Lu, Erpeng Qi, Yunkai Chen, Zhongya Xue, Qitong Wang, Peng Wang, and Wei Wang. Multi-period learning for financial time series forecasting, In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 2848-2888 2859, 2025a.
Xu Zhang, Peng Wang, Chen Wang. Zhe Xu, Xiaohua Nie, and Wei Wang. Global feature enhancing and fusion framework for strain gauge status recognition. In Companion Proceedings of the ACM on Web Conference 2025, pp. 611-620, 2025b.
Xu Zhang, Qitong Wang, Peng Wang, and Wei Wang. A lightweight sparse interaction network for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pp. 15304-13312 2025c.
Xu Zhang, Junwei Deng, Chang Xu, Hao Li, and Jiang Bian. Diff-mn: Diffusion parameterized moe-nede for continuous time series generation with irregular observations, 2026a. URL https
/arxiv.org/abss.
13534
Xu Zhang, Peng Wang, Yichen Li, and Wei Wang. Amortized predictabilit--wwar training framework for time series forecasting and classification. arXiv preprint arXiv:2602.1622. 16224, 2026b.
Xu Zhang, Qitong Wang, Peng Wang, and Wei Wang. Semixer: Semantics enhanced mlp-mixer for multiscale mixing and long-term time series forecasting. arXiv preprint arXiv:2602.1623 16220, 2026c.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems,
2023.
13

---

Published as a conference paper at ICLR 2026
A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
A. PROOF FOR THEOREMS ABOUT SHARP MINIMA AND FLAT MINIMA
We begin by performing second-order Taylor expansion of the loss function [((*) around the minimum point :
L( + _(0") + C2
2
(9)
~ - v"
where a small parameter perturbation (e.g.. |6| < 1).
The loss change AC after perturbation (interpolation) be formulated as:
1 _ H5 AC L(O' + -(0)) - v"
(10)
The largest AC is governed by the Amax of the Hessian H v²
Let the unit vector Umax be the cigenvector corresponding to the eigenvalue Amux. Perturbing in the direction of Umax yields the largest AC, because Umax represents the direction of maximum
curvature (Foret et al. (020), i.e., the steepest direction in the loss landscape. Let = c Uma with |e| < Then Eq, 10 can be reformulated as:
AC = c. V He Umax
(11)
According to the eigenvalue equation Umax Armux Umaxx we can further derive:
1 2 max Amaa Uraax AL ~
2 Umax Amnax 2 Ammxxx
(12)
2
2
2
Hence, AL is closely related to Amix a larger Amax corresponds to a sharper loss landscape, where perturbations along many directions significantly increase the loss. Conversely, smaller Amax indicates a flatter landscape, in which perturbations have little effect on the loss (Hochreiter & Schmidhuber 1997).
A.2 RELATED WORKS
Fine-tuning large time series models. Most works focus designing pre-training architecture and collecting large-scale time series data Das et al. (2024); Goswami et al. (2024); Woo al. (2024b); Liu et al. 2024) enrich and improve the foundations of LTSM. The fine-tune mne-tunnng techmique has received relatively less attention in the field of large time series models. The traditional fine-tuning strategies either full fine-tuning (adjusting all model parameters) or only fine-tuning the prediction head (also called linear probing). In the computer vision domain, Kumar et al. points out that full fine- tuning can achieve worse accuracy than linear probing in the condition of meeting (OOD) data when the pretrained features are good and the distribution shift is large. To address this, they propose to linear probing first and then full fine-tuning (LP-FF) to improve the fine-tuning performance of the pre-trained model in the OOD condition. In our study, we also apply LP-FF to fine-tune the LTSM as baseline to compare with our method.
-iittridiitiittiinn
Various optimization strategies can also be used for fine-tuning, Specifically, SAM (Sharpness-Awee) Minimization) (Foret et al., 2020) updates parameters not only by the loss at the current point, but also by the flatness in small neighborhood, so the optimizer is less likely to fall into high-curvature sharp minima. Note, however, that SAM needs one extra forward-backwand pass, so its training cost is twice that of ordinary training. SWA (Stochastic Weight Averaging) (Izmailov et al.) 2018 is an ensemble- like technology: saves several weight snapshots taken after the model has converged and averages them produce the final weights, reducing randomness and over-fitting Mixout (Lee) et al.) randomly replaces subset of the fine-tume ine-tuned weights with their pre-trained counterparts during training, mitigating catastrophic forgetting and over-fitting I.2-SP (Xuhong al. 2018)
14

---

Published as a conference paper at ICLR 2026
adds an L2 penalty between the current weights and the pre- vre-trained weights to the loss, preventing the fine-tuned model from drifting too far away from the pre-trained solution and thus preserving pre-trained knowledge while curbing over-fitting
Large time series models (LTSM), Existing efforts toward LTSMs can be categorized into two groups, with one being large language models for time series. FPT Zhou et al. (2023) partially fine-tunes GPT-2 on different downstream tasks. LLM4TS Chang et al. (2023) encodes time series into numerical tokens to utilize LLMs for time series forecasting. TimeLLM Jin al. aligns the text prompt with time series to enhance prediction. These methods demonstrate the potential of LLMs for time series analysis. Another category includes pre-trained models on large-scale time series. Moirai Woo et al. (2024b), an encoder-only architecture. transforms time series into varied token sizes for better handling varied frequencies and then performs the pre-training strategy of mask modeling for time series forecasting. MOMENT Goswami et al. (2024), an encoder-decode architecture, adopts a BERT-style mask modeling pre-training strategy and supports various downstream time series tasks. TimesFM Das et al. (2024) is decoder-only Transformer pre-trained on Google Trends for forecasting, exhibiting notable zero-shot ability. Timer [iu et al. (2024) conducts GPT-style pre-training the carefully processed and collected UTSD dataset and has achieved advanced accuracy on various tasks, including forecasting (Zhang et al.) [026b (025c), imputation Zhang et al. (026aa), and anomaly detection Liu al. 2024); 2 et al. (2025b).
A.3 PYTORCH CODES TO SMOOTH THE LOSS LANDSCAPE OF THE PRETRAINED LTSM
We show the example codes to use the randomly initialized LTSM to smooth the loss landscape of the pretrained one in Algorithm] I By combining the strengths of the randomly initialized LTSM (good trainability with smoother loss landscape) and the pretrained LTSM (good pretrained knowledge), the convergence of the smoothed LTSM can be improved during fine-tuning.
Algorithm 1: Smoothing the loss landscape of the pre-trained LTSM for fine-tuning
def Smoothing
(modell, (modell,mcde12)> model2);
"modell: pre-trained LTSM, model2: randomly initialized LTSK""""
for paraml, param2 in zip (model1, parameters (), model2, aodel2.parameters. ())
[modell.parametess
# Smoothing the loss landscape of modell through interpolation
cope__ (modell (modell Next, the smoothed modell is applied for fine-tuning thout increasing ng memory and computati onal overhead
alpha model2 (1 alpha)))
return modell
A.4 MORE DETAILS ABOUT REPRODUCING PAPER RESULTS
Datasets. In forecasting and imputation, we conduct extensive experiments eight well-known datasets, including Exchange rate, Weather, Electricity, Traffic, and four ETT datasets (ETThI, ETTh2, ETTml, ETTm2). Details be seen in Appendix A.5 and Table In the anomaly detection task, following previous work Liu et al. y Wu & Kcogh (202I). we use UCR Anomaly Archive (containing 250 datasets) Wu & Keogh (2021) for anomaly detection.
Evaluations. Following Timer Liu et al. (2024). we uniformly use MSE (Mean Squared Error) and MAE (Mean Absolute Error) evaluate the performance of methods on forecasting, imputation, and anomaly detection tasks. In forecasting, we investigate the performance of the proposed smoothed fine-tuning under different proportions of available fine-tuning data. The proportions range from 1% to 100%. The prediction length is 96 or 720. In anomaly detection, similar to Timer Liu et al. (2024), MSE used as confidence level to evaluate the effectiveness of anomaly detection. The higher the predicted MSE of the anomalous segments, the better, as this reduces the risk of normal segments being misjudged as anomalies.
Implementation details. The interpolation coefficient is selected from 0.3, 0.5, 0.7, 0.9 for all tasks. For fairness, we use the source codes of each baseline and follow their recommended settings. Within each baseline, the configurations for "baseline" and Praseinnefinettningg are kept identical, ensuring that any performance change is attributable solely to the fine-tuning method (c.g., direct full FT vs. smoothed full FT). Settings may differ across baselines due to their public implementations,
15

---

Published as a conference paper at ICLR 2026
accuracies between baselines not directly comparable. When the recommended input length causes out-of-memory issues (e.g., Sundial), we reduce while still keeping "baseline" and "baseline- finetuning" consistent. All experiments run four random seeds with NVIDIA 3090 GPUs using PyTorch, reporting the mean and standard deviation.
Following [Liu al. 2024). the input and prediction lengths of Timer fixed at 672 and 96. Based on the limited computing resources and settings supported by each model, the input lengths for MOMENT and TimesFM are 512 and 256, while the forecast lengths are 96, and 128, respectively. Following Timer Liu et al. (2024), the fine-tuning epochs are fixed at 10, and we report the best metric all epochs. The learning rate is 3e-5 and the optimizer Adam. More details can be found in our source code.
When implementing the baseline LTSMs, we download the weights from their official links, c.g.. which are listed as follows:
Timer's codes and pre-trained weights can be downloaded from the links
TimesEM's s codes and pre-trained weights can be downloaded from the link The model weight path on Hugging Face is http:/imeesmmm-.-000------------
We adopt the latest
2.0 version.
MOMENT's codes and pre-trained weights can be downloaded from the link The model weight path on Hugging Face is "AutonLab/MOMENT I-large..
A.5
MORE DETAILS ABOUT PUBLIC DATASETS
The statisties of the eight well-known datasets are shown in Table covering a range of variables and sampling frequency. These datasets involve applications in industrial machines, energy, and weather domains. They have been widely employed in the literature for time series analysis tasks Nie et al.
(2023): :: Liu et al. (2024); Goswami et al. (002)) Woo et al. (0024b) Zhou al. (2023): Chang et al. 20233
Jin et al.
Table 9: Statistics of eleven public datasets. Data size denotes the number of samples in train, validation, and test set for the single variable. In the experiment, each variable separately split to construct samples and then merged, so the total number of samples needs to be multiplied by the number of variables. Frequency denotes the sampling interval of time points.
Datasets
Variable
Data size (single variable)
Frequency
ETTHLETTI2
(8545, 2881, 2881)
Hourly
ETTmI,ETTm2
(34465, 11521, 11521)
15min
Weather
(36792, 5271, 10540)
10min
Exchange rate
(5120, 665, 1422).
Daily
Blectricity
321
(18317, 2633, 5261)
Hourly
Trailic
(12185, 1757,
Hourly
These datasets used in this paper are extensively used for TSF algorithm evaluation, including exchange rate forecasting in the financial field, electricity consumption forecasting in the energy field, climate parameter forecasting in the weather domain, and machine parameter (e.g., loads and oil temperature) forecasting in the industrial field:
Electricity datase collects the electricity consumption (kWh) every 15 minutes of 321 clients from 2012 to 2014.
16

---

Published as a conference paper at ICLR 2026
ETT datase comprises two sub-datasets, ETTI and ETT2, collected from two separate counties. Each sub-dataset offers two versions with varying sampling resolutions (15 minutes and 1 hour). ETT dataset includes multiple time series of electrical loads and single time sequence of oil temperature.
Weather datase ④ contains 21 meteorological indicators, such as air temperature, humidity, etc, recorded every 10 minutes for the entirety of 2020.
Traffic TO
dataset contains the occupation rate of freeway systems in California, USA. 5).
All datasets can be downloaded from the link 11
A.6 ADDITIONAL EXPERIMENT RESULTS AND DISCUSSIONS
The experiments in the appendix serve as supplements to those in the main paper, including complete standard deviations, MAE results, and interpolation experiments. All figures and tables in the appendix have been appropriately linked and referenced in the main paper. They can be located by clicking the hyperlinks in the main paper while reading
We hope that these extensive experiments can help demonstrate that directly fine-tuning pre-trained LTSMs may indeed lead limited performance, as they may overfit during pre-training, resulting in steep and unsmoothed loss landscape and poor trainability, thereby degrading and limiting the fine-tuning performance of pre-trained LTSMs on downstream tasks. Meanwhile, Our proposed smoothed finetuning can indeed help pre-trained LTSMs achieve better fine-tuning performance on downstream tasks.
A.6.1 COMPARISONS BETWEEN SFF AND MORE BASELINES
Comparison with LoRA. We additionally add LoRA fine-tuning as a baseline to highlight the contri- bution of our method. Since LTSM is mostly < B parameters, we follow the official recommendation and set the low-rank factor == 8. As reported in Table [TO] our approach outperforms LoRA. This is reasonable: LoRA trades full fine-tuning for a low-rank constraint, achieving appealing parameter efficiency, yet this restriction can limit the model's fine-tuning capacity.
Comparison with popular optimization strategies. We further compare our method with several widely used optimization strategies during training. As shown in Table I0 while some of these approaches offer modest improvements over standard full fine-tuning, their performance still falls far short of that achieved by our proposed SEF fine-tuning. The key limitation is that they don't address the underlying problem, namely, the highly steep and non-smooth loss landscape of the pre-trained model. In contrast, SFF explicitly mitigates this problem by first smoothing the landscape and then fine-tuning, resulting in substantially stronger fine-tuning and adaptation performance.
Influence of different parameter initialization schemes. We have conducted ablations with several perturbation-bast based smoothing strategies: standard Gaussian noise (mean = 0, variance 1). Xavier Gaussian, Xavier uniform, Kaiming Gaussian, and Kaiming uniform. Table IT shows that Xavier- and Kaiming-bars based schemes maintain stable performance improvements. Because they consider the stable gradient variance and can supply flat loss landscape (demonstrated by (Fort Scherlis 2019)) that is used to smooth the sharp landscape of the pre-trained model for better fine-tuning effect, which also aligned with our Theoretical analysis. In contrast, standard Gaussian initialization without variance control often pushes parameters into sharper regions of the loss landscape, leading to weaker smoothing and noticeably degraded downstream performance. These findings provide strong empirical support for design choice.
Influence of random seeds on parameter initializations. To assess sensitivity, we further evaluate SFF under different random seeds while using widely adopted initialization schemes (e.g., Kaiming uniform). The results in Table 12 indicate that improved performance remains highly stable across
hhtpp://pems.dot.caa.uu
htttss///orree..pppglc.ooo/rrr/e cCex (C2x.ccmmmyrllxxgfvawwwyy
17

---

Published as a conference paper at ICLR 2026
seeds. This suggests that SFF does not rely on carefully engineered initialization. Instead, the mainstream initialization strategy suffices to obtain consistent smoothing and fine-tuning gains. This reasonable because the prior work (Fort & Scherlis 2019) has proven that the underlying design of mainstream initialization methods ensures the initialized parameters indeed lie in the flat region of the loss landscape, without being influenced by the random states (seeds). We believe this robustness is a desirable property for practical deployment.
Table 10: Comparison with more baselines on the LTSM Timer with prediction length 96. We independently run four times with four random seeds to enhance the solidity the results and report the mean value and standard deviation.
Exchange
ETIbl
ETTh2
ETTml
ETTm2
Weather
Original full fine-tuning
0.09#0 0007
0.36740.0027
0. 0,30420.004 0049
0.312+0.0008
0.176+0.00/
0.158#00.00 0012
LoRA
0.12220.0003
0.41820 0005
0.30
0.401
0.19/=000001
0.155#000004
smoothing
0,0920.0018
0.364200.0036
0.30320.0043
0.31240.0011
0.177
0.158a0.0008
SAM
0.08%
0.362=0.003
0.296
0.309:0.0002
0.175:0.0017
0.157
SWA
0.09440.0017 0017
0,366+0.0024
0.304=0.0045
0.319+0.0009
0.17840.0014
0.162+0.0005
MixOut
0.09#0.0000
0.376#0:0006
0.297=0.0001
0.348:0.0018
0.18440 0006
0,16+0.000
0.09
0.36810000031
0.30420.0055
0.31520.0007
0.17740.0013
0.1620.00044
Ours (SFF)
00,000
0,35520.0013
0.274:0.0008
0,29750.0016
0,161=0.0009
0.145#0,0006
Table 11:
The effectiveness of SFF (smoothed full fine-tuning) across different parameter initialization schemes on the LTSM Timer.
Exchange
ETThl
ETTh2
ETTmi
ETTm2
Weather
Original full fine-tuning
0.09=0:0007
0.367=0.0027
0,304=0.0049
0,31240.0008
0,176+0.0013
0.1 0.158+0.0012
Standard Gaussian perturhation SPF
5.986a0.261
0.723#0.003
1.879=0.275
3.876=00.128
19.031=0.963
0.447=0.03
Kaiming Normal Distribution SFF
0.081=0.0006
0.353=0.0009
0.277=0.001
0.299+0.0011
0,164+0.0016
0.146+0.0008
Kaiming Uniform Distribution-SFF
0.081=0.0008
0.35520.000 0013
0.274=0.0008
0,297:0.0016
0,161 #0.0009
0. 0.145+0.0000 0006
Xavier Normal Distribution-33
0.081=0.0007
0.353:1.000
0.27620.0012
0.320.0009
0.16240.0001
0.14510.0002
Xavier Uniform Distribation SIFF
0.082:0.0008
0.353=0.0007
0.27720.0006
0.3g0.0003
0.162+0.0001
0.14510.0002
Experiments with four different random seeds (rl, r2.r3, and r4 here) on the LTSM Timer. The results show that SFF (smoothed full fine-tuning) is insensitive to the choice of random initialization distribution. FF denotes original full fine-tuning.
Table 12:
Exchange
ETThi
ETTh2
ETTml
ETTm2
Weather
SFF (Ours)-el
0,07996
0.3547
0,27379
0,29542
0,16003
0.14662
FF-l
0.08937
0.36941
0.31021
0.31134
0.17645
0.16067
(Ours)-2
0.35772
0.29902
0.14432
FF-r2
0.09071
0.36245
0,3035
0,31282
0,17843
0.15912
SFF (Ours)-r3
0.08101
0.35766
0.27453
0.29824
0,16129
0,14481
FF-r3
0.09102
0.36848
0.29699
0.31
0,17515
0,85
SEF (Ours)
0.08191
0.15588
0,27571
0,29938
0,16173
0,14525
FF-r4
0,08978
0,36815
0.30666
0.31057
0.17546
0.15649
A.6.2 GUIDANCE FOR SELECTING &
In practice, we suggest the following guidance to select oc:
(1) Empirically recommended values: Empirically Recommended Values: As shown in Figure 6 and Figure the main text, while different values introduce some variation, the sensitivity analysis demonstrates that SFF consistently outperforms vanilla fine-tuning across a wide range. These values can thus serve as a reference and be prioritized in experimental setups.
(2) Validation-basel tuning: We observe that the trend of test performance with respect to closely mirrors that on the validation set. Therefore, once a candidate search range is defined, selecting the that minimizes validation error provides straightforward and computationally efficient strategy.
(3) Data-driven selection: Automatically learning is indeed a promising direction. In the current framework, however, the interpolation weights fixed prior to fine-tuning, which makes adaptive selection non-trivee triviaii Approaches such as meta-learning could potentially be explored to determine optimal values across models and datasets. We regard this as important avenue for future research.
18

---

Published as a conference paper at ICLR 2026
A.6.3 INFLUENCE OF SFF ON NORMALIZATION OR SCALE BETWEEN LAYERS
We discuss this influence in two scenarios:
(1) Fine-tuning after loss landscape smoothing: When weights are smoothed, and the model is subsequently fine-tuned, the potential mismatch in normalization or scale is negligible. The model free to update the relevant parameters during fine-tuning, effectively correcting any minor discrepancies introduced by interpolation.
(2) Zero-shot forecasting after loss landscape smoothing: First, the random initializations used for smoothing follow standard schemes (e.g.. Kaiming, Xavier), whose typical scales consistent with the learnable scale parameters in normalization layers. Second, the "flat" and "sharp" minima analyze encompass the entire parameter space, including the weight matrices of normalization layers. Consequently, in theory, our method does not introduce significant scale or aligmment inconsistencies. Instead, smooths sharp minima located in suboptimal regions without harming flat minima, effectively relocating the sharp minima of the normalization layers to more favorable convergence points and thereby achieving improved and more generalizable performance. Moreover, we have included formal theoretical derivations and analyses demonstrating that the interpolation technology improves sharp minima while not harming flat minima.
Morcover, empirically, as shown in Tables 6 and 7 consistently demonstrates accuracy improvements, supporting the theoretical reasoning outlined above.
zero-shot forecasting following weight smoothing
A.6.4
MORE CASES ABOUT THE LOSS LANDSCAPE THE PRETRAINED LTSM (FIGURE TO AND FIGURE 8
As shown in Figure T0jin main paper and Figure and Figure 8 in the Appendix, we visualize the loss landscape of different datasets and empirically find that LTSMs initialized randomly typically have smooth loss landscape. In contrast, pre-trained LTSMs consistently exhibit steep and non-smooth loss landscapes, indicating that this a random occurrence. Hence, after pre-training, LTSMs may indeed show lower trainability, which can affect their fine-tuning performance on downstream tasks.
(a) Original pre-trained LTSM
Random Initialization
(c) Uuilizing (b) to smooth (a)
(d) Placing (b) und semi- transparent (a) in the coordinate system
(e) Placing (b) and semi- transparent (c) in the same coordinate system
Figure 7: Loss landscape comparisons based on the LTSM Timer and weather dataset. The smoother the surface, the better.
(u) Original pre-trained LTSM
(b) Random Initialization
(c) Utilizing (b) to smooth
(d) Placing (b) and semi- transparent the same coordinate system
(e) Placing (b) and semi- transparent (c) the same coordinate system
Figure 8: Loss landscape comparisons based the LTSM Timer and electricity dataset. The smoother the surface, the better.
19

---

Published as a conference paper at ICLR 2026
Training from scratch
Directly fine-tuning
Smoothing landscape suning
0.3
MSE
0.2
MSE
MSE
MSE
0.4
0.2
0.1
0.2
0.1
8
8
4
10
Epochs (Exchange rate)
Epochs (ETTh1)
Epochs (ETTm1)
Epochs (Weather)
Training
Directly fine-turing
Smoothing landscape fire-tuning
0.09
0.367
0.312
0.150
MSE
MSE
0.362
MSE
MSE
0.152
0.354
0.295
Exchange rate
ETTh1
ETTm1
Weather
Figure 9: Time series forecasting of the LTSM Timer on various datasets with 100% available data proportion. We show the comparisons of training and testing losses for training from scratch (black lines and bars), direct full fine-tuning (green lines and bars), and smoothing the loss landscape then full fine-tuning (blue lines and bars).
A.6.5 TRAINING LOSS AND TEST LOSS DURING FINE-TUNING THE PRE-TRAINED LTSM (FIGURA 9
We also empirically observe severe overfitting during the fine-tuning of pre-trained LTSM on down- stream tasks, which is consistent with our analysis of the loss landscape. steep and non-smooth loss landscape may cause the model to fall into poor local optmm,, leading to severe overfitting Li al. (2018). Specifically,
shown in Figure 9 the training loss directly fine-tuning the Timer (green lines) is significantly the lowest. However, the test MSE of the fine-tuned Timer (green bars) is even significantly worse than that of training from scratch on the Timer (black bars) on the downstream datasets (e.g., ETTh1, ETTml, and Weather) without pre-training. This suggests that directly fine-tuning Timer leads to severe overfitting Hastie (2009), which causes pre-trained knowledge of it not to be fully utilized for improving the accuracy of downstream tasks.
A.6.6 APPLYING smoothed mioetted..inetuumieee tuning FOR TIME SERIES IMPUTATION TASK (FIGURE TO
As shown in Figure 10 our method also shows improvement for the imputation task. This indicates that the poor trainability of the LTSM, caused by overfitting during the pre-training phase, may impact their performance detection, and imputation tasks. Our proposed method offers a potential solution to address this issue and provides new perspective for fine-tuning the pretrained LTSMs.
various downstream time series tasks, including forecasting, anomaly
Training from scratch
Directly fine-tuning
Smocthing the loss landscope then full fine-tuning
302
0.71
0.385
0.112
MSE
MSE
MSE
0.105
0.109
0.283
0.102
0.107
0,279
0,292
ETT1
ETTh2
ET7m1
ETTmZ
Weather
Figure 10: Time series imputation task on Timer with the mask ratio 25%. The experimental settings follow the Timer's codes: https: //gi com/thumlma.. / Ia rge- Time- Seri es Model
tab readre ov file
A.6.7 SUMMARY OF THE CONTENT IN THE FOLLOWING SECTIONS
The experimental results in the following sections serve supplements to the experiments in the main paper regarding complete standard deviations and MAE results under different available data proportions. Through multiple experiments with different random seeds, ensure that our proposed fine-tuning method indeed helps the LTSM achieve better fine-tuning performance and that this is not random occurrence. Moreover, method consistently outperforms other fine-tuning methods
20

---

Published as a conference paper at ICLR 2026
in terms of the MAE metric and across different data proportions, which further demonstrates the effectiveness our approach. Our work provides new insights for fine-tuning large models. In the subsequent content, provide relevant titles (in table of contents format) for reference to the supplementary experimental results without further redundant explanations:
Fintuning MSE with prediction length 720 for more LTSMs (Table 13).
Zero-shot forecasting MSE with prediction length 720 for more LTSMs (Table 14)
Forecasting MSE and standard deviations under 1 1%, 2%, and 4% proportion of available data (Table T5p.
Forecasting MSE and standard deviations under 5%, 10%, 15% and 20% proportion of available data (Table T6).
Forecasting MSE and standard deviations under 25%, 50%, 75% and 100% proportion of available data (Table 17).
Forecasting MAE and standard deviations under 1%, 2%, 3% and 4% proportion of available data (
Forecasting MAE and standard deviations under 5%, 10%, 15% and 20% proportion of available data (Table 19p.
Forecasting MAE and standard deviations under 25%, 50%, 75% and 100% proportion of available data (Table 20)..
Complete MSE of anomaly detection results on 250 datasets (Table 22
Complete standard deviations anomaly detection results on 250 datasets (Table 23
Complete MSE and standard deviations of applying the pretrained LTSMs TimesFM and MOMENT for time series forecasting (Table 24).
Complete MAE and standard deviations applying the pretrained LTSMs TimesFM and MOMENT for time series forecasting (Table 25).
Complete MSE and standard deviations of applying other fine-tuning methods, LP and LPFF, under grouped data proportions for time series forecasting (Table 26
Complete MAE and standard deviations of applying other fine-tuning methods, LP and LPFF, under grouped data proportions for time series forecasting (Table 27
Complete MSE and standard deviations of applying other fine-tuning methods, LP and LPFF, under 1 1%, 2%. 3%, and 4% proportion of available data for time series forecasting (Table 28).
Complete MSE and standard deviations of applying other fine-tuning methods, L.P and LPFF, under 5%, 10%, 15%, and 20% proportion of available data for time series forecasting (Table 29).
Complete MSE and standard deviations of applying other fine-tuning methods, LP and LPFF, under 25% 50%, 75%, and 100% proportion of available data for time series forecasting (Table 30).
Complete MAE and standard deviations of applying other fine-tuning methods, LP and LPFF, under 1 1,, 2%, 3%, and 4% proportion of available data for time series forecasting (Table
Complete MAE and standard deviations of applying other fine-tuning methods, LP and LPFF, under 5%, 10%, 15%, and 20% proportion of available data for time series forecasting (Table
Complete MAE and standard deviations of applying other fine-tuning methods, LP and LPFF, under 25%, 50%, 75%, and 100% proportion of available data for time series forecasting (Table 33).
Complete MAE and standard deviations of zero-shot forecasting after smoothing the loss landscape (Table 21
21

---

Published as a conference paper at ICLR 2026
Table 13: MSE of fine-tuning more LTSMs for the TSF task with prediction length 720. SFF, and FF are smoothed full fine-tuning and full fine-tuning.
UniTS-SFF
UniTS+tF MOIRAI-SFF MOIRAI-FT C Chronos-SFt SFF Chrones-FF (TTMMss-SF ITMs-FF (Sundial-StF Sundial-FF
ETTb1
0.704
0,741
0.582
0.634
0.421
0.424
0,485
0.49
ETTh2
0.431
0.436
0.582
0.634
0.402
0.404
ETTml
0.496
0.625
0.402
0.451
0.433
0.435
0.552
0.562
ETTm2
0.416
0.419
0,361
0.356
0,374
0.376
0.376
0,387
Weather
0.324
0.346
0.319
0.33
0.328
0.328
0.36
0.362
Elect.
0.355
0.495
0.843
0.999
0.241
0.24
Trallic
1.01
1.307
0.554
0.576
0.612
0.611
Table 14: Zhoo forecasting MSE of more LTSMs with prediction length 720. "_" indicates that the preprocessed dataset not included (Chronos) or out of memory (Sundial).
MOIRAI
+Smooth
Chromos
+Smooth
TIMs
+Smooth
Sundial
+Smooth
ETTh1
0.439
0.447
0.421
0.424
0.437
0.453
ETTh2
0.386
0.4
0.404
0.408
0.409
0.417
ETTml
0.435
0.439
0.417
FTTm2
0.407
0.438
0.409
0.41
0.397
0.408
Weather
0.399
0.424
0.328
0.328
0.354
0.357
Elect.
0.253
0.255
0.257
0.255
Traffic
0.622
0.622
Table 15: MSE of fine-tuning LTSM Timer for time series forecasting under 1%, 2 2% 3% and 4 % proportion of available data. SFF, FF, and TFS are smoothed full fine-tuning, full fine-tuning, and training from scratch, respectively.
Data proportion
1%
2%
3%
4%
Methods
SFF
FF
TFS
SFF
FF
TFS
SFF
FF
TFS
SFF
FF
TFS
Exchange
0,0853
0.0887
0.2831
0,0858
0.0884
0.2808
0.0842
0.0876
0.2803
0,085
0.0879
0.2796
Standard deviation
+332e-4
19.0c4 4
3..2
1442e-4
4...6.4
16.88.3
4.5.8c-4
18.1e4
16.86
12.4e-
16.2e-4
16.76:
0.3649
0.3861
0.5614
0.3608
0.3846
0.5588
0.3606
0.3823
Standard deviation
+6.36-3
+1 +.266-
+9.8e-3
36.66-3
+3.60-4
11.le
16.60-3
47.6c-4
1e.1e
+7.50-3
41-4
4447e-3
ETTh2
0.2825
0.2945
0.363
0.2747
0.2893
0,362
0.2772
0.2896
0.3617
0.2752
0.2872
0.3434
Standard deviation
+100-3
4..4c-4
3
11.76-3
31.56-3
+2.3e-3
11-96-3
47.0c-4
4.276-3
1.277-3
12.1e-3
4-236-3
ETTml
0,364
0.3826
0.5342
0.3304
0.3484
0.4193
0.3252
0.3418
0.4159
0.3179
0.3312
0.3884
Standard deviation
14.4e-3
1.336-
1.11e-
+:2.7633
1:1.003
+:3.003
4:344-3
18.00-4
+e3.2
+3356-3
17.9c-4
+:2.6633
ETTm2
0.1709
0.1897
0.2531
0,173
0.184
0.2524
0.1635
0.1758
0.2157
0.1654
0.1755
0,216
Standard deviation
32.4e-3
+ 3177-3
3:3.2e-3
1:2.ee3
47
1:3.46-3
1.2.4c33
4.3.4-44
11.76-3
11.4e-3
16.1e-
-1.56.3
Weather
0.1537
0.1564
0.2403
0.1505
0.153
0.2259
0.1489
0.1525
022162
0.147
0.1492
0.2111
Standard doviation
+39904
#6.50-
33.00-3
11.504
19.104
#2.36-3
48.76-
1.20-3
45.76-4
+2.904
28.26-4
#5.00-
Electricity
0.1369
0.1393
0.2285
0.1342
0.1367
0.2035
0.133
0.1356
0.1857
0.1331
0.1358
0.1708
Standard deviation
+7.805
+7.60-4
+1.003
-190
17.104
+3.30-4
#6.90-4
11.3
+2.504
47.404
#6.4c-4
0.3743
0.3768
0.3671
0.3698
0.4794
0.3594
0.3623
0.4199
Standard deviation
4..6 4
16.8e-4
3
1.000
1.9.9e4
3.2.4e-3 3
1.2.4e4 4
1.7.9e44
1.8.0e
1.2.3e4
1.9.9e
1.1.2e-4
Avg. Improvements
4.07%
37.9
3.64%
34.04
3.91%
31.47
3.4%
28.96
Max. Improvements
9.91%
69.87
5.98%
69.44
7.0%
69.96
5.75%
69.6
22

---

Published as a conference paper at ICLR 2026
Table 16: MSE of fine-tuning LTSM Timer for time series forecasting under 5%, 10% 15% and 20 proportion of available data. SFF, FE, and TFS are smoothed full fine-tuning, full fine-tuning, and training from scratch, respectively.
Data propartion
5%
10%
15%
20%
Methods
SFF
Exchunge
0.0854
0.0883
0.2721
0.0829
0.0854
0.1919
0.0815
0.0845
0.1715
0.0805
0.0858
0.157
Standard deviation
+1.60-
4840-4
16.40-3
+7.204
#6.9-4
#1.663
#3.104
#940-4
19.1e4
#110-3
#140-3
+2.50-3
ETTh1 0,3582
0.3745
0,500
0.3539
0.3654
04162
0.3528
0.3615
0.3963
0.3483
0.3565
0,382
Standard deviation
+4.60-3
27700-
+3.20-3
+2.1e3
+1.003
#1.003
#17033
11.2033
113633
#1900
44.6e-4
+1-2-3
ETTh2
0.272
0.2866
0.3288
0.2757
0.2855
0.3157
0.2728
0.2854
0.3038
0.2765
0.2865
0.2929
Standard deviation
33.5e33
4: 3.44
3:22
3.2.2.3
166.2644
1.7.0e4
3.3.8-3
1..3
4.1.7e44
ETTml
0,3152
0,3273
0.385
0.3046
0.3115
0.3532
0.3016
0.3067
0.3428
0.2992
0.3059
0.3378
Standard deviation
+2.60-3
#9.404
+2770-3
117003
28.70-
#1.303
#1.0003
#6.1004
#1.1e-3
#1.e-3
#7.1c-4
#1.1--
ETTm2
0.1637
0.1745
0.2151
0.1614
0.1714
0.1943
0.1614
0.1729
0.1846
0.162
0.1763
0.1793
Standard deviation
3
19.4e 4
3
1..88-3
1..99 4
1.5.6e
1.1.3-3
11.1e.3
1.29e.4
1.7.8e44
1.
1.3.7
Weather
0.1459
0.1478
0.2024
0.1446
0.1464
0.1852
0.1442
0.1489
0.1739
0,1442
0.1466
0.166
Standard deviation
11.6e.4
16.5e-4
1:3.6e-4
1.12e.4
1.4.9e4
1.2.0e4
1.1.2e4
1.1.2e3
1...e.5
1.29e-5
1.2.ee4
1.2.6644
Electricity
0,1319
0.1346
0,1621
0,1309
0.1338
0,1461
0,1309
0.1342
0.1411
0,1306
0.1345
0.1378
Standard deviation
+2.604
77.004
#4.404
1.8004
#:5.20-4
#2.00-4
+2.30-4
#5.50-4
40-4
31.304
#7.80-4
4-.50-5
Traffie
0,3574
0,3604
0,4095
0,3518
0.3582
0.3874
0.3508
0.3596
0.3788
0,349
0.3579
0,373
Standard deviation
4
19.4c-4
1:2.7e-4
1.85e-4
1.7.ee4
1:1.6e-4
1:1.e:3
1.8.6e-4
1:1.0e44
1:.76:3
1:4.4e44
1:1.8e4
Avg. Improvements
3.34%
25.97
2.84%
19.58
3.34%
16.24
3.66%
13.63
Max. Improvements
6.19%
68.61
5.83%
56.8
6.65%
52.48
8.11%
48.73
Table 17: MSE of fine-tuning LTSM Timer for time series forecasting under 25% 50%, 75% and 100% % proportion of available data. SFF, FF, and TFS are smoothed full fine-tuning, full fine-tuning, and training from scratch, respectively.
Data proportion
25%
50%
75%
100%
Metbads
SFF
TFS
Exchange
0.0805
0.0865
0.1441
0.0802
0.0891
0.114
0.0802
0.0914
0.1026
0.08
0.091
0.0981
Standard deviation
+4.50-4
11.004
+2.003
45
+2.363
#9.904
1.60-
8o-4
47.604
11.2
0.3506
0.355
0.3788
0.3494
0.3573
0.367
0.3493
0.358
0.3593
0.3547
0.3709
0.36
Standard deviation
16.104
+5.80-4
11.003
+1.363
#844-4
+9.30-
+3.60-3
#1.26-3
ETTh2
0.271
0.2866
0.2891
0.273
0.2905
0,2775
0.2772
0.3042
0.2796
0.2737
0.3117
0.2777
Standard deviation
3:.5ee33
1.6e 3
4.3.6e-4
1:2.0e-3
4...6-4
1.2.9c4
4
1.4.8e4 4
1.5.2e-4
4.7.6e-4
1.3.8e44
1.6.0e.3
1..66
ETTml
0.248
0.3049
0,333
0.2955
0.3069
0.3189
0.2956
0.3092
0.3116
0.2954
0.3128
0.3093
Standard deviation
11.1e-3
+5.6e-4
+9.3e-4
11.0-3
#7.1e-4
11.36-3
47.36-
51-5-3
45.56-4
-1.le-3
ETTm2
0.1594
0.1707
0.1741
0.1605
0.1718
0.1627
0.1623
0.1838
0.1651
0.16
0.1784
0.1644
Standard deviation
19.7e-4
1.3e.-
4
1:3.8e4
4.5.964
1.43e-4
4:2.5-33
19.3e-
+1.00-3
1:1.4-3
11.le-3
Weather
0.144
0.1472
0.1627
0.1441
0.1523
0.1538
0.1466
0.1665
0.1559
0.1443
0.1612
0.1526
Standard deviation
+.22e-5
16.1e-
4:57e-5
+:2.ee4
4744-4
1.4.6-4
1:1.664
E1.le-3
17.36-
19.2e-
Electricity
0,1303
0.1344
0.1365
0.1301
0.1347
0.1327
0.13
0.1367
0.1326
0.1304
0.1344
0.1324
Stundard deviation
+17e-4
19.4c-4
+5.76-5
42.4c-4
19.50-4
11.3
4:3.6--4
4-..8c-4
16.5e-4
42.0c-4
45.40-4
18.0c-4
Traffic
0.3488
0.3582
0.3688
0.3497
0.3586
0.3552
0.3478
0.361
0.3606
0.3551
0.3599
0.3609
Standard deviation
+2.003
+9.20-4
+-20004
1..663
#-5.8e-4
+:3.2033
+:3.0e-3
+:277-4
#:2.20-4
#:2.5e-3
Avg. Improvements
3.79%
12.28
4.97%
6.82
7.52%
5.47
7.41%
4.64
Max. Improvements
6.94%
44.14
9.99%
29.65
12.25%
21.83
12.19%
18.45
23

---

Published as a conference paper at ICLR 2026
Table 18: MAE of fine-tuning LTSM Timer for time series forecasting under 1%, 2%, 3% and 4 % proportion of available data. SFE, FE, and TFS are smoothed full fine-tuning, full fine-tuning, and truining from scratch, respectívely.
Data propartion
1%
2%
3%
4%
Methods
SFF
Exchunge
0,2048
0.208
0.3946
0.2045
0.2083
0.3928
0.2036
0.2067
0.3923
0.2038
0.2071
0.3919
Standard deviation
+2.304
+3.904
3:336-3
14480-
4540-
#5.1o-3
+3.504
1140-
15.1e:3
14.30-
420004
#5.0c-3
ETTh1
0.3973
04101
0,5202
0.3962
04085
0.5184
0.3943
0.4072
0.5168
0.3939
0.4044
0.5055
Standard deviation
3336--
+1.60-
+:.50-3
+340-3
14.50-
#5.20.3
+3.50-
+6.20-5
15.1e
+5.30-3
#4.7e-
#4.1e:3
ETTh2
0.3362
0.3433
0.4109
0.3319
0.3401
0.3322
0.3394
0.4098
0.3339
0.3392
0.3944
Standard deviation
3..0e
1..1e-4
3.
1.3.2e44
45..4.44 4
3:200e3
1:5.2e-4 4
166.5e-5
3.2.4e33
1.5.2e-4
1e-3
1.27e-3
ETTml
0.4085
0.4166
0.5138
0.3838
0.3952
0,4507
0.3807
0.3909
0.4488
0.3755
0.3824
0.4303
Standard deviation
+2,50-3
+3.80-
+.000-
+:2.6033
4..80-4
11.90-3
1.9003
33.80-4
+:200-3
1188033
1.6604
#177-3
ETTm2
0.2586
0,2738
0.3275
0.2591
0.2679
0.3271
0.2523
0.262
0.2979
0.2541
0.2609
0.2982
Standard deviation
13.0e 3
1.22e
3
1.9ee
1..7e. 4
1..77.3
1.2.5e33 3
18.55 4
1.1e-
1. 3
1.7.8e55
1.
Weather
0.2028
0.2062
0.2942
0.1996
0.2022
0.2808
0.1978
0.2015
0.2704
0.1954
0.1984
0.265
Standard deviation
4
1.40e-4
1.2.7e33
1.2.6e44
1...e-
1.2.5e33
1.7.0e 4
1.5.4e44
1.5.8e44
1.2.3e44
1.3.4e-5
1.4.ee4
Electricity
0.2342
0.2367
0.3243
0.2307
0.2334
0,2976
0.229
0.2315
0,2817
0,2292
0.2318
0.2702
Standard deviation
+8.30-5
#6.80-5
1o-3
+3.4044
#8.80-5
1.2033
#5.00c4
1o-1
#9.60-4
#2.70-
45.50-4
Traffic
0,2672
0.27
0.3921
0,2611
0.2636
0,3452
0.2575
0.26
0.3179
0.2559
0,2585
0.3037
Standard deviation
4
12.4c-4
3:177:
1.21e.4
1.1.9644
1::.4e33
1.6.8-5
1:7.4e:5
1.5.0e-4
1.9.9e5
4.2.5e-4
4:2.0ee4
Avg. Improvements
2.25%
27.77
2.1%
25.24
2.12%
23.22
1.72%
21.26
Max. Improvements
5.55%
48.1
3.28%
47.94
3.7%
48.1
2.61%
48.0
Table 19: MAE of fine-tuning LTSM Timer for time series forecasting under 5%, 10% 15% and 20 % proportion of available data. SFF, FF, and TFS are smoothed full fine-tuning, full fine-tuning, and training from scratch, respectively.
Data propottion
5%
10%
15%
20%
Metbads
SFF
TFS
Exchange
0.2044
0,2078
0.3867
0.2019
0,2059
0.3252
0.2009
0.2047
0.3085
0,2008
0.2072
0.2936
Standard deviation
+2.004
1.700
+4.963
+9.004
1.80
#1.663
16.1o4
1180
#100-3
#1563
#2.96-3
0.3923
0.011
0.462
0.3882
0.3952
0.4222
0.3884
0.3946
0.4244
0.3856
0.3915
0.4173
Standard deviation
+3,003
+3,804
11.663
1180-4
#6.80-4
11.503
-5.20-4
11.3
1o.
11.1
ETTh2
0.3325
0.3399
0.3826
0.3327
0.339
0.3686
0.3348
0.3385
0.3583
0.3332
0.3382
0.3512
Standard deviation
3
1.9.ee4
3.
1.2,ee
19.5e-4
4
4
1.17e-3
1..99-
12.le.
12.6e-3
1...9 4
1...e.
ETTml
0.3735
0,3801
0.4283
0.3663
0.3706
0.4054
0.3644
0.3678
0.3967
0.3631
0.3679
0.3926
Standard deviation
11.333
+ 1..
+2.00-3
48.20-4
1-456-5
13.1e-4
47.60-5
4--8
#8.20-4
+2.56-5
1.00-3
ETTm2
0.2522
0.2599
0.2976
0.2516
0.2584
0.2787
0.2532
0.2576
0.2707
0.2527
0.2622
0.2672
Standard deviation
1.66--
1..66-5
1.666-
1..3e-
1.000044 4
1:..ee-
4:5.0c-4
4:2.9e4
4:2.0e-4
1.4.e.4
1:4.0e4
4.2.5e4
Weather
0.1936
0.1968
0,257
0.1928
0.1967
0.2385
0.1928
0.1978
0.2271
0.1932
0.1988
0.2198
Standard deviation
+2.3e-
12.2e-5
4:3.ee-
4:14e-4
#:.36-55
16.96-4
d:2.9e-5
+:300e4
11122-3
4-3.8644
Electricity
0.2273
0,2303
0.2616
0.2263
0.2319
0.2444
0,226
0.2312
0.2384
0,225
0.23
0.2346
Stundard deviation
19.1e-5
45.4655
14.5e-4
43.2e-4
4:3.0e-
11.80-4
1e.3
1o-
44.5c-4
45-00e4
11.40-4
Traffic
0.2537
0.2571
0.2948
0.2501
0.2554
0.2754
0.2488
0.2573
0.2678
0.2473
0.2573
0.2631
Standard deviation
+6.5e-5
+1.804
+2290-4
+9.1e-4
199.ee55
#:2.3e4
+1.36-3
+1.5
21-20-3
Avg. Improvements
1887%
19.39
1.98%
14.37
1.9%
11.57
2.48%
9.94
Max. Improvements
2.96%%
47.14
2.63%
37.92
3.3%
34.88
3.89%
31.61
24

---

Published as a conference paper at ICLR 2026
Table 20: MAE of fine-tuning LTSM Timer for time series forecasting under 25% 50%, 75% and 100% proportion of available data. SFF, FF, and TFS are smoothed full fine-tuning, full fine-tuning. and training from scratch, respectively.
Data proportion
25%
50%
75%
100%
Methods
SFF
FF
TFS
SEF
FF
TFS
SEF
FF
TFS
SFF
FF
TFS
Exchange
0.1997
0,2095
0,2782
0,2122
0.2421
0.2002
0.2136
0.2266
0.2006
0.2153
Standard deviation
4.
1.33e 4
3.
11.0e-
12.6e
1.22e
11.7e.3
17.8e-4
11.2e-4
1..3e.
1..ee
14.00.4
0.3879
0.3858
0.3907
0.3956
0.3921
0.3955
Standard deviation
14.5e.4
1.5.e 4 1.9.8e4
1.3.0e 4 1.2.4e4 4 1.6.6e44
1.2.4-3 3
1.5.0e4 4
1.5.5e44
1..4e:3
1.l.le3
1..66.4
ETTh2
0.3325
0.337
0.3467
0.3327
0.3421
0.3378
0.3387
0.3516
0.3435
0.3353
0.3531
0.3436
Standard deviation
1133-3
4.770-
Ae-4
# 4e-3
# 1.-4
#7.80-4
#557c-4
1.2-0
22.4e-3
ETTml
0.3599
0.3656
0.3877
0.3576
0.3667
0.374
0.3566
0.3698
0.3696
0.3558
0.3703
0.3669
Standard deviation
3177e3
1.7e7-
4.87e-4
1.88--3
1:1.0.4
16..ee4
1:2.0e-3
d.5.1e-4
1:.9c-4
1:166-3
d-2.Ne-4
1:4.e-4
ETTm2
0.249
0.2562
0.2626
0.2497
0.2566
0,253
0.2534
0.2635
0.2571
0.2472
0.2591
0.2533
Standard deviation
16.4-4
14.66e-
4e2.e-4
4:58e-44
45.66--
#:l +..66-4
1:3.8c-4
+: 1.-33
46-4
11-5
1e-1
45.7e-4
Weather
0.1921
0.1961
0.2146
0.1929
0.199
0,2037
0.1971
0.217
0.2098
0.192
0.2046
0,203
Standard deviation
+2.9c4
14.36-4
19.0e-5
12.1
11.5633
4.2.c-4
+: 1.-33
17.30-4
11.50-3
16
Eloctricity
0.2245
0,2289
0,2328
0.2247
0,2273
0,228
0.2245
0.2292
0,2274
0.2239
0.2273
0.2268
Standard deviation
+4.2e-4
184c-4
+1.6e4
#2.70-4
#2.36-4
1-900-5
+.4.ce4
1:4.ee4
#2.1e-3
+44.204
#3.3e-
#22.c-4
0,2463
0.2599
0.2486
0.2512
0.2517
0.2553
Standard deviation
3199.
11.8e.3
1.55e
11.4ee3
1.1.7
:1.66e-4
1.4.6e4
1.9.0e-4
1.2.6e-3
4.1.0.4
-22.0-3
11.33-3
Avg. Improvements
2.27%
8.8
2.56%
4.58
3.99%
3.9
3.97%
3.72
Max. Improvements
4.68%
28.22
6.08%
17.68
9.17%
11.65
6.83%
9.19
Table 21: MAE of Smoothing the loss landscape then perform zero-shot forecasting.
ETTh1
ETT12
ETTml
ETIm2 +Smooth
Weather +Smooth
Electricity Timer
Traffic Tiner
Timer
+Smooth
Timer
+Smooth
Timer
+Smwoth
+Smooth)
+Smooth
MAE
0.434
0.418
0.359
0.352
0.61
0.607
0,294
0.236
0.231
0.312
0.308
0.343
0.335
=0
-500-4
+0
+0
+2.76-3
+1.20-3
=0
1o-3
#0
+ Ae-4
#0
Imp
3.69%
0.49%
2.0%
1.28%
2.33%
TimesPM 48mooth TimesPM 4Smooth TimesPM 4Smooth /Tm..M +Smooth Time FM Smooth Tim +mmoot FM +Smooth
MAE
0.559
0.55
0.541
0.419
0.749
0.682
0.404
0.335
0.278
0.262
0.756
0.747
0.867
0.834
Std.
-0
-144-3
1:
#2.1e-3
1:0
+:30e-3
=0
4-8.ee-4
=0
4:3.6e33
10
+:4.0e-3
10
+:400-3
Imp.
1.61%
22.55%
8.95%
17.08%
5.76%
1.19%
3.81%
25

---

Published as a conference paper at ICLR 2026
Table 22: The complete anomaly detection results on 250 datasets, reporting the average MSE values of anomalous segments in each dataset under four random seeds, where higher values are better, because this reduces the risk of normal segments being misjudged as anomalies.
Index
11
14
(SFF)
0.051
0.01
0.004
0.435
0.011
0.09
0.112
0.016
0.046
0.166
0.129
0.045
1.112
0.042
0.13
1 (FF)
0.031
0.003
0.002
0.706
0.004
0.002
0.039
0.006
0.005
0.003
0.055
0.033
0.071
0.024
0.096
(TFS))
0.019
0.005
0.003
0,262
0.005
0.003
0.032
0.009
0.028
0.002
0.139
0.008
0.652
0.034
0.1
(SFF)
0.107
0.077
0.002
0.164
0.005
0.002
0.093
0.004
0,082
0.065
0.071
0.053
0.27
0.324
0,043
2 (FF)
0.012
0.024
0.0
0.11
0.005
0.0
0.036
0.002
0.024
0.038
0.069
0.013
0.231
0,213
0.023
(TFS)
0.012
0.026
0.001
0.103
0.006
0.001
0.055
0.002
0,043
0.049
0.02
0.02
0.13
0.232
0.028
(SFF)
0.024
0.059
0.007
0,138
0.203
0.583
0.085
0.04
0.314
0.238
0.118
0.24
0.004
0.07
0.029
(FFF)
0.001
0.014
0.003
0.092
0.114
0.206
0.038
0.01
0.297
0.195
0.1
0.096
0.002
0.05
0.011
3 (TFS)
0.001
0.013
0.004
0.094
0.124
0.529
0.05
0.026
0.081
0.117
0.108
0.144
0.001
0.05
0.0
4 (SFT)
0.121
0.32
0.106
0.085
0
0.348
0,512
0.129
0.54
0.122
0.027
0.413
0.36
0,033
0.006
(FF)
0.026
0.095
0.073
0.013
0.016
0.062
0.259
0.003
0.223
0.013
0.026
0.396
0.352
0.005
0.003
(TFS)
0.087
0.184
0.175
0.031
0.079
0.095
0.323
0.005
0.213
0.012
0.015
0.212
0.218
0.004
0.003
(SH9)
0.912
1.184
0.043
0,255
0.017
0,072
0.093
0.042
0.074
0.032
0.062
0.331
0.255
0.058
0,021
5 (ITF)
0.618
0.078
0.014
0.114
0.001
0.038
0.029
0.014
0.026
0.014
0.007
0.224
0.007
0.063
0.003
(TFS)
0.602
0.133
0.036
0.113
0.0
0.029
0.053
0.029
0.019
0.015
0.02
0.238
0.004
0.019
0.013
(SFF)
0.29
0.015
0.034
0.142
0,105
0.497
0.352
0.134
0.013
0.021
0.141
0.002
0.14
0.011
0.068
6(FF)
0,334
0.004
0.011
0.12
0.007
0.432
0.156
0.002
0.005
0,14
0.001
0.135
0.006
0.002
(TFS)
0.158
0.005
0.007
0.074
0.007
0.169
0.051
0.001
0.002
0.001
0.081
0.0
0.104
0.007
0.003
7 (SFF)
0,345
0.003
0.013
0.039
1.293
0.131
0.055
0.318
0.06
0,111
0.097
0,026
0.82
0.198
0,314
7 (FFF)
0.028
0.001
0.006
0.025
0.779
0.063
0.005
0.065
0.016
0.11
0.031
0,006
0.583
0,085
0.055
(TFS)
0.13
0.001
0.014
0.037
0.159
0.114
0.036
0.332
0.015
0.073
0.026
0.006
0.267
0.078
0.094
(SFF)
0.182
0.007
0.119
0,135
0.182
0.004
0.233
0.073
0.005
0.117
0.081
0.045
0.518
0.014
8 (FF)
0.155
0,004
0.036
0.058
0.201
0.003
0.07
0.157
0.051
0,007
0.04
0.04
0.038
0.41
0,008
(TFS)
0.105
0.009
0.111
0.142
0.145
0.003
0.135
0.073
0,046
0.001
0.105
0.039
0.036
0,402
0.014
9 (SFT)
2,133
0.067
0.126
0.118
0.151
0.728
0.213
0.174
0.141
0.035
0.07
0.082
0.05
0.471
0.008
9(FF)
0.795
0.03
0.009
0.026
0.072
0.458
0.067
0.151
0.077
0.008
0.035
0,029
0.006
0.478
0.002
9 (TFS)
1,599
0.031
0.096
0.03
0.144
0.437
0.079
0.089
0.08
0.024
0.048
0.055
0.027
0.203
0.002
(SFF)
0.066
0.031
0.478
0.029
0.055
0,004
0.087
0.069
0,202
0,423
0.835
0,103
0.981
0.085
10 (FF)
0.028
0.015
0.094
0.004
0.014
0.001
0.043
0.02
0.017
0.135
0.431
0.008
0.056
0,255
0.008
(TFS)
0.027
0.167
0.005
0.06
0.001
0.045
0.014
0.043
0.273
0.462
0.008
0.012
0,382
0.005
(SFF)
0.054
0.073
0.485
0.005
0.143
0.005
0.054
0.176
0.003
1.351
0.048
0.007
0.003
0.046
0.596
(FF)
0.014
0.057
0.453
0.049
0.023
0.031
0.143
0.002
0.077
0.009
0.001
0.02
0.158
11FS))
0.049
0.05
0.35
0.001
0.017
0.001
0.027
0.108
0.002
0.115
0.007
0.003
0.005
0.039
0.371
(2FF)
0.511
0.026
0.184
0.082
0.085
0.26
0.637
0.221
0.092
0.096
0.13
0.014
0.002
0.011
0.137
12 (FF)
0.349
0.016
0.1
0.016
0.006
0.281
0.091
0.085
0.026
0.102
0.132
0.013
0.001
0.003
0.059
(TFS)
0.126
0.025
0.107
0.022
0.159
0.034
0.297
0.071
0.04
0.057
0.07
0.012
0.002
0.004
0.091
13 (SFP)
0.045
0.067
0.003
0.005
0.075
0.001
0.005
0.045
0.16
0.172
0.034
0.022
0.204
0.023
0.033
13 (FF)
0.026
0.025
0.001
0.002
0.01
0.002
0,002
0.028
0.01
0:015
0.021
0.018
0.118
0.001
0.015
(TFS)
0.019
0.038
0.001
0.004
0.006
0.002
0.002
0.021
0.014
0.005
0.034
0.016
0.162
0.005
0.03
14 (SFF)
0.98
0,579
0.012
0.022
0.014
0.003
0.728
0.02
0.002
0.354
0.042
0.023
0.055
0.068
0.132
14 (PF)
0.248
0.226
0.003
0.009
0.006
0.001
0.683
0.009
0.0
0.244
0.041
0.018
0.032
0.059
0.074
14 (TFS)
0.166
0.406
0.01
0.009
0.011
0.002
0.803
0.008
0.0
0.43
0.025
0.009
0.053
0.041
0.039
(SFF)
0.007
0.052
0.094
0.471
0.19
0.003
0.182
0.522
0.193
0.003
0.028
0.31
0.005
0.006
0.356
15 (PF)
0.004
0.013
0.072
0.263
0.015
0.002
0.0988
0.315
0.038
0.001
0.016
0.183
0.001
0.0
0.073
(TFS)
0.007
0.014
0.001
0.447
0.035
0.001
0.114
0.186
0.182
0.003
0.023
0.222
0.001
0.0
0.222
(SFF)
0.003
0,283
0.335
0.013
0.141
0.185
0.165
0.638
1.156
0.14
0.009
0.043
0.105
0.015
0.013
16 (FF)
0.001
0.175
0.275
0.03
0.049
0.129
0.016
0.299
0,694
0.048
0.006
0.031
0.073
0.009
0.004
16 (TFS)
0.001
0.102
0.335
0.009
0.142
0.109
0.1
0.153
0.774
0.004
0.012
0.033
0.026
0.009
0.008
(SFF)
0.067
0.049
0.358
0.008
0.031
0.107
0.09
0.321
0.024
0.005
17 (FF)
0.031
0.02
0.405
0.004
0.023
0.091
0.044
0.051
0.015
0.001
(TFS)
0.024
0.032
0.211
0.002
0.026
0.075
0.084
0.122
0.022
0.002
26

---

Published as a conference paper at ICLR 2026
Table 23: Standard deviations 250 anomaly detection datasets under four random seeds.
Index
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(SFF)
36.8e
3.30e.
d 1133e
1
1..2e.
1.0.12
1.0e22
3.
3.6.
0.11
1.6e.2
2.9e.2
0.68
9.2e.3
6.8e-3
(FF)
1.2e
42.3e-3
479c4
+043
#1.763
41.96-3
5.0e-2
1.3e-3
2.26-5
1.3e-3
7.56-4
1.8e22
5.8c-2
1.0e-3 8.2c-4
(TFS)
111-2 #1.4033
+244-4
#0.15
#270--
#1.70-3
2.0-2
4.60-
1.80-2
144-3
6.36-2
3.90-3
0.74
5.30-3
3.40-2
(SFF)
15.66-2
#39e-2
#1.46:3
11.3e-2
#1.663
#6.944
1.562
3.8o-4
2.8e-2
2.1e-2
1.0e-2
2.0e-2
2.76-2
4.1e-2 3.1e-3
(
3
1.4.p.3
13.9e4
121ee4
12.ee3
1.3.e.4
2.6
8.7e44
1,7e:2
3.7e:3
1.7e:2 7.3e:3
4.2e/2
6.3e 4.0e:3
(TFF))
1e-e 4:88e-4
48.5e4
17.e-2
44.5e-
447cc4
6.3e-3
3.0e-4
3.0e-2
4.7e-3
4.7e-3
1.5e-2
5.7e-2
0.12
2.4e-3
(SFF)
11.le2 12.6e22
1..led
1:.4e.2
12.4e2
1996e
2.4e:2
6.9e
2.3e:2
3.9e-2
4,9e-3
5.3e:2
1.1e:3
1.3e22
1.6e-2
((FF))
42.6e44
16.2e-3
439.a4
17.7e4
23.2622
42.2e4
1.2e-2
7.3e-4
5.5e-2
1.96-2
3.5e22
5.4e22
2.7e-5
2.3e-2
8.4e-3
(IFS)
1844e- 16.9e:
4:2.2e:
122222
16.4e22
10.18
1.0e22
4.5e:3
1.5e=2
4.1e:2
4.5e:3
7.5e:2
1.2e-4
2.3e:3
1.4e-4
(SFF)
+:4.2.2
+3.40-2
+1.3
+3.0022
+176-2
#7.8-3
0.1
9.1c-2
0.23
7.70-2
3.80-3
3.60-2
0.11
3.1o-2
3.40-4
4 (FFF)
3:5.8e3
11.5e:2
17.7e4
1:44e3
33.0e3
4:2.e22
0.18
1.9-3
1.9e-3
5.2e-3
5.7e-5
5.5e-2
2.6e-2
4.3e-3 3.4e-4
(TFS)
45.80-3
#5.20-2
#7.80-
#2.102
#5.30-
#1.102
0.22
9.804
3,90-2
5.70-3
3,50-3
2.56-2
5.50-2
2.50-3
9.9c-
(SFF)
10.14
10.4
23.ee3
4333-3
#2.6-2
1l.le22
3.4-2
1.5e-2
1.96-2
8.7e-3
4.6e-2
2.4e-2 0.35
2.0e-3 1.0e-3
(FF)
+0.15
#390-2
+5.1
#5.0c2
=5.704
118--2
1.0-2
1,6-2
1.36-2
9.36-3
4.60-
6.96-2
4.80-3
1.20-3 2.6c-3
(IFS)
=0.1
20.11
42,4e22
14.6e-2
#7.ee5
17.2e-3
1.5e-3
4.7e-3
6.3e-3
3.0e-
4.60-3
8.Ne-2
2.0e-3
4,4e-3 5.9e-3
6 (SFF)
141e22
14.6e33
11.7e2
11.622
254e2
11.2e22
0.2
9.4e
1.6e:2
1.3e-2
5.1e:3
7.5e:5
4.0e:3
1.0e:3 9.2e-2
(FFF)
16.8e-
129e3
48.264
16.9-
16.2633
47.6-2
5.6e-2
1.7e33
6.0e-3
5.4e3
4.7e-3
9.2e-5
3.3e-4
3.4e-5 1.8c-3
6 6(TPS)
+68co2 15.10
+236-3
#436-3
=5.003
7.8-3
4,9c4
3,904
3.0e4
1.1o-2
2,90-4
1.402
1.30:3
1.1o-3
(SFF))
+0.28
#6.804
#3.5e-3
14403
+1.0
19.30-3
1.1e2
4.9022
199-2
4.4e-3
6.16-3
6.76-3
0.16
1.8e-
6.2c-3
(FF))
1:20e:3
17.6e4
13.1e:
127e3
1041
12.4e22
2.
5.1e:2
1.9e-4
2.7e:2
3.4e-2
2.1e:3
0.1
5.4e-2
2.2e:2
(TFS)
13776. 1..3e.
19.1e.3
1.99.
118822
17.0a3
1.4e:2
2.76/2
3.0e-3
2.5e-2
2.1e:2
1.5e-3
4.0e:2
5.1e:2
1.1e.2
(SFF)
#2.304
45.86-3
#2.8022
#320-2
11.704
2.8-22
1.2o-2
2,5e-2
1,5e:3
1.5e2
5.1c-2
5.4e33
4,9e-2
4.5c-
(FF))
+4.8e-2
19.80-5
#1.20-
+1.102
-2.102
#1.503
100-2
2.60-2
3.80-3
5.60-3
1.36-2
2.00-2
8.404
1.1e-2 1.4--3
(IFS)
+5.9e-3
#5.5ee3
#1.6e:3
1.00e22
=20ee2
19.2e44
2.0e2
4.9e-2
1.0e22
1:2e-4
2.7e-2
2.2e-2
1.6e:3
8.6e-2
3.8e:3
(SFF)
10.36
15.66
10.1
2
129
1..5ee2
2.
2.4e
3.1e:2
1.2e-2
1.8e-2
3.4e-2
3.3e-2
1.6e.3 5.5e.3
(.FF
+0.46
#2.0e3
1346-22
#2.36-2
18.5e44
7.9622
3,2e-2
2.8e22
9.5c-4
5.46-4
4.8c:3
1.96:3
1.3e:3 6.1e-
(TFS)
+0.96
-2.302
+0.12
#4.06-2
-9.20-3
#8.76-2
8.76-2
2.60-2
2.40-2
1.76-2
1.50-2
6.96-3
1.8o-2
4.7o-2
6.4c-
(SFF)
16.5e-
17.5ea
10.27
12.36-2
12.9-2
43.4e44
2.1e22
2.26-2
5.0e-2
2.4e-2
0.13
6.3e-2
4.9e-3
0.16
9.7c-2
(FF)
129e:
3
17.96
4
49.2.44
120.e
5.4e:3
2.2e-2
3.5e
2,9e-3
4.8e:2
4.3e:3
9.0e-3
3.3e.3
(TFS)
11.5e.2 1..e
10.16
11.e
25.36.2
42.9c.
6.8e.3
7.0e:3
8.6e:3
0.15
0.24
2.8e:3
2.2e:3
0.18
1.3e-3
11 (SFF)
15.4e:2 1:.6e:2
13.2e.2
16.6
15.0e
1.7ee
3.6e:
2.4e-2
1.3e:3
0.86
4.7e-2
3.6e:
7.6e-4
6.4e:3
0.11
(HB)
+1.2.22
#6.8c4
+3.1e-2
+1.633
+3700-
11.ee2
4.7e-3
6.1e-3
1.1e-3
7.50-2
4.60-3
9.30-4
2.96-4
1.3e-2
0.19
(TFS)
16.3e-2 11.1e2
13.3e.
13464
19.3e3
#2.6c4
7.4e-3
2.5e-2
8.6e-4
2.96-2
3.1e:3
1.0e-
3.6e:3
9.9c-3
8.9c-2
(SFF)
+5.20-2
#1.433
+2.20-3
#41e02
#580-2
+2.6-2
0.39
1.50-2
2.76-2
4.50-3
1.20-2
1.50-3
2.30-4
6.3c-4
2.0e-2
(PP)
11.7e22
#5.2e
11.22.2
3388e
=20ee
13.3e-2
7.9-3
5.2e-2
9.20-3
4.8e-2
8.3e:3
7.4e:5
6.3e:5
1.le44 2.8-2
(TFS)
+0.15
#8.76-3
=1.8c-2
+38803
=3.6e-2
#4.1e-
0.35
7.3022
2.0e-2
2.00-2
1.4e-2
1.10-3
6.90-4
2.1e-4 1.1e-2
(SFF)
18.6e33
#3.9-22
46.5e44
11.0e-
#38e-2
+2.2e4
7.7e-4
2.56:3
0.1
0.19
3,5e:
2.1e:3
3.9e-2
1.6e-2 6.8c-3
(FF)
+1.602
#7.76-
#3.604
#4.644
-2.663
+28644
4.60-1
1.4-2
1.40-
5.50-3
1.46-
7.40-3
1.60-2
7.60-4 3.1e-3
13 (TFS)
118e-2
#3.3e-2
#3.le-
+2.5e:3
#22e33
#6.2e4
2.2e44
5.4e-3
2.3e-3
6.1e-4
2.1033
5.90-3
5.9e-2
5.4e:3 5.8e-3
(SFF)
10.91
10.15
13.ee3
136e
3
4
2.4e
3.6e
5.1e
2.9e-2
6.6e
2.8e
1.0e-2
6.5e.3 2.4e-2
14 (PP)
45.46-2
14.4622
12.le.
129.
43.7e4
2.4e.3
8.8e
2.3e-4
8.8e-2
2.5e-2
1.0e-2
5.6e-3
1.3e-2 4.3e-3
(TFS)
-3.1
+0.19
35.6-3
39.063
=326-
45.36-4
1.36-2
1.30-3
1.064
0.4
1.60-2
5.56-3
9.46-3
1.4e-2 2.56-2
(SFF)
+
+3.962
18.633
#1762
26.76-2
#1.4e33
9.9-3
0.11
5.70-3
8.1e4
8.004
3.6e-2
2.00-3
3.1e-3
9.76-2
(FF)
18les
17.e3
170c4
11263
1.22
3.6e
0.13
3,2e-2
1.5e-4
3.5e:
9.7e-2
3.0e-4
1.8c-4 1.9c22
(TFS)
3..3e.3 45.9e.3
45.le4
1377222
2200e
43.6e
5.1e.2
9.3e.2
1.5e:2
1.4e:3
8.2e.3
5.1e:2
1.2e:3
1.6e-4
7.8e-2
(6
13.644 14.26-2
-1.5e-2
#7.204
#226-2
#3.0-3
1.9e-2
0.13
0.12
5.8c-4
1.8e-3
8.4e-4
1.0c-2
2.56-3
1.8c-3
(FF)
1644-4
+1.70-
+3.5e-2
#3.7622
=1.322
+1.0-
4,50-
0.2
8.60-2
6.50-2
2.00-3
1.8o-2
4.8o-2
1.le-2
4.6c-3
(IFS)
154e-4
#2462
10.13
117763
11882
268-2
5.7e-3
0.18
3.8e-2
1.1e-3
6.6e-3
1.1e-2
2.1e22
5.5c-4
L.le2
(SFF)
11.e
11.22
177e2
1300e
134e3
168e
4.2e
0.13
1.4e3
4.2e-3
(FF)
118e-2
33.0c-3
13.4-3
241e3
11.66
1.8e-2
5.76-3
5.6e-4
8.5e-5
17 (TFS)
+1.le2
#1.663
#7.76-2
438c-4
3300-
#6.56-3
4.70-3
4.90-3
7.9-3
1.4033
27

---

Published as a conference paper at ICLR 2026
Table 24: Complete standard deviation and MSE of applying our smoothed fine-tuning (SFF) on other LTSMs TimesFM and MOMENT.
Data proportion
25% (TimesFM)
100% (TimesFM)
25% (MOMENT)
100% (MOMENT)
Methods
SFF
SFF
Exchange
0.1139
0.1276
0.1209
0.1149
0.1452
0.1199
0.1502
0,2648
0.1564
0.1064
0.1448
0.1091
Standard deviation
2.0e-3
2.0-4
6.3c-4
1.7e22
2.3e-3
2.4e-3
4.6o-4
3.60-3
5.8c-4
1.4e41
2.6c-4
ETTh1
0.3955
0.4322
0.4638
0.406
0.5101
0.4358
0.4287
0.4454
0.454
0.3757
0.3951
0.387
Standard deviation
1.9e3
2.4e-2
6.0c.4
3.6e-3
8.8e-3
2.0g-3
2.1
9.9c-4
1.8e-3
4.0c-4
6.5e.5
1.4e3
ETTh2
0.3232
0.3325
0,3198
0.3483
0.347
0,3199
0.3328
0.3326
0.2818
0.2936
Standard deviation
2.90-3
9.003
9.50-4
2.00-3
3,3c-3
4.7c-3
1,40-3
2.60-4
1.5c-3
7.8c-4
4.0p-5
1,6c-3
ET
0.3429
0,4001
0.3903
0.3478
0.3756
0.3926
0.3487
0.3587
0.3538
0.3139
0,3148
0.3272
Standard deviation
3.60-3
7.30-3
7.20-4
2,90-3
3.0c-2
4.8c-4
1,20-3
1.20-4
6.6c-4
1.0c-4
3.10-5
2.0c-3
ETTm2
0,1983
0,2061
0,2091
0,2026
0.2122
0.225
0,1793
0.192
0.1846
0.1692
0.172
0.1736
Standard deviation
3.7e.3
2.6e-3
4.0e
1.5e 3
6.8e
3.8e
2.1e
5.0e
6.
3.7e-4
3.7e
9.5e
Weather
0.0865
0.0885
0.1995
0.082
0.1184
0.1902
0.1673
0.1682
0.169
0.1548
0.1558
0.161
Standard deviation
4.76-
5.6e-3
4.50-3
1.1o-2
3.0c-2
4.2c-3
1.104
1.4-4
1.7c-4
1.9c-4
2.20-4
1.4c4
Avg. Improvements
7.55%
16.21
15.35%
16.18
10.28%
3.25
6.34%
3.54
Max. Improvements
30.74%
43.28%
26.52%
Table 25: Complete standard deviation and MAE of applying our smoothed fine-tuning (SFF) on other LTSMs TimesFM and MOMENT.
Data proportion
25% (TimesFM)
100% (TimesFM)
25% (MOMENT)
100% (MOMENT)
Methods
SEF
FF
TFS
Exchange
0,2414
022519
0.2497
0,2422
0.2703
0.2472
0.282
0.3844
0.2894
0,2322
0.2751
0.2369
Standard deviation
9.9044
4.36-3
4.2-4
8.40-4
1.8o2
1.70-3
2.38-3
3.30-4
3.4o-3
5.80-4
1.0c4
3.1o-4
ETTh1
0.405
0.4226
0.4526
0.4149
0.4567
0.4344
0.4386
0.4455
0.4559
0.4022
0.4144
0.4122
Standard deviation
4.3e
3.0e
4.5e-4
3.0e
2.7e-3
5.3e
1.2c.3
7.4e-4
8.3e-4
4.2e-4
2.6e:5
1.3e-3
ETTh2
0.3704
0.3782
0.3784
0.35
0.3514
Standard deviation
1.26-3
3.90-3
9.30-
8.36-4
2.6e-3
1.4-3
1.40-3
3,50-4
9.3c-4
3.30-4
4.16-5
8.5c-4
FTTml
0.3851
0.4119
0.4223
0,3892
0.3992
0.4227
0.3938
0.4054
0.4013
0.3783
0.3787
0.3854
Standard deviation
2.1e-3
3.8c-3
4.3e-4
3.20-3
1.5e-2
7.6c-4
1.30-3
1.3e-4
4.6e-4
1.6e-4
2.4e-5
7.5e-4
ETTm2
0.2823
0,2847
0.2925
0.2748
0.2851
0.3119
0.2671
0.2769
0.2724
0.2587
0.2613
0.2638
Standard deviation
1.0e-
1.4e33
3.8c-4
1.9e-3
5.1e:3
4.4e-3
6.3e-5
4.8c-4
4.9c44
1.4e-3
1.2e-5
7.5e-4
Weather
0.1135
0.1161
0.2523
0.1025
0.152
0.2424
0,2213
0.2247
0.2231
0.2107
0.2114
0.2142
Standard deviation
3.3e-3
6.7-3
4.2c-3
1.46-2
4.0e-2
4.2c-3
1.1e-3
1.2e-4
3.6c-4
1.
4.0c-4
2.5e-4
Avg. Improvements
3.04%
13.84%
10.05%
14.88%
6.46%
2,22%
3.78%
2.12%
Max. improvements
26.64%
3.79%
3.13%
28

---

Published as a conference paper at ICLR 2026
Table 26: Full standard deviation and MSE of comparing smoothed full fine-tuning (SFF) with linear probbng (LP) and linear-probing then full fin-ttning (LPFF). The average performance is reported for each group of three different data proportions, e.g., "Avg. on 1 1%, 2%, 3%"
Data proportion
Avg. on 18, 2%.
Avg. on 4%, 5%,
Avg. on 20%
Avg. on 50%, 75%,
100%
Methods
SFF
LP
LPFF
SFF
LP
LPHF
SFF
LP
LPFF
SFF
LP
LPFF
Exchange
0.0856
0.5943
D
0,0848
0,5906
0.4186
0,0816
0.563
0.1743
0,0812
0.474
0.0962
Standard deviation
7.3e-3
3.9e-3
3.7e-4
7:2e-3
6.76-3
6.1e4
6.6e-3
7.4e-3
8.3e-4
4.7e-3
1.9e-3
ETThl
0.3722
0,8806
0.7171
0,3641
0.8594
0.6367
0,3523
0,7955
0.4127
0,3529
0.6356
0.3731
Standard deviation
6.50-3
9.20-3
3.60-3
4.70-3
8.20-3
9.20-3
1.4e-3
6.7e-3
3.40-3
1.30-3
4.5c-3
1.80-3
ETTh2
0.28
0.4427
0.4026
0.278
0,4375
0.3707
0.2768
0,4234
0.3113
0.2758
0,3849
0.3001
Standard deviation
1.5c-3
7.3c-3
3.80-3
2.80-3
6.8c-3
1.60-3
2.60-3
5,8c-3
1.5c-3
9.70-
3.1e-3
1.76-3
ETTml
0.3448
1,466
0.7038
0.3162
1.0043
0,4772
0.301
0,8975
0.3245
0.2976
0.66088
0,3124
Standard deviation
3.5e-3
2.6e
3.3e-3
2.6e-3
2.3e:2
4.5e.3
1.0e 3
1.8e-2
6.8e
1.5e-3
9.1e-3
1,2e-3
ETTm2
0.1723
0.3555
0.3024
0.1663
0.3499
0,2559
0.1623
0,3297
0.1847
0.1616
0.2771
0,1804
Standard deviation
2.3c-3
6.2c-3
3.10-3
2.10-3
5,90-3
2.00-3
1.0c-3
4.90-3
9.8c-4
6,20-4
2,70-3
1.60-3
Weather
0.1515
0.324
0.2478
0,146
0,3082
0,1741
0.1441
0,2699
0.1481
0,1453
0.2013
0,1565
Standard deviation
4.7e-4
4.2e:3
4.1e:3
1.9e
3.4e-3
3.5e:
6.62
1.9e-3
5.6e
3.6e-4
8.2e-4
1.1e:3
Electricity
0.1346
0.6069
0.181
0.132
0,3242
0.1398
0.1305
0,2023
0.132
0.1301
0.1561
0.1335
Standard deviation
2.0e
1.1e:3
8.6e-4
2.3e-4
4.3e-4
2.6e
1.8e-4
1.9e
2.6e-4
2.7e-4
8.9e.5
5.8e-4
Traffic
0.3678
0.9577
0.4081
0.3562
0.5999
0.3638
0.3494
0,4529
0.3572
0.3516
0.4079
0.3575
Stundard deviation
1.3e4
2.2c-3
4.4c-4
4.20-4
1.9c-3
3.7e-4
1.80-3
4.4e-4
7.4c-4
1.40-3
1.5c-4
3.40-4
Avg. Improvements
41.14%
30.02%
13.04%
6.95%
Max. Improvements
82.17%
79.749
53.18%
15,595
Table 27: Full standard deviation and MAE of comparing our smoothed full fine-tuning (SFF) with linear- precar-poobingg (LP) and linear-probing then full fine-tuning (LPFF). The average performance is reported for each group of three different data proportions, e.g., "Avg. on 1 1%, 2%, 3%"
Data proportion
Arg. 15, 2
3%
Avg. on 4%,
10%
Avg. on 15% 20% 25%
Avg. 50%, 75%, 100%
Methods
SEF
LPFF
SFF
LP
LPFF
SFF
LP
LPFF
SPF
LP
LPFF
Exchange
0.2047
0,5867
0.5287
0,2039
0.5848
0.48%6
0,2013
0,5714
0.3048
0,2014
0.5244
0.2202
Standard deviation
3.5e.4
2.
5.1e.
2.6e-3
3.9e
7.2e4
2.5e.
6.6e.3
1.le3
2.1e.3
ETTh
0.4006
0.644
0.5865
0,3963
0.6364
0,5505
0.3886
0.6134
0.4376
0.3905
0.5504
0.4097
Standard deviation
3.4e-3
4.8c-3
1.36-3
3,56-3
4.4o-3
3.96-3
1.16-3
3.8o-3
1.
1.46-3
2.8c-3
1.0c-3
ETTh2
0.3345
0.46
0.4366
0.3347
0.457
0,4154
0.3358
0.4488
0.3671
0.3365
0.4248
0.3558
Standard deviation
9.5c-4
4.1e:3
2.1e:3
1.4e:3
3.96:
9.5e-4
1.8e
3.3e:
1.3e-3
7.3e-4
1.9e:3
1.2c.3
ETTml
0,3942
0.7198
0.597
0.3735
0.7061
0,4873
0.3637
0.6695
0,3874
0.3592
0.5751
0.3759
Standard deviation
2.3e-3
8.9e-3
2.1e:3
1.3e-3
8.3e-3
1.2e-3
9.5e-4
6.7e-3
3.5e-4
1.8e-3
3.6e-3
6.5e-4
ETTm2
0.2601
0.3958
0.3636
0,2547
0.3926
0.3328
0.2523
0.381
0.2778
0.2512
0.349
0.2711
Standard deviation
2.5c-3
3.5c-3
1.7e-3
1,5e-3
3.4e-3
1.1e-3
5.2c-4
2.96-3
1,36-3
8.20-4
1.7e-3
1,56-3
Weather
0,2005
0.3584
0,299
0,1941
0,3472
0,2298
0.1929
0,3189
0.2007
0.1946
0.2598
0.2072
Standard deviation
5.1e-4
2.8e-3
2.90-3
2.0c-4
2.36-3
3.1e-3
2.4e-4
1.3e-3
6.5c-4
6.8e-4
6.2e-4
1.7e-3
Electricity
0.2312
0.6057
0.2824
0.2275
0.411
0,2387
0.2251
0.3048
0.2269
0.2243
0.2599
0.227
Standard deviation
3.le-4
2.3e-3
6.4e-4
2.2e.4
8.3e-4
2.9e-4
3.5e
6.6e-5
1.2c-4
3.7e-4
7.8e-5
6.9e-4
Traffie
0.2619
0.6062
0.3052
0,2532
D.4331
0.263
0.2474
0.3423
0.2519
0.2474
0.2972
0,2486
Standard deviation
3.9e-5
4.8c-4
2.2c-4
4.0e-4
9.fi-4
1.0c-4
1.8e.3
2.0e44
1.
6.2e-4
1.6e-4
3.3e-4
Avg. Improvements
30.51%
22,06
22.06%
9.43%
4.77%
Max. Improvements
61.28%
58.27%
33.96%
8.54%
29

---

Published as a conference paper at ICLR 2026
Table 28: Full standard deviation and MSE of comparing our smoothed full fine-tuning (SFF) with ennear-probing (LP) and linear-probing then full fine-tuning (LPFF) under 1 %, 2%, 3 3%, and 4% proportion of available data.
Data proportion
1%
2%
3%
4%
Methods
I.P
1.PFF
SFF
I.P
LPFF
SFF
LPFF
SFF
1.FFF
Exchange
0.0857
0.5944
0.4841
0.0863
0.5943
0.4788
0.085
0.5943
0.4773
0.0852
0.5943
0.4763
Standard deviation
3.20-4
7.30-3
4.50-3
4.2c-4
7.3c-3
3.0e-3
5.8c-4
7.30-3
4.20-3
2,40-4
7.3e-3
3.9c-3
ETThl
0,3737
0.8809
0,7219
0.3731
0.8806
0,7159
0.37
0.8804
0.7134
0.371
0.8731
0.7677
Standard deviation
6.30-3
9.10-3
3.0e-3
6.60-3
9.20-3
3:4e-3
6.6e-3
9.20-3
4.30-3
7.50-3
7.70-3
5,3e-3
ETTh2
0,2839
0,4428
0.4047
0,2769
0.4427
0.402
0.2797
0.4427
0.4012
0.279
0.4394
0.3874
Standard deviation
1.0e:3
7.3e-1
4.1e:3
1,7e-3
7.3e-3
3.7e-1
1.9e-3
7.3e.3
3.7e-3
2,7e:3
6.9e-3
2.5e.3
FTTml
0,3702
1,0885
0.8141
0,3343
1.0401
0.6548
0,3301
1.0393
0.6424
0.3229
1.0215
0,5353
Standard deviation
4.403
2.70-2
8.40-3
2.70-3
2.6c-2
6.6c-3
3.40-3
2.60-2
6.70-3
3,50-3
2.4e-2
5.00-3
ETTm2
0.1744
0,3569
0.3154
0.1789
0.3568
0.3139
0,1668
0.3528
0.2779
0.1674
0.3527
0.2757
Standard deviation
3
6.3e 3
3.5e
2.1e:3
6.3e
3.0e
2.4e
6.0e 3
3.6e:3
1.4e
6.1e.3
2.1e:3
Weather
0.1541
0,3273
0,2726
0.1507
0.324
0.2477
0.1499
0.3207
0,223
0.1474
0.3175
0.2
Standard deviation
4
4.4e-3
2.Ne.3
4.2e-3
4.5e.3
8.7e-4
4.0e-3
4.9e-3
2.9e
3.8e.3
6.5e.3
Flectricity
0.1369
0.7664
0,2197
0.1342
0.5866
0.1669
0.133
0.4678
0.1535
0,1331
0.3911
0,1464
Standard deviation
7
7,904
9.60-4
1.9c-
1.4c-3
9.8c-4
3.3c-4
1,20-3
6.50-4
2,50-4
5.6o-4
4.6o-4
Traffic
0.3743
1.1903
0.4536
0.3671
0.917
0.3941
0.3622
0.7658
0.3767
0.3594
0.6768
0.3688
Standard deviation
1.0e44
2.4e-3
1.1e:3
1.4e:5
1.7e:3
1.7e-4
2.7e-4
2.4e:3
3.3e:5
2.6e-4
2.6e-3
2.2e-4
Avg. Improvements
62.35%
44.78%
61.21%
40.11%
59.87%
37.4%
58.38%
34.83%
Max. Improvements
85,58%
82.3%
85.48%
81.98%
85.7%
89.
85.66%
82. 11%
Table 29: Full standard deviation and MSE of comparing our smoothed full fine-tuning (SFF) with linear-probing (LP) and kinear-probing then full fine-tuning (LFFF) under 5%, 10 %, 15 %, and 20 % proportion of available data.
Data propartion
5%
10%
15%
20%
Methods
I.P
LPFF
SFF
L.
LPFT
SFF
LPFF
SFF
LPFF
Exchange
0.0856
0.5936
0.468
0.0838
0.5838
0.3114
0.082
0.5737
0.2205
0.0818
0,5613
0.1646
Standard deviation
1.60-4
7.40-3
3.00-3
7.20-4
7.0c-3
1.3e-2
3.10-
6.80-3
1.1o2
1.10-3
6.70-3
6.20-3
ETTh!
0.3646
0.8618
0.6158
0,3568
0.8434
0.5265
0.3551
0,8139
0.4433
0.3509
0.7926
0.4029
Standard deviation
4.60-3
8,70-3
1.26-2
2.10-3
8.2c-3
1.1e-2
1.7c-3
6.60-3
4.2o-3
1.90-3
6.90-3
3,56-3
ETTh2
0.277
0.4386
0.3743
0,2782
0.4346
0.3503
0.2776
0.4279
0,3223
0,2785
0.423
0,3116
Standard deviation
3.5e:3
7.
1.7e:3
2.2e-3
6.6e:
6.9e
3.8e:3
6.2e
1.6e:3
1.5e
5.7e.3
1.5e:3
ETTml
0.319
1.0209
0.5224
0_307
0.9706
0.374
0.303
0,9274
0.3353
0.3006
0.8982
0,3229
Standard deviation
2.6e-3
2.40-2
5.20-3
1.76-3
2.1c-2
3:20-3
1.0c-3
1.9e-2
5.1c-
1.16-3
1.8c-2
8.8c-4
ETTm2
0.1679
0,3525
0,2738
0.164
0,3444
0,2182
0,1633
0.3368
0,1942
0.1631
0.3297
0.1862
Standard deviation
2
6.1e-3
2.20-3
1.8e-3
5.6e-3
1.6e-3
1.3e-3
5.3e-3
1.3e-3
7.8c-4
4.9e-3
8.5e-1
Weather
0.1461
0,3116
0,1721
0.1447
0.2956
0.1501
0.1443
0.2823
0.1476
0.1443
0,2654
0,1488
Standard deviation
1.6e44
3.5e-3
3.20-3
1.2c-4
2.Ne-3
6.2c-4
1.2e-4
2.2e-3
7.4e-4
2.9e-5
2.0e-3
7.5e-1
Electricity
0.1319
0.3355
0.1402
0.1309
0.246
0.1329
0.1309
0.2179
0.1321
0.1306
0.1989
0.1321
Standard deviation
2.6c-4
1.4e-4
2.9c-4
1.8c-4
5.8c-4
1.9e-5
2.3e-4
2.7e-4
2.6e-4
1.3e-4
1.50-4
3.20-4
Traffic
0.3576
0.6198
0.3639
0.3518
0.503
0.3588
0.3508
0.4671
0.3578
0.349
0.4505
0.3571
Standard deviation
2.80-5
2.30-3
7.10-5
9.8c-4
8.0c-4
8.3c-1
1.20-3
4.40-4
5.80-4
1.90-3
4.2g-4
1.20-3
Avg. Improvements
57.17%
31.11%
53.5%
21.96%
51.22%
15.9%
49. 49.36%
12.45%
Max. Improvements
85.58%
81.71%
85.65%
73.09%
85.71%
62.81%
85.43%
50.3%
30

---

Published as a conference paper at ICLR 2026
Table 30: Full standard deviation and MSE of comparing our smoothed full fine-tuning (SFF) with linear- proeaing (LP) and linear- linear-probing then full fine-tuning (LPFF) under 25%, 50% %,, 75 %, and 100% proportion of available data.
Data propoction
25%
50%
75%
100%
Methods
SFF
LP
LPFF
SPF
LP
LPFF
SPF
LP
LPFF
SFF
LP
LPFF
Exchange
0.0811
0.5541
0.1377
0,0809
0,5163
0.1011
0.0819
0.4667
0.0958
0.0809
0.439
0.0917
Standard deviation
4.504
6.1e.3
4.7c-3
5.4c-4
5:4e-
2.76:3
1.20-3
4.8c-3
2.0-3
7.6c-4
3.8c-3
9.2c-4
ETTal
0.351
0.7801
0.3918
0.3509
0.695
0.3724
0,3513
0.623
0.3769
0.3567
0.5888
0.37
Standard deviation
6.1e4
6.6r-3
3.4e-3
1.1e.3
5.1e-3
1.2e.3
1.4e-3
4.4e-3
1.5e-3
1.4e-3
4.0e-3
3.8c-3
ETTh2
0,2745
0,4192
0.3001
0.2757
0.3985
0.2977
0.2778
0,3843
0,3034
0.274
0.372
0.2991
Standard deviation
2.50-3
5,50-3
1.4e-3
2,0c-3
4.0c-3
1.6c-3
4.80-4
3.0p-3
1.90-3
3,8c-4
2.4c-3
1.70-3
ETTml
0.2995
0,8669
0.3152
0.2979
0,7422
0.3103
0.2974
0.6467
0.3136
0,2976
0.5935
0.3132
Standard deviation
3.
1.6e
6.4e
1.6e
1.1e-2
8.8e 4
1.3e-3
8.7e-3
1.5e:3
1.5e-3
7.2e-3
1,3e-3
ETTm2
0,1608
0.3226
0.1738
0.1609
0,2947
0.1751
0.1627
0,2771
0.1863
0,1613
0.2595
0.1798
Standard deviation
9.7e4
4.6e:3 3
7.9c 4
3.8e
3.4e-3
8.4e
4.3e
2.7e:3
1.2e-3
1.0e-3
2.1e.3
2.9e-3
Weather
0.144
0,2619
0.1479
0.1443
0,2248
0.1497
0.1468
0.192
0,1624
0.1451
0.1871
0,1573
Standard doviation
5.20-5
1.50-3
1.9c-4
2.1c-4
1.0c-3
7.20-4
1.30-4
7.9c44
1.70-3
7.3c-4
6.5c-4
9.1c-4
Electricity
0.1303
0.1902
0.1317
0,1301
0.1654
0.1324
0.13
0.1538
0.1345
0.1304
0.1491
0.1335
Standard deviation
1.5e.
2.0c-4
2.4c-4
1.0e-4
3.6e-4
9.5e-5
9.7e
2.0c-4
6.9e-5
6.4e-4
Traflic
0,3488
0,441
0.3567
0.3497
0.4164
0.3564
0,3502
0.406
0.3587
0.3551
0.4014
0.3574
Standard deviation
3
4.5e-4
4.8c-4
1.7e:3
2.5e-4
7.4e-4
2.5e:3
1,2c
3.3e-4
7.1e:5
7.5e:5
5.7e-4
Avg. Improvements
48.49%
9.8%
42.89%
6.56%
37.73%
7.86%
19%
6.22%
Max. Improvements
85.36%
41.1%
84.33%
19.98%
82.45%
14.51%
81.57%
11.78%
Table 31: Full standard deviation and MAE of comparing our smoothed full fine-tuning (SFF) with linear-probing probing (LP) and linear- mneear-poobingg then full fine-tuning (LPFF) under 1 1,, 2%, 3 %, and 4% proportion of available data.
Data propartion
1%
2%
3%
4%
Methods
SFF
LP
LPFF
SFF
LP
LPFF
SFF
LP
LPFF
SFF
LP
LPFF
Exchange
0,2051
0.5867
0.5309
0,2052
0.5867
0.528
0,204
0.5867
0.5272
0,2041
0,5866
0.5267
Standard deviation
2.3e.4
3.6e
3.5e
4.8c
3.
1.1e.3
3.5e
2.6e
3.5e:3
4.3e
2.6e
1.4e 3
ETThl
0,4019
0.6441
0,5884
0.401
0.644
0.5861
0.3992
0.6439
0.5851
0.4012
0.6409
0.6001
Standard deviation
3,30-3
4,80-3
1.20-3
3.40-3
4.8c-3
1.20-3
3.50-3
4.8c-3
1.4c-3
5,30-3
4,4o-3
1,50-3
ETTh2
0,3389
0.4601
0.4379
0.3323
0.46
0.4362
0.3326
0.46
0.4356
0.3344
0.4581
0.4267
Standard deviation
2.0c-3
4.10-3
2.30-3
3.20-4
4.1c-3
2.0c-3
5.20-4
4.1o-3
2.00-3
5,20-4
3.90-3
1.50-3
ETTml
0.412
0,7239
0.6419
0,3875
0.7179
0.5773
0,3835
0.7176
0,5717
0,378
0.7119
0.5213
Standard deviation
3.5e.
9.1e-3
3.6e
2.6e-3
8.8e.3
1.4e-3
1.9e-3
8.9e-3
1.3e:3
1.8e-3
8.6e-3
8.9e-4
ETTm2
0.2628
0.3966
0.3717
0.2617
0.3965
0.3709
0.2559
0,3942
0.3483
0.2564
0.3942
0.347
Standard deviation
3.00-3
3.6e-3
2.0e-3
1.96-3
3.6e-3
1.7e-3
2.5c-3
3.56-3
1.5c-3
1.6e-3
3.56-3
1.16-3
Weather
0.2034
0,3607
0.3198
0.1999
0.3584
0,2994
0.1987
0.3562
0.2779
0.1957
0,3539
0.2567
Standard deviation
5.6e-4
3.0e-3
1.7e:3
2.6e-4
2.Ne-3
3.2e:3
7.0e-4
2.7e-3
3.7e-3
2.3e-4
2.6e-3
3.8e-3
Electricity
0.2342
0.6951
0,3184
0.2307
0.5986
0.2729
0.229
0.5233
0,256
0.2292
0.4676
0.2476
Standard deviation
8.3e-5
2.5e-3
5.1e-4
3.4c-4
2.3e-3
8.5c-1
5.0e-4
2.0e-3
5.6e-1
2.7e-4
1.4e-3
4.4e-1
Traflic
0.2672
0.7112
0.3433
0.2611
0.59
0.2946
0.2575
0.5175
0.2776
0.2559
0.4736
0.2694
Standard deviation
I.le4
5.1e-4
4.6e-4
4.76-6
1.5e.4
1,5e-4
4.7c-6
7.8c-4
7.1e-5
9.40-5
1.20-3
8.5e-5
Avg. Improvements
47.27%
33.23%
46.5%
29.86%
45.41%
27. 87.81%
44.14%
25.73%
Max. Imprevements
66.31%
61.37%
65.02%
61.14%
63.23%
61.31%
63.21%
61.25%
31

---

Published as a conference paper at ICLR 2026
Table 32: Full standard deviation and MAE of comparing our smoothed full fine-tuning (SFF) with linear-probing (LP) and linear-probing then full fine-tuning (LPFF) under 5%, 10 10, 15 15,, and 20 % proportion of available data.
Data proportion
5%
10%
15%
20%
Methods
I.P
1.PFF
SFF
I.P
LPFF
SFF
LPFF
SFF
L.FFF
Exchange
0.2046
0.5863
0.5221
0.2031
0.5816
0.4169
0.2017
0.5766
0.3476
0.2019
0.5705
0.298
Standard deviation
2.0044
2.60-3
1.003
9.0c-4
2.6c-3
9.3e-3
6.1e-4
2.60-3
8.96-3
1.00-3
2.5c-3
5.7c-3
ETThl
0,3971
0,6374
0,5467
0,3907
0.6308
0.5046
0.3904
0.6203
0.4575
0.3874
0.6122
0.4314
Standard deviation
3.4e-3
4.62-3
4.90-3
1.80-3
4.4e-3
5.1e-3
1.5e-3
3.80-3
2.80-3
1.4e-3
3,9e-3
2.1e-3
ETTh2
0.3339
0.4576
0.4186
0,3359
0.4554
0.4008
0.3365
0.4515
0.3769
0.3367
0.4485
0.3674
Standard deviation
1,2e:3
3.9e.1
8.9e
2.3e-1
3.7e.3
4.2e
1.7e.3
3.6e:3
1.3e-3
2.6e:3
3.2e-3
1,3e.3
FTTml
0,3753
0,7116
0.5149
0,3674
0.6949
0,4256
0,3646
0.6801
0.3959
0.3642
0.6701
0,3869
Standard deviation
1,30-3
8.60-3
8.90-4
8.20-4
7.8c-3
1.8c-
3.10-4
7.10-3
1.50-4
8,20-4
6.8c-3
5,40-4
ETTm2
0.2544
0.3941
0.3458
0,2534
0.3896
0.3057
0.2539
0,3852
0.2866
0,2533
0.3811
0.2807
Standard deviation
3.6e
3.5e 3
1.le3
1.3e-3
3.3e
1.0e
5.0e 4
3.1e:3
1.4e:3
4.1e 4
2.9e.3
3.4e
Weather
0.1939
0,3497
0,2289
0.193
0.3381
0.2038
0.193
0.3282
0.1996
0.1935
0.3161
0.2025
Standard deviation
2.3e.4
2.4e-3
3.0e-3
1.9e-3
6.9e 4
1.5e.3
4.3e.4
3.0e
1.4e.3
9.5e
Flectricity
0,2273
0,4226
0,2394
0.2263
0.3427
0.2291
0.226
0.3184
0.2274
0.225
0.3021
0.2266
Standard deviation
9.16-5
7,8c4
3.20-4
3,20-4
3.10-4
1.2o-4
1.8c-4
1,50-4
1.6o-4
4.50-4
4.50-5
2.6o-5
Traffic
0,2537
0,4449
0,2639
0.2501
0.3808
0.2557
0.2488
0.3548
0.2515
0,2473
0,3404
0.2511
Standard deviation
5
1.3e-3
5.8c
1.0e:3
3.1e-4
1,9c-4
1.4e-3
9.9e:5
3.3e:5
1.6e:3
2.0c-4
1.8e:3
Avg. Improvements
43.28%
23.27%
40.34%
16.19%
38.52%
11.46%
37.14%
9.14%
Max. Improvements
65.1%
60.81%
65.08%
51.28%
65.02%
41.97%
64.61%
32.25%
Table 33: Full standard deviation and MAE of comparing our smoothed full fine-tuning (SFF) with linear- probing (LP) and ahear-prohingg then full fine-tuning (L.FFF) under 25,, 50 % %, and 100% proportion of available data.
Data proportion
25%
50%
75%
100%
Methods
I.P
LPTT
I.P
L.FTT
SFT
L.
LPFF
SEF
L.P
LPFF
Exchange
0.2004
0.567
0,2689
0.2003
0.5476
0.2251
0.2026
0.5206
0.2204
0,2013
0.505
0.2151
Standard deviation
5.30-
2.30-3
5.30-3
1.00-3
2.30-3
3.76-3
1.70-3
2.20-3
2.50-3
5.30-4
2.0e-3
2.00-3
0,3883
0.6078
0.4238
0,3883
0,5753
0.4091
0,3892
0.5457
0.4127
0,394
0.5301
0.4074
Standard deviation
4.50-4
3.70-3
1.3e-3
3.00-4
3.00-3
1.5c-3
2.40-3
2.8c-3
9.06-4
1.46-3
2.70-3
6.8c-4
ETTh2
0,3343
0.4463
0.3571
0,3346
0,4337
0.3536
0.339
0.4245
0.3599
0.3359
0.4163
0.3539
Standard deviation
1.3e.3
3.1e:3
1.3e.3
1.4e:3
2.4e:3
1.4e
1.8e
1.8e
1.4e-3
5.7e
1.5e.3
8.3e-4
FTTml
0.3623
0.6584
0.3794
0,3602
0.6101
0.3731
0.3595
0.5709
0,3797
0.358
0.5444
0.3748
Standard deviation
1.7c-3
6.20-3
3.5c-4
1.80-3
4.4o-3
5.6c-4
2.0s-3
3.4e-3
7.20-4
1.6e-3
2.9c-3
6.76-4
ETTm2
0,2499
0,3768
0.2661
0,2504
0,3601
0.2653
0,2539
0.3494
0.2797
0,2493
0.3375
0.2683
Standard deviation
6.4g-4
2.7e-3
1.2e-3
5.8c-4
2.1e-3
9.2e-4
3.8e-4
1.7e-3
1.3e-3
1.5e-3
1.4e-3
2.3e-3
Weather
0.1925
0,3125
0.1999
0.193
0,2815
0,2002
0.1975
0.252
0.2164
0.1934
0.2459
0.2049
Standard deviation
2.9c-
1.0e-3
5.9c-4
2.1e-4
6.9e-4
1.4e-3
2.9c-4
6.3e-4
2.3e-3
1.5e-3
5.2c-4
1.5e-3
Electricity
0.2245
0.2939
0.2268
0,2247
0.2704
0.2266
0.2245
0.2578
0.2278
0.2239
0.2516
0.2266
Standard deviation
4.2c-4
4.7c-6
1.7c-4
2.7e-4
5.4e-5
1.1e-3
4.3o-4
1.0c-4
6.0c-4
4.2o-4
7.9e-5
3.9e-4
Traffic
0.2463
0.3316
0.2532
0,2486
0,3062
0,2465
0.2485
0.2951
0,2498
0.245
0.2902
0.2494
Standard deviation
2.20-3
2.9c-4
1.4e-3
1.4e-3
2.50-4
4.70-4
4.62-4
1.4e-4
3,50-4
0.0c+00
8.5e-5
1.80-4
Avg. Improvements
36.53%
7.28%
32.17%
4.27%
28.07%
5.6%
26.68%
4.36%
Max. Improvements
64.66%
25.47%
63.42%
11.02%
61.08%
9.22%
60. 14%.
7.08%
32


Reviews:
[{"review_id": "UsDBkDEzPl", "reviewer": "Reviewer_hGpf", "summary": "This paper investigates the fine-tuning challenges of large time series models and identify non-convex loss landscapes in pre-trained LTSM is the key root cause that lead to poor trainability and overfitting during fine-tuning stage. To alleviate the challenge, the authors propose smoothed full fine-tuning, which linearly interpolates the weights of a pre-trained LTSM with a randomly initialized auxiliary model to smooth the loss landscape, to improve trainability while preserve pre-trained knowledge. The method evaluation on forecasting, imputation, and anomaly detection tasks using eight existing LTSMs across multiple benchmark datasets shows consistent improvements over baselines (i.e., full fine-tuning, training from scratch, linear probing, and linear probing then full fine-tuning) with no added computational overhead.", "questions": "1. How does SFF compare to simply adding random noise to pre-trained weights (e.g., Gaussian perturbation)?\n2. How sensitive is SFF to the random init distribution?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies an interesting issue that pre-trained LTSMs may have sharp loss landscapes that hinder fine-tuning with empirical evidence and visualization.\n\n2. The proposed method is simple yet effective, requiring only a one-time weight interpolation before fine-tuning.\n\n3. The experiment is comprehensive, covering wide range of LTSM architectures and tasks.", "weaknesses": "1. Lack of novelty: The core idea of linear weight interpolation to smooth landscapes is not a new idea. Weight averaging/interpolation [1] has been widely studied in model merging [2], continual learning [3], yet none of these have been discussed in related work. Although the paper claims to be \"the first\" for LTSMs, this domain-specific application does not justify novelty, which seems more like a repackaging of existing optimization tricks with application on LTSMs.\n\n2. Lack of theoretical justification: The paper attempts to explore the challenge of model fine-tuning from optimization theory perspective. However, the discussion in Section 3.1 seems hand-wavy. Specifically, the claim, \"interpolation perturbs sharp minima without harming flat regions\" is intuitive but lacks rigor. Providing some theoretical proofs or connection to sharpness-aware minimization can better provide theoretical merit to the readers.\n\n3. Baselines for comparison: None of the parameter-efficient methods like LoRA, QLoRA are included, these are standard for fine-tuning large models. Also, there is no ablation on other smoothing techniques such as Gaussian noise or label smoothing. Without those baselines, it is hard to evaluate the contribution of this paper as the claim of paper is to tackle the \"fine-tuning\" problem of time series foundation model.\n \n[1] Vlaar, Tiffany J., and Jonathan Frankle. \"What can linear interpolation of neural network loss landscapes tell us?.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" International conference on machine learning. PMLR, 2022.\n\n[3] Kozal, Jędrzej, et al. \"Continual learning with weight interpolation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjTjoGNiwK", "reviewer": "Reviewer_t8mv", "summary": "This paper identifies a critical problem in the practical application of large time series models: pre-trained models often exhibit poor trainability when fine-tuned on downstream tasks. The authors attribute this to the models converging to sharp minima during pre-training, resulting in a non-convex loss landscape that leads to overfitting during fine-tuning—sometimes to the point of underperforming models trained from scratch. To address this, the paper proposes a simple and effective method called Smoothed Full Fine-tuning. SFF works by first creating an auxiliary, randomly initialized model, which possesses a smooth loss landscape  but no pre-trained knowledge. Before fine-tuning, SFF performs a single linear interpolation between the weights of the pre-trained model and this auxiliary model. The resulting \"smoothed\" model is shown to retain the valuable knowledge of the pre-trained model while inheriting the superior trainability of the random model , allowing it to escape sharp minima and find better, flatter basins. The authors provide extensive empirical validation, showing that SFF consistently improves the performance of eight different LTSMs  on forecasting and anomaly detection tasks compared to standard fine-tuning methods, without incurring any additional memory or computational overhead during the fine-tuning step.", "questions": "Listed in weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles a significant and practical limitation of pre-trained LTSMs. The observation that direct fine-tuning can lead to overfitting and even perform worse than training from scratch is a crucial finding that motivates the need for better fine-tuning strategies. The problem is clearly illustrated using loss landscape visualizations.\n\nThe proposed SFF method is exceptionally simple, consisting of a single linear interpolation of model weights before fine-tuning begins. This makes it easy to implement and, importantly, it adds no additional memory or computational overhead to the actual fine-tuning process, making it a highly practical solution.\n\nIn a particularly strong finding, the smoothing process by itself is shown to improve zero-shot forecasting performance, suggesting it guides the model to a better, more generalizable basin in the loss landscape even without fine-tuning.", "weaknesses": "The theoretical motivation in Section 3.1 is more of a high-level, intuitive argument rather than a formal analysis. While the explanation is plausible and well-supported by citations to related work, the paper does not provide a rigorous theoretical proof for why this specific form of interpolation is an optimal or principled way to achieve this smoothing\n \nThe interpolation coefficient $\\alpha$ is a new, critical hyperparameter introduced by SFF. The paper shows SFF is robust, outperforming FF across a wide range of $\\alpha$ values However, the paper also shows that the optimal $\\alpha$ for zero-shot performance differs from the optimal range for fine-tuning. The paper provides limited guidance on how $\\alpha$ should be selected for a new model or dataset, other than selecting from a predefined set.\n\nThe method relies on constructing an \"auxiliary LTSM\" through random initialization of the same architecture. The paper does not explore or justify this specific choice. It is unclear if a different, or perhaps simpler, randomly initialized model could achieve a similar or even better smoothing effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ibjSDQSsnM", "reviewer": "Reviewer_wYp2", "summary": "The paper proposes Smoothed Full Fine-tuning (SFF), a lightweight method to improve fine-tuning of large time-series models by interpolating pretrained weights with a randomly initialized model before training. This simple step smooths sharp loss landscapes, enhancing optimization without extra computational cost. Tested on eight major LTSMs across forecasting and anomaly detection tasks, SFF consistently improves fine-tuning and often boosts zero-shot accuracy. The approach is practical and broadly effective, though the paper’s novelty is mainly within time-series models and lacks comparisons to established flatness-based fine-tuning methods.", "questions": "Please address the identified weaknesses and limitations noted above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is simple and inexpensive. Smoothing is done with a one-shot linear interpolation using a randomly initialized copy before fine-tuning. It only needs a few lines of PyTorch and does not add compute or memory cost.\n- The motivation is clear. The paper argues that pretraining can leave large time-series models stuck in sharp, non-convex regions, and interpolation helps escape these while keeping flat regions stable.\n- The evaluation covers many settings. The authors test eight large time-series models across forecasting, anomaly detection, and imputation tasks.", "weaknesses": "- The novelty claim may be overstated. The idea of mixing weights for fine-tuning already exists in other domains, so the contribution should be limited to time-series foundation models.\n- The baselines are limited. The comparisons use full fine-tuning and linear probing but omit other regularization or smoothing approaches such as SAM, SWA, Mixout, or L2-SP.\n- Some zero-shot results decline after smoothing. Models like MOIRAI and Chronos perform worse on certain datasets.\n- Interpolation with a random model might cause misalignment. It can disturb normalization or scale between layers, and the paper does not analyze these risks in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9J9tNRKub0", "reviewer": "Reviewer_r8RN", "summary": "The paper proposes the Smoothed Full Fine-tunning (SFF) to finetune LTSMs. SFF linearly interpolate the parameters of a pretrained LTSM with a randomly initialized auxiliary model, with the goal of smoothing sharp, non-convex regions of the loss landscape. The main contribution lies in the interpolation that perturbs the parameter of the model to reach flatter regions. Such perturbation towards flat minima is a well-known strategy for enhance generalization. Experimental results show the method can obtain better results than compared baselines and that SFF can perform fine-tuning with minimal memory and computational overhead.", "questions": "Refer to the weakness part. \n\n- Can the authors provide a more detailed explanation or empirical evidence as to why a randomly initialized LTSM is guaranteed to have a smoother loss landscape than a pre-trained one?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors identify a significant and practical problem, the poor fine-tuning performance of modern LTSMs due to the non-convexity of their loss landscape.  \n- SFF is simple to implement, requiring only a one-time linear interpolation of weights before fine-tuning. It adds no computational or memory overhead during the actual fine-tuning process.", "weaknesses": "- Exploration of interpolation strategies. The paper states that $\\alpha$ controls the proportion of pre-trained knowledge retained, and the experiments show a sensitivity to $\\alpha$. However, the optimal $\\alpha$ appears to be model-dependent. A more robust, perhaps adaptive, method for selecting or tuning $\\alpha$ would significantly strengthen the work.\n\n- The auxiliary model is defined as a randomly initialized LTSM. The paper claims this model has a smooth loss landscape, however is not clear why a randomly initialized model is guaranteed to be smoother than a pre-trained one, especially considering the vast differences in initialization schemes and model architectures. A more rigorous analysis or reference to why random initialization leads to flat minima would be beneficial.\n\n- The theoretical analysis in Section 3.1 is relatively brief and qualitative, relying heavily on existing literature. I suggest the authors to present a more in depth analysis of the loss landscape for finetuning, such analysis would provide strong theoretical justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)