# Are Time Series Foundation Models Susceptible to Catastrophic Forgetting?

Nouha Karaouli, Denis Coquenet, Elisa Fromont, Martial Mermillod, Marina Reyboz

► To cite this version:

Nouha Karaouli, Denis Coquenet, Elisa Fromont, Martial Mermillod, Marina Reyboz. Are Time Series Foundation Models Susceptible to Catastrophic Forgetting?. 2025. hal-05306471

HAL Id: hal-05306471

https://hal.science/hal-05306471v1

Preprint submitted on 9 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Are Time Series Foundation Models Susceptible to Catastrophic Forgetting?

Nouha Karaouli¹, Denis Coquenet², Elisa Fromont¹, Martial Mermillod³, and Marina Reyboz⁴

¹Univ. Rennes, CNRS, Inria, IRISA - UMR 6074, F-35000 Rennes, France

²Univ. Rennes, CNRS, IRISA - UMR 6074, F-35000 Rennes, France

³Univ. Grenoble Alpes, Univ. Savoie Mont Blanc, CNRS, LPNC, Grenoble, France

⁴Univ. Grenoble Alpes, CEA, LIST, 38000 Grenoble, France

## Abstract

Time Series Foundation Models (TSFMs) have shown promising zero-shot generalization across diverse forecasting tasks. However, their robustness to continual adaptation remains underexplored. In this work, we investigate the extent to which TSFMs suffer from catastrophic forgetting when fine-tuned sequentially on multiple datasets. Using synthetic datasets designed with varying degrees of periodic structure, we measure the trade-off between adaptation to new data and retention of prior knowledge. Our experiments reveal that, while fine-tuning improves performance on new tasks, it often causes significant degradation on previously learned ones, illustrating a fundamental stability-plasticity dilemma.

## 1 Introduction

Foundation models have revolutionized Natural Language Processing (NLP) and Computer Vision (CV) by enabling strong zero-shot and few-shot generalization through large-scale self-supervised pretraining [1, 2, 3, 4, 5]. Despite their success, these models are known to suffer from catastrophic forgetting, the tendency to lose previously acquired knowledge when fine-tuned on new tasks [6, 7, 8].

Recently, similar foundation models have emerged in Time Series Forecasting (TSF), aiming to transfer the benefits of pretraining to temporal tasks. Notably, TimesFM [9] adapts the transformer architecture [10] to model temporal dependencies across a wide range of time series data. These TSFMs promise strong generalization, including zero-shot forecasting capabilities, especially on synthetic or structured datasets.

However, time series data presents unique challenges compared to text or images: it is continuous, often noisy, and prone to non-stationarity. Transformer-based TSFMs are particularly sensitive to such characteristics, frequently overfitting short-term fluctuations instead of learning long-range temporal patterns [11, 12, 13].

This work focuses on systematically evaluating catastrophic forgetting in TimesFM through a two-stage continual learning setup. Our results reveal a clear stability-plasticity dilemma [14], while adaptation to new datasets is possible through fine-tuning, it often comes at the cost of erasing knowledge learned from earlier tasks.

Our main contributions are as follows:

* We introduce an evaluation framework designed to quantify catastrophic forgetting for time series forecasting.

* We demonstrate that TSFMs are susceptible to catastrophic forgetting for the time series forecasting task.
## 2 Related Work

Recent advances have introduced several TSFMs designed to generalize across forecasting tasks, domains, and time scales [15, 16]. Notable examples include TimeGPT [17], PatchTST [18], FEDformer [19], and TimesFM [9]. These models differ in architecture and objectives but share the aim of delivering strong zero-shot and fine-tuning performance across diverse benchmarks.

To evaluate such models under realistic deployment conditions, several benchmarking protocols have emerged. GIFT-eval [20] assesses generalization and transferability across domains. OpenTS [21] complements this by offering a reproducible suite spanning datasets, metrics, and scenarios. Nixtla [22] further expands this with a comprehensive protocol assessing generalization across forecasting horizons and frequencies.

Across these benchmarks, TimesFM consistently emerges as a strong baseline, cited in GIFT-eval, OpenTS, and Nixtla for its robust performance, scalability, and transparent design. Its open-source availability and pretrained weights make it a practical and reproducible reference for TSFM research.

However, despite the rise of TSFMs and standardized benchmarks, most evaluations rely on static protocols, assessing zero-shot or fine-tuned performance on fixed tasks [20, 21]. This overlooks a key challenge in real-world deployment, namely continual learning. In particular, it remains unclear whether TSFMs can retain prior knowledge when fine-tuned sequentially.

In this work, we address this gap by studying catastrophic forgetting in TimesFM. We analyze how its pretrained capabilities degrade when exposed to new tasks in a continual learning setup. To the best of our knowledge, this is the first empirical study to demonstrate the stability-plasticity trade-off in univariate time series forecasting with foundation models, laying the groundwork for future improvements in time series continual learning.

## 3 Methodology

We focus on evaluating the TimesFM's retention capacity, to determine whether it is prone to catastrophic forgetting. TimesFM is a decoder-only transformer pretrained on large, diverse time series datasets and publicly available for fine-tuning.

The experimental protocol follows a two-stage continual learning setup. In **Stage One**, TimesFM is fine-tuned on a source dataset (A). In **Stage Two**, it is sequentially fine-tuned on a different target dataset (B). After each stage, we evaluate performance on dataset A to assess knowledge retention and quantify forgetting, and on dataset B to measure task adaptability. We vary fine-tuning hyperparameters (e.g., number of epochs, learning rate) and the domain shift between datasets A and B to analyze their impact on forgetting.

To ensure unbiased evaluation, we use synthetic datasets specifically designed to prevent any overlap with TimesFM's original training data. This approach guarantees that the model is tested on unseen data. We design synthetic multi-sinusoidal datasets with varying complexity and periodic structure:

* **D1** and **D2** feature 4 and 3 sine waves respectively, with harmonically aligned periods, allowing the model to observe full cycles within the dataset. This setup tests the ability to learn and predict fully repetitive patterns.

* **D3** and **D4** contain 10 sine waves with randomly sampled, non-harmonic periods, producing very long global cycles that exceed the dataset length. These datasets simulate real-world scenarios where only partial signal cycles are observed, challenging the model to generalize and extrapolate from incomplete information.

Each dataset contains 2,688 time steps (8 weeks of continuous data), which we partition into 70% training, 15% validation, and 15% testing sets, see Appendix A for more details on the dataset creation. This controlled setup allows us to sequentially fine-tune TimesFM on different datasets and measure performance retention, thereby quantifying catastrophic forgetting. We evaluate performance using Mean Absolute Error (MAE) and Backward Transfer (BWT) [23].
# 4 Results

The fine-tuning setup employed the Adam optimizer (learning rate = $1 \times 10^{-4}$, weight decay = 0.01), batch size of 64, and training for 5 epochs. All data was preprocessed using standardization and fed into the model using a sliding window with sequence length 256 and prediction length 128.

**Experiment 1 (D1 → D2):** MAE on D1 rose sharply from 0.15 to 1.60 after fine-tuning on D2, indicating severe forgetting, while D2's error improved from 1.27 to 0.08, as shown in Figure 1.

<figure><img src="image_2.png" /><figcaption>Figure 1: Forecasting results on D1 and D2 at each fine-tuning stage. left Panel shows degradation on D1 due to catastrophic forgetting, while right panel illustrates improved adaptation to D2 after fine-tuning.</figcaption></figure>

**Experiment 2 (D3 → D4):** MAE on D3 increased from 0.56 to 0.76 after fine-tuning on D4, showing moderate forgetting, with D4 improving from 0.90 to 0.47.

<table><caption>Table 1: Results on D1–D4 at each fine-tuning stage. The table reports MAE and BWT to track performance and forgetting throughout sequential training.</caption><thead><tr><th>Experiment</th><th>Dataset</th><th>Stage one</th><th>Stage two</th><th>BWT</th></tr></thead><tbody><tr><td rowspan="2">D1 → D2</td><td>D1</td><td>0.15</td><td>1.60</td><td>+1.45</td></tr><tr><td>D2</td><td>1.27</td><td>0.08</td><td>—</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.56</td><td>0.76</td><td>+0.20</td></tr><tr><td>D4</td><td>0.90</td><td>0.47</td><td>—</td></tr></tbody></table>

The results in table 1 demonstrate significant catastrophic forgetting in TimesFM during sequential fine-tuning, reflecting the stability-plasticity dilemma in foundation models.

To strengthen the proposed evaluation, we analyzed the impact of the learning rate ($10^{-4}$, $10^{-5}$, $10^{-6}$, $10^{-7}$) and number of epochs (5, 10, 15) on catastrophic forgetting and adaptation using the D1 → D2 and D3 → D4 sequential fine-tuning tasks. Tables 2 and 3 present MAE after each fine-tuning stage.

Based on the results shown in Tables 2 and 3, our study reveals a clear trade-off between adaptation and forgetting influenced by the learning rate and number of fine-tuning epochs. High learning rates (e.g., $10^{-4}$) enable rapid and effective adaptation to new data but cause severe catastrophic forgetting. Conversely, very low learning rates ($\le 10^{-6}$) substantially reduce forgetting but limit the model's ability to learn new patterns. Increasing the number of epochs intensifies forgetting without providing significant adaptation gains beyond approximately 10 epochs. Although an intermediate learning rate of $10^{-5}$ combined with 5 to 10 epochs offers the best observed balance, allowing effective adaptation while preserving prior knowledge, this tuning does not fully resolve the inherent stability-plasticity dilemma. These findings highlight the need for advanced continual learning methods to better manage this fundamental trade-off in time-series foundation models.
<table><caption>Table 2: Impact of the learning rate (LR) on catastrophic forgetting, with 5 epochs maintained across all settings. MAE is reported for the D1 → D2 and D3 → D4 experiments.</caption><thead><tr><th>LR</th><th>Experiment</th><th>Dataset</th><th>Stage One</th><th>Stage Two</th><th>BWT</th></tr></thead><tbody><tr><td rowspan="4">1 × 10<sup>-4</sup></td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.15</td><td>1.60</td><td>+1.45</td></tr><tr><td>D2</td><td>1.27</td><td>0.08</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.56</td><td>0.76</td><td>+0.20</td></tr><tr><td>D4</td><td>0.90</td><td>0.47</td><td>–</td></tr><tr><td rowspan="4">1 × 10<sup>-5</sup></td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.12</td><td>0.21</td><td>+0.09</td></tr><tr><td>D2</td><td>0.56</td><td>0.05</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.49</td><td>0.62</td><td>+0.13</td></tr><tr><td>D4</td><td>0.64</td><td>0.39</td><td>–</td></tr><tr><td rowspan="4">1 × 10<sup>-6</sup></td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.12</td><td>0.10</td><td>-0.02</td></tr><tr><td>D2</td><td>0.34</td><td>0.053</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.50</td><td>0.49</td><td>-0.01</td></tr><tr><td>D4</td><td>0.51</td><td>0.48</td><td>–</td></tr><tr><td rowspan="4">1 × 10<sup>-7</sup></td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.24</td><td>0.22</td><td>-0.02</td></tr><tr><td>D2</td><td>0.32</td><td>0.24</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.51</td><td>0.49</td><td>-0.02</td></tr><tr><td>D4</td><td>0.55</td><td>0.39</td><td>–</td></tr></tbody></table>

<table><caption>Table 3: Impact of the number of epochs on catastrophic forgetting, with a learning rate of 10⁻⁵ maintained across all settings. MAE is reported for the D1 → D2 and D3 → D4 experiments.</caption><thead><tr><th># Epochs</th><th>Experiment</th><th>Dataset</th><th>Stage one</th><th>Stage two</th><th>BWT</th></tr></thead><tbody><tr><td rowspan="4">5</td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.12</td><td>0.21</td><td>+0.09</td></tr><tr><td>D2</td><td>0.56</td><td>0.05</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.49</td><td>0.62</td><td>+0.13</td></tr><tr><td>D4</td><td>0.64</td><td>0.39</td><td>–</td></tr><tr><td rowspan="4">10</td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.10</td><td>0.26</td><td>+0.16</td></tr><tr><td>D2</td><td>0.62</td><td>0.04</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.49</td><td>0.61</td><td>+0.12</td></tr><tr><td>D4</td><td>0.65</td><td>0.39</td><td>–</td></tr><tr><td rowspan="4">15</td><td rowspan="2">D1 → D2</td><td>D1</td><td>0.10</td><td>0.35</td><td>+0.25</td></tr><tr><td>D2</td><td>0.69</td><td>0.074</td><td>–</td></tr><tr><td rowspan="2">D3 → D4</td><td>D3</td><td>0.48</td><td>0.62</td><td>+0.14</td></tr><tr><td>D4</td><td>0.66</td><td>0.41</td><td>–</td></tr></tbody></table>

## 5 Conclusion

Our evaluation reveals a fundamental challenge for foundation models like TimesFM: catastrophic forgetting during sequential fine-tuning. While these models can adapt effectively to new datasets, this adaptation often comes at the cost of significantly degraded performance on previously learned tasks. This highlights the inherent plasticity-stability trade-off, where learning new information disrupts retention of prior knowledge. Overcoming catastrophic forgetting is therefore critical to enable foundation models to continuously learn from evolving data without losing valuable insights from past experience. Developing robust continual learning strategies will be essential for deploying these models reliably in dynamic real-world forecasting environments where data distributions shift over time.
References

[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. *OpenAI Blog*, 1(8):9, 2019.

[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186. Association for Computational Linguistics, 2019.

[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In *Proceed- ings of the 38th International Conference on Machine Learning*, volume 139 of *PMLR*, pages 8748–8763, San Francisco, CA, USA, 2021. PMLR. Equal contribution. OpenAI.

[4] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. *arXiv preprint arXiv:2205.01917*, 2022. Version 2, 14 Jun 2022.

[5] Johannes Schneider, Christian Meske, and Paul Kuss. Foundation models: A new paradigm for artificial intelligence. *Business & Information Systems Engineering*, 66(2):221–231, 2024.

[6] Vinay Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of model and pretraining scale on catastrophic forgetting in neural networks. In *Proceedings of the International Conference on Learning Representations (ICLR)*. Google Research, Blueshift, 2022.

[7] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. *arXiv preprint arXiv:2308.08747v5*, 2025. arXiv:2308.08747v5 [cs.CL].

[8] Naimul Haque. Catastrophic forgetting in llms: A comparative analysis across language tasks. *arXiv preprint arXiv:2504.01241*, 2025.

[9] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In *Proceedings of the 41st International Conference on Machine Learning (ICML)*, volume 235 of *Proceedings of Machine Learning Research*, pages 4599–4623. PMLR, 2024.

[10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In *Advances in Neural Information Processing Systems*, volume 30, 2017.

[11] Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In *Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)*. NeurIPS, 2023.

[12] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? *arXiv preprint*, arXiv:2205.13504, August 2022. Version 3.

[13] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, pages 11106–11115. AAAI Press, 2021.

[14] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. *Frontiers in Psychology*, 4:504, 2013.

[15] Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, 2024.
[16] John A. Miller, Mohammed Aldosari, Farah Saeed, et al. A survey of deep learning and foundation models for time series forecasting. arXiv preprint arXiv:2401.13912, 2024.

[17] Azul Garza, Cristian Challu, and Max Mergenthaler-Canseco. Timegpt-1. arXiv preprint arXiv:2310.03589, 2024. Version 3, 27 May 2024.

[18] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.

[19] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the 39th International Conference on Machine Learning, pages 27268–27286, 2022.

[20] Taha Aksu et al. GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation. arXiv, November 2024.

[21] DecisionIntelligence. OpenTS – A Comprehensive and Fair Benchmark for Time Series Analytics. https://decisionintelligence.github.io/OpenTS/algorithms/methods/index.html, October 2024. Accessed 25 October 2024.

[22] Nixtla. Foundation time series arena – benchmarking foundation models for time series. https://github.com/nixtla/experiments/tree/main/foundation-time-series-arena, 2024. GitHub repository.

[23] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. In Advances in Neural Information Processing Systems (NeurIPS) 35, 2022. Main Conference Track.
# A Synthetic Dataset Generation

To evaluate our models under controlled but realistic scenarios, we generated four synthetic time series datasets (D1 to D4) by summing multiple sine waves with different periods and random phases. This approach allows us to simulate complex temporal patterns with known ground-truth periodicities.

## Key Function for Multi-Sinusoidal Time Series

def multSinFnGen(T_list, phase_div=12):
    sampled_freq_list = [1/t for t in T_list]
    phase_list = [2 * np.pi * np.random.random(10, phase_div) / phase_div
                  for _ in T_list]

    def multSinFn(x):
        return sum(np.sin(2 * np.pi * f * x + phi)
                    for f, phi in zip(sampled_freq_list, phase_list))

    return multSinFn

This function constructs a composite sinusoidal function by summing sine waves whose frequencies correspond to the inverses of the given periods in T_list. Each sine component is assigned a random phase uniformly sampled from discrete increments of 2π/phase_div (default 12).

## Data Generation Pipeline

For each dataset, we sample this composite function at uniform time steps (every 30 minutes), covering a total of 2688 time points (corresponding to 8 weeks). We pair these sampled values with datetime indices starting from January 1, 2000, resulting in a CSV file with two columns: date and values.

## Datasets and Their Periods

* D1: Periods = [21, 84, 336, 2688]

* D2: Periods = [42, 168, 1344]

* D3: Periods = [1260, 296, 1114, 1120, 325, 458, 105, 67, 911, 522]

* D4: Periods = [674, 570, 71, 726, 709, 1127, 226, 1198, 1282, 358]

These period selections reflect increasing complexity and variability from D1/D2 to D3/D4.