Title: Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time series forecasting
Abstract: Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting performance. Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. 
A key question is how to realize effective adaptation of TSFMs for a target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned to activate task-relevant network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on a more relevant and compact parameter space.  
Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This ``prune-then-finetune'' paradigm often enables TSFMs to achieve state-of-the-art performance and surpass strong specialized baselines. 
Source code is made publicly available at \url{https://github.com/SJTU-DMTai/Prune-then-Finetune}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to address an important problem in the time series forecasting domain: How to prune the pre-trained models during Full-Shot fine-tuning (referring to usage of 100% of the downstream data for fine-tuning). The paper proposes a pruning and then fine-tune approach, which essentially masks some of the neural network connections and thus create a more compact model. The paper evaluates the proposed approach in six time series forecasting datasets which are widely employed in this domain.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe problem is pretty important in the time series domain, where, often fine-tuning cannot match the performance of a model trained from scratch on the entire data.\n2.\tRigorous experimental evidence is shown for existing SOTA models that possess the addressed problem.\n3.\tMinimal necessary ablation studies have been done.\n\nWeaknesses:\n\n1.\tUndoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.\n2.\tThe pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? Have the author(s) tried to apply varying dropout (best value chosen using a validation set) during the fine-tuning. A full comparison with dropouts should be performed for all models and datasets.\n3.\tSome of the TSFMs suggest fine-tuning only the model’s head for the downstream task (e.g., the HeadProbing in TTM). How does the pruning method perform when only applied on the head (or other layers for other TSFMs)? \n4.\tThe improvement in the MSE and MAE scores are very insignificant, often 2nd and 3rd decimal places. This questions the efficacy of the proposed method.\n5.\tOnly three unique data sources have been used (ETT, electricity, weather). More variety of datasets should be included in the analysis, e.g, the traffic dataset widely used for evaluation TSFMs.\n\nQUESTIONS:\n1.\tUndoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.\n2.\tThe pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? Have the author(s) tried to apply varying dropout (best value chosen using a validation set) during the fine-tuning. A full comparison with dropouts should be performed for all models and datasets.\n3.\tSome of the TSFMs suggest fine-tuning only the model’s head for the downstream task (e.g., the HeadProbing in TTM). How does the pruning method perform when only applied on the head (or other layers for other TSFMs)? \n4.\tThe improvement in the MSE and MAE scores are very insignificant, often 2nd and 3rd decimal places. This questions the efficacy of the proposed method.\n5.\tOnly three unique data sources have been used (ETT, electricity, weather). More variety of datasets should be included in the analysis, e.g, the traffic dataset widely used for evaluation TSFMs.\n\nLIMITATIONS:\nNo. \nLimitations of the proposed method should be discussed. See the weaknesses section for details. A summary is provided here:\n1. How does the method perform in few-shot setting compared to a model trained from scratch with few-shot data as well?\n2. Comparison with HPO-selected dropouts.\n3. Comparison when only head is pruned.\n4. Wider variety of data sources.\n5. Insignificant improvements.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pruning and fine-tuning strategy for Time Series Foundation Models (TSFMs). The authors argue that directly fine-tuning pretrained TSFMs often leads to suboptimal performance and overfitting, frequently falling short of smaller models such as PatchTST. Through empirical analysis, they observe that a large portion of attention heads and fully connected channels are redundant, with only a sparse subset of weights are significant to forecasting tasks. Based on the importance of parameters, the model can be pruned and finetuned with only significant parameters.  Experimental results demonstrate that the pruned models consistently outperform those that are directly fine-tuned.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n1. The paper is well-written and easy to follow, with a clear structure and precise mathematical expressions.\n2. The proposed method is simple to understand and general to all TSFMs.\n3. Efficient fine-tuning of TSFMs is a timely and important direction to explore, with a great significance in real-world applications.\n4. The experiments are comprehensively conducted. The authors validate the proposed method on various TSFMs. \n\n### Weakness\n1. The originality of the proposed method seems not very significant. It appears to apply an established method from LLM-pruner to calculate importance of parameters and prune the insignificant ones.\n2. Some analysis could be more in-depth. Please refer to the Questions section for further details.\n\n\n[1] LLM-pruner: on the structural pruning 356 of large language models, NeurIPS 2023\n\nQUESTIONS:\n1. The authors define the **average relative output norm** to reflect the importance of an attention head. Is this importance metric originally proposed? This metric leads to 2 questions:\n* Why does the norm of output embedding $\\mathbf{o}$ reduce significantly (generally < 5-10 %) compared to that of input embedding $\\mathbf{x}$?\n* Can we interpret this metric as an indicator of the model’s forecasting capability on a given dataset? If so, why does the performance of large TSFM variants have a better zero-shot performance even they have a smaller proportion of significant heads compared to their smaller variants? \n\n2. For the results in Table 2, except for Chronos and TTM, the winning rates of other TSFMs are relatively low. Could the authors explain why fine-tuning these models fails to outperform PatchTST?\n\n3. For Table 2, it would be helpful to include the zero-shot performance, which can help to analyze how much improvement is achieved via fine-tuning. If space is limited, it could be provided in Appendix.\n\n4. In Figure 4, Moirai_large shows an extremely high proportion of pruned-out parameters, with both input and output channels in deeper layers reaching nearly 100%. In such cases, does the forward pass rely solely on the residual connection in these layers?\n\n5. Since overfitting is identified as a major issue, how does fine-tuning with dropout affect the performance? Have the authors conducted any comparisons or ablation studies in this regard?\n\n6. How one epoch is defined in the experiments? Is it a fixed number of batches, e.g. 100, or a full iteration of all the training samples?\n\n7. The authors use an extremely large batch size (8192) for pruning. Does batch size have a significant effect on the pruning results?\n\nSome errors:\n* Typos in the caption of Fig 2 (a)\n* The TopK operation in Algorithm 1 should select $s$ with the lowest importance. It appears that the current version selects the ones with the highest importance.\n\nLIMITATIONS:\nYes, the authors discuss the limitations of their work. However, one important aspect they overlook is the training efficiency of the proposed method. While they show the inference efficiency, it is also crucial to understand how the pruning process impacts training efficiency.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate why large pre-trained Time Series Foundation Models (TSFMs) often fail to outperform smaller task-specific models on downstream forecasting tasks, and propose a prune-then-finetune approach to help address this gap. They leverage the inherent sparsity and redundant components in a pre-trained TSFM by structured pruning (removing less-important channels and attention heads) before fine-tuning on a target dataset. This focuses adaptation on a compact, task-relevant subnetwork, aiming to preserve valuable pre-trained representations while avoiding overfitting the full model. Experiments across six forecasting benchmarks, and 7 recent TSFMs using the prune-then-finetune approach show improved forecasting performance, while also achieving up to 7 times inference speed-ups. The work positions structured pruning not merely as a compression tool but as an effective regularization approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Well-written and motivated paper: The paper is well-written, detailed, with a proper flow motivating the hypothesis and methodology, and then substantiating it with the results.\n\n2. Strong Empirical Evidence: Extensive experiments on 7 TSFMs and 6 benchmarks show the efficacy of the proposed approach.\n\n3. Method is Simple & Model-Agnostic: The authors show that the method works across encoder-only, decoder-only, and MoE TSFMs without architectural changes, thus making it easy to adopt.\n\n4. Performance and Efficiency: The approach benefits both performance and efficiency simultaneously.\n\nWeaknesses:\n\n1. Incremental Novelty: Their pruning metric adapts from LLM-Pruner, and the overall novelty lies more in application and context than in approach itself.\n\n2. Benchmark Diversity: Core results focus on ETT, Weather, Electricity, which aren't sufficient to fully justify the generalizability claims. Experiments on more diverse set of datasets would rectify this weakness.\n\n3. Hyper-parameter Sensitivity: The paper provides limited ablation on pruning ratio and pruning method, thus calling into question how robust performance gains are to these choices.\n\n4. The paper lacks other simple baselines like layer-freezing, adapters, or distillation that could isolate the benefit of removing parameters versus just not updating them.\n\nAuthors, please note: I am very open to increasing my score if these above weaknesses are answered/rectified.\n\nQUESTIONS:\n1. Generalizability to Other Datasets: Can the authors comment on the performance of their approach on additional benchmarks beyond those reported, like healthcare? The paper uses six benchmarks, which are fairly standard among time series forecasting papers. Demonstrating the method on a more diverse set (especially where characteristics differ, like highly non-stationary, noisy or shorter series) would strengthen confidence that the approach generalizes widely.\n\n2. Choice of Pruning Ratio and Strategy: How sensitive are the results to the chosen pruning fraction and strategy? How would the model perform under extreme pruning? For example, pruning 90% of the model weights?\n\n3. Effect of Pre-training Quality: The authors assume that the pre-trained TSFMs used have learned useful representations for the task. What about the case where the assumption fails, like when the TSFM exhibits poor zero-shot performance on the target domain? While the authors touch on this in the limitations, it would be interesting to see a concrete experiment where a TSFM pre-trained on electricity data and fine-tuned (with v/s without pruning) on a very different domain (say financial time series). Would pruning still help if the model’s pre-training knowledge is not directly relevant?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes a prune-then-finetune approach for adapting pre-trained large time-series forecasting models. They prune the TSFM first by removing redundant channels/heads guided by a loss-based importance score, and then fine-tune the smaller, specialized subnetwork on the downstream data. Empirical results show that pruning + fine-tuning reduces error significantly vs fine-tuning alone, and faster inference.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The problem of adapting a large TSFM is very important in the time-series field, and the proposed approach is promising.
2. Both the forecasting performance gains and computation efficiency improvements are significant
3. The overall approach is simple, and in principle applicable to other domains.
4. The paper is well written, and the experiments are adequate

Weaknesses:
1. No repeated trials for statistical significance analysis, however that is acceptable for large-scale models like these
2. It would be interesting to see if the same approach results in similar improvements for tasks such as classification, imputation, and anomaly detection using a foundation model such as MOMENT [1].

[1] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. (2024). Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885.

QUESTIONS:
Did you explore how sensitive your pruning decisions and final forecasting performance are to different random seeds? 
Is the set of pruned channels largely the same across them, or do you see very different subnetworks? 
It might be interesting to visualize the difference, perhaps for only a small subset of the complete set of experiments if the resources allow for it.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03