Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
Authors: Lifan Zhao
Yanyan Shen
Zhaoyang Liu
Xue Wang
Jiaji Deng
Venue: NeurIPS 2025 poster ()
Keywords: Time series forecasting
Primary area: deep_learning
TL;DR: We propose a prune-then-finetune paradigm to specialize time series foundation models in both levels of architecture and parameter weights, effectively improving downstream forecasting performance.

Abstract: Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting performance. Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. 
A key question is how to realize effective adaptation of TSFMs for a target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned to activate task-relevant network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on a more relevant and compact parameter space.  
Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This ``prune-then-finetune'' paradigm often enables TSFMs to achieve state-of-the-art performance and surpass strong specialized baselines. 
Source code is made publicly available at \url{https://github.com/SJTU-DMTai/Prune-then-Finetune}.

Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
Lifan Zhao' Yanyan Shen' Zhaoyang Liu², Xue Wang", Jiaji Deng",
Shanghai Jiao Tong University, AAlibaba Group
{mogician233 edu. cn, (jingmu. lzy, xue
Abstract
Scaling laws motivate the development of Time Series Foundation Models (TSFMs) that pre-train vast parameters and achieve remarkable zero-shot forecasting perfor- mance, Surprisingly, even after fine-tuning, TSFMs cannot consistently outperform smaller, specialized models trained on full-shot downstream data. A key question is how realize effective adaptation of TSFMs for target forecasting task. Through empirical studies on various TSFMs, the pre-trained models often exhibit inherent sparsity and redundancy in computation, suggesting that TSFMs have learned activate task-relevan network substructures to accommodate diverse forecasting tasks. To preserve this valuable prior knowledge, we propose a structured pruning method to regularize the subsequent fine-tuning process by focusing it on more relevant and compact parameter space. Extensive experiments on seven TSFMs and six benchmarks demonstrate that fine-tuning a smaller, pruned TSFM significantly improves forecasting performance compared to fine-tuning original models. This perune-then-fineture paradigm often enables TSFMs achieve state-of-the-art performance and surpass strong specialized baselines. Source code made publicly available httpt:: :/ wtt.:l.ccc/SUTUUUTTS -ttp:/SUUU----//PPAA then- Fann-Finntue
Introduction
Time Series Foundation Models (TSFMs) are large forecasting models pre-trained on massive datasets. Recent researches on TSFMs 1
30 have demonstrated their impressive zero-shot forecasting capabilities across various domains without being specifically trained for those particular prediction tasks. Time series forecasting has a distinct advantage compared other machine learning tasks. That is, new data naturally accumulates over time, providing fresh training samples without manual labeling. In many industrial sectors such traffic monitoring, energy consumption, weather prediction, and financial analysis, organizations continuously collect extensive time series data. This data serves two purposes: it provides long historical context for zero-shot forecasting and abundant samples for model training. Given this data availability, it becomes crucial to evaluate whether generic TSFMs offer real advantages over specialized models trained on ssecisccc datasets.
Our performance evaluations in Fig. demonstrate that even state-of-the- pretrained TSFMs oper- ating zero-shot mode frequently underperform compared specialized models like PatchTST 24).. This performance gap primarily stems from statistical differences between the diverse pre-training data and the specific patterns in downstream time series. Although fine-tuning TSFM parameters on target datasets improves prediction accuracy. our experiments show that even these adapted TSFMs struggle consistently outperform PatchT'T across various benchmarks. This raises crucial research question: how can we more effectively leverage these powerful TSFMs to achieve superior performance in specific forecasting applications?
This work done during his internship Alibaba Group. corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

130%
TTM, Time-MoFas
Time-MoFurg
Moirafuas
Moirälurge
Chrono
TimesFM2osoo
Pretrained
Fine- Fine-tuned
MSE relative to Patch1S
120%
11036
90%
80%
70%
ETTh1
ETTh2
ETTm1
ETTm2
Weather
Fig. 1: Performance comparison between TSFMs and PStchTST trained on full-shot training data of each benchmark. We calculate the average MSE of forecasting 96, 192, 336, and 720 steps, which is further divided by that of PatchTST with well-tuned hyperparameters (26). Weather is used for pre-training TimesFM and is not evaluated. The red dashed line represents the PatchTST baseline.
To answer the question, we investigate the internal characteristics of TSFMs that help explain their behaviors. Our analysis reveals that the pre-trained TSFMs often exhibit inherent sparsity hidden states and redundancy in parameters. For instance, up to 50% of attention heads yield negligible outputs throughout downstream task, and up to 60% of intermediate channels in feed-forward networks never activated. These sparsity patterns vary across datasets, suggesting that TSFMs activate different subnetworks depending on the specific task. While conventional the original model architecture, it may disrupt these task-relevant activation patterns by modifying all parameters indiscriminately. This approach risks overfitting and fails to capitalize on the valuable task-specific substructures identified during pre-training.
retains
In this paper, we propose decomposing TSFM deployment into two stages: architectural specialization through stractured pruning 3 [66 and weight specialization through fine-tuning. While structure pruning by removing redundant components like channels or attention heads commonly used to compress large models for lower latency and memory usage [22 29), our primary objective is to facilitate fine-tune -uning in a narrowed parameter space for stable performance improvement. Specifically, for each downstream task, analyze channel importance across all linear transformation layers of TSFM using a loss-guided metric [22] to identify and remove unimportant model parameters. This results in sparse model whose remaining parameters fine-tuned on the downstream data.
We conduct extensive experiments on various TSFMs and six popular time series forecasting bench- marks. The results demonstrate that fine-tuning pruned TSFM yields better forecasting performance than fine-tuning the original model in most cases, achieving up to 22% error reduction. In particular, when selecting the best TSFM on a specific benchmark, our the--iittueee approach increases the win rate against PStchTST from 90% to 100%. Moreover, pruned model achieves superior zero-shot performance when forecasting other time series within the same domain, indicating that the identified model substructure transfers effectively across similar datasets. These findings highlight the essential role of architectural specialization for TSFMs and establish the wrune-thens finettinetune paradigm as a promising strategy for effective TSFM deployment. The major contributions of the paper summarized in the following.
Identifying the TSFM specialization bottleneck: we provide comprehensive and fair benchmark comparisons showing that even fine-tuned TSFMs struggle to outperform specialized baselines in full-shot forecasting scenarios, highlighting TSFM specialization as a critical unresolved challenge.
Uncovering inherent sparsity in TSFMs: our empirical analysis reveals that pre-trained TSFMs, regardless of whether they utilize sparse Mixture-of of of-Extetss architectures, activate only task- relevant parameter subsets during downstream inference. We argue that these specialized architec- tural components represent valuable prior knowledge acquired during pre- pre-trainin and should be preserved when forecasting time series from specific domains.
Pioneering pruning for TSFMs: propose novel prune-then- some--then-ineture finetunn paradigm an effec- tive post-training pipeline for TSFMs. To the best of our knowledge, this is the first work that explores pruning TSFMs to enhance forecasting performance. Remarkably, find that up to 90% parameters can be removed while simultaneously improving performance.
Demonstrating performance and efficiency gains: we conduct extensive experiments on various TSFMs and datasets. The results demonstrate that our minimalist pruning approach frequently enables TSFMs to surpass state-of-the-art baselines in full-shot forecasting scenarios. Additionally, our structured pruning technique delivers up to 7x speedup in inference efficiency.

---

2 Related Works
2.1 Time Series Foundation Models
Recently, increasing efforts have been devoted to pre-training TSFMs large-scale time series data for a versatile starting point of various prediction tasks. Timer [21] and Timer-XL. (20] I stack multiple Transformer layers with causal attention to capture temporal correlations. Furthermore, several specialization modules are designed to handle the heterogeneity of time series. For frequency- specific specialization, Moirai (30) differentiates input/output projection layers among frequencies, while TTM 7 and TimesFM [S] include categorical frequency embeddings as input. Time-MoB [28 and Moirai-MoF: [9] adopt sparse mixture-of mixture-of-experts oepettsss (MoE) design where tokens processed by different expert feed-forward networks (FFNs). Though only few experts are activated for each token, all experts can be used and fine-tuned on the whole dataset.
While some TSFMs claim state-of-the-an performance, their comparisons set the lookback length to 96 for specialized models (e.g., PatchTST [4 while thousands for TSFMs, [ et al. demonstrate that specialized models with well-tuned hyperparameters outperform TSFMs, while they did not tune TSFM hyperparameters. In this work, after extensively tuning hyperparameters of all models, our benchmark results confirm that TSFMs still cannot surpass specialized baselines. Also, we find that applying parameter-efficient fine-tuning [13] to TSFMs is not superior full fine-tuning (see Appendix C.3 and cannot resolve the limitations in specialization. A concurrent work (25) also emphasizes that how to effectively fine-tuning TSFMs is under-explored yet important problem.
2.2 Model Pruning
Pruning conventional technique for network compression and acceleration, is increasingly applicd to Large Language Models (LLMs) for efficient inference. Based on pruning units, pruning has two categories. Unstructured pruning removes unimportant elements of weight matrices, while the sparse matrices require specialized hardware or software optimizations for acceleration (36]. Structured pruning removes entire components (e.g., channels and attention heads with the lowest importance) and is hardware-friendll. Numerous pruning metrics have been proposed to determine whether pruning unit is essential and should be pruned not. Various metrics guide pruning; for example, LLM-Pruner 22 assesses a component's importance by its impact on loss. Typically, LLMs after structure pruning struggle regain original performance even after retraining 22 36) The potential reason behind lossy compression of LLMs is that various neurons can preserve language skills and vast factual knowledge, which is critical to commonsense questions.
Crucially, the motivation our work differs from the predominant use of pruning for LLMs. Our main purpose is not to pursue inference efficiency. Instead, we devoted to improving the fine-tuning performance TSFMs by pruning neurons that are less relevant to the downstream task. The rationale that time series characterized by strong heterogeneity across domains and frequencies [1)] and specific task may only need to activate a certain fraction of model parameters (28). Moreover, due to the nature of non-stationarity, time series usually exhibit various data patterns over time, and there are scarce latent features that maintain long-term effectiveness for time series forecasting. Thus, pruning has the potential focus TSFMs on the most critical channels for given task.
Preliminary
In this section, we first formalize the notations of key modules in Transformers, which constitute the backbone of most existing TSFMs. Then, conduct a preliminary study on TSFMs in aspects of attention head outputs and FFN activations. Additionally, analysis of parameter magnitudes is provided in Appendix B.I
3.1 Transformer Architecture
Multi-Head A Attention (MHA). Typically, TSFMs organize time series observations as sequence of tokens (a.k.a,, patches). Let denote the number of input tokens and X 7'xd denote the token embeddings. Suppose there are H attention heads and each head has dи dimensions. For each i {1...., let W.K w?, w! dxdd denote key, query, value projections for the i-th head,
3

---

Table Statistics of popular TSFMs and PatchTST. dan denote the intermediate dimension of FFN.
Model
TIMA
Time-MoEsss
Time-MoEun
Moiraive
Moiraimo
Chronoswvi-he:
TimesFMa.
PatchTST
Decnder Only
Decnder- Only
Encoder-
Encoder-
Encoder-
Decoder Only
Encoder
Architecture
MLP
Only
Only
Decoder
Only
Model Size
5M
113M
453M
9IM
311M
205M
500M
500K
#laayer
12
#Head
12
16
12
12
26-384
384
768
768
1024
1024
1280
128
dm
52-768
1536
3072
3072
4096
3072
1280
256
and 1 E Rdux xd for output projections. Formally, MHA computation can be defined as below:
(XW?
XW!) (XW,))
MHA(X) = -o
softmax
w
(I)
vdH
where O; E IR Txd di is the output of the i-th head.
ford-Forward Network (FFN). Each FFN involves at least two linear layers parameterized by
Wup bup Wdoun, Idown FFN computation can be written
(
Wdown
(XW
bup +
down
(2)
activations
where a is the activation function (c.g., ReLU, GELU, SwiGLU) that can deactivate some intermediate hidden states zeros tiny negative values. TTM [7]. though not based on Transformers, also incorporates MLPs with GELU activation. For the i-th intermediate FFN channel, the probability of producing positive activations
dataset reflect its relevance the downstream task.
Residual Connection. enable deeper architectures and stabilize training, Transformer employs residual connections and normalizations. Formally, the output of a Transformer can be written as
X X + MHA(Nor)XX)), X FFN(Nor((X))
where Norm can be implemented by layer normalization or RMSNorm.
Empirical Analysis of TSFMs
According to the theory of bias-variane variance trade-offs 9.. large model can induce high variance risks predictions, which seems contradictory to the skilled zero-shot performance of TSFMs on unseen tasks. To ascertain the mechanism of TSFMs, we conduct an investigation into TSFMs through the lens attention heads and activations in FFNs. We perform 96-step forecasting on the Weather (31) and ETTml (35] datasets by PatchTST and seven pre-trained TSFMs, including TTM (7]. Time-MoEnun Time-MoElarge (28), Moirainese Moiraūjaree
(30), Chronos- TimesFMM-.00-000M [5). The key statistics of the TSFMs are provided in Table I
[[. and
3.2.1 Insignificant Outputs of Attention Heads
Empirically, we find that there are few attention heads TSFMs that usually produce insignificant modifications on residuals. For each head, we calculate its relative output norm averaged over the downstream dataset, as defined below.
Definition Given the i-th head output o, E Rd and the residual x E Rd of each input token, the average relative output norm of the i-th head is defined as Ex( Ex(1|o,112 / over a downstream dataset, which can reflect the importance of the i-th head when modifying the residual.
Fig. 2 illustrates diverse distributions of average relative output norms. Compared to PatchTST where all heads actively contribute 2 3% relative norm, few heads of pre-trained TSFMs give nearly zero norms, and larger TSFM variants tend to exhibit greater sparsity. For instance, 50% (resp. 30%) heads of TTime-MoEaa. (resp. Moirainage produce outputs with negligible relative norms (< 1%), while their smaller counterparts have fewer insignificant heads. Nevertheless, with parameters more than Time-MoFaus and Moiraisse- Chronos exhibits lower sparsity, suggesting that the sparsity degree varies with both model sizes and architectures.

---

Cumulative proportion
Cumulative proportion
0.8
0.8
Time-MoEuaa Time Mo
Time-MoEs Time Moo
0.6
0.6
Moiralaue Mairaliage
Moiral
0.4
0.4
Moiral
ChronoSto
Chronos
0.2
TimesFM
0.2
TimesFM M
0.0
0.0
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
0%
2%
4%
6%
8%
10%
12%
14% 16% 18%
/ / |4la
||l|
|x4
(a) Weather
(b) ETTml
Fig. 2: Cumulative distribution the average relative output norm one attention head over the Weather and ETTml datasets.
1.0
1.0
Cumulative proportion
Cumulative proportion
TTM
TTM
MaE
MoE
0.6
Time-MoEurg
0.6
Time-MoE Time-MoElurge
Blose
0.4
Molral arge
0.4
Molralurge
Chranas pomnorsgct.eeee base TimesFM PatchTST
Chrontsp
0.2
0.2
TimesFM
PatchTST
0.0
0.0
40%
60%
80%
100%
40%
60%
100%
Plactivation
0)
Plactivation >- 0)
(a) Weather
ETTml
Fig. 3: Cumulative distribution of the activation probability of one FFN intermediate channel over the Weather and ETTml datasets.
1.0
TTM,
TTMA
Time MoE
Time Ma
0.8
0.8
Trme-MoEarge
TTme-MoElgge
Proportion
Proportion
0.6
Moirahase
Moirahaw
Molralurge
Moiralurge
Chronos
0.4
Chronasuattorr
TimesFM
TimesFFM
0.2
PatchTST
0.2
10
20
30
40
50
10
30
40
50
Layer ID
(a) Weather
(b) ETTml
Fig. Proportion of sparsely activated channels in each FFN layer. A channel identified as sparsely activated if its activation probability is less than 5% over the downstream dataset.
3.2.2 Sparse Activations in FFNS
Shi et al. and Liu et al. found that MoE-based TSFMs tend to allocate different expert FFNs for samples from diverse datasets, and sparsely activated experts address the heterogeneity of time series. We assume that there is also sparse activation pattern at the granularity of FFN channels.
Definition 2
The activation probability of the i-th intermediate channel of FFNs is defined the frequency its activations (denoted by a:) being positive, i.c., P(a. > 0), over the dataset.
Fig. 3 highlights distinct FFN activation patterns. PatchTST utilizes their FFN channels relatively densely, with channels active least 20% computations. By contrast, several TSFMs exhibit extreme sparsity: 20-60% of FFN channels in Chronos and Time-MoF, models remain entirely inactive on the downstream data. Notably, even non-MoE architectures like Chronos and Moirai demonstrate this ability to selectively activate FFN parameter subgroups for specific tasks. This
5

---

sparsity tends to increase with model size, as seen with Moirainage being sparser than Moirai Even the smaller TTM (5M parameters) shows some FFN channels with low activation probabilities. Defining channels with <5% activation probability as unimportant, Fig. A further illustrates that significant FFN sparsity patterns are prevalent across different layers within TSFMs.
According to our analysis on attention heads and FFN activations, most pre-trained TSFMs exhibit inherent sparsity during inference, with a substantial portion of parameters being either redundant or inactive. Moreover, TSFMs exhibit different sparsity patterns over the Weather and ETTml datasets. We hypothesize that their impressive zero-shot forecasting performance stems from the ability to selectively activate a task-relevant subset of parameters, leveraging prior knowledge. This encourages us explore more tne-grained sparsity estimations and universal pruning methods.
Methodology
In this section, we first define the pruning unit and its importance score, followed by a progressive pruning and fine-tuning method to specialize TSFMs for downstream tasks.
4.1 Pruning Unit
For a comprehensive removal of task-irrelevant parameters, we define a pruning unit (i.c., the minimal pruning elements) the input/output channel of any linear layer, covering self-attention modules and FFNs. Formally, a linear layer f(-) is parameterized by transformation matrix W E Rd X and (optionally) a bias term b E Rda,, where dnn and dout are the dimension of the input and output channels, respectively. Additionally, we apply two binary mask vectors m {0, di and m E {0, 1) ddwe to the input and output of f()), respectively, which are initialized as 1. Given an input x E IR ig the masked output h E ooor defined as below:
h f(x m
m
W
(m
m out) + m
where 3 corresponds to the element- product. We can prune the i-th of W by setting m the i-th element of m to zero. Likewise, the j-th column of W and the j-th element of will be pruned if mu- O. It is worth mentioning that due to the residual connection and multi-head attention in Transformers, masking the output channels of a given layer not equivalent to masking the input channels in the subsequent layer. Therefore, treat the masking operations input and output channels independently.
4.2 Importance Score of Pruning Unit
From the perspective of forecasting loss, redundant channel, if pruned, would result in negligible loss changes. Following prior works [22], we define the channel- wise importance score as follows:
Definition 3 The importance score of channel i, denoted as 8;, is defined by 8; = |C (m e.) L (m)), where e; denote one-hot vector with the i-th element equal 1.
For computational efficiency, we derive an approximation of 8; by:
dcn,
02
8,
E, V m C
_
(V
e,
(5)
0m;
2N
dm?
acn,
@cn, Om,;
2
2
(6)
0m;
2N
x=1
where N is the number of samples involved in calculating forecast loss C, and C,, is the forecasting loss the n-th sample. In Eq. (5), we apply the second-order Taylor expansion to the forecasting loss L to efficiently evaluate the loss change. In Eq. (6). employ Fisher information to approximate the second-ordes derivative by the square the first-order derivative [8]
The derivative w.r.t. mf is computed by OL/Om," x; mL.)) In special case when ar; always zero activation, the i-th input channel has zero importance score and will be pruned. The derivative of m can be computed by @C/dmout out f(x @ m² ac/Oh. The output channels

---

TTMA
Moiraieae
Cumulative proportion
0.8
Moiralage
Time-MoEgann
0.6
TTMA
Moiral
MoEu
Moiral
0.4
Chronospat luse
Time- Time-MoE,
TimesFM
0.2
Chronos
TimesFM M
PatchTST
Patch TST
0%
2%
4%
6%
8%
10%
0%
2%
4%
66
8%
10%
(Normalized) importance score
(Normalized) importance
Fig. 5: (Left) Boxplot of importance scores, where the largest box encompasses the middle 50% of the data distribution. (Right) Cumulative distribution of importance scores, The scores are calculated based on the Weather dataset and further divided by the maximum value. Distributions on the ETTm! dataset are provided in Fig. 8
of an attention head will also have low importance scores if the outputs are marginal and the loss derivatives are small. On top of the inherent sparsity in TSFMs, the importance score metric can also measure other redundant channels whose outputs are of great value but get counteracted by other channels or activation functions deeper layers. Therefore, we adopt the importance score for comprehensive pruning to extract the critical network structure.
Analysis. Fig. 5 illustrates the distribution of channel importance scores over the Weather dataset. The Long-tailes distribution of these scores within TSFMs demonstrates significant redundancy in channels. This channel redundancy is notably more pronounced in larger TSFM variants. This TSFM behavior caters to the nature of temporal data, where the presence of concept drift 66 TO 14 32 implies that enduringly effective data patterns are scarce. Consequently, structured pruning emerges as compelling strategy selectively restrain the number of exploited latent features.
4.3 Progressive Pruning and Then Finetune
To avoid over-pruning at one time, it is desir- able to remove a small number of channels at one pruning step and re-calculate the loss at the next step. Given the high expense of inferring the whole training repeatedly, we adopt pruning based on batches. First, sam- pling the j-th batch, estimate the current importance score g03) by Eq. (6) Second, given previously calculated scores, we esti- mate an exponential moving average (EMA) by أ(t) as()) + 0:)(1-1), where a: is hyperparameter. Third, globally across all model layers, we rank g0) in ascending order and remove unpruned channels with the low- est importance based on predefined pruning ratio. Finally, we obtain pruned model after one or a few epochs over the training set. Op- tionally for acceleration, one can first prune out insignificant attention heads and sparse FFN channels discovered in the forward pass based on a predefined threshold. Then, we in- volve gradients compute importance scores of other channels for fine-grained pruning. After pruning, fine-tune the remaining pa- rameters on the downstream data.
Algorithm Prune-then-Finetune
Require:
Forecaster F with channels, data D, pruning number per batch K, EMA coefficient for = to C' do Initialize mask m; +- and score sf 0:
end for for 1 B do
Sample batch Dj from D;
Forecast D, with masks {m:}e
Calculate loss gradients
each m,;
for i 1 to C do
Calculate scores 5,6) using Eq. 6
3(1) + as! + (1 -a)) 1-1)
end for for
1 to C do if s TopK -s(i)) (f-
my 1}) then
+ 0:
end end for end for Prune channels of which 0: Fine-tune the unpruned parameters of F on
7

---

Experiments
In this section, we present extensive experiments evaluate the effectiveness of structured pruning when adapting TSFMs to downstream tasks. We compare full-shot forecasting performance with and without pruning, analyze the resulting sparsity patterns and inference speedups, and assess the transferability of the pruned TSFMs to other datasets. Additional experimental results are provided in Appendix C including short-term forecasting results on the M4 benchmark [23 and ablation studies and the pruning methods (Table
on the tuning methods (Table g)
5.1 Experimental Settings
Foundation Models. Our experiments are conducted on range of pre-trained Transformer-basedd time series forecasting models (TSFMs). including Time-MoEond Time-MoEjua Timer-XI and TTMA We inherit the open-source codes and pre-trained checkpoints from their official repositories.
Moirainas Moiraijarg 3 Chronos-boltt
Datasets. We evaluate long-term forecasting performance on three widely used benchmarks: ETT 351,
Weather [3I] and Electricity [[S]. Dataset statistics and train/validation/eest splits are summarized in Table Pruning and fine-tuning are performed on the training sets, and we report forecasting accuracy on the test sets using Mean Squared Error (MSE) and Mean Absolute Error (MAE) over horizons of 96, 192, 336, and 720 steps.
Implementation Details. To ensure fair comparison, we carefully tune all models' hyperparam- eters (e.g., learning rate, pruning ratio) on the validation set. For autoregressive TSFMs, a single model is trained and pruned per dataset and reused across all forecasting lengths. In contrast, for non-autoregressiive models (e.g.. Moirai and TTM), we train separate models optimized for each fore- casting horizon. Further details on the hyperparameter search process are provided Appendix (A.2 facilitate reproducibility. For simplicity, our main pruning strategy relies solely on importance scores. Additional results using alternative pruning criteria are presented in the supplementary material.
5.2 Forecasting Performance
As shown in Table 2 our prune-then-inettue paradigm achieves higher predictive performance than directly fine-tuning TSFMs in 83% of forecasting tasks. The prediction errors are reduced by an average of 4.4%, 4.0%, 2.5%, 3.2%, 1.3%, 3.3%, 1.0% for Time-MoBnase Time-Mo Timer-XI., Moiraiaases Moirai Chronoshott TTM A, respectively. The maximum reduction is 22.8% (for Time-MoFaas: on FTTm2), while the maximum performance drop is capped at only 1 1%. Notably, our method significantly increases the win rate several TSFMs against PStchTST, with the strong baseline surpassed all benchmarks, showcasing the remarkable potential of structured pruning. As for autoregressive predictions TimeMoE, Timer-XL, and Chronos, their performance gains from pruning become more pronounced with longer forecasting horizons, due reduced error accumulation. We posit that our pruning method goes beyond the inherent sparsity of TSFMs by deactivating more channels to regularize exploiting latent features, while fine-tuning the original model may mistakenly focus on outdated or noisy features. Different from random dropout, pruning focuses tesk-relevant subspace of parameters consistently across fine-tuning and inference.
5.3 Sparsification Analysis
As shown in Fig. 6 we compare the sparsification results in terms of the pruning ratios across input and output channels. Although both models consist of the same number of Transformer layers, Moiraiaage adopts an encoder-only architecture, while Chronospott-use follows an encoder-decoder
200M

---

Table 2: Full results of forecasting performance. "FT" is short for fine-tuning. Results of PatchTST are obtained from Li al.] A lower MSE or MAE indicates a better prediction. For each comparison between results without and with pruning, we mark the winner in bold. For each prediction task, we mark the best results in yellow. Zero-shot performance is shown in Table 10
Timer-Xi
Maieahuse
Moiral
Chrentmaa ver
PalchIST
Mathoad Tasks
FT
Pi]
FT
PrumosFF) [
FT
PrunseFT
FT
Prums=FT]
FT
Prams+F
FT
Pruns+FT
FT
Prum-FF]
JISE MAE MSE MARjESE MAE MSE MAEJMSE MAE MSE MAE MSE MAE MSE MAEJMSE MAA MSE MAEJMSE MAE MSE MAE MSE MAE MSE MAEJMSE MAE
95.
3:340 577 in Sa0 i. shap ST1 8.777 0.320 D TTPp D. TVS ILASK IL WI II 3ST a 196 RUMS Toep 591 o. THE 6,70 U. NoT y the 1L376 105 1TS iI 1SP IMs 8,38 0.wap, 17t 196
192
1.350 0.408 0.57 8,844 5.571 R,305 9.30 0,359 2.00 0416 3L359 0.433 FLASS 8.618 1LA04
0.41 14416 CA16 9.13 BA166418 DJA2 8.417 DLADS 0.392 0417 0,391 0.4000339 0.39 0.16
3.40 4:412 11,99
0.41 9.NW
3-19 041 ILA19 (.426
IL429
8,4%
BAS
0-45 DAY
B.449
8.402
2.14
0.43
9.57 0.409
0,434
0419 4 0.433
B.27
0.57
9454 0.41
B437 6 55%
(L38
8.416
0.41
8,400
0.43341111111
67%
36%--79%
91%5-195
95.
8..27j
273.0 N7700377 D. Topp-767 345 IL275 ILSTT I 1127X 516 IL272 D. USE 288 0.1 6,287 I 541 11 266-22 21 (1
2.200.22222 D
ETTR
192
1.341
1.366 0.415
0.380
0.331 3
0,383 0.558
8.411
0.382 0.355 11334 1L373 385 (U5S: 0.55 0.38 0.350 0.357 2.124 3.3
8.188 1-75 0 1.368
D.
0-499
0.371 0.46 1.778
0.378
0.331 0.35%
0.321
0L372
0.324
0.33 0.381
0.3%
0.4190.45:
0.54/2411 0.45
0.435
1.395
9.30
0.357
0.336
0.405
0.375
EL337
0.351
0.368
0.383
77%
68%
55%-78%6
56%
95.
1:220 902.0 a 200 WPi 214 MIS 8:211
8.29p TIS D. 11297 IL85 390 SI ILMA B. 512 0.345
0.
115 0,291
3.3410.7000 145
192
1.258 3.347 0.281 S84 2.65 0,545
9.257
0.333 0.355 133 8.31 1. 367 I.SI 9.5
343 C.30
0.342 2 SAS
0.525
0.327 0.333. 16
1:349 392
0.385
3o5
0,300
2-122 6.0 0.367
SSS ILARE 0.364
0.359
0.555 3331
0.300
1.452
0.406
0415
0.387
0.353
8,375
0,378
0.357
0.373
92%
64%
51%
51% 51%-52%
18%
95.
1.151 0.255 ILESS
i 166
0.255
8. 0.2566225.....0 DI
0:260 ILISI 8.273 261 IL172 8.259 INS 0:25E OLESS 2.1 I. 1.10 ILISK 01 160 1.200.2.600... 165 0:354
192
1.229 30300
1.217 8,29
1.24 0.315
9.2% 2.3 0.330 2.235 01310 11 305 FLZSI D. 2.E34 THE
0.252 19301 1I2E5 272 0.21 FL775 0123 112
0.21
0.286 0.221 0.292
356
0.313
0.273
6.342
0,381
0.356 0.283 0.344 D
ILE52 0.337
0.3-3 0:257 0.340
8.268
0.313
8.268
0.322
0.710.38
0.343
0.159
0.262
0.312
0.297
0.306 0.255
75%
57%
64%
30%
18%
86%5-9995
95. p.141
0.194 i 141 & 1950 140 193
8.139
0.190). LI9
0.226 ILUSS 0.219 1L17 226 1.1S 8.222 14T 1.0 OL147 0.19 F ILL ELERR & 150 8.145 0.166.0 9.
Weitter
192
1.196 11255
11191 8.25 191 0,211
199 0.241 2217 0.271 11209 11261 2210
1.216 9.26
2E 26 I 1L255 2I11 2EN ILIN ELEEA II O 0.000.2000 19100
356
0.250
0.309
1255 0.297
0.292
0.306
0.255
0.235
0.315
0.27
296 0.255 0.362
6.236 0.244
0.236
225
0.7990229
0.291 0.238
0.284
0,200
0.242
0.254
0.215 0,343
0.221 0.262
48%
180%
75%
20%
20%
44%
17%
91%--145
95.
K.139 0:257 02337.1180003371111111 2.2.122.....2222 0.135 8.236).148 0.243
0.138
2.66. 0.206
BLE20 ELD06 DL 1.129 0,129 0.229.22 0:235
197
1.150 0:256 ILEST 0.288 11 1701 1L162
0.261 17200777 0.164 206
226
IL11N IL22A (L347 21210 0. 168 0.355
R.177 0.275 0.176 275
0.176 0.275
0.17
0.175
0.253
0.155 0.342
0.167..366 0.165 0.303000
0.171
0.173
0.152
0.337
0,100
0.256 0.16
56%
40% 42%
3% 40%
53%
94%.
Win Rase|
46.0%
68.0%
46.0%
52.0%
11.7%
26.7%
25.0%
33.3%
18.3%
20.0%
50.0%
98.3%
85.0%
93.3%
" "Bp"" is the proportion of the remaining parameters after pruning relative to the original ones. a "Win Rate" is based on comparisons between each TSPM and PatcTTTT over all datasets.
Electricity included in the pre- trainii of Time-MoE is not evaluated, denoted by a dash (-).
8 We present performance comparison about TimesHMan sooM and the Traffic 31 dataset in Table
design. Both models generally exhibits more aggressive pruning in their deeper layers, with non- negligible pruning also observed in the shallow layers. In contrast, pruning in Chronosholt-base is more concentrated in the decoder (i.c., the last 12 layers) than the encoder. We attribute this to its use of much smaller patch size for time series tokenization, which requires the encoder to capture and aggregate more contextual information across tokens.
It is also important to note that channel pruning occurs across various layer types. Importance-score- based pruning successfully removes certain attention heads and sparsely activated FFN channels, as evidenced by the pruned input channels of Wo and droou, respectively, Furthermore, additional sparsification is introduced in other projection matrices, including W W WV. and W The observed improvements in fine-tuning performance confirm the effectiveness of our method in systematically identifying and pruning redundant components using w importance- based criteria.
5.4 Inference Efficiency
As shown in Table 3 in addition to the improve- ment in predictive performance, our proposed prune-then- teett--nn-nnnneeeee approach also gives a bonus in the speed of inference. We evaluate the com- putational efficiency of pruned TSFMs using the Weather benchmark. At each time step, all vari- ates are processed as a batch, and we report the av- erage CPU runtime during forecasting 720 steps, measured over 50 batches. Our results show that the pruned TSFMs achieve up a 7.4x speedup, with the largest gain observed for Moirai
Table 3: Inference time (in seconds) of TSFMs w/ and w/o pruning (horizon=720))
Original
Pruned
Speedup
TimeMoEguse TimeMoEange
72
70
1.0x
110
103
1.1x
Timer-X0 XL
0.18
0.16
1.1x
Moiraiguse
0.37
0.14
2.6x
Moirainage
1.86
0.25
7.4x
Chronoshatt-usee
8.51
1.26
6.5x
TTMA
0.01
0.01
1.0x
9

---

Sparsification of input channels in ChronoSpok base
(b) Sparsification of imput channels in Moirajagge
w 25 25 5% L2 20130
LoWrD
(c) Sparsification of output channels in ChronOSeat: base
(d) Sparsification of output channels in Moiraiaage
Fig. 6: Proportions of input/output channels pruned in different layers of Moiraitarge and ChronOSpott-base on the Weather dataset. Results of other models are provided in Appendix C.2
5.5
Transferability
We also observe that the pruned model exhibits strong transferability across related downstream tasks. We study Chronos using the ETTh and ETTm datasets from the electricity domain, with results summarized in Table A Notably, with appropriate source-target task pairs, the pruned model transferred from the source domain can even outperform the original (unpruned) model on the target task. This suggests that sparsity patterns learned in one domain may generalize well to semantically related tasks. Consequently, in data- scarce settings, one can leverage similar datasets to guide pruning and architecture specialization, potentially leading to improved performance.
Table 4: Transferability study (horizon=720)
Model Methods
Chronospott-buee
Zero-shot
Prune
Source- Target
MSE
MAE
MSE
MAE
0.420
0.421
0.389
0,410
ET1h2-ETTI
0.473
0.447
0.465
0.444
ETTmI-ETTIE -ETTm2
0,420
0.395
0.396
0.385
HTTm22-ETTH
0.504
0.430
0,501
0,427
ETTml =TTTI-
0.420
0.421
0.408
0,418
ETTm2-#ETTH
0.473
0.447
0.469
0.446
0.420
0,395
0.376
0.376
ETTh2-9ETTm
0.504
0.430
0.491
0.426
Conclusion
This paper sheds light the critical challenge of adapting pre-trained TSFMs for superior perfor- mance on full-shot downstream forecasting tasks. Investigating inherent sparsity and redundancy within TSFMs, we suggest that the activated task-relevant subnetwork is valuable prior knowledge of a starting point for fine-tuning. We proposed a "prume-then-finttnnè paradigm, employing structured pruning identify and extract the critical subnetwork. Extensive experiments across seven TSFMs and six popular benchmarks compellingly demonstrated that our method significantly improves forecasting performance, enabling pruned TSFMs to surpass strong specialized baselines. This work pioneers structured pruning primarily for performance enhancement via architectural specialization, offering promising pathway unlock the full potential of TSFMs in practical applications.
Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (No. 2023YFBB400202), National Natural Science Foundation of China (No. 62522211), and Key Research and Development Program of Xinjiang Uygur Autonomous Region (Grant No. 2023B01027, 2023B01027-I).
References
[1] Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix. Hao Wang, Michael W. Mahoney, Kari
10

---

Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneice and Yuyang Wang. 2024. Chronos: Learning the Language of Time Series. Transactions on Machine Learning Re-
search (2024). https:///pennrevixx.. net. http://ww..etttooommoorreeeeeeeo
[2] Saleh Ashkboos, Maximilian L. Croci, Marcelo Genmari do Nascimento, Torsten Hoefler, and James Hensman. 2024. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. In The Tweifth International Conference on Learning Representations. https:
//openreview
[3] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2024. A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations, IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 12 (2024), 10558-10578, doi:10. 1109/TPAMI
2024. 3447085
[4] Yann Le Cun, John S. Denker, and Sara A. Solla. 1990. Optimal brain damage. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 598-605.
[5] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. 2024. A decoder-only foundation model for time-series forecasting. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Fe- lix Berkenkamp (Eds.). PMLR, 10148-10167. httpa://proceedings mlr press/v235/
das24c.
[6] Dexin Duan, Peilin Liu, Bingwei Hui, and Fei Wen, 2025. Brain-inspired online adaptation for remote sensing with spiking neural network. IEEE Transactions on Geoscience and Remote Sensing (2025).
[7] Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam H. Nguyen, Wesley M. Gifford, Chandra Reddy, and Jayant Kalagnanam. 2024. Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few- Fhooht Forecasting of Multivariate Time Series. In Advances in Neural Information Processing Systems (NeurIPS 2024).
[8] Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations. https
/ //openreview wtt./eeoroommdrrrrru b3RcF7
[9] Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. 2001. The elements of statistical learning. Vol. Springer series in statisties New York.
[10] Joāo Gama, Indrundefined Zliobaitundefined, Albert Bifet, Mykola Pechenizkiy, and Abdel- hamid Bouchachia. 2014. A survey on concept drift adaptation. ACM Comput. Surv. 46, 4, Article 44 (March 2014), 37 pages. doi:10. 1145/2522811
[11] Azul Garza, Cristian Challu, and Max Mergenthaler-Canseen 2023. arXiv:2310.03589 [cs.LG]
TimeGPT-1.
[12] Mononito Goswami. Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. 2024. MOMENT: A Family of Open Time-series Foundation Models. In International Confer- ence on Machine Learning.
[13] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. (June 2021). arXiv:2106.09668 [cs.CL]
[14] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. 2021. Reversible Instance Normalization for Accurate Time-Serie Forecasting against Distribution Shift. In International Conference on Learning Representations. https://openrevien net/ forum?id==cDAAc00110.
[15] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In The 41st International ACM SIGIR Conference Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR '18) Association for Computing Machinery, New York, NY, USA, 95-104. doi:10. 1145 1145//2099778..... 3210006
11

---

[16] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning Filters for Efficient ConvNets. In International Conference on Learning Representations. https //openreview mttt//oorummmdddrppcc1a
[17] Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Qingsong Wen, Christian S. Jensen, and Bin Yang. 2024. FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting. arXiv:2410.1 1802 [es.LG]
[18) Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021. Group Fisher Pruning for Practical Network Compression. In Proceedings of the 38th International Conference Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 7021-7032. https://proceeddnggs mlr press/v139/
1iu21ab html
[19] Xu Liu. Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann. Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. 2024. Moirai-MoF: Empowering Time Series Foundation Models with Sparse Mixture of Experts. urXiv:2410.10466 [cs.LG]
https://arxiv: org/ abs///bs//210 10469
[20] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2025. Timer-XI.: Long-Context Transformers for Unified Time Series Forecasting. In The Thirteenth Interna- tional Conference on Learning Representations. hatts:://pperevieer
KMCJXjIDDr
[21] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024. Timer: Generative Pre-trained Transformers Are Large Time Series Models. In Proceedings the the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 02369-32399 https://proceeedings mlr grrsss/v335/lu22e5 html
[22] Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. LLM-pruner: on the structural pruning of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 950, 19 pages.
[23] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos, 2018. The M4 Compe- tition: Results, findings, conclusion and way forward. International Journal of Forecasting 34, 4 (2018), 802-808. doi:10. 1016/j iijoorccast. 2012 06. 001
[24] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023, Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In International Conference Learning Representations.
[25] Zhongzheng Qiao, Chenghao Liu, Yiming Zhang, Ming Jin, Quang Pham, Qingsong Wen, N. Suganthan, Xudong Jiang, and Savitha Ramasamy. 2025. Multi-Scale Finetuning for Encoder-based Time Series Foundation Models. arXiv:2506. 14087 [cs.LG] https://arxiv
org/abs/2506. 14087
[26] Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, and Bin Yang. 2024. TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. Proc. VLDB Endow. 17, 9 (2024). 2363-2377.
[27] Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. 2023. Lag- Llama: Towards Foundation Models for Time Series Forecasting. In RO-FoMo: of Few-shot and Zero-shot Learning in Large Foundation Models. https: attps://openrevieee forum?ide i YluZCLFDM
12

---

[28] Xiaoming Shi, Shiyu Wang. Yuqi Nie, Dianqi Li. Zhou Ye, Qingsong Wen, and Ming Jin. 2025. Time-MoE: Sillion-Scale Time Series Foundation Models with Mixture of Experts. In The Thirteenth International Conference on Learning Representations. https://openrevvex het/forrmm2d=<11oDDFnaaa
[29] Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long. Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. 2024. Model Compression and Efficient Inference for Large Language Models: A Survey. arXiv:2402.09748 [cs.CL] https://arxix: org htgg/bbb/2422
09748
[30] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Unified Training of Universal Time Series Forecasting Transformers. In Proceedings of the 41st International Conference
Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 53140-53166. //proceedings.nit mlr
https
html
[31] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, Vol. 34. 22419-22430.
[32] Lifan Zhao, Shuming Kong, and Yanyan Shen. 2023. DoubleAdapt: A Meta-learning Approach to Incremental Learning for Stock Trend Forecasting. In Proceedings of the 29th ACM SICKDD Conference Knowledge Discovery and Data Mining (KDD '23). ACM. doi:10. 1145/
3580305 3599315
[33] Lifan Zhao and Yanyan Shen. 2024. Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators. The Tweifth International Conference on Learning Representations.
[34] Lifan Zhao and Yanyan Shen. 2025. Proactive Model Adaptation Against Concept Drift for Online Time Series Forecasting. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM. doi:10. 1 1011/114499000 11445/6690624.311 3709210
[35] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang. Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time series forecasting. In Proceedings of the AAAI conference artificial intelligence, Vol. 35. 1110-111155
[36] Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A Survey Model Compression for Large Language Models. Transactions of the Association for Computational Linguistics 12 (2024), 1556-1577. doi:10.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Answer: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
14

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
15

---

reproducible, they Please see the NeurIPS code and data submission guidelines (https://nips.co httpa///guiieee/ooeuuuuinssssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ htttps.ccc/ubblcgguuuiie
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The computational cost is unaffordable to repeat extensive experiments on 7 x 4x across seven large models, six datasets, and four prediction horizons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
16

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
17

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
18

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Only for writing
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https /tap:/:////rri.... http://ce.......eeeeeeee//////// for what should or should not be described.
19

---

A Implementation Details
A.1 Benchmark Details
Table 5: Downstream forecasting dataset descriptions. Split denotes the number of time points in (train, validation, test) sets.
Dataset
Variate
Dataset Split
Frequency
Variate Types
ETThl
(8545, 2881, 2881)
Hourly
electricity load, oil temperature electricity load, oil temperature electricity load, oil temperature electricity load, oil temperature temperature, wind speed, pressure, rain, etc. electricity load
ETTh2
(8545, 2881, 2881)
Hourly
ETTml
(34465, 11521, 11521)
15 min
ETTm2
7
(34465, 11521, 11521)
15 min
Weather
21
(36792, 5271, 10540)
10 min
Electricity
(12280. 1755, 3509)
Hourly
We evaluate the long-term forecasting performance across six well-established datasets, including
Weather RI) Electricity [15], and ETT datasets (ETThl, ETTh2, ETTml, ETTm2) R5 A detailed
description of each dataset is provided in Table We use the full test set without the "drop-last" strategy used in PatchTST (24). We borrow benchmark results of PatchTST from TFB [26 that carefully tuned the hyperparamcters.
A.2 Experimental Details
We conduct extensive experiments based on 40 Nvidia V100 GPUs, 32 Nvidia 3090 GPUs, and Intel(R) Xeon(R) Platinum 8163 CPUs. The random seed is set to The search grid of hyperparame- ters is provided in Table Particularly, and Moirai; otherwise, these models after multi-epoch fine-tuning usually achieve lower validation MSE but higher test MSE due overfitting. To reduce tuning cost, do not run all combinations of hyperparameters. For instance, we tune the hyperparameters of pruning and fine-tuning separately. As for pruning, we first tune the coefficient of exponential moving average that is optimal for specific pruning ratio per epoch. Given four pruning ratios, we obtain four pruned models and tune the fine-tuning learning rates for each model. Alternatively, we can also select one pruned model that performs the best on the validation set and tune the learning rate of only a single model. The pruned models always follow the same batch size for fine-tuning the original model. The hyperparameter tuning for the most expensive model (e.g., Moiraijage) on the largest Electricity dataset consumes about day.
set the fine-tuning epoch to for Time-MoE, Chronos,
Table 6: Grid of hyperparameters search
Time-MoB
TTM
Time-XI.
Moirai
Chronos
Learning rate
[le-2, le-3, le-4, 1e-5, le-6, le:7, 1e-8, 1e-9)
Context length
4096
1536
2880
(2048, 3072, 4096)
2048
Patch size
1
96
[32, 64, 128)
16
Batch size for fine-tuning
(32, 64)
{256, 512)
Max. fine-tuning epoch
1
10
10
Patience for carly stop
Pruning ratio per epoch
(1 16, 2%, 5%, 10%, 15%, 20%)
Exponential moving average
(0.1, 0.2, 0.4, 0.5, 0.6, 0.8)
Batch size for pruning
8192
Max. pruning epoch
10
10
10
20

---

B Additional Empirical Study
B.1
Insignificant Weight Magnitudes
Cumulative proportion
propartion
0.8
T
TTM,
Time-Mot Tise-MoEusse
Time
MoE
0.6
MoEu
Molraliase
Moiraliave
0.4
Moiral bage
0.4
arge
Chronofuos
ChronoSsaxe
0.2
TimesFM
TimesFM
PatchTST
PatchTST
0.0
0.0
0.4
0.8
0.4
0.6
[w]] max/w
[w] / max [W]
(a) Absolute value of weight elements
(b) L2 norm of weight columns/rows
Fig. 7: Cumulative distribution of weight magnitudes, which are divided by the maximum value for comparable study across models.
On top of the analysis of sparse attention heads and FFN channels, another question is whether all parameters of TSFMs are vital importance. To ascertain it, we analyze the distribution of parameter magnitudes, which defined as the absolute value of each element in weight matrices. For comparable study across models, we normalize the magnitude by dividing each by the maximum value, ensuring all values fall within the [0, 1] range. As shown in Fig. 7a Chronos and Moirai, based on Transformer encoders, have a considerable proportion of tiny parameter weights. In contrast, well. PStchTST has nearly uniform distribution of weight magnitudes, though it is encoder-based As for decoder-only TSFMs, Timer-XL and Time-MoE also have a skewed distribution of weight magnitudes. As for the vector-wise magnitude of parameter weights (i.e., the rows columns of weight matrices), we depict the cumulative distribution in Fig. (7 where the difference between TSFMs and PatchTST becomes more significant. These findings suggest that zero-shot forecasting may be based on portion of parameters in TSFMs instead of all parameters. Nevertheless, it is worth mentioning that pruning based on magnitude is task-agnostic and may result suboptimal downstream performance. This is why we adopt loss change to compute importance scores.
B.2 Importance Scores over the ETTml dataset
We depict the distribution of importance scores over the ETTml dataset in Fig. Compared with results on Weather as shown Fig. []] the redundancy patterns of the pre-trained TSFMs vary on the datasets, indicating that there some task-specific parameters.
C
Additional Experimental Results
C.1
Short-term Forecasting Performance
Table 7 compares short-term forecasting performance of Chronos with different tuning methods over the M4 benchmark 233. Better performance can be indicated by smaller evaluation metrics including MSE (mean -quare error), MAE (mean absolute error). MASE (mean absolute scaled error), CRPS (continuous ranked probability score), and NRMSE (normalized aat--man---qaaae error), where MSE and MAE are calculated based proposed method can lead to better short-term forecasting performance than the naive full fine-tuning method.
values rather than normalized ones. In most cases,
C.2 Sparsification Visualization
We visualize more sparsification results in Fig. 9 Moirainase and Timer-XI. exhibit sparsification in input channels of output projections and intermediate channels of FFNs, which confirms that pruning based solely on importance scores can handle the inherent sparsity of TSFMs. In contrast, the sparsification of Time-Mod is distinct by mainly pruning the FFNs. This can be attributed to the fact
21

---

Moiraleae
proportion
0.8
Moiralage
Time-MoEgan
0.6
TTMA
Moiral
MoEu
Moiral
0.4
Chronospat
Time-Me Pane-MoEgg argg
TimesFM
0.2
Chronos
TimesFM M
PatchTST
Patch TST
0%
2%
4%
6%
8%
10%
0%
2%
4%
66
8%
10%
(Normalized) importance score
(Normalized) importance
Fig. 8: (Left) Boxplot of importance scores, where the largest box encompasses the middle 50% of the data distribution. (Right) Cumulative distribution of importance scores, The scores are calculated based on the ETTml dataset and further divided by the maximum value.
Table 7: Short-term Forecasting Performance over the M4 benchmark.
Chronos
MSE
MAE
MASE
CRPS
NRMSE
Zero-shot
3926423
943
3.507
0.121
0.318
Yearly (H=6)
Fune
3226941
833
3,089
0.107
0.309
Prune+FT
3125399
816
3.007
0.104
0.283
Zero-shot
1878597
578
1.224
0.077
0.229
Quarterly (H=8)
Fine-Tune
1775734
571
1.221
0.075
0.223
Prune+FT
1749123
1,225
0.074
0.221
Zero-shot
1934416
565
0.949
0.094
0.289
Monthly (H=18)
Fine-Tune
1894232
0.946
0.095
0.286
Prune+FT
1820581
556
0,920
0.091
0.280
Zero-shot
240658
258
2.08
0.038
0.089
Weekly (H=13)
Fine-Tune
239732
257
2.07
0.038
0.089
Prune+FT
238142
253
1.99
0.037
0.089
Zero-shot
359007
172
3.20
0.093
0.021
Daily (H=14)
Fine-Tune
511617
175
3.21
0.110
0.021
Prunc+FT
536501
168
3.05
0.113
0.021
Zero-shot
1154587
244
0.837
0.025
0.147
Hourly
Fine-Tume
1032867
239
0.943
0.026
0.139
(H==4)
Prune+FT
937494
227
0.805
0.024
0.131
that the usage of expert FFNs in MoE models less frequent than the multi-head attention modules, resulting in less contribution to the forecast loss over a batch. Moreover, the multiple experts may share redundancy with each other, and pruning an expert may not lead to significant loss changes. In our supplementary materials, we will provide results of pruning the attention heads based on threshold of the relative output norm.
C.3 Performance of Other Tuning Methods
Cerameter-ffficien Fine- Sine-Tuning (PEFT) has become popular approach to efficient LLM fine-tuning by reducing the number of trainable parameters. LoRA [3] one of the most prevalent PEFT methods that learns two additional low-rank weight matrices an adjustment to the original full-rank weight matrix at each layer. In this section, we apply LoRA to Chronos and TTM with the rank hyperparameter chosen from (32, 64, 128). As shown in Table|8] we do not observe considerable performance improvement of LoRA compared to full fine-tuning in most cases.
It is noteworthy that our fomeetthen-finturre paradigm is orthogonal to any model tuning methods. We conduct additional experiments by incorporating LoRA into our paradigm. Concretely, the dimensions of LoRA matrices will depend on the remaining input/output dimensions of each linear
22

---

301055 115 C% 4%
5%
(a) Input, Time-MoFeus Heuse
(b) Input, Time-MoFiar Harge
(c) Input, TimerXI.
(d) Input, Moirai
2%
w
(c) Output, Time- MoErase
(f) Output, Time-MoBusg
(g) Output, TimerXL
(h) Output, Moiralsase
Fig. 9: Proportions of input/output channels pruned out in different layers of TSFMs.
Table 8: Performance comparisons between different tuning methods. H denotes the forecast horizon. "FT" is short for fine-tuning.
Models
Chronos
TIM,
Methods
LoRA
PrunesLoKA
Prune-Full
LoRA
Prunc-LoRA
PrunesFull
Daatssets
MISK
MSE
MAK
MSE
MAE
MSS
MSK
MSK
MAE
MSK
M.A.
0.366
0.374
0.575
0.366
0.375
0.395
0.358
0.393
0.358
0,393
0.420
0.405
0,405
0.417
0.405
0.417
0.405
0.420
0.392
0.417
0.388
0.411
0.391
0.430
0:456
0.425
0.450
0.422
0.449
0.406
0.429
0.402
0.424
0.454
0.451
0.437
0.451
0.437
0.464
0.440
0.465
Avg.
0.425
0.411
0.422
0.411
0.421
0.410
0.421
0.410
0.430
0.402
0.427
0.400
0.424
0.400
0,423
0:767
0.312
0
0.311
0.266
0
0.333
0.364
0.383
0.262
0.929
0.362
0.328
0.235
0.356
0,321
0.572
0.324
0.370
0.391
0.376
0.389
0.370
0,351
0.397
0.352
0.396
0.386
0.395
0.418
0,438
0,242
0.434
0.389
0.415
11,05
0.539
0.389
0.432
0.393
0.437
Avg.
0.348
0.370
0,369
0.342
0.368
0.339
0,368
1.333
0.385
0.332
0.382
0.333
01.855
0,315
0.285
0.315
0.233
0.315
0.36
11295
0.346
0.259
0.339
0.291
0.340
0.310
6.344
0
0.445
0,20
0.344
0.326
0.24
0.368
0332
0,36
0.327
0.365
0-327
0.363
FT7ml
0.363
0.968
02600
0362
0.367
0.356
0,67
6.58
0,356
0:357
0.382
0.356
0.381
720
0.434
0.009
0.434
0110
0420
0.405
0.123
0000
0.894
0
0,409
0.020
0.396
0.305
0.365
0.406
Avg
0.353
0.359
0.352
0,361
0352
0.358
0.347
0:358
0.345
0.77
0.348
0.379
0.342
0.373
0.342
0.373
96
0.161
0,212
0.160
0,231
0.159
0.232
0.157
0:231
0.161
0.254
0.160
0.251
0.161
0.248
0.160
0.248
192
0.227
0.277
0.225
1.276
0125
0.276
0.119
0.274
0,213
0.290
11213
0.289
0.212
0.286
0.213
0.286
ETTm2
336
0.287
0.518
0.286
0,317
0.280
0.315
0.376
0.312
0.362
0.323
(1269)
0.326
0.266
0.320
0.268
0.322
720
0.383
0.579
0.381
0,377
0.368
0.371
0.363
0.369
0.341
0.373
0.346
0.379
0.357
0.379
0.338
0.369
Avg:
0:265
0.302
0.263
0.300
0.258
0.295
0.254
0.297
0.244
0.310
0.247
0.311
0.249
0.308
0.245
0.306
96
0.156
0.190
0.174
0.209
0.152
0.191
[.14]
1.175
0.151
0.208
0.150
0.207
0.145
0.195
0.146
0.195
192
0.198
0.233
0.231
0.258
0.268
0.239
(L186
P.200
0.192
0.245
0.192
0.243
0.189
0.241
0.188
I..240
Wenther
336
0.248
0.272
0.289
0.258
0.267
0.283
0.339
0262
0,240
0.261
0.237
0.279
0.238
0.278
0.236
0.277
720
0.324
0.323
0.376
0.347
0.330
0.329
0.319
0.315
0.301
0.323
0.314
0.335
0.314
0.358
0.31.
0.335
Avg.
0.232
0.255
0.268
0.278
0.238
0.251
0.221
0.245
0.222
0.264
0.225
0.265
0.222
0.263
0,231
0.362
transformation matrix, instead of the original ones. Similar to the performance gap between full fine-tuning and LoRA over the original TSFMs, applying full fine-tuning pruned TSFMs is usually better than applying LoRA to pruned TSFMs. As for comparisons between LoRA and Prune+LoRA the pruned ones can yield better performance except for a few cases. It is noteworthy that most GPÜ memory overhead during time series forecasting is caused by self-attention computations over long contexts, rather than the model parameters and the optimizer. Since the model sizes of existing TSFMs are much smaller than LLMs, parameter-efffcient fine-tuning methods do not significantly reduce memory. Thus, we would like adopt full fine-tuning for better performance.
C.4 Performance of Variant Pruning Methods
In light of the inherent sparsity of TSFMs, we introduce a variant of TSFM pruning, denoted as Pruncesat- This method prunes TSFMs based on statistics during the forward pass, specifically including the relative output norm of attention heads and the activation probability of FFN channels. We collect these statistics over the whole training set and subsequently remove insignificant heads and channels whose statistics fall below a predefined threshold. The threshold for relative output norm is selected from (0%, 0.5%, 1%, 2% J. while that for activation probabilities is selected from (0%, 1 1%, 2%, 5% As a counterpart, the pruning method based on importance scores is denoted as Prumeimp
23

---

Table 9: Performance comparison between different pruning methods. H denotes the forecast horizon. "FT" is short for fine-tuning.
Medel:
Timehlako.r
Tirand
TIM
Metbosh
F-
Prun
Prent.g
+FT
F1
Data
1M5E
MAE
MSS
MAE
MSE
MAA
MISE
MAE
MSE
MAE
MSE
MAE
MISE
MIA
MISE
MAA
MBE
MAE
MISE
MAB
MSE
MAE
MSSE
MAA
13330
8:374
0.403
0/844
0:417
B411
12418
14933
0415
0.423
6A11
0,833
0.3M
11434
0.391
8.40
ETTT
1.000
0.333
0.393
0.413
1.09
0.45
0.400
6.43
0435
0.333
0.819
6,42
0.45
0,24
0.453
0.334
9.409
6,422
8.000
0431
0.008
0.631
9.603
10.91
18:22
9.378
01417
0388
0406
0:466
DA1
16467
1842
1643
0:488
0/211
6.40
0442
0:39
16425
0440
6422
0.233
0.318
01311
01:01
0.331
0.754
0:383
0.30
0.334
0:357
I.385
0.31
0.191
0.333
0.521
11:33
0351
6,70
E1112
336
0.380
16415
0,62
01441
8.912
8.13
0.661
0,311
8.50
14389
0.300
1.912
0.75
03311
0.307
0.387
0.70
0,385
0.551
0337
0.300
11337
0.51
1.3%
10:553
0.393
0,400
01432
0.366
0.32
0.36
0.314
0339
0.312
0.35
0.575
10346
03399
0.386
0.367
0.39
0.965
0.355
0.332
01315
0.355
0
13.384
11:47
9.27H
0.543
0.34
0356
0.364
0.300
13329
16345
0.344
0.36
03352
0.300
0.501
11:30
0.57
0.363
0.340
0.393
0.383
0.382
0.555
0.311
0:500
0.355
0.583
16383
0.48
0.985
036
0.363
0.361
0.36
0.366
6.30
0.386
0.33
0.588
00354
0.566
0.31
10:39
0:315
11:59
0.36
0.76
13355
1033
0.347
0.351
11379
10:35
9.42
03755
0.161
0.100
0.330
0.384
8.317
6.39
0335
0.41
0.379
0.528
077%
0.756
0.734
0.119
6.754
0.384
0,315
0351
0211
6.36
336
033113
14367
0,316
0.353
8.73
8.342
0.91
0.313
0.297
1L314
0.79
1.332
0.266
16313
0.287
0.313
0.76
0.312
0.209
0.33
0.71
1L335
8366
1322
10.297
003340
9.291
01335
8.266
0.32
0235
0.317
0.265
0.316
00.60
10333
08302
0.244
0,277
01311
0.296
00334
0348
1.39
1.100
0.306
11:351
8.191
0,349
8.191
0.29]
13251
11256
01314
0.166
0.311
0.191
11315
0.18
1.30
Woodter
736
0.260
10.300
0:241
11321
8:49
6,301
11389
14204
0.365
0:3W
0.362
0.33
0.77
0:336
1L376
0.366
0.393
6,270
0.34)
0.19
0.335
0.336
10256
0:235
10.58
0:21
0,322
0.335
11:29
0:33
0.500
6.331
0.569
0.31
0.65
Thalk:
330
03377
0.775
11231
0.574
8,211
0:387
0.373
0.305
1L.11
0.085
0.356
The Wesller and Trallic clttascts incladed in the pre-triizing sec nut coulustod, Jennted by dash
130%
TTM, Time-MoF towe
Time-MoFssg
Moiralgae
Moiraligg Chrono6
TimesFMy assok
B Pretroined
Fine tuned
MSE relatvv Patch15)
120%
1100%
100%
90%
80%
70%
ETTh1
ETTh2
ETTml
ETTm2
Weather
(a) Horizon=96
130%
MSE relatie to Patch15T
120%
110%
100%
90%
80%
70%
ETTh1
ETTh2
ETTm1
ETTm2
Weather
(b) Horizon=720
Fig. 10: MSE of TSFMs and PatchTST when forecasting 96 and 720 steps.
Table 9 presents forecasting performance of TSFMs specialized by variant pruning methods and full fine-tuning. The method "Prunesta++FT"" usually achieves better performance than fine-tuning, which confirms the effectiveness of architecture specialization. Nevertheless, "Pruncetat+FT" cannot outperform "prune FFT" in most cases, indicating that only pruning attention heads or intermediate FFN channels is not sufficient for specialization. In few cases, "Prunc.np +FT" underperforms "Prunesta:+FT" +FT", e.... given Time-MoFRase on ETTh1 where "Prunestat removes several attention heads while "Pruneisp+F7 +FT" primarily focus on FFNs. We conjecture that more comprehensive pruning strategy, combining both statistical and importance score-based criteria, could potentially lead to better performance.
C.5 Zero-Shot Performance
We provide comparisons between zero-shot performance and full-shot performance in Table T0
24

---

D Limitations and Future Works
Our pruning metrics are calculated based on the training set, which may incur overfitting risks in extremely non-stationary time series data. To obtain a robust pruned model, possible solutions include data augmentation and adversarial attack techniques that keep the pruming metries insensitive noise. Furthermore, our pruning method works on TSFMs with strong zero-shot forecasting performance, since our basic assumption is that the prior knowledge of architecture specialization acquired by pre-training is effective for downstream forecasting. As for some tasks where pre-trained TSFMs show poor capabilities and cannot deliver an effective subnetwork, one possible solution is to apply pruning fine-tuned TSFM, rewind remaining parameters to the pre-trained weights, and fine-tune the pruned model. This follows prior works based on the classic Lottery Ticket Hypothesis [8), while would like to leave it as future work.
25

---

Table 10: Full results of forecasting performance. "FT" is short for fine- mee-tunnng. Results of Patch are obtained from Li et al. A lower MSE or MAE indicates better prediction. For each comparison between results without and with pruning, we mark the winner in bold. For each prediction task, we mark the best results in yellow.
MAE
6.455 0450 440
0LA0) 1642
R. 022777
321 0.372 0.321 01372 0.334 6,50 0.30
0.400
1.55 6,59 O 9.56 0-404
0457 9457 0.505 0-33
(300)
S17 291 0.50 316 U.51 0 36N 1. 6.37 613S8 O 3500 WO O SH 33 63F 0.30 199 3.000 300 O 0S 1566 3555 U USS O 331 USS 0312 0.33 o 3es 0 u 333 3 U F I 2.0 30 1:351 8.30 12
E300
38 01411 1 SHO WE S5 H.MI 371 9 AND 380 NAS 80 KOM O DE aIS O de A RHIC wss Neo o 0388 U 128 S00 SW to U 0350 385 o 400 18 US 1S9 355 0330 352 0556 0,337 8.1378N0 8N0 0. 0.130
58 0385 O 35 058 0.55% 8:81 8 0.500 190
Mo B RE RE 839 Rali 0416 015.4222
57 88 0,57 0:549 0230
The den UESS 11100 11100 2228 11000155
1 NEW 0313 2.20 0221 0.200
0255 0313
02440000
0215 614 614003 035 0178 0110 02130 0100022
WW 0200 16 1 0.16 0.263 0 JA 0 0-363
0:322 0.295
0:201
0.54
63F FOM NISS ARE 1H 135 0.13 0.35 0:35033
6.26202225
6,363 002302255
AL DESI 022 122 8L19 0.295 N
(1277 12 D2ST 0.00 UISSI 0L140 0.350 0.35 0.16
91%-455 28--88
369 ILEM (L30 333 035 0 ILES 1133 0 6.N5
95% -W9
MWS -9%
8,277
6.263
LAM 0-0000 o tieo 358 0-100 18 a 409 9770 4 133 n 63771 0.48 1 400 TH 5 9 9.9 (.3 (LAN 3SI ( 0.15 595 (351
in 557 a ahn S& 434 048) 0.454 0 I 4 D o 445 8339% 0.33 410 att 322 0444 I MENN 4442 o a 30 5000400 402 n443 03%% 0.43 D 4330 2300-221 a 0418 (L3R (415 MS 04 MAS n. EE 0,303
0. 200 UI 1337 0:37 0:40 2.20000 uEE 0331 Tr U:31 20 0E3 DO ULEST
04370-007
FraneFF Zero deoi MIEN
(1466
98.N 71.%%
IN
(44180
0.41 11.40 (14380
3ES 1377 6353 0 31399 uSS 0.34 1.777
55%
50%
1%%
55%
20
Chruneh to
1AE 0411 0411
032 08
Zevestol
aaai Me0 39 339 a 3914 1EE LSM 355 SSSUSS 395 D3S LOW 0351 0.33 0 0
0.4770 O AP 6. 3H1 EV OR CSRO t nE +3 8 LASH an o ai all GND B UED da 2 In nam naw 0400 a-s 10000 0-8
2010
T0 0 USS 11N0 EE 172 10000000 Q.M 0 02e0 Meeeo1s 139053
LESS LESS 1 22 INT FL ALTE 0 9e3 arssorrsss USSTO
650F R USE 0 SE 01 1 0E15 NESSSEEOOO 1O LI9
0177 71 LLST 2 2 0 ME USF 2 1E5
500%
0.5700 322 o 331 0:331 03335 8.30 SN 0.3 0.W a 0330 t. 31t
0-43)0
0335 1-57 6454 0,44
0.42
0 2200 2 SWNING 11 2AW WID W OF 2100 SOOEE 3 N 01 ate e 3100 1000000 a dE 01330 www G 300 HWEES 10 022 TH NOW 0222 AN
27% 27% 2o0g
SSS 43%
Prineet
20.0%
R25 S 1E RE9N S O RO I C S1 6357 1.1 020
Metcalow
0 1 DEu 2 239 133 ust U22 US0
1U0 1..%
45-0-55
0.43100
Zero
NEW 8100
Praner
58 60%
55% 55% 78%
5t -465
c2 in in 120 nL 1020
MN -5% -7%
0.55 11-427
ar mo 273 0:35 12
f 115 11 nor VEMI 19 0 12 12 59 T 1 D 10 0 W 030 WE
0285 0
3
Frune-1 Zere shot Marak... ar
0,355
418 0.434
02
61000 12 1 11% 01 00000010000000000000000000 0410 102
6.453 (.65 100....0....... 4420
0. ana 422
0
21.8
62H4 W2 WE 0 1 NE 0E1E
01800 S3 01S CN NEE 01260
8.30 400041110000000 0 42n 0.3%
67%
26.7%
Time.t
6 46
10-180 41 510372 4 411 a 415 o 4000 MM 6-111 ILEIN 01515 1150-100 4100 369
3M
11.7%
na a53 03250 200 12
Zero-bat
433 42300-44
3307 00 35T 0. 315 1.. I 3I ARS 4515 u. 3060 ANN 0.375 (L397 O 0.35 300
ANT Tu 3 20T0 212 RO 0.311 (L390 D.30T 3S
cn 20
577 313 The 3T66 an 300 STH 6.20 635 3000
100E
383 202 212 0. 0.20
022
9
MAK
Prenect
67%
3%
77%
WIS THTH
5200
NISE Prunest Zere TimeMal. Larra
312322
403 0.3
SMD 11500 123
H 10 1 3 0 W WI
1000
12
MAE
0
0
394
01700-77.4443 499 11-111444444 1L4S7
311 1
0 11377 8.23
354 129 & 117 NEW
15
17%
45%
32%
90%
15 13
18015
(0:57 SEERUC 10
3090 2E
311 O 2000
Timelats
0
201
10
1679
Zero dhot
30
20 2030
10:33
192 0.557:0300 106
38 10:055
0.276617:00
17:000
13
0.351
50 0.00
0
Aparam
Aparins
Apammm
Modes
Mechoni
20
192
536
720
Ang.
192
596
720
Ang.
192
596
730
Avg.
536
720
Ang.
MID Count
Datnsers
ETMM
Briml
ETima
Weater
BCL.
26


Reviews:
[{"review_id": "uNGSsxLgD4", "reviewer": "Reviewer_3GcC", "summary": "The paper proposes a prune-then-finetune approach for adapting pre-trained large time-series forecasting models. They prune the TSFM first by removing redundant channels/heads guided by a loss-based importance score, and then fine-tune the smaller, specialized subnetwork on the downstream data. Empirical results show that pruning + fine-tuning reduces error significantly vs fine-tuning alone, and faster inference.", "questions": "Did you explore how sensitive your pruning decisions and final forecasting performance are to different random seeds? \nIs the set of pruned channels largely the same across them, or do you see very different subnetworks? \nIt might be interesting to visualize the difference, perhaps for only a small subset of the complete set of experiments if the resources allow for it.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The problem of adapting a large TSFM is very important in the time-series field, and the proposed approach is promising.\n2. Both the forecasting performance gains and computation efficiency improvements are significant\n3. The overall approach is simple, and in principle applicable to other domains.\n4. The paper is well written, and the experiments are adequate\n\nWeaknesses:\n1. No repeated trials for statistical significance analysis, however that is acceptable for large-scale models like these\n2. It would be interesting to see if the same approach results in similar improvements for tasks such as classification, imputation, and anomaly detection using a foundation model such as MOMENT [1].\n\n[1] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. (2024). Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Sensitivity to random seeds**\n\nThanks for your advice! We prune Chronos-bolt-base in one epoch with two different random seeds, and the Jaccard similarity between the two sets of pruned channels is reported in the following table.\n\n|                    | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather |\n| ------------------ | ----- | ----- | ----- | ----- | ------- |\n| Jaccard similarity | 97.6% | 98.3% | 95.1% | 94.7% | 93.8%   |\n\nWe do not observe significantly different subnetworks. One reason is that, during pruning, we set a large batch size (e.g., 8192 as recommended) to ensure that each sampled batch could cover diverse data patterns. Consequently, randomness mostly lies in the order of training batches, of which the total number is small (e.g., ETTh1 has 14 batches).\n\nWe will visualize the layer-wise and channel-wise difference by figures in our future manuscript.\n\n**W1: Repeated trials**\n\nWe repeat runs with 5 different random seeds, and we report MSE with standard deviations in the following table.\n\n|                                 | ETTh1          | ETTh2          | ETTm1          | ETTm2          | Weather        |\n| ------------------------------- | -------------- | -------------- | -------------- | -------------- | -------------- |\n| Chronos-bolt-base (horizion=96) | 0.366 ± 0.0005 | 0.416 ± 0.0007 | 0.282 ± 0.0011 | 0.157 ± 0.0009 | 0.142 ± 0.0015 |\n\nThe standard deviations are considerably small. We posit that fine-tuning foundation models have fewer randomness than traditional models since all trials are based on the same pretrained weights without random initialization.\n\n**W2:  Other tasks using MOMENT**\n\nDue to the time limit, we are preparing codes and experiments, and we may provide results in the next week. Thanks!"}, {"review_id": "Ygqpy3QL5a", "reviewer": "Reviewer_ujDP", "summary": "The authors investigate why large pre-trained Time Series Foundation Models (TSFMs) often fail to outperform smaller task-specific models on downstream forecasting tasks, and propose a prune-then-finetune approach to help address this gap. They leverage the inherent sparsity and redundant components in a pre-trained TSFM by structured pruning (removing less-important channels and attention heads) before fine-tuning on a target dataset. This focuses adaptation on a compact, task-relevant subnetwork, aiming to preserve valuable pre-trained representations while avoiding overfitting the full model. Experiments across six forecasting benchmarks, and 7 recent TSFMs using the prune-then-finetune approach show improved forecasting performance, while also achieving up to 7 times inference speed-ups. The work positions structured pruning not merely as a compression tool but as an effective regularization approach.", "questions": "1. Generalizability to Other Datasets: Can the authors comment on the performance of their approach on additional benchmarks beyond those reported, like healthcare? The paper uses six benchmarks, which are fairly standard among time series forecasting papers. Demonstrating the method on a more diverse set (especially where characteristics differ, like highly non-stationary, noisy or shorter series) would strengthen confidence that the approach generalizes widely.\n\n2. Choice of Pruning Ratio and Strategy: How sensitive are the results to the chosen pruning fraction and strategy? How would the model perform under extreme pruning? For example, pruning 90% of the model weights?\n\n3. Effect of Pre-training Quality: The authors assume that the pre-trained TSFMs used have learned useful representations for the task. What about the case where the assumption fails, like when the TSFM exhibits poor zero-shot performance on the target domain? While the authors touch on this in the limitations, it would be interesting to see a concrete experiment where a TSFM pre-trained on electricity data and fine-tuned (with v/s without pruning) on a very different domain (say financial time series). Would pruning still help if the model’s pre-training knowledge is not directly relevant?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Well-written and motivated paper: The paper is well-written, detailed, with a proper flow motivating the hypothesis and methodology, and then substantiating it with the results.\n\n2. Strong Empirical Evidence: Extensive experiments on 7 TSFMs and 6 benchmarks show the efficacy of the proposed approach.\n\n3. Method is Simple & Model-Agnostic: The authors show that the method works across encoder-only, decoder-only, and MoE TSFMs without architectural changes, thus making it easy to adopt.\n\n4. Performance and Efficiency: The approach benefits both performance and efficiency simultaneously.\n\nWeaknesses:\n\n1. Incremental Novelty: Their pruning metric adapts from LLM-Pruner, and the overall novelty lies more in application and context than in approach itself.\n\n2. Benchmark Diversity: Core results focus on ETT, Weather, Electricity, which aren't sufficient to fully justify the generalizability claims. Experiments on more diverse set of datasets would rectify this weakness.\n\n3. Hyper-parameter Sensitivity: The paper provides limited ablation on pruning ratio and pruning method, thus calling into question how robust performance gains are to these choices.\n\n4. The paper lacks other simple baselines like layer-freezing, adapters, or distillation that could isolate the benefit of removing parameters versus just not updating them.\n\nAuthors, please note: I am very open to increasing my score if these above weaknesses are answered/rectified.", "was_revised": true, "final_justification": "I have increased my score from 4 to 5. The authors answered all questions with strong empirical results, and systematically tackled the weaknesses I mentioned in their rebuttal. The methodology is strong, while a little derivative, and the experiments show strong empirical performance. The authors cover a wide range of experiments and justify design decisions empirically too.", "rebuttal": "**Q1 & Q3 & W2: Generalizability to Other Datasets in the financial and healthcare domains. Would pruning still help if the model’s pre-training knowledge is not directly relevant?**\n\nThanks for your advice! As for healthcare, we additionally perform short-term forecasting over the Covid Death benchmark borrowed from [GIFT-EVAL](huggingface.co/datasets/Salesforce/GiftEval/). As for financial time series, we perform short-term forecasting over the M4 Yearly benchmark. It is worth mentioning that Chronos included merely no *yearly* financial time series, and thus M4 Yearly is distinct to its pretraining knowledge.\n\nThe results of Chronos-bolt-base are reported in the following table. Our proposed method outperforms the fine-tuning baseline on almost all the metrics and datasets, and the improvement on Covid Death is more significant than M4 Yearly. \n\n|                                          | Chronos-bolt-base   | MSE         | MAE     | MASE      | CRPS      | NRMSE     |\n| ---------------------------------------- | ------------------- | ----------- | ------- | --------- | --------- | --------- |\n| M4 Yearly (context length$\\in[19, 284]$) | Zero-shot           | 3926423     | 943     | 3.507     | 0.121     | 0.318     |\n|                                          | Fine-Tune           | 3226941     | 833     | 3.089     | 0.107     | 0.309     |\n|                                          | Prune-then-Finetune | **3125398** | **816** | **3.007** | **0.104** | **0.283** |\n| Covid Death (context length=212)         | Zero-shot           | 641888      | 164     | 38.9      | 0.047     | 0.301     |\n|                                          | Fine-Tune           | 577110      | 157     | **37.5**  | 0.046     | 0.286     |\n|                                          | Prune-then-Finetune | **239526**  | **93**  | 49.8      | **0.034** | **0.184** |\n\nAs for the MLP-based TTM model, its released version suitable for M4 Yearly only has 935 parameters. Applying pruning to this tiny model, we observe a performance decline on the validation set, i.e., the optimal pruning ratio after hyperparameter selection is 0. Thus, we only report the performance of TTM on Covid Deaths as follows:\n\n|                                 | TTM                 | MSE        | MAE     | MASE     | CRPS      | NRMSE     |\n| ------------------------------- | ------------------- | ---------- | ------- | -------- | --------- | --------- |\n| Covid Death (context length=90) | Zero-shot           | 2181495    | 327     | 53.5     | 0.123     | 0.56      |\n|                                 | Fine-Tune           | 688993     | 125     | 30.8     | 0.047     | 0.361     |\n|                                 | Prune-then-Finetune | **584696** | **111** | **29.2** | **0.042** | **0.287** |\n\nFrom the results, we can see that pruning still brings consistent benefits even when the model's pre-training knowledge is not directly relevant. The sparsification effect essentially acts as an implicit regularizer, helping the model adapt more effectively to new domains with limited or different characteristics from pretraining data.\n\n**Q2 & W3: Sensitivity to the chosen pruning fraction and strategy? Performance under extreme pruning, e.g., pruning 90%?**\n\nThat is a valuable question! We studied only moderate pruning ratios (5%-20% per epoch) in our previous hyperparameter tuning and now extend the pruning ratio to 90% according to your advice. In the following table,  we report the average MSE of Chronos-bolt-base with the horizon in {96, 192, 336, 720}.\n\n| Pruning ratio | 5%        | 10%       | 15%   | 20%   | 90%       | Zero-shot |\n| ------------- | --------- | --------- | ----- | ----- | --------- | --------- |\n| ETTh1         | **0.417** | 0.419     | 0.421 | 0.424 | 0.457     | 0.439     |\n| ETTh2         | 0.342     | **0.339** | 0.340 | 0.345 | 0.362     | 0.361     |\n| ETTm1         | 0.347     | 0.348     | 0.347 | 0.348 | **0.344** | 0.391     |\n| ETTm2         | 0.256     | 0.255     | 0.254 | 0.254 | **0.247** | 0.283     |\n| Weather       | 0.226     | 0.220     | 0.220 | 0.221 | **0.215** | 0.242     |\n| Electricity   | 0.151     | 0.150     | 0.150 | 0.151 | **0.149** | 0.154     |\n\nThe results are not much sensitive to small pruning ratios. Performance after extreme pruning cannot get recovered after finetuning on small datasets (ETTh1 and ETTh2), where we can observe worse performance than zero-shot forecasting. However, given abundant training samples in other datasets, finetuning the model with 90% parameters pruned (from 207M to 30M) achieves the best performance. This is an interesting phenomenon that can also be observed at Moirai-large on Electricity (see Table 2 in our paper).\n\nAs for the pruning strategy, please refer to Appendix C.4 in our supplemental materials. The optimal pruning strategy siginificantly varies by models and datasets, while both pruning strategies outperforms the fine-tuning baseline in most cases.\n\n\n\n**W4: Baselines that just not updating weights**\n\nThanks for your insightful advice! We implement the classic head probing baseline that only fine-tunes the final projection head and freezes all other layers. As for parameter-efficient fine-tuning, we report results of LoRA in Appendix C.3 in our supplementary materials, where full fine-tuning often outperforms LoRA though insignificantly. In the following table, we compare the average MSE and MAE with the horizon in {96, 192, 336, 720}.\n\nChronos-bolt-base:\n\n| Methods  | Probing |       | LoRA  |       | Full      | Finetune  | Prune-then- | Finetune  |\n| -------- | ------- | ----- | ----- | ----- | --------- | --------- | ----------- | --------- |\n| Datasets | MSE     | MAE   | MSE   | MAE   | MSE       | MAE       | MSE         | MAE       |\n| ETTh1    | 0.420   | 0.411 | 0.424 | 0.411 | 0.420     | **0.410** | **0.417**   | **0.410** |\n| ETTh2    | 0.348   | 0.373 | 0.344 | 0.370 | **0.339** | **0.368** | **0.339**   | **0.368** |\n| ETTm1    | 0.356   | 0.360 | 0.353 | 0.359 | 0.345     | 0.357     | **0.344**   | **0.356** |\n| ETTm2    | 0.252   | 0.301 | 0.265 | 0.302 | 0.255     | 0.297     | **0.247**   | **0.296** |\n| Weather  | 0.226   | 0.250 | 0.232 | 0.255 | 0.261     | 0.268     | **0.215**   | **0.243** |\n| ECL      | 0.152   | 0.238 | 0.154 | 0.240 | 0.154     | 0.240     | **0.150**   | **0.237** |\n\nTTM:\n\n| Methods  | Probing |       | LoRA  |       | Full  | Finetune | Prune-then- | Finetune  |\n| -------- | ------- | ----- | ----- | ----- | ----- | -------- | ----------- | --------- |\n| Datasets | MSE     | MAE   | MSE   | MAE   | MSE   | MAE      | MSE         | MAE       |\n| ETTh1    | 0.403   | 0.424 | 0.403 | 0.430 | 0.402 | 0.427    | **0.399**   | **0.423** |\n| ETTh2    | 0.333   | 0.385 | 0.333 | 0.385 | 0.333 | 0.385    | 0.333       | **0.383** |\n| ETTm1    | 0.353   | 0.377 | 0.345 | 0.377 | 0.348 | 0.379    | **0.342**   | **0.373** |\n| ETTm2    | 0.252   | 0.307 | 0.244 | 0.310 | 0.247 | 0.311    | **0.245**   | **0.306** |\n| Weather  | 0.225   | 0.264 | 0.222 | 0.264 | 0.223 | 0.265    | **0.221**   | **0.262** |\n| ECL      | 0.162   | 0.261 | 0.192 | 0.287 | 0.161 | 0.258    | **0.160**   | **0.256** |\n\nThe results demonstrate that full finetuning is a strong baseline, and neither head probing nor LoRA exhibit significant performance improvement. In a few cases, head probing or LoRA outperforms full finetuning, while our proposed prune-then-finetune method still achieves the best performance."}, {"review_id": "JHtxUhZpeo", "reviewer": "Reviewer_EirF", "summary": "This paper proposes a pruning and fine-tuning strategy for Time Series Foundation Models (TSFMs). The authors argue that directly fine-tuning pretrained TSFMs often leads to suboptimal performance and overfitting, frequently falling short of smaller models such as PatchTST. Through empirical analysis, they observe that a large portion of attention heads and fully connected channels are redundant, with only a sparse subset of weights are significant to forecasting tasks. Based on the importance of parameters, the model can be pruned and finetuned with only significant parameters.  Experimental results demonstrate that the pruned models consistently outperform those that are directly fine-tuned.", "questions": "1. The authors define the **average relative output norm** to reflect the importance of an attention head. Is this importance metric originally proposed? This metric leads to 2 questions:\n* Why does the norm of output embedding $\\mathbf{o}$ reduce significantly (generally < 5-10 %) compared to that of input embedding $\\mathbf{x}$?\n* Can we interpret this metric as an indicator of the model’s forecasting capability on a given dataset? If so, why does the performance of large TSFM variants have a better zero-shot performance even they have a smaller proportion of significant heads compared to their smaller variants? \n\n2. For the results in Table 2, except for Chronos and TTM, the winning rates of other TSFMs are relatively low. Could the authors explain why fine-tuning these models fails to outperform PatchTST?\n\n3. For Table 2, it would be helpful to include the zero-shot performance, which can help to analyze how much improvement is achieved via fine-tuning. If space is limited, it could be provided in Appendix.\n\n4. In Figure 4, Moirai_large shows an extremely high proportion of pruned-out parameters, with both input and output channels in deeper layers reaching nearly 100%. In such cases, does the forward pass rely solely on the residual connection in these layers?\n\n5. Since overfitting is identified as a major issue, how does fine-tuning with dropout affect the performance? Have the authors conducted any comparisons or ablation studies in this regard?\n\n6. How one epoch is defined in the experiments? Is it a fixed number of batches, e.g. 100, or a full iteration of all the training samples?\n\n7. The authors use an extremely large batch size (8192) for pruning. Does batch size have a significant effect on the pruning results?\n\nSome errors:\n* Typos in the caption of Fig 2 (a)\n* The TopK operation in Algorithm 1 should select $s$ with the lowest importance. It appears that the current version selects the ones with the highest importance.", "limitations": "Yes, the authors discuss the limitations of their work. However, one important aspect they overlook is the training efficiency of the proposed method. While they show the inference efficiency, it is also crucial to understand how the pruning process impacts training efficiency.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strength\n1. The paper is well-written and easy to follow, with a clear structure and precise mathematical expressions.\n2. The proposed method is simple to understand and general to all TSFMs.\n3. Efficient fine-tuning of TSFMs is a timely and important direction to explore, with a great significance in real-world applications.\n4. The experiments are comprehensively conducted. The authors validate the proposed method on various TSFMs. \n\n### Weakness\n1. The originality of the proposed method seems not very significant. It appears to apply an established method from LLM-pruner to calculate importance of parameters and prune the insignificant ones.\n2. Some analysis could be more in-depth. Please refer to the Questions section for further details.\n\n\n[1] LLM-pruner: on the structural pruning 356 of large language models, NeurIPS 2023", "was_revised": true, "final_justification": "I have carefully reviewed the manuscript and the rebuttal. All of my concerns have been clearly and satisfactorily addressed. This paper tackles a timely and practically important problem, i.e. fine-tuning time series foundation models (TSFMs) effectively and efficiently. While the methodological novelty is not particularly groundbreaking, the application of pruning to TSFMs is exploratory and appears to be practically useful. The experimental evaluation is comprehensive and robust, especially with the inclusion of additional results on more datasets. Overall, I consider this a high-quality paper in the time series forecasting domain, and I will maintain my original scores.", "rebuttal": "**Q1.1: About the average relative output norm. Why does the output norm reduce significantly compared to the input embedding.**\n\nOur inspiration comes from Figure 5c in [1]. Due to residual connections (as formulated in our Eq. 3 and Eq. 1), the relatively small norm of head outputs does not mean significant reduction on the results, since each head output is added into the input hidden states. We define the relative norm to quantify the significance of attention heads when modifying the input hidden states. The small output norm of most attention heads implies that not all heads are necessary in forecasting a specific downstream time series. As it is well accepted that multi-head attention can model heterogenous patterns, we assume that each attention head may play distinct role in handling time series hidden states from a specific domain. \n\n> [1] Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML'23.\n\n**Q1.2 Why does large TSFM variants have better zero-shot performance even with a smaller proportion of significant heads compared to smaller variants? **\n\nThough Moirai-large has a smaller proportion of significant heads, the absolute number of significant ones may be greater than Moirai-base, since Moirai-large has 384 heads in total and Moirai-base has 144 heads in total. Increasing model size during pretraining enhances the capacity of time series foundation models to memorize patterns more effectively, which in turn improves their zero-shot forecasting performance.\n\n**Q2: Why fine-tuning Moirai and Timer fails to outperform PatchTST?**\n\nFine-tuning Moirai and Timer fails to outperform PatchTST because these foundation models already lag behind in zero-shot performance, as shown on the GIFT-Eval leaderboard. Compared to other state-of-the-art time series foundation models, Moirai and Timer exhibit a larger performance gap relative to PatchTST, making it more difficult to close this gap through fine-tuning alone.\n\n**Q3: include zero-shot performance in table.**\n\nThanks for your advice! We will include it in our future manuscript. You can also refer to Figure 11-14 in our supplementary materials for a brief comparison.\n\n**Q4: Moirai_large shows an extremely high proportion of pruned-out parameters. Does the forward pass rely solely on the residual connection in these layers?**\n\nThanks for your valuable question! When most channels in the deeper layers are pruned, these layers have minimal impact on the hidden states, meaning that the forward pass primarily relies on the residual connections to carry information forward. As a result, the final output representation is largely determined by the shallower layers, which appear sufficient for accurate prediction. As it is an interesting phenomenon that is also observed in LLMs, we may also apply early-exit techniques [2] to TSFMs, which are promising to reduce computation costs.\n\n> [2] EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism.\n\n**Q5: How does fine-tuning with dropout affect the performance?**\n\nThanks for your suggestions! We compare the average MSE and MAE of Chronos-bolt-base without and with dropout in the following table.\n\n| Methods  | Fine-Tune | (dropout=0) | Fine-Tune | (dropout=0.1) | Prune-then- | Finetune  |\n| -------- | --------- | ----------- | --------- | ------------- | ----------- | --------- |\n| Datasets | MSE       | MAE         | MSE       | MAE           | MSE         | MAE       |\n| ETTh1    | 0.420     | **0.410**   | 0.422     | 0.411         | **0.417**   | **0.410** |\n| ETTh2    | **0.339** | **0.368**   | 0.343     | 0.369         | **0.339**   | **0.368** |\n| ETTm1    | 0.345     | 0.357       | 0.352     | 0.361         | **0.344**   | **0.356** |\n| ETTm2    | 0.255     | 0.297       | 0.263     | 0.300         | **0.247**   | **0.296** |\n| Weather  | 0.261     | 0.268       | 0.268     | 0.278         | **0.215**   | **0.243** |\n| ECL      | 0.154     | 0.240       | 0.153     | 0.239         | **0.150**   | **0.237** |\n\nThe results demonstrate that dropout can result in negative effects. We assume that the dropout technique is not suitable for time series. In computer vision tasks, it is naturally required and relatively easy to conduct classification or reconstruction even with noisy pixels. By contrast, it is not suitable to always consider all latent features in time series, which may contain noisy fluctuations or outdated information.\n\n**Q6: How one epoch is defined in the experiments?**\n\nWe define one epoch as a full iteration of all the training samples.\n\n**Q7: Does batch size have a significant effect?**\n\nIntuitively, a very large batch size (e.g., approaching the full data size) can capture the comprehensive patterns of the dataset, while smaller batch sizes can increase risks of mistakenly pruning importance channels. We use the batch size to control the iteration numbers and strike a balance between pruning efficiency and effectiveness."}, {"review_id": "UO6bdDupzN", "reviewer": "Reviewer_SzHa", "summary": "The paper attempts to address an important problem in the time series forecasting domain: How to prune the pre-trained models during Full-Shot fine-tuning (referring to usage of 100% of the downstream data for fine-tuning). The paper proposes a pruning and then fine-tune approach, which essentially masks some of the neural network connections and thus create a more compact model. The paper evaluates the proposed approach in six time series forecasting datasets which are widely employed in this domain.", "questions": "1.\tUndoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.\n2.\tThe pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? Have the author(s) tried to apply varying dropout (best value chosen using a validation set) during the fine-tuning. A full comparison with dropouts should be performed for all models and datasets.\n3.\tSome of the TSFMs suggest fine-tuning only the model’s head for the downstream task (e.g., the HeadProbing in TTM). How does the pruning method perform when only applied on the head (or other layers for other TSFMs)? \n4.\tThe improvement in the MSE and MAE scores are very insignificant, often 2nd and 3rd decimal places. This questions the efficacy of the proposed method.\n5.\tOnly three unique data sources have been used (ETT, electricity, weather). More variety of datasets should be included in the analysis, e.g, the traffic dataset widely used for evaluation TSFMs.", "limitations": "No. \nLimitations of the proposed method should be discussed. See the weaknesses section for details. A summary is provided here:\n1. How does the method perform in few-shot setting compared to a model trained from scratch with few-shot data as well?\n2. Comparison with HPO-selected dropouts.\n3. Comparison when only head is pruned.\n4. Wider variety of data sources.\n5. Insignificant improvements.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe problem is pretty important in the time series domain, where, often fine-tuning cannot match the performance of a model trained from scratch on the entire data.\n2.\tRigorous experimental evidence is shown for existing SOTA models that possess the addressed problem.\n3.\tMinimal necessary ablation studies have been done.\n\nWeaknesses:\n\n1.\tUndoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.\n2.\tThe pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? Have the author(s) tried to apply varying dropout (best value chosen using a validation set) during the fine-tuning. A full comparison with dropouts should be performed for all models and datasets.\n3.\tSome of the TSFMs suggest fine-tuning only the model’s head for the downstream task (e.g., the HeadProbing in TTM). How does the pruning method perform when only applied on the head (or other layers for other TSFMs)? \n4.\tThe improvement in the MSE and MAE scores are very insignificant, often 2nd and 3rd decimal places. This questions the efficacy of the proposed method.\n5.\tOnly three unique data sources have been used (ETT, electricity, weather). More variety of datasets should be included in the analysis, e.g, the traffic dataset widely used for evaluation TSFMs.", "was_revised": true, "final_justification": "While the rebuttal has clarified many of my concerns, I still believe the evaluation is weak as I mentioned in my replies. The proposed method of pruning is model agnostic, which is good for its applicability, but the evaluation setting and reported metrics limit the understanding on how much gain is possible with the help of pruning.", "rebuttal": "**Q5: More variety of datasets should be included**\n\nThanks for your advice! We additionally perform short-term forecasting over Traffic and the M4 Yearly and Covid Death benchmark borrowed from GIFT-EVAL. The results of Chronos-bolt-base are reported in the following table. Our proposed method outperforms the fine-tuning baseline on almost all the metric. \n\n|                                          | Chronos-bolt-base   | MSE         | MAE     | MASE      | CRPS      | NRMSE     |\n| ---------------------------------------- | ------------------- | ----------- | ------- | --------- | --------- | --------- |\n| M4 Yearly (context length$\\in[19, 284]$) | Zero-shot           | 3926423     | 943     | 3.507     | 0.121     | 0.318     |\n|                                          | Fine-Tune           | 3226941     | 833     | 3.089     | 0.107     | 0.309     |\n|                                          | Prune-then-Finetune | **3125398** | **816** | **3.007** | **0.104** | **0.283** |\n| Covid Death (context length=212)         | Zero-shot           | 641888      | 164     | 38.9      | 0.047     | 0.301     |\n|                                          | Fine-Tune           | 577110      | 157     | **37.5**  | 0.046     | 0.286     |\n|                                          | Prune-then-Finetune | **239526**  | **93**  | 49.8      | **0.034** | **0.184** |\n\n**Q1: Undoubtedly, the main application of time series foundation models (TSFMs) are in the zero-shot or few-shot settings. Hence, the same experiments and analysis should be done in few-shot fine-tuning settings with different percentages of training data employed (20%, 40%, …, 100%). Then only, the efficacy of the proposed approach will be clear.**\n\nAs shown in the table above, fine-tuning significantly outperforms zero-shot forecasting, and our proposed approach delivers considerable performance gains. We believe that it is time to improve full-shot performance of TSFMs for wider applications.\n\nAs for few-shot performance (with 20% training data), the following tables show the average MSE and MAE with the horizon in {96, 192, 336, 720}. FT is short for Fine-tuning, and Prune+FT is short for our proposed Prune-then-Finetune.\n\n|          | Chronos     | -bolt | -base     |           |             | TTM   |           |           |\n| -------- | ----------- | ----- | --------- | --------- | ----------- | ----- | --------- | --------- |\n|          | **FT** |       | **Prune** | **+FT**   | **FT** |       | **Prune** | **+FT**   |\n| Datasets | MSE         | MAE   | MSE       | MAE       | MSE         | MAE   | MSE       | MAE       |\n| ETTh1    | 0.430       | 0.414 | **0.428** | **0.413** | 0.408       | 0.432 | **0.402** | **0.426** |\n| ETTh2    | 0.343       | 0.369 | **0.339** | **0.368** | 0.333       | 0.385 | 0.333     | **0.383** |\n| ETTm1    | 0.359       | 0.359 | **0.350** | **0.358** | 0.350       | 0.379 | **0.342** | **0.373** |\n| ETTm2    | 0.271       | 0.305 | **0.254** | **0.297** | 0.245       | 0.310 | **0.244** | **0.306** |\n| Weather  | 0.248       | 0.269 | **0.221** | **0.263** | 0.225       | 0.267 | **0.221** | **0.263** |\n| ECL      | 0.154       | 0.240 | **0.153** | **0.239** | 0.167       | 0.265 | **0.165** | **0.263** |\n\nAs shown in the table, our method can still outperforms fine-tuning all parameters in few-shot scenarios. \n\n**Q2: The pruning method is intuitively similar to dropouts as it essentially drops some network connections. Is that correct? A full comparison with dropouts should be performed for all models and datasets.**\n\nThanks for your question! Pruning and dropout are essentially different. Dropouts randomly corrupt hidden states during training, while the inference time involves the complete network without droping any connections. By contrast, connections are dropped permanently after pruning, and we consistently use the same subnetwork during finetuning and inference.\n\nWe compare the average MSE and MAE of Chronos-bolt-base without and with dropout in the following table.\n\n| Methods  | Fine-Tune | (dropout=0) | Fine-Tune | (dropout=0.1) |\n| -------- | --------- | ----------- | --------- | ------------- |\n| Datasets | MSE       | MAE         | MSE       | MAE           |\n| ETTh1    | **0.420** | **0.410**   | 0.422     | 0.411         |\n| ETTh2    | **0.339** | **0.368**   | 0.343     | 0.369         |\n| ETTm1    | **0.345** | **0.357**   | 0.352     | 0.361         |\n| ETTm2    | **0.255** | **0.297**   | 0.263     | 0.300         |\n| Weather  | **0.261** | **0.268**   | 0.268     | 0.278         |\n| ECL      | 0.154     | 0.240       | **0.153** | **0.239**     |\n\nThe results demonstrate that dropout can result in negative effects. We assume that the dropout technique is not suitable for time series. In computer vision tasks, it is naturally required and relatively easy to conduct classification or reconstruction even with noisy pixels. By contrast, it is not suitable to always consider all latent features in time series, which may contain noisy fluctuations or outdated information.\n\n**Q3: Some of the TSFMs suggest fine-tuning only the model’s head (e.g., the HeadProbing in TTM)**\n\nThanks for your advice! In the following tables, we compare performance of probing and full finetuning in terms of average MSE and MAE with the horizon in {96, 192, 336, 720}. The dropout hyperparameter has been tuned based on the validation set.\n\nChronos-bolt-base:\n\n| Methods  | Probing |       | Full      | Finetune  | Prune-then- | Finetune  |\n| -------- | ------- | ----- | --------- | --------- | ----------- | --------- |\n| Datasets | MSE     | MAE   | MSE       | MAE       | MSE         | MAE       |\n| ETTh1    | 0.420   | 0.411 | 0.420     | **0.410** | **0.417**   | **0.410** |\n| ETTh2    | 0.348   | 0.373 | **0.339** | **0.368** | **0.339**   | **0.368** |\n| ETTm1    | 0.356   | 0.360 | 0.345     | 0.357     | **0.344**   | **0.356** |\n| ETTm2    | 0.252   | 0.301 | 0.255     | 0.297     | **0.247**   | **0.296** |\n| Weather  | 0.226   | 0.250 | 0.261     | 0.268     | **0.215**   | **0.243** |\n| ECL      | 0.152   | 0.238 | 0.154     | 0.240     | **0.150**   | **0.237** |\n\nTTM:\n\n| Methods  | Probing |       | Full  | Finetune | Prune-then- | Finetune  |\n| -------- | ------- | ----- | ----- | -------- | ----------- | --------- |\n| Datasets | MSE     | MAE   | MSE   | MAE      | MSE         | MAE       |\n| ETTh1    | 0.403   | 0.424 | 0.402 | 0.427    | **0.399**   | **0.423** |\n| ETTh2    | 0.333   | 0.385 | 0.333 | 0.385    | 0.333       | **0.383** |\n| ETTm1    | 0.353   | 0.377 | 0.348 | 0.379    | **0.342**   | **0.373** |\n| ETTm2    | 0.252   | 0.307 | 0.247 | 0.311    | **0.245**   | **0.306** |\n| Weather  | 0.225   | 0.264 | 0.223 | 0.265    | **0.221**   | **0.262** |\n| ECL      | 0.162   | 0.261 | 0.161 | 0.258    | **0.160**   | **0.256** |\n\nThe results demonstrate that only fine-tuning the final head does not exhibit significant performance improvement. In a few cases, head probing outperforms full finetuning, while our proposed prune-then-finetune method still achieves the best performance.\n\nAs for TTM, their [official code](https://github.com/ibm-granite/granite-tsfm/blob/63c962fee413ed53789ea0d16119218d09f28d1a/notebooks/hfdemo/tinytimemixer/full_benchmarking/gift_leaderboard/ttm.ipynb) suggests finetuning full parameters to achieve higher performance on benchmarks including [GIFT-EVAL](huggingface.co/datasets/Salesforce/GiftEval/). We would like to consider head probing as an approach to training efficiency."}]

Author rebuttal:

Meta-review: This paper addresses the important and timely problem of efficiently adapting large pre-trained TSFMs to downstream forecasting tasks. The authors propose a prune-then-finetune framework, which removes redundant channels and attention heads based on a loss-based importance score, and then fine-tunes the resulting compact subnetwork. This approach is simple, model-agnostic, and applicable to encoder-only, decoder-only, and MoE TSFMs. Experiments across seven TSFMs and six benchmarks show consistent performance improvements and significant inference speed-ups.

Strengths:

1. Tackles a timely and practical problem in TSFM adaptation.

2. Method is simple, broadly applicable, and empirically effective.

3. Demonstrates both performance gains and computational efficiency.

4. Well-written with extensive experimental validation.

Weakness:
1.  Algorithmic sensitivity & hyperparameters: The pruning-then-finetune method requires gradually expanding the scope of pruned components and accumulating sufficient feedback over batches to stabilize the importance score. These design choices introduce multiple hyperparameters and can make the algorithm potentially unstable and difficult to use reliably in practical deployments.

2.  Benchmark diversity: Experiments are concentrated on ETT, Weather, and Electricity; generalizability to other datasets (e.g., traffic) remains unclear.

The work is solid, well-motivated, and demonstrates promising empirical results in adapting large TSFMs using a prune-then-finetune approach. Although questions remain regarding novelty, robustness, hyperparameter sensitivity, and generalization to broader benchmarks, the method offers a clear and valuable practical contribution to the community. Considering these factors, I recommend acceptance.
Decision: Accept (poster)