SEMPO: Lightweight Foundation Models for Time Series Forecasting
Authors: Hui He
Kun Yi
Yuanchi Ma
Qi Zhang
Zhendong Niu
Guansong Pang
Venue: NeurIPS 2025 poster ()
Keywords: time series
foundation model
forecasting
lightweight
Primary area: deep_learning
TL;DR: SEMPO is a lightweight time series foundation model with energy-aware spectral decomposition and mixture-of-prompts enabled Transformer for effective and efficient zero-shot and few-shot forecasting.

Abstract: The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resource-constrained environments. In response to this growing tension between versatility and affordability, we propose **SEMPO**, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) _energy-aware **S**p**E**ctral decomposition module_, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) _**M**ixture-of-**P**r**O**mpts enabled Transformer_, that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.

SEMPO: Lightweight Foundation Models for Time Series Forecasting
Hui Hel Kun Yi,, Yuanchi Ma', Qi Zhang', Zhendong Niu'; Guansong Pang2t
'Beijing Institute of Technology. 'Singapore Management University, State Information Center, "Tongji University
(hehui617, yma, zniu)@bit. edutt.ddu.com en, kunyi cn@gmail com [hangqi_cs$tong]] edu. cn, gspangōsmu. leteesmueedu..ge..... edu.sg
Abstract
The recent boom of large pre-trained models witnesses remarkable success in de- veloping foundation models (FMs) for time series forecasting, Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre- pre-training on large-scale datasets, which significantly hinders their deployment in resource- constrained environments. In response this growing tension between versatility and affordability, we propose SEMPO, a novel lightweight foundation model that requires pretraining relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) energy- aware SpEctral decomposition module, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that ignored in current methods; and 2) Mixture-of-PP -frio-ff-s--ss enabled Transformer, that learns heteroge- neous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt- bastd experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale bench- marks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at
Introduction
Time series forecasting constitutes a fundamental analytical tool within dynamic real-world systems, underpinning diverse array of contemporary domains, such urban computing [1], inventory optimization [2], energy demand estimation [3], and climate system modeling [4]. Forecasting has often been approached in task-specific and end-to-end fashion. A wide range of methods have been developed in this paradigm, from classic statistical models, such as Exponential Smoothing [5] and Gaussian Processes [6], modern deep learning approaches. including Transformer-bases architectures [7-11] and those based on Multilayer Perceptrons (MLPs) [12-15]. One key issue shared by these approaches is that their effectiveness often relies heavily on large in-distribution training data and extensive domain expertise.
To mitigate this issue, profound paradigm shift has been recently ushered in by the emergence of foundation models (FMs) for general-purgee purpose forecasting, which leverage large-scale pre-training
The work done when Hui He visited Singapore Management University
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prediction
Ground
Imput
Number of Parameters (Millions)
Time Mafr
Morra
Zargffew Peweenaa Aveerage
0.450
0.425
Choops
Chrondk
Mime
Chroros,
Chronos,
SEMPO
0.400
Morai
Morely GPT4TS
0.77-
0.350
SEMPO IZeoo
MEMPO
20 50
50
100
120 F00g
20
60 100
120 Prou
10t
30
10t
10*
Figure 1: Left: Most time series FMs cannot effectively utilize low-energy frequency signals, exemplified by Chronoss [20] on the ETTh1 dataset [24]. Middle: The EASD module in SEMPO helps mitigate this issue. Right: The number of parameters in SEMPO and state-of-thh-an (SOTA) time series FMs against average zero-/few-shot performance on the TSLib benchmark [25].
on massive amounts of time series data to enable zero- and fnd- few-shot generalization across broad spectrum of downstream tasks. In line with this trend, research efforts have increasingly emphasized scaling up pre- pre-training datasets and model sizes to push the frontier of transferability in diverse time series domains [16-21]. However, excessively large pre-trained models often impose substantial computational burdens during both training and inference, hindering its application to computational resource-constrained environments [22, 23]. This motivates the question: Can we inslead devise "small" FMs to serve as an ulternative to these "large" FMs for zero- and few-shol forecasting in such Aisource-constraingg environments? A few studies [22,23] have explored this question very recently. They showed remarkable success in reducing the model size while maintaining its generalization, but they still rely computationally costly pre-training on large-scale data. We take this question step further: Can we substantially reduce both model size and pre-training data size while maintaining strong generalization ability?
To answer this question, we propose SEMPO, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Two key challenges on the way to this goal include 1) significantly improved utilization of pre-training data and 2) lightweight model architecture with strong generalizability. Two novel modules- aware spectral decomposition (EASD) and ceatturr-of-promppe enabled Transformer architectures (MoPFormer) are introduced in SEMPO to respectively address each of these two challenges.
Specifically, modeling full energy spectrum of temporal signals from pre-training data is essential for learning a comprehensive knowledge base of temporal patterns to underpin strong transferability across different datasets [26-28]. However, current time series FMs [17, 18, 20, 21] often suffer from bottlenecks in utilizing these pre-training data, since their pre-training can ignore low-energy frequency signals characterized by small amplitudes and distributed across mid- high- or even low- frequency bands (see Figure Left). Despite being subtle, such signals are persistent and can encode stable temporal dynamics that are crucial for forecasting generalization. In light of this, we propose the EASD module that transforms each time series into the frequency domain and decomposes it along energy and frequency axes. To prevent low-energy components from being overwhelmed by high-energy ones, we perform adaptive spectral partitioning using learnable energy threshold conditioned on the input's 's spectral characteristics. As shown in Figure Middle, leveraging this partitioning, we apply dual-branch spectral masking on the frequency axis, where independently parameterized multi-ban band masks can selectively suppress high- and low-frequency bands in each energy branch.
To accommodate a wide range of heterogeneous input patterns from different domains, most time series FM solutions [ [16, 19, 20, 29] opt for a large Transformer architecture with vast amounts of parameters, e.g., hundreds or thousands of millions of parameters, as shown in Figure Right. Some recent works explore model specialization (i.e., model customization for specific tasks or datasets) with sparse activation at various levels-such as dataset [30], frequency [17, 18], and token [21,31] - to alleviate the computational burdens while maintaining its generalization over the heterogeneous temporal patterns. However, the architecture of their base model remains large. To address this issue, propose the MoPFormer module that learns mixture of lightweight prompt-based experts and adaptively routes different time series tokens to these experts. This enables cost-effective learning of the heterogeneous input patterns without relying large network architectures such the costly mixture of experts in Transformers [1] in current approaches (see Figure 1 Right).
In summary, this work makes the following four main contributions:

---

We propose SEMPO, novel time series FM with significantly reduced model size and pre- trainini scale, yet demonstrating superior generalization ability on diverse downstream forecasting tasks.
We reveal the bias towards high-energy frequency signals in the pre-training of current solutions and introduce the EASD module to mitigate this bias, enabling significantly improved data exploitation.
We then introduce MoPFormer, a Transformer with mixture of small learnable prompts, to learn the heterogeneous temporal patterns from diverse imput datasets, offering an effective lightweight alternative to popular Transformers with large mixture-of-eeppert networks.
Comprehensive experiments two large-scale benchmarks show that SEMPO, with only 6.5M parameters pre-traie pre-trained on 83M time points, surpasses SOTA time series FMs with hundreds of millions of parameters pre-trained on billions of time points, demonstrating an average reduction of 12% and 22% in forecasting errors under zero- and few-shot scenarios, respectively.
2 Related Work
LLM-based Time Series FMs. The emergence of Large Language Models (LLMs), empowered by in-context learning and emergent abilities, has catalyzed a growing body of research on transferring their powerful pre-trained knowledge to the time series domain. One line of research [32,33] leverages pre-trained LLMs as zero-shot learners by directly converting numerical time series into textual inputs. Another line of research [34-37], such as GPT4TS [34], Time-LLM [30], and I-IP-LLM [36], adapts LLMs to time series forecasting through prompting or fine-tuning to bridge the modality gap. Time-LLM [30] reprograms time series using text prototypes and prepends textual prompts to provide enriched contextual information. IP [36] aligns time series embeddings with the pre- pre-trained semantic space of LLMs by retrieving prompts from semantic anchors. Despite their positive impact on generalization compared to task-specific models (see Appendix A), these models rely on substantial computational resources.
Pre-trained FMs on Time Series. Foundation models pre-trained from scratch on large-scale, multi-domain time series data have recently attracted widespread attention 216-21,29,311 Building on this paradigm, Moment [16] and Moirai [18] adopt encoder-only architectures with masked pre- pre-training objectives, focusing on leveraging self-supervised learning to capture intricate temporal dependencies. TimesFM [17] and Timer [19] employ decoder-only architectures under a GPT-style causal modeling paradigm for autoregressive forecasting. Chronos [20] adopts an encoder-decoder architecture that tokenizes numerical time series via scaling and quantization into fixed vocabulary. Most of these models [16, 2,220,29 rely on large Transformer architectures with numerous parameters to memorize the heterogencous input patterns across diverse domains, which can increase both the learning complexity and the costs associated with training and inference.
Lightweight Time Series FMs. Recently, sparse mixture of experts (MoE) has been introduced into Transformers [2],31] to enhance generalization across the heterogeneous temporal patterns by routing diverse time series tokens to specialized experts. Despite simultaneously alleviating the computational budget with sparse activation, the architecture of their base model remains large. With the emergence of small language models [38, 39]. there is also growing interest in small time series FMs [22,23], addressing practical computational resource and cost constraints. For example, TTM [23] leverages lightweight mixer-style architecture and pre-trains family of compact models for general forecasting. Chronos [20] and Moirai [18] also provide small variants to tackle such real-world settings. However, they still require substantial pre- mreitraining large-scale time series data, significantly hindering their applications in resource- coo.urcce-ottriineddd environments.
Methodology
3.1 Problem Statement
Given a time series input X = [X]., Xi. X E N with the number of vari- ates N and the lookback window length L, where Xi, denotes the historical observations of the i-th variate, the task of time series forecasting is to predict the future H time steps Y = [X]+14+ X X.N 1 E NxH This work focuses on developing generalist time series forecaster pre-trained on collection of J datascts from diverse domains,
3

---

Turain = {Dirain) train; D Diyain .... train where each traip = =X², Y represents the j-th dataset. For downstream evaluation, we consider two settings, including zero- and few-shot forecasting. In the few-shot setting, the pre-trained model is further adapted using a small target dataset Dtune. where Drune < Ttrain. and evaluated on unseen Deest from the same domain. In the zero-shot setting, the model is tested directly on Dest without any additional adaptation. Following standard zero- and sew-shot settings, Drune and Drest are drawn from entirely new domains that are not presented in Tuainn To support any-variate forecasting, we apply channel independence [10] to decompose multivariate inputs into univariate series.
Praiin.....
3.2 Overview of SEMPO
As illustrated in Figure 2, SEMPO is built upon encoder-decoder architecture, which comprises four key components: energy-aware spectral decomposition (EASD), patchify and project, mixture-of- prompts enabled Transformer (MoPFormer), and reconstruction and prediction heads. EASD include two steps: energy-wise spectral partitioning and frequency-wwis spectral masking, while MoPFormer includes a mixture of prompts (MoP) and MoP block. The training of SEMPO follows two-stage paradigm, as outlined in Section 3.5. Below we introduce the EASD and MoPFormer components in detail.
3.3 EASD: awergy-aware Spectral Decomposition
Rooting in the weighting characteristics of self-attention, the Transformer backbones widely used in time series FMs [17,18, 20,2] often exhibit energy bias [40], being prone to frequency components with high energy while overlooking those with low energy. Existing forecasting methods [22,26, 27] primarily focus on the bias between high- and low-frequeney components along the frequency axis. To address this issue, we introduce EASD to improve spectral modeling by mitigating energy bias and enhancing the subtle yet persistent temporal patterns. It achieves this by sequentially performing energy-wise spectral partitioning and frequency-wist spectral masking as follows.
Energy-wise Spectral Partitioning. Given a univariate time series X RR,, after instance normal- ization [.1], the normalized input is transformed into the frequency domain by applying Fast Fourier Transform (FFT) along the temporal dimension. The spectral energy at each frequency is defined as the squared magnitude of the corresponding Fourier coefficient [40], i.e., Energylfi where Z denotes the complex spectrum. We then perform spectral partitioning in the energy axis by adjusting a learnable energy threshold r according to the input's 's spectral characteristies to delineate high- and low-energy components. This separation ensures low-energy components are not over- whelmed by high-energy components, thus enabling the extraction of frequency-localize patterns for modeling complex temporal dynamics. The process can be formulated as:
[2]
Z (No(No))( ZHee Z (Energy (Z) > T): ZLec = Z ZHee Z-Zeee
where represents element-wise multiplication along the frequency dimension, (Energy >) yields a binary mask that identifies frequency components to be retained in ZHoe if their energy exceeds the threshold T, while the remaining low-energy components are assigned Zt
Frequency-vee wise Spectral Masking. Existing FMs [16, 17,42] primarily leverage random patch masking the temporal domain to expose models varying context lengths, enabling the learning of multi-seen scale temporal dependencies and improving generalization. While this strategy enhances contextual diversity, it fails to explicitly model the structured and periodic nature inherent in time series. Inspired by the frequency decomposition learning [22], we propose a frequency-axis spectral masking module. In both the high- and low-energy branches, high- and low-frequency bands are selectively suppressed through independently parameterized multi-band masks. Concretely, for each energy branch, we generate a set of frequency band masks (M, M2, ..., .N ) by sampling a frequency threshold 6; and a direction indicator d, for each mask M,. The threshold 6, defines the cutoff frequency, and the binary variable d, determines the masking direction: masking frequencies
less than 8, if d; = 1, and greater than 8, if d, 0. The corresponding mask M, E (0, 1/2+1 is constructed as:
8, Uniform (0. a), d, Bernoullipp). i= 1.,, Nm. M, = 1 1/, (f, S 6.) + (1 d:) (fr 2 6,)l. j j= 1,2, L/2 +
(2)

---

Prompt Experts
Adaptive Router
Self-attention
Weight
Softmax
Router
Pred Head
Head
Scale
MLP & Reshape
Softmax & Linear
Recon Decoder
Copy
Pred Decoder
K
mask
Add & iFFT
MoP
Transformer
Encoder
Myse
Zione
MicC
Zoeo
Zuec
Patchify Projection
Zytac
Mal WWW
d₂
Mn mmma
5
d
d2
62
Frequency-wise Spectral Masking
Frea
Threshold
Energy wise Spectral Partitioning Xu
O bg
IThreshold
Leee Threshold & hhocholld 8o ZHoc
Input Time Series
X1.1
FFT
Copy
MoPFormer
Energy pretraining
MoP tuning
Weight copying
Figure 2: SEMPO follows an encoder-decodes architecture. Given a univariate time series, ① its EASD module transforms it into the frequency domain and partitions into high- and low-energy components, @ These components are independently masked across multiple frequencies, fused, and transformed back the time domain. After patching and projection, the masked representation fed to backbone layers and a reconstruction head as part of the energy-aware pre-training. During MoP tuning, all previous modules are frozen; only the MoP and the prediction head are trainable. The masked input is then 3 routed to token-dependent prompts via an adaptive router and @ concatenated into the key and value of self-attention, enabling compt-guided reconstruction and prediction.
where a < L/2 1, and fy denotes the j-th frequency point. Stacking all Nm such masks for each energy branch yields two multi-band masking matrices: MHese MLee {0, 1] xLL2++1), Each row represents distinct frequency band mask, and each element maj E {0, 1} indicates whether the j-th frequency point is masked under the i-th mask. Correspondingly, we expand ZHoe and Ziec along new dimension and replicate them Nr, times, resulting in ZHec Zeec C... (1./2+1) that are aligned with the masking matrices. We then apply element-wise multiplication between each replicated representation and its corresponding mask, and convert the aggregated result back to the time domain using the inverse Fast Fourier Transform (iFFT):
Maskk iFFT C MHec + Zt.0c O Miac).
This masking strategy incorporates two core insights. First, MHес and MLec are separately generated without shared sampling parameters between the high- and low-energy branches. Such a decoupled design promotes spectral diversity and inherently mitigates energy bias. Second, by adaptively learning the sampling parameters, the masking process can be tailored to each specific time series dataset, thereby enhancing the model's ability generalize across diverse data scenarios.
Then standard patchify and projection step applied. As illustrated in Figure 2, after obtaining the masked series Xmak NR.... L, we divide it into P non-overlapping patches of equal length L.p. where P P= [./Lpp Each patch is then projected into a patch token by linear layer R4, D2 where Dp the dimension of Transformers. The positional embedding for the p-th patch xp denoted by pc,, Accordingly, the position-encos encoded patch token b, is given by b, = Dropout(xx + pe,,) and B (b), b2,... br} R
3.4 MoPFormer: Mixture-of-prompts Enabled Transformer
A significant challenge in pre-training multi-domain time series datasets lies in their high degree of cross-domain heterogeneity, including differences sampling resolution, noise levels, variable semantics, etc. To overcome this limitation, we introduce Mixture of Prompts (MoP) and incorporate
5

---

it into stacked Transformer blocks, which learns dataset-specifie prompts to guide the model's adaptation to new data, thereby promoting generalization across diverse target domains.
To account for the diversity of multi-domain time series data, we randomly initialize prompt-based expert pool E {el, e2, .... er} 1xxx; where each prompt-based expert is E IR with dimension Dp. Applying all prompt- based experts indiscriminately to each patch token may dilute the effectiveness of prompt information, particularly under significant cross- domain distributional shifts in time series data. Inspired by the conditional computation mechanism MoE [21,31], we introduce a woken-dependent adaptive router that dynamically assigns relevant prompt-based experts conditioned on the input token. Concretely, the router computes expert-specific gating scores a linear-softmax transformation over the input token, and performs soft merging of expert vectors through weighted combination guided by these scores. To encode a shared structure between prompt key and value matrices [43], we further adopt reparameterization strategy consisting of two-layer MIP followed by reshape operation, transforming the aggregated expert representations into structured key-value pairs. Given the masked patch token b, E R2, and expert e, E RD for i = 1,2,..., .......,,, I, the complete routing and merging process can be formalized as:
Mixture of Prompts.
Si. = (G(tmax((incar(bbp))
ē = RRsshapoeMMPPP
Σ
1=1 Si.p c;)):
where Si.p E RR denotes the gating scores over the expert pool, indicating the contribution of the i-th expert c; to the p-th input token by- The mixed prompt op E Rx22xDp represents a structured prompt valuv pairs of by distributed across S Transformer layers.
MoPFormer Block. Since self-attention (SA) can interpreted as a specialized MoE [44], the mixture of prompts is integrated into the SA mechanism to inject new experts, which can be achieved via the modification of key and value matrices. Specifically, the mixed prompt Emix =êe, @é1,ē,,... .., ēp} E R N,. Px Dy is first decomposed into E E R xx.xPPxx, and El E RN In the SA layer, let Q, K, V denote the query, key, and value matrices respectively, the key and value matrices are then constructed by appending the prompt E and Fy to the input series B E RN xPxDp along the patch dimension P as follows:
SA Attention(Q = B, K = Concat((F,, mix: B), V Concat(E Fl B)).
(5)
We employ Transformer encoder [45] in both the encoder and decoder. Following SOTA time series
FMs [18, 21, we integrate recent advancements including RMSNorm [46], pre-normalization
scheme [47], and SwiGLU [48] into our architecture. Each block at the s-th layer is defined as:
U SA(RMSNNrm
B²-1, Ü RMSNorm(U*)) 8 FFN(U*) + U (6)
Both the encoder and decoder are constructed by stacking S MoPFormer blocks. SEMPO is then pre-trained using the above steps via reconstruction and prediction objectives. More details the MoPFormer block during pre-training and inference are provided Appendix B.
3.5
Model Training
To improve the model's generalization ability in downstream zero-shot and few-shot forecasting tasks, we adopt two-stage training paradigm: energy energy-aware pre-training and MoP tuning, as discussed in the following:
Energy-aware Pre-training. During energy-aware pre-training, SEMPO is pre-trained on multi- domain time series data via self-supervised reconstruction objective, enabling it to capture transfer- able cross-domain commonalities. As shown in Figure 2 (black arrows), the series representation fed into the reconstruction decoder without incorporating the mixture of prompts. The decoder generates the reconstructed series X R through linear reconstruction head. The training objective is the Mean Squared Error (MSE) between the input and its reconstruction:
Epretraining = -1|X1:L 21.1..

---

Table 1: Zero-shot results on the TSLib benchmark. MSE and MAE are averaged errors over forecasting horizons H E {96, 192, 336, 720). '-' denotes dataset used in pre-training and excluded. Values in (.) denote model size and pre-training data size respectively. 'S' Small, 'B": Base, ' Large. Red: the best, Blue: the second best. Full table is in Appendix D.
SEMPO Time-MoE, Time-MuE,
Mairai
Muiraige
Muiraiz Chremoss Chromass er Chrunew,, TimesFM
Mument
(6.5ML83M) (113M,309)) (453M_309B) (67.4M228B) (148,227) (912,278) (311M2227)) (46M.84B3 (20088,843) (71080.848) (200M) (AS5ML11333
Metries
MSR MAR MSE MSR MSE MAR MSR MAF MAE MSE MAF MSE MAR MSE MAE MSE MAR MAR
MAR
ETTbi
[0.410
0,430 0,45, 0.449 0445 0,449 0,451 0,463 0.48 0.432 0.33 6.431 0,166 0,443 0.551 0.163 0,524 0.429 0.541 0.443 0,489 0.466
0,708 0,580
ETTh2
[0.34
0,391 0,566 0.479 0.477 0,452 0,366 0:408 0,355 0:401 O 360 O. 0.382 0,397 0.394 O 0,409 0,392 0.401 0,385 0:400 0.396 0.405 0,392 0.430
ETTml
[0.503
0.406 0.507 0.480 0.483 0.471 0.544 0.476 0.554 0.477 0.566 0.464 0.601 0,468 0.625 0.487 0,506 0,405 0,521 0.448 0.434 0.419
0,697
0.555
FTTm2
[0.256
0.343 0.538 0.463 0.509 0,452 0.298 0.346 0.323
0.356 0.334 0.352 0.320 355 O.R 0,344 0,315 0.350 0,330
0:353
0,319 0.360
[0.248
0.287 0.279
0.318
0.334 0.292 0,312 0.267 0.306
0.295
0.289 0.295 3.2 0,283 0.295 0.292 0:297
0.291
0.323
Bleeteeit
(0.1%
0.295
0.297 0.575 0.245 0.329
0.246 0.234 0.309 0.246 0,336 0.329 0.326
0.328
0,861 0.766
Traticc
[0.466
0,344
0,613 0.407
0.614
0.600
0.413 0,600 0,411
1.411 0.804
1d Cound]
13
O
0
MoP Tuning. During MoP tuning, SEMPO is jointly trained with supervised forecasting and self-supervised reconstruction to adapt to domain-specific variations by tuning both the mixture of prompts and the prediction heads, while keeping the Transformer backbone frozen, as shown in Figure (red arrows). We adopt a multi-resolution forecasting strategy [21] with each prediction head implemented as a single linear layer. A composite loss, aggregating forecasting errors over different horizons, is computed to enhance model generalization. Let H = { H1 H2, ...... Ha} denote the set of forecasting horizons, where H, is the number of future steps for the r-th resolution. Given the ground truth X H, for 1,2, ..., R. the training loss is then defined as:
Lruning
Σ ||X -XLI1LtH 112 +1| X1.211B
(8)
H,EH
4 Experiments
Datasets.
Leveraging large-scale publicly available time series collection UTSD [19], we curate a diverse subset covering multiple domains, totaling ~083 million time points. We then set the pre-training trainin--valiattion validation split to 9:1, following [19]. For zero-/few-shot forecasting. evaluate SEMPO on the Lime-Series-Library (TSLib) benchmark [25]. which includes seven datasets: ETTHI, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic. Given that most real-world scenarios involve limited data, we draw on prior works [16, 17,20, 22] and set the lookback window length L, = 512 on TSL.b. We also include GIFT-Eval [49], the largest benchmark for zero-shot forecasting [50]. To avoid data leakage, we remove datasets overlapping with pre-training corpus and TSLib, and select datasets from the GIFT-Eval benchmark spanning diverse domains. For GIFT-Eval, we adhere to its default evaluation protocol [49]. More details are given in Appendix C.I.
Baselines. We evaluate SEMPO against 17 latest open-sourced SOTA forecasting models, grouped in three categorized: (1) Pre-trained FMS on time series: TTM [23]. Time-MoE [21], Timer [19], Moirai [18], Chronos [20]. TimesFM [17]. and Moment [16]; (2) LLM-based time series FMs:
Time-LLM [30], GPT4TS [34], and I'IP-LLM [36]; (3) Task-specific models: iTransformer [1]], DLinear [12], PatchTST [10], TimesNet [25], Stationary [9], FEDformer [8], and Autoformer [7].
Among these, the TTM family and small variants of Moirai and Chronos are lightweight pre- trained
FMs on time series. More baseline details are provided in Appendix C.2.
Implementation Details. Using 83M pre-training datasets, the entire two-stage pre-training process takes 10 hours on 4 A6000-48G GPUs with BF32 precision and a batch size of 2,048. By default, we set layer_ number 5 = 6, head number=16, latent, dimension D2 = 256, patch size Lp = 64, prompt_ number I = 128, and mask number NM 4. The hyperparameter analysis of I, NNA. and D,, as well as the scaling analysis of the model are given in Appendix D. Regarding optimization, we use AdamW W optimizer with hyperparameters: caaring_rate=-le 32 = 0.95. A constant learning rate used after a linear warmup for the first 10,000 steps. More implementation details are provided in Appendix C.3.
weight_ ddecttddd__11,1 B1 = 0.9.
7

---

Table 2: Few-shot results the TSLib benchmark with 5% training data. MSE and MAE averaged over forecasting horizons E {96, 192, 336, 720), where lower values indicate better prediction. Red: the best, Blue: the second best. Full table is in Appendix D.
Models
SEMPO
TTM
Time-LLM GPT4TS 8'I--LIM iTrandormer DLincar
PatehTST
TimesNet Stationary FEDlormer Autoformer
Metries
IMSE
MAR
MAE
MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MSE MAE
[0.406 0,382
0,710 0.750 0.694 0,569 0.925 0.647 0.943 0,646 0.658 0.562 0.722 0,508
ETTh2
[0.320
0.372 0.333 0.376 0,352 0,438 0,400 0:433 0.380
0,415 0.888 0.475 0.684 0.577 0,827 0,615 0,439 0,448 0.470 0.439 0:463 0:454 0.441 0.457
ETTal
[0.363
0,385 0,389 0,389 0.425 D:434 0,472 D,400
0.416
0.421 0.784 0.597 0,400
0,417 0,525
0,476 0,737 01561 0,857 0.598 0.730 0,592 0.796 0,620
6TTm2
[0.256
0.315 0,265 0.328 0,274 0,323 0,308 0,346 0.279 0.325 0.356 0.388 0.399 0:426 0,314 0,352 0,344 0,372 0,341 0.372 0.381 0,404 0.388 0,433
Weather
|0.230
0.268
0.256 0.273 0,200 0,509 0,263 0,301 0.257 0.295 0.309 0.339 0.263 0,508 0,269 0,303 0,298 0,318 0.327 0.328 0.309 0.355 0.310 0.353
Electricity
(0.165
Traflic
[0.410 0,427
0,303
0/423 0.298 0.434
0.208
0.324
0.450
0.317
0,418 0.29% 0,867 01403 1.526 0.676 0:425 0:533 0,502
1s Count
4.1 Zero-shot Forecasting
Setup. In this section, evaluate the performance of SEMPO for zero-shot forecasting on two representative benchmarks: TSLib [25] and GvFl-Eval [49], using MAE (Mean Absolute Error) and MSE for TSLib, and SMAPE (Symmetric Mean Absolute Percentage Error) and NRMSE (Normalized Root Mean Squared Error) for GIFT-Eval.
Results. Table reports the average zero-shot forecasting results across multiple horizons H E (96, 192, 336, 720) for Transformer-based FMs. Ad- ditional results comparing the MLP-baaed FMs TTM and different lookback window lengths are provided in Appendix D. Despite containing only 6.5M param- eters and using only 83M pre-training data, SEMPO is consistently the best performer on the majority of datasets, yielding average reductions of 23.1% in MSE and 10.8% in MAE across all methods. In par- ticular, compared lightweight FMs with similar pa- rameter scale, e.g., Moirais (14M, 27B), SEMPO ex- hibits markedly superior performance. Remarkably, even when benchmarked against large-scale time series FMs such as Chronos,, (710M, 84B), Time- MoE, (453M, 309B) and Moment (385M, I.13B). SEMPO achieves substantial MSE improvements of 19.9%, 19.2%, and 36.0%, respectively; these improvements
SEMPO Chronos,
Chronase
1.6
Chronos:
SMAPE
1.2
0.4
0.0
dednee
loop
tax/
solar
Lit Hpp
bb I2c
bbb sve
Jena
Figure 3: Zero-shot results on the GIFT-Eval benchmark. Full details in Appendix D.
more pronounced when comparing the pre-training data scale: 83M of time points in SEMPO vs. 1.13B to 309B in these large FMs. Besides, Figure presents a comparison between SEMPO and Chronos family Gval-Eval. SEMPO achieves the best performance in most cases (6 out of 9). This promising result also demonstrates that SEMPO learns a powerful knowledge base, which serves solid foundation for its remarkable capabilities in knowledge transfer.
4.2 Few-shot Forecasting
Setup. We fine-tune the pre-trained SEMPO and baseline models on the training splits of seven datasets from the TSL.b benchmark. Following the few-shot protocols in [23,30,34] we consider two training scenarios using 5% and 10% of the training data, respectively. both cases, only the prediction head and the MoP module are fine-tuned.
Results. Table 2 presents the average results under the 5% few-shot setting, with the corresponding 10% results provided in Appendix D. SEMPO establishes its superiority by outperforming 11 recent deep models across both 5% and 10% scenarios, achieving an average MSE reduction of 32.3%. In particular, SEMPO significantly outperforms the LLMM-bsed pre-trained models Time-LLM (1 11.4%)) GPT4TS (15.9%), and S2IP-LLMM (10.7%), and surpasses the lightweight MI.P-based FM TTM by 4.6%. These results underscore the central contribution of SEMPO's pre-trained initialization.
8

---

Table 3: Ablation studies. Zero-shot MSE and MAE averaged over H {96, 192, 336, 720) on the TSLib benchmark, evaluated with different model components. Full table is in Appendix D.
ETThI
ETTH2
ETTml
ETTm2
Weather
Electricity
Traffic
Design
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
SEMPO
0.410 0.341
0.503
0.286
0.248
0.287
0.295
0.344
Multi bund Masking
0.462 0.461
0.423
0.562
0.342
0.376
0.261
0.308
0.204
0.306
0.356
A.2 Random Patch Masking
0.446 0.460
0:400
0.428
0.574
0.498
0.340
0.381
0.261
0,313
0.243
0.345
0.647
0.404
B.I Sparse MoE (3 experts, activated
| 0.441 0.452
0.358
0,402
0.515
0.485
0.308
0.360
0.253
0.292
0.223
0.321
0.532
0.391
B.2 Prefix Tuning
0.430
0.448 0.359
0.404 0.513
0:475
0.309
0.361
0.268
0,306
0.217 0.315
0.494 0.365
Prodition
Ground Toth
Prodiciion
Ground
Prodetion
Gound Toch
Free
40
60
100
120/
123 Freq.
(a) Chronoss
(b) Moiraiz,
(c) SEMPO
Figure 4: Spectrum visualizations of predictions from Chronoss, Moirai,,, and SEMPO on Electricity.
which equips the model with strong inductive biases and significantly enhances its effectiveness in resource-constrained scenarios. Compared to the zero-shot results in Table 1, fine-tuning yields only marginal gains on ETThl and ETTh2, suggesting that adaptation is less effective when the target dataset is limited in scale or variability.
4.3 Model Analysis
Ablation Study. To verify the design of SEMPO, we perform a series of ablation studies under
the zero-shot setting, each targeting key model module. As shown in Table 3, replacing the default dual-branch spectral masking leads to consistent performance degradation: multi-band masking [22] (A.1) and random patch masking [10] (A.2) increase the average MSE from 0.350 to 0.399 and 0.416, respectively. The degradation in A.2 stems from its unstructured nature, which disrupts temporal dependencies and corrupts spectral coherence. While A.I retains frequency-wiss modeling, the lack of energy-based partitioning allows the dominant high-energy components to eclipse subtle yet informative low-energy signals, resulting in large information loss. Furthermore, we evaluate the impact of replacing the MoP module with two variants. The first (B.1) substitutes MoP with a sparse MoE [21] with three experts and one activated, totaling 8.5M parameters. Despite the larger model size, it underperforms SEMPO, highlighting the superior efficiency and expressiveness of the MoP design. In the second variant (B.2), we remove the adaptive router in MoP and replace it with conventional prefix tuning [51], which leads to further degradation and underscores the importance of data-dependent adaptive router in enhancing generalization across diverse time series patterns.
Analysis of Low-energy Components. We further investigate SEMPO's capability to model low- energy components in temporal signals. Figure 4 shows the zero-shot prediction results in the spectral domain for the Electricity dataset with prediction length H 336. As shown in Figure 4, Chronos and Moiraiz-thh small and large variants of the Chronos and Moirai families, respectively-tent tend to focus on dominant high-energy components. Chronos largely neglects subtle yet persistent low-energy components. Moirai,, exhibits negliggiie responses in low-energy regions, yet these do not consistently align with the ground-truth spectral peaks. This indicates that while Moirai,, can partially capture low-energy signals, it still predominantly attends to high-energy components, resulting less accurate learning of low-energy features such as important positional information. In contrast, SEMPO consistently attends to signals across the full energy spectrum, more effectively preventing low-energy components from being overwhelmed than both compact and large FMs. Besides, as shown in Appendix D, Figure 8, predictions ETThl dataset across various prediction

---

lengths demonstrate SEMPO's strong ability to identify and leverage low-energy regions, regardless of whether they reside high- mid-, or even low- frequuenncy bands of the spectrum.
Visualization of MoP. To analyze MoP's ability to learn dataset-specific knowledge, visualize the soft gating scores across different datasets in Figure 5.
Tokens
Within each dataset, different patch tokens attend to different subsets of prompt-based experts, with each ex-
ETTmZ
Tokens
pert specializing in distinct knowledge, indicating that MoP enables fine-grained, token-level composition of prompt information. When comparing across datasets, same-domain cases (e.g., ETTh] and ETTm2) exhibit similar prompt routing patterns, while cross-domain datasets (e.g., Traffic and Weather) reveal distinct token- to-prompt mappings, reflecting nomain--spcciff adap- tation. The heterogeneous routing patterns indicate that the model dynamically adapts its representations to the specific characteristics of each dataset, supporting SEMPO's strong generalization and transferability.
Weather
Tokens
Traffic
Tokens
20
40
600
00
100
120
Experts
Figure 5: Gating scores of various experts.
Efficiency Analysis. To highlight the perfor- mance and efficiency benefits of SEMPO, com- pare its inference costs with other FMs and assess ETThl. The zero-shot results are shown in Figure 6. SEMPO, a lightweight gen- eral model with only 6.5M parameters and fast
SEMPO
Chrones
Zero shot MSE on ETTh1
Moiral;
their performance
0.70
Morral,
(986c)
0.55
Chronos:
Chroncte
Time-Mofa Time-Mo
inference time of 22s, significantly outperforms all other FMs. Compared Moiraig, the smallest model among them, SEMPO achieves nearly 10 times faster inference speed while delivering supe- rior prediction performance. In contrast to large- scale FMs, SEMPO precisely fulfills the urgent need for general models in resource-constrained real-world scemarios, where both exceptional effi- ciency and superior accuracy are indispensable. More efficiency results are in Appendix D.
311M
119/3st
(14185g)
710M
(2054)
14M
(2556)
(4134)
113M
(4946)
(18/9g
200M
6.5M
91M
453M
1225
(518g)
22
205
414
695
987
1880
1973
5575
16185
Inference
(Seconds)
Figure 6: Efficiency comparison on ETThl.
5 Conclusion
In this paper, we propose SEMPO, novel lightweight foundation model for general time series forecasting, with relatively small data scale and model size while exhibiting strong generalization capabilities. SEMPO leverages novel modules: energy-aware spectral decomposition for signifi- cantly improved data exploitation during pre-training, and centurreof-promppe enabled Transformer for parameter-efficien model adaptation across different dataset and domains. Extensive experiments on two widely-used benchmarks spanning 16 datasets highlight the superiority of SEMPO in term of both effectiveness and efficiency for zero- and few-shot forecasting tasks. This work paves the way for future ad advancements in considering multivariate interactions and flexible distribution forecasting, which will empower the model structure with more time series capabilities.
Acknowledgments
This research is supported by the National Natural Science Foundation China (62272048). A*STAR under its MTC YIRG Grant (M2448c0103), the Ministry of Education, Singapore under its Tier- Academic Research Fund 14-SIS-SSMU-OE and the Lee Kong Chian Fellowship.
References
[1] Hui He, Qi Zhang, Simeng Bai, Kun Yi, and Zhendong Niu. CATN: cross attentive tree-aware network for multivariate time series forecasting. In AAAL, pages 4030-4038. AAAI Press, 2022.
10

---

1,21
[2] Massil Hihat, Stéphane Gaiffas, Guillaume Garrigos, and Simon Bussy. Online inventory problems: Beyond the i..d. setting with online convex optimization. In NeurIPS, 2023.
[3] Kun Yi, Qi Zhang. Hui He, Kaize Shi, Liang Hu, Ning An, and Zhendong Niu. Deep coupling network for multivariate time series forecasting. ACM Trans. Inf. Syst., 12:5):12:11-127:22 2024.
[4] Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nat. Mac. Intell., 5(6):602 611, 2023.
[5] Ed. McKenzie. General exponential smoothing and the equivalent arma process. Journal of Forecasting, 3(3):333-344, 1984.
[6] David Salinas, Michael Bohlke-Schneide Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. In NeurIPS, pages 6824-6834, 2019.
[7] Haixu Wu, Jiehui Xu, Jianmin Wang. and Mingsheng Long. Autoformer: Decomposition transformers with correlattn for long-term series forecasting. In NeurlPS, pages 22419-
22430, 2021. 1, 7, 21, 24
[8] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 27268-27286. PMLR, 2022. 21, 24
[9] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity time series forecasting. In NewrIPS, 2022. 1, 7, 21, 24
[10] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR, 2023. 1, 4, 7, 9, 21, 24
[1I] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR, 2024. 1, 7, 21, 24
[12] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, pages 11121-11128 AAAI Press, 2023. 1, 7, 21, 24
[13] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In KDD, pages ACM, 2023. 1, 21
[14] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. crqquency-domale mips are more effective learners in time series forecasting. In NeurlPS, 2023. 1,
[15] Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo. Lintao Ma, James Y. Zhang, and Jun Zhou Timemixer: Decomposable multiscale mixing for time series forecasting. In ICLR, 2024. 2,
[16] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MOMENT: family of open time-series foundation models. In ICML, 2024. 2, 3, 4, 7, 23, 24
[17] Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In ICML, 2024. 2, 3, 4, 7, 23, 24
[18] Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In ICML, 2024. 2, 3, 4, 6, 7, 21, 22, 23,24 24
[19] Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In JCML, 2024. 2, 3, 22, 23, 24, 25
[20] Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Türkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Araddo. Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, An- drew Gordon Wilson, Michael Bohike-Schneidder, and Yuyang Wang. Chronos: Learning the language of time series. CoRR, abs/2403.07815. 2024. 2, 4, 23, 24

---

[21] Xiaoming Shi, Shiyu Wang. Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Time- moe: Billion-scale time series foundation models with mixture of experts. In The Thirteenth International Conference on Learning Representations, 2025. 2, 3, 4, 6, 7, 9, 21, 22, 23, 24, 25,
31
[22] Yihang Wang. Yuying Qiu, Peng Chen, Kai Zhao, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, and Chenjuan Guo. ROSE: register assisted general time series forecasting with decomposed frequency learning. CoRR, abs/2405.17478, 2024. 2, 4, 9,
[23] Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wesley M. Gifford, Chandra Reddy, and Jayant Kalagnanam. Tiny time mixers (tims): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series. In NeurlPS, 2024. 2,
3, 7, 8, 23, 24, 25
[24] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In
AAAI, pages 11106-11115 AAAI Press, 2021. 2
[25] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR, 2023. 2, 7, 2,,, 22,
24, 28, 32, 33, 34
[26] Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, and Yasushi Sakurai. Fred- former: Frequency debiased transformer for time series forecasting. In KDD, pages 2400-2410.
ACM, 2024. 2,4
[27] Kun Yi, Jingru Fei, Qi Zhang, Hui He, Shufeng Hao, Defu Lian, and Wei Fan. Filternet: Harnessing frequency filters for time series forecasting. In NeurIPS, 2024. 2, 4, 28
[28] Jingru Fei, Kun Yi, Wei Fan, Qi Zhang, and Zhendong Niu. Amplifier: Bringing attention to neglected low-energy components in time series forecasting. AAAI, pages 1645-116633.
AAAI Press, 2025. 2
[29]
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilame Towards foundation models for time series forecasting. CoRR, abs/2310.0833 0.08278, 2023. 2, 3
[30] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-Ilm: Time series forecasting
by reprogramming large language models. In ICLR, 2024. 2, 3, 7, 8, 23, 24
[31] Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann. Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. CoRR, abs/2410. 10469, 2024. 2, 6, 21
[32] Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. JEEE Trans. Knowl. Data Eng., 36 36(11):6851-686 2024. 3
[33] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models zero-shot time series forecasters. In NewrIPS, 2023. 3
[34] Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In NeurIPS, 2023. 3, 7, 8, 23, 24
[35] Defu Cao, Furong Jia, Sercan Ö. Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. TEMPO: prompt-based generative pre-trained transformer for time series forecasting, In ICLR, 2024.
[36] Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. S2IP-LLM: semantic space informed prompt learning with LLM for time series forecasting. In ICML, 2024. 3, 7, 23, 24
[37] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes:
Autoregressive time series forecasters via large language models. In NeurIPS, 2024. 3
[38] Zhengqing Yuan. Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. CoRR, abs/2312. 2023. 3
[39] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACI-HIIT, pages 2352. Association for Computational Linguistics, 2021. 3
12

---

[40] Guoqi Yu, Yaoming Li, Juncheng Wang. Xiaoyu Guo, Angelica I. Avilés-Rivero, Tong Yang. and Shujun Wang. Refocus: Reinforcing mid-frequency and key-frequency modeling for multivariate time series forecasting. CoRR, abs/2502. 1602200..6890, 2025. 4, 21
[41] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang- Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICIR, 2022. 4,
[42] Qichao Shentu, Beibu Li, Kai Zhao, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, and Chen- juan Guo. Towards a general time series anomaly detector with adaptive bottlenecks and dual adversarial decoders. In The Thirteenth International Conference on Learning Representations,
2025.
[43] Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth
International Conference Learning Representations, 2025. 6
[44] Minh Le, An Nguyen The, Huy Nguyen, Trang Nguyen. Trang Pham, Linh Ngo Van, and Nhat
Ho. Mixture of experts meets prompt-based continual learning. In Neur/PS, 2024. 6, 21
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008,
2017. 6, 21
[46] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurlPS, pages 13360-1237,, 2019. 6, 21
[47] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 10524-10533. PMLR, 2020. 6, 21
[48] Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. 6, 21
[49] Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. CoRR, abs/2410. 10393, 2024. 7, 8, 22, 23,29
[50] Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Sundial: A family of highly capable time series foundation models.
CoRR. abs/2502.000 00000000086,, 16, 2025. 7
[51] Xiang Lisa Li and Percy Liang. Prefix-tuning:
Optimizing continuous prompts for generation. In ACL/JCNNPP (1). pages 4582-4597. Association for Computational Linguistics, 2021. 9, 21
[52] Bryan Lim and Stefan Zohren. Time series forecasting with deep learning: A survey. CoRR, abs/2004. 13408, 2020. 21
[53) Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR, pages 95-104. ACM, 2018. 21
[54] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilis- tie forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181-1191 2020. 21
[55] Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NeurlPS, pages 4838 4847, 2019. 21
[56) Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In NeurlPS, 2022. 21,21 22
[57] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In NeurlPS Datasets and Benchmarks, 2021. 22
[58] Hoang Anh Dau, Anthony J. Bagnall, Kaveh Kamgar, Chin-Chin Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh. The UCR time series archive IEEE CAA J. Autom. Sinica,
2019. 22
[59] Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I. Webb. Time series extrinsic regression. Data Min. Knowl. Discov., 3533):0032-106 2021. 22
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We included the implementation details in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
The source data and code are available at https://githul com cwm/mala mala- aall-lab/laaaaaaaaaaaaaaaaaanaaa SEMPO.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Setpi//gguidee/odeSuueeeee soiobiiisin for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.cccpubiic//eec S ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The average improvement of SEMPO over all baseline models is statistically significant the confidence of 95%.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our model is implemented Pytorch 2.1.2 with Python 3. 10 and all the experiments run on 4 A6000-48G GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //nuuriuusp... cc/public//eiicccccccccccc ctwt:/tthhhitiieeeeiieeeeeeeeeee
Answer: [Yes]
Justification: Our use of public datasets complies with NeurlPS Ethics Guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model is not high-risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited the data sources.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Not applicable (no new assets introduced in the paper).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects or crowdsourcing was used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Not applicable (no human subjects research).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: LLMs were only used for grammar checks (Grammarly). No impact onmethod- ology.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http:/rips.coooofeeeeeeeeee cc//oonferecccc/225/L for what should or should be described.
20

---

A Further Related Work
Task-specific Deep Models for Time Series. In recent years, deep learning has revolutionized time series forecasting by enabling models to capture complex temporal dynamics and nonlinear dependencies [52]. Early efforts explored various neural architectures for modeling temporal patterns, such Recurrent Neural Networks (RNNs) [1,53,44], Convolution Neural Networks (CNNs) [25, 55,56) 56], etc. Thereafter, the limited ability of these models to capture long-range dependencies has motivated a shift towards Transformer-basee which have achieved SOTA performance across a wide range of forecasting tasks. While these models demonstrate competitive in-distribution performance, they typically require meticulous hyperparameter tuning across different datasets and exhibit limited flexibility and generalizability when applied to -iittiiiiiiitto tasks, particularly under few-shot or zero-shot scenarios. Time series foundation models help address this challenge.
[7-11] and MLP-based [12-15] architectures,
B
More Details about MoPFormer Block
Recent insights from vision research [44] reveal that self-attention (SA) can be interpreted as a specialized MoF architecture, characterized by linear experts and quadratic gating score functions. Building on this connection, we propose that applying a mixture of prompts (MoP)-in in a manner similar to prefix-tuning [51]-within pre-trained Transformer backbones serves as a flexible and effective mechanism for injecting new experts. These newly added experts work in conjunction with the pre-trained experts, facilitating efficient adaptation of the model to new and unseen datasets.
8 LL
L+1:
Recon Head
Pred Head
Reconstruction Head
Prediction Head
Copy
Reconstruction Decoder
Recon Decoder
Pred
Prediction Decoder
Decoder
Eiix
B
m
Emix
Transformer Encoder
Transformer Encoder
Transformer Encoder
MoP
MoP
B
B Emix
tB
B
E
Emix
B
EASD & Patchify & Projection
EASD & Patchify & Projection
EASD Patchifyy & Projection
Learnable
X
Frozen
X
Figure 7: MoPFormer block details at different stages. Left: Energy aware pre-training stage.
Middle: MoP tuning stage. Right: Inference stage.
As shown in Figure 7 Left and Middle above, MoP is introduced during the MoP tuning stage. Both the encoder and decoder utilize the Transformer [45] encoder structure, as adopted [22]. They are constructed by stacking S MoPFormer blocks. Fach MoPFormer block integrates recent advancements from SOTA time series foundation models [18,21,31] Specifically, all LayerNorm layers replaced with RMSNorm [46] and applied pre-normalization scheme [47] to mitigate training instability in deep architectures. The standard non-linearity in FFN layers is replaced with SwiGLU [48], which improves expressiveness and stability while preserving parameter efficiency. During the tuning stage, given a univariate time series X E RR', instance normalization is first applied with zero mean and unit standard deviation to mitigate distribution shifts [41]. Recent work [40] has theoretically shown that normalization scales the absolute spectral energy by the variance but does not affect its relative distribution. Then, after the proposed EASD module and the Patchify & Projection module, the encoder processes the masked series representation B E N xPxD, along with the mixed prompt Fmix E Ik Sx2xx, PxD,, The decoder first aggregates information over the Nm dimension via an averaging operator and then takes the aggregated series B PxD, along with its corresponding mixed prompt Émix Sx2xPxx, input.
In downstream fine-tuning, the model is trained on the target dataset using the same joint training strategy in the MoP tuning stage, in sho-shot setting. For zero-shot inference, the reconstruction
21

---

decoder and head are removed, and all modules are frozen. A greedy scheduling algorithm [21] is then applied generate predictions for arbitrary target lengths, as shown Figure Right.
C
Experimental Details
C.1 Datasets
Pretraining Datasets. Large-scale publicly available time series collections, e.g, Time-300B [21], UTSD [19]. and LOTSA [18], have resolved data availability issues for building general time series forecasting models. Leveraging UTSD [19], we curate diverse subset covering multiple domains, totaling ~83 million time points. We then set the pre-training
validation split to 9:1, following [19]. Below, we outline the key properties of the datasets, including their domain, resolution, number of time points, total file size, Augmented Dickey-Fuller (ADF) test statistics, forecastability, and data source.
Table 4: List of pre-training datasets. All datasets are drawn from subset of the UTSD [19] data repositories, and all attribute specifications (e.g., resolution, time points, file size, ADF,, forecast) adhere to the original UTSD schema.
Datatset
Domain
Resolution Time Points File Size
ADF.
Forecast.
Source
Australian Electricity Demand
Energy
30 Min
1.16M
5M
-27.554
0.730
Monash [57]
Wind
Energy
4 Sec
7.40M
29M
-29.174
0.811
Monash [57]
KIDD Cup 2018
Nature
Hourly
2.94M
12M
-10.107
0.362
Monash [57]
Temperature Rain
Nature
Daily
23.25M
93M
-10.952
0.133
Monash [57]
Saugeen River Flow
Nature
Daily
0.02M
1M
-19.305
0.300
Monash [57]
Sunspol
Nature
Daily
0.07M
-7.866
0.287
Monash [57]
PIGCVP
Health
0.62M
-4855
0.577
UCR [58]
US Births
Health
Daily
0.00M
IM
-3.352
0.675
Monash [57]
PEMS04
Transport
5 Min
15.65M
60M
-15.192
0.494
[56]
PEMS07
Transport
5 Min
24.92M
96M
20.603
0.466
[56]
Pedestrian Counts
Transport
Hourly
3.13M
12M
-23.462
0.297
Monash [57]
Beijing PM25 Quality
Environment
Hourly
3.66M
14M
-31.415
0.404
[59]
Evaluation Datasets. For zero- and few-shot forecasting, we evaluate SEMPO on the widely- used Seee--eeress--bbrry (TSLib) benchmark [25], which includes seven datasets: ETThl, ETTh2, ETTml, E77m2, Weather, Electricity, and Traffic. In detail, the ETT (Electricity Transformer Temperature) dataset is collected from electric power industry and contains 2 years of data from two separate counties China. It includes temperature measurements and power load features. The dataset is further divided into subsets for different time granularities: (ETTh1, ETTh2) for --hour intervals and (,TTml, ETTm2) for 15-minute intervals. The Weather' dataset contains 21 meteorological indicators such as air temperature and humidity, is collected every 10 minutes from the Weather Station of the Max Planck Institute for Biogeochemistry 2020. The Electricity" dataset includes hourly electricity consumption data from 321 clients, measured in kWh. Due to missing data, the dataset was adjusted to an hourly granularity over two years. The Traffic dataset collects 48 months (2015-2016) of hourly data from the California Department of Transportation, which tracks road occupancy rates on San Francisco Bay Area freeways. The details about these datasets summarized in Table 5.
We also include GIFT-Eval [49] for evaluation. This comprehensive benchmark is designed to evaluate general-purpose time series forecasting models, particularly foundation models. It com- prises evaluation datasets, comprising approximately 144,000 time series and 177 million data points, spanning seven paniaiinssccooomc///iaaaaaa and Web/CloudOps and covers ten sampling frequencies, ranging from seconds to yearly. The benchmark supports both univariate (15 datasets) and multivariate (8 datasets) forecasting tasks with short-, medium- and long-term prediction lengths. GIFT-Eval also provides a rich set of statistical
energy, healthcare, nature, sales, transportation,
http:://www.bhcjjee.. jet.eejnnnaaaaaaapaeeee dc/wette/
"hupss//archive.ics. ics.uu uci. wtt.eu/////aansee/eeeeeittt aadDiagamm220111111111 12014
"http://pems.. dot.ca..oooooooooooooooo
22

---

Table 5: List of datasets TSLib benchmark.
Dataset
ETTh1
ETTh2
ETTml
ETTm2
Weather
Electricity
Traffic
Variables
7
7
7
21
321
862
Time Points
17420
17420
69680
69680
52696
26304
17544
Resolution
Hourly
Hourly Electricity
15 Min
15 Min
10 Min
Hourly
Hourly
Domain
Electricity
Electricity
Electricity
Weather
Electricity
Traffic
features (e.g., trend, seasonality, entropy, stability) for in-depth data analysis. To avoid data leakage,
we remove datasets overlapping with pre-traiming corpus and TSLib, and select datasets including dense, loop_sed seattle, taxi, solar, bizitobs application, bizitobs 12c, bitbrains service, car_ par_psts, and jenaaeeeaaae weather For GIFT-Eval, we adhere to its default evaluation protocol [49].
C.2 Baselines
Pre-trained FMs on time series. The models include TTM [23]. Time-MoB [21]. Timer [19], Moirai [18], Chronos [20], TimesFM [17], and Moment [16], detailed follows. A comparison between these pre-trained FMs is presented in Table 6. Notably, although SEMPO is evaluated with fixed context lengths in this paper- similar TTM [23] its Transformer based architecture is inherently insensitive to context length variations, as it can accommodate different context lengths through preprocessing techniques such as padding [20].
TTM [23] is MLP-based compact model tailored for zero- and few-shot multivariate time series forecasting. The TTM family includes three variants: TTMg (1M), TTMg (4M), and TTMA (5M). The official checkpoint is available at https://buggingaaee ct.cco///i/oee www.oo/o/e///oooooooooeee// lbm-granite/ granite-timeseries
Time-MoE [21] introduces a MoE structure into the Transformer framework, enabling sparse activation where only subset of experts is activated per input token. The Time-MoE family includes three variants: Time-MoE,, MoEaaee (1 (113M), Time-MoBjarge (453M), and Vime-MoBulle (2.4B). The official checkpoint is available https: https:/ htt.///ghggggg..eeee/////// co co/Maple728.
decoder-only Transformer architecture and pre-trained with next token prediction objective on massive time series corpora. The offi available at
Timer [19] adopts
cial checkpoint
cnteeries-fonddatiin moooommooolss--oooo-ooo-oooeoomm hwt.mmmdddccc-eceeeeeee------e-e
Moirai [18] is universal forecasting model designed handle diverse time series forecasting tasks in zero-shot manner. The Moirai family includes three variants: Moiraismall (14M), Moiraiçase (91M), and Moiraitarge (M). The official checkpoint is available at https: //tuggggggage.....c// cocooleccions// hwtms//alsfffo//ooionoooo mowoori----ooeellsss-----------e 4tt.a/00000051141134340000000000
/
Chronos [20] is designed for zero-shot and in-domain forecasting, and demonstrates that by learning the 'language' of time series. The Chronos family includes three variants: Chronos (46M), Chronosbase (200M), and Chronostarge (710M). The of-
ficial checkpoint is available at
httpre/innaaacccccoleeeeeeeeeeee wwc.oooections//aaaaeeeeeeeeeeee amazoo//
TimesFM [17] introduces input patching to represent sequences efficiently, leverages longer output patches to reduce autoregressive steps, and employs random masking learn from variable-length contexts. The official checkpoint is available at https: h/thgginggaaaeeeeeeee// ct/ google/timesseiee 0-200m.
Moment [16] leverages a patch-based Transformer encoder trained with masked time series modeling to learn robust representations across diverse domains. The Moment family includes three variants: Momentsmalt (40M), Momentaaee (125M), and Momentaagge (385M). The official checkpoint is available at https://huggingtin
co/AutonLab
L.MM-based time series FMs. LLM [36], detailed follows:
The models include Time-LLM [30], GPT4TS [34], and 12IP.
23

---

Table 6: Comparison between pre-trained FMs time series.
Method
SEMPO
TTM
Time-MoB
Timer
Moirai
Chronos
TimesFM
Moment
Architecture
Eacoder-only
Mixer-style
Decodor-only Decodor-only Bacoder-only Encodor-dcoder Decode-only
Encoder-only
(Max) Model Size
9.9M
SM
2,48
67M
311M
710M
200M
385M
(Min) Model Size
6.5M
IM
113M
67M
14M
46M
200M
40M
Input Token
Patch
Patch
Point
Patch
Paich
Point
Patch
Parch
Dataset Scale
83M
1B
3098
28B
27B/231B
84B
100B
1.13B
Length
512/10241536 512/1024/1536
44499
1440
50000
$$12
512
Source
[23]
[21] [19]
[18] [20] [17] [16]
Time-LLM [30] is a reprogramming framework that adapts off-the-shelf LLMs to perform time series forecasting. Time-LLM (LLaMA full, 3.4B). Time-LLM (LLaMA-8, 976M), Time-LLM (GPT-2, 124M). The official implementation is available at https://
/ai//Kiiieennnnne LLM.
GPT4TS [34] fine-tunes the input embedding, positional embeddings, layer normalization, and output layer to adapt to time series data. It has about 4M learnable parameters (total 87M). The official implementation available at https:// /tops:///itii httg/ggttthb.co/ wwt.://o//DDO--//iiii Lttpa//OOe_FFs__LL.____
I'IP-LMM [36] aims to enhance time series forecasting by aligning time series embeddings with the semantic space of pre-trained large language model. The total parameters are 117M. The official implementation is available at /ctps://githut com/panzi 18825/22IP- LLM.
Task-specific Models. The models include iTTranffrmee [11], DLinear [12], PatchTST [10),
TimesNet [25], Stationary [9], FEDformer [8], and Autoformer [7], detailed as follows:
iTTransfommer [11] inverts the conventional design by embedding the entire time series of each variate as a token, enabling attention mechanisms to directly model multivariate correlations. The official implementation is available at https: /tt:::///tttttt www/tttmmmiieeeeeeeeeeeeeeeeee /ttiirraanafrree
DLinear [12] directly maps the historical input sequence to future outputs via temporal linear projection, without modeling enter-variabl dependencies. The official implementation is available at https://g// tttbb.......... wtt.ttt...c..e.eeeeeoeoo...ttttt Lat.se-LaaLLLS---Lii
PatchTST [10] segments each univariate series into subseries-level patches and allows each time series channel to be processed separately using shared Transformer backbone. The official imple- mentation is available at https: thub weamm/uqqniiee////
TimesNet [25] introduces temporal 2D-variation modeling by transforming ID time series into structured 2D tensors based on automatically discovered periods. The official implementation is available at https://github com/thuml/ /Tiiem/emmmmet........m
Stationary [9] integrates series stationarization module to normalize input sequences and the de- stationary attention mechanism by re-incorporating statistical properties into the attention computa- tion. The official implementation available at https: //gtthub.. Transformers. ormers.
thub. http///mmm//ootttttionrrrrr
FEDformer [8] combines seasonal-trend decomposition with frequency-domanl modeling to enhance the global understanding of temporal dynamics. The official implementation is available at https: //github wtg.thu..com//iiiiii..... www.///iiiigg//aaaaooaaoe
Autoformer [7] replaces point-wise self-attention with a scries-wise correcorreltion operation that captures periodic dependencies across sub-series using fast Fourier transform. The official implementation available at https://githee com/thuml/Autoforae
C.3 Implementation Details
SEMPO and all other baselines are conducted on 4x NVIDIA A6000-48G GPUs. The feedforward dimension d_ff is set to 256, and head dropout is set to 0.2. During energy-aware pre-training, the model is trained for 10 epochs, with batch size of 2,048. MoP tuning is performed for 20 epochs, with the same batch size of 2,048. For few-shot and zero-shot settings, the batch size is reduced to 32. Early stopping is applied with patience of 6. All experimental results obtained by averaging

---

Priddion
Ground Trush
Pricidio
Droaan Truts
Precatio
Ground Trchh
Mid- eequernyy
Low traquunccyyyy
High-fraquancy
60
100
Rrg.
54
100
150
Froq
100
100
Frod.
(a) 512 + 192
(b) 512
336
(c) 512 - 720
Figure 8: Spectrum visualizations of predictions on ETTh 1 with different prediction lengths.
0.44
612-666
0.25
512-000
512
012
512--102
512--182
512-430
312-330
512-000e
51Pw220
5196-770
0.45
S1Pu7790
20.30
0.4
0.54
52
64
50
125
192
255
64
128
205
517
700
1024
(a) Nu on ETTh1/h2
(b) on Weather
(c) Dp on ETTm1/m2
Figure 9: Sensitivity analysis of mask_numbee Nas, prompt_numbe 1, and latent dimension Dp.
over three runs with different random seeds. In deep time series forecasting, performance variance across seeds is typically small, and following prior work [19,21,23], we report only the mean values.
D
Additional Results
D.1 Analysis of Low-energy Components
We further investigate SEMPO's capability to model low-energy components in temporal signals across diverse prediction lengths. Experiments are conducted on the ETThl dataset, with the results presented in Figure 8. These figures demonstrate that SEMPO exhibits full energy spectrum learning capability, applicable to both short- and long-term forecasting, Notably, SEMPO effectively identifies and leverages low-energy regions, regardless of whether they reside in high-, mid-, even low- frequency bands of the spectrum.
D.2 Sensitivity Analysis
The parameters mask_ number (Nas in Equation 2), prompt_nnmmine (1 in Equation 4), and la- tent dimension (D, in Patchify & Projection module) play a pivotal role in shaping the functionality of EASD and MoPFormer. In this section, conduct comprehensive experiments ETTm1/m2, and Weather datasets to investigate the impact of these parameters on forecasting perfor- mance. We explore range of values from the sets 2, 3, 5, 6). (32, 64, 96, 128, 192, 256), and (64, 128, 256, 512, 768, 1024] for NM. I, and D,. respectively, while strictly controlling for look- back window and prediction length. Specifically, lengths, i.e., 512 - (96, 192, 336, 720), and present the results in Figure 9. Figure 9a illustrates that, as the number of masked series increases, the forecasting performance initially improves due to the capture of more decomposed frequency patterns in the two energy branches, but eventually decreases due to information redundancy, Figure 9b shows that increasing the prompt number results in minimal changes in forecasting performance, with the optimal result achieved when the prompt number is set 128. Insufficient prompts fail to capture enough domain information, while too many introduce redundancy, hindering the model's 's efficient adaptation to new and unseen datasets. Figure 9c demonstrates that the forecasting performance fluctuates slightly before reaching 256, but drops sharply thereafter. For small datasets such ETTm 1/m2, compact latent dimension suffices to capture the data's underlying characteristics: therefore, we opt for dimension of 256 to strike an optimal balance between model capacity and expressive power.
the ETTh1/h2,
examine the impact of four different prediction
25

---

D.3 Scaling Analysis
To investigate the scalability SEMPO, we scaled both model size and dataset size and evaluated its predictive performance on two ETTh datasets. As shown in Figure 10 Left, expanding the dataset size consistently improves model performance, in line with the scaling law. As the dataset size increases, the performance improvement plateaus, indicating diminishing returns from further expansion. How- ever, the performance remains consistently below the MSE of the SOTA FMs Moirais and Moiraig with a dataset size of 27B. Regarding model size, as seen Figure 10 Right, SEMPO follows sim- ilar trend. Performance declines in the later stages, likely due to a mismatch between model capacity and dataset size. Despite this, SEMPO continues to surpass Moiraig and Moirai with 14M and 91M parameters, respectively.
Moiisl (0-02) 1490
0430
Kany (UR77
0.43
Zalbahe Moriee MSC
0.348
(S3M)
14 FM)
(5.5M)
0.35
10000 2732
0,98
(654))
(141M)
.5.5ML
45%
75%
100%
170% 2005
Precrate Percenlage
Layer Mumber
Figure 10: Scalability analysis on 1/Th
D.4 Efficiency Analysis
Since many Transformer-based FMs process data in a purely univariate fashion, the dataset size and inference costs directly proportional. To further highlight the performance and efficiency benefits, in addition to the smaller ETThl dataset (see Figure 6), we also compare the inference costs of SEMPO and other FMs on the Weather and ETTm2 datasets. All experiments are conducted using a batch size of 32 on a single A6000 48G GPU, as higher batch sizes often lead to out-of-memory (OOM) errors in many FMs, such Chronos,, The zero-shot results are shown in Figure 11, where SEMPO significantly outperforms all other FMs in both efficiency and accuracy.
0.54
SEMPO Timer
Chronos, Moirals
0.48
S11M (4525g)
SEMPO Timar
Moirais
Zero-shot MSE on ETTm2
(378g)
TimesIM
Zero-shot MSE on Weather
0.34
453M
Moment
Moirala
Moiral
453M
(1575))
MoE,
(584g)
Chronoss
Time-MoCo
0.32
Chronose
Time MoE;
91M
311M
200M
Chronoss
Time-Modd
0.30
(455%)
67
113M
(21624)
91M
Chronos,
(5175)
(17046)
(5578s)
(42110s)
0.28
(1020a)
385M
(217736)
710M
(8540%)
200M (61509s)
(156511s)
6.5M
6.5
0.30
67.4M
(225%)
(292%)
385M
(5857%)
46M
(16549g)
200M
0.26
(196s)
(12434)
14M
0.28
0.24
50
200
500
1000
3000
10000-40000
200500
2000
20000
50000
Inference Time (Seconds)
Inference Time (Seconds)
(a) ETTm2
(b) Weather
Figure 11: Additional efficiency comparison the ETTm2 and Weather datasets.
Beyond the comparisons among Transformer-based FMs, we have additionally included an efficiency comparison with TTM on the ETTh1 and ETTm2 datasets, summarized in Table right. While SEMPO is less competitive in parameter size and in- ference time due based architecture, it uses only 83M pre-training data (vs. TTM's IB) and still achieves lower MSE on both datasets. Moreover, SEMPO outperforms TTM in 31 out of 50 cases across five diverse datasets (see Table 10 in Appendix D.6), with par- ticularly substantial advantages on large-scale datasets such as Traffic and Electricity, demonstrating stronger knowledge transfer capability and superior sample efficiency.
Table 7: Efficiency comparison with TTM on the ETThl and ETTm2 datasets. Red: the best.
Datasets
Models
Inference Time (Seconds)
Parameters (M)
more complex Transformer-
ETTHI
TTM SEMPO
0.411
6.5
0.410
BTTm2
TTM SEMPO
0.288
67
6.5
0.286
26

---

D.5 Visualization of Predictions
To provide clearer comparison between SEMPO and other foundation models, we present prediction case studies on the ETTh2 dataset, as illustrated in Figure 12. We select several representative time series FMs as baselines, including Moiraiz, Timer, Time-MoEz, Time-MoEg, and Chronos L. SEMPO consistently delivers more accurate forecasting of future series variations compared to these diverse FMs, underscoring its superior predictive capability.
Input
Input
Ground Truth
Prediction
1.5
M www moll mmw I
wWW W/ll mmw
Veluee
Values
-1
-0.5
20
40
60
80
100
120
140
160
20
40
80
1000
120
140
160
(a) SEMPO
(b) Moiraiz
Input
Ground Truth
Prediction
Inpul
Ground Truth
Prediction
1.5
wMW W/ mmwl mn
MMW W mmw W
0.5
8..5
-0.5
0.5
40
60
100
120
140
160
80
140
Time
(c) Timer
(d) Time-MoE,,
Input
Ground Truth
Prediction
Impul
Ground Truth
Prediction
1.5
1.5
M MMW Wil mmm armaurmam Mymnn MM
wWW W/ mnN M Wo
0.5 Volugs
Values
0.5
-0.5
0.5
W
-1
100
0
20
40
60
80
100
120
140
160
Time
Time
(e) Time-MoFg
(f) Chronosz
Figure 12: Visualization of zero-shot prediction cases on the ETTh2 dataset by various FMs, using lookback window length L 512 and prediction length H 336.
D.6 Full Results Tables
Here, we present the comprehensive versions of the tables in the main text. These versions include the full results for multiple prediction lengths across all datasets. To save space in the main paper, some of these results are averaged across different prediction lengths.
Full Results for Zero-shot Forecasting. Table and Table 9 report the zero-shot results on the TSLib and GIFT-Eval benchmarks, respectively. It is observed that with only 6.5M parameters and using 83M pre-training data, SEMPO consistently outperforms other time series FMs on the
27

---

Table 8: Full zero-shot results on the TSLib benchmark [25]. Lower MSE/MAE values indicate better prediction. '-' denotes dataset used in pre-training and excluded in the evaluation. Values in ( ) denote model size and pre-training data size, respectively. 'S': Small, 'B' Base, 'L': Large. Red: the best, Blue: the second best.
Models
(6. (6.5M,85M) (113M,3191)) (455ML3151B) (67,4M)288) (14M.278)) (91M,177)) (311M227)) (46M.84B) (200M1848) (2NMMM (280MLU038) (185M11138)
SEMPO My Time-MoRF, Timer
Mairai
Moiraigz
Moirai
Choomes: Charenos Cheonos,
TimesFM
Momcnt
Metries
MSE MAE MSE MAE MSE MAR MISE MAR MSE MAK MSE MAE MIS MAE MSE MAK MSE MAE MSE MAK MSE MAE MSE
MAE
96
[0.384 0.408 0.355 0.382 0.350 0,382 0.414 0.499 0:407 0.405 o 394 0,399 0.410 O 403 0.476 0.411 0.452 D. 5.506 0.453 0-401 0.432 0:405 0.706
0.561
192
0.4499
01426 0.414 0.417 0.402
0.423
0.400
0:455 0:443 0:425 0.431 0.430 0.459 O 0.434 0.547 0.452 0,513 0:429 01525 0.426 0,492
0.438
0.716
0.579
0.417
0.449
0.454
0.447
0:459
0,435
0.463
0,438
0.450 0.437
0.481 0,553 0:450 (1570) 0.452
0.519
0.458
0.583
720
00.422
0.454
0:406
0.461 0.456 0.457
0.
0.578 0:484 0,619
0.477
0.597
Arg
[0.410
0,430
0,485
0.489
0.435 0,419
0:451
0.163
0.948 0,452 0.433 0,431 0,166 0.443 0.551 E 0,963 0.524 0:439 0.541 0,463 0.439 0.464
0,708
0.580
96
00.282
11.42 0,302 0.355
0.301
0.354
12305
0,255 0,258 0.345 0.2H5 0,342 0.254 0.344 O. 0.354 0.30T 11552 (1297 0.344 0.311 0.345
0.573
0.416
B1122
192
0.334
0.384 0:438
0.431 0.407
0,417
0:365
0.406 0,352 O 0.352 O 0.391 0.368 0.387 O. 386 399 O 3.3 0,588 0.3 0.379 0 0.392 0.401
0.397
0.
0.423
0.355
01403
0.617
0.509
0.476
0.5TH
0.426 0/422 0.414 0,412 0.466
0.490
0.426
00.395 11435 0.907
0.622
0.689
0.561 0.414
0.457
0.405 0.442 0-188 0.446
0.454 0:455 0:442 0:443 (145) 0.452 0.437 0.450
0.425
0.454
[0.343 0,391
0,566
0.479
0.477
0,452
U,306 OJAIN 0,355 0.403
3.399 0,397 0,394 0,409 0.392 0:401 01385 0,400 113396 0.405
0.392
0.430
96
10.465
0:443 0.339 0.368
0.309
0.357
0:440
0.422 O. SO9 0:558 O 528 0.33 0.512 0.433 O. SO8 0:421 0:447 0:400 0.451 O. 0.306 0. 0.566 0.374
0.679
0.544
0.484
0:455
0.443
0.440
0.435
0,505
0,458 0.471
0.455
0.474 0,535 0:451 0.464
0.401
0.600
0.550
11469
0.582
0.502
0.516
0.495
0.5TO
OLAMII 0.563 0.479 0.579 0.473 0.620
0,500
OL6IZ
13ARS (154)
0.487
0.445
0.429
0.701
0.557
720
0.557
0,.99
0.716
0.610
0,683
0.597
0.659
0.534 O 0-490 O O 674
SO8 O. 0.545 01682
0,516 0.630 0.507
0.513
0.470
0.719
0.569
2org
[0:503
0.507
0.483
0.471
0.544 0.476
966 0.457 0,565
0.465
0.419
0.555
00.1% 1.286 0.197
0.287
0.197
0.285
0,203
0.285
0. 0. O 223 288 0,209
0.287
0.2144 0.250
01201 0.389)
0.257
0.230
0.305
B11m2
0.252
0.323
0.376
0.265
0.327
2.250
0.325
0.285
0.338
0.306
0,354
0.586
0.501
0.534
0.481
0.319
0.361
03335 0,360
0.575
354
764
0,371
0.327
0.339
0.365
0.350
0.381
0.319
0.369
720
(0.391 01.04 1.034
0.683
0.978
0.668
01,05
0.410 03-53 0.425 0-556 0.4 0.656 0.423 0.448 0,430 0.482 0:435 0.437 0.420 0.464
0.445
0.423
0.424
Xung
00.256
11:341 0.538
0.463
0.509
0.452
0.29H
0.346
0:351 0.3 0.356
0:355 0.30S 11.444 01315 0,350 0.330
0.353
0.319
0.360
00.171
0.208 0.159
0.213
0.159
0.214
0.236 150 0,220
22121 0.21 3 0.21 0,258 198 0:231 0.207 0,230
0.216
0.271
Welther
192
0.2118
0.359 0.214
0.267
0.217
0.271
0.261
0.293
0.231 0.279 0-281 0.270
0.283 0.248 0.276 0.260 0.277
0.364
0.306
336
00.267
11304
0.292
0.326
0.312
0.343
0.332
0.340
(22899 O. 320 0.313 01527 0.312 0.521 0.321 0.301 0.514 (L310) 0,513
0.313
0.336
720
00.36
0,350 0:453
0.430
0.586
0.506
0,585
0.781 0.367 0,385 O 4-420 O 0.376 O. 8.6 O. 369 0,393 0,367 O. 1.336 0,561 0.390 0.365
0.369 0.380
Avg
(0.248 0.287 0.279 0.389
0.318
0.334 0.292 0.312 0.267 0,305 0.312 0.295 0.477 0.289 0.298 0,302 0.283 0,295 0.292 0.297
0.291 0.323
Eleetility
192
0.183
0.271
0,283
0,210 0.212 0,212 0.304 0.169
0.239 0.357 0:224 0.315 0.186 0.285 O. 0.186 295 0.2 280 0.207 0.285 0.214 0,294
0.267 0.196 0.273 0.198
0.850 0.762
0,761
336
0.198
0.297
0.284 0.372 0.244 0.331 0.215 0.299 0.221 O. 0.244 0.321 0.258 0:314 0.239 0.314
0.862 0.766
720
0.233
11329
0.456 0.479 0.291 0,365 0.257 0,332 0.296 0.355 0,524 0,371 0,310 0:355 0.316 0,362
0.888 0.774
Are
00.19
0.295
0,297 0L375 0,243 0,329 0.207 0.296 0.224 0.309 U.246 0,312 0.356 0.520 01326 0.328
0.361 0.766
00.441
0,333
0.536 0.368
0.562 0,378 0.558 0.575 0.541 0.364
1,300 0.800
Traflic
192
0.456 11.39
0.561 0L385
0.579 0.412 0.560 D. 8.509 0:570 0-406
1.403 0.802
336
0.467
11.444
0.614 0.412
0.5%4 0.420 0.584 0.433 (.571 0-404
1.455
0.044
720
00.503
0.360
0,749 0.464
0,723 0,472 0,711 0:464 0.719 0,489
1.477
0.808
Aor
[0.466
0.544
0.613 0.407
0,624 0,420 00603 0,413 00000 0,413
1.411 0.804
1d Count]
49
12
0
10
majority of datasets. Given that the Gval benchmark includes short- (i.e., H = L), medium- (i.e., H = 10 x L), and long-term (i.e. H = 15 x L) prediction lengths, the results in Table 9 further demonstrate SEMPO's s strong adaptability to a wide range of prediction horizons, achieving outstanding forecasting performance on most datasets.
Full Results for Zero-shot Comparisons with TTM. Since Table 8 provides the comparison with taassfommr-bbased time series FMs, Table 10 further reports results the comparison to the lightweight MI.P-based FM, TTM. Existing research [27] has experimentally demonstrated that simpler structures, such as MLPs, may be more suitable for smaller datasets, while larger datasets, due to their complex relationships, require more contextual structures, such as Transformers. To ensure a fair comparison, we exclude two small datasets, ETTh1 and ETTm1, from the TSLib benchmark to maintain a balanced dataset ratio for evaluation. The results are shown in Table 10. Despite TTM having only 1M parameters, SEMPO outperforms in the majority of cases (31 1 out of 50) utilizing only 83M data, showcasing its better capabilities in knowledge transfer than TTM. Notably, SEMPO demonstrates substantial advantages on large datasets, such as Traffic and Electricity, which is consistent with our earlier analysis.
Full Results for Zero-shot Comparisons with Chronos-Bolt and TimesFM (500M). Since new versions of Chronos and TimesFM are available, we have added zero-shot results for the latest Chronos-Bolt and TM (500M), summarized in Table I1 for up-to-date comparison. Across
28

---

Table 9: Full zero-shot results on the GIFT-Eval benchmark [49]. Lower NRMSE/SMAPE values indicate better prediction. 'S': Small, 'B:: Base, 'L' Large. Red: the best. Abbr. stands for abbreviation.
Datasets
Metrics SEMPO
Chronos
Chronosy
Chronoss
NRMSE 0.392
0,319
0.314
0.317
m_ dense
SMAPE
0.295
0,186
0.181
0.183
NRMSH 0.160
0.175
0.177
loop_seattee seattle (abbr. loop)
SMAPE
0.131
0,132
0.133
0.134
NRMSE 0.369
0.372
0.375
0.380
Laxi (abbr. laxi)
SMAPE
0.406
0,425
0.418
0.450
NRMSE 1.084
1.144
1.086
1.144
solar
SMAPE
1.217
1,270
1.230
1.252
NRMSE 0.213
0.160
0.155
0.147
bizitohs_applicaton (ahbr. biz_app)
SMAPH
0.162
0.098
0.110
0.107
NRMSH 0.780
1,010
1.040
0,999
bizitobs_124 (abbr. 12c)
SMAPE
0.880
1.140
1.135
1.076
NRMSE 0.299
0.236
0.233
0.227
bitbrains service (abbr. bb svc)
SMAPE
0.228
0.139
0.150
0.140
MASE SMAPE
3.075
3.013
3.008
2.958
car_ parts (abbr. car)
1.796
1.862
1.871
1.871
NRMSE
0.236
0,241
0,243
0,261
jena_ weather (abbr. jena)
SMAPH
0.638
0.657
0.653
0.649
Table 10: Zero-shot comparison with TTM on the ETTh2/m2, Weather, Electricity, and Traffic datasets. Lower MSE/MAE values indicate better prediction. Values in (.) denote model size and
pre-training data size, respectively. Red: the best.
Datasets
ETTh2
ETTm2
Weather
Electricity
Traffic
1 Count
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.276
0.335
0.177
0.259
0.151
0.196
0.182
0.274
0.508
0.342
TTM
(IM, IB)
0.346
0.383
0.246
0.307
0.195
0.241
0.196
0.287
0.538
0.355
336
0.385
0.417
0.324
0.350
0,256
0.285
0,219
0.307
0.574
0.373
720
0.420
0.450
0.406
0.405
0,319
0.333
0.261
0,342
0.617
0,390
Avg
0.357
0.396
0.288
0.330
0,230
0.264
0.215
0.303
0.559
0.365
(6.5M. 83M)
96
0.282
0.342
0.196
(1.286
0.171
0.228
0.168
0.271
0.441
0.333
SEMPO
192
0.334
0.384
0.252
0.323
0.218
0.269
0.183
0.283
0.456
0.339
336
0.355
0.403
0.306
0.354
0.267
0.304
0.198
0.297
0.467
0.344
31
0.395
0.435
0.391
0.404
0.336
0.350
0.238
0.329
0.503
0.360
Avg
0.341
0.391
0.286
0.341
0.248
0.287
0.196
0.295
0.466
0.344
five datasets, Chronos-Bol Bolt achieves the best performance in 20 cases, SEMPO 16, and TimesFM (500M) in 7. We can observe that SEMPO significantly outperforms TimesFM (500M). It should be noted that, even with the small version of Chronos-Boo (i.e., Chronos Chronos-Bolts ), it is trained on nearly 100B time points with 48M parameters. In contrast. SEMPO is trained only 83M points with just 6.5M parameters, yet it still ranks in second in terms of the overall performance. This compelling data Peleciency--erffrmaae for resource-conee resource-constrained environments, to which the other FMs such as Chronos-Bas and TimesFM (500M) are infeasible.
balance highlights SEMPO's promise
effective lightweight FM
Full Results for Different Lookback Window Lengths. Given that existing time series FMs, such as Time-MoE and TTM, utilize larger lookback window lengths, we additionally pre-train two variants: HEMPO-EEnanced (SEMPOg) with I, = 1024, I, 128 (parameters: 7.3M), and SEMPO- Advanced (SEMPO with 1536, Iap 128 (parameters: 9. 9.9M). The default configuration, with L 512, Lp 64 (parameters: 6. 6.5M), is denoted as SEMPO To avoid data leakage, exclude
29

---

Table 11: Zero-shot comparison with Chronos-Bolt and TimesFM (500M) on the ETTh1/h2/m//mg 1/m2 and Weather datasets. Lower MSE/MAE values indicate better prediction. Values in denote model size and pre- pre-training data size, respectively. Red: the best.
Dutavets
ETThl
ETTh2
ETTml
ETTm2
Weather
1st Count
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
(S0IM, >> >100B)
192
0.428
0.409
0.370
0.385
0.388
0.376
0.268
0.307
TimesFM
336
0.451
0.424
0.416
0.420
0.426
0.402
0.327
0.347
0.463
0.447
0.443
0.448
0.491
0.421
0.407
Avg
0.447
0.427
0.410
0:418
0.435
0.407
0.339
0.354
Chromas Bolks
192
0.451
0.415
0.351
0.371
0.382
0,366
0.239
0.291
0.218
0.250
0.497
0.396
0.405
0.433
0.306
0.334
0.275
(48M.
720
0.499
0.456
0.413
0.429
0.530
0.447
0.413
0.398
0.354
0.338
Avg
0.482
0.387
0.402
0.448
0.319
0.341
0.282
0.293
20
SEMPO Chromux- Boll
(205M, 1008)
192
0.435
0.412
0.355
0.371
0.381
0.366
0.247
0.295
0.222
0.258
0.400
0.403
0.432
0.393
0.314
0.336
0.282
720
0.480
0.449
0.414
0.422
0.525
0.439
0.415
0.395
0.366
0.348
Ave
0.464
0.431
0.390
0.399
0.446
0.399
0.325
0,342
0.290
0.301
(6.5M. 83M)
192
0.409
0.334
0.384
0.484
0.252
0.323
0.216
0.269
336
0.417
0.433
0.355
0.403
0.506
0.469
0.305
0.354
0.267
0.304
16
720
0.432
0.454
0.395
0.435
0.557
0.498
0.391
0.404
0.336
0.350
Avg
0.419
0.438
0.361
0.407
0.516
0.474
0.316
0.360
0.273
0.308
Table 12: Zero-shot comparison with Mime-MoE on the TSLib benchmark, excluding Electricity and Traffic datasets, under different lookback widnow lengths. Lower MSE/MAE values indicate better prediction. 'B: Base, 'E': Enhanced, "A': Advanced, LL': Large, *#*; L 1024, "t': L. 1536. Valuu in (. ) denote model size and pre- pre-training data size, respectively. Red: the best.
Models
SEMPO, (6.5M, 83M)
SEMPO (7.3M, 83M)
SEMPO (9.9M, 83M)
Time-MoE;; (453M, 3098)
Time-MoE'g (113M, 30913)
Time-Mok! (453M, 309B)
Time-MoE;
(113M, 309B)
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0.384
0.408
0.392
0.419
0.392
0.423
0.339
0.375
0.343
0.373
0,338
0.375
0,343
0.375
B11bl
192
0,409
0.426
0.415
0.433
0.422
0.443
0.389
0.412
0.385
0.405
0.391
0.415
0.387
0.407
336
0.417
0.433
0.423
0.440
0.435
0.452
0.429
0.443
0.421
0.437
0.428
0.445
0.420
0.438
720
0.432
0.454
0.440
0.465
0.447
0.468
0.498
0.504
0.507
0.511
0.482
0.497
0.493
0.507
Avg
0:410
0.430
0.417
0.439
0.424
0.441
0.414
0.433
0:414
0.431
0.409
0.433
0.410
0.431
96
0.282
0.342
0.309
0.365
0.308
0.362
0,286
0.338
0.271
0.332
0.277
0.335
0.266
0.329
ETT2
192
0.334
0.384
0,362
0.400
0.370
0.405
0.364
0.385
0.348
0.383
0.359
0.385
0.341
0.381
336
0.355
0.403
0.378
0.415
0.388
0.420
0.413
0.420
0.393
0.416
0.411
0.423
0.385
0.414
720
0.395
0.435
0.399
0.417
0.409
0.440
0.463
0.460
0.433
0.453
0.492
0.478
0.412
0.448
Avg
0.341
0.391
0.362
0.399
0.368
0.406
0.381
0.400
0.361
0.396
0.384
0:405
0.351
0.393
96
0.466
0.443
0.454
0.447
0.319
0,266
0.327
0.232
0.315
ETTmll
192
0.484
0.455
0.473
0.453
0.471
0.462
0.347
0.381
0.353
0.388
0.314
0.364
0.318
0.370
336
0.506
0.469
0.491
0.464
0,482
0.470
0.416
0,441
0.448
0.392
0.416
0,402
720
0.557
0.498
0.527
0.487
0.495
0.479
0.564
0.525
0.616
0.555
0.517
0.499
0.559
0.524
Avg
0,503
0.466
0.486
0.459
0.475
0.464
0.398
0.415
0.419
0.429
0.363
0.396
0.379
0.408
96
0.196
0.286
0.194
0.284
0.191
0.288
0.170
0.260
0.170
0.264
0.167
0.258
0.165
0.257
ET1m2
192
0,252
0.323
0.245
0.318
0.247
0.320
0.250
0.321
0.254
0.330
0.246
0.317
0.237
0.315
336
0.306
0.354
0.292
0.347
0,286
0.345
0.341
0.381
0.378
0.407
0.338
0.377
0.336
0.380
720
0.391
0.404
0.372
0.395
0.360
0.390
0.522
0.483
0.597
0.524
0.531
0.483
0.517
0.483
Avg
0.286
0.341
0.275
0.336
0.271
0.334
0.320
0.361
0.349
0.381
0.320
0.358
0.313
0.358
96
0.171
0.228
0.174
0.231
0.177
0.234
0.153
0.206
0.153
0.204
0.150
0.202
0.151
0.201
Weather
192
0.218
0.269
0.217
0.268
0.278
0.267
0.210
0.260
0.206
0.253
336
0.267
0.304
0.261
0.299
0.273
0.307
0.315
0.341
0.287
0.318
0,288
0.323
0,270
0.306
720
0.336
0.350
0.325
0.343
0.334
0.346
0.566
0.489
0.433
0.411
0,489
0.453
0.395
0.391
Avg
0.248
0.287
0.244
0.285
0.253
0.291
0.312
0.325
0.270
0.298
0.283
1"r Count
25
25
30

---

Table 13: Full-shot results the ETTh2, ETTm2, Weather and Traffic datasets. Lower MSE/MAE values indicate better prediction. Red: the best.
Models
SEMPO
GPT4TS
S-P-LLM
iTransformer
DLinear
PatchTST
TimesNet
Metrics
MSE
MAE
MSE
MSE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
96
0.273
0.334
0.285
0.342
0.278
0.340
0.297
0.348
0.302
0.368
0.274
0.337
0.351
0.399
BTTh2
192
0.333
0.376
0.354
0.389
0.246
0.385
0.371
0.403
0.404
0.433
0,341
0.382
0.394
0.429
336
0.355
0.400
0.373
0.407
0.367
0.406
0.404
0.428
0.511
0.498
0.329
0.384
0.415
0.443
720
0.399
0.435
0.406
0.441
0,400
0.436
0.424
0.444
0.815
0.640
0.379
0.422
0,477
0.481
Avg
0.340
0.386
0.355
0.395
0.323
0.392
0.374
0.406
0.508
0.485
0.351
0.381
0.409
0:438
96
0.160
0.251
0.173
0.262
0.165
0.257
0.175
0.266
0.164
0.255
0.166
0.256
0.233
0.305
ETTm2
192
0.221
0.294
0.229
0.301
0.222
0.299
0.242
0.312
0.224
0.304
0.223
0.296
0.265
0.328
336
0.273
0.328
0.286
0.341
0.277
0.330
0.282
0.340
0.277
0.339
0.274
0.329
0,379
0.392
720
0.349
0.380
0.378
0.401
0.363
0.390
0.378
0.398
0.371
0.401
0.362
0.385
0.390
0.407
Avg
0.251
0,313
0.266
0.326
0.257
0.319
0.209
0.329
0.259
0.325
0.256
0.317
0,317
0.358
96
0.143
0.193
0.162
0.212
0.145
0.195
0.159
0.208
0.170
0.230
0.149
0.198
0.193
0.244
Weather
192
0.189
0..34
0.204
0.248
0.212
0.267
0.194
336
0.240
0.280
0.254
0.286
0.243
0.280
0,253
0.289
0,257
0.305
0.245
0.282
0,363
0.366
720
0.325
0.338
0.326
0.337
0.338
0.318
0.356
0.314
Avg
0.224
0.261
0.2236
0.271
0:271
0.239
0.289
0.225
0.264
0.336
96
0.355
0.246
0.388
0.282
0.379
0.274
0.363
0.265
0.411
0.294
0.360
0.249
0.611
0.323
Trutle
192
0.373
0.253
0.407
0.290
0.397
0.282
0.385
0.273
0.421
0.298
0.379
0.256
0.609
0.327
336
0.384
0.260
0.412
0.294
0.407
0.289
0.396
0.277
0:431
0.304
0.392
0.264
0.616
0.335
720
0.427
0.286
0.450
0.312
0.440
0.301
0.445
0.312
0.468
0.325
0.432
0.286
0.656
0.349
Avg
0.385
0.261
0.414
0.294
0.406
0.286
0.397
0.282
0.433
0.305
0.391
0.264
0.623
0.334
Count
29
the Traffic and Electricity datasets, as they are included in the pre-training data for Time-MoE. The zero-shot comparison results with Time-MoE family are shown in Table 12. From the table, it is evident that, despite having significantly fewer parameters and less pre-training data, the variants of SEMPO achieve comparable forecasting performance to the large-scale Time-MoE variants across these five datasets.
Full Results for Full-shot Experiment. To ensure fairesss and enhance the comprehensiveness of our evaluation, we have added in-distribution experiments under the standard supervised setting adopted by Time-MoB [21]. Specifically, we compare SEMPO against LLM--based time series FMMs IS'MP-LLM and GPT4TS) as well as strong task-specific models (iTransformer, DLinear, PatchTST and TimesNet). Results on four datasets of different scales are presented in Table 13. The results show that SEMPO consistently outperforms six advanced time series models, demonstrating that its pre-training framework effectively equips the model with strong inductive biases for downstream adaptation, despite its lightweight design.
Full Results for Few-shot Experiment. Table 14 and Table 15 show the 5% and 10% few-shot results for all prediction lengths on the TSLib benchmark.
Full Results for Ablation Study. Table 16 presents comprehensive analysis of the impact of EASD and MoPFormer within SEMPO. As shown in the table, Group A replaces the dual- branch spectral masking in EASD with multi-band masking and random patch masking. Group B replaces MoP with Sparse MoE and prefix tuning, where sparse MoE is designed in two versions: experts with activated and 3 experts with 1 activated. EASD, by performing frequency decomposition across two energy branches, helps SEMPO capture low-energy components that are easily overlooked by existing methods such the alternative methods, leading to more accurate forecasts. The incorporation of the mixture of prompts significantly benefits SEMPO by effectively and efficiently decoupling the weights across diverse domains.
31

---

Table 14: Full few-shot results on the TSLib benchmark [25] with 5% training data. MSE and MAE are averaged over forecasting horizons H7 E (96, 192, 336, 720), where lower values indicate better prediction. Red: the best, Blue: the second best.
Models
SEMPO
TTM
TIm--LAM
GPT4TS
SPIP-ELM Transformrr
Di.inar
PatchTST
TimesNet
Stationary
FED Autoformer
Metries
MSF
MAR
MSR MAE MSE MAF MSF MAR
MSF
MAF MSE MAE
MAE MSF MAR
96
10,383
0.408
0,362 0,389 0,683 0,461 0.543 0,506 0.475 0.158 0.805 0.610
0,503 0.557 0,519 0.892 0,625 0.952 0.650 0.593 0.529 0.681 0,570 0.720
0.547
192
0.409
0,540 0.748
0.570 0,940 0,665
0.563
0.725 0.600 0.594 0.761 0.624
236
0.425
0.439
0,400
0,768 0,626 0.754 0,595 0,760
1,475
0.861 0.984
0.727
0.816 0.619 0,945 0,653 0.925
0.644
720
Avg 10.406
0.423
0,580
0/750
0.569 0.925 0.647
0.562
0.598
96
10.272
0,338
0.272
0,336
0.397
0.376
0,421
0,323
0.345
0.442
0,401
0.421
0.419
0,420
0.408
0.423
0,390
0.424
0.428 0.460
192
0.333
0.375
0.346
0,425
0.118
0.441
0,403
0.617
0.455
0.483
0,464
0.497
0.496
0.504
336
0.355
0.402
0,383
0.415
0,405
0,432 0.108 0.439
0,415
0.440
0.631
0.553
1.424
0.849
0,464
0.469 0.499
0.479 0.507 0.481
0.477
0.483 0.185 0:496
720
Avg
|0.320
0.372
0,418
4.400
380
0.475
0.837
0.439
0.448
0.461
0.454
(:441 0/457
10.307
0.354
0.341
0.357
0.386
0.005
0.232
0.414
0.606
0,518
0.540
0.726
0,578
192
0.344
0.375
0.384
0.380
0,450
0.464
0.440
0.438
0:437
0.430
700
0.585
0.358
0.390
0,441
0.436
0.681
0,539
0.844
0.59]
0.606
0.566
0.750
0.591
336
0.376
0.393
0,589
0.396
0,450
0,424
0.485
0.459
0.445
0.425
0.895
0.641
(L402
0.416
0,899
0.467
0.786
0.597
0.870
0.603
0,807
0.628
0.851
0.009
720
0.427
0.421
0.442
0.423
0.483
0,471
0.577
0.499
0.479
0.169
0.948
0.671
0.511
0.489
0,767
0.587
0.796
0.593
0.895
0.611
0.822
0.633
0.857
0.655
Arg
10.363
0.385
0,389
0.389
0,425
0.434
0.472
0,450
0.416
0.421
0,784
0.597
0.400
0.417
0.526
0:476
0.717 0.561
0.857
0.598
0,730
0.592
0.795 0.500
10.167
0.25%
0.376
0,174
01261
0.199
0,280
0.265
01236
0.206
0.220
0.299
0.229
0.252 0.322
BTTn2
192
0.225
0.295
0.244
0.305
0.215
0.287
0.256
0,316
0.228
0.297
0,510
0.362
0.206
0.373
0,264
0.324
0,511
0.361
0.298
0.349
0.394
0.261
0.291
0.357
336
0.274
0.325
0.319
0.347
0,273
0,330
0.315
0.353
0.294
0.347
0,373
0.399
0.280
0.423
0,334
0.367
0.338
0,366
0.255
0.380
0,578 0.427
0.478 0,517
720
0.359
0.381
0.402
0.403
0,433
0,412
0,460
0:436
0.425
0.404
0,478
0.454
0.674
0.563
0,454
0.432 0.509
0,465
0.475
0.445
0,523 0.510 0.583 0.538
Avg
10.256
0.315
0.785 0 328 0,274
0,323
300 346
0.2
0. 325
0. 3.356
0 388
0.309
0. 0.425 D. 314 0 0.352 314 0,372
341 0 373 D. 0.581 0.404 0 0.388 0:433
96
10.154
0.205
0.155
0.203
0,172
0.263
0.175
0.230
0.167
0.218
0,264
0.307
0.184
0.242 0.171 0.224
0.207
0.253
0.215 0,252
0.229
0.309
0.227
0.299
Weather
192
0.19%
0.246
11.206
0.257
0.224
0..771
0.227
0.276
11.255
0.275
0,284
0.326
11222
0.285
0.230
0.277
0.272
0.307
0
0.30T
0.265
0.317
0.275
0.533
336
0.249
0.285
0.255
0.292
0.282
(1.211
0.226
0,322
1.280
0.320
0.323
0.349
(.279
0,322
0.294
0.326
0,313
0,328
0.355
0.348
0.353
0.392
0.351
0,393
720
0.321
0.337
0.330
0.345
0,366
0.381
0.366
0,379
0.359
0.371
0,366
0.375
0.264
0.388
0,384
0.387
0.400
0,385
0.452
0.391
0.294
0.387
0.389
Asg
[0.230
0.268
0.236
0.273
0,250
0,309
0.263
0.501
0.257
0.295
0,309
0.339
0.263
0,308
0,269
0.303
0.298
0,318
0.327
0.328
0,309
0.353
0,310
0.353
Electreity
192
96
1.136
0.153
0.254
0.249
0.146
0.106
0.262
0.241
0.158
0.147
0,241
0.242
0.159
0.145
0.255
0.241
11169
11153
0.251
0.208
0.150
0.102
0.278
0.264
111501
0.163
0.251
0.263
0.163
0.145
0.244
0.260
0.315
0,315
0.396
0.389
0.484
0.501
0.518
0.551
0.247
0.235
01322
0.241
0.297 0.367
0.305
0.375
336
0.168
0.267
0.185
0.282
0,175
0.277
0.179
0.271
0.183
0.281
0.207
0.305
0.175
0.278
0,153
0.281
0.340
0.415
0.574
0.578
0.267
0.256
0.354
0.411
0.205
0.303
0.236
0.224
0,312
0.235
11.39
0.219
0.233
0.655
0.613
0.952
0.318
0.426
0.466
Avz
|0.165
0.263
0.183
0.276
0,179
0,268
0.178 0:273
D. 186
0.281 0.201
0 1.99
0.176
0.275
0,151 0.277 O. 402 0,453
0.627
0 0,366 0.353
0.345 0:404
96
10,390
0,283
0.411
0.293
0,414
0.291
0.419
0.298
0.410
0.289
0,431
0.312
0.427
0,304
0,404
0.286 0.854 0.492
1.468
0.821
0,670
0.421
0.795
0.451
Traffie
192
0.411
0.287
0.425
0.300
0.419
11291
01434
0,305
11415
0.295
0.456
0.326
0.447
0,315
0,412
0.204
0.894 0,517 1.509
0.836
0:653
0.405
0.837 0.508
336
0.421
0.292
0.449
01478
0,439
0.310
0.853
0.471
0.445
0.867 0.523
720
Asg
16.410 0.287
0.427 0.305 0,423 0.298 0.434 0,305 0.419 0.298 0,450 0.324
0.450
0.317
0,418 0.296 0.867 0,493 1.526 0.839 0.676 0.425 0.833 0.502
12 Count
48
II
2
32

---

Table 15: Full few-shot results on the TSLib benchmark [25] with 10% training data. MSE and MAE are averaged over forecasting horizons H E (96, 192, 33,, 720], where lower values indicate better prediction. Red: the best, Blue: the second best.
Models
SRMPO
TTM
TIm--LAM
GPT4TS
S'P-LMM Transformrr Di.inar
PatchTST
TemesNet
Stationary
FED Autoformer
Metries
MSF
MAR
MSR
MSE MAF MSF MAR
MAE MSF
MSE MAF MSE MAE
MSE MAF MSF MAR
96
60.378
0.405 0,362
0.389 0,668 0,660 0.458 0,456 0,463 0.159
0,790 0.586
0.492
0.095
0,516 0:485 0,861 0,628 0.918
0.639
0,512
0.499 0.613 0.552
192
0.407
0.570
0.565
0.797 0.503
0.722 0.598
236
0.426
0.439
0,399
0.589
01540
0.608
0.535
0,603
0.750
0.721
0.687
0.550
0941
0.648
0.574
0,750 0.619
720
0.456
0.474
0,465
0.477 0,700
0,501
0.725 0,591
0.713
0.388
1,234
0.811
0.986
0,743
0,762
0.610
0.877
0.641
0.587
0.645
0.728
0.614
0.721 0.516
Avg
[0.417
0.435
0,522
0.525
0.565
0.800
0.691
0.542
0,628
0,702 0.596
96
00.276
0,340
0.275
0.275
0.326
0.331
0,574
0,300
0.360
0,404
1.257
0,353
0.389
0.378
0,409
0.389
0.382
0.416
0.413 0.451
192
0.332
0.376
0,345
0.374
0.373
0.402
0.411
0.372
0,414
0,590
0.467
0.423
0.474
0.474
0.477
336
0.354
0.399 0.384 0.416 0,606
0,429
0.106
0.433
0.389
0.413
0.689
0.885
0.671
0.572
0,426
0:441
0.537
0.694
0.507
0,480
0.504
0.501
0.547 0.543
720
0.395
0.433 0.419
0 450
0,427
0,449
0.449
0.464
0.403 0.426
0.593
0. 538
0.824
0.648
0,477 0.480
0,510
0.491
0.477
0.472
0,499
0.509
0.516 0,503
Avg
|0,339
0.387
0,970
397
0.421 366
0 392 0.480
0,505
0.41
0.479 0.465
0.466
0.475
(1488 0.409
96
10.304
0.355
0.346
0,388
0.353
0.252
0.419
0,583
0,501
0.568
0.518
0.774
0.614
ETTml
192
0.340
0.375
0.375
0.395
0.373
0.416
0.429
0,423
0.368
0.403
0.717
0.568
0.382
0,412
0,437
0.434
0.630
0,508
0.781
0.574
0.617
0.546
0.754 0.592
336
0.372
0.392
0.599
0.392
0,413
0.426 0.469
0.439
0.417 0.428 0,735
0.575
1.419
0.434
0.476
0.454
0.725
0.508
01303
0.587
0.998
0.375
0.869 0.677
720
0.427
0.423
0.443
0.429
0.465
0.476
0.569
0.098
0.473
0.168
0,752
0.584
0.490
0.477
0.681
0.556
0.769
0,549
0.846
0.581
0,690
0.579
0,810 0.630
Arg
|0.360
0.386
0,390
0.394
0,804
0:427
0.164
0,541
0.402
0.112
0,728
0.565
0:411
0.119
0.501 0:466
0.677
0.537
0.797
0.578
0,722
0.605
0.802 0.528
00.166
0.256
0.176
0,177
01261
0.188
0.209
0.245
0,213
0,303
0.191
0.274
0,212
0.265
0.30S
0.209
0.352
0.454
BTTn2
192
0.223
0.295
0.242
0.304
0,241
0.314
0.251
0,309
0.207
0.293
0,274
0.338
(1278
0,345
0,252
0.317
0.270
0.323
0.291
0.343
0,307
0.379
0.694
0.691
336
0.276
0.329
0.315
0.345
0.274
0.327
0.307
0,346
0.264
0.231
0.361
0.394
0.238
0.385
0,306
0.355 0.323 0.353 0.348
0.376
0,543
0.559
2.408
1.400
720
0.357
0.381
0.394
0.399
0,417 0.300
0.426 0:417
0,381
0.387
0,467
0.442
0.436
0.440
0,433
0.427
0.474 0,449
0.461
0.438
0,712 0.614
1.913
1,166
Avg
10.255 0.315
0.781 0 327 0.277 0. 0.303 2.25 03335
0.248
0.313
0.
0.336
0.373
0.316
O. 368
D. 2.2566 0 0.343 O. 0.300 0.35
332 0 366 0/463 0.488
1.342
0:910
96
10.152
0.204
0.152
0.199
0,151 0.210
0.163 0.215
0,154
0.201
0,253
0.307
0.171
0.224 0.165 0.215
0.184
0.230
0.1 0.234
0.188
0.253
0.221
0.297
Weather
192
0.196
0.243
0.199
0.245
0,204
01.44
0.210
0.254
11195
0.241
0,292
0.328
(1215
0.265
0.210 0.257
0.245
0.263
0.269
0.295
0.250
0.304
0,270
0.522
336
0.246
0.282
0.246
0.282
0.261
113002
0.256
0.292
11,600
0L302
0.322
0.346
01258
0,299
0.259
0.297
0.305
0,321
0.370
0.357
0,312
0.340
0.320
0.351
720
0.319
0.336
0.336
0.346
0,309
0.332
0.321
0.339
0,303
0.329
0,365
0.374
0.320
0.345
0,332
0.346
0.381
0,351
0.441
0.405
0,387
0.295
0.390
0.396
Asg
[0.228
0,266
0.232
0.268
0.234
0,273
0.238
0.275
0,228
0.268
0,308
0.338
0.241
0.283
0,242 0:279
0,279
0,301
0.318 0.323
0.284
0.324
0,300
0,342
Electreity
192
96
0.134
0.151
0.251
0.247
0.140
0.163
0.259
0.237
0.151
0.1.9
0,248
0.241
0.156
0.139
0:252
0.237
1.142
0.163
0.260
01245
0.154
0.171
0.212
0.257
111501
0.164
0.253
0.264
0.100
0.140
0.255
0.233
0,305
0.299
0,379
0,373
0.420
0.411
0.459
0.466
0.261
0.231 0.323 0.261 0.348
0.256
0,338
0.406
336
0.167
0.267
0.150
0.275
0.169
0.270
0.175
0.270
0.173
0.270
0,196
0.295
0.181
0.282
0.150
0.276
0.319
0,391
0.434
0.473
0,360
0.445
0.410 0:474
0.201
0.298
0.241
0.240
0.233
0.223
0.369
0.426
0.510
0,530
0.585
0.715 0.685
Avz
|0.163
0.261
0.181
0.274
0,175
0.270
0.176
0,269
D.
0.178
0.275
0,
0.196
0.293
0.1
0.280
0, 0,150
0.273 O. 0.323
0,392
0.444
0.480 0. 0,346 0.427 0:431 0:478
96
10.355
0,277
0.408
0.292
0.418
0.291
0.414
0.297
0.,01
0.285
0.448
0.329
0.419
0.295
0,003
0.289
0.719
0.416
1.412
0.802
0.039 0.400
0.672
0,408
Tralfic
192
0.407
0.282
0.417
0.
0.295
0.414
(1.206
01426
0.301
11.100
0.293
0.487
0.360
01434
0.305
0.415
0.25%
0.74%
0.438
1.419
0.806
0:637
0.416 0.727
0.434
336
0.417
0.287
0.434
01449
0.426
01304
0.471
0.655
0.427
0.749 0.454
720
0.450
0.304
0.476
0.462
0.327
0.487
0.337
0.470
0.532
0.383
0.484
0.336
1,485
0.825
1.539
0.456
0.847 0.499
Asg
16.417
0.287
0.432
0.305
0,429 0.306
0.450
0,310
0,426
0.305
0,095
0.36]
0.447
0,313
0,420
0.305
0.951
0,535
1.455
0.815
0,663
0.425
0.749 0.446
12 Count
45
15
13
0
33

---

Table 16: Full results for our ablation studies. MSE and MAE for zero-shot forecasting on the TSLib benchmark [25], evaluated with different model components. Group A replaces dual-branch spectral masking with alternative masking methods, while Group replaces MoP with alternative specialization methods.
Datasets
ETTh1
ETTh2
ETTml
ETTm2
Weather
Electricity
Traffic
Metrics
MAE
MAE
MSE
MSE
MSE
MAE
10334
0:408
0.282
0.342
0.466
0.443
0.196
0.286
0.171
0.228
0.168
0.271
0.441
0.333
192
0.409
0.426
0.334
0.384
0.484
0.455
0.252
0.323
0.218
0.269
0.183
0.283
0.456
0.339
SEMPO
336
0.417
0:433
0.355
0.403
0.506
0.469
0.306
0.354
0.267
0.304
0.198
0.297
0.467
0.344
720
0.432
0,454
0.395
0.435
0.557
0.498
0.391
0.404
0.336
0.350
0.238
0.329
0.503
0.360
0.410
0.430
0.341
0.391
0.503
0.466
0.286
0.341
0.248
0,287
0.196
0.295
0.466
0.344
0:438
0,441
0.348
0.389
0.533
0,476
0.239
0.317
0.183
0.248
0.180
0.284
0.518
0.348
192
0.455
0.453
0.403
0.421
0.538
0.480
0.286
0.343
0.231
0.286
0.192
0.295
0.524
0.351
Multi-band Masking
336
0.463
0,461
0.424
0.438
0.491
0.338
0.376
0.279
0.324
0.205
0.308
0.535
0.355
720
0.495
0.492
0.519
0.498
0.620
0.527
0.506
0.471
0.352
0.377
0.240
0.338
0.572
0.372
avg
0.462
0.461
0.423
0.436
0.562
0.493
0,342
0.376
0.261
0.308
0.204
0.306
0.537
0.356
96
0.422
0.437
0.327
0.381
0.546
0,480
0.246
0.326
0.189
0,256
0.218
0.324
0.625
0.397
0.450
0.487
0.631
A.3 Random Patch Masking
336
0.449
0.461
0.417
0.438
0.573
0.498
0.381
0.388
0,280
0.328
0.247
0.349
0.648
0.404
0.492
0.526
0.449
0.343
0.278
0.686
avg
0.446
0.460
0.428
0.574
0.498
0.340
0.381
0.261
0.313
0.243
0.647
0.404
96
0.648
0.532
0.354
0.405
0.610
0.509
0.246
0.340
0.185
0.248
0.211
0.316
0.600
0.394
Spurse MoE
192
0.689
0.551
0.416
0.439
0.598
0.509
0.291
0.364
0.235
0,289
0.226
0.328
0.618
0.400
(8 experts, 2 activated)
336
0.692
0.558
0.448
0.461
0.610
0.518
0.350
0.398
0.283
0.326
0.241
0.342
0.626
0.401
720
0.720
0.583
0.521
0.509
0.659
0.545
0.491
0.477
0.350
0.376
0.274
0.369
0.664
0.416
avg
0.687
0.556
0.434
0.453
0.619
0.520
0.344
0.394
0.263
0.309
0.238
0.338
0.627
0.402
10.419
0.434
0.295
0.358
0.482
0.456
0.220
0.306
0.178
0.236
195
0.299
0.506
0,380
192
0.435
0.445
0.356
0.396
0.498
0.479
0.277
0.343
0,225
0.274
0.211
0.312
0.522
0,388
B.1 Sparse MoE
336
0.444
0.453
0.378
0.414
0.485
0.327
0.372
0.273
0.308
0.224
0.322
0.531
0.390
(3 experts,
activated)
720
0.468
0.478
0.404
0.441
0.566
0.520
0.410
0.422
0.339
0.353
0.263
0.352
0.569
0.406
avg
0.441
0.452
0.358
0.402
0.515
0.485
0.308
0.360
0,253
0.292
0.223
0.321
0.532
0.391
96
0.402
0.426
0.298
0.357
0.486
0.457
0.220
0.307
0.192
0.248
089
0.291
0.468
0.354
192
0.425
0.441
0.351
0.394
0,491
0.462
0.277
0.342
0.240
0.289
0.203
0.485
0.362
B.2 Prefix Tuning
336
0.437
0.451
0.389
0.431
0.513
0.476
0.328
0.373
0,286
0.322
0.218
0.318
0.495
0.365
720
0.456
0.476
0.398
0.437
0.565
0.507
0.413
0.422
0.356
0.368
0.258
0.349
0.529
0.379
avg
0.430
0:448
0.359
0.404
0.513
0.475
0.309
0.361
0,268
0.306
0.217
0.315
0.494
0.365
34


Reviews:
[{"review_id": "o7v91wgWGh", "reviewer": "Reviewer_Y38u", "summary": "The paper proposes SEMPO, a lightweight foundation model designed for general time series forecasting that significantly reduces both model size and pre-training data requirements, while still achieving state-of-the-art performance in zero-shot and few-shot scenarios.", "questions": "In lines 126–130, the paper introduces the concepts of $D_tune$ and $D_test$ in the context of few-shot and zero-shot evaluation. However, it is unclear whether these datasets are drawn from entirely new domains or from domains already included in the pre-training set $T_train$. Please clarify this distinction to help assess the generalization capabilities of the proposed model more accurately.\n\n\tIn Figure 1 (right), SEMPO (Zero) and SEMPO (Few) are marked separately, while other models such as PatchTST and Time-LLM are not differentiated by evaluation setting. This inconsistency may lead to confusion when interpreting the results. It would be clearer to present two separate figures: one for zero-shot comparisons and another for few-shot comparisons, ensuring a fair and consistent visual comparison across models.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1.\tThe paper is well-structured, with clear organization and a logical flow that makes it easy to follow.\n\n2.\tContribution: The paper explores an efficient way to address time series forecasting through Energy-aware Spectral Decomposition and Mixture-of-Prompts Transformer. Extensive experiment results demonstrate that SEMPO is able to outperform baselines with only 6.5M parameters which is pre-trained on 83 million time points\n\nWeakness\n\n1.\tThe core contribution of SEMPO is to present a lightweight yet effective foundation model for time series forecasting. However, the experimental evaluation primarily focuses on zero-shot and few-shot settings, where many of the baseline methods were not originally designed to perform. This may limit the fairness and comprehensiveness of the comparison. It would strengthen the empirical analysis if the authors could additionally include in-distribution evaluations—similar to what was done in Time-MoE—to demonstrate that SEMPO also achieves competitive or superior performance in standard supervised settings, despite its reduced model size.\n\n2.\tIn lines 51–53, the authors state: “Specifically, modeling full energy spectrum of temporal signals from pre-training data is essential for learning a comprehensive knowledge base of temporal patterns to underpin strong transferability across different datasets.” This is an important claim, but it lacks empirical evidence or citations to support it. \n\n\n3.\tWhile the proposed Mixture-of-Prompts (MoP) mechanism contributes to model adaptability, the overall technical novelty of SEMPO is limited. The use of FFT for spectral decomposition is a well-established technique in time series analysis, and the design of the model architecture—particularly the encoder-decoder structure and masking strategies—follows relatively standard practices. As a result, the contribution of the model as a whole may not be sufficient for a top-tier venue.\n\n4.\tThere are several formatting issues in the current draft, such as broken links to corresponding figures and missing or improperly linked citations. These issues hinder readability and should be addressed to improve the overall presentation quality.", "was_revised": true, "final_justification": "I would like to raise my score as my concerns are mostly addressed, although I am still have reservation on novelty.", "rebuttal": "We sincerely appreciate your constructive and positive comments on the clear presentation and technical contribution. Please see our response to your comments one by one below.\n\n>**Weakness #1** Lack of in-distribution evaluations similar to Time-MoE\n\nThank you very much for the insightful suggestion. To ensure fairness and enhance the comprehensiveness of our evaluation, we have added in-distribution experiments under the standard supervised setting adopted by Time-MoE. Specifically, we compare SEMPO against LLM-based time series foundation models (FMs) (S2IP-LLM and GPT4TS) as well as strong task-specific models (iTransformer, DLinear, PatchTST and TimesNet). Results on four datasets of different scales are presented in **Table A1** below:\n\n```\nTable A1. In-domain forecasting results on four datasets. \n```\n|Dataset |Horizon|SEMPO||GPT4TS||S2PLLM||iTransformer||DLinear||PatchTST||TimesNet||\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|||MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|\n|ETTh2|96|**0.273**|**0.334**|0.285|0.342|0.278|0.340|0.297|0.348|0.302|0.368|0.274|0.337|0.351|0.399|\n||192|0.333|**0.376**|0.354|0.389|**0.246**|0.385|0.371|0.403|0.404|0.433|0.341|0.382|0.394|0.429|\n||336|0.355|0.400|0.373|0.407|0.367|0.406|0.404|0.428|0.511|0.498|**0.329**|**0.384**|0.415|0.443|\n||720|0.399|0.435|0.406|0.441|0.400|0.436|0.424|0.444|0.815|0.640|**0.379**|**0.422**|0.477|0.481|\n|ETTm2|96|**0.160**|**0.251**|0.173|0.262|0.165|0.257|0.175|0.266|0.164|0.255|0.166|0.256|0.233|0.305|\n||192|**0.221**|**0.294**|0.229|0.301|0.222|0.299|0.242|0.312|0.224|0.304|0.223|0.296|0.265|0.328|\n||336|**0.273**|**0.328**|0.286|0.341|0.277|0.330|0.282|0.340|0.277|0.339|0.274|0.329|0.379|0.392|\n||720|**0.349**|**0.380**|0.378|0.401|0.363|0.390|0.378|0.398|0.371|0.401|0.362|0.385|0.390|0.407|\n|Weather|96|**0.143**|**0.193**|0.162|0.212|0.145|0.195|0.159|0.208|0.170|0.230|0.149|0.198|0.193|0.244|\n||192|**0.189**|**0.234**|0.204|0.248|0.190|0.235|0.200|0.248|0.212|0.267|0.194|0.241|0.320|0.329|\n||336|**0.240**|**0.280**|0.254|0.286|0.243|0.280|0.253|0.289|0.257|0.305|0.245|0.282|0.363|0.366|\n||720|0.325|0.338|0.326|0.337|**0.312**|**0.326**|0.321|0.338|0.318|0.356|0.314|0.334|0.440|0.404|\n|Traffic|96|**0.355**|**0.246**|0.388|0.282|0.379|0.274|0.363|0.265|0.411|0.294|0.360|0.249|0.611|0.323|\n||192|**0.373**|**0.253**|0.407|0.290|0.397|0.282|0.385|0.273|0.421|0.298|0.379|0.256|0.609|0.327|\n||336|**0.384**|**0.260**|0.412|0.294|0.407|0.289|0.396|0.277|0.431|0.304|0.392|0.264|0.616|0.335|\n||720|**0.427**|**0.286**|0.450|0.312|0.440|0.301|0.445|0.312|0.468|0.325|0.432|**0.286**|0.656|0.349|\n|**1$^{st}$ Count**||**25**||0||3||0||0||5||0||\n\nThe results show that SEMPO consistently outperforms six advanced time series models, demonstrating that its pre-training framework effectively equips the model with strong inductive biases for downstream adaptation, despite its lightweight design. These results will be included in the final version to address this concern.\n\n>**Weakness #2** Lack empirical evidence or citations to support claim on ''modeling full energy spectrum of temporal signals from pre-training data...''\n\nLow-energy frequency components represent subtle yet informative short-term variations, such as periodicities over short durations, which serve as good indicators for forecasting. Prior studies [Refs1-3] have revealed that attending to these low-frequency components with substantial spectral energy and mitigating this energy bias are critical for enhancing performance in **task-specific forecasting domain**. In particular, Fredformer [Ref1] and FilterNet [Ref2] provide **compelling empirical evidence** that the dominance of low-frequency components with substantial spectral energy can create a bottleneck to full-spectrum information utilization, ultimately hindering forecasting performance.\n\nBuilding upon this, multi-source time series pre-training can be viewed as an **aggregation** of learning processes across multiple individual datasets. If energy bias occurs within each dataset, it can compound during pre-training, leading to irreversible information loss. Therefore, we extend these prior findings to **general-purpose forecasting domain**, arguing that modeling the full energy spectrum is also essential for learning comprehensive forecasting patterns from a large collection of pre-training datasets to underpin strong generalization across datasets. \n\nTo further address your concern regarding empirical evidence, we conducted experiments comparing SEMPO with three strong FMs (Chronos$_L$, TimesFM, and Timer) under two settings: with and without low-energy components, to highlight the indispensable role of low-energy pattern modeling, please see our response to **Reviewer uD87**’s **Weakness #4** for the detailed results and discussion. We will include the references and empirical evidence in the final version.\n\n>**Weakness #3** The use of FFT and the encoder-decoder architecture with masking strategies appears standard and lacks novelty\n\nFFT-based spectral decomposition is indeed widely adopted in time series analysis, such as in [Refs1-4]. However, it is important to clarify that these prior works primarily address the **learning bias between high- and low-frequency components**, a phenomenon characterized by the prioritization of low-frequency features at the expense of high-frequency features. \nIn contrast, our work goes a step further by empirically revealing that existing time series FMs [Refs5-6] are generally biased toward high-energy components, while neglecting low-energy components that may distribute across mid-, high-, or even low-frequency bands (**see Figures 1, 4, 7**). The coarse-grained frequency bias correction is insufficient in this context, as it can lead to significant information loss of low-energy frequency information. To address this, we propose the novel EASD module, which performs adaptive spectral partitioning and dual-branch spectral masking to model low-energy yet informative frequency signals that are ignored in current FMs.\n\nAdditionally, most existing FMs adopt Transformer-based architectures, such as Moirai [Ref5] and Chronos [Ref6]. Leveraging a well-established encoder-decoder framework and following standard design practices does **not diminish** the novelty of our work in the two proposed modules. Instead, it enables us to clearly identify the above **critical limitation** in current Transformer-based FMs, namely **energy bias**. Building upon this mature architecture, we further propose the novel MoPFormer module, which enables parameter-efficient model adaptation across different datasets and domains. Together, these two innovative modules enable SEMPO to substantially reduce both the pre-training data scale and model size, while achieving strong generalization. \n\n>**Weakness #4** Several formatting issues in the current draft\n\nThank you for pointing this out. The broken links may be due to the separation between the main text and appendix. We will address the formatting issues, including figure links and citations, in the final version to ensure preferred readability.\n\n>**Question #1** Clarify whether $D_{tune}$ and $D_{test}$ are drawn from new domains or overlap with the pre-training set $T_{train}$\n\nFollowing standard few-shot and zero-shot settings, $D_{tune}$ and $D_{test}$ are drawn from entirely new domains that are not presented in $T_{train}$. We will clarify this point in our final version.\n\n>**Question #2** Fair and consistent visual comparison in Figure 1 (right)\n\nWe would like to clarify that Time-LLM, GPT4TS, and PatchTST are used only in the few-shot setting, while other FMs are evaluated in the zero-shot setting (see **Table 1 and Table 2**). Accordingly, we distinguish SEMPO (Zero) and SEMPO (Few) for fair comparison in each setting. \n\nTo address your concern regarding consistency, we will revise the figure and split it into two subplots for zero- and few-shot settings respectively in the final version.\n\n>**References**:\n- [Ref1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024\n\n- [Ref2] CATCH: Channel-aware Multivariate Time Series Anomaly Detection via Frequency Patching, ICLR 2025\n\n- [Ref3] Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer, ICML 2025\n\n- [Ref4] FilterNet: Harnessing Frequency Filters for Time Series Forecasting, NeurIPS 2024\n\n- [Ref5] Unified training of universal time series forecasting transformers, ICML 2024\n\n- [Ref6] Chronos: Learning the language of time series, TMLR 2024"}, {"review_id": "jOoA2tUlNg", "reviewer": "Reviewer_3Jvu", "summary": "This paper tackles the time series forecasting task by utilising a lightweight foundation model, involving 2 key modules, one decomposes the time series in the spectral domain by thresholding the frequencies and masking the frequencies randomly; and one mixture of prompts block. It is claimed that with such modules the model improves the time series forecasting performance in MSE, MAE or sMAPE.", "questions": "- What do Hec and Lec mean? What would you expect, if the energy-wise spectral partitioning leads to more than 2 partitions? \n\n- In figure 4, what is \"input\", and what is the point of plotting \"input\" in the figure?\n\n- In figure 8a in Appendix, could you explain the peak at N_m=5? Does it make sense to show the cases with N_m > 6?", "limitations": "Although claimed in the checklist, I failed to find a proper discussion of limitations in the conclusion.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strength\n\n  The paper is well written and easy to follow. The approach is clearly described, and the experimental setup and the results are well illustrated. \n\n  The proposed approach is to the best of my knowledge novel and sound. \n\n- Weakness\n\n  It seems the results in the experiments are from only one run. Since the masks are generated randomly, it would be more convincing if multiple runs are conducted and the standard deviations are reported. \n\n  It is claimed in the checklist 7, but I failed to see a discussion about the statistical test in the paper.", "was_revised": true, "final_justification": "My questions are mostly answered.", "rebuttal": "Thank you very much for the constructive comments. We are grateful for the positive comments on our methodological innovation, paper clarity and empirical justification. Please see our response to your comments one by one below.\n\n>**Weakness #1** It seems the results in the experiments are from only one run. Since the masks are generated randomly, it would be more convincing if multiple runs are conducted and the standard deviations are reported\n\nAll experimental results are obtained by averaging over three runs with different random seeds. In deep time series forecasting, performance variance across seeds is typically small, and following prior work [Refs1-3], we report only the mean values. To address your concern regarding random sampling, we additionally report the standard deviation of the 5\\% few-shot experiments to demonstrate the stability of our method in **Table A1** below. \n\n```\nTable. A1. Statistical results (mean ± std) of SEMPO and TTM under the 5% few-shot setting. The horizon length H is all set to 336.\n```\n|Models|SEMPO||TTM||\n|---|---|---|---|---|\n|Metrics|MSE|MAE|MSE|MAE|\n|ETTh1|0.425 $\\pm$ 0.0006|0.439 $\\pm$ 0.0000|**0.400 $\\pm$ 0.0010**|**0.421 $\\pm$ 0.0006**|\n|ETTh2|**0.355 $\\pm$ 0.0005**|**0.402 $\\pm$ 0.0000**|0.383 $\\pm$ 0.0012|0.415 $\\pm$ 0.0008|\n|ETTm1|**0.376 $\\pm$ 0.0012**|**0.393 $\\pm$ 0.0005**|0.389 $\\pm$ 0.0005|0.396 $\\pm$ 0.0008|\n|ETTm2|**0.274 $\\pm$ 0.0000**|**0.328 $\\pm$ 0.0000**|0.319 $\\pm$ 0.0008|0.347 $\\pm$ 0.0000|\n|Weather|**0.249 $\\pm$ 0.0005**|**0.285 $\\pm$ 0.0000**|0.255 $\\pm$ 0.0000|0.292 $\\pm$ 0.0000|\n|Electricity|**0.168 $\\pm$ 0.0000**|**0.267 $\\pm$ 0.0000**|0.185 $\\pm$ 0.0005|0.282 $\\pm$ 0.0005|\n|Traffic|**0.421 $\\pm$ 0.0008**|**0.292 $\\pm$ 0.0000**|0.446 $\\pm$ 0.0000|0.315 $\\pm$ 0.0000|\n\nThe results show that both SEMPO and TTM exhibit stable performance across different seeds. Furthermore, SEMPO demonstrates its superiority by outperforming TTM in both stability and predictive accuracy in 12 out of 14 cases.\n\n>**Question #1** What do Hec and Lec mean? What would you expect, if the energy-wise spectral partitioning leads to more than 2 partitions?\n\nHec refers to high-energy components, and Lec refers to low-energy components. If energy-wise spectral partitioning yields more than two groups, it effectively creates multiple patches along the energy axis.  This is similar to the frequency patching technique. The latter offers finer granularity to capture energy patterns but introduces significant computational overhead during masking.\n\n>**Question #2** In figure 4, what is \"input\", and what is the point of plotting \"input\" in the figure?\n\nIn **Figure 4**, \"input\" refers to the spectrum of the lookback window $X_{1:L}$. In time series forecasting, it is a common practice to include the lookback window in both time- and frequency-domain visualizations to present a complete forecasting sample, see similar examples in [Ref1, Ref4]. More importantly, it provides essential contextual information that helps assess whether the model's predicted spectrum preserves the spectral structure of the input and whether the model tends to emphasize certain frequency components.\n\n>**Question #3** In figure 8a in Appendix, could you explain the peak at $N_m=5$? Does it make sense to show the cases with $N_m>6$?\n\nIn principle, increasing $N_m$ provides the model with more complementary frequency perspectives via multiple random samplings, facilitating the learning of diverse spectral features. However, the spike observed at $N_m=5$ is likely caused by stochastic noise rather than a stable trend. Such fluctuations may arise as redundant or conflicting frequency views introduce noise or disrupt convergence, ultimately leading to performance degradation.\n\nUnfortunately, it does not make sense to show cases with $N_m>6$. This is because, as shown in **Figure 8(a)**, performance does not consistently improve beyond $N_m=4$, and it may even degrade due to increased randomness in masking and potential over-fragmentation of frequency views. Meanwhile, a larger $N_m$ imposes considerable memory and computational overhead, as frequency masking is applied separately to $N_m$ duplicated inputs in both high- and low-energy branches. Considering the trade-off between performance and efficiency, we choose not to explore $N_m>6$.\n\n>**Limitation** Although claimed in the checklist, I failed to find a proper discussion of limitations in the conclusion. \n\nWe briefly touched on this aspect in **Lines 360-362**, where we discussed future directions that reflect the current limitations of SEMPO. Specifically, one limitation is that the channel-independent mechanism may hinder the modeling of critical multivariate interactions. Another is that supporting only point forecasting reduces the SEMPO’s flexibility in capturing uncertainty through probabilistic forecasting. These limitations may constrain the SEMPO’s ability to capture richer time series characteristics.\n\nWe will add all the above empirical results and clarifications in the final version and hope we have addressed your concerns.\n\n>**References**:\n- [Ref1] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts, ICLR 2025\n\n- [Ref2] Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-shot Forecasting of Multivariate Time Series, NeurIPS 2024\n\n- [Ref3] Timer: Generative Pre-trained Transformers Are Large Time Series Models, ICML 2024\n\n- [Ref4] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024"}, {"review_id": "XpoAPz8nU9", "reviewer": "Reviewer_uD87", "summary": "The paper introduces SEMPO, a lightweight foundation model for general-purpose time series forecasting that achieves strong performance while using fewer parameters and less pre-training data than existing large-scale models. SEMPO integrates two key features: (1) an Energy-Aware Spectral Decomposition (EASD) module that transforms time series into the frequency domain and models both high- and low-energy frequency signals, improving utilization of pre-training data; and (2) a Mixture-of-Prompts-enabled Transformer (MoPFormer) that uses dataset-specific prompts to capture heterogeneous temporal patterns via a lightweight routing mechanism. SEMPO achieves zero-shot and few-shot forecasting on various datasets using less parameters and pre-training data points. Experiments show that it outperforms several LLM-based models.", "questions": "Q1. The claimed advantage of using less pre-training data is not convincingly demonstrated. The paper does not clearly specify how the number of datasets or total data volume used for pre-training compares to other models. Additionally, the number of pre-training epochs significantly affects the effective data utilization and should be reported to ensure a fair and meaningful comparison.\n\nQ2. MoE architectures typically rely on sparse routing to activate only a subset of expert networks, while maintaining diversity among the experts is essential for generalization. It remains unclear how SEMPO addresses these challenges, particularly how it ensures both routing efficiency and functional diversity in the proposed prompt-based expert system.\n\nQ3. Please respond to the other concerns raised in W1-W4.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. SEMPO achieves state-of-the-art results in zero-shot and few-shot time series forecasting across 16 datasets, despite using only 6.5M parameters and a modest 83M time points for pre-training, which are substantially smaller than competing models that use hundreds of millions of parameters and billions of data points.\n\nS2. The proposed EASD module effectively captures both high- and low-energy frequency components of time series. \n\nS3. The paper provides extensive experiments on benchmark time series datasets covering various domains. It includes zero-shot, few-shot, ablation, and efficiency analyses that can demonstrate the superiority of SEMPO.\n\n\\\nWeaknesses: \n\nW1. Some claims are not sufficiently justified. One notable example is the claim that existing time series FMs suffer from energy bias and fail to effectively handle low-energy frequency components is primarily supported by a case study. A more rigorous justification for this limitation by other models would be necessary to convincingly support this claim.\n\nW2. The presentation of SEMPO’s data efficiency is unclear. While the model is positioned as requiring less pre-training data, the paper does not clearly quantify the significance of this reduction compared to baselines. It is unclear how many data points are utilized in other models. Also, the epochs for pre-training play an important role in its real data utilization. \n\nW3. There is a lack of clarity regarding the relationship between the pre-training and inference stages. There’s a gap between training and inference stage. It is better to illustrate the process of this two stages in details. \n\nW4. The analysis of low-energy components in the experiments is weak. The paper includes only a qualitative case study to demonstrate SEMPO’s effectiveness in modeling such signals. However, a more comprehensive quantitative evaluation is preferred to be provided.", "was_revised": true, "final_justification": "The authors have sucessfully addressed my concerns. I would adjust my score accordingly", "rebuttal": "Thank you very much for the constructive and positive comments on our technical contribution and empirical justification. Please see our response to your comments one by one below.\n\n>**Weakness #1** Lack of sufficient justification for the energy bias issue.\n\nThank you very much for the suggestion. We’d like to clarify that the claim regarding energy bias has been widely justified in the **task-specific forecasting domain** in the literature [Refs1-3], and we explained the basis of this claim in **Lines 145-148**. This phenomenon reflects a learning bias in Transformers, where the self-attention tends to prioritize low-energy components over low-energy ones. Such limitation has been **rigorously supported** by both empirical and theoretical studies. For example, Fredformer [Ref1] and FilterNet [Ref2] provide comprehensive empirical evidence showing that the dominance of low-frequency components, due to their **substantial spectral energy**, can lead to frequency bias, as Transformer-based forecasting models tend to disproportionately attend to these signals during learning; Amplifier [Ref3] complements this by offering a theoretical justification that frequency components with low spectral energy contribute minimally to the training loss and receive slower parameter updates compared to high-energy components.\n\nBuilding upon these previous findings, and given that most existing foundation models (FMs) [Refs4-6] are built upon Transformer architectures, we empirically investigate whether energy bias persists in **general-purpose forecasting domain**. Our experiments on ETTh1 and Electricity datasets confirm its presence in foundation models, such as Chronos$_S$ [Ref6] and Moirai$_L$ [Ref5], as illustrated in **Figures 1, 4**. To further address your concern regarding this claim, we have added additional quantitative results in our response to **Weakness #4**, showing that other FMs like TimesFM and Timer also struggle to effectively capture low-energy frequency components. \n\nTogether, prior studies and our quantitative and qualitative results provide a rigorous justification for our claim that existing FMs exhibit energy bias.\n\n> **Weakness #2 and Question #1** Unclear presentation of SEMPO’s data efficiency, such as pre-training data size and epochs.\n\nWe’d like to clarify that both the **model size and pre-training data size** of SEMPO and all baseline FMs have been reported in **Table 1 in main text and Table 6 in Appendix C.2** (in the $(\\cdot,\\cdot)$ values just below the names of the models). SEMPO achieves strong zero-shot forecasting performance with just 83M time points, underscoring its remarkable data efficiency compared to baselines trained on data with hundreds of millions of time points to billions (or even hundreds of billions) of time points. \n\nAdditionally, the number of pre-training epochs have been provided in **Lines 1010-1012 in Appendix C.3**: in SEMPO, the energy-aware pre-training is conducted for 10 epochs, followed by 20 epochs of MoP tuning. To enable a straightforward comparison, Table A1 summarizes the pre-training epochs and data sizes of SEMPO and other four strong FMs. Notably, SEMPO requires only 83M time points,  with 30 epochs of training, achieving a substantial reduction in training data and demonstrating superior data efficiency compared to baseline FMs trained on billions of time points. \n\n```\nTable. A1. Pre-training epochs of SEMPO and four representative FMs.\n```\n|Models|SEMPO|TTM|Timer|Chronos|Moment|\n|-|-|-|-|-|-|\n|Data size|83M|1B|28B|84B|1.13B|\n|Epoch|30|20|10|200|2|\n\nTo further validate SEMPO’s data efficiency, we extended the pre-training by 5 additional epochs in either the pre-training or the tuning stage, with the results presented in Table A2 below. TTM with 20 training epochs is used as baseline. SEMPO reaches optimal performance within 30 total epochs under 83M time points, indicating that SEMPO has effectively converged and fully utilized the pre-training data. \n\n```\nTable. A2. Zero-shot performance with increased pre-training epochs, averaged over horizons of {96,192,336,720}.\n```\n|Datasets|ETTh2||ETTm2||Electricity||Traffic||\n|-|-|-|-|-|-|-|-|-|\n|Metrics|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|\n|SEMPO (10,20)|**0.342**|**0.391**|**0.286**|0.342|**0.197**|**0.295**|**0.467**|**0.344**|\n|SEMPO (15,20)|0.348|0.396|0.288|0.340|0.206|0.304|0.497|0.363|\n|SEMPO (10,25)|0.349|0.398|0.287|0.340|0.202|0.300|0.483|0.353|\n|TTM (20)|0.357|0.396|0.288|**0.330**|0.215|0.303|0.559|0.365|\n\n> **Weakness #3** Unclear relation between pre-training and inference stages.\n\nWe’d like to clarify that we’ve provided the **relationships/differences** for the energy-aware pre-training stage, the MoP Tuning stage, and the inference stage in **Figure 2 (Details) in Appendix B**.  \nAs mentioned in **Lines 867–881 of Appendix B.1**, the energy-aware pre-training does not include the MoP module, which is introduced and used only in the MoP tuning.\n\nAs also discussed in **Lines 886–902 of Appendix B.2**, after pre-training, the model enters the fine-tuning phase, where the model is trained on the target dataset using the same joint optimization strategy as in the MoP tuning. In this stage, only the parameters in the mixture of prompts and prediction heads are updated, while the Transformer backbone remains frozen. During zero-shot inference, the reconstruction decoder and head are removed, and there are no parameter updates in all modules.  We will clarify these relationships and differences in the final version.\n\n> **Weakness #4** Weak analysis of low-energy components.\n\nWe appreciate your constructive suggestion. We’d like to clarify that our analysis of low-energy components is not limited to the qualitative case study in Figure 4. In fact, we’ve also investigated SEMPO’s capability to model low-energy signals across **diverse prediction lengths**, with the results shown in **Figure 7 in Appendix D.1**. \n\nIn terms of quantitative analysis, we conducted ablation studies comparing SEMPO with its variant **A.1 Multi-band Masking**, which removes the energy-based partitioning mechanism (i.e., a variant that also ignores the energy bias, as what is done in existing FMs). The results across **all** datasets and forecasting horizons in Table 3 demonstrate the superior performance of SEMPO over this variant, in which modeling low-energy components is the only difference between SEMPO and this variant. The same observation can be found in the full ablation study results in **Table 13 in Appendix D.6**.\n\nWhile these results provide meaningful evidence, we agree that more quantitative evaluations would strengthen the argument. In response, we justified the importance of low-energy modeling from another perspective, i.e., whether including such a component in the pre-training could improve the performance of existing FMs.  The results of SEMPO and three strong FMs under two settings: with and without low-energy modeling are given in **Table A3** below. Note that as the energy distribution varies across time windows and **the distinction between high- and low-energy components is inherently relative**, applying a unified threshold for statistical evaluation across the entire dataset is infeasible. We therefore randomly selected five batch samples from Traffic and ETTm1 and report their averaged results in **Table A3**.  \n\n```\nTable A3. Impact of modeling low-energy components (LEC) on MAE across different FMs.\n```\n|Datasets|LEC Setting|Chronos$_L$|TimesFM|Timer|SEMPO|\n|-|-|-|-|-|-|\n|Traffic|w/o LEC|0.381|-|0.391|0.304|\n||with LEC|0.341|-|0.357|**0.256**|\n||Improvement|10.5%|-|8.7%|**15.8%**|\n|ETTm1|w/o LEC|0.502|0.450|0.491|0.472|\n||with LEC|0.455|**0.394**|0.434|0.409|\n||Improvement|9.4%|12.4%|9.6%|**13.3%**|\n\nTwo key observations can be found in **Table A3**. 1) Adding the low-energy modeling consistently results in very significantly decreased MAE across all FMs, highlighting the positive role these components play in enhancing the prediction accuracy; 2) SEMPO achieves the largest performance gains from incorporating low-energy components, with a 13.3% improvement on ETTm1 and 15.8% on Traffic, indicating its superior ability to effectively mitigate energy bias and model low-energy signals. We will add this quantitative evaluation to the final version.\n\n> **Question #2** How it ensures routing efficiency and functional diversity？\n\nUnlike traditional MoE that requires sparse activation over large expert networks to ensure routing efficiency and reduce computation, SEMPO adopts **lightweight, learnable prompt vectors** as experts. Based on these lightweight experts, SEMPO applies soft gating via a simple linear-softmax function over input tokens, enabling efficient expert selection without the need for sparse activation. These prompts are prepended to the key-value pairs in self-attention, allowing all tokens to share the same Transformer backbone. This avoids multi-branch execution, resulting in minimal computational overhead only, and well supports parameter-efficient adaptation.\n\nTo ensure functional diversity, each prompt-based expert is tailored to a specific small dataset, independently learned, and assigned input-adaptive weights via soft gating. A reparameterization strategy further transforms each expert into structured key-value pairs through a two-layer MLP, enhancing its expressiveness and promoting diverse functional roles across the expert set.\n\n>**References**:\n- [Ref1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024\n- [Ref2] FilterNet: Harnessing Frequency Filters for Time Series Forecasting, NeurIPS 2024.\n- [Ref3] Amplifier: Bringing Attention to Neglected Low-Energy Components in Time Series Forecasting, AAAI 2025.\n- [Ref4] Timer: Generative Pre-trained Transformers Are Large Time Series Models, ICML 2024\n- [Ref5] Unified training of universal time series forecasting transformers, ICML 2024\n- [Ref6] Chronos: Learning the language of time series, TMLR 2024"}, {"review_id": "9zuKigHQzN", "reviewer": "Reviewer_EgfG", "summary": "The paper proposes a lightweight foundation model for time series forecasting, SEMPO, which includes an Energy-Aware Spectral Decomposition (EASD) module and a Mixture-of-Prompts enabled Transformer (MoPFormer) module. The former leverages high- and low-energy frequency signals to improve data utilization, while the latter achieves parameter-efficient adaptation through a lightweight prompt-based expert system. Experiments show that SEMPO, with 6.5 million parameters and pre-trained on 83 million data samples, outperforms mainstream models in terms of prediction accuracy under zero-shot and few-shot scenarios, demonstrating its efficiency and generalization capability.", "questions": "The authors can refer to the weakness listed above.", "limitations": "The authors can refer to the weakness listed above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Proposes a lightweight SEMPO model that achieves zero-/few-shot prediction advantages with only 6.5 million parameters and 83 million pre-training data, offering high resource efficiency and practical value.  \n2. The experiments cover multiple datasets and baseline models, including ablation studies and visualization analyses, providing thorough validation.  \n3. The paper is well-structured, with clear formulas, standardized figures, precise technical descriptions, and smooth logical flow.\n\n**References**\n\n1. It is recommended to include full-shot experiments to evaluate model performance with the full dataset, thereby validating the benefits of pre-training.  \n2. A comparison with frequency debiasing models such as Fedformer [1] should be added to highlight the superiority of the EASD module.  \n3. The description of the two-stage training paradigm in the main text is too brief; more details should be provided.  \n4. The EASD module uses a masking mechanism for data augmentation, which actually increases the amount of training data. Therefore, further verification is needed to confirm whether the claimed reduction in pre-training cost is valid.  \n5. The zero-shot experiment in Table 1 and the efficiency analysis in Figure 6 do not include the lightweight model TTM. It is suggested to add its results to fully illustrate the performance gradient of the lightweight methods. Additionally, since new versions of Chronos and TimesFM are available, it is advisable to update the comparisons to the latest state-of-the-art models to ensure the conclusions remain current with field advancements.  \n6. The visualization of $Z_{Lec}$ in Figure 2 appears to contain an error; the complementary relationship between $Z_{Lec}$ and $Z_{Hec}$ should be corrected.  \n7. In Figure 4(b), the low-energy feature response of Moirai seems to reflect prediction inaccuracies rather than being completely neglected. The authors are encouraged to provide a more reasonable explanation for this phenomenon.\n\n**Weaknesses**\n\n[1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024", "was_revised": true, "final_justification": "The paper proposes a lightweight foundation model for time series forecasting, SEMPO, which includes an Energy-Aware Spectral Decomposition (EASD) module and a Mixture-of-Prompts enabled Transformer (MoPFormer) module. The former leverages high- and low-energy frequency signals to improve data utilization, while the latter achieves parameter-efficient adaptation through a lightweight prompt-based expert system. Experiments show that SEMPO, with 6.5 million parameters and pre-trained on 83 million data samples, outperforms mainstream models in terms of prediction accuracy under zero-shot and few-shot scenarios, demonstrating its efficiency and generalization capability. The author's response has largely addressed my concerns. Considering the significance of SEMPO for lightweight time series forecasting foundation models, I have raised my score to 5.", "rebuttal": "Thank you very much for the constructive comments. We are grateful for the positive comments on the technical contribution, sound experiments and well-structured presentation. Please see our detailed response below.\n\n>**Weakness #1** Include full-shot experiments to evaluate model performance.\n\nPlease see our detailed response to **Reviewer Y38u**’s **Weakness #1**, where we added the full-shot experiments on four representative datasets of different scales from the TSLib benchmark.\n\n>**Weakness #2** Compare with frequency debiasing model Fredformer.\n\nThank you for highlighting the debiasing model Fredformer. Following your suggestion, we added a comparison between SEMPO and Fredformer under 10\\% few-shot and full-shot settings, as summarized in **Table A1** below. SEMPO demonstrates strong capabilities, consistently outperforming Fredformer under both settings in both MSE and MAE. While Fredformer shifts attention toward mid- and high-frequency components, it overlooks energy-level bias, as low-energy signals can occur across all frequency bands. In contrast, the EASD module in SEMPO  suppresses dominant high-energy components to better capture subtle yet informative low-energy patterns, thereby reducing information loss and improving the forecasting generalization.\n\n```\nTable A1. SEMPO vs. Fredformer with 10% and 100% training data. Results are averaged over horizons of {96,192,336,720}.\n```\n|Dataset|Metrics|ETTh1|ETTh2|ETTTm1|ETTm2|Weather|Electricity|Traffic|\n|---|---|---|---|---|---|---|---|---|\n|SEMPO (10\\%)|MSE|**0.417**|**0.339**|**0.361**|**0.256**|**0.228**|**0.163**|**0.417**|\n||MAE|**0.436**|**0.387**|**0.386**|**0.315**|**0.266**|**0.261**|**0.288**|\n|Fredformer (10\\%)|MSE|0.592|0.393|0.473|0.279|0.229|0.187|0.428|\n||MAE|0.527|0.425|0.458|0.332|0.279|0.279|0.299|\n|SEMPO (100\\%)|MSE|**0.419**|**0.340**|0.356|**0.250**|**0.224**|0.172|**0.385**|\n||MAE|**0.432**|**0.386**|0.383|**0.313**|**0.261**|0.270|**0.261**|\n|Fredformer (100\\%)|MSE|0.424|0.348|**0.350**|0.256|0.225|**0.161**|0.394|\n||MAE|0.435|0.390|**0.379**|0.317|0.265|**0.256**|0.273|\n\n>**Weakness #3** Overly brief description of the two-stage training paradigm in the main text.\n\nWe have explained the details of two-stage training paradigm in **Lines 882-902 of Appendix B.2**. For clarity, we will move this to the main text in the final version.\n\n>**Weakness #4** The claimed reduction in pre-training cost needs further verification.\n\nThe pre-training data reduction we argue in the paper is mainly focused on the original data size, not considering the data size resulting from data augmentation techniques. However, even we take into account of the data size increment due to the data augmentation, our data reduction argument still holds.  Specifically, random masking is a widely adopted strategy in recent time series foundation models (FMs), such as TimesFM [Ref1] and Moment [Ref2], meaning that their pre-training data size is even larger than the one, TimesFM (100B) and Moment (1.13B), we reported in Table 1 if we consider the augmented samples as part of the pre-training data size.\n\nUnlike these FMs that merely apply masking on original samples, SEMPO duplicates each sample $N_m$ times across both energy branches, yielding a training size of $83M \\times N_m \\times 2$. Yet, despite this expansion, the total pre-training data of SEMPO remains substantially smaller than the billion-scale data required by other FMs such as TimesFM (100B) and Moment (1.13B). More importantly, SEMPO requires only 83M time points as original pre-training data, making it well-suited for deployment in resource-constrained environments. The cost introduced by duplication is already reflected in the reported memory usage and computational overhead.\nTherefore, our claim of reduced pre-training cost remains **valid**. We will add these discussions in the final version.\n\n\n>**Weakness #5** Lack of comparison to TTM in zero-shot and efficiency comparisons, and results of new versions of Chronos and TimesFM.\n\nTo clarify, we have provided zero-shot comparison results with the lightweight **MLP-based** FM TTM in **Table 9 in Appendix D.6**, as explained in **Lines 264-267**, since **Table 1** focuses on zero-shot results against **Transformer-based** FMs. \n\nFollowing your suggestion, we have additionally included an efficiency comparison with TTM on ETTh1 and ETTm2, summarized in **Table A2** below. While SEMPO is less competitive in parameter size and inference time due to its more complex Transformer-based architecture, it uses only 83M pre-training data (vs. TTM's 1B) and still achieves lower MSE on both datasets. Moreover, SEMPO outperforms TTM in **31 out of 50 cases** across five diverse datasets (see **Table 9 in Appendix D.6**), with particularly substantial advantages on large-scale datasets such as Traffic and Electricity, demonstrating stronger knowledge transfer capability and superior sample efficiency.\n\n```\nTable A2. Efficiency comparison with TTM on ETTh1 and ETTm2.\n```\n|Datasets|Models|Inference Time (Seconds)|Parameters (M)|MSE|\n|-|-|-|-|-|\nETTh1|TTM|3.2|1|0.411|\n||SEMPO|22|6.5|**0.410**|\nETTm2|TTM|5.2|1|0.288|\n||SEMPO|67|6.5|**0.286**|\n\n\nTo address your concern regarding the updated FMs, we added zero-shot results for the latest **Chronos-Bolt and TimesFM (500M)**, summarized in **Table A3** below for an up-to-date comparison. Across five datasets, Chronos-Bolt achieves the best performance in **16** cases, SEMPO in **12**, and TimesFM (500M) in **5**. We can observe that SEMPO significantly outperforms TimesFM (500M). It should be noted that, even with the small version of Chronos-Bolt (i.e., Chronos-Bolt$_S$), it is trained on nearly **100B** time points with **48M** parameters. In contrast, SEMPO is trained on only **83M** points with just **6.5M** parameters, yet it still ranks in second in terms of the overall performance. This compelling data efficiency–performance balance highlights SEMPO’s promise as an effective lightweight FM for resource-constrained environments, to which the other FMs such as Chronos-Bolt and TimesFM are infeasible.\n\n```\nTable A3. Zero-shot comparison on five datasets with Chronos-Bolt and TimesFM (500M).\n```\n\n|Dataset|Horizon|SEMPO||Chronos-Bolt$_S$||Chronos-Bolt$_L$||TimesFM (500M)||\n|---|---|---|---|---|---|---|---|---|---|\n|||MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|\n|ETTh1|192|**0.409**|0.426|0.451|0.415|0.435|0.412|0.428|**0.409**|\n||336|**0.417**|0.433|0.497|0.437|0.476|0.433|0.451|**0.424**|\n||720|**0.432**|0.454|0.499|0.456|0.480|0.449|0.463|**0.447**|\n|ETTh2|192|**0.334**|0.384|0.351|**0.371**|0.355|**0.371**|0.370|0.385|\n||336|**0.355**|**0.403**|0.396|0.405|0.400|**0.403**|0.416|0.420|\n||720|**0.395**|0.435|0.413|0.429|0.414|**0.422**|0.443|0.448|\n|ETTm1|192|0.484|0.455|0.382|**0.366**|**0.381**|**0.366**|0.388|0.376|\n||336|0.506|0.469|0.433|0.396|0.432|**0.393**|**0.426**|0.402|\n||720|0.557|0.498|0.530|0.447|0.525|**0.439**|**0.491**|0.444|\n|ETTm2|192|0.252|0.323|**0.239**|**0.291**|0.247|0.295|0.268|0.307|\n||336|**0.305**|0.354|0.306|**0.334**|0.314|0.336|0.327|0.347|\n||720|**0.391**|0.404|0.413|0.398|0.415|**0.395**|0.421|0.407|\n|Weather|192|**0.216**|0.269|0.218|**0.250**|0.222|0.258|-|-|\n||336|**0.267**|0.304|0.275|**0.290**|0.282|0.298|-|-|\n||720|**0.336**|0.350|0.354|**0.338**|0.366|0.348|-|-|\n|**1$^{st}$ Count**||12||**16**||||5||\n\n>**Weakness #6** The visualization of $Z_{Lec}$ in Figure 2 appears to contain an error.\n\nWe have carefully checked Figure 2, and confirm that $Z_{Lec}$ and $Z_{Hec}$ are indeed complementary, with $Z_{Lec}=Z-Z_{Hec}$.\n\n>**Weakness #7** In Figure 4(b), the low-energy feature response of Moirai seems to reflect prediction inaccuracies rather than being completely neglected.\n\nYes, your observation is correct. As shown by the spectral discrepancy between the prediction and ground truth, Moirai$_L$ exhibits non-negligible responses in low-energy regions, in contrast to Chronos$_S$, which largely neglects low-energy components. However, these responses in Moirai$_L$ do not consistently align with the spectral peaks of the ground truth. This indicates that while Moirai$_L$ is able to partially capture low-energy signals, it still predominantly attends to high-energy components, resulting in less accurate learning of low-energy features, such as the important positional information. We will clarify this in the final version.\n\n>**References**:\n\n- [Ref1] A Decoder-only Foundation Model for Time-Series Forecasting, ICML 2024\n\n- [Ref2] MOMENT: A Family of Open Time-series Foundation Models, ICML 2024"}]

Author rebuttal:

Meta-review: This paper proposes a new lightweight time series foundation model with strong experimental results across 16 datasets in 0 and few short forecasting tasks. The reviewers mostly found the proposed approach to be novel and the experiments to be good although (prior to the author/reviewer discussion completing) not thorough enough. However, after the author/reviewer discussion, the bulk of reviewer concerns were addressed, and the reviewers unanimously favored acceptance for this paper (3 accept, 1 borderline accept). I am recommending acceptance for this paper as a result.
Decision: Accept (poster)