Title: SEMPO: Lightweight Foundation Models for Time Series Forecasting
Venue: NeurIPS 2025
Accepted: yes
Keywords: time series

foundation model

forecasting

lightweight
Abstract: The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resource-constrained environments. In response to this growing tension between versatility and affordability, we propose **SEMPO**, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) _energy-aware **S**p**E**ctral decomposition module_, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) _**M**ixture-of-**P**r**O**mpts enabled Transformer_, that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.

SEMPO: Lightweight Foundation Models for Time Series Forecasting

Abstract

The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resourceconstrained environments. In response to this growing tension between versatility and affordability, we propose SEMPO , a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) energyaware S p E ctral decomposition module , that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) M ixture-ofP r O mpts enabled Transformer , that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.

1 Introduction

Time series forecasting constitutes a fundamental analytical tool within dynamic real-world systems, underpinning a diverse array of contemporary domains, such as urban computing [1], inventory optimization [2], energy demand estimation [3], and climate system modeling [4]. Forecasting has often been approached in a task-specific and end-to-end fashion. A wide range of methods have been developed in this paradigm, from classic statistical models, such as Exponential Smoothing [5] and Gaussian Processes [6], to modern deep learning approaches, including Transformer-based architectures [7-11] and those based on Multilayer Perceptrons (MLPs) [12-15]. One key issue shared by these approaches is that their effectiveness often relies heavily on large in-distribution training data and extensive domain expertise.

To mitigate this issue, a profound paradigm shift has been recently ushered in by the emergence of foundation models (FMs) for general-purpose forecasting, which leverage large-scale pre-training

Input

3

2

3

2

1

2

3

Prediction

Input

0

20

40

60

80

100

0

1

2

3

Prediction

Ground Truth

Input

0

20

40

60

80

100

0

1

SEMPO

0

20

40

60

80

100

0

1

2

3

Prediction

Ground Truth

Input

0

20

40

60

80

100

0

1

Time-LLM

GPT4TS

PatchTST

SEMPO (Few)

ChronosB

ChronosL

MoiraiS

MoiraiB

MoiraiL

Time-MoEB

Time-MoEL

SEMPO (Zero)

on massive amounts of time series data to enable zero- and few-shot generalization across a broad spectrum of downstream tasks. In line with this trend, research efforts have increasingly emphasized scaling up pre-training datasets and model sizes to push the frontier of transferability in diverse time series domains [16-21]. However, excessively large pre-trained models often impose substantial computational burdens during both training and inference, hindering its application to computational resource-constrained environments [22, 23]. This motivates the question: Can we instead devise 'small' FMs to serve as an alternative to these 'large' FMs for zero- and few-shot forecasting in such resource-constrained environments? A few studies [22,23] have explored this question very recently. They showed remarkable success in reducing the model size while maintaining its generalization, but they still rely on computationally costly pre-training on large-scale data. We take this question a step further: Can we substantially reduce both model size and pre-training data size while maintaining strong generalization ability?

To answer this question, we propose SEMPO , a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Two key challenges on the way to this goal include 1) significantly improved utilization of pre-training data and 2) lightweight model architecture with strong generalizability. Two novel modules-energyaware spectral decomposition ( EASD ) and mixture-of-prompts enabled Transformer architectures ( MoPFormer )-are introduced in SEMPO to respectively address each of these two challenges.

Specifically, modeling full energy spectrum of temporal signals from pre-training data is essential for learning a comprehensive knowledge base of temporal patterns to underpin strong transferability across different datasets [26-28]. However, current time series FMs [17, 18, 20, 21] often suffer from bottlenecks in utilizing these pre-training data, since their pre-training can ignore low-energy frequency signals characterized by small amplitudes and distributed across mid-, high-, or even lowfrequency bands (see Figure 1 Left ). Despite being subtle, such signals are persistent and can encode stable temporal dynamics that are crucial for forecasting generalization. In light of this, we propose the EASD module that transforms each time series into the frequency domain and decomposes it along energy and frequency axes. To prevent low-energy components from being overwhelmed by high-energy ones, we perform adaptive spectral partitioning using a learnable energy threshold conditioned on the input's spectral characteristics. As shown in Figure 1 Middle , leveraging this partitioning, we apply dual-branch spectral masking on the frequency axis, where independently parameterized multi-band masks can selectively suppress high- and low-frequency bands in each energy branch.

To accommodate a wide range of heterogeneous input patterns from different domains, most time series FM solutions [16, 19, 20, 29] opt for a large Transformer architecture with vast amounts of parameters, e.g. , hundreds or thousands of millions of parameters, as shown in Figure 1 Right . Some recent works explore model specialization ( i.e. , model customization for specific tasks or datasets) with sparse activation at various levels-such as dataset [30], frequency [17, 18], and token [21, 31]to alleviate the computational burdens while maintaining its generalization over the heterogeneous temporal patterns. However, the architecture of their base model remains large. To address this issue, we propose the MoPFormer module that learns a mixture of lightweight prompt-based experts and adaptively routes different time series tokens to these experts. This enables cost-effective learning of the heterogeneous input patterns without relying on large network architectures such as the costly mixture of experts in Transformers [21,31] in current approaches (see Figure 1 Right ).

In summary, this work makes the following four main contributions:

Prediction

Ground Truth

Prediction

Ground Truth

Input

We propose SEMPO, a novel time series FM with significantly reduced model size and pre-training scale, yet demonstrating superior generalization ability on diverse downstream forecasting tasks.

We reveal the bias towards high-energy frequency signals in the pre-training of current solutions and introduce the EASD module to mitigate this bias, enabling significantly improved data exploitation.

We then introduce MoPFormer, a Transformer with mixture of small learnable prompts, to learn the heterogeneous temporal patterns from diverse input datasets, offering an effective lightweight alternative to popular Transformers with large mixture-of-experts networks.

Comprehensive experiments on two large-scale benchmarks show that SEMPO, with only 6.5M parameters pre-trained on 83M time points, surpasses SOTA time series FMs with hundreds of millions of parameters pre-trained on billions of time points, demonstrating an average reduction of 12% and 22% in forecasting errors under zero- and few-shot scenarios, respectively.

2 Related Work

LLM-based Time Series FMs. The emergence of Large Language Models (LLMs), empowered by in-context learning and emergent abilities, has catalyzed a growing body of research on transferring their powerful pre-trained knowledge to the time series domain. One line of research [32,33] leverages pre-trained LLMs as zero-shot learners by directly converting numerical time series into textual inputs. Another line of research [34-37], such as GPT4TS [34], Time-LLM [30], and S 2 IP-LLM [36], adapts LLMs to time series forecasting through prompting or fine-tuning to bridge the modality gap. Time-LLM [30] reprograms time series using text prototypes and prepends textual prompts to provide enriched contextual information. S 2 IP-LLM [36] aligns time series embeddings with the pre-trained semantic space of LLMs by retrieving prompts from semantic anchors. Despite their positive impact on generalization compared to task-specific models (see Appendix A), these models rely on substantial computational resources.

Pre-trained FMs on Time Series. Foundation models pre-trained from scratch on large-scale, multi-domain time series data have recently attracted widespread attention [16-21,29,31]. Building on this paradigm, Moment [16] and Moirai [18] adopt encoder-only architectures with masked pre-training objectives, focusing on leveraging self-supervised learning to capture intricate temporal dependencies. TimesFM [17] and Timer [19] employ decoder-only architectures under a GPT-style causal modeling paradigm for autoregressive forecasting. Chronos [20] adopts an encoder-decoder architecture that tokenizes numerical time series via scaling and quantization into a fixed vocabulary. Most of these models [16,19,20,29] rely on large Transformer architectures with numerous parameters to memorize the heterogeneous input patterns across diverse domains, which can increase both the learning complexity and the costs associated with training and inference.

Lightweight Time Series FMs. Recently, sparse mixture of experts (MoE) has been introduced into Transformers [21, 31] to enhance generalization across the heterogeneous temporal patterns by routing diverse time series tokens to specialized experts. Despite simultaneously alleviating the computational budget with sparse activation, the architecture of their base model remains large. With the emergence of small language models [38, 39], there is also a growing interest in small time series FMs [22,23], addressing practical computational resource and cost constraints. For example, TTM [23] leverages a lightweight mixer-style architecture and pre-trains a family of compact models for general forecasting. Chronos [20] and Moirai [18] also provide small variants to tackle such real-world settings. However, they still require substantial pre-training on large-scale time series data, significantly hindering their applications in resource-constrained environments.

3 Methodology

3.1 Problem Statement

Given a time series input X = [ X 1 1: L , X 2 1: L , ..., X N 1: L ] ∈ R N × L with the number of variates N and the lookback window length L , where X i 1: L denotes the historical observations of the i -th variate, the task of time series forecasting is to predict the future H time steps Y = [ X 1 L +1: L + H , X 2 L +1: L + H , ..., X N L +1: L + H ] ∈ R N × H . This work focuses on developing a generalist time series forecaster pre-trained on a collection of J datasets from diverse domains,

T train = {D 1 train , D 2 train , ..., D J train } , where each D j train = { X j , Y j } represents the j -th dataset. For downstream evaluation, we consider two settings, including zero- and few-shot forecasting. In the few-shot setting, the pre-trained model is further adapted using a small target dataset D tune , where D tune ≪ T train , and evaluated on a unseen D test from the same domain. In the zero-shot setting, the model is tested directly on D test without any additional adaptation. Following standard zeroand few-shot settings, D tune and D test are drawn from entirely new domains that are not presented in T train . To support any-variate forecasting, we apply channel independence [10] to decompose multivariate inputs into univariate series.

3.2 Overview of SEMPO

As illustrated in Figure 2, SEMPO is built upon an encoder-decoder architecture, which comprises four key components: energy-aware spectral decomposition (EASD), patchify and project, mixture-ofprompts enabled Transformer (MoPFormer), and reconstruction and prediction heads. EASD include two steps: energy-wise spectral partitioning and frequency-wise spectral masking , while MoPFormer includes a mixture of prompts (MoP) and a MoPFormer block . The training of SEMPO follows a two-stage paradigm, as outlined in Section 3.5. Below we introduce the EASD and MoPFormer components in detail.

3.3 EASD: Energy-aware Spectral Decomposition

Rooting in the weighting characteristics of self-attention, the Transformer backbones widely used in time series FMs [17,18,20,21] often exhibit an energy bias [40], being prone to frequency components with high energy while overlooking those with low energy. Existing forecasting methods [22,26,27] primarily focus on the bias between high- and low-frequency components along the frequency axis. To address this issue, we introduce EASD to improve spectral modeling by mitigating energy bias and enhancing the subtle yet persistent temporal patterns. It achieves this by sequentially performing energy-wise spectral partitioning and frequency-wise spectral masking as follows.

Energy-wise Spectral Partitioning. Given a univariate time series X ∈ R L , after instance normalization [41], the normalized input is transformed into the frequency domain by applying Fast Fourier Transform (FFT) along the temporal dimension. The spectral energy at each frequency is defined as the squared magnitude of the corresponding Fourier coefficient [40], i.e. , Energy[ f ] = | Z [ f ] | 2 , where Z denotes the complex spectrum. We then perform spectral partitioning in the energy axis by adjusting a learnable energy threshold τ according to the input's spectral characteristics to delineate high- and low-energy components. This separation ensures low-energy components are not overwhelmed by high-energy components, thus enabling the extraction of frequency-localized patterns for modeling complex temporal dynamics. The process can be formulated as:

where ⊙ represents element-wise multiplication along the frequency dimension, (Energy( Z ) > τ ) yields a binary mask that identifies frequency components to be retained in Z Hec if their energy exceeds the threshold τ , while the remaining low-energy components are assigned to Z Lec .

Frequency-wise Spectral Masking. Existing FMs [16,17,42] primarily leverage random patch masking in the temporal domain to expose models to varying context lengths, enabling the learning of multi-scale temporal dependencies and improving generalization. While this strategy enhances contextual diversity, it fails to explicitly model the structured and periodic nature inherent in time series. Inspired by the frequency decomposition learning [22], we propose a frequency-axis spectral masking module. In both the high- and low-energy branches, high- and low-frequency bands are selectively suppressed through independently parameterized multi-band masks. Concretely, for each energy branch, we generate a set of frequency band masks { M 1 , M 2 , ..., M N M } by sampling a frequency threshold δ i and a direction indicator d i for each mask M i . The threshold δ i defines the cutoff frequency, and the binary variable d i determines the masking direction: masking frequencies less than δ i if d i = 1 , and greater than δ i if d i = 0 . The corresponding mask M i ∈ { 0 , 1 } L/ 2+1 is constructed as:

X"

Energy-wise Spectral

Partitioning

Frequency-wise Spectral

Masking

Pred

Decoder

Recon

Decoder

Recon

Head

Pred

Head

Transformer

Encoder

𝐞 $

𝐞 %

𝐞 &

Router

∑

𝑋$:"

Patchify & Projection

…

𝐾

𝑄

𝑉

Scale

Softmax

⨂

⨂

0

50

100

0

2

4

6

8

0

50

100

0

2

4

6

8

𝒅𝟏

= 𝟏

𝜹 𝟏

𝒅𝟏

= 𝟏

𝜹 𝟏

0

50

100

0

2

4

6

8

0

50

100

0

2

4

6

8

𝜹 𝟑

𝒅𝟑

= 𝟏

𝒅𝟑

= 𝟎

𝜹 𝟑

0

50

100

0

2

4

6

8

0

50

100

0

2

4

6

8

𝜹 𝟐

𝒅𝟐 = 𝟎

𝒅𝟐 = 𝟏

𝜹 𝟐

0

50

100

0

2

4

6

8

𝑍 789

0

50

100

0

2

4

6

8

𝑍 :89

Threshold

𝜹𝟏

Threshold

𝜹𝟐

Threshold

𝜹𝟑

c

𝑀789

⨀

𝑍 789

𝑀:89

⨀

𝑍 :89

Add & iFFT

𝑋=>?@

…

0

50

100

0

2

4

6

8

0

50

100

0

2

4

6

8

Threshold

𝝉

0

50

100

0

2

4

6

8

⊖

𝑋$:"

FFT

Input Time Series

𝑍 :89

𝑍

𝑍 789

c

Softmax & Linear

𝐛D

MLP & Reshape

𝐞 E D

Weight

…

c

𝐸 =GH

I

𝐵

𝐵

𝐸 =GH

K

𝐵

𝑋 L

$:"

𝑋 L

"M$:"MN O

Copy

Self-attention

①

②

③

④

MoP

Prompt Experts

EASD

Energy-aware pretraining

MoP tuning

Copy

Weight copying

Adaptive Router

MoPFormer

where α < L/ 2 + 1 , and f j denotes the j -th frequency point. Stacking all N m such masks for each energy branch yields two multi-band masking matrices: M Hec , M Lec ∈ { 0 , 1 } N m × ( L/ 2+1) . Each row represents a distinct frequency band mask, and each element m ij ∈ { 0 , 1 } indicates whether the j -th frequency point is masked under the i -th mask. Correspondingly, we expand Z Hec and Z Lec along a new dimension and replicate them N m times, resulting in Z Hec , Z Lec ∈ C N m × ( L/ 2+1) that are aligned with the masking matrices. We then apply element-wise multiplication between each replicated representation and its corresponding mask, and convert the aggregated result back to the time domain using the inverse Fast Fourier Transform (iFFT):

This masking strategy incorporates two core insights. First, M Hec and M Lec are separately generated without shared sampling parameters between the high- and low-energy branches. Such a decoupled design promotes spectral diversity and inherently mitigates energy bias. Second, by adaptively learning the sampling parameters, the masking process can be tailored to each specific time series dataset, thereby enhancing the model's ability to generalize across diverse data scenarios.

Then a standard patchify and projection step is applied. As illustrated in Figure 2, after obtaining the masked series X mask ∈ R N m × L , we divide it into P non-overlapping patches of equal length L p , where P = ⌊ L/L p ⌋ . Each patch is then projected into a patch token by a linear layer R L p → R D p , where D p is the dimension of Transformers. The positional embedding for the p -th patch x p is denoted by pe p . Accordingly, the position-encoded patch token b p is given by b p = Dropout( x p +pe p ) and B = { b 1 , b 2 , ..., b P } ∈ R N m × P × D p .

3.4 MoPFormer: Mixture-of-prompts Enabled Transformer

A significant challenge in pre-training on multi-domain time series datasets lies in their high degree of cross-domain heterogeneity, including differences in sampling resolution, noise levels, variable semantics, etc. To overcome this limitation, we introduce Mixture of Prompts (MoP) and incorporate

it into stacked Transformer blocks, which learns dataset-specific prompts to guide the model's adaptation to new data, thereby promoting generalization across diverse target domains.

Mixture of Prompts. To account for the diversity of multi-domain time series data, we randomly initialize a prompt-based expert pool E = { e 1 , e 2 , ..., e I } ∈ R I × D p , where each prompt-based expert is e i ∈ R D p with dimension D p . Applying all prompt-based experts indiscriminately to each patch token may dilute the effectiveness of prompt information, particularly under significant crossdomain distributional shifts in time series data. Inspired by the conditional computation mechanism in MoE [21,31], we introduce a token-dependent adaptive router that dynamically assigns relevant prompt-based experts conditioned on the input token. Concretely, the router computes expert-specific gating scores via a linear-softmax transformation over the input token, and performs soft merging of expert vectors through a weighted combination guided by these scores. To encode a shared structure between prompt key and value matrices [43], we further adopt a reparameterization strategy consisting of a two-layer MLP followed by a reshape operation, transforming the aggregated expert representations into structured key-value pairs. Given the masked patch token b p ∈ R D p and expert e i ∈ R D p for i = 1 , 2 , ..., I , the complete routing and merging process can be formalized as:

where s i,p ∈ R I denotes the gating scores over the expert pool, indicating the contribution of the i -th expert e i to the p -th input token b p . The mixed prompt ˜ e p ∈ R S × 2 × D p represents a structured prompt key-value pairs of b p distributed across S Transformer layers.

MoPFormer Block. Since self-attention (SA) can be interpreted as a specialized MoE [44], the mixture of prompts is integrated into the SA mechanism to inject new experts, which can be achieved via the modification of key and value matrices. Specifically, the mixed prompt E mix = { ˜ e 1 , ˜ e 2 , ..., ˜ e P } ∈ R S × 2 × N m × P × D p is first decomposed into E K mix ∈ R N m × P × D p and E V mix ∈ R N m × P × D p . In the SA layer, let Q , K , V denote the query, key, and value matrices respectively, the key and value matrices are then constructed by appending the prompt E K mix and E V mix to the input series B ∈ R N m × P × D p along the patch dimension P as follows:

We employ Transformer encoder [45] in both the encoder and decoder. Following SOTA time series FMs [18,21,31], we integrate recent advancements including RMSNorm [46], pre-normalization scheme [47], and SwiGLU [48] into our architecture. Each block at the s -th layer is defined as:

Both the encoder and decoder are constructed by stacking S MoPFormer blocks. SEMPO is then pre-trained using the above steps via reconstruction and prediction objectives. More details on the MoPFormer block during pre-training and inference are provided in Appendix B.

3.5 Model Training

To improve the model's generalization ability in downstream zero-shot and few-shot forecasting tasks, we adopt a two-stage training paradigm: energy-aware pre-training and MoP tuning, as discussed in the following:

Energy-aware Pre-training. During energy-aware pre-training, SEMPO is pre-trained on multidomain time series data via a self-supervised reconstruction objective, enabling it to capture transferable cross-domain commonalities. As shown in Figure 2 ( black arrows ), the series representation is fed into the reconstruction decoder without incorporating the mixture of prompts. The decoder generates the reconstructed series ˆ X ∈ R L through a linear reconstruction head. The training objective is the Mean Squared Error (MSE) between the input and its reconstruction:

MoP Tuning. During MoP tuning, SEMPO is jointly trained with supervised forecasting and self-supervised reconstruction to adapt to domain-specific variations by tuning both the mixture of prompts and the prediction heads, while keeping the Transformer backbone frozen, as shown in Figure 2 ( red arrows ). We adopt a multi-resolution forecasting strategy [21] with each prediction head implemented as a single linear layer. A composite loss, aggregating forecasting errors over different horizons, is computed to enhance model generalization. Let H = { H 1 , H 2 , ..., H R } denote the set of forecasting horizons, where H r is the number of future steps for the r -th resolution. Given the ground truth ˆ X L +1: L + H r for r = 1 , 2 , ..., R , the training loss is then defined as:

4 Experiments

Datasets. Leveraging large-scale publicly available time series collection UTSD [19], we curate a diverse subset covering multiple domains, totaling ∼ 83 million time points. We then set the pre-training training-validation split to 9:1, following [19]. For zero-/few-shot forecasting, we evaluate SEMPO on the Time-Series-Library (TSLib) benchmark [25], which includes seven datasets: ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic. Given that most real-world scenarios involve limited data, we draw on prior works [16,17,20,22] and set the lookback window length L = 512 on TSLib. We also include GIFT-Eval [49], the largest benchmark for zero-shot forecasting [50]. To avoid data leakage, we remove datasets overlapping with pre-training corpus and TSLib, and select 9 datasets from the GIFT-Eval benchmark spanning diverse domains. For GIFT-Eval, we adhere to its default evaluation protocol [49]. More details are given in Appendix C.1.

Baselines. We evaluate SEMPO against 17 latest open-sourced SOTA forecasting models, grouped in three categorized: (1) Pre-trained FMs on time series : TTM [23], Time-MoE [21], Timer [19], Moirai [18], Chronos [20], TimesFM [17], and Moment [16]; (2) LLM-based time series FMs : Time-LLM [30], GPT4TS [34], and S 2 IP-LLM [36]; (3) Task-specific models : iTransformer [11], DLinear [12], PatchTST [10], TimesNet [25], Stationary [9], FEDformer [8], and Autoformer [7]. Among these, the TTM family and small variants of Moirai and Chronos are lightweight pre-trained FMs on time series. More baseline details are provided in Appendix C.2.

Implementation Details. Using 83M pre-training datasets, the entire two-stage pre-training process takes 10 hours on 4 A6000-48G GPUs with BF32 precision and a batch size of 2,048. By default, we set layer_number S = 6 , head_number=16, latent_dimension D p = 256 , patch_size L p = 64 , prompt_number I = 128 , and mask_number N M = 4 . The hyperparameter analysis of I , N M , and D p , as well as the scaling analysis of the model are given in Appendix D. Regarding optimization, we use AdamW optimizer with hyperparameters: learning_rate=1e-3, weight_decay=0.1, β 1 = 0 . 9 , β 2 = 0 . 95 . A constant learning rate is used after a linear warmup for the first 10,000 steps. More implementation details are provided in Appendix C.3.

4.1 Zero-shot Forecasting

Setup. In this section, we evaluate the performance of SEMPO for zero-shot forecasting on two representative benchmarks: TSLib [25] and GIFT-Eval [49], using MAE (Mean Absolute Error) and MSE for TSLib, and SMAPE (Symmetric Mean Absolute Percentage Error) and NRMSE (Normalized Root Mean Squared Error) for GIFT-Eval.

Results. Table 1 reports the average zero-shot forecasting results across multiple horizons H ∈ { 96 , 192 , 336 , 720 } for Transformer-based FMs. Additional results comparing the MLP-based FMs TTM and different lookback window lengths are provided in Appendix D. Despite containing only 6.5M parameters and using only 83M pre-training data, SEMPO is consistently the best performer on the majority of datasets, yielding average reductions of 23.1 % in MSE and 10.8 % in MAE across all methods. In particular, compared to lightweight FMs with similar parameter scale, e.g. , Moirai S (14M, 27B), SEMPO exhibits markedly superior performance. Remarkably, even when benchmarked against large-scale time series FMs such as Chronos L (710M, 84B), TimeMoE L (453M, 309B) and Moment (385M, 1.13B),

Best

SEMPO achieves substantial MSE improvements of 19.9 % , 19.2 % , and 36.0 % , respectively; these improvements are even more pronounced when comparing the pre-training data scale: 83M of time points in SEMPO vs. 1.13B to 309B in these large FMs. Besides, Figure 3 presents a comparison between SEMPO and Chronos family on GIFT-Eval. SEMPO achieves the best performance in most cases (6 out of 9). This promising result also demonstrates that SEMPO learns a powerful knowledge base, which serves as a solid foundation for its remarkable capabilities in knowledge transfer.

4.2 Few-shot Forecasting

Setup. We fine-tune the pre-trained SEMPO and baseline models on the training splits of seven datasets from the TSLib benchmark. Following the few-shot protocols in [23,30,34], we consider two training scenarios using 5% and 10% of the training data, respectively. In both cases, only the prediction head and the MoP module are fine-tuned.

Results. Table 2 presents the average results under the 5% few-shot setting, with the corresponding 10% results provided in Appendix D. SEMPO establishes its superiority by outperforming 11 recent deep models across both 5% and 10% scenarios, achieving an average MSE reduction of 32.3 % . In particular, SEMPO significantly outperforms the LLM-based pre-trained models Time-LLM (11.4%), GPT4TS (15.9%), and S 2 IP-LLM (10.7%), and surpasses the lightweight MLP-based FM TTM by 4.6%. These results underscore the central contribution of SEMPO's pre-trained initialization,

4

3

4

3

4

3

0

20

40

60

80

100

0

1

2

3

4

5

Prediction

Ground Truth

Input

0

20

40

60

80

100

0

1

2

(a) Chronos

S

0

20

40

60

80

100

120Freq.

0

1

2

3

4

5

Prediction

Ground Truth

Input

0

20

40

60

80

100

120Freq.

0

1

2

(b) Moirai

L

0

20

40

60

80

100

0

1

2

3

4

5

Prediction

Ground Truth

Input

0

20

40

60

80

100

0

1

2

(c) SEMPO

which equips the model with strong inductive biases and significantly enhances its effectiveness in resource-constrained scenarios. Compared to the zero-shot results in Table 1, fine-tuning yields only marginal gains on ETTh1 and ETTh2, suggesting that adaptation is less effective when the target dataset is limited in scale or variability.

4.3 Model Analysis

Ablation Study. To verify the design of SEMPO, we perform a series of ablation studies under the zero-shot setting, each targeting a key model module. As shown in Table 3, replacing the default dual-branch spectral masking leads to consistent performance degradation: multi-band masking [22] (A.1) and random patch masking [10] (A.2) increase the average MSE from 0.350 to 0.399 and 0.416, respectively. The degradation in A.2 stems from its unstructured nature, which disrupts temporal dependencies and corrupts spectral coherence. While A.1 retains frequency-wise modeling, the lack of energy-based partitioning allows the dominant high-energy components to eclipse subtle yet informative low-energy signals, resulting in large information loss. Furthermore, we evaluate the impact of replacing the MoP module with two variants. The first (B.1) substitutes MoP with a sparse MoE [21] with three experts and one activated, totaling 8.5M parameters. Despite the larger model size, it underperforms SEMPO, highlighting the superior efficiency and expressiveness of the MoP design. In the second variant (B.2), we remove the adaptive router in MoP and replace it with conventional prefix tuning [51], which leads to further degradation and underscores the importance of data-dependent adaptive router in enhancing generalization across diverse time series patterns.

Analysis of Low-energy Components. We further investigate SEMPO's capability to model lowenergy components in temporal signals. Figure 4 shows the zero-shot prediction results in the spectral domain for the Electricity dataset with prediction length H = 336 . As shown in Figure 4, Chronos S and Moirai L -the small and large variants of the Chronos and Moirai families, respectively-tend to focus on dominant high-energy components. Chronos S largely neglects subtle yet persistent low-energy components. Moirai L exhibits non-negligible responses in low-energy regions, yet these do not consistently align with the ground-truth spectral peaks. This indicates that while Moirai L can partially capture low-energy signals, it still predominantly attends to high-energy components, resulting in less accurate learning of low-energy features such as important positional information. In contrast, SEMPO consistently attends to signals across the full energy spectrum, more effectively preventing low-energy components from being overwhelmed than both compact and large FMs. Besides, as shown in Appendix D, Figure 8, predictions on ETTh1 dataset across various prediction

Input

lengths demonstrate SEMPO's strong ability to identify and leverage low-energy regions, regardless of whether they reside in high-, mid-, or even low-frequency bands of the spectrum.

Visualization of MoP. To analyze MoP's ability to learn dataset-specific knowledge, we visualize the soft gating scores across different datasets in Figure 5. Within each dataset, different patch tokens attend to different subsets of prompt-based experts, with each expert specializing in distinct knowledge, indicating that MoP enables fine-grained, token-level composition of prompt information. When comparing across datasets, same-domain cases ( e.g. , ETTh1 and ETTm2) exhibit similar prompt routing patterns, while cross-domain datasets ( e.g. , Traffic and Weather) reveal distinct tokento-prompt mappings, reflecting domain-specific adaptation. The heterogeneous routing patterns indicate that the model dynamically adapts its representations to the specific characteristics of each dataset, supporting SEMPO's strong generalization and transferability.

Efficiency Analysis. To highlight the performance and efficiency benefits of SEMPO, we compare its inference costs with other FMs and assess their performance on ETTh1. The zero-shot results are shown in Figure 6. SEMPO, a lightweight general model with only 6.5M parameters and a fast inference time of 22s, significantly outperforms all other FMs. Compared to Moirai S , the smallest model among them, SEMPO achieves nearly 10 times faster inference speed while delivering superior prediction performance. In contrast to largescale FMs, SEMPO precisely fulfills the urgent need for general models in resource-constrained real-world scenarios, where both exceptional effi-

ETTh1

ETTm2

Weather

Traffic

Experts

Tokens

Tokens

Tokens

Tokens

01234567

01234567

01234567

0

20

40

60

80

100

120

6.5M

(22s)

46M

(1973s)

200M

(1879s)

710M

(14185s)

200M

(5574s)

385M

(986s)

311M

(494s)

453M

(518s)

67.4M

(255s)

14M

(205s)

91M

(284s)

113M

(413s)

ciency and superior accuracy are indispensable. More efficiency results are in Appendix D.

5 Conclusion

In this paper, we propose SEMPO, a novel lightweight foundation model for general time series forecasting, with relatively small data scale and model size while exhibiting strong generalization capabilities. SEMPO leverages two novel modules: energy-aware spectral decomposition for significantly improved data exploitation during pre-training, and mixture-of-prompts enabled Transformer for parameter-efficient model adaptation across different dataset and domains. Extensive experiments on two widely-used benchmarks spanning 16 datasets highlight the superiority of SEMPO in term of both effectiveness and efficiency for zero- and few-shot forecasting tasks. This work paves the way for future advancements in considering multivariate interactions and flexible distribution forecasting, which will empower the model structure with more time series capabilities.

Acknowledgments

This research is supported by the National Natural Science Foundation of China (62272048), A*STAR under its MTC YIRG Grant (M24N8c0103), the Ministry of Education, Singapore under its Tier-1 Academic Research Fund (24-SIS-SMU-008), and the Lee Kong Chian Fellowship.

References

Hui He, Qi Zhang, Simeng Bai, Kun Yi, and Zhendong Niu. CATN: cross attentive tree-aware network for multivariate time series forecasting. In AAAI , pages 4030-4038. AAAI Press, 2022.

1, 21

Massil Hihat, Stéphane Gaïffas, Guillaume Garrigos, and Simon Bussy. Online inventory problems: Beyond the i.i.d. setting with online convex optimization. In NeurIPS , 2023. 1

Kun Yi, Qi Zhang, Hui He, Kaize Shi, Liang Hu, Ning An, and Zhendong Niu. Deep coupling network for multivariate time series forecasting. ACM Trans. Inf. Syst. , 42(5):127:1-127:28, 2024. 1

Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nat. Mac. Intell. , 5(6):602-611, 2023. 1

Ed. McKenzie. General exponential smoothing and the equivalent arma process. Journal of Forecasting , 3(3):333-344, 1984. 1

David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. In NeurIPS , pages 6824-6834, 2019. 1

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS , pages 2241922430, 2021. 1, 7, 21, 24

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 27268-27286. PMLR, 2022. 1, 7, 21, 24

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. In NeurIPS , 2022. 1, 7, 21, 24

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR , 2023. 1, 4, 7, 9, 21, 24

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR , 2024. 1, 7, 21, 24

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI , pages 11121-11128. AAAI Press, 2023. 1, 7, 21, 24

Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In KDD , pages 459-469. ACM, 2023. 1, 21

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. In NeurIPS , 2023. 1, 21

Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting. In ICLR , 2024. 1, 21

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MOMENT: A family of open time-series foundation models. In ICML , 2024. 2, 3, 4, 7, 23, 24

Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. In ICML , 2024. 2, 3, 4, 7, 23, 24

Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In ICML , 2024. 2, 3, 4, 6, 7, 21, 22, 23, 24

Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In ICML , 2024. 2, 3, 7, 22, 23, 24, 25

Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Türkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda-Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang. Chronos: Learning the language of time series. CoRR , abs/2403.07815, 2024. 2, 3, 4, 7, 23, 24

Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, and Ming Jin. Timemoe: Billion-scale time series foundation models with mixture of experts. In The Thirteenth International Conference on Learning Representations , 2025. 2, 3, 4, 6, 7, 9, 21, 22, 23, 24, 25, 31

Yihang Wang, Yuying Qiu, Peng Chen, Kai Zhao, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, and Chenjuan Guo. ROSE: register assisted general time series forecasting with decomposed frequency learning. CoRR , abs/2405.17478, 2024. 2, 3, 4, 7, 9, 21

Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam Nguyen, Wesley M. Gifford, Chandra Reddy, and Jayant Kalagnanam. Tiny time mixers (ttms): Fast pre-trained models for enhanced zero/few-shot forecasting of multivariate time series. In NeurIPS , 2024. 2, 3, 7, 8, 23, 24, 25

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , pages 11106-11115. AAAI Press, 2021. 2

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR , 2023. 2, 7, 8, 21, 22, 24, 28, 32, 33, 34

Xihao Piao, Zheng Chen, Taichi Murayama, Yasuko Matsubara, and Yasushi Sakurai. Fredformer: Frequency debiased transformer for time series forecasting. In KDD , pages 2400-2410. ACM, 2024. 2, 4

Kun Yi, Jingru Fei, Qi Zhang, Hui He, Shufeng Hao, Defu Lian, and Wei Fan. Filternet: Harnessing frequency filters for time series forecasting. In NeurIPS , 2024. 2, 4, 28

Jingru Fei, Kun Yi, Wei Fan, Qi Zhang, and Zhendong Niu. Amplifier: Bringing attention to neglected low-energy components in time series forecasting. In AAAI , pages 11645-11653. AAAI Press, 2025. 2

Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Bilos, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag-llama: Towards foundation models for time series forecasting. CoRR , abs/2310.08278, 2023. 2, 3

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-llm: Time series forecasting by reprogramming large language models. In ICLR , 2024. 2, 3, 7, 8, 23, 24

Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. CoRR , abs/2410.10469, 2024. 2, 3, 6, 21

Hao Xue and Flora D. Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Trans. Knowl. Data Eng. , 36(11):6851-6864, 2024. 3

Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In NeurIPS , 2023. 3

Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: Power general time series analysis by pretrained LM. In NeurIPS , 2023. 3, 7, 8, 23, 24

Defu Cao, Furong Jia, Sercan Ö. Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. TEMPO: prompt-based generative pre-trained transformer for time series forecasting. In ICLR , 2024. 3

Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. S2IP-LLM: semantic space informed prompt learning with LLM for time series forecasting. In ICML , 2024. 3, 7, 23, 24

Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Autoregressive time series forecasters via large language models. In NeurIPS , 2024. 3

Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. CoRR , abs/2312.16862, 2023. 3

Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In NAACL-HLT , pages 2339-2352. Association for Computational Linguistics, 2021. 3

Guoqi Yu, Yaoming Li, Juncheng Wang, Xiaoyu Guo, Angelica I. Avilés-Rivero, Tong Yang, and Shujun Wang. Refocus: Reinforcing mid-frequency and key-frequency modeling for multivariate time series forecasting. CoRR , abs/2502.16890, 2025. 4, 21

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In ICLR , 2022. 4, 21

Qichao Shentu, Beibu Li, Kai Zhao, Yang Shu, Zhongwen Rao, Lujia Pan, Bin Yang, and Chenjuan Guo. Towards a general time series anomaly detector with adaptive bottlenecks and dual adversarial decoders. In The Thirteenth International Conference on Learning Representations , 2025. 4

Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth International Conference on Learning Representations , 2025. 6

Minh Le, An Nguyen The, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Ngo Van, and Nhat Ho. Mixture of experts meets prompt-based continual learning. In NeurIPS , 2024. 6, 21

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017. 6, 21

Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurIPS , pages 12360-12371, 2019. 6, 21

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In ICML , volume 119 of Proceedings of Machine Learning Research , pages 10524-10533. PMLR, 2020. 6, 21

Noam Shazeer. GLU variants improve transformer. CoRR , abs/2002.05202, 2020. 6, 21

Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Gift-eval: A benchmark for general time series forecasting model evaluation. CoRR , abs/2410.10393, 2024. 7, 8, 22, 23, 29

Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Sundial: A family of highly capable time series foundation models. CoRR , abs/2502.00816, 2025. 7

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/IJCNLP (1) , pages 4582-4597. Association for Computational Linguistics, 2021. 9, 21

Bryan Lim and Stefan Zohren. Time series forecasting with deep learning: A survey. CoRR , abs/2004.13408, 2020. 21

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104. ACM, 2018. 21

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting , 36(3):1181-1191, 2020. 21

Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NeurIPS , pages 4838-4847, 2019. 21

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. In NeurIPS , 2022. 21, 22

Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In NeurIPS Datasets and Benchmarks , 2021. 22

Hoang Anh Dau, Anthony J. Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn J. Keogh. The UCR time series archive. IEEE CAA J. Autom. Sinica , 6(6):1293-1305, 2019. 22

Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I. Webb. Time series extrinsic regression. Data Min. Knowl. Discov. , 35(3):1032-1060, 2021. 22

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state our contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our work in Conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We included the implementation details in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The source data and code are available at https://github.com/mala-lab/ SEMPO .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The average improvement of SEMPO over all baseline models is statistically significant at the confidence of 95%.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Our model is implemented in Pytorch 2.1.2 with Python 3.10 and all the experiments are run on 4 A6000-48G GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our use of public datasets complies with NeurlPS Ethics Guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our model is not high-risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cited the data sources.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not applicable (no new assets introduced in the paper).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects or crowdsourcing was used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable (no human subjects research).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: LLMs were only used for grammar checks (Grammarly). No impact onmethodology.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Further Related Work

Task-specific Deep Models for Time Series. In recent years, deep learning has revolutionized time series forecasting by enabling models to capture complex temporal dynamics and nonlinear dependencies [52]. Early efforts explored various neural architectures for modeling temporal patterns, such as Recurrent Neural Networks (RNNs) [1,53,54], Convolution Neural Networks (CNNs) [25, 55, 56], etc. Thereafter, the limited ability of these models to capture long-range dependencies has motivated a shift towards Transformer-based [7-11] and MLP-based [12-15] architectures, which have achieved SOTA performance across a wide range of forecasting tasks. While these models demonstrate competitive in-distribution performance, they typically require meticulous hyperparameter tuning across different datasets and exhibit limited flexibility and generalizability when applied to out-of-distribution tasks, particularly under few-shot or zero-shot scenarios. Time series foundation models help address this challenge.

B More Details about MoPFormer Block

Recent insights from vision research [44] reveal that self-attention (SA) can be interpreted as a specialized MoE architecture, characterized by linear experts and quadratic gating score functions. Building on this connection, we propose that applying a mixture of prompts (MoP)-in a manner similar to prefix-tuning [51]-within pre-trained Transformer backbones serves as a flexible and effective mechanism for injecting new experts. These newly added experts work in conjunction with the pre-trained experts, facilitating efficient adaptation of the model to new and unseen datasets.

file:/

EASD & Patchify & Projection

Reconstruction Decoder

Reconstruction Head

Transformer Encoder

𝑋":$

𝑋 %

":$

EASD & Patchify & Projection

Pred

Decoder

Recon

Decoder

Recon

Head

Pred

Head

Transformer

Encoder

𝑋":$

𝑋 %

":$

𝑋 %

$&":$&' (

Copy

MoP

EASD & Patchify & Projection

Prediction Decoder

Prediction Head

Transformer

Encoder

𝑋":$

𝑋 %

$&":$&' (

MoP

𝐵

𝐵 *

𝐸,-.

𝐸 *

,-.

𝐵

𝐵

𝐸 *

,-.

𝐵 *

𝐸 *

,-.

𝐸,-.

𝐵

𝐵

𝐵 *

Learnable

Frozen

2025/5/19 15:47

flames.svg

2025/5/19 15:47

flames.svg

file:/

Users/hui/Downloads/flames.svg

1/1

2025/5/19 15:47

flames.svg

Users/hui/Downloads/flames.svg

1/1

2025/5/19 15:47

flames.svg

2025/5/19 15:47

flames.svg

2025/5/19 15:47

flames.svg

2025/5/19 15:47

flames.svg

file:/

Users/hui/Downloads/flames.svg

1/1

2025/5/19 15:59

雪花

.svg

2025/5/19 15:59

雪花

.svg

file:/ /Users/hui/Downloads/

雪花

.svg

1/1

2025/5/19 15:59

雪花

.svg

file:/ /Users/hui/Downloads/

雪花

.svg

1/1

2025/5/19 15:59

雪花

.svg

2025/5/19 15:59

雪花

.svg

2025/5/19 15:59

雪花

.svg

file:/ /Users/hui/Downloads/

雪花

.svg

1/1

2025/5/19 15:59

雪花

.svg

file:/ /Users/hui/Downloads/

雪花

.svg

1/1

2025/5/19 15:59

雪花

.svg

file:/ /Users/hui/Downloads/

雪花

.svg

1/1

2025/5/19 15:59

雪花

.svg

2025/5/19 15:59

雪花

.svg

2025/5/19 15:59

雪花

.svg

file:/

Users/hui/Downloads/flames.svg

1/1

file:/ Users/hui/Downloads/flames.svg 1/1 file:/ /Users/hui/Downloads/ 雪花 .svg 1/1 file:/ /Users/hui/Downloads/ 雪花 .svg 1/1 file:/ /Users/hui/Downloads/ 雪花 .svg 1/1 As shown in Figure 7 Left and Middle above, MoP is introduced during the MoP tuning stage. Both the encoder and decoder utilize the Transformer [45] encoder structure, as adopted in [22]. They are constructed by stacking S MoPFormer blocks. Each MoPFormer block integrates recent advancements from SOTA time series foundation models [18,21,31]. Specifically, all LayerNorm layers are replaced with RMSNorm [46] and applied pre-normalization scheme [47] to mitigate training instability in deep architectures. The standard non-linearity in FFN layers is replaced with SwiGLU [48], which improves expressiveness and stability while preserving parameter efficiency. During the tuning stage, given a univariate time series X ∈ R L , instance normalization is first applied with zero mean and unit standard deviation to mitigate distribution shifts [41]. Recent work [40] has theoretically shown that normalization scales the absolute spectral energy by the variance but does not affect its relative distribution. Then, after the proposed EASD module and the Patchify & Projection module, the encoder processes the masked series representation B ∈ R N m × P × D p along with the mixed prompt E mix ∈ R S × 2 × N m × P × D p . The decoder first aggregates information over the N m dimension via an averaging operator and then takes the aggregated series ˜ B ∈ R P × D p , along with its corresponding mixed prompt ˜ E mix ∈ R S × 2 × P × D p as input.

In downstream fine-tuning, the model is trained on the target dataset using the same joint training strategy as in the MoP tuning stage, in a few-shot setting. For zero-shot inference, the reconstruction

decoder and head are removed, and all modules are frozen. A greedy scheduling algorithm [21] is then applied to generate predictions for arbitrary target lengths, as shown in Figure 7 Right .

C Experimental Details

C.1 Datasets

Pretraining Datasets. Large-scale publicly available time series collections, e.g. , Time-300B [21], UTSD [19], and LOTSA [18], have resolved data availability issues for building general time series forecasting models. Leveraging UTSD [19], we curate a diverse subset covering multiple domains, totaling ∼ 83 million time points. We then set the pre-training training-validation split to 9:1, following [19]. Below, we outline the key properties of the datasets, including their domain, resolution, number of time points, total file size, Augmented Dickey-Fuller (ADF) test statistics, forecastability, and data source.

Evaluation Datasets. For zero- and few-shot forecasting, we evaluate SEMPO on the widelyused Time-Series-Library (TSLib) benchmark [25], which includes seven datasets: ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic . In detail, the ETT 3 (Electricity Transformer Temperature) dataset is collected from electric power industry and contains 2 years of data from two separate counties in China. It includes temperature measurements and power load features. The dataset is further divided into subsets for different time granularities: {ETTh1, ETTh2} for 1-hour intervals and {ETTm1, ETTm2} for 15-minute intervals. The Weather 4 dataset contains 21 meteorological indicators such as air temperature and humidity, is collected every 10 minutes from the Weather Station of the Max Planck Institute for Biogeochemistry in 2020. The Electricity 5 dataset includes hourly electricity consumption data from 321 clients, measured in kWh. Due to missing data, the dataset was adjusted to an hourly granularity over two years. The Traffic 6 dataset collects 48 months (2015-2016) of hourly data from the California Department of Transportation, which tracks road occupancy rates on San Francisco Bay Area freeways. The details about these datasets are summarized in Table 5.

We also include GIFT-Eval [49] for evaluation. This comprehensive benchmark is designed to evaluate general-purpose time series forecasting models, particularly foundation models. It comprises 23 evaluation datasets, comprising approximately 144,000 time series and 177 million data points, spanning seven domains-economics/finance, energy, healthcare, nature, sales, transportation, and Web/CloudOps-and covers ten sampling frequencies, ranging from seconds to yearly. The benchmark supports both univariate (15 datasets) and multivariate (8 datasets) forecasting tasks with short-, medium, and long-term prediction lengths. GIFT-Eval also provides a rich set of statistical

features ( e.g. , trend, seasonality, entropy, stability) for in-depth data analysis. To avoid data leakage, we remove datasets overlapping with pre-training corpus and TSLib, and select 9 datasets including m_dense, loop_seattle, sz_taxi, solar, bizitobs_application, bizitobs_12c, bitbrains_service, car_parts, and jena_weather . For GIFT-Eval, we adhere to its default evaluation protocol [49].

C.2 Baselines

Pre-trained FMs on time series. The models include TTM [23], Time-MoE [21], Timer [19], Moirai [18], Chronos [20], TimesFM [17], and Moment [16], detailed as follows. A comparison between these pre-trained FMs is presented in Table 6. Notably, although SEMPO is evaluated with fixed context lengths in this paper-similar to TTM [23]-its Transformer-based architecture is inherently insensitive to context length variations, as it can accommodate different context lengths through preprocessing techniques such as padding [20].

TTM [23] is a MLP-based compact model tailored for zero- and few-shot multivariate time series forecasting. The TTM family includes three variants: TTM B (1M), TTM E (4M), and TTM A (5M). The official checkpoint is available at https://huggingface.co/ibm-granite/ granite-timeseries-ttm-r2 .

Time-MoE [21] introduces a MoE structure into the Transformer framework, enabling sparse activation where only a subset of experts is activated per input token. The Time-MoE family includes three variants: Time-MoE base (113M), Time-MoE large (453M), and Time-MoE ultra (2.4B). The official checkpoint is available at https://huggingface.co/Maple728 .

Timer [19] adopts the decoder-only Transformer architecture and is pre-trained with a next token prediction objective on massive time series corpora. The official checkpoint is available at https://huggingface.co/collections/thuml/ time-series-foundation-models-67c80ace73299239b651d954 .

Moirai [18] is a universal forecasting model designed to handle diverse time series forecasting tasks in a zero-shot manner. The Moirai family includes three variants: Moirai small (14M), Moirai base (91M), and Moirai large (311M). The official checkpoint is available at https://huggingface. co/collections/Salesforce/moirai-r-models-65c8d3a94c51428c300e0742 .

Chronos [20] is designed for zero-shot and in-domain forecasting, and demonstrates that by learning the 'language' of time series. The Chronos family includes three variants: Chronos small (46M), Chronos base (200M), and Chronos large (710M). The official checkpoint is available at https://huggingface.co/collections/amazon/ chronos-models-and-datasets-65f1791d630a8d57cb718444 .

TimesFM [17] introduces input patching to represent sequences efficiently, leverages longer output patches to reduce autoregressive steps, and employs random masking to learn from variable-length contexts. The official checkpoint is available at https://huggingface.co/google/timesfm-1. 0-200m .

Moment [16] leverages a patch-based Transformer encoder trained with masked time series modeling to learn robust representations across diverse domains. The Moment family includes three variants: Moment small (40M), Moment base (125M), and Moment large (385M). The official checkpoint is available at https://huggingface.co/AutonLab .

LLM-based time series FMs. The models include Time-LLM [30], GPT4TS [34], and S 2 IPLLM [36], detailed as follows:

Time-LLM [30] is a reprogramming framework that adapts off-the-shelf LLMs to perform time series forecasting. Time-LLM (LLaMA full, 3.4B), Time-LLM (LLaMA-8, 976M), Time-LLM (GPT-2, 124M). The official implementation is available at https://github.com/KimMeen/Time-LLM .

GPT4TS [34] fine-tunes the input embedding, positional embeddings, layer normalization, and output layer to adapt to time series data. It has about 4M learnable parameters (total 87M). The official implementation is available at https://github.com/DAMO-DI-ML/One_Fits_All .

S 2 IP-LLM [36] aims to enhance time series forecasting by aligning time series embeddings with the semantic space of a pre-trained large language model. The total parameters are 117M. The official implementation is available at https://github.com/panzijie825/S2IP-LLM .

Task-specific Models. The models include iTransformer [11], DLinear [12], PatchTST [10], TimesNet [25], Stationary [9], FEDformer [8], and Autoformer [7], detailed as follows:

iTransformer [11] inverts the conventional design by embedding the entire time series of each variate as a token, enabling attention mechanisms to directly model multivariate correlations. The official implementation is available at https://github.com/thuml/iTransformer .

DLinear [12] directly maps the historical input sequence to future outputs via a temporal linear projection, without modeling inter-variable dependencies. The official implementation is available at https://github.com/cure-lab/LTSF-Linear .

PatchTST [10] segments each univariate series into subseries-level patches and allows each time series channel to be processed separately using a shared Transformer backbone. The official implementation is available at https://github.com/yuqinie98/PatchTST .

TimesNet [25] introduces temporal 2D-variation modeling by transforming 1D time series into structured 2D tensors based on automatically discovered periods. The official implementation is available at https://github.com/thuml/TimesNet .

Stationary [9] integrates a series stationarization module to normalize input sequences and the destationary attention mechanism by re-incorporating statistical properties into the attention computation. The official implementation is available at https://github.com/thuml/Nonstationary_ Transformers .

FEDformer [8] combines seasonal-trend decomposition with frequency-domain modeling to enhance the global understanding of temporal dynamics. The official implementation is available at https: //github.com/MAZiqing/FEDformer .

Autoformer [7] replaces point-wise self-attention with a series-wise auto-correlation operation that captures periodic dependencies across sub-series using fast Fourier transform. The official implementation is available at https://github.com/thuml/Autoformer .

C.3 Implementation Details

SEMPO and all other baselines are conducted on 4 × NVIDIA A6000-48G GPUs. The feedforward dimension d_ff is set to 256, and head dropout is set to 0.2. During energy-aware pre-training, the model is trained for 10 epochs, with a batch size of 2,048. MoP tuning is performed for 20 epochs, with the same batch size of 2,048. For few-shot and zero-shot settings, the batch size is reduced to 32. Early stopping is applied with a patience of 6. All experimental results are obtained by averaging

3

2

Prediction

Ground Truth

3

2

Input

Prediction

Ground Truth

10

0

20

40

60

80

0

1

2

3

Prediction

Ground Truth

Input

0

20

40

60

80

0

1

0

20

40

60

80

0

1

2

3

Prediction

Ground Truth

Input

Mid-frequency

(a)

512

→

192

0

50

100

150

0

1

2

3

Prediction

Ground Truth

Input

0

20

40

60

80

100

0

1

Low-frequency

0

20

40

60

80

0

1

2

3

Prediction

Ground Truth

Input

(b)

512

→

336

0

50

100

150

0

5

10

Prediction

Ground Truth

Input

0

50

100

150

0

5

High-frequency

0

20

40

60

80

0

1

2

3

Prediction

Ground Truth

Input

(c)

512

→

720

1

2

3

4

5

6

N M

0.34

0.36

0.38

0.4

0.42

0.44

MSE

512->96

512->192

512->336

512->720

(a)

N

M

on ETTh1/h2

32

64

96

128

192

256

I

0.2

0.25

0.3

0.35

MSE

512->96

512->192

512->336

512->720

(b)

I

64

128

256

512

768

1024

D p

0.35

0.4

0.45

0.5

MSE

512->96

512->192

512->336

512->720

64

128

256

512

768

1024

D p

0.35

0.4

0.45

0.5

MSE

512->96

512->192

512->336

512->720

(c)

D

p

on ETTm1/m2

over three runs with different random seeds. In deep time series forecasting, performance variance across seeds is typically small, and following prior work [19, 21, 23], we report only the mean values.

D Additional Results

D.1 Analysis of Low-energy Components

We further investigate SEMPO's capability to model low-energy components in temporal signals across diverse prediction lengths. Experiments are conducted on the ETTh1 dataset, with the results presented in Figure 8. These figures demonstrate that SEMPO exhibits full energy spectrum learning capability, applicable to both short- and long-term forecasting. Notably, SEMPO effectively identifies and leverages low-energy regions, regardless of whether they reside in high-, mid-, or even lowfrequency bands of the spectrum.

D.2 Sensitivity Analysis

The parameters mask_number ( N M in Equation 2), prompt_number ( I in Equation 4), and la-tent_dimension ( D p in Patchify & Projection module) play a pivotal role in shaping the functionality of EASD and MoPFormer. In this section, we conduct comprehensive experiments on the ETTh1/h2, ETTm1/m2, and Weather datasets to investigate the impact of these parameters on forecasting performance. We explore a range of values from the sets {1, 2, 3, 4, 5, 6}, {32, 64, 96, 128, 192, 256}, and {64, 128, 256, 512, 768, 1024} for N M , I , and D p , respectively, while strictly controlling for lookback window and prediction length. Specifically, we examine the impact of four different prediction lengths, i.e. , 512 → {96, 192, 336, 720}, and present the results in Figure 9. Figure 9a illustrates that, as the number of masked series increases, the forecasting performance initially improves due to the capture of more decomposed frequency patterns in the two energy branches, but eventually decreases due to information redundancy. Figure 9b shows that increasing the prompt number results in minimal changes in forecasting performance, with the optimal result achieved when the prompt number is set to 128. Insufficient prompts fail to capture enough domain information, while too many introduce redundancy, hindering the model's efficient adaptation to new and unseen datasets. Figure 9c demonstrates that the forecasting performance fluctuates slightly before reaching 256, but it drops sharply thereafter. For small datasets such as ETTm1/m2, a compact latent dimension suffices to capture the data's underlying characteristics; therefore, we opt for a dimension of 256 to strike an optimal balance between model capacity and expressive power.

Input

Prediction

Ground Truth

Input

D.3 Scaling Analysis

To investigate the scalability of SEMPO, we scaled both model size and dataset size and evaluated its predictive performance on two ETTh datasets. As shown in Figure 10 Left , expanding the dataset size consistently improves model performance, in line with the scaling law. As the dataset size increases, the performance improvement plateaus, indicating diminishing returns from further expansion. However, the performance remains consistently below the MSE of the SOTA FMs Moirai S and Moirai B with a dataset size of 27B. Regarding model size, as seen in Figure 10 Right , SEMPO follows a similar trend. Performance declines in the later stages,

Moirai

B

(0.397, 27B)

Moirai

S

(0.400, 27B)

0.420

(37M)

0.406

(60M)

0.376

(83M)

0.387

(141M)

0.388

(164M)

0.376

(6.5M)

0.386

(5.5M)

0.384

(7.5M)

0.389

(4.5M)

Moirai

S

(0.400, 14M)

Moirai

B

(0.397, 91M)

likely due to a mismatch between model capacity and dataset size. Despite this, SEMPO continues to surpass Moirai S and Moirai B with 14M and 91M parameters, respectively.

D.4 Efficiency Analysis

Since many Transformer-based FMs process data in a purely univariate fashion, the dataset size and inference costs are directly proportional. To further highlight the performance and efficiency benefits, in addition to the smaller ETTh1 dataset (see Figure 6), we also compare the inference costs of SEMPO and other FMs on the Weather and ETTm2 datasets. All experiments are conducted using a batch size of 32 on a single A6000 48G GPU, as higher batch sizes often lead to out-of-memory (OOM) errors in many FMs, such as Chronos L . The zero-shot results are shown in Figure 11, where SEMPO significantly outperforms all other FMs in both efficiency and accuracy.

6.5M

(67s)

67.4M

(225s)

385M

(2929s)

14M

(517s)

91M

(831s)

311M

(1704s)

200M

(5578s)

46M

(5857s)

200M

(16549s)

710M

(42110s)

113M

(378s)

453M

(584s)

(a) ETTm2

6.5M

(196s)

14M

(1243s)

67.4M

(455s)

113M

(1020s)

91M

(2162s)

311M

(4525s)

710M

(156511s)

385M

(8540s)

46M

(21773s)

200M

(61509s)

453M

(1575s)

(b) Weather

Beyond the comparisons among Transformer-based FMs, we have additionally included an efficiency comparison with TTM on the ETTh1 and ETTm2 datasets, summarized in Table 7 right. While SEMPOisless competitive in parameter size and inference time due to its more complex Transformerbased architecture, it uses only 83M pre-training data (vs. TTM's 1B) and still achieves lower MSE on both datasets. Moreover, SEMPO outperforms TTM in 31 out of 50 cases across five diverse datasets (see Table 10 in Appendix D.6), with par-

ticularly substantial advantages on large-scale datasets such as Traffic and Electricity, demonstrating stronger knowledge transfer capability and superior sample efficiency.

D.5 Visualization of Predictions

To provide a clearer comparison between SEMPO and other foundation models, we present prediction case studies on the ETTh2 dataset, as illustrated in Figure 12. We select several representative time series FMs as baselines, including Moirai L , Timer, Time-MoE L , Time-MoE B , and Chronos L . SEMPO consistently delivers more accurate forecasting of future series variations compared to these diverse FMs, underscoring its superior predictive capability.

0

20

40

60

80

100

120

140

160

Time

-0.5

0

0.5

1

1.5

Values

Input

Ground Truth

Prediction

(a) SEMPO

0

20

40

60

80

100

120

140

160

Time

-1

0

1

2

Values

Input

Ground Truth

Prediction

(b) Moirai

L

0

20

40

60

80

100

120

140

160

Time

-0.5

0

0.5

1

1.5

Values

Input

Ground Truth

Prediction

(c) Timer

0

20

40

60

80

100

120

140

160

Time

-0.5

0

0.5

1

1.5

Values

Input

Ground Truth

Prediction

(d) Time-MoE

L

0

20

40

60

80

100

120

140

160

Time

-0.5

0

0.5

1

1.5

Values

Input

Ground Truth

Prediction

(e) Time-MoE

B

0

20

40

60

80

100

120

140

160

Time

-1

-0.5

0

0.5

1

1.5

Values

Input

Ground Truth

Prediction

(f) Chronos

L

D.6 Full Results Tables

Here, we present the comprehensive versions of the tables in the main text. These versions include the full results for multiple prediction lengths across all datasets. To save space in the main paper, some of these results are averaged across different prediction lengths.

Full Results for Zero-shot Forecasting. Table 8 and Table 9 report the zero-shot results on the TSLib and GIFT-Eval benchmarks, respectively. It is observed that with only 6.5M parameters and using 83M pre-training data, SEMPO consistently outperforms other time series FMs on the

majority of datasets. Given that the GIFT-Eval benchmark includes short- ( i.e. , H = L ), medium-( i.e. , H = 10 × L ), and long-term ( i.e. , H = 15 × L ) prediction lengths, the results in Table 9 further demonstrate SEMPO's strong adaptability to a wide range of prediction horizons, achieving outstanding forecasting performance on most datasets.

Full Results for Zero-shot Comparisons with TTM. Since Table 8 provides the comparison with Transformer-based time series FMs, Table 10 further reports results the comparison to the lightweight MLP-based FM, TTM. Existing research [27] has experimentally demonstrated that simpler structures, such as MLPs, may be more suitable for smaller datasets, while larger datasets, due to their complex relationships, require more contextual structures, such as Transformers. To ensure a fair comparison, we exclude two small datasets, ETTh1 and ETTm1, from the TSLib benchmark to maintain a balanced dataset ratio for evaluation. The results are shown in Table 10. Despite TTM having only 1M parameters, SEMPO outperforms in the majority of cases (31 out of 50) utilizing only 83M data, showcasing its better capabilities in knowledge transfer than TTM. Notably, SEMPO demonstrates substantial advantages on large datasets, such as Traffic and Electricity, which is consistent with our earlier analysis.

Full Results for Zero-shot Comparisons with Chronos-Bolt and TimesFM (500M). Since new versions of Chronos and TimesFM are available, we have added zero-shot results for the latest Chronos-Bolt and TimesFM (500M), summarized in Table 11 for an up-to-date comparison. Across

five datasets, Chronos-Bolt achieves the best performance in 20 cases, SEMPO in 16, and TimesFM (500M) in 7. We can observe that SEMPO significantly outperforms TimesFM (500M). It should be noted that, even with the small version of Chronos-Bolt ( i.e. , Chronos-Bolt S ), it is trained on nearly 100B time points with 48M parameters. In contrast, SEMPO is trained on only 83M points with just 6.5M parameters, yet it still ranks in second in terms of the overall performance. This compelling data efficiency-performance balance highlights SEMPO's promise as an effective lightweight FM for resource-constrained environments, to which the other FMs such as Chronos-Bolt and TimesFM (500M) are infeasible.

Full Results for Different Lookback Window Lengths. Given that existing time series FMs, such as Time-MoE and TTM, utilize larger lookback window lengths, we additionally pre-train two variants: SEMPO-Enhanced (SEMPO E ) with L = 1024 , L p = 128 (parameters: 7.3M), and SEMPOAdvanced (SEMPO A ) with L = 1536 , L p = 128 (parameters: 9.9M). The default configuration, with L = 512 , L p = 64 (parameters: 6.5M), is denoted as SEMPO B . To avoid data leakage, we exclude

the Traffic and Electricity datasets, as they are included in the pre-training data for Time-MoE. The zero-shot comparison results with Time-MoE family are shown in Table 12. From the table, it is evident that, despite having significantly fewer parameters and less pre-training data, the variants of SEMPO achieve comparable forecasting performance to the large-scale Time-MoE variants across these five datasets.

Full Results for Full-shot Experiment. To ensure fairness and enhance the comprehensiveness of our evaluation, we have added in-distribution experiments under the standard supervised setting adopted by Time-MoE [21]. Specifically, we compare SEMPO against LLM-based time series FMs (S 2 IP-LLM and GPT4TS) as well as strong task-specific models (iTransformer, DLinear, PatchTST and TimesNet). Results on four datasets of different scales are presented in Table 13. The results show that SEMPO consistently outperforms six advanced time series models, demonstrating that its pre-training framework effectively equips the model with strong inductive biases for downstream adaptation, despite its lightweight design.

Full Results for Few-shot Experiment. Table 14 and Table 15 show the 5% and 10% few-shot results for all prediction lengths on the TSLib benchmark.

Full Results for Ablation Study. Table 16 presents a comprehensive analysis of the impact of EASD and MoPFormer within SEMPO. As shown in the table, Group A replaces the dual-branch spectral masking in EASD with multi-band masking and random patch masking. Group B replaces MoP with Sparse MoE and prefix tuning, where sparse MoE is designed in two versions: 8 experts with activated and 3 experts with 1 activated. EASD, by performing frequency decomposition across two energy branches, helps SEMPO capture low-energy components that are easily overlooked by existing methods such as the alternative methods, leading to more accurate forecasts. The incorporation of the mixture of prompts significantly benefits SEMPO by effectively and efficiently decoupling the weights across diverse domains.

Models

SEMPO

TTM

Time-LLM

GPT4TS

S

2

IP-LLM iTransformer

DLinear

PatchTST

TimesNet

Stationary

Metrics

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE

MAE

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ETTh1

96

0.383

0.408

0.362

0.389

0.483

0.464

0.543

0.506

0.475

0.458

0.808

0.610

0.547

0.503

0.557

0.519

0.892

0.625

0.952

0.650

0.593

0.529

0.681

0.570

192

0.409

0.424

0.386

0.408

0.629

0.540

0.748

0.580

0.693

0.562

0.928

0.658

0.720

0.604

0.711

0.570

0.940

0.665

0.943

0.645

0.652

0.563

0.725

0.602

336

0.425

0.439

0.400

0.421

0.768

0.626

0.754

0.595

0.760

0.618

1.475

0.861

0.984

0.727

0.816

0.619

0.945

0.653

0.935

0.644

0.731

0.594

0.761

0.624

720

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

Avg

0.406

0.423

0.382

0.405

0.627

0.543

0.681

0.560

0.642

0.546

1.070

0.710

0.750

0.611

0.694

0.569

0.925

0.647

0.943

0.646

0.658

0.562

0.722

0.598

ETTh2

96

0.272

0.338

0.272

0.331

0.336

0.397

0.376

0.421

0.323

0.385

0.397

0.427

0.442

0.456

0.401

0.421

0.409

0.420

0.408

0.423

0.390

0.424

0.428

0.468

192

0.333

0.378

0.346

0.383

0.406

0.425

0.418

0.441

0.403

0.420

0.438

0.445

0.617

0.542

0.452

0.455

0.483

0.464

0.497

0.468

0.457

0.465

0.496

0.504

336

0.355

0.402

0.383

0.415

0.405

0.432

0.408

0.439

0.415

0.440

0.631

0.553

1.424

0.849

0.464

0.469

0.499

0.479

0.507

0.481

0.477

0.483

0.486

0.496

720

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

Avg

0.320

0.372

0.333

0.376

0.382

0.418

0.400

0.433

0.380

0.415

0.488

0.475

0.694

0.577

0.827

0.615

0.439

0.448

0.470

0.489

0.463

0.454

0.441

0.457

ETTm1

96

0.307

0.354

0.341

0.359

0.316

0.377

0.386

0.405

0.303

0.362

0.589

0.510

0.332

0.374

0.399

0.414

0.606

0.518

0.823

0.587

0.628

0.544

0.726

0.578

192

0.344

0.375

0.384

0.380

0.450

0.464

0.440

0.438

0.437

0.430

0.703

0.565

0.358

0.390

0.441

0.436

0.681

0.539

0.844

0.591

0.666

0.566

0.750

0.591

336

0.376

0.393

0.389

0.396

0.450

0.424

0.485

0.459

0.445

0.423

0.898

0.641

0.402

0.416

0.499

0.467

0.786

0.597

0.870

0.603

0.807

0.628

0.851

0.659

720

0.427

0.421

0.442

0.423

0.483

0.471

0.577

0.499

0.479

0.469

0.948

0.671

0.511

0.489

0.767

0.587

0.796

0.593

0.893

0.611

0.822

0.633

0.857

0.655

Avg

0.363

0.385

0.389

0.389

0.425

0.434

0.472

0.450

0.416

0.421

0.784

0.597

0.400

0.417

0.526

0.476

0.717

0.561

0.857

0.598

0.730

0.592

0.796

0.620

ETTm2

96

0.167

0.258

0.176

0.259

0.174

0.261

0.199

0.280

0.170

0.255

0.265

0.339

0.236

0.326

0.206

0.288

0.220

0.299

0.238

0.316

0.229

0.320

0.232

0.322

192

0.225

0.295

0.244

0.305

0.215

0.287

0.256

0.316

0.228

0.297

0.310

0.362

0.306

0.373

0.264

0.324

0.311

0.361

0.298

0.349

0.394

0.361

0.291

0.357

336

0.274

0.328

0.319

0.347

0.273

0.330

0.318

0.353

0.294

0.347

0.373

0.399

0.380

0.423

0.334

0.367

0.338

0.366

0.353

0.380

0.378

0.427

0.478

0.517

720

0.359

0.381

0.402

0.403

0.433

0.412

0.460

0.436

0.425

0.404

0.478

0.454

0.674

0.583

0.454

0.432

0.509

0.465

0.475

0.445

0.523

0.510

0.553

0.538

Avg

0.256

0.315

0.285

0.328

0.274

0.323

0.308

0.346

0.279

0.325

0.356

0.388

0.399

0.426

0.314

0.352

0.344

0.372

0.341

0.372

0.381

0.404

0.388

0.433

Weather

96

0.154

0.205

0.155

0.203

0.172

0.263

0.175

0.230

0.167

0.218

0.264

0.307

0.184

0.242

0.171

0.224

0.207

0.253

0.215

0.252

0.229

0.309

0.227

0.299

192

0.198

0.246

0.206

0.257

0.224

0.271

0.227

0.276

0.225

0.273

0.284

0.326

0.228

0.283

0.230

0.277

0.272

0.307

0.290

0.307

0.265

0.317

0.278

0.333

336

0.249

0.285

0.255

0.292

0.282

0.321

0.286

0.322

0.280

0.320

0.323

0.349

0.279

0.322

0.294

0.326

0.313

0.328

0.353

0.348

0.353

0.392

0.351

0.393

720

0.321

0.337

0.330

0.343

0.366

0.381

0.366

0.379

0.359

0.371

0.366

0.375

0.364

0.388

0.384

0.387

0.400

0.385

0.452

0.407

0.391

0.394

0.387

0.389

Avg

0.230

0.268

0.236

0.273

0.260

0.309

0.263

0.301

0.257

0.295

0.309

0.339

0.263

0.308

0.269

0.303

0.298

0.318

0.327

0.328

0.309

0.353

0.310

0.353

Electricity

96

0.136

0.234

0.146

0.241

0.147

0.242

0.143

0.241

0.153

0.251

0.162

0.264

0.150

0.251

0.145

0.244

0.315

0.389

0.484

0.518

0.235

0.322

0.297

0.367

192

0.153

0.249

0.166

0.262

0.158

0.241

0.159

0.255

0.169

0.268

0.180

0.278

0.163

0.263

0.163

0.260

0.318

0.396

0.501

0.531

0.247

0.341

0.308

0.375

336

0.168

0.267

0.185

0.282

0.178

0.277

0.179

0.274

0.183

0.281

0.207

0.305

0.175

0.278

0.183

0.281

0.340

0.415

0.574

0.578

0.267

0.356

0.354

0.411

720

0.205

0.303

0.236

0.320

0.224

0.312

0.233

0.323

0.239

0.324

0.258

0.339

0.219

0.311

0.233

0.323

0.635

0.613

0.952

0.786

0.318

0.394

0.426

0.466

Avg

0.165

0.263

0.183

0.276

0.179

0.268

0.178

0.273

0.186

0.281

0.201

0.296

0.176

0.275

0.181

0.277

0.402

0.453

0.627

0.603

0.266

0.353

0.346

0.404

Traffic

96

0.399

0.283

0.411

0.293

0.414

0.291

0.419

0.298

0.410

0.289

0.431

0.312

0.427

0.304

0.404

0.286

0.854

0.492

1.468

0.821

0.670

0.421

0.795

0.481

192

0.411

0.287

0.425

0.300

0.419

0.291

0.434

0.305

0.415

0.295

0.456

0.326

0.447

0.315

0.412

0.294

0.894

0.517

1.509

0.838

0.653

0.405

0.837

0.503

336

0.421

0.292

0.446

0.315

0.437

0.314

0.449

0.313

0.433

0.310

0.465

0.334

0.478

0.333

0.439

0.310

0.853

0.471

1.602

0.860

0.707

0.445

0.867

0.523

720

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

Avg

0.410

0.287

0.427

0.303

0.423

0.298

0.434

0.305

0.419

0.298

0.450

0.324

0.450

0.317

0.418

0.296

0.867

0.493

1.526

0.839

0.676

0.423

0.833

0.502

1

st

Count

48

11

4

0

2

0

0

0

0

0

0

0

Models

SEMPO

TTM

Time-LLM

GPT4TS

S

2

P-LLM

iTransformer

DLinear

PatchTST

TimesNet

Stationary

Metrics

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE

MAE

MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE

ETTh1

96

0.378

0.405

0.362

0.389

0.448

0.460

0.458

0.456

0.463

0.459

0.790

0.586

0.492

0.495

0.516

0.485

0.861

0.628

0.918

0.639

0.512

0.499

0.613

0.552

192

0.407

0.424

0.386

0.408

0.484

0.483

0.570

0.516

0.482

0.487

0.837

0.609

0.565

0.538

0.598

0.524

0.797

0.593

0.915

0.629

0.624

0.555

0.722

0.598

336

0.426

0.439

0.399

0.420

0.589

0.540

0.608

0.535

0.603

0.543

0.780

0.575

0.721

0.622

0.657

0.550

0.941

0.648

0.939

0.644

0.691

0.574

0.750

0.619

720

0.456

0.474

0.465

0.477

0.700

0.604

0.725

0.591

0.713

0.588

1.234

0.811

0.986

0.743

0.762

0.610

0.877

0.641

0.887

0.645

0.728

0.614

0.721

0.616

Avg

0.417

0.435

0.403

0.423

0.556

0.522

0.590

0.525

0.565

0.524

0.910

0.860

0.691

0.600

0.633

0.542

0.869

0.628

0.915

0.639

0.639

0.561

0.702

0.596

ETTh2

96

0.276

0.340

0.275

0.335

0.275

0.326

0.331

0.374

0.300

0.360

0.404

0.435

0.357

0.411

0.353

0.389

0.378

0.409

0.389

0.411

0.382

0.416

0.413

0.451

192

0.332

0.376

0.345

0.383

0.374

0.373

0.402

0.411

0.372

0.371

0.470

0.474

0.569

0.519

0.403

0.414

0.490

0.467

0.473

0.455

0.478

0.474

0.474

0.477

336

0.354

0.399

0.384

0.416

0.406

0.429

0.406

0.433

0.389

0.413

0.489

0.485

0.671

0.572

0.426

0.441

0.537

0.494

0.507

0.480

0.504

0.501

0.547

0.543

720

0.395

0.433

0.419

0.450

0.427

0.449

0.449

0.464

0.403

0.426

0.593

0.538

0.824

0.648

0.477

0.480

0.510

0.491

0.477

0.472

0.499

0.509

0.516

0.523

Avg

0.339

0.387

0.355

0.391

0.370

0.394

0.397

0.421

0.366

0.392

0.489

0.483

0.605

0.538

0.415

0.431

0.479

0.465

0.462

0.455

0.466

0.475

0.488

0.499

ETTm1

96

0.304

0.355

0.346

0.363

0.346

0.388

0.390

0.404

0.353

0.390

0.709

0.556

0.352

0.392

0.410

0.419

0.583

0.501

0.761

0.568

0.578

0.518

0.774

0.614

192

0.340

0.375

0.375

0.395

0.373

0.416

0.429

0.423

0.368

0.403

0.717

0.548

0.382

0.412

0.437

0.434

0.630

0.528

0.781

0.574

0.617

0.546

0.754

0.592

336

0.372

0.392

0.398

0.392

0.413

0.426

0.469

0.439

0.417

0.428

0.735

0.575

0.419

0.434

0.476

0.454

0.725

0.568

0.803

0.587

0.998

0.775

0.869

0.677

720

0.427

0.423

0.443

0.429

0.485

0.476

0.569

0.498

0.473

0.468

0.752

0.584

0.490

0.477

0.681

0.556

0.769

0.549

0.844

0.581

0.693

0.579

0.810

0.630

Avg

0.360

0.386

0.390

0.394

0.404

0.427

0.464

0.441

0.402

0.422

0.728

0.565

0.411

0.429

0.501

0.466

0.677

0.537

0.797

0.578

0.722

0.605

0.802

0.628

ETTm2

96

0.166

0.256

0.176

0.260

0.177

0.261

0.188

0.269

0.140

0.242

0.245

0.322

0.213

0.303

0.191

0.274

0.212

0.285

0.229

0.308

0.291

0.399

0.352

0.454

192

0.223

0.295

0.242

0.304

0.241

0.314

0.251

0.309

0.207

0.293

0.274

0.338

0.278

0.345

0.252

0.317

0.270

0.323

0.291

0.343

0.307

0.379

0.694

0.691

336

0.276

0.329

0.315

0.345

0.274

0.327

0.307

0.346

0.264

0.331

0.361

0.394

0.338

0.385

0.306

0.353

0.323

0.353

0.348

0.376

0.543

0.559

2.408

1.407

720

0.357

0.381

0.394

0.399

0.417

0.390

0.426

0.417

0.381

0.387

0.467

0.442

0.436

0.440

0.433

0.427

0.474

0.449

0.461

0.438

0.712

0.614

1.913

1.166

Avg

0.255

0.315

0.281

0.327

0.277

0.323

0.293

0.335

0.248

0.313

0.336

0.373

0.316

0.368

0.296

0.343

0.320

0.353

0.332

0.366

0.463

0.488

1.342

0.930

Weather

96

0.152

0.204

0.152

0.199

0.161

0.210

0.163

0.215

0.154

0.201

0.253

0.307

0.171

0.224

0.165

0.215

0.184

0.230

0.192

0.234

0.188

0.253

0.221

0.297

192

0.196

0.243

0.193

0.245

0.204

0.248

0.210

0.254

0.195

0.241

0.292

0.328

0.215

0.263

0.210

0.257

0.245

0.283

0.269

0.295

0.250

0.304

0.270

0.322

336

0.246

0.282

0.246

0.282

0.261

0.302

0.256

0.292

0.260

0.302

0.322

0.346

0.258

0.299

0.259

0.297

0.305

0.321

0.370

0.357

0.312

0.346

0.320

0.351

720

0.319

0.336

0.336

0.346

0.309

0.332

0.321

0.339

0.303

0.329

0.365

0.374

0.320

0.346

0.332

0.346

0.381

0.371

0.441

0.405

0.387

0.393

0.390

0.396

Avg

0.228

0.266

0.232

0.268

0.234

0.273

0.238

0.275

0.228

0.268

0.308

0.338

0.241

0.283

0.242

0.279

0.279

0.301

0.318

0.323

0.284

0.324

0.300

0.342

Electricity

96

0.134

0.231

0.140

0.237

0.139

0.241

0.139

0.237

0.142

0.243

0.154

0.257

0.150

0.253

0.140

0.238

0.299

0.373

0.420

0.466

0.231

0.323

0.261

0.348

192

0.151

0.247

0.163

0.259

0.151

0.248

0.156

0.252

0.163

0.260

0.171

0.272

0.164

0.264

0.160

0.255

0.305

0.379

0.411

0.459

0.261

0.356

0.338

0.406

336

0.167

0.267

0.180

0.275

0.169

0.270

0.175

0.270

0.173

0.270

0.196

0.295

0.181

0.282

0.180

0.276

0.319

0.391

0.434

0.473

0.360

0.445

0.410

0.474

720

0.201

0.298

0.241

0.326

0.240

0.322

0.233

0.317

0.237

0.320

0.263

0.348

0.223

0.321

0.241

0.323

0.369

0.426

0.510

0.521

0.530

0.585

0.715

0.685

Avg

0.163

0.261

0.181

0.274

0.175

0.270

0.176

0.269

0.178

0.273

0.196

0.293

0.180

0.280

0.180

0.273

0.323

0.392

0.444

0.480

0.346

0.427

0.431

0.478

Traffic

96

0.395

0.277

0.408

0.292

0.418

0.291

0.414

0.297

0.401

0.285

0.448

0.329

0.419

0.298

0.403

0.289

0.719

0.416

1.412

0.802

0.639

0.400

0.672

0.405

192

0.407

0.282

0.417

0.295

0.414

0.296

0.426

0.301

0.410

0.293

0.487

0.360

0.434

0.305

0.415

0.296

0.748

0.428

1.419

0.806

0.637

0.416

0.727

0.424

336

0.417

0.287

0.430

0.302

0.421

0.311

0.434

0.303

0.425

0.314

0.514

0.372

0.449

0.313

0.426

0.304

0.853

0.471

1.443

0.815

0.655

0.427

0.749

0.454

720

0.450

0.304

0.476

0.331

0.462

0.327

0.487

0.337

0.470

0.330

0.532

0.383

0.484

0.336

0.474

0.331

1.485

0.825

1.539

0.837

0.722

0.456

0.847

0.499

Avg

0.417

0.287

0.432

0.305

0.429

0.306

0.440

0.310

0.426

0.305

0.495

0.361

0.447

0.313

0.430

0.305

0.951

0.535

1.453

0.815

0.663

0.425

0.749

0.446

1

st

Count

45

15

4

0

13

0

0

0

0

0

0

0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a lightweight foundation model for time series forecasting, SEMPO, which includes an Energy-Aware Spectral Decomposition (EASD) module and a Mixture-of-Prompts enabled Transformer (MoPFormer) module. The former leverages high- and low-energy frequency signals to improve data utilization, while the latter achieves parameter-efficient adaptation through a lightweight prompt-based expert system. Experiments show that SEMPO, with 6.5 million parameters and pre-trained on 83 million data samples, outperforms mainstream models in terms of prediction accuracy under zero-shot and few-shot scenarios, demonstrating its efficiency and generalization capability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Proposes a lightweight SEMPO model that achieves zero-/few-shot prediction advantages with only 6.5 million parameters and 83 million pre-training data, offering high resource efficiency and practical value.  \n2. The experiments cover multiple datasets and baseline models, including ablation studies and visualization analyses, providing thorough validation.  \n3. The paper is well-structured, with clear formulas, standardized figures, precise technical descriptions, and smooth logical flow.\n\n**References**\n\n1. It is recommended to include full-shot experiments to evaluate model performance with the full dataset, thereby validating the benefits of pre-training.  \n2. A comparison with frequency debiasing models such as Fedformer [1] should be added to highlight the superiority of the EASD module.  \n3. The description of the two-stage training paradigm in the main text is too brief; more details should be provided.  \n4. The EASD module uses a masking mechanism for data augmentation, which actually increases the amount of training data. Therefore, further verification is needed to confirm whether the claimed reduction in pre-training cost is valid.  \n5. The zero-shot experiment in Table 1 and the efficiency analysis in Figure 6 do not include the lightweight model TTM. It is suggested to add its results to fully illustrate the performance gradient of the lightweight methods. Additionally, since new versions of Chronos and TimesFM are available, it is advisable to update the comparisons to the latest state-of-the-art models to ensure the conclusions remain current with field advancements.  \n6. The visualization of $Z_{Lec}$ in Figure 2 appears to contain an error; the complementary relationship between $Z_{Lec}$ and $Z_{Hec}$ should be corrected.  \n7. In Figure 4(b), the low-energy feature response of Moirai seems to reflect prediction inaccuracies rather than being completely neglected. The authors are encouraged to provide a more reasonable explanation for this phenomenon.\n\n**Weaknesses**\n\n[1] Fredformer: Frequency Debiased Transformer for Time Series Forecasting, KDD 2024\n\nQUESTIONS:\nThe authors can refer to the weakness listed above.\n\nLIMITATIONS:\nThe authors can refer to the weakness listed above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SEMPO, a lightweight foundation model for general-purpose time series forecasting that achieves strong performance while using fewer parameters and less pre-training data than existing large-scale models. SEMPO integrates two key features: (1) an Energy-Aware Spectral Decomposition (EASD) module that transforms time series into the frequency domain and models both high- and low-energy frequency signals, improving utilization of pre-training data; and (2) a Mixture-of-Prompts-enabled Transformer (MoPFormer) that uses dataset-specific prompts to capture heterogeneous temporal patterns via a lightweight routing mechanism. SEMPO achieves zero-shot and few-shot forecasting on various datasets using less parameters and pre-training data points. Experiments show that it outperforms several LLM-based models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. SEMPO achieves state-of-the-art results in zero-shot and few-shot time series forecasting across 16 datasets, despite using only 6.5M parameters and a modest 83M time points for pre-training, which are substantially smaller than competing models that use hundreds of millions of parameters and billions of data points.\n\nS2. The proposed EASD module effectively captures both high- and low-energy frequency components of time series. \n\nS3. The paper provides extensive experiments on benchmark time series datasets covering various domains. It includes zero-shot, few-shot, ablation, and efficiency analyses that can demonstrate the superiority of SEMPO.\n\n\\\nWeaknesses: \n\nW1. Some claims are not sufficiently justified. One notable example is the claim that existing time series FMs suffer from energy bias and fail to effectively handle low-energy frequency components is primarily supported by a case study. A more rigorous justification for this limitation by other models would be necessary to convincingly support this claim.\n\nW2. The presentation of SEMPO’s data efficiency is unclear. While the model is positioned as requiring less pre-training data, the paper does not clearly quantify the significance of this reduction compared to baselines. It is unclear how many data points are utilized in other models. Also, the epochs for pre-training play an important role in its real data utilization. \n\nW3. There is a lack of clarity regarding the relationship between the pre-training and inference stages. There’s a gap between training and inference stage. It is better to illustrate the process of this two stages in details. \n\nW4. The analysis of low-energy components in the experiments is weak. The paper includes only a qualitative case study to demonstrate SEMPO’s effectiveness in modeling such signals. However, a more comprehensive quantitative evaluation is preferred to be provided.\n\nQUESTIONS:\nQ1. The claimed advantage of using less pre-training data is not convincingly demonstrated. The paper does not clearly specify how the number of datasets or total data volume used for pre-training compares to other models. Additionally, the number of pre-training epochs significantly affects the effective data utilization and should be reported to ensure a fair and meaningful comparison.\n\nQ2. MoE architectures typically rely on sparse routing to activate only a subset of expert networks, while maintaining diversity among the experts is essential for generalization. It remains unclear how SEMPO addresses these challenges, particularly how it ensures both routing efficiency and functional diversity in the proposed prompt-based expert system.\n\nQ3. Please respond to the other concerns raised in W1-W4.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper tackles the time series forecasting task by utilising a lightweight foundation model, involving 2 key modules, one decomposes the time series in the spectral domain by thresholding the frequencies and masking the frequencies randomly; and one mixture of prompts block. It is claimed that with such modules the model improves the time series forecasting performance in MSE, MAE or sMAPE.

STRENGTHS AND WEAKNESSES:
- Strength

  The paper is well written and easy to follow. The approach is clearly described, and the experimental setup and the results are well illustrated. 

  The proposed approach is to the best of my knowledge novel and sound. 

- Weakness

  It seems the results in the experiments are from only one run. Since the masks are generated randomly, it would be more convincing if multiple runs are conducted and the standard deviations are reported. 

  It is claimed in the checklist 7, but I failed to see a discussion about the statistical test in the paper.

QUESTIONS:
- What do Hec and Lec mean? What would you expect, if the energy-wise spectral partitioning leads to more than 2 partitions? 

- In figure 4, what is "input", and what is the point of plotting "input" in the figure?

- In figure 8a in Appendix, could you explain the peak at N_m=5? Does it make sense to show the cases with N_m > 6?

LIMITATIONS:
Although claimed in the checklist, I failed to find a proper discussion of limitations in the conclusion.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SEMPO, a lightweight foundation model designed for general time series forecasting that significantly reduces both model size and pre-training data requirements, while still achieving state-of-the-art performance in zero-shot and few-shot scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1.\tThe paper is well-structured, with clear organization and a logical flow that makes it easy to follow.\n\n2.\tContribution: The paper explores an efficient way to address time series forecasting through Energy-aware Spectral Decomposition and Mixture-of-Prompts Transformer. Extensive experiment results demonstrate that SEMPO is able to outperform baselines with only 6.5M parameters which is pre-trained on 83 million time points\n\nWeakness\n\n1.\tThe core contribution of SEMPO is to present a lightweight yet effective foundation model for time series forecasting. However, the experimental evaluation primarily focuses on zero-shot and few-shot settings, where many of the baseline methods were not originally designed to perform. This may limit the fairness and comprehensiveness of the comparison. It would strengthen the empirical analysis if the authors could additionally include in-distribution evaluations—similar to what was done in Time-MoE—to demonstrate that SEMPO also achieves competitive or superior performance in standard supervised settings, despite its reduced model size.\n\n2.\tIn lines 51–53, the authors state: “Specifically, modeling full energy spectrum of temporal signals from pre-training data is essential for learning a comprehensive knowledge base of temporal patterns to underpin strong transferability across different datasets.” This is an important claim, but it lacks empirical evidence or citations to support it. \n\n\n3.\tWhile the proposed Mixture-of-Prompts (MoP) mechanism contributes to model adaptability, the overall technical novelty of SEMPO is limited. The use of FFT for spectral decomposition is a well-established technique in time series analysis, and the design of the model architecture—particularly the encoder-decoder structure and masking strategies—follows relatively standard practices. As a result, the contribution of the model as a whole may not be sufficient for a top-tier venue.\n\n4.\tThere are several formatting issues in the current draft, such as broken links to corresponding figures and missing or improperly linked citations. These issues hinder readability and should be addressed to improve the overall presentation quality.\n\nQUESTIONS:\nIn lines 126–130, the paper introduces the concepts of $D_tune$ and $D_test$ in the context of few-shot and zero-shot evaluation. However, it is unclear whether these datasets are drawn from entirely new domains or from domains already included in the pre-training set $T_train$. Please clarify this distinction to help assess the generalization capabilities of the proposed model more accurately.\n\n\tIn Figure 1 (right), SEMPO (Zero) and SEMPO (Few) are marked separately, while other models such as PatchTST and Time-LLM are not differentiated by evaluation setting. This inconsistency may lead to confusion when interpreting the results. It would be clearer to present two separate figures: one for zero-shot comparisons and another for few-shot comparisons, ensuring a fair and consistent visual comparison across models.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}