Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

## Abstract

Abstract Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in language and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecasting models in real-world applications. In response, we introduce Time-MoE , a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. Time-MoE comprises a family of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time-300B , which spans over 9 domains and encompassing over 300 billion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number of activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements position Time-MoE as a state-of-the-art solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility.

## 1 Introduction

Time series data is a major modality in real-world dynamic systems and applications across various domains (Box et al., ; Zhang et al., ; Liang et al., ). Analyzing time series data is challenging due to its inherent complexity and distribution shifts, yet it is crucial for unlocking insights that enhance predictive analytics and decision-making. As a key task in high demand, time series forecasting has long been studied and is vital for driving various use cases in fields such as energy, climate, education, quantitative finance, and urban computing (Jin et al., ; Nie et al., ; Mao et al., ). Traditionally, forecasting has been performed in a task-specific, end-to-end manner using either statistical or deep learning models. Despite their competitive performance, the field has not converged on building unified, general-purpose forecasting models until recently, with the emergence of a few foundation models (FMs) for universal forecasting (Das et al., ; Woo et al., ; Ansari et al., ). Although promising, they are generally small in scale and have limited task-solving capabilities compared to domain-specific models, limiting their real-world impact when balancing forecasting precision against computational budget.

Figure: Figure 1: Performance overview. (Left) Comparison between Time-MoE models and state-of-the-art time series foundation models, reporting the average zero-shot performance across six benchmark datasets. (Right) Comparison of few- and zero-shot performance between Time-MoE and dense variants, with similar effective FLOPs per time series token, across the same six benchmarks.
Refer to caption: x1.png

Increasing model size and training tokens typically leads to performance improvements, as known as scaling laws, which have been extensively explored in the language and vision domains (Kaplan et al., ; Alabdulmohsin et al., ). However, such properties have not been thoroughly investigated in the time series domain. Assuming that scaling forecasting models with high-quality training data follows similar principles, several challenges remain: *Dense versus sparse training.* Most time series forecasting models compose of dense layers, which means each input time series tokens requires computations with all model parameters. While effective, this is computationally intensive. In contrast, sparse training with mixture-of-experts (MoE) is more flop-efficient per parameter and allows for scaling up model size with a fixed inference budget while giving better performance, as showcased on the right of Figure 1. However, optimizing a sparse, large-scale time series model faces another challenge of *stability and convergency.* Time series are highly heterogeneous (Woo et al., ; Dong et al., ), and selecting the appropriate model design and routing algorithm often involves a trade-off between performance and computational efficiency. Sparse solutions for time series foundation models have yet to be explored, leaving a significant gap in addressing these two challenges. While time series *pre-training datasets* are no longer a major bottleneck, most existing works (Das et al., ; Woo et al., ; Ansari et al., ) have not extensively discussed their in-model data processing pipelines or mixing strategies. Answering this is particularly important, given that existing data archives are often noisy and largely imbalanced across domains.

On the other hand, most time series FMs face limitations in *flexibility and generalizability*. General-purpose forecasting is a fundamental capability, requiring a model to handle any forecasting problems, regardless of context lengths, forecasting horizons, input variables, and other properties such as frequencies and distributions. Meanwhile, achieving strong generalizability pushes the boundaries further that existing works often fail to meet simultaneously. For instance, Timer (Liu et al., ) has limited native support for arbitrary output lengths, which may lead to truncated outputs, while Moment (Goswami et al., ) operates with a fixed input context length. Although Moirai (Woo et al., ) achieves universal forecasting, it depends on hardcoded heuristics in both the input and output layers.

The recognition of the above challenges naturally raises a pivotal question:

Answering this question drives the design of Time-MoE, a scalable and unified architecture for pre-training larger, more capable forecasting FMs while reducing computational costs. Time-MoE consists of a family of decoder-only transformer models with a mixture-of-experts architecture, operating in an auto-regressive manner to support any forecasting horizon and accommodate context lengths of up to 4096. With its sparsely activated design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without significantly increasing inference costs. Our proposal is built on a minimalist design, where the input time series is point-wise tokenized and encoded before being processed by a sparse transformer decoder, activating only a small subset of parameters. Pre-trained on large-scale time series data across 9 domains and over 300 billion time points, Time-MoE is optimized through multi-task learning to forecast at multiple resolutions. During inference, different forecasting heads are utilized to enable forecasts across diverse scales, enabling flexible forecast horizons. For the first time, we scale a time series FM up to 2.4 billion parameters, achieving substantial improvements in forecasting precision compared to existing models, as shown on the left of Figure 1. Compared to dense models with the same number of activated parameters or equivalent computational budgets, our models consistently outperform them by a large margin. Our contributions lie in three aspects:

- 1.
We present Time-MoE, a universal decoder-only time series forecasting foundation model architecture with mixture-of-experts. To the best of our knowledge, this is the first work to scale time series foundation models up to 2.4 billion parameters.
Time-MoE achieves substantial improvements in forecasting accuracy and consistently outperforms dense models with comparable computational resources, while maintaining high efficiency.
- 2.
We introduce Time-300B, the largest open-access time series data collection, comprising over 300 billion time points spanning more than nine domains, accompanied by a well-designed data-cleaning pipeline. Our Time-MoE models and Time-300B data collection are open-sourced.
- 3.
Trained on Time-300B, Time-MoE models outperform other time series foundation models with a similar number of activated parameters across six real-world benchmarks, achieving reductions in forecasting errors by an average of 20% and 24% in zero-shot and in-distribution scenarios, respectively.

## 2 Related Work

Time Series Forecasting. Deep learning models have become powerful tools for time series forecasting over the past decade, which can be broadly categorized into two types: (1) *univariate models*, such as DeepState (Rangapuram et al., ), DeepAR (Salinas et al., ), and N-BEATS (Oreshkin et al., ), which focus on modeling individual time series, and (2) *multivariate models*, which include both transformer-based approaches (Wen et al., ; Zhou et al., ; Nie et al., ; Liu et al., ; Wang et al., ; Chen et al., ) and non-transformer models (Sen et al., ; Jin et al., ; Wang et al., ; Hu et al., ; Qi et al., ), designed to handle multiple time series simultaneously. While these models achieve competitive in-domain performance, many are task-specific and fall short in generalizability when applied to cross-domain data in few-shot or zero-shot scenarios.

Large Time Series Models. Pre-training on large-scale sequence data has significantly advanced modality understanding in language and vision domains (Dong et al., ; Selva et al., ). Building on this progress, self-supervised learning has been extensively developed for time series (Zhang et al., ), employing masked reconstruction (Zerveas et al., ; Nie et al., ) or contrastive learning (Zhang et al., ; Yue et al., ; Yang et al., ). However, these methods are limited in both data and model scale, with many focused on in-domain learning and transfer. Recently, general pre-training of time series models on large-scale data has emerged, though still in its early stages with insufficient exploration into sparse solutions. We discuss the development more in Appendix A. Unlike these dense models, Time-MoE introduces a scalable and unified architecture for pre-training larger forecasting foundation models, which is also more capable while maintaining the same scale of activated parameters or computational budget as dense models.

Sparse Deep Learning for Time Series. Deep learning models are often dense and over-parameterized (Hoefler et al., ), leading to increased memory and computational demands during both training and inference. However, sparse networks, such as mixture-of-experts models (Jacobs et al., ), which dynamically route inputs to specialized expert networks, have shown comparable or even superior generalization to dense models while being more efficient (Fedus et al., ; Riquelme et al., ). In time series research, model sparsification has received less attention, as time series models have traditionally been small in scale, with simple models like DLinear (Zeng et al., ) and SparseTSF (Lin et al., ) excelling in specific tasks prior to the advent of large-scale, general pre-training. The most relevant works on this topic include Pathformer (Chen et al., ), MoLE (Ni et al., ), and IME (Ismail et al., ). However, none of them delve into the scalability of foundation models with sparse structures. Besides, MoLE and IME are not sparse models, as input data is passed to all heads and then combined to make predictions.

## 3 Methodology

Our proposed Time-MoE, illustrated in Figure 2, adopts a mixture-of-experts-based, decoder-only transformer architecture, comprising three key components: (1) *input token embedding*, (2) *MoE transformer block*, and (3) *multi-resolution forecasting*. For the first time, we scale a sparsely-activated time series model to 2.4 billion parameters, achieving significantly better zero-shot performance with the same computation. This marks a major step forward in developing large time series models for universal forecasting.

Problem Statement.
We address the problem of predicting future values in a time series: given a sequence of historical observations $\mathbf{X}_{1:T}=\left(x_{1},x_{2},\ldots,x_{T}\right)\in\mathbb{R}^{T}$ spanning $T$ time steps, our objective is to forecast the next $H$ time steps, i.e., $\hat{\mathbf{X}}_{T+1:T+H}=f_{\theta}\left(\mathbf{X}_{1:T}\right)\in\mathbb{R
}^{H}$. Here, $f_{\theta}$ represents a time series model, where $T$ is the context length and $H$ is the forecasting horizon. Notably, both $T$ and $H$ can be *flexible* during Time-MoE inference, distinguishing it from task-specific models with fixed horizons. Additionally, channel independence (Nie et al., ) is adopted to transform a multivariate input into univariate series, allowing Time-MoE to handle *any-variate* forecasting problems in real-world applications.

Figure: Figure 2: The architecture of Time-MoE, which is a decoder-only model. Given an input time series of arbitrary length, 1 we first tokenize it into a sequence of data points, 2 which are then encoded. These tokens are processed through $N$-stacked backbone layers, primarily consisting of causal multi-head self-attention and 3 sparse temporal mixture-of-expert layers. During training, 4 we optimize forecasting heads at multiple resolutions. For model inference, Time-MoE provides forecasts of flexible length by 5 dynamically scheduling these heads.
Refer to caption: x2.png

### 3.1 Time-MoE Overview

Input Token Embedding.
We utilize *point-wise tokenization* for time series embedding to ensure the completeness of temporal information. This enhances our model’s flexibility and broad applicability in handling variable-length sequences. Then, we employ SwiGLU (Shazeer, ) to embed each time series point:

$$ $\mathbf{h}^{0}_{t}=\operatorname{SwiGLU}(x_{t})=\operatorname{Swish}\left(Wx_{ t}\right)\otimes\left(Vx_{t}\right),$ (1) $$

where $W\in R^{D\times 1}$ and $V\in R^{D\times 1}$ are learnable parameters, and $D$ denotes the hidden dimension.

MoE Transformer Block.
Our approach builds upon a decoder-only transformer (Vaswani, ) and integrates recent advancements from large language models (Bai et al., ; Touvron et al., ). We employ RMSNorm (Zhang & Sennrich, ) to normalize the input of each transformer sub-layer, thereby enhancing training stability. Instead of using absolute positional encoding, we adopt rotary positional embeddings (Su et al., ), which provide greater flexibility in sequence length and improved extrapolation capabilities. In line with (Chowdhery et al., ), we remove biases from most layers but retain them in the QKV layer of self-attention to improve extrapolation. To introduce sparsity, we replace a feed-forward network (FFN) with a mixture-of-experts layer, incorporating a shared pool of experts that are sparsely activated.

$$ $\displaystyle\mathbf{u}^{l}_{t}$ $\displaystyle=\operatorname{SA}\left(\operatorname{RMSNorm}\left(\mathbf{h}^{l -1}_{t}\right)\right)+\mathbf{h}^{l-1}_{t},$ (2) $\displaystyle\mathbf{\bar{u}}^{l}_{t}$ $\displaystyle=\operatorname{RMSNorm}\left(\mathbf{u}^{l}_{t}\right),$ (3) $\displaystyle\mathbf{h}^{l}_{t}$ $\displaystyle=\operatorname{Mixture}\left(\mathbf{\bar{u}}^{l}_{t}\right)+ \mathbf{u}^{l}_{t}.$ (4) $$

Here, $\operatorname{SA}$ denotes self-attention with a causal mask, and $\operatorname{Mixture}$ refers to the mixture-of-experts layer. In practice, $\operatorname{Mixture}$ comprises several expert networks, each mirroring the architecture of a standard FFN. An individual time series point can be routed to either a single expert (Fedus et al., ) or multiple experts (Lepikhin et al., ). One expert is designated as a shared expert to capture and consolidate common knowledge across different contexts.

$$ $\displaystyle\operatorname{Mixture}\left(\mathbf{\bar{u}}^{l}_{t}\right)$ $\displaystyle=g_{N+1,t}\operatorname{FFN}_{N+1}\left(\mathbf{\bar{u}}^{l}_{t} \right)+\sum_{i=1}^{N}\left({g_{i,t}\operatorname{FFN}_{i}\left(\mathbf{\bar{u }}^{l}_{t}\right)}\right),$ (5) $\displaystyle g_{i,t}$ $\displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\operatorname{Topk}(\{s_{j,t}|1 \leq j\leq N\},K),\\ 0,&\text{otherwise},\end{cases}$ (6) $\displaystyle g_{N+1,t}$ $\displaystyle=\operatorname{Sigmoid}\left(\mathbf{W}^{l}_{N+1}\mathbf{\bar{u}} ^{l}_{t}\right),$ (7) $\displaystyle s_{i,t}$ $\displaystyle=\operatorname{Softmax}_{i}\left(\mathbf{W}_{i}^{l}\mathbf{u}_{t} ^{l}\right),$ (8) $$

where $\mathbf{W}_{i}^{l}\in\mathbb{R}^{1\times D}$ denotes the trainable parameters, and $N$ and $K$ respectively denote the numbers of non-shared experts and activated non-shared experts per MoE layer.

Multi-resolution Forecasting.
We introduce a novel multi-resolution forecasting head, which allows for forecasting at multiple scales simultaneously, in contrast to existing foundation models that are limited to a single fixed scale. This capability enhances Time-MoE ’s flexibility by enabling forecasting across various horizons. The model employs multiple output projections from single-layer FFNs, each designed for different prediction horizons. During training, Time-MoE aggregates forecasting errors from different horizons to compute a composite loss (Section 3.2.2), thereby improving the model generalization. By incorporating a simple greedy scheduling algorithm (see Appendix B), Time-MoE efficiently handles predictions across arbitrary horizons. This design also boosts prediction robustness through multi-resolution ensemble learning during inference.

### 3.2 Model Training

#### 3.2.1 Time-300B Dataset

Training time series foundation models require extensive, high-quality data. However, well-processed large-scale datasets are still relatively scarce. Recent advancements have facilitated the collection of numerous time series datasets from various sources (Godahewa et al., ; Ansari et al., ; Woo et al., ; Liu et al., ). Nonetheless, data quality still remains a challenge, with prevalent issues such as *missing values* and *invalid observations* (Wang et al., ) that can impair model performance and destabilize training. To mitigate these issues, we developed a streamlined *data-cleaning pipeline* (Appendix C) to filter and refine raw data, and constructed the largest open-access, high-quality time series data collection named Time-300B for foundation model pre-training. Time-300B is composed of a diverse range of publicly available datasets, spanning multiple domains such as energy, retail, healthcare, weather, finance, transportation, and web, as well as a portion of synthetic data to enhance the data quantity and diversity. Time-300B covers a wide spectrum of sampling frequencies from seconds to yearly intervals, and has over 300 billion time points after being processed by our data-cleaning pipeline, as summarized in Table 1.

**Table 1: Key statistics of the pre-training dataset Time-300B from various domains.**
|  | Energy | Finance | Healthcare | Nature | Sales | Synthetic | Transport | Web | Other | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Seqs. | 2,875,335 | 1,715 | 1,752 | 31,621,183 | 110,210 | 11,968,625 | 622,414 | 972,158 | 40,265 | 48,220,929 |
| # Obs. | 15.981 B | 413.696 K | 471.040 K | 279.724 B | 26.382 M | 9.222 B | 2.130 B | 1.804 B | 20.32 M | 309.09 B |
| % | 5.17 % | 0.0001% | 0.0001% | 90.50 % | 0.008 % | 2.98% | 0.69 % | 0.58 % | 0.006 % | 100% |

**Table 2: A high-level summary of Time-MoE model configurations.**
|  | Layers | Heads | Experts | $K$ | $\bm{d}$model | $\bm{d}$ff | $\bm{d}$expert | Activated Params | Total Params |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | 12 | 12 | 8 | 2 | 384 | 1536 | 192 | 50 $\mathrm{M}$ | 113 $\mathrm{M}$ |
| Time-MoElarge | 12 | 12 | 8 | 2 | 768 | 3072 | 384 | 200 $\mathrm{M}$ | 453 $\mathrm{M}$ |
| Time-MoEultra | 36 | 16 | 8 | 2 | 1024 | 4096 | 512 | 1.1 $\mathrm{B}$ | 2.4 $\mathrm{B}$ |

#### 3.2.2 Loss Function

Pre-training time series foundation models in large scale presents significant challenges in training stability due to the massive datasets and the vast number of parameters involved. To address this, we use the Huber loss (Huber, ; Wen et al., ), which provides greater robustness to outliers and improves training stability:

$$ $\mathcal{L}_{\text{ar}}\left(x_{t},\hat{x}_{t}\right)=\begin{cases}\frac{1}{2} \left(x_{t}-\hat{x}_{t}\right)^{2},&\text{if }\left|x_{t}-\hat{x}_{t}\right| \leq\delta,\\ \delta\times\left(\left|x_{t}-\hat{x}_{t}\right|-\frac{1}{2}\times\delta\right ),&\text{otherwise},\end{cases}$ (9) $$

where $\delta$ is a hyperparameter that balances the L1 and L2 loss components.

When training the model with a MoE architecture, focusing solely on optimizing prediction error often leads to load imbalance issues among the experts. A common problem is routing collapse (Shazeer et al., ), where the model predominantly selects only a few experts, limiting training opportunities for others. To mitigate this, following the approaches of (Dai et al., ; Fedus et al., ), we achieve expert-level balancing with an auxiliary loss to reduce routing collapse:

$$ $\displaystyle\mathcal{L}_{\text{aux}}=N\sum_{i=1}^{N}f_{i}r_{i},$ $\displaystyle\quad f_{i}=\frac{1}{KT}\sum_{t=1}^{T}\mathbb{I}\left(\text{Time point }t\text{ selects Expert }i\right),\ \ r_{i}=\frac{1}{T}\sum_{t=1}^{T}s_{ i,t},$ (10) $$

where $f_{i}$ represents the fraction of tokens assigned to expert $i$, and $r_{i}$ denotes the proportion of router probability allocated to expert $i$. $\mathbb{I}$ is the indicator function. Finally, we combine the auto-regressive losses across all multi-resolution projections with the auxiliary balance loss to form the final loss:

$$ $\mathcal{L}=\frac{1}{P}\sum_{j=1}^{P}\mathcal{L}_{\text{ar}}\left(\mathbf{X}_{ t+1:t+p_{j}},\hat{\mathbf{X}}_{t+1:t+p_{j}}\right)+\alpha\mathcal{L}_{\text{ aux}},$ (11) $$

where $P$ is the number of multi-resolution projections and $p_{j}$ is the horizon of the $j$-th projection.

#### 3.2.3 Model Configurations and Training Details

Informed by the scaling laws demonstrated by Llama (Dubey et al., ; Touvron et al., ), which show that a 7- or 8-billion parameter model continues to improve performance even after training on over one trillion tokens, we chose to scale Time-MoE up to 2.4 billion parameters with around 1 billion of them activated. This model, Time-MoEultra, supports inference on consumer-grade GPUs with less than 8GB of VRAM. We have also developed two smaller models: Time-MoEbase, with 50 million activated parameters, and Time-MoElarge, with 200 million activated parameters, both specifically designed for fast inference on CPU architectures. These streamlined models are strategically developed to ensure broader accessibility and applicability in resource-constrained environments. The detailed model configurations are in Table 2. Each model undergoes training for $100,000$ steps with a batch size of $1024$, where the maximum sequence length is capped at $4096$. This setup results in the consumption of $4$ million time points per iteration. We choose $\left\{1,8,32,64\right\}$ as different forecast horizons in the output projection and set the factor of the auxiliary loss $\alpha$ to $0.02$. For optimization, we employ the AdamW optimizer, configured with the following hyperparameters: $\text{lr}=1\mathrm{e}\text{-}{3}$, $\text{weight_decay}=0.1$, $\beta_{1}=0.9$, $\beta_{2}=0.95$. A learning rate scheduler with a linear warmup over the initial $10,000$ steps followed by cosine annealing is also utilized. Training is conducted on 128 $\times$ NVIDIA A100-80G GPUs using BF16 precision.

## 4 Main Results

Time-MoE consistently outperforms state-of-the-art forecasting models by large margins across six well-established benchmarks and settings (Appendix B).
To ensure a fair comparison, we adhered to the experimental configurations from (Woo et al., ) for out-of-distribution forecasting and (Wu et al., ) for in-distribution forecasting with a unified evaluation pipeline we developed. Specifically, we evaluate Time-MoE against 16 different baselines, representing state-of-the-art models in long-term forecasting. They are categorized into two groups: 1. zero-shot forecasting evaluation group, includes pre-trained foundation models such as Moirai (), TimesFM (),
Moment (), and Chronos (); 2. in-distribution (full-shot) forecasting evaluation group, consists of modern time series models such as iTransformer (), TimeMixer (), TimesNet (), PatchTST (), Crossformer (), TiDE (), DLinear (), and FEDformer ().

### 4.1 Zero-shot Forecasting

**Table 3: Full results of zero-shot forecasting experiments. A lower MSE or MAE indicates a better prediction. TimesFM, due to its use of Weather datasets in pretraining, is not evaluated on these two datasets and is denoted by a dash ($-$). Red: the best, Blue: the 2nd best.**
| Models | Time-MoE (Ours) | Zero-shot Time Series Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | Time-MoElarge | Time-MoEultra | Moiraismall | Moiraibase | Moirailarge | TimesFM | Moment | Chronossmall | Chronosbase | Chronoslarge |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| ETTh1 | 96 | 0.357 | 0.381 | 0.350 | 0.382 | 0.349 | 0.379 | 0.401 | 0.402 | 0.376 | 0.392 | 0.381 | 0.388 | 0.414 | 0.404 | 0.688 | 0.557 | 0.466 | 0.409 | 0.440 | 0.393 | 0.441 | 0.390 |
| 192 | 0.384 | 0.404 | 0.388 | 0.412 | 0.395 | 0.413 | 0.435 | 0.421 | 0.412 | 0.413 | 0.434 | 0.415 | 0.465 | 0.434 | 0.688 | 0.560 | 0.530 | 0.450 | 0.492 | 0.426 | 0.502 | 0.424 |  |
| 336 | 0.411 | 0.434 | 0.411 | 0.430 | 0.447 | 0.453 | 0.438 | 0.434 | 0.433 | 0.428 | 0.495 | 0.445 | 0.503 | 0.456 | 0.675 | 0.563 | 0.570 | 0.486 | 0.550 | 0.462 | 0.576 | 0.467 |  |
| 720 | 0.449 | 0.477 | 0.427 | 0.455 | 0.457 | 0.462 | 0.439 | 0.454 | 0.447 | 0.444 | 0.611 | 0.510 | 0.511 | 0.481 | 0.683 | 0.585 | 0.615 | 0.543 | 0.882 | 0.591 | 0.835 | 0.583 |  |
|  | AVG | 0.400 | 0.424 | 0.394 | 0.419 | 0.412 | 0.426 | 0.428 | 0.427 | 0.417 | 0.419 | 0.480 | 0.439 | 0.473 | 0.443 | 0.683 | 0.566 | 0.545 | 0.472 | 0.591 | 0.468 | 0.588 | 0.466 |
| ETTh2 | 96 | 0.305 | 0.359 | 0.302 | 0.354 | 0.292 | 0.352 | 0.297 | 0.336 | 0.294 | 0.330 | 0.296 | 0.330 | 0.315 | 0.349 | 0.342 | 0.396 | 0.307 | 0.356 | 0.308 | 0.343 | 0.320 | 0.345 |
| 192 | 0.351 | 0.386 | 0.364 | 0.385 | 0.347 | 0.379 | 0.368 | 0.381 | 0.365 | 0.375 | 0.361 | 0.371 | 0.388 | 0.395 | 0.354 | 0.402 | 0.376 | 0.401 | 0.384 | 0.392 | 0.406 | 0.399 |  |
| 336 | 0.391 | 0.418 | 0.417 | 0.425 | 0.406 | 0.419 | 0.370 | 0.393 | 0.376 | 0.390 | 0.390 | 0.390 | 0.422 | 0.427 | 0.356 | 0.407 | 0.408 | 0.431 | 0.429 | 0.430 | 0.492 | 0.453 |  |
| 720 | 0.419 | 0.454 | 0.537 | 0.496 | 0.439 | 0.447 | 0.411 | 0.426 | 0.416 | 0.433 | 0.423 | 0.418 | 0.443 | 0.454 | 0.395 | 0.434 | 0.604 | 0.533 | 0.501 | 0.477 | 0.603 | 0.511 |  |
|  | AVG | 0.366 | 0.404 | 0.405 | 0.415 | 0.371 | 0.399 | 0.361 | 0.384 | 0.362 | 0.382 | 0.367 | 0.377 | 0.392 | 0.406 | 0.361 | 0.409 | 0.424 | 0.430 | 0.405 | 0.410 | 0.455 | 0.427 |
| ETTm1 | 96 | 0.338 | 0.368 | 0.309 | 0.357 | 0.281 | 0.341 | 0.418 | 0.392 | 0.363 | 0.356 | 0.380 | 0.361 | 0.361 | 0.370 | 0.654 | 0.527 | 0.511 | 0.423 | 0.454 | 0.408 | 0.457 | 0.403 |
| 192 | 0.353 | 0.388 | 0.346 | 0.381 | 0.305 | 0.358 | 0.431 | 0.405 | 0.388 | 0.375 | 0.412 | 0.383 | 0.414 | 0.405 | 0.662 | 0.532 | 0.618 | 0.485 | 0.567 | 0.477 | 0.530 | 0.450 |  |
| 336 | 0.381 | 0.413 | 0.373 | 0.408 | 0.369 | 0.395 | 0.433 | 0.412 | 0.416 | 0.392 | 0.436 | 0.400 | 0.445 | 0.429 | 0.672 | 0.537 | 0.683 | 0.524 | 0.662 | 0.525 | 0.577 | 0.481 |  |
| 720 | 0.504 | 0.493 | 0.475 | 0.477 | 0.469 | 0.472 | 0.462 | 0.432 | 0.460 | 0.418 | 0.462 | 0.420 | 0.512 | 0.471 | 0.692 | 0.551 | 0.748 | 0.566 | 0.900 | 0.591 | 0.660 | 0.526 |  |
|  | AVG | 0.394 | 0.415 | 0.376 | 0.405 | 0.356 | 0.391 | 0.436 | 0.410 | 0.406 | 0.385 | 0.422 | 0.391 | 0.433 | 0.418 | 0.670 | 0.536 | 0.640 | 0.499 | 0.645 | 0.500 | 0.555 | 0.465 |
| ETTm2 | 96 | 0.201 | 0.291 | 0.197 | 0.286 | 0.198 | 0.288 | 0.214 | 0.288 | 0.205 | 0.273 | 0.211 | 0.274 | 0.202 | 0.270 | 0.260 | 0.335 | 0.209 | 0.291 | 0.199 | 0.274 | 0.197 | 0.271 |
| 192 | 0.258 | 0.334 | 0.250 | 0.322 | 0.235 | 0.312 | 0.284 | 0.332 | 0.275 | 0.316 | 0.281 | 0.318 | 0.289 | 0.321 | 0.289 | 0.350 | 0.280 | 0.341 | 0.261 | 0.322 | 0.254 | 0.314 |  |
| 336 | 0.324 | 0.373 | 0.337 | 0.375 | 0.293 | 0.348 | 0.331 | 0.362 | 0.329 | 0.350 | 0.341 | 0.355 | 0.360 | 0.366 | 0.324 | 0.369 | 0.354 | 0.390 | 0.326 | 0.366 | 0.313 | 0.353 |  |
| 720 | 0.488 | 0.464 | 0.480 | 0.461 | 0.427 | 0.428 | 0.402 | 0.408 | 0.437 | 0.411 | 0.485 | 0.428 | 0.462 | 0.430 | 0.394 | 0.409 | 0.553 | 0.499 | 0.455 | 0.439 | 0.416 | 0.415 |  |
|  | AVG | 0.317 | 0.365 | 0.316 | 0.361 | 0.288 | 0.344 | 0.307 | 0.347 | 0.311 | 0.337 | 0.329 | 0.343 | 0.328 | 0.346 | 0.316 | 0.365 | 0.349 | 0.380 | 0.310 | 0.350 | 0.295 | 0.338 |
| Weather | 96 | 0.160 | 0.214 | 0.159 | 0.213 | 0.157 | 0.211 | 0.198 | 0.222 | 0.220 | 0.217 | 0.199 | 0.211 | - | - | 0.243 | 0.255 | 0.211 | 0.243 | 0.203 | 0.238 | 0.194 | 0.235 |
| 192 | 0.210 | 0.260 | 0.215 | 0.266 | 0.208 | 0.256 | 0.247 | 0.265 | 0.271 | 0.259 | 0.246 | 0.251 | - | - | 0.278 | 0.329 | 0.263 | 0.294 | 0.256 | 0.290 | 0.249 | 0.285 |  |
| 336 | 0.274 | 0.309 | 0.291 | 0.322 | 0.255 | 0.290 | 0.283 | 0.303 | 0.286 | 0.297 | 0.274 | 0.291 | - | - | 0.306 | 0.346 | 0.321 | 0.339 | 0.314 | 0.336 | 0.302 | 0.327 |  |
| 720 | 0.418 | 0.405 | 0.415 | 0.400 | 0.405 | 0.397 | 0.373 | 0.354 | 0.373 | 0.354 | 0.337 | 0.340 | - | - | 0.350 | 0.374 | 0.404 | 0.397 | 0.397 | 0.396 | 0.372 | 0.378 |  |
|  | AVG | 0.265 | 0.297 | 0.270 | 0.300 | 0.256 | 0.288 | 0.275 | 0.286 | 0.287 | 0.281 | 0.264 | 0.273 | - | - | 0.294 | 0.326 | 0.300 | 0.318 | 0.292 | 0.315 | 0.279 | 0.306 |
| Global Temp | 96 | 0.211 | 0.343 | 0.210 | 0.342 | 0.214 | 0.345 | 0.227 | 0.354 | 0.224 | 0.351 | 0.224 | 0.351 | 0.255 | 0.375 | 0.363 | 0.472 | 0.234 | 0.361 | 0.230 | 0.355 | 0.228 | 0.354 |
| 192 | 0.257 | 0.386 | 0.254 | 0.385 | 0.246 | 0.379 | 0.269 | 0.396 | 0.266 | 0.394 | 0.267 | 0.395 | 0.313 | 0.423 | 0.387 | 0.489 | 0.276 | 0.400 | 0.273 | 0.395 | 0.276 | 0.398 |  |
| 336 | 0.281 | 0.405 | 0.267 | 0.395 | 0.266 | 0.398 | 0.292 | 0.419 | 0.296 | 0.420 | 0.291 | 0.417 | 0.362 | 0.460 | 0.430 | 0.517 | 0.314 | 0.431 | 0.324 | 0.434 | 0.327 | 0.437 |  |
| 720 | 0.354 | 0.465 | 0.289 | 0.420 | 0.288 | 0.421 | 0.351 | 0.437 | 0.403 | 0.498 | 0.387 | 0.488 | 0.486 | 0.545 | 0.582 | 0.617 | 0.418 | 0.504 | 0.505 | 0.542 | 0.472 | 0.535 |  |
|  | AVG | 0.275 | 0.400 | 0.255 | 0.385 | 0.253 | 0.385 | 0.285 | 0.409 | 0.297 | 0.416 | 0.292 | 0.413 | 0.354 | 0.451 | 0.440 | 0.524 | 0.311 | 0.424 | 0.333 | 0.431 | 0.326 | 0.431 |
| Average | 0.336 | 0.384 | 0.336 | 0.380 | 0.322 | 0.372 | 0.349 | 0.377 | 0.347 | 0.370 | 0.359 | 0.373 | 0.396 | 0.413 | 0.461 | 0.454 | 0.428 | 0.420 | 0.429 | 0.412 | 0.416 | 0.405 |  |
| $1^{\text{st}}$ Count | 3 | 10 | 28 | 2 | 11 | 10 | 1 | 4 | 0 | 0 | 1 |  |  |  |  |  |  |  |  |  |  |  |  |

Setup. Time series foundation models have recently demonstrated impressive zero-shot learning capabilities (Liang et al., ). In this section, we conducted experiments on the six well-known long-term forecasting benchmarks for which datasets were *not* included in the pre-training corpora.
We use four different prediction horizons, which are $\{96,192,336,720\}$, with the corresponding input time series lengths $\{512,1024,2048,3072\}$. The evaluation metrics adopt mean square error (MSE) and mean absolute error (MAE).

Results. Detailed results of zero-shot forecasting are in Table 3. Time-MoE *achieves consistent state-of-the-art performances, improving a large margin as MSE reduction in average exceeding 20% over the other most competitive baselines.* Importantly, as the model size scales (e.g., base $\rightarrow$ ultra), it continuously exhibits enhanced performance across all datasets, affirming the efficacy of scaling laws within our time series foundation models. Furthermore, in comparisons with robust baselines that have a similar number of activated parameters,
Time-MoE demonstrates significantly superior performance.
The largest models among the state-of-the-art baselines are Chronoslarge, Moment and Moirailarge. Compared to those models, Time-MoE achieved average MSE reductions of 23%, 30% and 11% respectively.

### 4.2 In-distribution Forecasting

**Table 4: Full results of in-domain forecasting experiments. A lower MSE or MAE indicates a better prediction. Full-shot results besides Global Temp are obtained from (Liu et al., ). Red: the best, Blue: the 2nd best.**
| Models | Time-MoE (Ours) | Full-shot Time Series Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | Time-MoElarge | Time-MoEultra | iTransformer | TimeMixer | TimesNet | PatchTST | Crossformer | TiDE | DLinear | FEDformer |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| ETTh1 | 96 | 0.345 | 0.373 | 0.335 | 0.371 | 0.323 | 0.365 | 0.386 | 0.405 | 0.375 | 0.400 | 0.384 | 0.402 | 0.414 | 0.419 | 0.423 | 0.448 | 0.479 | 0.464 | 0.386 | 0.400 | 0.376 | 0.419 |
| 192 | 0.372 | 0.396 | 0.374 | 0.400 | 0.359 | 0.391 | 0.441 | 0.436 | 0.436 | 0.429 | 0.421 | 0.429 | 0.460 | 0.445 | 0.471 | 0.474 | 0.525 | 0.492 | 0.437 | 0.432 | 0.420 | 0.448 |  |
| 336 | 0.389 | 0.412 | 0.390 | 0.412 | 0.388 | 0.418 | 0.487 | 0.458 | 0.484 | 0.458 | 0.491 | 0.469 | 0.501 | 0.466 | 0.570 | 0.546 | 0.565 | 0.515 | 0.481 | 0.459 | 0.459 | 0.465 |  |
| 720 | 0.410 | 0.443 | 0.402 | 0.433 | 0.425 | 0.450 | 0.503 | 0.491 | 0.498 | 0.482 | 0.521 | 0.500 | 0.500 | 0.488 | 0.653 | 0.621 | 0.594 | 0.558 | 0.519 | 0.516 | 0.506 | 0.507 |  |
|  | AVG | 0.379 | 0.406 | 0.375 | 0.404 | 0.373 | 0.406 | 0.454 | 0.447 | 0.448 | 0.442 | 0.454 | 0.450 | 0.468 | 0.454 | 0.529 | 0.522 | 0.540 | 0.507 | 0.455 | 0.451 | 0.440 | 0.459 |
| ETTh2 | 96 | 0.276 | 0.340 | 0.278 | 0.335 | 0.274 | 0.338 | 0.297 | 0.349 | 0.289 | 0.341 | 0.340 | 0.374 | 0.302 | 0.348 | 0.745 | 0.584 | 0.400 | 0.440 | 0.333 | 0.387 | 0.358 | 0.397 |
| 192 | 0.331 | 0.371 | 0.345 | 0.373 | 0.330 | 0.370 | 0.380 | 0.400 | 0.372 | 0.392 | 0.402 | 0.414 | 0.388 | 0.400 | 0.877 | 0.656 | 0.528 | 0.509 | 0.477 | 0.476 | 0.429 | 0.439 |  |
| 336 | 0.373 | 0.402 | 0.384 | 0.402 | 0.362 | 0.396 | 0.428 | 0.432 | 0.386 | 0.414 | 0.452 | 0.541 | 0.426 | 0.433 | 1.043 | 0.731 | 0.643 | 0.571 | 0.594 | 0.541 | 0.496 | 0.487 |  |
| 720 | 0.404 | 0.431 | 0.437 | 0.437 | 0.370 | 0.417 | 0.427 | 0.445 | 0.412 | 0.434 | 0.462 | 0.657 | 0.431 | 0.446 | 1.104 | 0.763 | 0.874 | 0.679 | 0.831 | 0.657 | 0.463 | 0.474 |  |
|  | AVG | 0.346 | 0.386 | 0.361 | 0.386 | 0.334 | 0.380 | 0.383 | 0.406 | 0.364 | 0.395 | 0.414 | 0.496 | 0.386 | 0.406 | 0.942 | 0.683 | 0.611 | 0.549 | 0.558 | 0.515 | 0.436 | 0.449 |
| ETTm1 | 96 | 0.286 | 0.334 | 0.264 | 0.325 | 0.256 | 0.323 | 0.334 | 0.368 | 0.320 | 0.357 | 0.338 | 0.375 | 0.329 | 0.367 | 0.404 | 0.426 | 0.364 | 0.387 | 0.345 | 0.372 | 0.379 | 0.419 |
| 192 | 0.307 | 0.358 | 0.295 | 0.350 | 0.281 | 0.343 | 0.377 | 0.391 | 0.361 | 0.381 | 0.374 | 0.387 | 0.367 | 0.385 | 0.450 | 0.451 | 0.398 | 0.404 | 0.380 | 0.389 | 0.426 | 0.441 |  |
| 336 | 0.354 | 0.390 | 0.323 | 0.376 | 0.326 | 0.374 | 0.426 | 0.420 | 0.390 | 0.404 | 0.410 | 0.411 | 0.399 | 0.410 | 0.532 | 0.515 | 0.428 | 0.425 | 0.413 | 0.413 | 0.445 | 0.459 |  |
| 720 | 0.433 | 0.445 | 0.409 | 0.435 | 0.454 | 0.452 | 0.491 | 0.459 | 0.454 | 0.441 | 0.478 | 0.450 | 0.454 | 0.439 | 0.666 | 0.589 | 0.487 | 0.461 | 0.474 | 0.453 | 0.543 | 0.490 |  |
|  | AVG | 0.345 | 0.381 | 0.322 | 0.371 | 0.329 | 0.373 | 0.407 | 0.409 | 0.381 | 0.395 | 0.400 | 0.405 | 0.387 | 0.400 | 0.513 | 0.495 | 0.419 | 0.419 | 0.403 | 0.406 | 0.448 | 0.452 |
| ETTm2 | 96 | 0.172 | 0.265 | 0.169 | 0.259 | 0.183 | 0.273 | 0.180 | 0.264 | 0.175 | 0.258 | 0.187 | 0.267 | 0.175 | 0.259 | 0.287 | 0.366 | 0.207 | 0.305 | 0.193 | 0.292 | 0.203 | 0.287 |
| 192 | 0.228 | 0.306 | 0.223 | 0.295 | 0.223 | 0.301 | 0.250 | 0.309 | 0.237 | 0.299 | 0.249 | 0.309 | 0.241 | 0.302 | 0.414 | 0.492 | 0.290 | 0.364 | 0.284 | 0.362 | 0.269 | 0.328 |  |
| 336 | 0.281 | 0.345 | 0.293 | 0.341 | 0.278 | 0.339 | 0.311 | 0.348 | 0.298 | 0.340 | 0.321 | 0.351 | 0.305 | 0.343 | 0.597 | 0.542 | 0.377 | 0.422 | 0.369 | 0.427 | 0.325 | 0.366 |  |
| 720 | 0.403 | 0.424 | 0.451 | 0.433 | 0.425 | 0.424 | 0.412 | 0.407 | 0.391 | 0.396 | 0.408 | 0.403 | 0.402 | 0.400 | 1.730 | 1.042 | 0.558 | 0.524 | 0.554 | 0.522 | 0.421 | 0.415 |  |
|  | AVG | 0.271 | 0.335 | 0.284 | 0.332 | 0.277 | 0.334 | 0.288 | 0.332 | 0.275 | 0.323 | 0.291 | 0.332 | 0.280 | 0.326 | 0.757 | 0.610 | 0.358 | 0.403 | 0.350 | 0.400 | 0.304 | 0.349 |
| Weather | 96 | 0.151 | 0.203 | 0.149 | 0.201 | 0.154 | 0.208 | 0.174 | 0.214 | 0.163 | 0.209 | 0.172 | 0.220 | 0.177 | 0.218 | 0.158 | 0.230 | 0.202 | 0.261 | 0.196 | 0.255 | 0.217 | 0.296 |
| 192 | 0.195 | 0.246 | 0.192 | 0.244 | 0.202 | 0.251 | 0.221 | 0.254 | 0.208 | 0.250 | 0.219 | 0.261 | 0.225 | 0.259 | 0.206 | 0.277 | 0.242 | 0.298 | 0.237 | 0.296 | 0.276 | 0.336 |  |
| 336 | 0.247 | 0.288 | 0.245 | 0.285 | 0.252 | 0.287 | 0.278 | 0.296 | 0.251 | 0.287 | 0.280 | 0.306 | 0.278 | 0.297 | 0.272 | 0.335 | 0.287 | 0.335 | 0.283 | 0.335 | 0.339 | 0.380 |  |
| 720 | 0.352 | 0.366 | 0.352 | 0.365 | 0.392 | 0.376 | 0.358 | 0.349 | 0.339 | 0.341 | 0.365 | 0.359 | 0.354 | 0.348 | 0.398 | 0.418 | 0.351 | 0.386 | 0.345 | 0.381 | 0.403 | 0.428 |  |
|  | AVG | 0.236 | 0.275 | 0.234 | 0.273 | 0.250 | 0.280 | 0.257 | 0.278 | 0.240 | 0.271 | 0.259 | 0.286 | 0.258 | 0.280 | 0.258 | 0.315 | 0.270 | 0.320 | 0.265 | 0.316 | 0.308 | 0.360 |
| Global Temp | 96 | 0.192 | 0.328 | 0.192 | 0.329 | 0.189 | 0.322 | 0.223 | 0.351 | 0.215 | 0.346 | 0.250 | 0.381 | 0.219 | 0.349 | 0.272 | 0.406 | 0.223 | 0.352 | 0.221 | 0.354 | 0.261 | 0.392 |
| 192 | 0.238 | 0.375 | 0.236 | 0.375 | 0.234 | 0.376 | 0.282 | 0.404 | 0.266 | 0.393 | 0.298 | 0.418 | 0.269 | 0.395 | 0.305 | 0.435 | 0.278 | 0.401 | 0.257 | 0.388 | 0.299 | 0.423 |  |
| 336 | 0.259 | 0.397 | 0.256 | 0.397 | 0.253 | 0.399 | 0.313 | 0.431 | 0.313 | 0.430 | 0.315 | 0.434 | 0.319 | 0.435 | 0.352 | 0.468 | 0.330 | 0.440 | 0.294 | 0.418 | 0.341 | 0.454 |  |
| 720 | 0.345 | 0.465 | 0.322 | 0.451 | 0.292 | 0.426 | 0.393 | 0.488 | 0.468 | 0.536 | 0.407 | 0.497 | 0.452 | 0.526 | 0.508 | 0.562 | 0.485 | 0.544 | 0.380 | 0.479 | 0.359 | 0.469 |  |
|  | AVG | 0.258 | 0.391 | 0.251 | 0.388 | 0.242 | 0.380 | 0.303 | 0.419 | 0.316 | 0.426 | 0.318 | 0.433 | 0.315 | 0.426 | 0.359 | 0.468 | 0.329 | 0.434 | 0.288 | 0.410 | 0.315 | 0.435 |
| Average | 0.306 | 0.362 | 0.304 | 0.359 | 0.301 | 0.358 | 0.349 | 0.382 | 0.337 | 0.375 | 0.356 | 0.400 | 0.349 | 0.382 | 0.560 | 0.516 | 0.421 | 0.439 | 0.387 | 0.416 | 0.375 | 0.417 |  |
| $1^{\text{st}}$ Count | 4 | 21 | 33 | 0 | 7 | 0 | 0 | 0 | 0 | 0 | 0 |  |  |  |  |  |  |  |  |  |  |  |  |

Setup. We fine-tune the pre-trained Time-MoE models on the train split of the above-mentioned six benchmarks and set the number of finetuning epochs to only one.

Results. The full results are in Table 4. Time-MoE *exhibits remarkable capabilities, comprehensively surpassing advanced deep time series models from recent years, achieving a MSE reduction of 24% in average.* Fine-tuning on downstream data with only one epoch significantly improves predictive performance, showcasing the remarkable potential of large time series models built on the MoE architecture. Similar to zero-shot forecasting, as the model size increases, the scaling law continues to be effective, leading to continuous improvements in the performance of the Time-MoE.

### 4.3 Ablation Study

**Table 5: Ablation studies. (Left) Average MSE for horizon-96 forecasting across six benchmarks, evaluated with different model components. (Right) Analysis of various multi-resolution forecasting configurations. Further details in Appendix D.1.**
|  | Average MSE |
| --- | --- |
| Time-MoEbase | 0.262 |
| w/o Huber loss | 0.267 |
| w/o multi-resolution layer | 0.269 |
| w/o mixture-of-experts | 0.272 |
| w/o auxiliary loss | 0.275 |

To validate our designs in Time-MoE, we conducted detailed ablation studies on key architectural components and loss functions across all experimental benchmarks, as shown in Table  5.

##### Model Architecture.

Replacing the MoE layers with standard FFNs (w/o mixture-of-experts) led to an average performance drop from $0.262$ to $0.272$, highlighting the performance boost provided by the sparse architecture. A detailed comparison of dense and sparse models is presented in Section  4.4. We retained only the horizon-32 output layer by eliminating the other multi-resolution output layers from the Time-MoEbase, excluding the multi-task optimization (w/o multi-resolution layer). Consequently, we observed that the performance of this modified model was slightly inferior compared to that of the Time-MoEbase.
Additionally, as shown in the right side of Table 5, our default selection of four multi-resolution output projections with receptive horizons of $\{1,8,32,64\}$ results in optimal predictive performance and inference speed. As we reduce the number of multi-resolution output projections, performance consistently declines, and inference speed significantly increases. This demonstrates the rationality of our multi-resolution output projection design.

##### Training Loss.

Models trained with Huber loss outperformed those using MSE loss (w/o Huber loss), due to Huber loss’s superior robustness in handling outlier time points. We also removed the auxiliary loss from the objective function, retaining only the auto-regressive loss (w/o auxiliary loss) while still using the MoE architecture. This adjustment caused the expert layers to collapse into a smaller FFN during training, as the activation score of the most effective expert became disproportionately stronger without the load balance loss. Consequently, the model’s performance was significantly worse than the Time-MoEbase.

### 4.4 Scalability Analysis

##### Dense versus Sparse Models.

To assess the performance and efficiency benefits of sparse architectures in time series forecasting, we replaced the MoE layer with a dense layer containing an equivalent number of parameters as the activated parameters in the MoE layer. Using identical training setup and data, we trained three dense models corresponding to the sizes of the three Time-MoE models. A zero-shot performance comparison between the dense and sparse models is shown in Figure 3. Our approach reduced training costs by an average of 78% and inference costs by 39% compared to dense variants. This clearly demonstrates the advantages of Time-MoE, particularly in maintaining exceptional performance while significantly reducing costs.

##### Model and Data Scaling.

We save model checkpoints at intervals of every 20 billion time points during training, allowing us to plot performance traces for models of different sizes trained on various data scales. The right side of Figure 3 shows that models trained on larger datasets consistently outperform those trained on smaller datasets, regardless of model size. Our empirical results confirm that as both data volume and model parameters scale, sparse models demonstrate continuous and substantial improvements in performance, as well as achieve better forecasting accuracy compared to the dense counterparts under the same
scales.

Figure: Figure 3: Scalability analysis. (Left) Comparison of dense and sparse models in terms of training and inference costs. (Right) Average MSE for 96-horizon forecasting across six benchmarks, comparing Time-MoE and dense models, both trained from scratch with varying data sizes.
Refer to caption: x3.png

##### Training Precision.

We trained a new model, Time-MoEbase (FP32), using identical configurations but with float32 precision instead of bfloat16. As shown in Table 6, the forecasting performance of both models is comparable. However, the bfloat16 model achieves a 12% improvement in training speed and reduces memory consumption by 20% compared to the float32 model. Moreover, the bfloat16 model can seamlessly integrate with flash-attention (Dao, ), further boosting training and inference speed by 23% and 19% respectively.

Table: Table 6: Comparison of BF16 and FP32 in terms of training and inference efficiency. Further details are provided in Table 13 of Appendix D.2. FA denotes flash-attention.

### 4.5 Sparsification Analysis

##### Activation Visualization.

As shown in Figure 4, Time-MoE dynamically activates different experts across various datasets, with each expert specializing in learning distinct knowledge. This leads to diverse activation patterns across datasets from different domains, showcasing Time-MoE’s strong generalization capabilities. The heterogeneous activations indicate that the model adapts its learned representations to the specific characteristics of each dataset, contributing to its great transferability and generalization as a large-scale time series foundation model.

Figure: Figure 4: Gating scores for experts across different layers in the six benchmarks.
Refer to caption: x4.png

##### Number of Experts.

**Table 7: Performance and inference speed across different topk setups. Average MSE for horizon-96 forecasting evaluated across six benchmarks. Lower values of inference speed (s/iter) indicate better performance.**
| Time-MoEbase | Average MSE | Inference Speed |
| --- | --- | --- |
| w/ {Top1} | 0.264 | 0.082 s/iter |
| w/ {Top2} | 0.262 | 0.095 s/iter |
| w/ {Top4} | 0.262 | 0.109 s/iter |
| w/ {Top6} | 0.265 | 0.120 s/iter |
| w/ {Top8} | 0.269 | 0.129 s/iter |

We performed a sensitivity analysis on the number of experts, represented as topk, within the Time-MoE architecture, as shown in Table 7. As $k$ increases, performance shows only marginal changes, with minimal improvements in average MSE. However, inference time increases noticeably as more experts are utilized. This indicates that increasing sparsity within the MoE architecture does not compromise performance but significantly enhances computational efficiency. This balance is critical for scaling time series foundation models, where optimizing performance and computational cost is essential. Sparse MoE architectures inherently offer advantages in these areas.

## 5 Conclusion

In this paper, we introduced Time-MoE, a scalable and unified architecture for time series foundation models that leverages a sparse design with mixture-of-experts to enhance computational efficiency without compromising model capacity. Pre-trained on our newly introduced large-scale time series dataset, Time-300B, Time-MoE was scaled to 2.4 billion parameters, with 1.1 billion activated, demonstrating significant improvements in forecasting accuracy. Our results validate the scaling properties in time series forecasting, showing that Time-MoE consistently outperforms dense models with equivalent computational budgets across multiple benchmarks. With its ability to perform universal forecasting and superior performance in both zero-shot and fine-tuned scenarios, Time-MoE establishes itself as a state-of-the-art solution for real-world forecasting challenges. This work paves the way for future advancements in scaling and enhancing the efficiency of time series foundation models.

## Appendix A Further Related Work

In this section, we delve deeper into the related work on large time series models. Current research efforts in universal forecasting with time series foundation models can be broadly classified into three categories, as summarized in Table 8: (1) *encoder-only models*, such as Moirai (Woo et al., ) and Moment (Goswami et al., ), which employ masked reconstruction and have been pre-trained on datasets containing 27B and 1B time points, respectively, with model sizes reaching up to 385M parameters; (2) *encoder-decoder models*, exemplified by Chronos (Ansari et al., ), which offers pre-trained models at four scales, with up to 710M parameters; and (3) *decoder-only models*, including TimesFM (Das et al., ), Lag-Llama (Rasul et al., ), and Timer (Liu et al., ), with the largest models containing up to 200M parameters. In contrast to these dense models, Time-MoE introduces a scalable, unified architecture with a sparse mixture-of-experts design, optimized for larger time series forecasting models while reducing inference costs. Trained on our Time-300B dataset, comprising over 300B time points, Time-MoE is scaled to 2.4B parameters for the first time. It outperforms existing models with the same number of activated parameters, significantly enhancing both model efficiency and forecasting precision, while avoiding limitations such as fixed context lengths or hardcoded heuristics.

**Table 8: Comparison between large time series models.**
| Method | Time-MoE | Moirai | TimesFM | Moment | Chronos | Timer | Lag-Llama | TimeGPT |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Architecture | Decoder- | Encoder- | Decoder- | Encoder- | Encoder- | Decoder- | Decoder- | Encoder- |
| Only | Only | Only | Only | Decoder | Only | Only | Decoder |  |
| (Max) Model Size | 2.4B | 311M | 200M | 385M | 710M | 67M | 200M | Unknown |
| Input Token | Point | Patch | Patch | Patch | Point | Patch | Point | Patch |
| Dataset Scale | 309B | 27B/231B^* | 100B | 1.13B | 84B | 28B | 0.36B | 100B |
| Max Context Length | 4096 | 5000 | 512 | 512 | 512 | 1440 | 1024 | Unknown |
| FFN | Sparse | Dense | Dense | Dense | Dense | Dense | Dense | Dense |
| Open-source Data | ✓ | ✓ |  |  | ✓ | ✓ |  |  |
| Source | Ours |  |  |  |  |  |  |  |
| ^* Depend on the way of calculation according to the original paper. |  |  |  |  |  |  |  |  |

## Appendix B Implementation Details

##### Training Configuration.

Each model is trained for 100,000 steps with a batch size of 1,024, and a maximum sequence length capped at 4,096. This setup processes 4 million time points per iteration. We use forecast horizons of $\left\{1,8,32,64\right\}$ in the output projection and set the auxiliary loss factor $\alpha$ to 0.02. For optimization, we apply the AdamW optimizer with the following hyperparameters: $\text{lr}=1\mathrm{e}\text{-}{3}$, $\text{weight_decay}=1\mathrm{e}\text{-}{1}$, $\beta_{1}=0.9$, and $\beta_{2}=0.95$. A learning rate scheduler with a linear warmup for the first 10,000 steps, followed by cosine annealing, is used. Training is performed on 128 $\times$ NVIDIA A100-80G GPUs with BF16 precision. To improve batch processing efficiency and handle varying sequence lengths, we employ sequence packing (Raffel et al., ), which reduces padding requirements.

##### Benchmark Details.

We evaluate the performance of various models for long-term forecasting across eight well-established datasets, including the Weather (Wu et al., ), Global Temp (Wu et al., ), and ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2) (Zhou et al., ). A detailed description of each dataset is provided in Table 9.

**Table 9: Detailed dataset descriptions. Dataset sizes are listed as (Train, Validation, Test).**
| Tasks | Dataset | Dim | Series Length | Dataset Size | Frequency | Forecastability$\ast$ | Information |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | ETTm1 | 7 | {96, 192, 336, 720} | (34465, 11521, 11521) | 15min | 0.46 | Temperature |
|  | ETTm2 | 7 | {96, 192, 336, 720} | (34465, 11521, 11521) | 15min | 0.55 | Temperature |
| Long-term | ETTh1 | 7 | {96, 192, 336, 720} | (8545, 2881, 2881) | Hourly | 0.38 | Temperature |
| Forecasting | ETTh2 | 7 | {96, 192, 336, 720} | (8545, 2881, 2881) | Hourly | 0.45 | Temperature |
|  | Weather | 21 | {96, 192, 336, 720} | (36792, 5271, 10540) | 10 min | 0.75 | Weather |
|  | Global Temp | 1000 | {96, 192, 336, 720} | (12280, 1755, 3509) | Hourly | 0.78 | Temperature |

##### Metrics.

We use mean square error (MSE) and mean absolute error (MAE) as evaluation metrics for time-series forecasting. These metrics are calculated as follows:

$$ MSE $\displaystyle=\frac{1}{H}\sum_{i=1}^{H}(x_{i}-\widehat{x}_{i})^{2},$ MAE $\displaystyle=\frac{1}{H}\sum_{i=1}^{H}|x_{i}-\widehat{x}_{i}|,$ $$

where $x_{i},\widehat{x}_{i}\in\mathbb{R}$ are the ground truth and predictions of the $i$-th future time point.

##### Multi-resolution Forecasting.

To construct the multi-resolution forecasting head, we define $P$ output projections, each corresponding to a distinct forecasting horizon, denoted as $\left(p_{1},p_{2},\ldots,p_{P}\right)$. The output projection for horizon $p_{j}$ is used to forecast the subsequent $p_{j}$ time steps, as follows:

$$ $\hat{\mathbf{X}}_{t+1:t+p_{j}}=\mathbf{W}_{p_{j}}\mathbf{h}^{L}_{t},$ (12) $$

where $\mathbf{W}_{p_{j}}\in\mathbb{R}^{p_{j}\times D}$ is the learnable parameter matrix for that horizon, and $\mathbf{h}^{L}_{t}$ represents the output hidden state from the last MoE Transformer block. All output projections are optimized simultaneously during model training.

During inference, we apply a greedy scheduling algorithm for arbitrary target output lengths $H$, as outlined in Algorithm 1. For each forecast operation in the auto-regressive process, we select a projection $p_{j}$ with the closest forecasting horizon that does not exceed the remaining forecast duration. This approach allows Time-MoE to extend predictions beyond the next immediate time step or fixed horizon, significantly improving both the model’s utility and overall forecasting accuracy.

Figure: Algorithm 1 Scheduling for the Multi-resolution Forecasting

## Appendix C Processed Data Archive

Going beyond previous work (Ansari et al., ; Woo et al., ; Liu et al., ), we organized a comprehensive large-scale time series dataset from a vast collection of complex raw data. To ensure data quality, we addressed issues by either imputing missing values or discarding malformed time series. Inspired by data processing techniques from large language models (Penedo et al., ; Computer, ; Jin et al., ), we developed a fine-grained *data-cleaning pipeline* specifically designed for time series data:

##### Missing Value Processing.

In time series data, missing values often appear as ‘NaN’ (Not a Number) or ‘Inf’ (Infinity). While previous studies commonly address this by replacing missing values with the mean, this may distort the original time series pattern. Instead, we employ a method that splits the original sequence into multiple sub-sequences at points where missing values occur, effectively removing those segments while preserving the integrity of the original time series pattern.

##### Invalid Observation Processing.

In some data collection systems, missing values are often filled with 0 0 or another constant, leading to sequences with constant values that do not represent valid patterns for the model. To address this, we developed a filtering method that uses a fixed-length window to scan the entire sequence. For each window, we calculate the ratio of first-order and second-order differences, discarding the window if this ratio exceeds a pre-specified threshold (set to 0.2 in our case). The remaining valid continuous window sequences are then concatenated into a single sequence. This process transforms the original sequence into multiple sub-sequences, effectively removing segments with invalid patterns.

Following the processing steps described above, we compiled a high-quality time series dataset named Time-300B, which spans a range of sampling frequencies from seconds to yearly intervals, encompassing a total of 309.09 billion time points. To optimize memory efficiency and loading speed, each dataset is split into multiple binary files, with a metafile providing details such as the start and end positions of each sequence. This setup allows us to load the data using a fixed amount of memory during training, preventing memory shortages. Datasets like Weatherbench, CMIP6, and ERA5 are particularly large, often leading to data imbalance and homogenization. To mitigate these issues, we apply down-sampling to these datasets. During training, we utilized approximately 117 billion time points in Time-300B, sampling each batch according to fixed proportions of domains and distributions of observation values.

Below, we outline the key properties of the datasets after processing, including their domain, sampling frequency, number of time series, total number of observations, and data source. Also, we present the KEY COMPONENT’S SOURCE CODE of the data-cleaning pipeline in Algorithm 2.

**Table 10: Datasets and key properties from Time-300B. For frequency: S = second, T = minute, H = hour, D = day, B = business day, W = week, M = month, Q = quarter, Y = year.**
| Dataset | Domain | Freq. | # Time Series | # Obs. | Source |
| --- | --- | --- | --- | --- | --- |
| Electricity (15 min) | Energy | 15T | 347 | 39,708,170 | Godahewa et al. () |
| Electricity (Weekly) | Energy | W | 318 | 49,608 | Godahewa et al. () |
| ERCOT Load | Energy | H | 152 | 1,238,832 | ourownstory () |
| Australian Electricity | Energy | 30T | 5 | 1,153,584 | Godahewa et al. () |
| Solar Power | Energy | 4S | 26 | 5,248 | Godahewa et al. () |
| Wind Farms | Energy | T | 43,246 | 39,705,317 | Godahewa et al. () |
| BDG-2 Bear | Energy | H | 215 | 1,422,320 | Emami et al. () |
| BDG-2 Fox | Energy | H | 179 | 2,285,288 | Emami et al. () |
| BDG-2 Panther | Energy | H | 136 | 893,840 | Emami et al. () |
| BDG-2 Rat | Energy | H | 455 | 4,596,080 | Emami et al. () |
| Borealis | Energy | H | 17 | 82,757 | Emami et al. () |
| Buildings900K | Energy | H | 2,464,188 | 15,124,358,211 | Emami et al. () |
| BDG-2 Bull | Energy | H | 464 | 501,832 | Wang et al. () |
| BDG-2 Cockatoo | Energy | H | 4 | 17032 | Wang et al. () |
| Covid19 Energy | Energy | H | 1 | 31,912 | Wang et al. () |
| Elecdemand | Energy | 30T | 1 | 17,520 | Godahewa et al. () |
| GEF12 | Energy | H | 20 | 788,280 | Wang et al. () |
| GEF17 | Energy | H | 8 | 140,352 | Wang et al. () |
| BDG-2 Hog | Energy | H | 152 | 365,304 | Wang et al. () |
| IDEAL | Energy | H | 225 | 1,253,088 | Emami et al. () |
| KDD Cup 2018 | Energy | H | 3,054 | 922,746 | Godahewa et al. () |
| KDD Cup 2022 | Energy | 10T | 8,554 | 2,332,874 | Zhou et al. () |
| London Smart Meters | Energy | 30T | 24,132 | 160,041,727 | Godahewa et al. () |
| PDB | Energy | H | 1 | 17,520 | Wang et al. () |
| Residential Load Power | Energy | T | 79,508 | 404,832,695 | Bergmeir et al. () |
| Residential PV Power | Energy | T | 248,888 | 184,238,228 | Bergmeir et al. () |
| Sceaux | Energy | H | 1 | 34,223 | Emami et al. () |
| SMART | Energy | H | 5 | 95,709 | Emami et al. () |
| Spanish | Energy | H | 1 | 35,064 | Wang et al. () |
| Exchange Rate | Finance | B | 13 | 56,096 | Ansari et al. () |
| CIF 2016 | Finance | M | 72 | 7,108 | Godahewa et al. () |
| Bitcoin | Finance | D | 29 | 68927 | Godahewa et al. () |
| FRED MD | Finance | M | 104 | 71,624 | Godahewa et al. () |
| NN5 Daily | Finance | D | 220 | 35,303 | Godahewa et al. () |
| Tourism Monthly | Finance | M | 359 | 98,867 | Godahewa et al. () |
| Tourism Quarterly | Finance | Q | 427 | 39,128 | Godahewa et al. () |
| Tourism Yearly | Finance | Y | 419 | 11,198 | Godahewa et al. () |
| COVID Deaths | Healthcare | D | 2 | 364 | Godahewa et al. () |
| Hospital | Healthcare | M | 727 | 55,224 | Godahewa et al. () |
| CDC Fluview ILINet | Healthcare | W | 286 | 220,144 | CDC () |
| CDC Fluview WHO NREVSS | Healthcare | W | 108 | 56,407 | CDC () |
| Project Tycho | Healthcare | W | 588 | 120,183 | van Panhuis et al. () |
| US Births | Healthcare | D | 1 | 7,275 | Godahewa et al. () |
| Weatherbench (Hourly) | Nature | H | 3,984,029 | 74,630,250,518 | Rasp et al. () |
| Weatherbench (Daily) | Nature | D | 301,229 | 3,223,513,345 | Rasp et al. () |
| Weatherbench (Weekly) | Nature | W | 226,533 | 462,956,049 | Rasp et al. () |
| Beijing Air Quality | Nature | H | 4,262 | 2,932,657 | Chen () |
| China Air Quality | Nature | H | 17,686 | 4,217,605 | Zheng et al. () |
| CMIP6 | Nature | 6H | 14,327,808 | 104,592,998,400 | Nguyen et al. () |
| ERA5 | Nature | H | 11,940,789 | 93,768,721,472 | Nguyen et al. () |
| Oikolab Weather | Nature | H | 309 | 615,574 | Godahewa et al. () |
| Saugeen | Nature | D | 38 | 17,311 | Godahewa et al. () |
| Subseasonal | Nature | D | 17,604 | 51,968,498 | Mouatadid et al. () |
| Subseasonal Precipitation | Nature | D | 13,467 | 4,830,284 | Mouatadid et al. () |
| Sunspot | Nature | D | 19 | 45,312 | Godahewa et al. () |
| Temperature Rain | Nature | D | 13,226 | 3,368,098 | Godahewa et al. () |
| Weather | Nature | D | 9,525 | 26,036,234 | Ansari et al. () |
| Dominick | Sales | D | 3,712 | 759,817 | Godahewa et al. () |
| Car Parts | Sales | M | 16 | 816 | Godahewa et al. () |
| Favorita Sales | Sales | D | 91,513 | 20,371,303 | Woo et al. () |
| Favorita Transactions | Sales | D | 258 | 81,196 | Woo et al. () |
| Hierarchical Sales | Sales | D | 215 | 114,372 | Mancuso et al. () |
| Restaurant | Sales | D | 155 | 30,289 | Woo et al. () |
| M5 | Sales | D | 14,341 | 5,011,077 | Alexandrov et al. () |
| Mexico City Bikes | Transport | H | 556 | 78,848 | Ansari et al. () |
| Traffic | Transport | H | 1,371 | 14,993,544 | Godahewa et al. () |
| Taxi (Hourly) | Transport | H | 2,433 | 1,762,024 | Ansari et al. () |
| Beijing Subway | Transport | 30T | 552 | 19,872 | Wang et al. () |
| Covid Mobility | Transport | D | 426 | 120,950 | Godahewa et al. () |
| HZMetro | Transport | 15T | 160 | 11,680 | Wang et al. () |
| LargeST | Transport | 5T | 1,208,997 | 4,175,062,621 | Liu et al. () |
| Loop Seattle | Transport | 5T | 1,809 | 33,700,832 | Wang et al. () |
| Los-Loop | Transport | 5T | 3,381 | 6,231,168 | Wang et al. () |
| Pedestrian Counts | Transport | H | 80 | 3,125,914 | Godahewa et al. () |
| PEMS Bay | Transport | 5T | 3,980 | 15,975,920 | Wang et al. () |
| PEMS03 | Transport | 5T | 1,651 | 9,210,432 | Wang et al. () |
| PEMS04 | Transport | 5T | 6,634 | 14,638,784 | Wang et al. () |
| PEMS07 | Transport | 5T | 3,828 | 23,789,760 | Wang et al. () |
| PEMS08 | Transport | 5T | 2,612 | 8,684,480 | Wang et al. () |
| Q-Traffic | Transport | 15T | 46,990 | 257,200,384 | Wang et al. () |
| SHMetro | Transport | 15T | 574 | 41,902 | Wang et al. () |
| SZ-Taxi | Transport | 15T | 156 | 464,256 | Wang et al. () |
| Rideshare | Transport | H | 1,352 | 192,949 | Godahewa et al. () |
| Taxi | Transport | 30T | 96,758 | 40,584,636 | Alexandrov et al. () |
| Traffic Hourly | Transport | H | 1,363 | 14,858,016 | Godahewa et al. () |
| Traffic Weekly | Transport | W | 821 | 78,816 | Godahewa et al. () |
| Uber TLC Daily | Transport | D | 235 | 42,533 | Alexandrov et al. () |
| Uber TLC Hourly | Transport | H | 344 | 510,284 | Alexandrov et al. () |
| Vehicle Trips | Transport | D | 10 | 1,626 | Godahewa et al. () |
| Wiki Daily (100k) | Web | D | 100,001 | 274,099,872 | Ansari et al. () |
| Alibaba Cluster Trace 2018 | Web | 5T | 48,640 | 83,776,950 | Woo et al. () |
| Azure VM Traces 2017 | Web | 5T | 263,928 | 880,648,165 | Woo et al. () |
| Borg Cluster Data 2011 | Web | 5T | 216,636 | 176,650,715 | Woo et al. () |
| Kaggle Web Traffic Weekly | Web | W | 133,388 | 15,206,232 | Godahewa et al. () |
| Extended Web Traffic | Web | D | 161,890 | 332,586,145 | Godahewa et al. () |
| Wiki-Rolling | Web | D | 47,675 | 40,619,100 | Alexandrov et al. () |
| TSMixup 10M | Synthetic | - | 10,968,625 | 8,198,358,952 | Ansari et al. () |
| KernelSynth 1M | Synthetic | - | 1,000,000 | 1,024,000,000 | Ansari et al. () |
| M1 Monthly | Other | M | 8 | 1,047 | Godahewa et al. () |
| M1 Quarterly | Other | 3M | 195 | 9,628 | Godahewa et al. () |
| M1 Yearly | Other | Y | 106 | 3136 | Godahewa et al. () |
| M3 Monthly | Other | M | 799 | 109,538 | Godahewa et al. () |
| M3 Quarterly | Other | 3M | 755 | 36,960 | Godahewa et al. () |
| M3 Yearly | Other | Y | 645 | 18,319 | Godahewa et al. () |
| M4 Daily | Other | D | 4,134 | 9,903,554 | Godahewa et al. () |
| M4 Hourly | Other | H | 415 | 352,988 | Godahewa et al. () |
| M4 Monthly | Other | M | 30,126 | 8,480,953 | Godahewa et al. () |
| M4 Quarterly | Other | 3M | 2,623 | 491,632 | Godahewa et al. () |
| M4 Weekly | Other | W | 293 | 348,224 | Godahewa et al. () |
| M4 Yearly | Other | Y | 106 | 3,136 | Godahewa et al. () |

Figure: Algorithm 2 Sample code of Data-cleaning Pipline

## Appendix D Additional Results

### D.1 Ablation Study

**Table 11: MSE for horizon-96 forecasting across six benchmarks, evaluated with different model components.**
|  | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Global Temp | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | 0.357 | 0.305 | 0.338 | 0.201 | 0.160 | 0.211 | 0.262 |
| w/o Huber loss | 0.365 | 0.309 | 0.344 | 0.205 | 0.163 | 0.217 | 0.267 |
| w/o multi-resolution layer | 0.358 | 0.313 | 0.348 | 0.212 | 0.164 | 0.217 | 0.269 |
| w/o mixture-of-experts | 0.370 | 0.317 | 0.347 | 0.212 | 0.163 | 0.223 | 0.272 |
| w/o auxiliary loss | 0.368 | 0.325 | 0.350 | 0.219 | 0.164 | 0.226 | 0.275 |

As shown in Table 11, replacing the MoE layers with standard FFNs (denoted as “w/o mixture-of-experts ”) led to a noticeable performance decline, with the MSE worsening from $0.262$ to $0.272$. This highlights the significant contribution of the sparse architecture to the model’s overall performance, as its dynamic routing enables more specialized processing of diverse input patterns.

We also conducted experiments by retaining only the horizon-32 forecasting head from the Time-MoEbase (denoted as “w/o multi-resolution layer”), excluding the multi-task optimization. The performance of this modified model was slightly inferior to the complete Time-MoEbase.

**Table 12: Full ablation results for different multi-resolution forecasting configurations.**
|  | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Global Temp | Average MSE | Inference Speed |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | 0.357 | 0.305 | 0.338 | 0.201 | 0.160 | 0.211 | 0.262 | 0.095 s/iter |
| Time-MoEbase w/ {1,8,32} | 0.353 | 0.316 | 0.370 | 0.225 | 0.161 | 0.213 | 0.273 | 0.130 s/iter |
| Time-MoEbase w/ {1,8} | 0.389 | 0.391 | 0.441 | 0.304 | 0.174 | 0.222 | 0.320 | 0.411 s/iter |
| Time-MoEbase w/ {1} | 1.071 | 0.920 | 2.098 | 2.320 | 1.500 | 0.383 | 1.382 | 2.834 s/iter |

As shown in Table 12, the default configuration of four multi-resolution forecasting heads with receptive horizons of ${1,8,32,64}$ delivers optimal predictive performance and inference speed. Reducing the number of heads consistently resulted in decreased performance and longer inference time. This inverse relationship highlights the effectiveness of our multi-resolution forecasting design, striking a balance between accuracy and computational efficiency in a decoder-only forecasting foundation model.

These findings highlight the importance of key architectural components in Time-MoE, such as the mixture-of-experts, multi-task optimization, and multi-resolution forecasting, in delivering state-of-the-art performance in universal time series forecasting.

### D.2 Training Precision Analysis

To optimize model performance and efficiency, we conducted a comparative study examining the impact of numerical precision during training. We trained two versions of our model under identical configurations, with the only difference being the precision: one using bfloat16 and the other using float32. The model trained with float32 precision is referred to as Time-MoEbase w/ FP32.

**Table 13: Full results of the comparison between BF16 and FP32 in terms of training and inference efficiency. FA denotes flash-attention.**
|  | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Global Temp | Average MSE | Training Speed | Inference Speed | Training Memory | Inference Memory |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Time-MoEbase | 0.357 | 0.305 | 0.338 | 0.201 | 0.160 | 0.211 | 0.262 | 0.84 s/iter | 0.095 s/iter | 1.77 GB | 226.70 MB |
| Time-MoEbase w/o FA | 0.357 | 0.305 | 0.338 | 0.201 | 0.160 | 0.211 | 0.262 | 1.09 s/iter | 0.118 s/iter | 1.77 GB | 226.70 MB |
| Time-MoEbase w/ FP32 | 0.358 | 0.303 | 0.342 | 0.198 | 0.158 | 0.208 | 0.261 | 1.24 s/iter | 0.133 s/iter | 2.21 GB | 453.41 MB |

As detailed in Table 6, our analysis reveals that the forecasting performances of these two models are remarkably comparable. This finding is significant as it demonstrates that the use of reduced precision (e.g., bfloat16) does not compromise the predictive capabilities of our model.

However, the similarities in performance belie the substantial differences in computational efficiency and resource utilization:

- •
Training Speed: Notably, the bfloat16 model demonstrates a 12% improvement in training speed compared to its float32 counterpart. This considerable acceleration in the training process can significantly reduce the time-to-deployment for large-scale models and facilitate more rapid experimentation and iteration.
- •
Memory Consumption: In terms of memory usage, the bfloat16 model exhibits superior efficiency, consuming substantially less memory than the float32 model. Specifically, we observed a reduction of 20% in memory usage. This memory optimization is crucial for scaling models to larger sizes or deploying them on memory-constrained hardware.
- •
Compatibility with Advanced Techniques: A key advantage of the bfloat16 model is its seamless integration with advanced optimization techniques. In particular, it can easily be combined with flash-attention (Dao, ), a state-of-the-art attention mechanism designed for better efficiency. This integration results in an additional 23% increase in training speed and a 19% boost in inference speed, further enhancing the already significant performance gains.

The implications of these findings are far-reaching:

- •
Resource Efficiency: The reduced memory footprint and increased training speed of the bfloat16 model translate to more efficient utilization of computational resources, potentially lowering infrastructure costs and energy consumption.
- •
Scalability: The memory savings offered by bfloat16 precision enable the training of larger, more complex models on the same hardware, potentially leading to improved model capabilities without increasing computational requirements.
- •
Faster Development Cycles: The substantial improvements in training speed can accelerate the research and development process, allowing for more rapid prototyping and experimentation.
- •
Inference Optimization: The compatibility with flash-attention not only benefits training but also enhances inference speed, which is crucial for real-time applications and large-scale deployments.

Our experiments show that adopting bfloat16 precision, combined with advanced techniques like flash-attention, provides a compelling balance between model performance, computational efficiency, and resource utilization. These optimizations enable the scalable and efficient deployment of large-scale time series forecasting models without sacrificing predictive accuracy.

## Appendix E Forecast Showcases

To visualize the performance differences among various large-scale time series models, we present the forecasting results of our model, Time-MoE, in comparison to the ground truth across six real-world benchmarks. These benchmarks include ETTh1, ETTh2, ETTm1, ETTm2, Weather, and Global Temp datasets. Alongside Time-MoE’s results, we also show the performance of other large-scale baseline models at different scales, providing a comprehensive view of their comparative capabilities (Figures 5 – 10). In all figures, the context length is set to 512, and the forecast horizon is 96. To enhance clarity and aesthetics, we display the full forecast output, complemented by a portion of the preceding historical input data, ensuring a more intuitive comparison.

The results clearly demonstrate the superiority of Time-MoE over the other foundational models. Its ability to consistently produce more accurate forecasts across a range of datasets underscores the effectiveness of its architecture and design. The performance gains are especially noticeable in long-term prediction scenarios, where Time-MoE’s handling of temporal dependencies proves more robust than its counterparts. These visual comparisons highlight the practical advantages of Time-MoE in large-scale time series forecasting, reinforcing its status as a state-of-the-art model.

Figure: Figure 5: Zero-shot forecasting cases from ETTh1 by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
Refer to caption: x5.png

Figure: Figure 6: Zero-shot forecasting cases from ETTh2 by different models, with forecast horizon 96.
Refer to caption: x6.png

Figure: Figure 7: Zero-shot forecasting cases from ETTm1 by different models, with forecast horizon 96.
Refer to caption: x7.png

Figure: Figure 8: Zero-shot forecasting cases from ETTm2 by different models, with forecast horizon 96.
Refer to caption: x8.png

Figure: Figure 9: Zero-shot forecasting cases from Weather by different models, with forecast horizon 96.
Refer to caption: x9.png

Figure: Figure 10: Zero-shot forecasting cases from Global Temp by different models, with forecast horizon 96.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please check the Weaknesses part!
soundness: 3
strengths: The originality of this work shines through its novel approach to applying MoE architecture in time series forecasting! By adopting a sparse framework, the authors not only boost computational efficiency but also extend the potential of large-scale models in this field! This unique integration of MoE principles, previously effective in fields like NLP and Computer Vision (CV), offers a new perspective, tailored to tackle the distinctive challenges of time series data! 

The work stands out for its rigorous methodology, which supports a reliable, in-depth analysis of the TIME-MOE architecture! The authors carefully describe each component, training process, and evaluation metric! This level of detail strengthens the study's findings, ensuring they are both robust and reproducible, an essential foundation for advancing knowledge and encouraging future work in the field!  

The paper stands out for its clarity, presenting complex ideas in a way that's easy to follow! The authors organize their work with well-defined sections, clear plots, and tables that effectively highlight the model's performance metrics! This structure makes it easier to understand the proposed architecture and broadens the paper's appeal, helping both experts and newcomers engage with the context! The study is particularly important because it tackles a crucial gap in time series forecasting research! By scaling a foundation model up to 2.4 billion parameters and showing significant accuracy gains, the authors demonstrate that larger models can boost performance without overwhelming computational demands! This insight is valuable for practical applications, indicating that organizations can adopt high-performance forecasting models without excessive resource costs! 

The paper's contributions go beyond theoretical insights, opening doors to practical applications across fields like finance, energy, and climate science! TIME-MOE's capability for universal forecasting and flexibility across different time horizons make it a valuable tool for improving decision-making in various industries! This potential for tangible impact highlights the paper's importance and its value to both researchers and industry professionals!
summary: This paper presents a new architecture designed to improve time series forecasting by utilizing large-scale foundation models! It addresses key challenges in time series data, like complexity and distribution shifts, which require strong forecasting solutions across fields such as finance, energy, and climate. TIME-MOE adopts a sparse mixture-of-experts (MoE) approach, activating only a subset of parameters during inference! This reduces computational demands while preserving performance! A notable focus is on scaling laws, which have seen success in language and vision research! By applying these principles to time series forecasting, the authors show that expanding model size and training data leads to substantial improvements, positioning TIME-MOE as a leading model that surpasses traditional dense architectures. The flexible design of TIME-MOE supports variable forecasting horizons and input context lengths, making it suitable for diverse applications! The paper also includes a thorough reproducibility statement detailing the architecture, implementation, and data processing steps, highlighting the authors' dedication to transparency and collaborative research! This work not only advances time series forecasting but also paves the way for future studies on scalable, efficient forecasting architectures!
weaknesses: (1) The originality of the model is somewhat limited by the lack of a detailed comparison with other SOTA models beyond the chosen benchmarks. While the authors report promising improvements, the analysis would benefit from a broader comparative study that includes contemporary forecasting models, such as [1] and [2]. Specifically, it would be helpful to compare aspects like model architecture, data handling techniques, and feature processing methods in these SOTA models to highlight where TIME-MOE uniquely contributes. Additionally, incorporating ablation tests could help clarify the impact of each component in the TIME-MOE architecture, allowing readers to understand which elements drive its performance gains. This would strengthen the claims of superiority over existing models and enhance the overall rigour of the evaluation!

(2) The paper highlights the benefits of sparsity and computational efficiency in the MoE architecture but overlooks certain limitations, notably the risk of underfitting when only a limited number of experts are activated. This issue is particularly relevant in time series forecasting, where the variability across datasets demands a careful approach to expert selection. I suggest that the authors provide an analysis examining how the number of activated experts impacts model performance across diverse time series datasets. This would directly address the concern of potential underfitting and add valuable insights. Additionally, referencing studies like [3] that discuss specific challenges in MoE models would strengthen the discussion!

(3) The experimental evaluation is limited by the narrow selection of datasets, primarily focusing on a restricted set of benchmarks. This approach may not capture the model's performance across diverse time series characteristics. To strengthen the study, I recommend that the authors consider expanding their dataset choices to include more varied features, such as seasonality, trend, and noise levels. Adding datasets like M4, those from the UCI time series repository, the Monash dataset [4], and LOTSA as used in [5] could offer a broader perspective on the model’s adaptability and robustness. Additionally, I encourage the authors to clarify their rationale for selecting the current benchmarks. Specifically, explaining how these datasets reflect a range of time series characteristics would help assess whether the chosen datasets sufficiently represent the intended applications or whether a more comprehensive selection is warranted. This would either justify their current dataset choice or highlight areas where further dataset diversity could enhance the model’s generalizability!

(4) The interpretability of the TIME-MOE model could be significantly strengthened by providing more detailed insights into expert activations for various types of time series inputs. Given the complexity of MoE architectures, it would be valuable for the authors to visualize which experts are activated for specific predictions across diverse input scenarios. Additionally, an ablation study illustrating how different experts contribute to predictions for various data types would offer a clearer understanding of each expert’s role. These additions would not only improve transparency but also increase the model's reliability and applicability in real-world settings!

(5) The paper mentions the efficiency of the TIME-MOE model but lacks specific, detailed analysis on computational costs, which would be essential for understanding its practical impact. I would recommend that the authors provide concrete metrics, such as training time per epoch, GPU memory usage, and inference time across different model sizes. Additionally, a direct comparison of these metrics with baseline models would add valuable context for practitioners, especially those working in resource-constrained environments, who need a clear understanding of the computational demands before implementing the model!

(6) The authors could strengthen their exploration of real-world implications by including a practical case study or detailed scenario demonstrating how TIME-MOE could be applied in a specific industry, such as energy forecasting or financial market prediction. Highlighting potential benefits and challenges within a concrete example would provide a clearer picture of how TIME-MOE fits into current forecasting practices, making the research more relatable and actionable. This approach would help bridge the gap between the theoretical aspects of their work and its practical implementation, enhancing the paper’s overall impact! 



[1] Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting, Lim et al., 2020

[2] N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, Oreshkin et al., 2020

[3] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Shazeer et al. (2017)

[4] Monash Time Series Forecasting Archive. Godahewa et al., 2021

[5] Unified Training of Universal Time Series Forecasting Transformers, Woo et al., 2024

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Expert Routing in MoE Layers: Could you provide more details on the gating mechanism and expert selection criteria? Specifically, how does the model ensure a balanced load across experts, and what mechanisms are in place to prevent routing collapse?
2. Handling Domain Imbalances in Time-300B: Given the wide range of domains in Time-300B, how does TIME-MOE adapt to data imbalances? Are there any mechanisms to ensure that underrepresented domains do not disproportionately impact model performance?
3. Choice of Rotary Positional Embeddings: Could the authors explain the choice of rotary embeddings over other positional encodings? Additionally, have the authors conducted ablation studies to assess the impact of this encoding on forecasting accuracy?
soundness: 3
strengths: 1. Scalability and Efficiency: By using sparse MoE, TIME-MOE achieves improved performance and efficiency, especially for larger models. It reduces the computational cost significantly compared to dense models of similar scale, enabling inference on hardware with lower memory requirements.
2. Multi-Resolution Forecasting: The multi-resolution forecasting component allows the model to handle various forecasting horizons efficiently. This flexibility is critical in real-world applications where forecasting requirements vary widely.
3. Empirical Validation Across Benchmarks: The model shows substantial improvements over state-of-the-art dense models in both zero-shot and fine-tuned settings across multiple time series datasets, providing robust evidence of its performance.
summary: The article titled "TIME-MOE: Billion-Scale Time Series Foundation Models with Mixture of Experts" introduces a scalable, efficient time series forecasting model known as TIME-MOE, designed to handle large datasets with diverse domains and flexible forecasting requirements.
weaknesses: 1. Complexity in Routing and Expert Selection: The process by which TIME-MOE assigns tokens to experts in the MoE layer lacks detailed explanation, particularly regarding the criteria for expert selection and load balancing across experts. Specifically, it would be helpful to include the mathematical formulation of the gating mechanism used to select the top-k experts, along with any thresholds or conditions applied during this process. Furthermore, could the authors clarify the techniques employed to prevent expert collapse, such as auxiliary loss functions or dynamic routing strategies? Including these details would provide a clearer understanding of how TIME-MOE maintains balanced and effective specialization among experts.
2. Dependence on Data Quality in Time-300B: The data-cleaning pipeline for Time-300B is briefly mentioned, but additional details on the metrics used to assess data quality before and after cleaning would provide valuable insights. Could the authors specify which metrics (e.g., missing data rate, outlier detection thresholds, signal-to-noise ratio) were used to evaluate the dataset’s quality at each stage? Additionally, it would be beneficial to understand if different domains in Time-300B required unique preprocessing strategies, given that they may have distinct characteristics (e.g., varying degrees of stationarity, seasonality, or noise levels). For instance, were there specific thresholds or criteria applied differently for stationary versus non-stationary time series, or domain-specific noise reduction techniques? Clarifying these details would help illustrate how the pipeline handles domain-specific data imbalances and noisy observations, which is crucial for the model’s generalizability across diverse applications.
3. Positional Encoding and Temporal Coherence: The paper’s use of rotary positional embeddings is interesting, but it would be beneficial to see a comparative analysis of rotary embeddings against other positional encoding methods (e.g., sinusoidal or learned embeddings) specifically for long-term forecasting. Could the authors provide performance metrics—such as error metrics over extended forecast horizons, robustness to long-range dependencies, and computational efficiency—across these encoding types? Additionally, were there specific challenges or performance trends observed when using rotary embeddings over extended timeframes? For example, did rotary embeddings show advantages or disadvantages in terms of capturing the temporal continuity needed in long-term forecasting, compared to sinusoidal or learned encodings? Providing these comparisons and insights would clarify why rotary embeddings were chosen and how they impact forecast accuracy and computational costs over long sequences.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. While point-wise tokenization is reasonable, neighboring time steps often encode similar information. Did you consider tokenizing multiple time steps as a unit? This might improve context efficiency and extend the effective sequence length.

2. Following from the above, would a hybrid tokenization (mixing single time step and multiple time step tokens) be feasible? Effectively increasing the vocabulary? As far as I see the vocab size is of size 10?

3. Could you present MAE results, particularly for ablations? Seeing those values would help evaluate the robustness of different configurations.

4. Regarding the ablation on loss functions: while L2 loss was tested, other simple losses (like L1) could have been competitive. Did you consider these alternatives before settling on Huber loss?

5. The inference speed results in Table 5 indicate that removing resolution layers increases speed. Could you clarify why retaining multi-resolution layers results in faster inference? This seems counterintuitive.

6. There appears to be some variability in performance between different model sizes (e.g., Time-MoE_large performing worse in some 100B cases than dense models). Do the authors have an intuition for why this happens?
soundness: 3
strengths: **Originality**: Introducing a sparse Mixture-of-Experts model for time series forecasting is, as far as I know, novel. This work provides the first large-scale configuration of sparse architectures for time series models, addressing efficiency issues as well as overfitting seen in dense approaches.

**Quality**: The manuscript includes multiple experiments, with good evaluation across zero-shot and fine-tuning settings on multiple (but limited domains) benchmarks (compared to multiple llms and pre-llm models), showcasing the generalizability and effectiveness of the approach. I really like that the authors conducted an scalability analysis with their model as well as the dense counterpart, which can also be seen as a contribution due to its sheer parameter size.

**Clarity**: The paper is clearly written and easy to follow, making it accessible to a broad audience. The use of detailed and easy-to-follow figures (e.g., Figure 2) supports understanding of complex architectural components.

**Significance**: Scaling time series forecasting models has traditionally been challenging due to high computational costs and the lack of proper training data. This work shows significant performance improvements over existing dense models as well as prior LLM-based approaches. Furthermore, due to similarities in its components, it represents a good step towards broadening time series research across research communities (i.e., NLP research).
summary: The paper introduces Time-MoE, a scalable foundation model for time series forecasting that leverages a sparse mixture-of-experts (MoE) architecture. This approach enhances computational efficiency by selectively activating a subset of expert networks for each prediction. The model was pre-trained on a newly proposed large-scale dataset, Time-300B, comprising over 300 billion time points/tokens from nine different domains. Time-MoE's decoder-only structure supports flexible forecasting horizons, and it scales up to 2.4 billion parameters without significant increases in inference costs. The paper demonstrates substantial improvements in forecasting accuracy, positioning Time-Moe as a state-of-the-art solution for time series forecasting.
weaknesses: While I liked the paper there are some areas which can be improved

### Areas for Improvement 
**W1**: The usage of rotary positional encoding is mentioned but not clearly represented in Figure 2. This leads to some confusion (i.e. at a first glance i thought the authors constructe a set transformer); it would be helpful to clarify this, possibly adding an explicit label to ensure better understanding.

**W2**: The recent work by Liu et al. (2024), which proposes a new one-for-all benchmark, is missing. Given its relevance, including this benchmark would provide a more comprehensive assessment of the model’s generalizability. In general, discussing differences in the related work would be a good idea.

**W3**: In my opinion, it is not necessary to prove again that flash attention is fast and BF16 is reasonable to use (the trade-off). These evaluations could be moved to the appendix for interested readers. Instead, I think it would be a good idea to see model performances over the course of the training with different data sampling strategies.

**W4**: There is limited ablation on certain design choices, specifically around the multi-resolution head and expert allocation/configuration strategies. This could strengthen the papers claims (see more in the questions 1,2,4,5)

**W5**: The lack of standard deviations weakens the paper's results and claims

**W6**: The forecastability table needs to proper reference its origin (Wang et al., 2024)

**W7**: I think it is not enough to evaluate on two domains i.e. Temperature and Weather which are already kind of related.

### Limitations
- I did not find any addressed limitations regarding their proposed dataset collection and what impact it would have to have datasets with Spurious correlations  in the collection such as P2S (Kraus et al.,)
- Currently, the decision on which datasets were excluded from the Time-300B collection appears somewhat arbitrary, without strong justification provided for the selection criteria.

Liu et al., AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024)

Wang et al., TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)

Kraus et al., Right on Time: Revising Time Series Models by Constraining their Explanations (2024)

Meta-review: This paper introduces Time-MoE, a mixture-of-experts model for timeseries forecasting. The constituent models operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. Additionally, they introduce a new large-scale dataset called Time-300B, which spans over 9 domains and encompasses over 300 billion time points. 

The reviewers were very enthusiastic about the work and commented on its clarity and their ability to present “complex ideas in a way that's easy to follow” (Ruyh). The main weaknesses that were noted by reviewers focused on the complexity of the routing mechanism and lack of clarity on the pre-processing and other quality metrics needed to produce their Time-300B dataset. However, the authors did a great job of addressing these concerns, and provide a number of new analyses in their revised paper. 

Overall, all reviewers agreed that the contributions of the work were highly significant, with high scores from a majority of reviewers. The introduction of their new dataset, Time-300B, will also be very valuable for the field.

Decision: Accept (Spotlight)