Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
Authors: Xiaoming Shi
Shiyu Wang
Yuqi Nie
Dianqi Li
Zhou Ye
Qingsong Wen
Ming Jin
Venue: ICLR 2025 Spotlight (2025)
Keywords: time series
foundation model
forecasting
Primary area: foundation or frontier models, including LLMs
TL;DR: Time-MoE is a family of time-series foundation models with the mixture-of-experts architecture. For the first time, Time-MoE has scaled up to 2.4 billion parameters, resulting in substantially improved zero-shot/full-shot forecasting performance.

Abstract: Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in language and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecasting models in real-world applications. In response, we introduce Time-MoE, a scalable and unified architecture designed to pre-train larger, more capable forecasting foundation models while reducing inference costs. By leveraging a sparse mixture-of-experts (MoE) design, Time-MoE enhances computational efficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows Time-MoE to scale effectively without a corresponding increase in inference costs. Time-MoE comprises a family of decoder-only transformer models that operate in an auto-regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time-300B, which spans over 9 domains and encompassing over 300 billion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number of activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements position Time-MoE as a state-of-the-art solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility. Code is available at https://github.com/Time-MoE/Time-MoE

Published as a conference paper at ICLR 2025
TIME-MoE: BILLION-SCALE TIME SERIES FOUN- DATION MODELS WITH MIXTURE OF EXPERTS
Xiaoming Shi Shiyu Wang Yuqi Nie Dianqi Li, Zhou Ye, Qingsong Wen Vi Ming Jin
Xiaohongshu Inc Priinceton University "Squirel Ai Learning *Griffith University
sxm mxm//2shhttmii.. com, kwukingg gmaal. com, ysle@princeton.co edu
{dianqili77, yezhou199032, giagsongedu, mingjinedu) com
ABSTRACT
Deep learning for time series forecasting has seen significant advancements over the past decades. However, despite the success of large-scale pre-training in lan- guage and vision domains, pre-trained time series models remain limited in scale and operate at a high cost, hindering the development of larger capable forecast- ing models real-world applications. In response, we introduce TIME-MoE, a capable fore-
scalable and unified architecture designed to pre-train larger, casting foundation models while reducing inference costs. By leveraging a sparse mixture-of-exxpeti (MoE) design, TIME-MoE enhances computational efficiency by activating only subset of networks for each prediction, reducing computa- tional load while maintaining high model capacity. This allows TIME-MOBE scale effectively without a corresponding increase in inference costs. TIME-MON comprises a family of decoder-only transformer models that operate in an auto- regressive manner and support flexible forecasting horizons with varying input context lengths. We pre-trained these models on our newly introduced large-scale data Time- 3003, which spans over 9 domains and encompassing over 300 bil- lion time points. For the first time, we scaled a time series foundation model up to 2.4 billion parameters, achieving significantly improved forecasting precision. Our results validate the applicability of scaling laws for training tokens and model size in the context of time series forecasting. Compared to dense models with the same number activated parameters or equivalent computation budgets, our models consistently outperform them by large margin. These advancements posi- tion TIME-MOE as a state-of-the-an solution for tackling real-world time series forecasting challenges with superior capability, efficiency, and flexibility.
Resources:
com / Time MoB/ Time MoE
INTRODUCTION
Time series data is major modality in real-world dynamic systems and applications across various domains (Box et al.] 2013) Zhang et al. 2024: Liang et al., 2024). Analyzing time series data is challenging due to its inherent complexity and distribution shifts, yet it is crucial for unlocking insights that enhance predictive analytics and decision-making. As a key task in high demand, time series forecasting has long been studied and is vital for driving various
cases in fields such as energy, climate, education, quantitative finance, cloud service, and urban computing, (Jin al. 2023 Nie et al.] 2024 Wang et al.) 2023c; Mao et al. 2024). Traditionally, forecasting has been performed in a task-specific, end-to-end manner using either statistical or deep learning models. Despite their competitive performance, the field has not converged on building unified, general-purpose forecasting models until recently, with the emergence of a few foundation models (FMs) for universal forecasting (Das et al. 2024: Woo et al., 2024 Ansari al. 2024). Although promising, they are generally small in scale and have limited task-solving capabilities compared to domain-specific models, limiting their real-world impact when balancing forecasting precision against computational budget.
Equal contribution
Project lead
Corresponding author

---

Published as a conference paper at ICLR 2025
ChroneSaral
Time-MoE (Zero-shot)
0.32
Dense (Few-shot)
Chromosowss
Chrondserge
0.27
Time-MoEE (Few-shot)
Average MSE
0.30
Moiralyyust
Average MSE
0.26-
Moiral
Moiralluoe
0.28
0.25
MoEsse
--- -
0.26
0.24
MoEa-
0.23
0.24
10
100
300
1000
Number activated parameters (millions)
GFLOPs per time point prediction
Figure I: Performance overview. (Left) Comparison between TIME- MoE models and state-of-the- art time series foundation models, reporting the average zero-shot performance across six benchmark datasets. (Right) Comparison of few- and zero-shot performance between TIME-MoR and dense variants, with similar effective FLOPss per time series token, across the same six benchmarks.
Increasing model size and training tokens typically leads to performance improvements, as known as scaling laws, which have been extensively explored in the language and vision domains (Kaplan et al. [2200] Alabdulmohsin et al.) 2022). However, such properties have not been thoroughly in- vestigated in the time series domain (Yao et al., 2025). Assuming that scaling forecasting models with high-quality training data follows similar principles, several challenges remain: Dense versus sparse training. Most time series forecasting models compose dense layers, which means each input time series tokens requires computations with all model parameters. While effective, this is computationally intensive. In contrast, sparse training with mixture-of-experts (MoE) is more flop- efficient per parameter and allows for scaling up model size with a fixed inference budget while giving better performance, as showcased on the right of Figure [ However, optimizing a sparse, large-scale time series model faces another challenge of stability and convergency. Time series are highly heterogeneous (Woo et al.) 2024 Dong et al. 2024), and selecting the appropriate model design and routing algorithm often involves trade-off between performance and computational efficiency. Sparse solutions for time series foundation models have yet to be explored, leaving significant gap in addressing these two challenges. While time series pre-training datasets are no longer major bottleneck, most existing works (Das et al.) 2024; Woo et al. 2024 Ansari et al.) 2024] have not extensively discussed their in-model data processing pipelines or mixing strategies. Answering this is particularly important, given that existing data archives often noisy and largely imbalanced across domains.
On the other hand, most time series FMs face limitations in flexibility and generalizability. General- purpose forecasting is a fundamental capability, requiring a model to handle any forecasting prob- lems, regardless of context lengths, forecasting horizons, input variables, and other properties such as frequencies and distributions. Meanwhile, achieving strong generalizability pushes the bound- aries further that existing works often fail to meet simultaneously. For instance, Timer (Liu al., 2024d) has limited native support for arbitrary output lengths, which may lead to truncated out- puts, while Moment (Goswami al.) 2024) operates with fixed input context length. Although Moirai (Woo et al. 2024) achieves universal forecasting, it depends on hardcoded heuristics in both the input and output layers.
The recognition of the above challenges naturally raises pivotal question:
How scale fime series foundation models achieve universal forecasting while balancing model capability and computational overhead, mirroring the success of foundation models in other domains?
Answering this question drives the design of TIME-MOE, scalable and unified architecture for pre- pre-training larger, more capable forecasting FMs while reducing computational costs. TIME-MoE consists of family of decoder-only transformer models with mixture-of-exppre architecture. op- erating in an ruto-regeessive manner to support any forecasting horizon and accommodate context lengths of up 4096. With its sparsely activated design, TIME-MOE enhances computational ef-
2

---

Published as a conference paper at ICLR 2025
ficiency by activating only a subset of networks for each prediction, reducing computational load while maintaining high model capacity. This allows TIME- MOME-MMEMTN to scale effectively without signifi- cantly increasing inference costs. Our proposal is built on a minimalist design, where the input time series is point-wise tokenized and encoded before being processed by sparse transformer decoder, activating only small subset of parameters. Pre-trained domains and over 300 billion time points, TIME-MoE is optimized through multi-task learning to forecast at multiple resolutions. During inference, different forecasting heads are utilized to enable forecasts across diverse scales, enabling flexible forecast horizons. For the first time, we scale time series FM up to 2.4 billion parameters, achieving substantial improvements in forecasting precision compared to existing models, shown the left of Figure Compared to dense models with the same number of activated parameters or equivalent computational budgets, our models consistently outperform them by a large margin. Our contributions lie in three aspects:
large-scale time series data across
1.
We present TIME-MoE, a universal decoder-only time series forecasting foundation model ar- chitecture with miixture-ff-xxeee To the best of our knowledge. this is the first work to scale time series foundation models up to 2.4 billion parameters. TIME-MoE achieves substantial im- provements in forecasting accuracy and consistently outperforms dense models with comparable computational resources, while maintaining high efficiency.
We introduce Time- 300B, the largest open-access time series data collection, comprising over 300 billion time points spanning more than nine domains, accompanied by a well-designed data- cleaning pipeline. Our TIME-MoE models and Time-300B data collection are open-sourced.
3. Trained on Time--30BB TIME-MoF. models outperform other time series foundation models with a similar number of activated parameters across six real-world benchmarks, achieving re- ductions in forecasting errors by an average of 20% and 24% in zero-shot and in-distribution scenarios, respectively.
RELATED WORK
Time Series Forecasting. Deep learning models have become powerful tools for time series fore-
casting over the past decade, which can be broadly categorized into two types: (1) univariate models,
such DeepState (Rangapuram al.] 2018). DeepAR (Salinas al. 2020), and N-BEATS Orr- shkin et al.) 2020). which focus on modeling individual time series, and (2) multivariate models,
which include both transformer- based approaches (Wen al.) 2023: Zhou et al., 2021; Nie al.)
2023 [iu et al. 2024b; Wang et al. 2024c Chen et 20.244224 Wang et al. 2022) and non-transformer
models (Sen et al.] 2019 Jin al.
2022 Wang et al. 2024b Hu et al. 2024, Qi al. 2024
Wang 2024), designed to handle multiple time series simultaneously. While these models achieve competitive in-domain performance (Wang et al. 2025), many are task-specific and fall short in generalizability when applied to cross-domain data in few-shot or zero-shot scenarios.
Large Time Series Models. Self-supervised learning has been extensively developed for time se- ries (Zhang al. 2024). employing masked reconstruction (Zerveas et al., 2021 Nie et al. 2023) or contrastive learning (Zhang et al. 2022] Wang et al.) (023bb) Yue et al. 2022). However, these methods limited in both data and model scale, with many focused on in-domain learning and transfer. Recently, general pre-training of time series models on large-scale data has emerged (Liang eet al., 20244). though still in its early stages with insufficient exploration into sparse solutions. See Appendix A for more information. Unlike these dense models, TIME-MOE introduces scalable, unified architecture for pre-training larger, more capable forecasting foundation models while main- taining the same scale of activated parameters and computational budget.
Sparse Deep Learning for Time Series. Deep learning models are often dense and over- parameterized (Hoefler et al.] (02T). leading to increased memory and computational demands during both training and inference. However, sparse networks, such as mixture-of-experts mod- els (Jacobs et al.] 1991). which dynamically route inputs to specialized expert networks, have shown comparable or even superior generalization to dense models while being more efficient (Fedus et al.) Riquelme et al. 2021). In time series research, model sparsification has received relatively less attention, as time series models have traditionally been small in scale, with simple models like DLinear (Zeng al. 2023) and SparseTSF (Lin et al.) 2024) excelling in specific tasks prior to the advent of large-scale, general pre- tr-itrniiin.. The most relevant works on this topic include Path- former (Chen et al. 2024). MoLE et al. 2024), and IME (Ismail et al. (023). However, none of
2022
3

---

Published as a conference paper at ICLR 2025
611
613
&y.
Output Patch 1
Output
Multi-tsk
Patch
Optimization
Buyu Monuoaa
FM Head
Head
Add
Output Embeddings
Multi-resolution Scheduling
Mixture-of-Expert
Miltture-of-Expert
Output Hidden
1xx
Output Embeddings
Add RMSNorm
Add RMSNom
M
Causal Attention
Cau sall Attention
km1
Output
Router
Embeddings
RMSNorm
Input Hidden
B
x
Input Layer
Inyer
Swish FEN
Point Tolienzztion
1
Time Tokens
Vu
The architecture of TIME-MOE, which is a decoder-only model. Given an input time series of arbitrary length, ! first tokenize it into sequence of data points, which are then encoded. These tokens are processed through N-stacked backbone layers, primarily consisting of causal multi-head selffattention and a sparse temporal mixture-of-expert layers. During training, A we optimize forecasting heads at multiple resolutions. For model inference, TIME-MoF provides forecasts of flexible length by ( dynamically scheduling these heads. Details about the causal multi- head self-attention are in Appendix B and illustrated in Figure
Figure 2:
them delve into the scalability of foundation models with sparse structures. Besides, MoLE and IME are not sparse models, as input data is passed all heads and then combined make predictions.
3 METHODOLOGY
Our proposed TIME-MoE, illustrated in Figure 2 adopts a mixture-of-cxperee based, decoder-only transformer architecture, comprising three key components: (I) input token embedding. (2) MoE transformer block, and (3) multi-resolution forecasting. For the first time, we scale a sparsely- activated time series model 2.4 billion parameters, achieving significantly better zero-shot perfor- mance with the same computation. This marks major step forward in developing large time series models universal forecasting.
Problem Statement. We address the problem of predicting future values in a time series: given a sequence of historical observations XI:T = objective is to forecast the next time steps, i.e., X441:+* = fo (X1.1) E IR.. Here, fo represents time series model, where T' the context length and H is the forecasting horizon. Notably, both T and H can be flexible during TIME-MoE inference, distinguishing it from task- specific models with fixed horizons. Additionally, channel independence (Nie et al. 2023) adopted to transform a multivariate input into univariate series, allowing TIME-MoN to handle any-variate forecasting problems real-worl! applications.
(T) RR spanning T time steps, our
3.1 TIME-MOE OVERVIEW
Input Token Embedding. We utilize point-wise tokenization for time series embedding to ensure the completeness of temporal information. This enhances our model's flexibility and broad applica- bility in handling variable-length sequences. Then, we employ SwiGLU (Shazeer) 2020) to embed each time series point:
h SwiGLU(<x)) Swish (Wxr) @ (Vx)),
4

---

Published as a conference paper at ICLR 2025
where W RD and V RDx are learnable parameters, and D denotes the hidden dimension.
MoE Transformer Block. Our approach builds upon a decoder-only transformer Vaswani 2017) and integrates recent advancements from large language models (Bai al., 2023 Touvron et al, 2023). We employ RMSNorm (Zhang & Sennrich 2019) to normalize the imput of each transformer sub-layer, thereby enhancing training stability. Instead of using absolute positional encoding, adopt rotary positional embeddings (Su et al. (024), which provide greater flexibility in sequence length and improved extrapolation capabilities. In line with (Chowdhery al., 2023), we remove biases from most layers but retain them in the QKV layer of self-attention to improve extrapolation. To introduce sparsity, we replace a feed-forward network (FFN) with a mixture-of-expers layer, incorporating shared pool of experts that are sparsely activated.
u = SA (RMSNorm (b/-")) + h/-1
(2)
uf RMSNorm (u')
(3)
h Mixture (u!) + u
(4)
Here, SA denotes self-attention with causal mask, and Mixture refers to the mixture-of-experts layer. In practice, Mixture comprises several expert networks, each mirroring the architecture of standard FFN. An individual time series point can be routed to either single expert (Fedus al.
2022) or multiple experts (Lepikhin et al., (020). One expert is designated as shared expert to capture and consolidate common knowledge across different contexts.
Mixture (u!) = IN+1, FFNN+1 (a))
Σ (g FFN, (u!)).
(5)
81,00
81. E (opp(((s.d)) S i NN,K) K),
9i,t
0.
otherwise,
(6)
IN+1,1 Sigmoid (W N+ m!)
81,7 = Softmax; (W'n!),
(8)
where w! e RIXD denotes the trainable parameters, and N and K respectively denote the numbers of non-shared experts and activated non-shared experts per mixture-of-exxpee layer.
Multi-resolution Forecasting. We introduce novel multi-resolution forecasting head, which al- lows for forecasting at multiple scales simultancously, in contrast to existing foundation models that are limited to single fixed scale. This capability enhances TIME-MoE flexibility by abling forecasting across various horizons. The model employs multiple output projections from single-layer FFNs, each designed for different prediction horizons. During training. TIME-MoB aggregates forecasting errors from different horizons to compute composite loss (Section 3.2.2 thereby improving the model generalization. By incorporating simple greedy scheduling algorithm (see Appendix(B B TIME-MoB efficiently handles predictions across arbitrary horizons. This design also boosts prediction robustness through melti-resolution ensemble learning during inference.
3.2
MODEL. TRAINING
3.2.1
TIME-30O8 3C0B DATASET
Training time series foundation models require extensive, high-quality data. Recent advancements have facilitated the collection of numerous time series datasets from various sources (Godahewa al. 202T Ansari et al., 2024 Woo al. 2024] Liu al. 2024d]a), a). Nonetheless, data quality still remains challenge, with prevalent issues such as missing values and invalid observations (Wang al. 2024a) that can significantly impair model performance and destabilize training- To mitigate these issues, we developed streamlined data-cleaning pipeline (Appendix () to filter and refine raw data, and constructed the largest open-access, high-quality time series data collection named Hime-30055 for foundation model pre-training. Time-300 comprises diverse array of publicly available datasets from domains such as energy, retail, healthcare, weather, finance, transportation, and web, augmented with synthetic data to enhance both quantity and diversity. It spans sam- pling frequencies from seconds to yearly intervals and, after processing through our data-cleaning pipeline, includes over 300 billion time points, summarized in Table
5

---

TIME-MOE,, Published as a conference paper at ICLR 2025
Table Key statisties of the pre-training dataset Time- 30CB from various domains.
Energy
Finance
Healthcare
Nature
Sales
Synthetic
Transport
Web
Other
Total
Seqs.
2,875,335
1.715 413.696
1.752 471 K
110,210 26.382 0.008 %
11998.622
622.414 2.130 B
972.158
40,265
18.220.929
Obs.
13.981
279.724
9.222
1.804
20.32 M
309.00 B
Percent%
5.17 %
0
0.0001%
90.50 %
2.98%
0.69 10
0.58 %
0.006
100%
3.2.2 Loss FUNCTION
Pre- Pre-training training time series foundation models in large scale presents significant challenges in training stability due to the massive datasets and the vast number of parameters involved. To address this, we use the Huber loss (Huber) 1992] Wen et al., (209)), which provides greater robustness to outliers and improves training stability:
[ (z, - 2z)²,
if | 2,1 6,
Lar (Ix,
8 (1ore fzl 8) otherwise,
where 8 a hyperparameter that balances the and L.2 loss components.
When training the model with MoE architecture, focusing solely optimizing prediction er- ror often leads to load imbalance issues among experts. A common problem routing col- lapse (Shazeer et al. 2017), where the model predominantly selects only few experts. limiting training opportunities for others. To mitigate this, following the approaches (Dai et al.] [2024 Fe- dus et al. 2022), we achieve expert-level balancing with auxiliary loss to reduce routing collapse:
T
Caux N Dfts f. = KT
Σ I (Time point t selects Expert i). r1 T 1=1
Σ
(10)
where f; represents the fraction of tokens assigned expert i, and r; denotes the proportion of router probability allocated to expert i. is the indicator function. Finally, we combine the auto-regressive losses across all multi-resolution projections with the auxiliary balance loss form the final loss:
L
E Car (Xit1atyy> Xesreets)) F=1
aLCaxx
(11)
where P the number of multi-resolution projections and pj is the horizon of the j-th projection.
3.2.3 MODEL CONHIURRATINN AND TRAINING DETAILS
Informed by the scaling laws demonstrated in Dubey al. 2024 Touvron al.) 2023 which show that a 7- or 8-billion parameter model continues to improve performance even after training on trillion tokens, chose to scale TIME-MoE up 2.4 billion parameters with around billion of them activated. This model, TIME-MOF.ass. supports inference on consumer-grade GPUs with less than 8GB of VRAM. We have also developed two smaller models: TIME-MOF with 50 million activated parameters, and TIME-MOEJane. with 200 million activated parameters, both specifically designed for fast inference on CPU architectures. The detailed model configurations are in Table 2 Each model undergoes training for 100, 000 steps with batch size of 1024, where the maximum sequence length capped at 4096. This setup results in the consumption of million time points per iteration. We choose {1, 8, 32, 64] } as different forecast horizons in the output projection and set the factor of the auxiliary loss a to 0.02. Refer Appendix B for optimization details.
Table 2: A high-level summary of TIME-MoE model configurations.
Layers
Heads
Experts
K
d
d
dagen
Activated Params
Total Params
TIME- MaRe-Morass
12
8
384
1536
50 M
113 M
TIME-MOEuE
8
768
3072
384
200
453 M
8
1024
4096
512
1.1 B
2.4 B

---

Published as a conference paper at ICLR 2025
4
MAIN RESULTS
TIME-MoR consistently outperforms state-of-the-art models by large margins across 6 well- established benchmarks and settings (Appendix [B). To ensure a fair comparison, we adhered to the configurations from (Woo al. 2024) for out-of-distribtioon forecasting and (Wu et al., (023a) for in-distribution forecasting with a unified evaluation pipeline we developed. Specifically, we evaluate TIME-MoN MOE against 16 different baselines, representing state-of-the- afftthe-art art forecasting foun- dation models. They are categorized into two groups: (1) zero-shot forecasting group, includes pre-trained models such as Moirai (2024), TimesFM (2024), Moment (2024), and Chronos (2024); (2) in-distribution (full-shot) forecasting group, consists of up-to-date models such as iTrans- former (2024b), TimeMixer (2024b), TimesNet (2023a), PatchTST (2023), Crossformer (2023),
TiDE (2023), DLinear FEDoorer (2022b). We also include addition comparisons with
Timer (2024d), TFT (2021), and N-BEATS (2020) in Appendix D.3
4.1 ZERO-SHOP FORECASTING
Table 3: Full results of zero-shot forecasting experiments. A lower MSE MAE indicates better prediction. TimesMM, due to its use of Weather datasets in pretraining, is not evaluated on this dataset and is denoted by dash (-). Red: the best, Blue: the 2nd best.
C TIME-MOE (Qurs)
Zere-shot Thnee Series Medels
Models
TIMF-MoF... Ae.
TIMS-Mol bay
TTME-MoRaa
Mairaluee
Mairaloung
TimedFM
Moment
Chronituess
Chriset
Chranen.aa
Metries
MAE
MSS
MAE
MAF
D1357
11:58
1.
017007
11.349
11.77
01:01
0.903
(0: 175
0 1.90
0.361
0.345
CC414
0-404
DANK
0.557
(0-666
0.399
11:400 [1:M1
(0,54]
0:350
ETTM
192
0.384
01.014
1.355
0,512
0.295
11.13
0.435
0-34
0.42
0.412
0.43
0.434
0,455
0.45
0.465
0.434
0.68%
0.560
0.530
0.46
0.450
11292
0.426
00,502
0/424
0.449
0.477
0.427
0.455
01457
0.462
0.439
0.611
0.985
0.355
0.615
01822
0.553
0.400
07:24
01394
0.419
0.412
01425
0.33%
0.227
10.481
0,419
0:480
0.419
0.415
0:445
0685
0,358
03565
0.472
0.991
01465
0.584
0.456
1.100
0.354
11.292
0.311
0.33
(6:706
0.310
10.115
0.340
13214
0:186
10100
0.316
0.54
(0.330
0.345
ETT1
0.391
0.385
14177
0.385
0.347
0.379
0,365
0,351
0.390
0.356
0.42
0425
130
(1419
01:54
0.557
0.996
10:33
01447
0.11
0.226
00.115
0.411
0.425
0.41%
0.445
0.454
0.395
0.712
0.604
0.511
0.501
01477
(.001
0,511
Aug
0.766
11
06405
0.415
0057
11:00
11361
01:34
00355
0.367
0.377
0.362
0:806
0:361
0.459
01424
01:30
11:05
01410
0,855
0/427
10.338
0,309
0.367
0.415
0.392
0,360
0.361
0.42
6T0ml
15g (1:66
01:13
117
0.08
0L569
0.305
0.355
0.91
0-411
0-13
00416
11.95
0.416
0.000
0-479
0662
DAT2
0.532
0.517
0.68
0.534
0.567
03677
01505
0.461
0.450
150
(1.504
11:00
1475
0.77
11:80
11477
11:66
0-331
0.460
11.15%
0.462
0.270
0.112
0-471
0,492
0.151
0.348
0.956
(1W0
(1:00
0.000
0.536
Avg.
0.394
0.415
0.33
0.405
0.356
0.201
0.435
0440
0.406
0.355
0,422
0.39]
0.433
0.418
0.678
0.536
0.640
0.299
01645
01500
0,585
CJ465
0.201
6T7m2
197
176
(1.1%%
01534
11373
115
1.40
0375
0329
121
1.203
11312
1L345
11311
0.364
0.963
10.714
11 190
0.515
0.361
0.341
0.31%
0.355
0.749
340
0 366
0.769
0.350
0.799
0.761
0.15%
0:54
0.900
0.361
1153
0,557
01766
01254
(1311
720
(3488
0.464
1480
0.661
0,22
0.425
0,802
0.415
0.437
0.411
0.455
0.425
0.462
0.430
0.384
0.409
0.553
0.299
[1,55
0.429
0.A16
C4115
Avg.
0.317
0.365
0.316
0.361
0.25
0.344
0.307
0.347
00.311
0.307
0.329
0.343
0.338
0.346
0L316
0.365
01349
0.380
0,310
0.350
00:100
01:14
1.191
0.11
16157
01211
0.155
0.223
0.220
0.217
10.199
0.211
10:455
0.255
0.343
10.01
0133
00.144
0.335
Weather
192
0.311
(3376
0.309
01300
0315
036
0.22
0.255
0.205
11.90
1.24%
0.247
0.243
0.265
0.301
(0277
102266
0.29
0.250
6.374
6.246
0.251
0.251
0.77% 0.379
0.306
0.346
0.321
0.263
0.539
0.251
11.5%
11514
112500
U336
01340
0.365
0.337
0.418
0.405
0.15
0.400
0.405
0.597
0.313
0.354
0.337
0.397
0,507
0.596
11.11
11.333
0.310
0.343
00214
0.345
0.237
0.354
00.234
0.351
10.234
0.35)
10.255
0.375
10365
0,422
10.354
0.36)
11230
11355
10.225
0.354
Globel Trmp
192 0.257
01281
11:055
0.405
1151
0.385
0.266
11.36
0.379
0.269
0.292
0.499
0.5%
0.266
0.354
0.267
0.477
0.355
0.315
0.455
0.430
0.387
0.517
0.489
0.314
0.276
0.000
0.304
11273
U.55
11.766
0.396
0.375
1774
0.385
0.255
11.85
0.405
0.290
0.415
0.314
0/400
0.526
01424
0333
Average
0.336
08.844
0.356
0.380
0.323
173
0.349
0.377
(0.341
0.370
0,359
0.373
0.356
0.413
0.461
0.854
(0/425
11:200
11.429
01412
0,416 0/406
Count
11
Setup. Time series foundation models have recently demonstrated impressive zero-shot learning capabilities (Liang et al., 2024: Liu al. (202c)). In this section, we conducted experiments on the six well-known long-term forecasting benchmarks for which datasets were not included in the pre-training corpora. We use four different prediction horizons, which are {96, 192, 336, 7203, with the corresponding input time series lengths (512, 1024, 2048, 3072). The evaluation metrics adopt mean square error (MSE) and mean absolute error (MAE).
Results. Detailed results of zero-shot forecasting are in Table 3 TIME-MoE achieves consis- tent state-nf-the-art performances, improving a large margin as MSE reduction in average ex- ceeding 20% over the other most competitive baselines. Importantly, as the model size scales it continuously exhibits enhanced performance across all datasets, affirming the efficacy of scaling laws within our time series foundation models. Further- more, in comparisons with robust baselines that have a similar number of activated parameters, TIME-MoE demonstrates significantly superior performance. The largest models among the state- oh-the-art baselines are Chronostages Moment and Moirai Compared to those models, TIME- MoE achieves average MSE reductions of 23%, 30% and 11 % respectively.
(e.g., TIME-MOEnss + TIME-MOE.tan
7

---

Published as a conference paper at ICLR 2025
Table 4: Full results of in-domain forecasting experiments. A lower MSE or MAE indicates better prediction. Full-shot results besides Global Temp are obtained from (Liu et al. 2024b). Red: the best, Blue: the 2nd best.
TIMF-MoF
Modele
MoMssMoPFaar
TiME-MoFas
ITaanfomer
Timehiiser MSS
TimeeNet
PaceTNT
Croedumner
Diineer
FEDlrrmer
Metries
MSR
MAF.
|MSE
MAR
MSR
MAF.
MSE
MAR
MSR
MAF
MSE
MAF
MSSE
MAF
MSE
MAR]
MSE
MAF]
MSR
MAF
[0.45
0.313
00.355
0,231
0.335
0.365
0.386
0,415
10.315
0.354
0.40C
0.414
[1423
0:448
(0.499
0.464
0.366
0,400
0.37% 04419
192
00.32
0.396
00.544
0.400
0.359
0.391
0,441
0.456
0.436
0.429
0.411
0.429
0.400
0.445
[.47]
0444
0.525
0.492
0.47
0.422
0.43)
0.448
(1
9,412
11.900
0.412
0,388
0.415
0.487
0.458
0.484
0.45%
0.911
0.499
0.501
1-466
0.500
0.546
(0.56
0.515
0.611
0.499
0.499
04465
10:10
0.413
11.012
0.43
9.455
(3450
0.50
0.488
0.521
0.500
01500
8851
0.621
(1.84
0.558
0.519
8.50
0.500
Ang.
11.79
0,850
0.575
0,81
0,454
0.447
0.454
0.450
0.468
0.529
0.522
0.540
0.507
0.455
0.451
0/459
0.226
0.340
10.25
0.335
0.214
0.358
0.297
0.349
10,239
0.341
(0.340
0,374
00302
0.348
0.745
0,584
10,420
0.440
10.313
0.357
03358
0.297
6TTM2
197
11.11
11
0.304
03545
8.5M0
0.362
0.196
0.171
0.900
0.428
0:40
0.313
0.346
0:900
0.400
0.459
0.11
0.541
0:426
0.588
14400
1877
0:751
0,066
13641
[0.578
0.971
8.500
0.417
0.541
0:476
8:49
0.49
0.419
0.482
11,414
0.411
0.43
0.417
0.427
0.462
0.657
07763
0.874
0.679
3.463
0.474
11:300
0.314
11.244
11
6.256
0.133
0.354
6. 3.388
(6. 1.770
0.357
0.375
0.115
0.77M
1.367
0:436
10.354
0.387
0.345
0.112
0.77M 01419
11:00
0.318
11:41
110
8.281
0.145
0.77
1.561
0.187
0.567
0.590
0:451
0.004
0.360
8:476
0.641
0.350
0.376
0.138
0.390
0.400
0411
0.515
0.428
0.455
04133
0.409
0.400
[1:22
0.351
0L:22
0.371
0119
0.341
0:400
0.405
01400
0:495
0:419
0,41
0.405
0.406
01448
04452
11:72
0.265
11.199
0.259
0.73
01190
0.264
17.
0.187
0.267
11175
0.364
10.207
0.305
0.119
0.252
0.287
0301
0.309
0.34
8414
6492
0.250
0.364
0.328
0.345
0.293
0.541
6.278
0.311
0.321
0.360
0.414
(0:41
01411
0475
0.454
0.417
0.407
0.46
0.40%
1.700
0.51%
0.931
0.554
0.112
9:421
0.415
Aug
[1.271
0.315
11284
013
0.377
0.35
00388
0.315
0.315
0.701
0.113
1.700
117
0.500
0.358
0.403
0.190
0.400
0:164
0.100
[1151
0.203
0.149
0.201
0.154
0.208
0.174
[0.16
0.172
0:220
0.17
0.18
0.188
6:230
0.202
0.261
0.000
0.117
0.200
Weater
334
11:07
0.34%
0.345
0.255
0.25
0.274
0.356
0.351
0.306
0.77
0.773
0.358
0:367
0.335
0.363
0.115
0.35
0.180
720
01352
0.366
0.352
0.365
0.392
03
0.558
0.340
0.339
0.341
0.365
0.359
0.354
0.345
0.598
6/18
0.351
0.386
0.345
0.381
0.403
0.425
Aug.
[1226
0.275
0.234
0.233
0.250
0.280
0.257
0.278
0.240
0.271
0.259
0.286
0.258
01280
01258
0.315
0.270
0.320
0.265
0.316
0.308
0.360
Global Tomp
336
192
0.259
11250
0.375
0.397
11:30
0.256
0.375
0.397
0.354
6,253
0.399
0.313
0.431
0.313
0.366
0.430
0.315
0.434
0.418
0.201
0.319
0455
13305
0.552
0.435
0468
0.27%
0.330
0.440
0.257
0.204
0.318
0.488
0.34
0.2M
0.454
0.425
720
(8.45
0.465
11:22
0.451
0.292
8.435
0.503
0.468
0.466
0.536
0.407
0.497
DA52
0.526
0.503
0.582
0.455
0.544
0.360
0.479
0.359
0.499
Avp
11:00
0.391
1.:4
0.342
0.161
11301
0.419
0.316
0.476
0.11N
0.435
0.315
01676
11309
01008
0.379
0.414
0.768
0:410
0:115
01435
Average
00300
0.362
0
0.384
0.356
0.400
0,516
0,421
0,439
0.367
0.375
01417
1" Crant
21
35
11
0
01
11
4.2 IN-DISTRIBUTION FORECASTING
Setup. We fine-tune the pre-trained TIME-MOE models on the train split of the above-mentione six benchmarks and set the number of finetuning epochs to only one.
The full results are in Table TIME-MOE exhibits remarkable capabilities, comprehen- sively surpassing advanced deep time series models from recent years, achieving a MSE reduction of 24% in average. Fine-tuning on downstream data with only one epoch significantly improves pre- dictive performance, showcasing the remarkable potential of large time series models built on the MoE architecture. Similar to zero-shot forecasting, as the model size increases, the scaling law con- tinues to be effective, leading to continuous improvements in the performance of the TIME-MoE.
Results.
4.3 ABLATION STUDY
Table 5: Ablation studies. (Left) Average MSE for horizon-96 forecasting across six benchmarks, evaluated with different model components. (Right) Analysis of various multi-resolution forecast- ing configurations. More details are Appendix D.1.
Average MSE
Averuge MSE
Inference Speed
TIME- MOE
w/o multi resolution layer
w/o Haber Joss
0.262
0.269
0.267
TIME-MoERus w/ (1.8)
TIME-MOFNs: w/ (18.32)
TOME-MOEiss (1,8.32,64)
0.320
0,273
0.262
0.095 xttee
0.411 s/ier
0.130 s/ier
w/o offeuneeoffeexptts
w/o suxiliary loss
0.272
0.275
TOME-MoENss w/ (1)
1.382
2.834 s/iter
To validate our designs in TIME-MOE, we conducted detailed ablation studies key architectural components and loss functions across all experimental benchmarks, as shown in Table
Model Architecture. Replacing the MoE layers with standard FFNs (w/o mixture-of-experts) led to average performance drop from 0.262 to 0.272, highlighting the performance boost provided by the sparse architecture. A detailed comparison of dense and sparse models is presented in Sec- tion 44 We retained only the horizon-32 output layer by eliminating the other multi-resolution output layers from the TIME-Mo excluding the multi-task optimization (w/o multi-resolution layer). Consequently, we observed that the performance of this modified model was slightly inferior
8

---

Published as a conference paper at ICLR 2025
30
Training cost (x109 TFLOPs)
24
0.34
Deniseaase Densearge
0.261
Inference cost (s/iter)
5
4,8
4,8
DensCutra
Average MSE
Time MaseeMoCaase
4-
0.30
Time-MoEuuge
MoEwer
0.28
0,550
0.147
2
0.26
1.2
00,095 *
0.24
5.30
0.22
Dense (200M)
Time-MoE (bees)
Dense (18)
Time-Mot (Large)
1008
2008
3008
4008
Training Time Points
Figure 3: Scalability analysis. (Left) Comparison of dense and sparse models in terms of training and inference costs. (Right) Average MSE for 96-horizon forecasting across six benchmarks, com- paring TIME-MOE and dense models, both trained from scratch with varying data sizes.
compared to that of the TIMEMMEEase: Additionally, as shown in the right side of Table 5] our default selection of four results in optimal predictive performance and inference speed. As we reduce the number of multi- resolution output projections, performance consistently declines, and inference speed significantly increases. This demonstrates the rationality of our multi-resolution output projection design.
output projections with receptive horizons of {1, 8, 64)
Training Loss. Models trained with Huber loss outperformed those using MSE loss (w/o Huber loss), due to Huber loss's superior robustness in handling outlier time points. We also removed the auxiliary loss from the objective function, retaining only the auto-regressive loss (w/o auxiliary loss) while still using the MoE architecture. This adjustment caused the expert layers collapse into smaller FFN during training, as the activation score of the most effective expert became disproportionately stronger without the load balance loss. Consequently, the model's performance was significantly worse than the TIME-MoEE.e
4.4 SCALABILITY ANALYSIS
Dense versus Sparse Models. To assess the performance and efficiency benefits of sparse archi- tectures in time series forecasting, we replaced the MoE layer with dense layer containing an equivalent number of parameters as the activated parameters in the MoE layer. Using identical training setup and data, we trained three dense models corresponding to the sizes of the three TIME- MoE models. A zero-shot performance comparison between the dense and sparse models is shown Figure Our approach reduced training costs by an average of 78% and inference costs by 39% compared to dense variants. This clearly demonstrates the advantages of TIME-MoE, particularly in maintaining exceptional performance while significantly reducing costs.
Model and Data Scaling. We save model checkpoints intervals every 20 billion time points during training, allowing to plot performance traces for models of different sizes trained on various data scales. The right side Figure 3 shows that models trained on larger datasets consistently outperform those trained smaller datasets, regardless of model size. Our empirical results confirm that as both data volume and model parameters scale, sparse models demonstrate continuous and substantial improvements in performance, as well as achieve better forecasting accuracy compared to the dense counterparts under the same scales.
Training Precision. We trained a new model, MOME-MEnasse (FP32), using identical configura- tions but with float32 precision instead of bfloat16. As shown in Table the forecasting performance of both models is comparable. However, the b6loat16 model achieves a 12% improvement in train- ing speed and reduces memory consumption by 20% compared to the float32 model. Moreover, the b model can seamlessly integrate with flash-attention (Dao 2024), further boosting training and inference speed by 23 % and 19 % respectively.

---

Published as a conference paper at ICLR 2025
Table 6: Comparison BF16 and FP32 in terms of training and inference efficiency. FA denotes
flash-attention. More details are in Table 13 of Appendix D.2
Average MSE
Training Speed
Inference Speed
Training Memory
Inference Memory
MM-Basc MoFeMe w/o FA
0.262
0.84 s/iter
0.095 siter
1.77 GB
226. MB
1.09 s/iter
viter
GB
226.70
TIME- TIME-MOERso FP32
0.261
1.24 s/iter
0.133 s/iter
2.21 GB
453.41 MB
4.5 SPARSIFICATION ANALYSIS
Activation Visualization. As shown Figure TIME-MoE dynamically activates different ex- perts across various datasets, with each expert specializing in learning distinct knowledge. This leads to diverse activation patterns across datasets from different domains, showcasing strong generalization capabilities. The heterogeneous activations indicate that the model adapts its learned representations to the specific characteristies of each dataset, contributing to its great trans- ferability and generalization as large-scale time series foundation model.
ETTh1
ETTh2
ET1m1
ETTm2
Weather
Global Temp
0.5
0.4
0.2
0.1
0.0
Experts
Experts
Experts
Experts
Experts
Experts
Figure 4: Gating scores experts across different layers the six benchmarks.
Number of Experts. We performed sensitivity analysis on the number of experts, represented as tops. within the TiME-MoE architecture, as shown in Table 7 As k increases, performance shows only marginal changes, with minimal improvements average MSE. However, inference time increases ticeably as more experts are utilized. This indicates that increasing sparsity within the MoE architecture does not compromise performance but significantly enhances computational efficiency. This balance is critical for scaling time series foundation models, where optimizing performance and computational cost is essential. Sparse MoE architectures inherently offer advantages in these areas.
Table 7: Performance and inference speed across different topı setups. Average MSE for horizon-96 forecasting evaluated across six benchmarks. Lower values of inference speed (s/iter) indicate better performance.
TIME-MoEis.:
Average MSE
Inference Speed
w/ (Top:) w/ Tops)
0.264
0.082 s/iter
0.262
0.095 s/iter
w/ (Tpp4
0.262
0.109 s/iter
w/ (Topa]
0.265
0,120 s/iter
w/ (Tops)
0.269
0.129 s/iter
5 CONCLUSION
In this paper, we introduced TIME-MoE, scalable and unified architecture for time series foun- dation models that leverages a sparse design with mixture-of-expert to enhance computational ef- ficiency without compromising model capacity. Pre-trained on our newly introduced large-scale time series dataset, me- 300R, TIME-MoE was scaled to 2.4 billion parameters, with 1. billion activated, demonstrating significant improvements in forecasting capabilities. Our results validate the scaling properties in time series forecasting, showing that TIME-MoE consistently outperforms dense models with equivalent computational budgets across multiple widely accepted benchmarks. With its ability to perform universal forecasting and superior performance in both zero-shot and fine- tuned scenarios, TIME-Mo MMME-MMBsC establishes itself as state-of-the the-art solution for real- aal-world forecasting challenges. This work lays the groundwork for future advancements in scaling and enhancing the efficiency of time series foundation models, paving the way toward time series general intelligence.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Y. Nie acknowledges financial support from Princeton Language and Intelligence at Princeton Uni- versity. M. Jin was supported in part by the NVIDIA Academic Grant Program and CSIRO National Science Foundation (US) AI Research Collaboration Program.
REFERENCES
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-2233 2022.
Alexander Alexandrov, Konstantinos Benidis, Michael SchnkerSe Valentin Flunkert, Jan Gasthaus. Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Tåkarkmen, and Yuyang Wang. Gluonts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 21(116):1-6, 2020.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur. Syama Sundar Rangapuram, Sebastian Pineda Arango. Shubham Kapoor, al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07881 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,
Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Christoph Bergmeir, Quang Bui. Frits de Nijs, and Peter Stuckey. Residential power and battery data, August 2023. URL htttts:://ooi.orrtttrrrr
5281/2enodo 8219786
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
CDC. Flu portal dashboard, 2017. Eliportallashboard html
URL httpss//giss.cd..goo//aaa//
Peng Chen, Yingying Zhang, Yunyao Cheng. Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Pathformer: Multi-scale transformers with adaptive pathways for time series forecasting. In International Conference on Learning Representations, 2024.
Song Chen. Beijing Multi-Site Air-Quality Data. UCI Machine Learning Repository, 2019. DOI: http:://dbb.rg//022433//555555
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https: /tg/ttttubbbb/g wtt.ah/..cmmoooggg..eerrrorrrrrr
Data
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint S0KV::2400000060 2024.
Tri Dao. F Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations (ICLR), 2024.
Abhimanyu Das, Weihao Kong, Andrew Leach. Shaan K Mathur, Rajat Sen, and Rose Yu. Long- term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Re- search, 2023.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. decoder-only only foundation model for time-series forecasting. In Forty-furst International Conference on Machine Learning, 2024.

---

Published as a conference paper at ICLR 2025
Zheng Dong, Renhe Jiang, Haotian Gao, Hangchen Liu, Jinliang Deng, Qingsong Wen, and Xuan Song. seaprogeeniit--iffmmen meta-parameter learning for spatiotemporal time series forecast- ing. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 631-641, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Lerman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Patrick Emami, Abhijeet Sahu, and Peter Graf. Buildingsbench: A large-scale dataset of 900k build- ings and benchmark for short-term load forecasting. Advances in Neural Information Processing Systems, 36:19823-1998 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 2022.
Azul Garza, Cristian Challu, and Max Mergenthaler-Caassco.. Timegpt-1. arXiv:2310.03589, 2023.
arXiv preprint
Rakshitha Wathsadini Godahewa, Christoph Bergmeir, Geoffrey Webb, Rob Hyndman, and Pablo Montero-Manno. Monash time series forecasting archive. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https:
//openreview.. wwt.//ooouueeeeee d=wEcl JU-
Georg Goerg. Forecastable component analysis. ICML. 2013.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In Forty-first International Conference on Machine Learning. 2024.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
Jiaxi Hu, Yuehong Hu, Wei Chen, Ming Jin, Shirui Pan, Qingsong Wen, and Yuxuan Liang. At- tractor memory for long-term time series forecasting: A chaos perspective. arXiv preprint arXiv:2402.11463 2024.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodol- ogy and distribution, pp. 492-518. Springer, 1992.
Aya Abdelsalam Ismail, Sercan O Arik, Jinsung Yoon, Ankur Taly, Soheil Feizi, and Tomas Pfister. Interpretable mixture of experts. Transactions on Machine Learning Research, 2023. ISSN 2835- 8856.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Ming Jin, Yu Zheng, Yuan-Fang Li, Siheng Chen, Bin Yang, and Shirui Pan. Multivariate time series forecasting with dynamic graph neural odes. IEEE Transactions on Knowledge and Data Engineering, 35(9):9168 (10))916899999 9180, 2022.
Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Yi Wang, Haifeng Chen, Xiaoli Li, et Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 2023.
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui Pan, and Qingsong Wen. Position: What can large language models tell us about time series analysis. In Forty-first International Conference on Machine Learning, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
12

---

Published as a conference paper at ICLR 2025
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 1001::0006.1666,, 2020.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6555- 6565, 2024.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Plister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4): 2021.
Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. Sparsetsf: Modeling long-term time series forecasting with Ik parameters. In Forty-first International Conference on Machine Learning. 2024.
Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang, Zhenguang Liu, Bryan Hooi, and Roger Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting. arXiv preprint 220i::23000002559 2023.
Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. Moirai-moe: Empowering time series foundation models with sparse mixture of experts. arXiv preprint arXiv:2410. 10469, 2024a.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth Inter- national Conference on Learning Representations, 2024b.
Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Autotimes: Au- toregressive time series forecasters via large language models. arXiv preprint arXiv:2402.02370, 2024c.
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In Forty-first Interna- tional Conference on Machine Learning, 2024d.
Paolo Mancuso, Veronica Piccialli, and Antonio M Sudoso. A machine learning approach for fore- casting hierarchical time series. Expert Systems with Applications, 182:115102 2021.
Shengzhong Mao, Chaoli Zhang, Yichi Song, Jindong Wang, Xiao-Jun Zeng, Zenglin Xu, and Qing- song Wen. Time series analysis for education: Methods, applications, and future directions. arXiv preprint arXiv:2408. 13960, 2024.
Soukayna Mouatadid, Paulo Orenstein, Genevieve Elaine Flaspohler, Miruna Oprescu, Judah Co- hen, Franklyn Wang, Sean Edward Knight, Maria Geogdzhayeva, Samuel James Levang, Ernest Fraenkel, and Lester Mackey. SubseasonalclimateUS.. A dataset for subseasonal forecasting and benchmarking. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Tung Nguyen, Jason Kyle Jewik, Hritik Bansal, Prakhar Sharma, and Aditya Grover. Climateleam:: Benchmarking machine learning for weather and climate modeling. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Ronghao Ni, Zinan Lin, Shuaiqi Wang, and Giulia Fanti. Mixture-of-linear linee-----e for long-term time series forecasting. In International Conference on Artificial Intelligence and Statistics, pp- 4672-4680. PMLR, 2024.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-tem forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M Mulvey, H Vincent Poor, Qingsong Wen, and Stefan Zohren. A survey of large language models for financial applications: Progress, prospects and challenges. arXiv preprint arXiv:2406. 2024.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural ba- sis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
ourownstory, Neuralprophet datasets, 2023. URL http:::/githuu..ccmmmoreeee neuralprophet data
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01/11 133306001166 2023.
Shiyi Qi, Zenglin Xu, Yiduo Li, Liangjian Wen, Qingsong Wen, Qifan Wang, and Yuan Qi. Pde- time: Rethinking long-term multivariate time series forecasting from the perspective of partial differential equations. arXiv preprint arXiv:2402.16913 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14)):1-67, 2020.
2020.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(1 20220000200,,,,,,,,,,,,, 2020.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin, Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag-Ilama: Towards foundation models for time series forecasting, 2023.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8555 2021.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International journal of forecasting, 36(3): 181- 1191, 2020.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach high-dimensional time series forecasting. Advances in neural information processing systems, 32, 2019.
N Shazeer, A Mirhoseini, K Maziarz, A Davis, Q Le, G Hinton, and Dean. The sparsely-gated mixture-of-eepprts layer. Outrageously large neural networks, 2017.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXhv:2002.05202, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1377, 2023,
Willem G van Panhuis, Anne Cross, and Donald S Burke. Project tycho 2.0: a repository to improve the integration and reuse of data for global population health. Journal of the American Medical Informatics Association, 25:1 160--1617 2018
14

---

Published as a conference paper at ICLR 2025
Ashish Vaswani. Attention is all you need. arXiv preprint
2017.
Jingyuan Wang, Jiawei Jiang, Wenjun Jiang, Chengkai Han, and Wayne Xin Zhao. Towards effi- cient and comprehensive urban spatial-tmmpora prediction: A unified library and performance benchmark. arXiv preprint arXiv:2304. 14343, 2023a.
Jun Wang, Wenjie Du, Wei Cao, Keli Zhang, Wenjia Wang, Yuxuan Liang, and Qingsong Wen. Deep learning for multivariate time series imputation: A survey, arXiv preprint arXiv:2402.04059, 2024a.
Shiyu Wang. Neuralreconeiler for hierarchical time series forecasting. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 731-739, 2024.
Shiyu Wang, Fan Zhou, Yinbo Sun, Lintao Ma, James Zhang, and Yangfei Zheng. End-to-end
modeling of hierarchical time series using autoregressive transformer and conditional normalizing flow-based reconciliation. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1087-1094. IEEE, 2022.
Shiyu Wang. Yinbo Sun, Xiaoming Shi, Zhu Shiyi, Lin-Tao Ma, James Zhang, YangFei Zheng, and Liu Jian. Full scaling automation for sustainable development of green data centers. In Proceedings of the Thirty-Secodd International Joint Conference on Artificial Intelligence, pp- 6264 6271, 2023b.
Shiyu Wang, Yinbo Sun, Yan Wang, Fan Zhou, Lin-Tao Ma, James Zhang, and YangFei Zheng- Flow-based end-to-end model for hierarchical time series forecasting via trainable attentive- reconeiliation. In International Conference on Database Systems for Advanced Applications, pp. 167-176. Springer, 2023c.
Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou Timemixer: Decomposable multiscale mixing for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024b.
Shiyu Wang, Jiawei Li, Xiaoming Shi, Zhou Ye, Baichuan Mo, Wenze Lin, Shengtong Ju, Zhixua Chu, and Ming Jin. Timemixer++: A general time series pattern machine for universal predictive analysis. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Xue Wang, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. In The Twelfth International Confer- ence on Learning Representations (ICLR), 2024c.
Zhixian Wang, Qingsong Wen, Chaoli Zhang, Liang Sun, Leandro Von Krannichfeldt, and Yi Wang. Benchmarks and custom package for electrical load forecasting. arXiv preprint arXiv:2307.07191, 2023d.
Qingsong Wen, Jingkun Gao, Xiaomin Song. Liang Sun, and Jian Tan. RobustTrend: huber loss with a combined first and second order difference regularization for time series trend filtering. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp. 3856-3862, 2019.
Qingsong Wen, Tian Zhou Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: survey. In Proceedings of the Thirty-Second Second International Joint Conference on Artificial Intelligence (IJCAI), pp. -7778-6766 2023.
Gerald Woo, Chenghao Liu, Akshat Kumar, and Doyen Sahoo. Pushing the limits of pre-training for time series forecasting in the cloudops domain. arXiv preprint 1r.iv02300005066 2023.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In Forty-first International Conference on Machine Learning, 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
15

---

Published as a conference paper at ICLR 2025
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d variation modeling for general time series analysis. In International Conference on Learning Representations, 2023a.
Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nature Machine Intelligence, 2023b.
Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, and Shirui Pan. Towards neural scaling laws for time series foundation models. The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8980-899 1880-8987,0 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Fick- hoff. A transformer- framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 2114-2124 2021.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Junbo Zhang. Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Kexin Zhang. Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Y Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, et al. Self-supervised learning for time series anal- ysis: Taxonomy, progress, and prospects. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised con- trastive pre-training for time series via time- fene-frreeency consistency. Advances in Neural Informa- fion Processing Systems, 45:3988-4000 2022.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations, 2023.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Fore- casting fine-grained air quality based on big data. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 2267-2276, 2015.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Jingbo Zhou, Xinjiang Lu, Yixiong Xiao, Jiantao Su, Junfu Lyu, Yanjun Ma, and Dejing Dou. Sdwpf: A dataset for spatial dynamic wind power forecasting challenge at kdd cup 2022. arXiv preprint arxiv:2208. 043600, 2022a.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022), 2022b.
16

---

Published as a conference paper at ICLR 2025
A FURTHER RELATED WORK
In this section, we delve deeper into the related work on large time series models. Current research efforts in universal forecasting with time series foundation models be broadly classified into three categories, as summarized Table (1) encoder-only models, such as Moirai (Woo et al.) 2024) and Moment (Goswami al.] 2024). which employ masked reconstruction and have been pre-trained on datasets containing 27B and IB time points, respectively, with model sizes reaching up 385M parameters; (2) encoder-decoder models, exemplified by Chronos (Ansari al. 2024), which offers pre-trained models at four scales, with to 7 710M parameters: and (3) decoder-only models, including TimesFM (Das al.] 2024), Lag-Llama (Rasul et al. 2023). and Timer (Liu et al. 2024d). with the largest models containing up to 200M parameters. The concurrent work, Moiral-MoE (Liu et al. 2024a). includes up 935M parameters but with a different expert and routing design. In contrast to these models, TIME-MoË introduces scalable, unified architecture with a sparse mixture-of-eret -expeersss design, optimized for larger time series forecasting models while reducing inference costs. Trained
Time- -anee-3300 300B dataset, comprising over 300B time points, TIME-MOE is scaled to 2.4B parameters for the first time. It outperforms existing models with the same number of activated parameters, significantly enhancing both model efficiency and forecasting precision, while avoiding limitations such as fixed context lengths or hardcoded heuristics.
Table 8: Comparison between large time series models.
Method
Time-Mob MoB
Moirai
TimesFM
Moment
Chronos
Timer
Ag-Llama
TimeGPT
Decoder-
Encoder-
Decoder-
Encoder-
Encoder-
Decoder-
Decoder-
Encoder-
Architecture
Only
Only
Only
Only
Decoder
Only
Only
Decoder
(Max) Model Size
2.4B
311M
200M
385M
710M
67M
200M
Unknown
Input Token
Point
Patch
Patch
Patch
Point
Patch
Point
Parch
Dataset Scale
309B
27B/231B"
100B
1,13B
84B
28B
0.36B
100B
Max Length
4096
5000
512
512
512
1440
1024
Unknown
FFN
Sparse
Dense
Dense
Dense
Dense
Dense
Dense
Dense
Open-source Data
Source
Ours
Woo
Das
Guswami Ansari al.
Liu al.
Rasul et
Garza
Depend on the way of calculation according to the original paper. indicates the total of the context and prediction lengths.
B IMPLEMENTATION DETAILS
Training Configuration. Each model is trained for 100,000 steps with a batch size of 1,024, and maximum sequence length capped at 4,096. This setup processes 4 million time points per iteration. We use forecast horizons {1, 8, 32, 64] in the output projection and set the auxiliary loss factor to 0.02. For optimization, = 1e-3, weight decay = 1c-1, 3 = 0.9, and B₂ = 0.95. A learning rate scheduler with linear warmup for the first 10,000 steps, followed by cosine annealing, is used. Training is performed 128 x NVIDIA A 1100-80G GPUs with BF16 precision. To improve batch processing efficiency and handle varying sequence lengths, we employ sequence packing (Raffel et al. 2022),, which reduces padding requirements.
apply the AdamW W optimizer with the following hyperparameters:
Benchmark Details. We evaluate the performance of various models for long-term forecasting across eight well cell-established datasets, including the Weather (Wu et al. (022). Global Temp (Wu (et al. 2023b), and ETT datasets (ETThI, ETTh2, ETTml, ETTm2) (Zhou et al. 2021). A detailed description of each dataset is provided in Table
17

---

Published as a conference paper at ICLR 2025
Table 9: Detailed dataset descriptions. Dataset sizes are listed as (Train, Validation, Test).
Tasks
Dataset
Dim
Series Length
Dataset Size
Frequency
Forecastability
Information
ETTml
7
(96, 192, 336, 720)
(34465, 11521, 11521)
15min
0.46
Temperature
ETTm2
7
(96, 192, 336, 720)
(34465, 11521, 11521)
15min
0.55
Temperature
Long-term
ETTh1
[96, 192, 336, 720)
(8545, 2881, 2881)
Hourly
0.38
Temperature
Forecasting
ETTh2
|
7
[96, 192, 336, 720]
(8545, 2881, 2881)
Hourly
0.45
Tempenature
Weather
21
(96, 192, 336, 720)
(36792, 5271, 10540)
10 min
0.75
Weather
Global Temp
1000
(96. 192, 336, 720)
(12280, 1755, 3509)
Hourly
0.78
Temperature
The forecastability is calculated by one minus the entropy of Fourier decomposition of time series larger value indicates better predictability.
Goergg 2013
Metrics. We use mean square error (MSE) and mean absolute error (MAE) evaluation metrics for sime-series forecasting. These metrics are calculated as follows:
H MSE
H MAE
Σ
- 2,
H i=1
H 1=1
where Ft, T. E are the ground truth and predictions of the i-th future time point.
Technical Details. Our mixture-of-experts layer consists of one shared expert and several isolated experts, each represented by feedforward network that is smaller than the standard FFN employed in dense models. In the formulation from Equations to 8 FFNN+1 denotes the shared expert, while FFN, to FFNN n correspond to the isolated experts. The weight JN+1.1 associated with the shared expert for token / is normalized using the Sigmoid function. In contrast, the weight Ji, for the i-th isolated expert of token t is normalized using the Softmax function. Furthermore, we retain only the top-k largest scores among the isolated experts and set the remaining scores to zero.
To prevent routing collapse among experts, we adopt the strategy proposed by (Fedus al., 2022), incorporating an auxiliary loss to ensure balanced expert load. The key aspect of this method is penalize experts with high gating scores. This helps prevent a scenario where stronger experts, being exposed to more tokens, become even stronger while weaker experts continue to fall be- hind. The mathematical formulation presented in Equa- tion 10 where fi represents the fraction of tokens as- signed to expert i, and r; denotes the proportion of router probability allocated to expert i. If one expert is assigned too many tokens and achieves a higher routing score, it will incur correspondingly higher loss.
Linear
Concat
Heads
Scaled Dot-Product Attention
RoPE
RoPE
Multi-resolution Forecasting. To construct the multi- resolution forecasting head, we define P output projec- tions, each corresponding to a distinct forecasting hori- zon, denoted (p1, P2. .........0 ..)). The output projection for horizon py is used to forecast the subsequent py time steps, as follows:
Linear
Linear
Linear
K
Figure Causal attention layer.
X bf
(12)
where Wp, E TR,, XD the learnable parameter matrix for that horizon, and hf represents the output hidden state from the last MoE Transformer block. All output projections are optimized simultaneously during model training.
During inference, we apply greedy scheduling algorithm for arbitrary target output lengths H, as outlined in Algorithm|] [ For each forecast operation in the rugo-regressive process, we select pro- jection P, with the closest forecasting horizon that does not exceed the remaining forecast duration. This approach allows TIME-MoE to extend predictions beyond the next immediate time step or fixed horizon, significantly improving both the model's utility and overall forecasting accuracy.
18

---

Published as a conference paper at ICLR 2025
Algorithm 1 Scheduling for the Multi-resolution Forecasting
Require: Target output length H, forecast horizon of each output projection {P1, P2. ... pp) in ascending order
Ensure: Combined output length = H, P1 =
1: H + 0
2: J J++ 0
3: while < H do
4: for j P down to do
5:
if H + P. - H then
6:
H H + Pi
7:
add pj to J 8: break
9: end if
10: end for
11:
end while
12:
return
C PROCESSED DATA ARCHIVE
Going beyond the previous work (Ansari et al. 2024 Woo al. 2024 Liu et al. 2024d).
organized a comprehensive large-scale time series dataset from a vast collection of complex raw data. We utilize the missing value ratio and the invalid observation ratio as metrics assess the quality the dataset. These two metrics can effectively identify data issues caused by the instability of data collection and artificially imputed values. The missing value ratio is defined as the proportion of 'nan' and 'inf" values present in the time series. Meanwhile, the invalid observation ratio refers the maximum proportion of zeros in the first- or second-orde differences of the time series. To address these issues and drawing inspiration from the data processing techniques of large language models (Penedo et al. 2023 Computer 2023; Jin al.) 2024), we developed fine-grained data- cleaning pipeline specifically designed for time series data:
Missing Value Processing. In time series data, missing values often appear as 'nan' (not a number) or 'inf" (infinity). While previous studies commonly address this by replacing missing values with the mean, this may distort the original time series pattern. Instead, we employ a method that splits the original sequence into multiple sub-sequences points where missing values occur, effectively removing those segments while preserving the integrity of the original time series pattern.
Invalid Observation Processing. In some data collection systems, missing values often filled with 0 or another constant, leading to sequences with constant values that do not represent valid patterns for the model. To address this, we developed filtering method that uses fixed-length window to scan the entire sequence. For each window, we calculate the ratio of first-order and second-order differences, discarding the window if this ratio exceeds pre-specified threshold (set to 0.2 in our case). The remaining valid continuous window sequences are then concatenated into single sequence. This process transforms the original sequence into multiple sub-sequences, effec- tively removing segments with invalid patterns.
Following the processing steps described above, we compiled a high-quality time series dataset named Time- 300B, which spans a range of sampling frequencies from seconds to yearly intervals, encompassing total of 309.09 billion time points. To optimize memory efficiency and loading speed, each dataset is split into multiple binary files, with a metañile providing details such as the start and end positions each sequence. This setup allows us to load the data using a fixed amount of memory during training, preventing memory shortages. Datasets like Weatherbench, CMIP6, and ERA5 are particularly large, often leading to data imbalance and homogenization. To mitigate these issues, we apply down-sampling to these datasets. During training, utilized approximately 117 billion time points in Time-3C08 sampling each batch according to fixed proportions of domains and distributions of observation values.
19

---

Published as a conference paper at ICLR 2025
Below, outline the key properties of the datasets after processing, including their domain, sam- pling frequency, number of time series, total number of observations, and data source. Also, we present the key component's source code of the data-cleaning pipeline in Algorithm 2
Table 10: Datasets and key properties from Time-300B. For frequency: $ = second, T minute, H hour, D day, B business day, W week, M month, Q quarter, Y year.
Dataset
Domain
Freq-
Time Series
# Obs.
Source
Eleetrictty (15 min)
Energy
15T
347
39,708,170
Godahewa al.
2021
Electricity (Weckly)
Energy
318
49,608
Godahewa et al.
2021
ERCOT Load
Energy
H
152
1,238,832
ourownstory
2023
Australian Flectricity
Energy
30T
5
1,153,584
Godahewa al.
2021
Solar Power
Energy
26
5,248
Godahewa al.
2021
Farms
Energy
43.246
39,705,117
Godahewa al.
2021
BDG-2 Bear
Energy
215
1,422,320
Emami et al.
2023
BDG-2 Fox
Energy
179
2,285,288
Emami al.
2023
BDG-2 Panther
Energy
H
136
893,840
Emami al.
2023
BDG-2 Rat
Energy
455
4,596,080
Emami
2023
Borealis
Energy
82,757
Emani
Building:900K
Energy
2,464,18%
15,124,358,21
Fmaai et al.
2023
BDG-2 Bull
Encrgy
464
501,832
Wang ct al
2023d
BDG-2 Cockatoo
Energy
H
4
17032
Wang al
2023d
Covid19 Energy
Energy
1
31,912
Wang al
2023d
Eleedemand
Energy
17,520
Godahewa
2021
GEF12
Energy
20
788,280
Wang ct al.
2023d
GEF17
Energy
H
8
140,352
Wang et al
2023d
BDG-2 Hoog
Energy
II
152
365,304
Wang et al
2023d
IDEAL
Energy
225
1,253,088
Emami al.
2023
KDD Cup 2018
Energy
3,054
922,746
Godahewa
202
KDD Cup 2022
Energy
10T
8,554
2,332,874
Zhou et al.
2022a
London Smart Meters
Energy
30T
24,132
160,041,727
Gudahewa et al
2021
PDB
Energy
H
17,520
Wang et al
2023d
Residential Load Power
Energy
T
79,508
404,832,695
Bergmeir et al.
2023
Residential Power
Energy
248,888
184,238.228
Bergmeir
2023
Sceaux
Energy
34,223
Emani
2023
SMART
Energy
5
95,709
Fmaai et al.
2023
Spanish
Energy
1
35,064
Wang ct al
2023d
Exchange Rate
Finance
13
56,096
Ansari al.
2024
CIF 2016
Finance
72
7,108
Godahewa al.
2021
Bitcnin
Finance
29
68927
Godahewa al.
2021
FRED MD
Finance
104
71,624
Godahewa al.
202
NN5 Daily
Finance
220
35,303
Godahewa et al.
2021
Tourism Monthly
Finance
M
359
98,867
Gudabewa al
2021
Tourism Quarterly
Finance
427
39,128
Godahewa al.
2021
Tourism Yearly
Finance
419
11,198
Godahewa al.
2021
COVID Deaths
Healtheare
364
Godahewa
2021
Hospital
Hcaltheare
727
55,224
Godahewa al.
2021
CDC Fluview ILINet
Healthcare
286
220.144
CDC
2017
CDC Fluview WHO NREVSS
Healtheure
W
108
56,407
CDC
2017
Project Tycho
Healthcare
W
588
120,183
Panhuis et al.
2018
20

---

Published as a conference paper at ICLR 2025
Table 10 continued from previous page
Dataset
Domain
Freq-
Time Series
# Obs.
Source
US Births
Healtheure
7,275
Gudahewa el al
2021
Weatherbench (Hourly)
Nature
H
3,984,029
74,630,250,518
Rasp et al.
2020
Weatherbench (Daily)
Nature
D
301,229
3,223,513.345
Rasp al.
2020
Weatherbench (Weekly)
Nature
226,533
462.956,049
Rusp ul
2020
Beijing Air Quality
Nature
4,262
2,932,657
Chen
2019)
China Quality
Nature
17,686
4,217,605
Zheng et al.
2015
CMIP6
Nature
6H
14,327.808
104,592,998.400
Nguycn ct al.
2023
ERAS
Nature
H
11,940,789
93,768,721,472
Nguyen et al
2023
Oikolab Weather
Nature
309
615,574
Godahewa al.
2021
Saugeen
Nature
38
17,311
Godahewa al.
2021
Subseasonal
Nature
17.604
51,988498
Mouatadid al.
2023
Subscusonal Precipitation
Nature
D
13,467
4,830,284
Mouatadid al
2023
Sunspol
Nature
D
19
45,312
Gudahewa al
2021
Temperature Rain
Nature
D
13,226
3,368,098
Godahewa al.
202
Weather
Nature
9,525
26,036,234
Ansari al.
2024
Dominick
Sales
3,712
759,817
Godahewa al.
2021
Car Parts
Sales
16
816
Godahewa et al
2021
Favorita Sales
Sales
91,513
20,371,303
Woo et al.
2024
Favorita Transactions
Sales
258
81,196
Woo et al.
2024
Hierarchical Sales
Sales
215
114,372
Mancuso al.
2021
Restaurant
Sales
155
30,289
Woo al.
2024
M5
Sales
14,341
5,011,077
Alexandrov ct
2020
Mexico City Bikes
Transport
556
78,848
Ansari ct al.
2024
Traffic
Transport
H
1,371
14,993,544
Godahewa al
202
Taxi (Hourly)
Transport
2,433
1,762,024
Ansari al
2024
Beijing Subway
Transport
30T
552
19,872
Wang al.
2023a
Covid Mobility
Transport
426
120,950
Godahewa al.
2021
HZMetro
Transport
15T
160
11,680
Wang et al
2023al
LargeST
Transport
5T
1,208.997
4,175,062,621
Liu al.
2023
Loop Seattle
Transport
5T
1,809
33,700.832
Wang al.
2023a
Los-Loop
Transport
3,381
6,231.168
Wang al.
2023a
Pedestrian Counts
Transport
3,125,914
Godahewa al.
2021
PEMS Bay
Transport
5T
3,980
15,975,920
Wang et al
2023a
PEMS03
Transport
5T
1,651
9,210,432
Wang ct al
2023a
PEMS04
Transport
5T
6,634
14.638,784
Wang al.
2023a
PEMS07
Transport
5T
3,828
23,78 23,789,760
Wang al.
2023a
PEMS08
Transport
2,612
8,684.480
Wang al.
2023a
Q-Traftic
Transport
46,990
257,200,384
Wang et al.
2023a
SHMetro
Transport
15T
574
41,902
Wang et al.
2023a
SZ-Taxi
Transport
15T
156
464,256
Wang et al
2023a
Rideshare
Transport
H
1,352
192.949
Godahewa al.
2021
Taxi
Transport
30T
96.758
40,584.636
Alexandrov
2020
Traffic Hourly
Transport
1,363
14,858,016
Gudahewa
2021
Traffic Weekly
Transport
78,816
Godahewa al.
2021
Uber TLC Daily
Transport
235
42,533
Alexandrov et al.
2020
Uber TLC Hourly
Transport
HI
344
510,284
Alexandrov et
2020
Vehicle Trips
Transport
D
10
1,626
Godahewa al
2021
Wiki Daily (100k)
Web
100,001
274,099,872
Ansari al.
2024
21

---

Published as a conference paper at ICLR 2025
Table 10 continued from previous page
Dataset
Domain
Freq.
Time Series
Obs.
Source
Alibuba Cluster Trace 2018
Web
5T
48,640
83,776,950
Woo cl al
2023
Azure VM Traces 2017
Web
5T
263,928
880,648,165
Woo et al
2023
Borg Cluster Data 2011
Web
5T
216,636
176,650,715
Woo ct al.
2023
Kaggle Web Traffic Weekly
Web
133,388
15,206,232
Godahewa al.
2021
Extended Web Traffic
Web
161,890
332,586,145
Gudahewa
2021
Wiki-Rolling
Web
47,675
40,619,100
Alexandrov ct
2020
TSMixup 10M
Synthetic
10,968.625
8,198,358,952
Ansari ct al.
2024
KemneSyynth IM
Synthetic
1,000,000
1,024,000,000
Ansari al.
2024
MI Monthly
Other
M
8
1,047
Godahewa al.
2021
MI Quarterly
Other
3M
195
9,628
Godahewa al.
2021
Yearly
Other
106
3136
Godahewa al.
2021
M3 Monthly
Other
799
109,538
Godahewa et al.
2021
M3 Quarterly
Other
3M
755
36,960
Gudahewa al
2021
M3 Yearly
Other
Y
645
18,319
Godahewa al.
2021
M4 Daily
Other
4.134
9,903.554
Godahewa al.
202
Hourly
Other
415
352,988
Godahewa al.
2021
M4 Monthly
Other
30,126
8,480,953
Godahewa al.
2021
M4 Quarterly
Other
3M
2,623
491,632
Godahewa et al.
2021
M4 Weekly
Other
W
293
348.224
Godahewa al.
2021
M4 Yeurly
Other
106
3,136
Godahewa al.
2021
22

---

Published as a conference paper at ICLR 2025
Algorithm 2 Code Snippet of Data-cleaning Pipline
Missing Value Processing
def split_ssq__bp _a________aa_niiie_e inf (seg, minimum_ length: int 1)
output
sublist [] for in seq:
rum is None or np.1anan (num) ns.isint (num):
if len (suallat) 5- sas__mmm___eeeeeeeeee_eeee
cutput. aatguut.apend(/ublleer.
else:
subliat.agpendd
len (sublisst) =intmon_soo_lengthn
output append
return output
Invalid Observation Processing
def spl it__tt_________ wywidobby_
window_siza: int 128, zero_threshold, int 256):
#it.imum_see__eene
if [en()eg) winéox_size:
flag, info = check_sequance
i: flag:
return [seq]
else: []
window_size
sub_seg
cut
while True:
1. i window_size (seg)
windaw_seg seg(i window_size: (seg)
(seg)
else:
window_seg segli window_sizes
flag, infa seeck_sequancelee (weceeewwniii) (window__og zaro@hhrasholdszzznntteeaeaa
1: flag:
sub_seq.extent (wo___ggeeceedddidddd.....
alse:
it len isua_seee)) >- see_uu_s___eggthh
out our__lss..apend (sub__eee
sub_seq
1f i >- len (seg):
broak window_size
if (an minimun_aaq longth: out app__llat.
cui_list
def @heck_sequence (seg, tero_threshold: flost):
import numpy not isinstance Iseg, np.ndarray):
seq = mp.arrayiseg
if (seq. raise
(E' Dimension, of the is, not equal, too 1;, Aseg. shapel")
flag
info
nan_count np. Lanan (eeq)
infol' raEEu'"a nan_ccuaa
if nan_count > 0:
return flag, info
inf_count (emnsniniiiiinf(eee infa("inf_sour:" inf_ceunt if iat_count 0:
isinf (seg(
flag False return flag,
zero_ratic ep.eum 0) lentseg mnfa['zero_ration if zero_ > sera_thresho d: flag False
zero
firat_diff = seq[t:] seq(:-1) ciret__litr_eerorratoo ap. 31 ienifirat_dirt)
futo[""irat_ddiffeeee reto_ratorroorttio""o ratior: first cirst_diff_eerr_reett if first_diff ratii zerc_chreshold:
flag Falsa
secoad_diff seq[2:] seg[:-2]
socand_dife secon_ddiff_zrrorrttti np. (socand diff 0) Ion (second_d:.f)
infol'secone diff if difond_diff_.co _it_dddff__oooaati____ tlag False
secend weceeudddffeeeeeeeeeee ratio
threshold:
23
flag, info

---

Published as a conference paper at ICLR 2025
D ADDITIONAL RESULTS
D.1 ABLATION STUDY
Table 11: MSE and MAE for horizon-96 forecasting across six benchmarks, evaluated with different model components.
ETThl
ETTh2
ETTml
ETTm2
Weather
Global
Tomp
MSE
MAE
MSE
MAE
MSE
MAE
MSF
MAE
MSE
MAF
MSR
MAE
TIME Mol who Huber
0.357
0.381
0.305
0.359
0.338
0.368
0.2011
0.291
0.160
0.214
0,211
0.343
0.369
0.163
0.221
0.359
wlo multi-resoluioon layer
0.358
0.379
0.313
0.362
0.348
0.377
0.212
0,301
0.164
0.219
0.217
0.354
wlo exxeeettsss experts
0.373
0.212
0.298
0.163
0.218
0.223
0.357
w/o auxiliary loss
0.368
0.394
0.325
0.387
0.350
0.377
0.219
0.304
0.164
0.220
0.226
0.363
replacing the MoE layers with standard FFNs (denoted as "w/o mixture-of- experts ") led to a noticeable performance decline, with the average MSE worsening from 0.262 to 0.272. This highlights the significant contribution of the sparse architecture to the model's overall performance, its dynamic routing enables more specialized processing of diverse input patterns.
As shown in Table IT
We also conducted experiments by retaining only the horizon-32 forecasting head from the TIME- MOE (denoted as "w'o multi-resolution layer"), excluding the multi-task optimization. The per- formance of this modified model was slightly inferior to the complete TIME-MoE.
Table 12: Full ablation results for different multi-resolltion forecasting configurations.
ETTh1
ETTh2
ETTml
ETTm2
Weather
Global Temp
Average MSF
Inference Speed
TIME- MOE TIME-MOE
W/ (1.8.32.44)
0.357
0.305
0.338
0.201
0.160
0,211
0,262
0.095 s/iter
TIME-MOEAse (1,832)
0.353
0.316
0.370
0.225
0.161
0,213
0.273
0.130 s/iter
TIME-MOEaue w/ (1.8)
0.389
0.391
0.441
0.304
0.174
0.222
0.320
0.411 s/iter
TIME-MOE w/ [}
1.071
0.920
2.098
2.320
1,500
0.383
1.382
2.834 s/iter
As shown in Table 12. the default configuration of four multi-resolution forecasting heads with receptive horizons of 8, 32, 64 delivers optimal predictive performance and inference speed. Re- ducing the number of heads consistently resulted in decreased performance and longer inference time. This inverse relationship highlights the effectiveness of our multi-resolution forecasting de- sign, striking balance between accuracy and computational efficiency a decoder-only forecasting foundation model.
These findings highlight the importance of key architectural components in TIME-MoE, such as the mixture-of-experts, multi-task optimization, and multi-resolution forecasting, in delivering state-of- the-art performance in universal time series forecasting.
D.2 TRAINING PRECISION ANALYSIS
To optimize model performance and efficiency, we conducted comparative study examining the impact of numerical precision during training. We trained two versions our model under identical configurations, with the only difference being the precision: one using 6 and the other using float32. The model trained with float32 precision referred to as TIME-MOEnss w/ FP32.
Table 13: Full results of the comparison between BF16 and FP32 in terms of training and inference efficiency. FA denotes flash-attention.
ETTat
Weather Global Temp
Average MSE
Training Speed Speed Memary Menory
ToMisMMB... Tome: MoEu FA TiMs-MoFF... FP32
0.388
0.201
0.160
0.11
0.362
sñtes
0.0055
1.77
0.211
0.263
0.138
1.77 GB
226.70
0.208
0.133
As detailed in Table 6 our analysis reveals that the forecasting performances of these two models are remarkably comparable. This finding is significant as it demonstrates that the use of reduced precision (e.g., 6 does not compromise the predictive capabilities of model.
24

---

Published as a conference paper at ICLR 2025
However, the similarities in performance belie the substantial differences in computational efficiency and resource utilization:
Notably, the b6loat16 model demonstrates a 12 improvement in training speed compared to its float32 counterpart. This considerable acceleration in the training process can significantly reduce the time-to-deployment for large-scale models and facilitate more rapid experimentation and iteration.
Training Speed:
Memory Consumption: In terms of memory usage, the 6 model exhibits superior effi ciency, consuming substantially less memory than the float32 model. Specifically, we observed reduction of 20% % in memory usage. This memory optimization is crucial for scaling models to larger sizes or deploying them on memory-constrained hardware.
Compatibility with Advanced Techniques: A key advantage of the bfloat model is its seam- less integration with advanced optimization techniques. particular, can easily be combined with flash-attention (Dao 2024), a state-of-the-arr attention mechanism designed for better effi- ciency. This integration results in an additional 23% increase in training speed and a 19 % boost in inference speed, further enhancing the already significant performance gains.
The implications of these findings are far-reaching:
Resource Efficiency:
The reduced memory footprint and increased training speed of the bfloat16 model translate to more efficient utilization of computational resources, potentially lowering infrastructure costs and energy consumption.
Scalability: The memory savings offered by bfloat16 precision enable the training of larger, more complex models on the same hardware, potentially leading to improved model capabilities without increasing computational requirements.
The substantial improvements in training speed can accelerate the research and development process, allowing for more rapid prototyping and experimentation.
Faster Development Cycles:
Inference Optimization: The compatibility with flash-attention not only benefits training but also enhances inference speed, which is crucial for real-time applications and large-scale de- ployments.
Our experiments show that adopting bfloat16 precision, combined with advanced techniques like flash-atennion, provides a compelling balance between model performance, computational effi- ciency, and resource utilization. These optimizations enable the scalable and efficient deployment of large-scale time series forecasting models without sacrificing predictive accuracy.
D.3 ADDITIONAL EXPERIMENTAL RESULTS
D.3.1 TAXIBJ DATASET
We include benchmark dataset, TaxiBJ (Zhang et al. 2017) for short-term forecasting evaluation.
This original dataset encompasses taxicab GPS data and metcorological information collected from Beijing four distinct intervals: July 2013 October 30, 2013; March 1, 2014 June 30, 2014; March 1, 2015 June 30, 2015; and November 1, 2015 April 10, 2016. We selected the in-flow data benchmark. This benchmark dataset consists of 1,024 time-series sequences derived from 32 x 32 grid cells.
from the period November 2015, April 10, 2016
We conducted evaluations all zero-shot models using this benchmark, and set the context length to 512 for all baselines. The results are summarized in Table [4]
Table 14:
Short-term zero-shot forecasting results in TaxiBJ: A lower MSE or MAE indicates better prediction. Red: the best, Blue: the 2nd best.
L ToME-MoE (Ous))
Zero-sot Time Series Medels
Models TrMr-MoR...
TIME-MoF..
TIME-MoR....
Muiraisuat
Muiraiu..
Mairalurye
TimIsFM
Mumnnt
Chrman
Chranos
Chrones
Metries
MMSE
MSS
MAR
MAR
MSE
MSE
0.214
0.204
0.214
0.292
88314
0.944
0.334 0.373
0.262 0.334 0.267
0.325
0.317
03355
1.865
0.75;
(0250
0.315
10255 0,36
0,20 0.563
Taxil
8
0.0
1.300
0.410
0.361
0.376
0.299
0.356
0.400
0.188
0.317
0.367
0.111
0660
0.88
0.50
0.670
0.477
0.590
0.477
0,422
0.588
0.431
0-488
01455
0.466
0.995
0.008
1.030
10805
1.994
0.964
0.799
0.341
04884
03300
0:400
(197
U311 0.357
04400
(.431
0.310
04418
6.811
48
0.425
0,443
0.414
0.480
0,42
0.64
0.626
0,542
0.559
0.497
0.563
0.500
0.524
0.515
[.892
0.765
0.502
0.47%
0.475
0.450
0.494
0.460
25

---

Published as a conference paper at ICLR 2025
The results indicate that models consistently outperform other baselines in short-term forecasting on the TaxiBJ dataset.
D.3.2 COMPARISON TO TIMER, TFT, AND N-BEATS
In this section, we incorporate additional baseline models for a more comprehensive evaluation. Specifically. Timer (2024d) is included for zero-shot forecasting (Table 15). while TFT (202T) and N-BEATS (2020) are included for in-domain forecasting (Table [6]. The results indicate that our models consistently demonstrate improved performance relative to these established approaches.
Table 15: Additional zero-shot forecasting results of Timer, with 1B, 16B and 28B representing the scale of pretraining datasets as presented in the original paper Liu et al. (2024d). A lower MSE or MAE indicates better prediction. Red: the best, Blue: the 2nd best.
TIEE-MOE (Oup)
Timer
Models
MoME
TOME-MoEee
TIME-MoFasa
Metries
MISE:
MAA
MSI
MAE
MSE
MAY
MSF
MSE
MAF
MAR
0.357
0.351
(.350
0.382
0.349
0.379
00488
0,425
0.364
0.305
01.30
0:421
ETThl
336
0.431
0.434
0.447
0.453
0.354
730
D.440
0.477
0.437
0.655
0.45
0.462
0.756
0.544
0.4%%
0.444
0.457
0.494
N96
(1850)
0/424
0.396
0.11
0412
0.426
0.352
0.478
B.406
0.816
01814
0.455
[0306
0.359
0,352
6,315
0.350
192
0.351
0.386
0.364
0,555
6.347
0,579
0.353
0.412
0.353
0.365
0.346
0.595
336
[U30]]
0/415
04417
BA25
6406
0411
0.422
0.775
0.400
0.366
0.513
730
0.419
0425
96 [.358
0.368
0.30
0157
5251
0.000
0.536
0/966 I.509
(1430 0.415
ETTml
192 11355
0.318
6.34
5200
6051
0.533
725
0.504
0.493
0415
047
6469
0393
0.627
0752
0.565
0.558
0.499
01394
94455
0.316
0.55
0771
0.572
07760
0.557
(1457
0:457
Bl1m2
336
192
11258
1.324
0.314
0.215
0317
0250
9255
1312
0234
0346
0.577
11539
8.28
0,30
0.543
(1315 0:38
0.304
0.38
730
0.400
9822
C-14
11:22
0.413
0.714
0.119
0111
N157
1111
0.181
0,207
0:305
0.251
(1.45 0.381
Weater
19h
11210
0.560
6266
1125
0.314
0.354
01356
130
730
0.405
0.415
0.408
0.405
01597
0.354
(1.50]
1.40% 0.395
0.374
Neg
11785
0.702
0.210
01400
0.256
11.300
0.389
11.00
0.394
01:21
01306
01331
Gloeal Temp
335
197
(.251
0.455
0.386
0.257
0.365
6.346
6.366
0.379
0
0.759
0.347
0.115
0.651
0
0.365
0:418
0.465
0.415
0.507
0.861
720
[U354
0.465
0.259
0.421
0.368
11421
0.522
0.521
0.542 0.585
(.579
0.61
A0g
0.275
0.400
0.255
0.355
0.385
0.317
0.363
0.415
0.504
Asaraage
00336
0.364
0.316
01:90
0,312
0,372
60394
0.406 6.7/8 0,30
0.344 0.391
Table 16: Additional results of in-domain forecasting baselines. A lower MSE or MAE indicates better prediction. Red: the best, Blue: the 2nd best.
TIME-MOR (Oam)
Vull-shot
Middls
Models
[iner-MoE
TOME-MOFag
TIME-MOF.n.
TFT
N-BEATS
Metries
MSE
MAR
MSE
MAR
MSB
MAE
MSE
MAF
MSE
MAF
0.545
0.57%
0.135
0.171
6.321
0.365
0:474
0.476
11S
BTTM
192 0,372
0
0.374
0.391
0.100
0.486
0.453
720
0L110
0.443
0.402
0.40
0.529
05155
0.55
0.359
0.408
0.375
0.404
8333
0.406
0.521
0.806
0:457
0/473
96 0.206
0.340
00278
0.335
0.274
0.358
0.350
0.357
0.362
0.384
192
0.399
0.371
0.474
0.41
0415
0.573
0.384
0.30
0.3%6
0.661
0.554
96
0.256
1.334
0.264
0.325
6.356
0.333
10463
0.444
0.334
0,372
ETTml
336
142
0.554
0.507
11390
0.55%
0. 1.99
9,33
0.15)
2122%
6:381
0:53
0.374
0357
0.811
0.538
0.438
[.42]
0.570
0,425
0.401
730
0443
0.445
0.4m9
0/4
0450
0.722
0.476
0.551
0.223
0/477
01506
11,005
0/417
96
0.265
1.109
0.399
111
632
10323
11.05
0.306
0.363
RTTm2
876
192
0.255
11306
0.545
0201
9321
0114
0.365
0300
0364
0281
0.329
0.355
0.344
0.334
0,372
0.183
720
0.013
1.424
0441
0.53
0424
04435
11:35
(1.60))
0.455
Avg.
0.274
0.335
0.244
3:77
035
0.358 03356
0.342
0.149
0.154
Weather
336
192
0.247
0.286
0.192
6.165
0.365
0330
0.00
6367
03302
0.240
0.317
0.309
0.361
0.310
720
1.361
11366
0.161
6376
0.308
11300
1.33%
0.362
Arg.
0.256
0.275
0.254
0.21
0.200
0:200
0.79
0.25%
0.243
0.291
0.31%
0.189
0.200
0.290
Global Temp
336
192
0125%
0.299
11397
0.74
6381
0.559
0.004
0.255
11:42
720
0.345
0.465
0.32
0.51
6200
0.4%
0.371
0.477
0.342
0.457
Ang.
0.284
11399
0.551
0.18
0.300
0,321
01800
11222
0:394
Average
0.305
01362
0.104
6.301
0.388
0.415
0.422
0.400
26

---

Published as a conference paper at ICLR 2025
E FORECAST SHOWCASES
To visualize the performance differences among various time series foundation models, we present the forecasting results of our model, TIME-MOE, comparison the ground truth across six real- world benchmarks. These benchmarks include ETTh1, ETTh2, ETTml, ETTm2, Weather, and Global Temp datasets. Alongside TIME- MoE's results, we also show the performance of other foundation models at different scales, providing a comprehensive view of their comparative capabil- ities (Figures 6 [T]. In all figures, the context length is set 512, and the forecast horizon 96. To enhance clarity and aesthetics, we display the full forecast output, complemented by a portion of the preceding historical input data, ensuring more intuitive comparison.
The results clearly demonstrate the superiority of TIME-MoE over the other foundational models. Its ability to consistently produce more accurate forecasts across a range of datasets underscores the effectiveness of its architecture and design. The performance gains are especially noticeable in long-term prediction scenarios, where TIME-MoE' handling of temporal dependencies proves more robust than its counterparts. These visual comparisons highlight the practical advantages of TIME- MoE in large-scale time series forecasting, reinforcing its status as a state-of-the-art model.
27

---

Published as a conference paper at ICLR 2025
Ground Truth
-0.5
Prediction
00.8
-0.5
1.2
-12-
1.2
-1.6
-1.0-
-1.6
Prediction
50
150
250
50
100
150
250
(a) Time-MoEs Mo- base
(b) Lomne-MoEtarga large
(c) Tiiee-MoE.in
Ground Truth
Ground Trith
Ground Iuth
-0.0
Prediction
-0.0
Peediction
Pred citon
-1.0
-1.2
-1.2
-1.2
Aur
-1.4-
1.6
-2.6
-1.6
250
300
30
100
1.50
200
250
(d) Moiraismatt
(e) Moiralpase
(f) Moiraitarge
Ground Truth Prediction
Ground
-0.8
-0.0
-30
-3.2
-1.2
-14
-1.4
-1.4
-1.8
=1.8
+1.8
Prodictt
50
1000
1.50
700
2500
100
200
250
100
50
100
1.90
2010
250
100
(g) Chronos
(h) Chronospase
Chronos
Lacund Iuth
Ground truth
Prediction
-0.0 irod
-10
0.00
-1.0
-14
-1.6
-1.8
50
1.00
150
200
250
300
30
100
150
250
250
200
(i) Moment
(k) TimesFM
Figure 6: Zero-shot forecasting cases from ETThl I by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
28

---

Published as a conference paper at ICLR 2025
-0.5
-3.0
-1.0
-.5
-1.9
-1.-
-2.0
-2.5
-2.5
-2.
Ground Tuuhh Prediction
Ground Truth
Ground Thuth Prodiction
-3.0
-3.0
Predictionn
-3.0
(a) Time Laree-MoEpace
(b) Time-MoE
(c) Time Mo
-0.5
-3.0
-1.0
-3.
-1.5
-1.5
-2.5
-7.5
+2.0
Ground Tuuh Prediction
-3.0
Ground Predicttin
Ground Truth
Prediction
50
100
350
200
250
(d) Moiraigmall
(e) Moira
(f) Moira Marraittage
-0.5
-0.5
-1.0
-1.0
-1.
-2.0
-2.0
-2.0
-7.5
-2.5
-2.5
-3.0
Ground Tuth Prediction
Ground Truth
Ground
50
100
150
200
250
100
50
1000
350
200
250
300
(g) Chronossmaul
(h) Chronospase
(i) Chronostarge
-0.
M
-0.5
h/
-15
1.5
MM
-2.0
-2.5
-300
Prediction
Ground Truth
-3.0
Prediction
(i) Moment
(k) TimesFM
Figure 7: Zero-shot forecasting cases from ETTh2 by different models, with forecast horizon 96. Blue lines are the ground truths and read lines the model predictions.
29

---

Published as a conference paper at ICLR 2025
-0.6
-0.6
-0.6
M
0.7
0.7
0.7
-0.8
M Mm Awn
-0.8
M Jmn
In
-0.8
-0.9
-0.9
-0.0
-.0
-1.0
-i.
-1.1
-1.1
-1.1
-1.2
-1.2
-1.2
50
100
150
200
250
300
50
100
150
200
250
200
50
100
150
200
250
300
(a) Time-MoEE hase
(b)
(c) Time-MooEute
Ground Truth
Ground Truth Prediction
M
Ground Truth Prediction
Prediction
Mn
Jwn JM
-0.8
M M
-0.9
-1.1
-1.2
-1.2
-1/2
100
350
250
50
100
1.50
200
250
300
50
100
150
200
250
200
(dl) Moiraigmall
(e) Moira
(f) Moira
-0.8
-0.6
-0.6
M
Ground Bruth Prediction
M
Ground Truth Frediction
Ground Truth Pred ction
-0.7
-0.7
M Mm
-0.8
MAN
M
(g) Chronos small
(h) Chronospase
(i) Chronos targe
-0.6
-0.6
-0.7
Prediction
-0.7
-0.0
Mn
-0.8
-0.9
-0.9
-1.0
-1.0
1.1
1.1
-1.2
-12
Prediction
100
1.50
200
250
50
100
150
200
230
300
(i) Moment
(k) TimesFM
Figure 8: Zero-shot forecasting cases from ETTml by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
30

---

Published as a conference paper at ICLR 2025
0.25
0.25
0.25
0.00
0.00
0.00
-0.50
-0.50
-0.50
-1.00
-1
-1.25
-1.25
-1.25
-1.50
Ground Tuthh
Ground Tuthh
-1.50
Ground Tuth Prediction
Prediction
Prediction
100
250
(a) Time-MoEpase
(b) Time
Time-MoEuaree MoEuarge
(c) Time-MoEutero
0.25
0.25
0.25
0.00
-0.25
lay raghte
-0.30
0.50
0.50
-0.75
0.75
-1.00
-1.25
-125
-1.19
-1.50
Ground Thut Ieediction
1.50
around Wuth Prediction
Gomund Wath Prodlction
100
250
(d) Moiraismall
(e) Moirai
(f) Moira
0.25
0.25
0.25
0.00
-0.25
-0.25
0.50
0.50
0.50
-0.75
-1.00
-1.00
-1.25
-125
-1.5
-1.50
Ground Tuuth
Ground
Ground Tuth Prediction
Presiction
Prediction
-1.75
-1.73
150
155
700
340
160
750
(g) Chronos
(h) Chronosmase
(i) Chronostarge
0.00
-0.55
-0.25
0.50
1.00
-1.00
-1.25
-1.00
-1.79
-1.75
50
100
150
200
250
300
50
100
15n
700
750
300
(i) Moment
(k) TimesFM
Figure 9: Zero-shot forecasting cases from ETTm2 by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
31

---

Published as a conference paper at ICLR 2025
Ground Thut
0.00
0.00
0.00
-0.00
-0.00
-0.50
1.00
1.00
-1.00
-1.25
-1.25
-1.25
-1.50
250
1.00
120
200
250
20
(a) Time-MoEpase
(b) Time-MoEEpn
(c) Time-MooEttre
Ground
0.25
0.00
0.00
0.00
-0.50
0.50
-1.00
1.00
-1.09
-1.25
-1.25
-1.50
-1.50
-1.50
100
350
1.00
(d) Moiraismall
(e) Moirai
(f) Moira
0.5 Ground Truth
Prediction
0.25
Prediiction
Ground Truth
0.25
Prediction
Ground Tuth
-0.25
-5.50
-0.50
-0.75
11.00
-1.00
-1.5
-1.50
-1.50
150
200
(g) Chronos
(h) Chronos pase
(i) Chronossarge
Prediction
Mey Mablh
Predictio
0.00
0.00
-0.0
-0.30
-3.75
-1.00
-1.00
-125
-125
100
150
200
250
100
(i) Moment
(k) TimesFM
Figure 10: Zero-shot forecasting cases from Weather by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
32

---

Published as a conference paper at ICLR 2025
colon
0.8-
0.8-
0.8
3.-
0.-
0.6
0.4
0.4
0.4
1
-0.0
-04
-0.
-0.6
-0.6
-0.6
20
100
150
200
250
300
50
300
150
200
250
300
100
150
200
250
300
(a) Time-MoEpe
(b) Momne-MoErarge
(c) Time Home-MooEttrer
1.0
1.0
Prediction
calon
0.8
0.8-
0.8
0.6
0.6
0.5
0.4
-07
44
-0.4
-04
-0.4
-0.6
-0.6
-0.6
250
30
100
150
200
250
300
100
150
200
250
300
(d) Moirai smatt
(e) Moiralbase
Moiraltarge
Ground
Groune Trubs
Grounc Truth
0.6
0.6-
0.6-
0.4
04-
0.4
50
100
150
200
250
3010
50
300
1521
2000
252
300
300
350
2010
250
500
(g) Chronossmalt
(h) Chronospase
Chronostarge
Ground Truth
Ground Bruth
Pred
Prediction
0.4
0.
0.2
0.0
-0.
50
100
150
200
250
300
300
350
()) Moment
(k) TimesFM
Figure 11: Zero-shot forecasting cases from Global Temp by different models, with forecast horizon 96. Blue lines are the ground truths and read lines are the model predictions.
33


Reviews:
[{"review_id": "AHH6qzjGiW", "reviewer": "Reviewer_Ruyh", "summary": "This paper presents a new architecture designed to improve time series forecasting by utilizing large-scale foundation models! It addresses key challenges in time series data, like complexity and distribution shifts, which require strong forecasting solutions across fields such as finance, energy, and climate. TIME-MOE adopts a sparse mixture-of-experts (MoE) approach, activating only a subset of parameters during inference! This reduces computational demands while preserving performance! A notable focus is on scaling laws, which have seen success in language and vision research! By applying these principles to time series forecasting, the authors show that expanding model size and training data leads to substantial improvements, positioning TIME-MOE as a leading model that surpasses traditional dense architectures. The flexible design of TIME-MOE supports variable forecasting horizons and input context lengths, making it suitable for diverse applications! The paper also includes a thorough reproducibility statement detailing the architecture, implementation, and data processing steps, highlighting the authors' dedication to transparency and collaborative research! This work not only advances time series forecasting but also paves the way for future studies on scalable, efficient forecasting architectures!", "questions": "Please check the Weaknesses part!", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The originality of this work shines through its novel approach to applying MoE architecture in time series forecasting! By adopting a sparse framework, the authors not only boost computational efficiency but also extend the potential of large-scale models in this field! This unique integration of MoE principles, previously effective in fields like NLP and Computer Vision (CV), offers a new perspective, tailored to tackle the distinctive challenges of time series data! \n\nThe work stands out for its rigorous methodology, which supports a reliable, in-depth analysis of the TIME-MOE architecture! The authors carefully describe each component, training process, and evaluation metric! This level of detail strengthens the study's findings, ensuring they are both robust and reproducible, an essential foundation for advancing knowledge and encouraging future work in the field!  \n\nThe paper stands out for its clarity, presenting complex ideas in a way that's easy to follow! The authors organize their work with well-defined sections, clear plots, and tables that effectively highlight the model's performance metrics! This structure makes it easier to understand the proposed architecture and broadens the paper's appeal, helping both experts and newcomers engage with the context! The study is particularly important because it tackles a crucial gap in time series forecasting research! By scaling a foundation model up to 2.4 billion parameters and showing significant accuracy gains, the authors demonstrate that larger models can boost performance without overwhelming computational demands! This insight is valuable for practical applications, indicating that organizations can adopt high-performance forecasting models without excessive resource costs! \n\nThe paper's contributions go beyond theoretical insights, opening doors to practical applications across fields like finance, energy, and climate science! TIME-MOE's capability for universal forecasting and flexibility across different time horizons make it a valuable tool for improving decision-making in various industries! This potential for tangible impact highlights the paper's importance and its value to both researchers and industry professionals!", "weaknesses": "(1) The originality of the model is somewhat limited by the lack of a detailed comparison with other SOTA models beyond the chosen benchmarks. While the authors report promising improvements, the analysis would benefit from a broader comparative study that includes contemporary forecasting models, such as [1] and [2]. Specifically, it would be helpful to compare aspects like model architecture, data handling techniques, and feature processing methods in these SOTA models to highlight where TIME-MOE uniquely contributes. Additionally, incorporating ablation tests could help clarify the impact of each component in the TIME-MOE architecture, allowing readers to understand which elements drive its performance gains. This would strengthen the claims of superiority over existing models and enhance the overall rigour of the evaluation!\n\n(2) The paper highlights the benefits of sparsity and computational efficiency in the MoE architecture but overlooks certain limitations, notably the risk of underfitting when only a limited number of experts are activated. This issue is particularly relevant in time series forecasting, where the variability across datasets demands a careful approach to expert selection. I suggest that the authors provide an analysis examining how the number of activated experts impacts model performance across diverse time series datasets. This would directly address the concern of potential underfitting and add valuable insights. Additionally, referencing studies like [3] that discuss specific challenges in MoE models would strengthen the discussion!\n\n(3) The experimental evaluation is limited by the narrow selection of datasets, primarily focusing on a restricted set of benchmarks. This approach may not capture the model's performance across diverse time series characteristics. To strengthen the study, I recommend that the authors consider expanding their dataset choices to include more varied features, such as seasonality, trend, and noise levels. Adding datasets like M4, those from the UCI time series repository, the Monash dataset [4], and LOTSA as used in [5] could offer a broader perspective on the model’s adaptability and robustness. Additionally, I encourage the authors to clarify their rationale for selecting the current benchmarks. Specifically, explaining how these datasets reflect a range of time series characteristics would help assess whether the chosen datasets sufficiently represent the intended applications or whether a more comprehensive selection is warranted. This would either justify their current dataset choice or highlight areas where further dataset diversity could enhance the model’s generalizability!\n\n(4) The interpretability of the TIME-MOE model could be significantly strengthened by providing more detailed insights into expert activations for various types of time series inputs. Given the complexity of MoE architectures, it would be valuable for the authors to visualize which experts are activated for specific predictions across diverse input scenarios. Additionally, an ablation study illustrating how different experts contribute to predictions for various data types would offer a clearer understanding of each expert’s role. These additions would not only improve transparency but also increase the model's reliability and applicability in real-world settings!\n\n(5) The paper mentions the efficiency of the TIME-MOE model but lacks specific, detailed analysis on computational costs, which would be essential for understanding its practical impact. I would recommend that the authors provide concrete metrics, such as training time per epoch, GPU memory usage, and inference time across different model sizes. Additionally, a direct comparison of these metrics with baseline models would add valuable context for practitioners, especially those working in resource-constrained environments, who need a clear understanding of the computational demands before implementing the model!\n\n(6) The authors could strengthen their exploration of real-world implications by including a practical case study or detailed scenario demonstrating how TIME-MOE could be applied in a specific industry, such as energy forecasting or financial market prediction. Highlighting potential benefits and challenges within a concrete example would provide a clearer picture of how TIME-MOE fits into current forecasting practices, making the research more relatable and actionable. This approach would help bridge the gap between the theoretical aspects of their work and its practical implementation, enhancing the paper’s overall impact! \n\n\n\n[1] Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting, Lim et al., 2020\n\n[2] N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, Oreshkin et al., 2020\n\n[3] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Shazeer et al. (2017)\n\n[4] Monash Time Series Forecasting Archive. Godahewa et al., 2021\n\n[5] Unified Training of Universal Time Series Forecasting Transformers, Woo et al., 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjQF2lay2h", "reviewer": "Reviewer_1Bqc", "summary": "The article titled \"TIME-MOE: Billion-Scale Time Series Foundation Models with Mixture of Experts\" introduces a scalable, efficient time series forecasting model known as TIME-MOE, designed to handle large datasets with diverse domains and flexible forecasting requirements.", "questions": "1. Expert Routing in MoE Layers: Could you provide more details on the gating mechanism and expert selection criteria? Specifically, how does the model ensure a balanced load across experts, and what mechanisms are in place to prevent routing collapse?\n2. Handling Domain Imbalances in Time-300B: Given the wide range of domains in Time-300B, how does TIME-MOE adapt to data imbalances? Are there any mechanisms to ensure that underrepresented domains do not disproportionately impact model performance?\n3. Choice of Rotary Positional Embeddings: Could the authors explain the choice of rotary embeddings over other positional encodings? Additionally, have the authors conducted ablation studies to assess the impact of this encoding on forecasting accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Scalability and Efficiency: By using sparse MoE, TIME-MOE achieves improved performance and efficiency, especially for larger models. It reduces the computational cost significantly compared to dense models of similar scale, enabling inference on hardware with lower memory requirements.\n2. Multi-Resolution Forecasting: The multi-resolution forecasting component allows the model to handle various forecasting horizons efficiently. This flexibility is critical in real-world applications where forecasting requirements vary widely.\n3. Empirical Validation Across Benchmarks: The model shows substantial improvements over state-of-the-art dense models in both zero-shot and fine-tuned settings across multiple time series datasets, providing robust evidence of its performance.", "weaknesses": "1. Complexity in Routing and Expert Selection: The process by which TIME-MOE assigns tokens to experts in the MoE layer lacks detailed explanation, particularly regarding the criteria for expert selection and load balancing across experts. Specifically, it would be helpful to include the mathematical formulation of the gating mechanism used to select the top-k experts, along with any thresholds or conditions applied during this process. Furthermore, could the authors clarify the techniques employed to prevent expert collapse, such as auxiliary loss functions or dynamic routing strategies? Including these details would provide a clearer understanding of how TIME-MOE maintains balanced and effective specialization among experts.\n2. Dependence on Data Quality in Time-300B: The data-cleaning pipeline for Time-300B is briefly mentioned, but additional details on the metrics used to assess data quality before and after cleaning would provide valuable insights. Could the authors specify which metrics (e.g., missing data rate, outlier detection thresholds, signal-to-noise ratio) were used to evaluate the dataset’s quality at each stage? Additionally, it would be beneficial to understand if different domains in Time-300B required unique preprocessing strategies, given that they may have distinct characteristics (e.g., varying degrees of stationarity, seasonality, or noise levels). For instance, were there specific thresholds or criteria applied differently for stationary versus non-stationary time series, or domain-specific noise reduction techniques? Clarifying these details would help illustrate how the pipeline handles domain-specific data imbalances and noisy observations, which is crucial for the model’s generalizability across diverse applications.\n3. Positional Encoding and Temporal Coherence: The paper’s use of rotary positional embeddings is interesting, but it would be beneficial to see a comparative analysis of rotary embeddings against other positional encoding methods (e.g., sinusoidal or learned embeddings) specifically for long-term forecasting. Could the authors provide performance metrics—such as error metrics over extended forecast horizons, robustness to long-range dependencies, and computational efficiency—across these encoding types? Additionally, were there specific challenges or performance trends observed when using rotary embeddings over extended timeframes? For example, did rotary embeddings show advantages or disadvantages in terms of capturing the temporal continuity needed in long-term forecasting, compared to sinusoidal or learned encodings? Providing these comparisons and insights would clarify why rotary embeddings were chosen and how they impact forecast accuracy and computational costs over long sequences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2t3Lxw0Lua", "reviewer": "Reviewer_uVef", "summary": "The paper introduces Time-MoE, a scalable foundation model for time series forecasting that leverages a sparse mixture-of-experts (MoE) architecture. This approach enhances computational efficiency by selectively activating a subset of expert networks for each prediction. The model was pre-trained on a newly proposed large-scale dataset, Time-300B, comprising over 300 billion time points/tokens from nine different domains. Time-MoE's decoder-only structure supports flexible forecasting horizons, and it scales up to 2.4 billion parameters without significant increases in inference costs. The paper demonstrates substantial improvements in forecasting accuracy, positioning Time-Moe as a state-of-the-art solution for time series forecasting.", "questions": "1. While point-wise tokenization is reasonable, neighboring time steps often encode similar information. Did you consider tokenizing multiple time steps as a unit? This might improve context efficiency and extend the effective sequence length.\n\n2. Following from the above, would a hybrid tokenization (mixing single time step and multiple time step tokens) be feasible? Effectively increasing the vocabulary? As far as I see the vocab size is of size 10?\n\n3. Could you present MAE results, particularly for ablations? Seeing those values would help evaluate the robustness of different configurations.\n\n4. Regarding the ablation on loss functions: while L2 loss was tested, other simple losses (like L1) could have been competitive. Did you consider these alternatives before settling on Huber loss?\n\n5. The inference speed results in Table 5 indicate that removing resolution layers increases speed. Could you clarify why retaining multi-resolution layers results in faster inference? This seems counterintuitive.\n\n6. There appears to be some variability in performance between different model sizes (e.g., Time-MoE_large performing worse in some 100B cases than dense models). Do the authors have an intuition for why this happens?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality**: Introducing a sparse Mixture-of-Experts model for time series forecasting is, as far as I know, novel. This work provides the first large-scale configuration of sparse architectures for time series models, addressing efficiency issues as well as overfitting seen in dense approaches.\n\n**Quality**: The manuscript includes multiple experiments, with good evaluation across zero-shot and fine-tuning settings on multiple (but limited domains) benchmarks (compared to multiple llms and pre-llm models), showcasing the generalizability and effectiveness of the approach. I really like that the authors conducted an scalability analysis with their model as well as the dense counterpart, which can also be seen as a contribution due to its sheer parameter size.\n\n**Clarity**: The paper is clearly written and easy to follow, making it accessible to a broad audience. The use of detailed and easy-to-follow figures (e.g., Figure 2) supports understanding of complex architectural components.\n\n**Significance**: Scaling time series forecasting models has traditionally been challenging due to high computational costs and the lack of proper training data. This work shows significant performance improvements over existing dense models as well as prior LLM-based approaches. Furthermore, due to similarities in its components, it represents a good step towards broadening time series research across research communities (i.e., NLP research).", "weaknesses": "While I liked the paper there are some areas which can be improved\n\n### Areas for Improvement \n**W1**: The usage of rotary positional encoding is mentioned but not clearly represented in Figure 2. This leads to some confusion (i.e. at a first glance i thought the authors constructe a set transformer); it would be helpful to clarify this, possibly adding an explicit label to ensure better understanding.\n\n**W2**: The recent work by Liu et al. (2024), which proposes a new one-for-all benchmark, is missing. Given its relevance, including this benchmark would provide a more comprehensive assessment of the model’s generalizability. In general, discussing differences in the related work would be a good idea.\n\n**W3**: In my opinion, it is not necessary to prove again that flash attention is fast and BF16 is reasonable to use (the trade-off). These evaluations could be moved to the appendix for interested readers. Instead, I think it would be a good idea to see model performances over the course of the training with different data sampling strategies.\n\n**W4**: There is limited ablation on certain design choices, specifically around the multi-resolution head and expert allocation/configuration strategies. This could strengthen the papers claims (see more in the questions 1,2,4,5)\n\n**W5**: The lack of standard deviations weakens the paper's results and claims\n\n**W6**: The forecastability table needs to proper reference its origin (Wang et al., 2024)\n\n**W7**: I think it is not enough to evaluate on two domains i.e. Temperature and Weather which are already kind of related.\n\n### Limitations\n- I did not find any addressed limitations regarding their proposed dataset collection and what impact it would have to have datasets with Spurious correlations  in the collection such as P2S (Kraus et al.,)\n- Currently, the decision on which datasets were excluded from the Time-300B collection appears somewhat arbitrary, without strong justification provided for the selection criteria.\n\nLiu et al., AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024)\n\nWang et al., TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting (ICLR 2024)\n\nKraus et al., Right on Time: Revising Time Series Models by Constraining their Explanations (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)