Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Authors: Xu Liu
Juncheng Liu
Gerald Woo
Taha Aksu
Yuxuan Liang
Roger Zimmermann
Chenghao Liu
Junnan Li
Silvio Savarese
Caiming Xiong
Doyen Sahoo
Venue: ICML 2025 poster (2025)
Keywords: Time Series Foundation Models
Sparse Mixture of Experts
Primary area: applications->time_series

Abstract: Achieving effective unified pretraining on large time series corpora remains an open challenge in developing time series foundation models. Existing methods, such as Moirai, introduce multiple projection layers for time series of different frequencies to account for high data heterogeneity. We identify major drawbacks to this human-imposed frequency-level model specialization. First, frequency is not a reliable indicator for grouping pretraining data. Second, time series can display varied distributions even within a short window. Frequency-level specialization overlooks the diversity at this granularity. To address these issues, this paper introduces Moirai-MoE, excluding human-defined data groupings while delegating the modeling of diverse time series patterns to the sparse mixture of experts (MoE) within Transformers. With this design, Moirai-MoE eliminates reliance on heuristics and enables automatic token-level specialization. Extensive evaluations on 39 datasets demonstrate the superiority of Moirai-MoE over state-of-the-art foundation models. This study also conducts comprehensive model analyses to explore the inner workings of time series MoE foundation models.

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Xu Liu
Juncheng Liu
Gerald Woo
Taha Aksu
Yuxuan Liang'
Roger Zimmermann' Doyen Sahoo !
Chenghao Liu'
Junnan Li
Silvio Savarese
Caiming Xiong'
Abstract
downstream forecasting tasks in a zero-shot manner, regard- less of variations in domain, frequency, dimensionality, con- text, or prediction length. This new paradigm significantly reduces the complexity of building numerous specialized models, paving the way for forecasting as-a-service.
Achieving effective unified pretraining large time series corpora remains an open challenge in developing time series foundation models. Ex- isting methods, such as MOIRAI, introduce mul- tiple projection layers for time series of differ- ent frequencies to account for high data hetero- geneity. We identify major drawbacks to this human-imposed frequency-level model specializa- tion. First, frequency is not reliable indicator for grouping pretraining data. Second, time series can display varied distributions even within a short window. Frequency-level specialization overlooks the diversity at this granularity. To address these issues, this paper introduces MOIRAI-MON MoE, ex- cluding human-defined data groupings while dele- gating the modeling of diverse time series patterns to the sparse mixture of experts (MoE) within Transformers. With this design, MOIRAI-MOO eliminates reliance on heuristics and enables auto- matic token-level specialization. Extensive evalu- ations on 39 datasets demonstrate the superiority of MOIRAI-MOE over state-of-the-ar foundation models. This study also conducts comprehensive model analyses to explore the inner workings of time series MoE foundation models.
However, unlike language and vision modalities which ben- efit from standardized input formats, time series corpora are highly heterogeneous, posing significant challenges dur- ing model pretraining. Existing solutions such as UniTime (Liu et al., 2024a) and TEMPO (Cao et al., 2024) lever- age language prompts to discern the source of data, thereby achieving dataset-level model specialization. MOIRAI (Woo et al., 2024) goes step further and proposes a more gran- ular categorization based on time series meta feature frequency. Specifically, they design multiple input/output projection layers with each layer specialized to handle data from a specific frequency, thereby enabling frequency-level specialization. TimesFM (Das et al., 2024) is at this level as well, grouping data via a frequency embedding mapping.
Given the heterogeneity of time series, acknowledge the value of model specialization; however, we argue that human-imposed frequency-leve specialization lacks gen- eralizability and introduces several limitations. First, fre- quency is not a reliable indicator for grouping pretraining corpora. As shown in Figure 1, time series with different frequencies can exhibit similar patterns, while those with the same frequency may display diverse and unrelated pat- terns. This human numan-imposed mismatch between frequency and pattern undermines the efficacy of model specializa- tion. Furthermore, real-world time series are inherently non- stationary (Liu et al., 2022), displaying varied distributions even within a short window. Frequency-level specialization overlooks the diversity at this granularity, underscoring the need for more fine-grained modeling approaches.
1. Introduction
Time series forecasting is experiencing major shift (Liang al., 2024). The traditional approach of developing sepa- rate models for each dataset is being replaced by the con- cept of universal forecasting (Woo et al., 2024), where a pretrained foundation model can be applied across diverse
To tackle the aforementioned issues, this paper introduces MOIRAI-MOE, an innovative solution for effective and efficient time series pretraining, inspired by recent devel- opments of Sparse Mixture of Experts (MoE) Transform- ers (Lepikhin et al., 2021; Fedus et al., 2022; Dai et al., 2024). The core idea of MOIRAI-MON is to exclude human- defined time series groupings while delegating the model-
'Salesforce AI Research NNaioonl University of Singa- pore 'The Hong Kong University of Science and Technology (Guangzhou). Correspondence Chenghao Liu <cheng- hao.liu@ salesforce.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Time series are highly varied within frequency
Monthly
Time series with similor patterns originate from different frequencies
Database
Varied distributions within a single time series
Dally Database
Www
Hourly
Database
Figure 1. An illustration of the challenges arising from grouping time series by frequency and imposing frequency-level model specializa- tion: the diversity of patterns within the same frequency group, the similarity patterns across different frequencies, and the variability of distributions within single time series. The examples presented are real time series in the Monash benchmark (Godahewa et al., 2021).
ing of diverse time series patterns to the sparsely activated experts in Transformer layers. With this design, specializa- tion of MOIRAI- MAIRRI--EEII is achieved in data-driven manner and operates at the time series token level. It also benefits from the enhanced computational efficiency and scalability brought by the MoE architecture. Moreover, this study in- vestigates existing expert gating functions that generally use a randomly initialized linear layer for expert assignments (Shazeer et al., 2017: Jiang et al., 2024) and proposes a new function that leverages cluster centroids derived from an existing pretrained dense model to guide expert allocations.
2. Related Work
Module Specialization for Time Series Foundation Mod- els A key challenge in pretraining time series foundation models lies in accommodating high data diversity, under- scoring the need for designing specialization modules. Ap- proaches like UniTime (Liu et al., 2024a) and TEMPO (Cao et al., 2024) utilize prompts to identify data sources, facil- itating dataset-level specialization. MOIRAI (Woo al., 2024) and TimesFM (Das et al., 2024) refine this further by grouping data based on time series meta feature fre- quency. While Moment (Goswami et al., 2024) and UniTS (Gao et al., 2024) employ different projection heads for var- ious time series analysis tasks, including classification and anomaly detection. Methods like Chronos (Ansari al., 2024), Lag-LLaMA (Rasul al., 2023), and Timer (Liu et al., 2024c) lack specialization modules, potentially in- creasing learning complexity and requiring more parameters to handle diverse inputs. In this work, we propose automatic time series token-level specialization, where diverse tokens are handled by different experts, while similar tokens share parameter space, reducing learning complexity.
We extensively evaluate MOIRAI-MOE using total of 39 datasets in in-distribution and zero-shot forecasting scenar- ios. The results confirm the superiority of MOIRAI-MOE over state-of-the-art foundation models. Additionally, we conduct comprehensive model analyses, as the first attempt, to explore the inner workings of time series MoE foun- dation models. It reveals that MOIRAI-MON acquires the capability to achieve frequency invariint representations and essentially performs progressive denoising throughout the model. Our contributions are summarized as follows:
We propose MOIRAI-MOE, pioneering mixture-of- experts time series foundation model, achieving token- level model specialization in data-driven manner. We introduce a new expert gating function for accurate expert assignments and improved performance.
Mixture-of-Experts Time Series Models In vision and language, sparse mixture of experts (MoE) has proven effec- tive for scaling Transformer capacity while minimizing com- putational overheads (Fedus et al., 2022; Dai et al., 2024; Zhu et al., 2024). The application of Mixture of Experts (MoE) in time series research remains limited, with only a few studies exploring MoE concepts in dataset-specifie train- ing (Ismail al., 2022; Ni et al., 2024). In these studies, the experts are typically linear models, such autoregressive models and DLiinarr (Zeng et al., 2023), which are small- scale models. In contrast, MOIRAI-MOE represents a time series foundation model capable of scaling efficiently with- out significant increase in computational costs. A concur- rent work, Time-MoB (Shi et al., 2024), also integrates MoE
Extensive experiments on 39 datasets reveal that MIORAI-MOE delivers up to 17% performance improve- ments over MOIRAI at the same level of model size, and outperforms other time series foundation models with up 65x x fewer activated parameters.
We conduct thorough model analyses to deepen under- standing of the inner workings of time series MoE foun- dation models and summarize insights for future research.
2

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Muili Oulput Projection
Single Output
Morthly
Daily
Hourly
Projection
All Data
FFN
FFN
FFN
+
t
Transformer
Transformer
Gating Function
Token Embedding
Token Embedding
Multi Input Projection
Single Input Projection
Monthly
Daily
Hourty
All Data
Self Attention
WWlWA
Figure 2. Comparison of MOIRAI (left) and MOIRAI-MOE (right).
into time series foundation models. However, it relies on existing gating functions without proposing new alternatives investigating the internal mechanisms of MoE models. Additionally, Time employs point-wise tokenization, which may fail to capture the underlying semantics of time series data (Zeng et al., 2023), limiting its performance.
ceeding patch, applying a causal normalizer each patch is the most effective way achieve accurate normalization. However, this approach generates N subsequences with dif- ferent lengths, diminishing the parallel training that decoder- only models typically offer. To address this, we introduce the masking ratio r as a hyperparameter, which specifies the portion of the entire sequence used exclusively for robust normalizer calculation, without contributing to the predic- tion loss. Finally, we forward the patches through single projection layer to generate time series tokens E RND. where D is the dimension of Transformers. We pass on the capability of learning diverse time series patterns to the vast number of parameters Transformers. This projection layer is implemented as residual multi-layer perceptron to enhance representation capacity (Das et al., 2023).
3. Methodology
In this section, present MOIRAI-MOF, scalable time series foundation model built upon MOIRAI (Woo et al., 2024). Figure 2 presents comparison. MOIRAI-MOE significantly improves MOIRAT by removing the heuris- tically defined input/output projection layers that handle data of varying frequencies. Instead, MOIRAI-MOE lever- ages sparse mixture of experts within Transformers au- tonomously capture the diverse patterns in time series data, allowing for greater flexibility. In addition, MOIRAI-MON proposes a novel gating function that routes tokens based on the knowledge of an existing pretrained dense model, and adopts the next-token prediction objective to improve pre- training efficiency by enabling parallel learning of various context lengths in a single model update.
3.2. Sparse Hixture-of-Exper Transformers
A decoder-only Transformer (Dubey et al., 2024) is con- structed by stacking L layers of Transformer blocks. The block at the 1-th layer is represented follows:
a! CSA(LN((2)) + x!
(1)
1+1 FFF((LN(E)) =
(2)
3.1. Time Series Token Construction
where CSA, FFN, and LN denote a causal self-attention module, feed-forward network, and the layer normal- ization, respectively. /' E R are the hidden states of tokens after the attention module of the 1-th layer and x' = x/+1 E NxD are the input and output hidden states of the 1-th layer. Note that MOIRAI-MOR considers multi- variate correlations by flattening all variates into a sequence. During causal attention, each token attends to both its pre- ceding tokens and those from other variates, enabling the model to capture intra- and inter-variate dependencies.
Patching techniques, first introduced in PatchTST (Nie et al., 2023), have become a prevalent method in time series fore- casting (Das et al., 2024; Liu al., 2024a; Woo et al., 2024). By aggregating adjacent time series data into patches, this technique effectively captures local semantic information and significantly reduces computational overhead when pro- cessing long inputs. Given a time series with length S, we segment it into overlaaalg patches of size P, resulting sequence of patches a E IR N x where N N=11 $1.
Next, we establish the mixture of experts by replacing each FFN with a MoE layer, which composed of M expert networks {E....., EM and a gating function G. Only subset of experts activated for each token, allowing
We then normalize the patches mitigate distribution shift issues (Liu et al., 2022; Wu et al., 2023). In decoder-only (autoregressive) model, where each patch predicts its suc-
3

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
experts to specialize in distinct patterns of time series data and ensuring computational efficiency. The output of the MoE layer is computed as:
Token Clusters as Gating Function Technically, we first pretrain MOIRAT model using single-patch input/output projection layers to remove the human-imposed frequency biases in MOIRAL. We then perform inference using our pretraining data LOTSA (Woo et al., 2024). For a hatch B containing T tokens, we extract the attention outputs á' E RT'xD each layer and perform mini-batch k-means clustering on them to continuously update cluster centroids C B xx at each layer, where the number of clusters is set to match the total number of experts. During MoE pre- training, for each layer, each token computes the Euclidean distance to the cluster centroids, and these distances serve as token-to-exper ex-ert affinity scores for expert assignments:
M
G(a") E,('') Σ
(3)
where E4 (a')) is the output of the i-th expert network, and G( is the i-th coken-to-expert affinity score generated by the gating function. Following (Lepikhin et al., 2021: Zhu et al., 2024; Jiang et al., 2024), we set the number of activated experts to K =. 2.
3.2. 1. GATING FUNCTION
G(ø²) Sottmax(TooKKEuullddee
Ci)))
The most common linear gating mechanism projects MoE inputs a' by weights W, E DxM, then selects the top-K logits and applies the softmax function over them (Shazeer et al., 2017; Jiang et al., 2024; Dai et al., 2024):
3.3. Pretraining Objective
Let T4-1+1:: = { ( 2t} denote the context win- dow of length for a token at position In this study, to facilitate both point and probabilistic forecasting, our goal is formulated as forecasting the distribution of the next token p( 16) by predicting the mixture distribution parameters I (Woo et al., 2024). These parameters are derived from the output tokens of the Transformer, followed by single out- put projection layer. The following negative oog-likelihoo is minimized during pretraining:
G( Softmax(TopK(a' Wg))
(4)
To mitigate the load balancing issue (Shazeer et al., 2017) with this sparse gating, an auxiliary loss is typically applied to encourage an even distribution of tokens across experts (Lepikhin al., 2021; ; Jiang et al., 2024; Dai et al., 2024). Formally, the load balancing loss for batch B containing T tokens is defined as:
T
Ppedd = - log (ogp((++1 お), و fo(w (o
(8)
Liond
Σ D.P., D. i=1
Σ
1
I (Toeen selects Expert i},
T
(5)
4. Experiments
T Σ G( P. = T 1=1
(6)
4.1. MOIRAI-MOE Setup
To ensure fair comparison with MOTRAT in terms of ac- tivated parameters, we set the number of activated experts to for MOIR resulting in 11M/86M activated parameters per token for closely matching the dense model MOIR M1SRAA//MMIRRAI that contains 14M/91M IM activated parameters. The total num- ber of experts is set to 32, yielding total parameter sizes of 117M for MOIRAI-MOEs and 935M for MOIRAI-MOEB. MOIRAI-MOE, is not presented due to the significant re- quirements of computational resources. The specific con- figurations are outlined in Table I. See the effects of the number of MoE layers in Appendix B.2.
where 1 is the indicator funetion, D, denotes the fraction of tokens routed to expert i, and P; indicates the proportion of the gating probability allocated to expert i. The loss Load is applied to each Transformer layer I. The loss is then aggregated by computing the mean across all layers and added the prediction loss fpeed with weight of 0.01 (Jiang et al., 2024; Dai et al., 2024).
While effective, these gating functions are learned entirely from scratch, and poor initialization may negatively impact their performance. Recent advances developing MoE leveraged pretrained dense models introduce inductive bias for enhancing performance (Komatsuzaki et al., 2022: Qwen, 2024). Inspired by these successes, propose novel gating mechanism using cluster centroids derived from dense model to guide expert allocations. The intu- ition is that clusters of pretrained token embeddings more closely reflect the real distribution of data, leading to more effective expert specialization and improved performance.
Table 1. Model configurations of MOTRAL and MOIRAI-MOF.
Model
Layers
dassal
dg
Activated Params
Total Params
MOIRAIs
384
1.024
14M
14M
MOIRAIS
768
2,048
91M
91M
MOIRALE
24
1,024
2,736
310M
310M
MOIRAI-MOEs
384
512
11M
117M
MOIRAI-MOEg
768
1,024
86M
935M

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
1.24
1.23
Baselines
Foundation Models
Aggregated MAE
1.08
Moirai MO
3.05
0.94
0.54
0.92
0.91
0.91
0.89
0.78
0.67
0.72
0.7
0.63
0.5
LLATiIe
Theaa
Arasa
Stasonal Halve
ET3
BEARS
Canteoee
Dhspult
Waverfee
Timesttee
Contras Smaafe
Chramar- Bate
Chenei Larger
Mairal smell
MoiratBase
Maitt Lage
Hoiral Hot Small
Hoinll MoE-Baa
Figure 3. In-distribution evaluation using 29 datasets from the Monash benchmark (Godahewa et al., 2021). * indicates that the methods includes the datasets here in their pretraining corpora. The aggregated MAE is reported, with cach dataset's 's MAE normalized by scasonal naive, followed by geometric mean. Datasct dotails, full results, and context lengths are in Appendix A.1 and A.4.
4.2. Main Results
all zero-shot performance, outperforming baselines that included partial test data in their pretraining corpora. When compared to all sizes of MOIRAI, MOIRAI-MOEs de- livers a 3%-14% improvement in CRPS and an 8%-16% im- provement in MASE. These improvements are remarkable, considering that MOIRAI-MOEs has only 11M activated parameters 28x fewer than MOIRAIL.
In-distribution Forecasting We begin with an in- distribution evaluation using total of 29 datasets from the Monash benchmark (Godahewa al., 2021). Their train- ing set are included in LOTSA (Woo et al., 2024), holding out the test set which we now use for assessments. Figure 3 summarizes the results based on the aggregated mean absolute error (MAE), in comparison with the baselines pre- sented in the Monash benchmark and following foundation models: LLMTime (Gruver al., 2023), TimesFM (Das al.. 2024), Chronos (Ansari al., 2024), and MOIRAI (Woo et al., 2024). Dataset details and full results are provided in Appendix A. The evaluation results show that MOIRAI-MOE beats all competitors. In particular, MOIRAI-MOEs drastically surpasses its dense counterpart MOIRAIS by 17%, and outperforms the larger MOIRAIg and MOIRAT, by 8% and 7%, respectively. MOIRAI-MOEg delivers further 3% improvement over MOIRAI-MOES. Compared to Chronos, which MOIRAI could not surpass, MOIRAI-MOE successfully bridges the gap and delivers superior results with up to 65x fewer activated parameters.
Summary Our extensive evaluation validates the effec- tiveness of MOTRAT-MOE's design and showcases its strong generalization abilities. It also highlights the superiority of token-level specialization over frequency-based approaches (TimesFM, MOIRAI) and models without specialization modules (Chronos, TTM). Additionally, MOIRAI-MOF sig- nificantly outperforms Time-MoE, underscoring the advan- tages of cluster-based gating and patch- match-based tokenization.
4.3. Ablation Studies
Model Design In the main results, we simultaneously enable the mixture of experts and switch the pretraining objective from a masked filling approach to a next-token prediction approach. To ensure more rigorous comparison, we conduct further experiments where only the learning ob- jective is changed. Table presents the Monash evaluation results, with the first and last rows representing MOIRATS and MOIRAI-MOESs respectively. This outcome suggests that altering the learning objective alone yields modest per- formance improvements, while the major gains stem from leveraging experts for automatic token-level specialization.
Zero-shot Forecasting Next, we conduct an out-of- distribution evaluation on 10 datasets not included LOTSA (see dataset details in Appendix A.2). To establish a compre- hensive comparison, we report results for both probabilis- tic and point forecasting using 6 metrics. The continuous ranked probability score (CRPS) and mean absolute scaled error (MASE) are shown in Table 2, with additional metrics in Table and 10. For baselines, we compare against foun- dation models: TTM (Ekambaram et al., 2024), Timer (Liu al., 2024c). TimesFM, Chronos, MOIRAL, and Time-Mof (Shi et al., 2024), as well as state-of-the-ar full-shot mod- els trained individual datasets: TiDE (Das et al., 2023), PST (Nie et al., 2023), iTransformer (Liu et al., 2024b), and MoLE-DLinear (Ni et al., 2024). Context lengths of all methods are provided in Appendix A.4. Due to page limits, partial baselines are presented in Table 2 and full results are in Table MOFRAI-MOF achieves the best over-
Prefraining Objective We adopt the next-token predic- tion objective for its superior pretraining efficiency com- pared to the masked filling. To illustrate this, we conduct experiments with varying pretraining steps, as shown in Figure 4 (left). The results show that the next-token pre- diction approach consistently outperforms masked filling at each evaluated step. Moreover, next- prediction pre- training with 50k steps achieves comparable performance
5

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table Zero-shot performance comparisons with * indicates the non-zero-shig datasets. The Avg column is computed by normalizing
each dataset's 's MAE by seasonal naive, then applying the geometric mean. Two Avg values are shown: one that averages all, and another (non-leak) excludes Electricity and Solar. Best results are in red. Second best results are in blue. Power: Turkey Power. Traffic: Istanbul
Traffic. Weather: Jena Weather. BizITObs: Biz/TObs-L22. Dataset details, full results, and context lengths are in Appendix A.2 and A.4,
Method
Metric
Electricity
Solar
Power
ETTI
ETT2
Traffic
MIDENSE
Walmart
Weather
BizITObs
Avg (all)
Avg (non-leak)
CRPS
0,070
0.512
0.085
0,515
0.205
0,257
0.294
0.151
0.068
0,262
1.000
1.000
Seasonal Naive
MASF
0.881
1,203
0.906
1.778
1,390
1.137
1.669
1,236
0.782
0.986
1.000
1.000
CRPS
0.057
0.443
0,056
0.344
0.129
0.105
0.072
0,070
0.053
0.077
0.540
0.483
iTransformer
MASE
0.875
1.342
1.076
2.393
1.841
0.581
0.727
0.761
0.623
0,271
0.767
0.708
CRPS
0,075
0,534
0.059
0,417
0.122
0,210
0.150
0.192
0.055
0.102
0.758
0.697
TTM
MASE
0.802
1.255*
0.898
1.934
1.547
0.901
1.195
1.477
0.506
0.308
0.831
0.798
TimesFM
CRPS
0,045*
0,456
0,037
0.280
0.113
0.131
0.070
0,067
0.042
0.080
0,488
0.439
MASE
0.655
1,391
0.851
1,700
1.644
0.678
0,702
0,735
0.440
0,310
0.689
0. 0.640
Chronoss
CRPS
0.043*
0,389
0.038
0,360
0.097
0.124
0.087
0.079
0.089
0.087
0.543
0.513
MASE
0.629
1,193*
0.717
1.799
1.431
0.622
0.834
0.849
0.606
0.301
0.694
0.661
Chronose
CRPS
0
0.341
0,039
0.387
0.092
0.109
0.075
0,080
0.058
0.084
0.499
0.471
MASE
0.617
1.002*
0.722
1,898
1.265
0.553
0.712
0,849
0,583
0.301
0.656
0.631
Chronos,
CRPS
0
0.339
0.038
0.404
0.091
0.117
0.075
0.073
0.062
0.084
0.500
0.473
MASE
0.615
0.987*
0.702
1.959
1,270
0.597
0.724
0,788
0.601
0.310
0.660
0.638
CRPS
0.072
0.471
0,048
0.275
0.101
0.173
0,084
0,103
0.049
0.081
0.578
0.507
MOIRAIs
MASE
0.981
1.465
0,948
1.701
1.417
0.990
0.836
1.048
0.521
0.301
0.798
0.726
MOIRAIs
CRPS
0,055
0.419
0,040
0,301
0.095
0.116
0.104
0.093
0.041
0.078
0.520
0.467
MASE
0.792
1.292
1.736
0.644
1.101
0.291
0.736
0.685
MOIRAI
CRPS
0.050
0,406
0.036
0,286
0.094
0.112
0.095
0.098
0.051
0.079
0.514
0.467
MASE
0.751
1.237
0.870
1.750
1.436
0.631
0.957
1.007
0.515
0.285
0.729
0.685
Time-MoFig
CRPS
0
0.230
0.044
0.392
0.125
0,152
0.099
0.100
0.070
0.112
0.583
0.586
MASE
0
0.535*
1.823
1.672
0.672
0.846
0.558
0.343
0.695
Time-MoFe
CRPS
0
0.294
0.045
0,386
0.131
0.172
0.090
0.097
0.058
0.111
0.589
0.576
MASE
0,581
0.689*
0,790
1.773
1.878
0.762
0.759
0.817
0.524
0.337
0.678
0.695
MOIRAI-MOES
CRPS
0.046
0.429
0.036
0,288
0.093
0.108
0.071
0.090
0.056
0.081
0.497
0.450
MASE
0.719
1.222
1.750
1.248
0.563
0.746
0.298
0.620
MOIRAI-MOFs
CRPS
0.041
0.382
0.034
0,296
0.091
0.100
0.071
0.088
0.057
0.079
0.478
0.439
MASE
0,638
1.161
0,725
1.748
1.247
0,510
0.721
0,918
0.509
0,290
0.651
0.611
to masked filling with 100k steps, highlighting the substan- tial efficiency gains provided by the next-token prediction objective. In Appendix B.I, we present further results il- lustrating how MOIRAI-MOE's downstream performance evolves during pretraining, compared to MOIRAL
4.4. Model Analyses
In this section, we delve deeper into the learned to- ken embeddings and expert assignment distribution of MOIRAI-MOn MOE shed light on the inner workings of the time series MoE foundation model.
Gating Function In Figure 4 (right), we vary the total number of experts and examine the impact of gating func- tions on performance. Across all gating functions, per- formance consistently improves as the number of experts increases until 32. Moreover, our proposed token clustering method proves to be consistently superior to other gating variants across all expert configurations. This indicates that the clustering approach aligns closely with the distribution of data representations that have been optimized in pretrain- ing, leading to more effective expert specialization than randomly initialized gating.
Obs 1: MOIRAI-MOE produces token embeddings in a data-driven way, effectively improving performance. In Figure 5, we utilize the T-SNE visualization (Van der Maaten & Hinton, 2008) compare the token embeddings generated from the input projection layers of MOIRAI and MOIRAI-MOE. (1) In the first row, we examine the NN5 Daily and Traffic Hourly datasets, which have different frequencies but exhibit similar underlying patterns (visual- izations of these patterns are in Appendix D). The figure illustrates that MOIRAI produces distinct embeddings due to the use of separate frequency projection layers, while MOTRAT- successfully blends their representations to- gether. Their inherent similarities are further demonstrated by their comparable expert distributions in the last two columns. (2) In the second row, we analyze another daily frequency dataset, Covid Daily Deaths, which shows dis-
Table 3. Model variants performance on Monash.
Model Variant
Aggregated MAE
Multi Projection w/ Masked Filling Multi Projection w/ Next- Token Prediction
0.78
0.75
Single Projection& & MoE w/ Next- Token Prediction
0.65
6

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
0.81
0.74
Projection w/ Masked Filling
Linear Projection
Aggregated MAE
Aggregated MAE
0.79
Multi Projection Next laken Prediction
0.72
Linear Projection w/ Load Balance Taken Clusters
0.70
0.75
0.68
0.73
0.66
0.71
0.64
25k
50k
75k
100k
125k
16
32
64
Training Steps
Number of Experts
Figure 4, Ablation studies of the pretraining objective and gating function using MOIRAI-
Moiral Taken Embedding
MoE Token Embedding
MoE Expert Allocation
MoE Expert Allocation
NNS Daiy Tullic Hourty
1.
MR5 Daty
VNS Duily
Tratic Hoaryy
Trafiic Hoarly
Token (%)
Token (%)
0.0
-1.0
-10
10
-1.5
-1.0
-0.5
0.0
0.5
1.0
10
20
0.0
20
30
0.3
tass Daily
tass Daily
VNS Daiy
Coo Da Deaths
Covid a Daiy Seaths
10-
Coun Da Deatrs
Taken (%6
Taken (%)
-5
-10
-5
10
-20
-20
30
20
10
20
30
1.
20
30
Figure 5. The first two columns are the comparison of embeddings from MOIRAIS and MOIRAI-MOES. The last two columns the expert assignment distributions of MORAI-MOOE in layer I: the x-axis corresponds to the 32 experts in a layer, and the y-axis is the proportion of tokens that choose experts.
tinct patterns compared to NN5 Daily. We observe that the embeddings of these two datasets overlap to some extent in MOIRAI but are effectively separated in MOIRAI-MOE. Moreover, the Covid Daily dataset shows a different ex- pert distribution than NNS Daily due to different token embeddings. The data-driven modeling paradigm of MOIRAI-MOE ultimately leads to significant perfor- mance boosts, reducing the NN5 Daily MAE from 5.37 to 4.04 (a 25% improvement), the Traffic Hourly MAE from 0.02 to 0.013 (a 35% improvement), and the Covid Daily Deaths MAE from 124.32 to 119 (a 4% improvement..
nize that some experts in MOIRAI-MOE are rarely selected during inference on Monash. Pruning these underutilized experts for inference efficiency is left for future work.
Obs 3: Shallow layers have more routing preferences than deep layers. According to Figure 6, in the shallow layers, the model relies multiple experts to manage the high level of short-term variability, such as cyclical, sea- sonal, or abrupt changes. As tokens are aggregated in deeper layers, the model shifts its focus to more generalizable tem- poral dependencies, such as broader trends and long-term patterns, that can be shared across different frequencies and leads to more concentrated experts being selected. This behavior contrasts with patterns observed in LLMs (Zhu et al., 2024), where earlier layers typically concentrate on limited number of experts to capture common linguistic features, while deeper layers target more task-specific char- acteristics. This divergence may stem from the dynamic and noisier nature of time series tokens, which are gen- erated from small time windows, unlike language tokens derived from a fixed vocabulary. Our findings suggest that denoising processes occur progressively through- out the model. This observation aligns with conclusions from GPT4TS (Zhou et al., 2023), which found that as the layer depth increases, token vectors are projected into the low-dimensional top eigenvector space of input patterns.
Obs 2: Different frequency data exhibit different expert selection distributions at shallow layers but similar distri- butions at deep layers. We present the expert allocation distributions on Monash grouped by frequency in Figure 6. In the shallow layers, expert selection is notably diverse across frequencies. While by the final layer (layer 6), expert allocation becomes nearly identical across all frequencies, suggesting that the model has abstracted time series into high-level representations largely independent of the fre- quency. This evidence indicates that MOIRAI-MOE effec- tively achieves frequency-ine irearancy-nvvariat hidden representa- tions, which are crucial for model generalization (Van Ness et al., 2023). The shared parameter space in the last layer also shows that it is sufficient for generating representations needed make diverse predictions. Additionally, we recog-
7

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
MoE Layer
MoE Layer
MoE Layer
0.4
0.6
0.6
Token (%)
Yearly Quarterly
0.2
0.3
0.3
0.0
0.0
0.0
10
20
30
10
20
30
10
20
30
0.4
0.6
0.6
Token (9%)
Monthly
0.-
0.3
0.3
0.0
0.0
0.0
0
10
20
30
0
10
30
10
20
30
0.4
0.6
0.6
Token (%)
Weekly
0.2
0.3
0.3
0.0
0.0
0.0
20
30
10
30
10
30
0.4
0.6
0.6
Token (%)
Daily
0.2
0,3
0.3
0.0
0.0
0.0
30
10
30
0.4
0.6
0.6
Token (%)
Hourly
0.
0.3
0.3
0.0
0.0
0.0
Expert Index
Expert Index
Expert Index
Figure 6. Visualization of the distribution of expert allocation for MOIRAI-MOEs layers 2, 4, and 6 (the last layer) using the Monash henchmark grouped by time series frequency.
Table 4. Inference cost evaluation. The values in brackets represent the parameter sizes of the foundation models. For MoE models, the two values indicate the number activated parameters and total number of parameters. The spent time is seconds.
Chronoss
Chronosg
Chronos,
MOIRAIS
MOIRAIg
MOIRA4,
MOIRAI- MoFF
MOIRAI-MOF (86M/935M)
Model
(46M)
(200M)
(710M)
(14M)
(9IM)
(310M)
(1 17M)
Spent Time (s)
551
1,177
2,780
264
358
537
273
370
4.5. Efficiency Analyses
tains the same level of efficiency as MOIRAT but also de- livers substantial performance improvements. Additionally, when comparing MOIRAI-MoR. to Chronos, which also ploys autoregressive inference, we find that MOIRAI-MOE is significantly faster. This speed advantage stems from the fact that MOIRAI-MOE generates predictions using patches of size 16, while Chronos uses patch size of 1, which greatly affects its inference efficiency.
In this section, we aim to validate whether the inference speeds of MOIRAT and MOIRAI-MOE are comparable, as have configured them with similar activated parame- ters. Additionally, due to the difference in the inference algorithms (the mask filling in MOTRAT predicts all tokens simultancously, while the next-token prediction approach in MOIR AI-MOE generates predictions autoregressively), we evaluate the inference cost on a subset of the Monash benchmark where the predicted token is one (correspond- ing 16 time steps) to eliminate this discrepancy. To also compare to the foundation model Chronos, we align with the setting in Chronos by setting the context length to 512 and the number of sampling samples to 20.
5. Conclusion
In this work, we introduce a pioneering time series MoE foundation model MOIRAI-MOF that utilizes sparse experts to model diverse time series patterns in a data-driven manner. Empirical experiments demonstrate that MOIRAI-MOE not only achieves significant performance improvements over all sizes of its predecessor MOIRAL, but also outperforms other competitive foundation models with much fewer acti- vated parameters. Moreover, comprehensive model analyses
We present the summarized results in Table and conclude that MOIRAI-MOEs and MOIRAI-MOER exhibit similar inference times MOIRAIS and MOIRAIg, respectively. These results highlight that AI-MAI-MOE not only main-
8

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
have been conducted to gain deeper understanding of time series MoE foundation models.
Gao, S., Koker, T., Queen, O., Hartvigsen, T., Tsiligkaridis, T., and Zitnik, M. Units: A unified multi-task time se- ries model. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Impact Statement
Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., and Montero-Manso, Monash time series forecasting archive. arXiv preprint arXiv:2105. 06640000066633.. 2021.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and
Dubrawski, A. Moment: family of open foundation models. arXiv preprint arXiv:2402.03885. 2024.
References
Ansari, A. F., Stella, L.,. Turkmen, C., Zhang, X., Mercado, P., Shen, H., Shchur, O., Rangapuram, S. S., Arango, S. P., Kapoor, S., et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403. 077851,, 2024.
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G. Large language models are zero-shot time series forecasters. In Advances in Neural Information Processing Systems, 2023.
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
Han, X., Zheng, H., and Zhou, M. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18100-18111 2022.
Cao, D., Jia, F., Arik, S. O., Plister, T., Zheng, Y., Ye, W., and Liu, Y. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. In International Conference Learning Representations, 2024.
Ismail, A. A., Arik, S. Ó., Yoon, J., Taly, A., Feizi, S., and Pfister, T. Interpretable mixture of experts. arXiv preprint arXiv:2206.00 02 2022.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.0444 040088488, 2024.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J. Zeng, W. Yu, X., Wu, Y., et al. Deepseekmoe: To- wards ultimate expert specialization in mixture-of-experr language models. In Association for Computational Lin- guistics, 2024.
Jiang, J., Han, C., Jiang, W., Zhao, W. X., and Wang, J. To- wards efficient and comprehensive urban spatial-temporal prediction: A unified library and performance benchmark. arXiv e-prints, pp. arXiv-2304, 2023.
Das, A., Kong, W., Leach, A., Mathur, S., Sen, R., and Yu, R. Long-term forecasting with tide: Time-series dense encoder. Transactions on Machine Learning Research, 2023.
Komatsuzaki, A., Puigcerver, J., Lee Lee-Thorp. J., Ruiz, C. R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture- of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055, 2022.
Das, A., Kong, W., Sen, R., and Zhou, Y. A decoder- only foundation model for time-series forecasting. In International Conference on Machine Learning, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR confer- ence on research development information retrieval, pp. 95 104, 2018
Ekambaram, V., Jati, A., Nguyen, N. H., Dayama, P., Reddy, C. Gifford, W. M., and Kalagnanam, J. Ttms: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. arXiv preprint 1.XXi:24011.39555 2024.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Rep- resentations, 2021.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, pp. 1-39, 2022.
Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., and Wen, Q. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6555-6565, 6565, 2024.
Shi, X., Wang, S., Nie, Y., Li, D., Ye, Z., Wen, Q. and Jin, M. Time-moe: Billion-scale time series founda- tion models with mixture of experts. arXiv preprint arXiv:2409. 16040, 2024.
Liu, X., Hu, J., Li, Y., Diao, S., Liang, Y., Hooi, B., and Zimmermann, R. Unitime: A language-empowered uni- fied model for cross-domain time series forecasting. In Proceedings of the ACM on Web Conference 2024, pp- 4095-4106, 2024a.
Trindade, A.
Electicitylaaddigraan22111111111 12014.
UCI Machine Learning Repository, 2015. DOI:
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. In Advances in Neural Information Process- ing Systems, pp. 9881-9893, 2022.
Van der Maaten, and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, (11), 2008.
Van Ness, M., Shen, H., Wang, H., Jin, X., Maddix, D. C.,
and Gopalswamy, K. Cross-freques fressffry time series meta- forecasting. arXiv preprini urXiv:2302.02077 2023.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. In International Conference on Learning Representations, 2024b.
Walmart Competition Admin, W. C. Walmart recruiting store sales forecasting, 2014.
Woo, G., Liu, C.,, Kumar, A., Xiong, C., Savarese, S., and Sahoo, D. Unified training of universal time series fore- casting transformers. In International Conference on Machine Learning. 2024.
Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., and Long, M. Timer: Generative pre-trained transformers are large time series models. In Forty-first International Conference on Machine Learning, 2024c.
Wu, H., Xu, J., Wang. J., and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term series forecasting. In Advances in neural information 2021.
Ni, R., Lin, Z., Wang, S., and Fanti, G. Mixture-of-linear experts for long-term time series forecasting, In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 4680, 2024.
processing systems, pp.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation modeling for gen- eral time series analysis. In International Conference on Learning Representations, 2023.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Leurning Representations, 2023.
Yao. J., Pan, W., Ghosh, S., and Doshi-Velez, F. Quality of uncertainty quantification for bayesian neural network 2019.
Palaskar, S., Fkambaram, V., Jati, A., Gantayat, N., Saha, A., Nagar, S., Nguyen, H., Dayama, P., Sindhgatta, R., Mohapatra, P., et al. Automixer for improved multivariate time-series forecasting on business and it observability data. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 22962-22968, 2024.
inference. arXiv preprint arXiv:/906.0000
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1121-11128, 2023.
Qwen. Qwen 1.5-mee: Matching model per- formance with 1/3 activated parameters", February 2024. URL https::/ /ttt:://qwnnlmm wwwreenmm..ithu..... wwt.gg.ithhoiooo io/blog/ qwen-moe/.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, pp. 11 11106- 11115, 2021.
Rasul, K., Ashok, A., Williams, A. R., Khorasani, A., Adamopoulos, G., Bhagwatkar, R., Biloš, M., Ghonia, H., Hassen, N. V., Schneider, A., et al. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint ar 110:22300000288. 2023.
Zhou, T., Niu, P., Sun, L., Jin, R. et al. One fits all: Power general time series analysis by pretrained Im. In Advances in neural information processing systems, pp. 43322-43357 2023.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gate mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Zhu, T. Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-moe: Building mixture-of-experts from llaaa with continual pre-training. arXiv preprint arXiv:2406. 16554, 2024.
10

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
A. Experimental Details
A.1. In-distribution Forecasting Datasets and Full Performance Results
Following MOIRAI (Woo et al., 2024), we perform evaluations on 29 datasets from the Monash benchmark (Godahewa et al., 2021), including M1 Monthly, M3 Monthly, M3 Other, M4 Monthly, M4 Weekly, M4 Daily, M4 Hourly, Tourism Quarterly, Tourism Monthly, CIF 2016, Australian Electricity Demand, Bitcoin, Pedestrian Counts, Vehicle Trips, KDD Cup 2018, Australia Weather, NN5 Daily, NN5 Weekly, Carparts, FRED-MD, Traffic Hourly, Traffic Weekly, Rideshare, Hospital, COVID Deaths, Temperature Rain, Sunspot, Saugeen River Flow, and US Births. The statistics of data are provided in Table 5 and full results of time series foundation models are shown in Table 6.
Table Summary of datasets used in in-distribution forecasting evaluations.
Dataset
Domain
Frequency
Number of Series
Prediction Length
MI Monthly
Econ/Fin
M
617
18
M3 Monthly
Econ/Fin
1,428
M3 Other
Econ/Fin
M
174
M4 Monthly
Econ/Fin
48,000
18
M4 Weekly
Econ/Fin
W
359
M4 Daily
Econ/Fin
D
4,227
14
M4 Hourly
Econ/Fin
414
Tourism Quarterly
Econ/Fin
Q
427
Tourism Monthly
Econ/Fin
366
CIF 2016
Econ/Fin
M
72
12
Aus. Elec. Demand
Energy
30T
5
336
Bitcoin
Econ/Fin
D
18
30
Pedestrain Counts
Transport
II
66
Vehicle Trips
Transport
D
329
30
KDD Cup 2018
Energy
H
270
168
Australia Weather
Nature
3,010
30
NN5 Daily
Econ/Fin
D
111
56
NN5 Weekly
Econ/Fin
III
Carparts
Sales
M
2,674
12
FRED-MD
Econ/Fin
107
Traffic Hourly
Transport
H
862
168
Traffic Weekly
Transport
W
862
Rideshare
Transport
2,304
168
Hospital
Hcalthcare
M
767
12
COVID Deaths
Hcalthcare
266
30
Temperature Rain
Nature
D
32,072
30
Sunspot
Nature
D
Saugeen River Flow
Nature
D
30
US Births
Healtheare
A.2. Zero-shot Forecasting Datasets and Full Performance Results
Dataset Details We conduct zero-shot evaluations on the datasets listed in Table 7, which cover five domains and span frequencies ranging from minute-level to weekly. We use a non-overlapping rolling window approach, where the stride equals the prediction length. The test set consists of the last / time steps, where h is the forecast horizon and r number of rolling evaluation windows. The validation is defined the last forecast horizon before the test set, while the training set includes all preceding data.
Dataset Selection Justification The benchmarks set by previous studies specifically the datasets ETTh1 (H), ETTh2 (H), ETTml (15T), ETTm2 (15T), Electricity (H), Weather (1OT), Traffic (H), and Solar (10T) exhibit a strong bias toward the electricity data and the frequencies of hourly and minutely. Additionally, ETTh1 and ETTml, as well as ETTh2 and ETTm2, are essentially derived from the same underlying data. Furthermore, some of these datasets were already part of our pretraining corpus, raising potential data leakage concerns.

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table Full MAE results of time series foundation models the Monash Benchmark. Other baseline results in (Woo et al., 2024).
Doraset
Seasonal Naive
LLMTime
TimesfM
MoIRAls
MoIRAls
MoIRAlaage
Chrontsiual
Chreorigue
Chronokaage
MOIRAl Mo
MOIRAL Mo
MI Monthly
2,011.96
2,562.54
1673.00
2,082.25
2.068.63
1,983.18
1.797.78
L,637.66
1.627.11
1,992.09
1.81194
Mosthl
783.95
713.41
664.03
644.38
622.27
619.79
617.31
263.54
196.59
179.92
M4 Moutlly
700.24
728.27
580.20
597.60
592.09
584.36
992.85
598.46
534.78
509, 25
544 DM
M4 Weelly
347.99
518.44
285.89
339.76
328.01
301.52
264.56
252.26
248.89
302
278.37
18210
169.91
172.4
Hourly
576.06
196.20
20804
197.29
214.18
230.70
201.14
241.58
217.35
Tourism Quarterly
11,00545
16,918.80
10,568.01
18,352.4
17,196.56
15,820.02
7,83.277
8.835.52
8.521.70
9, 3,508
7.374.27
Tourism Moothly
1,980.21
6.60861
2,422.01
3,500,85
2,862.06
2,688.55
2,465.10
2,358.67
2,140.73
2,523.66
2
649.110.99
453,611.21
Aus. Flee. Demund
453.96
760.81
525.73
266.57
201.30 39
177.68
267. 18
27
215:28
327 92
Diizzin
7,78C+17
1,74++18
7,78E+17
1.76C+18
1622++18
1.870+18
2,341+18
2.270+188
1888+18
1.552+18
1 1.00++18
Pedestiaan Counts
54.88
27.34
43.35
Nehacle Trigs
31,45
24-46
21.85
21,65
KDD Cup 2018
47.09
42.72
40.86
39.1
38,65
39.09
38,60
42.36
38.83
40.21
40.95
Australia Waather
2.36
2,17
2.07
1.96
1,80
1.75
1.96
1.54
1.55
1.76
1.75
NNS
NNS Weekly
15.76
15.07
15.12
15.74
Carguns
0.67
0.44
0,50
0.53
0.47
0.49
0.52
0.54
0.53
0.45
0.44
FRED-MD
5,385.53
6400464
2,37763
2,568.48
2,699.20
2,792.15
938.46
1,036.67
863.99
1,651.76
2,275.61
Hourly
Traffic Woekly
1.15
1.06
1.17
1.13
1.14
1.12
Rideshare
1.60
6,28
1,36
1.35
1,39
1.29
1.27
1.33
1.30
1.26
1.26
Hospin]
20.01
25.68
18.54
23.00
19,40
19.44
19.74
19.75
19.88
20.17
19.60
COVID Deatts
124.32
119:00
Temperature Rain
9.39
6.37
5:27
5.30
3.08
5.27
5.35
5.17
5.19
5.33
5.36
Sumspo
3.9
5,077
1,07
0.31
0,08
01.33
0.20
2.45
3.45
0.10
11,05
Saugren River Flow
34,84
25.16
24.07
24.40
23.57
25.54
26.25
23.05
24.40
Rartss
1,152.67
1,374.99
872.51
476.50
432.14
420008
385.24
To address these issues and establish more diverse zero-shot evaluation scenario without data leakage, we adopt six zero-shot datasets from the MOIRAI paper and incorporate four additional datasets. These added datasets enhance the diversity of our evaluation, making our evaluation cover five domains and include frequencies ranging from minute-level to weekly. However, due to the inherent nature of some datasets, such as limited test set lengths, it is not feasible to apply the same prediction length uniformly across all data. Consequently, the prediction lengths vary across datasets in this study.
Full Performance Results Measured by Metries To comprehensively evaluate performance, we use three point forecasting metries: mean absolute scaled error (MASE), mean squared error (MSE), and mean absolute error (MAE). For probabilistic forecasts, evaluate using continuous ranked probability score (CRPS), prediction interval coverage probability (PICP) (Yao et al., 2019), and quantile interval coverage error (QICE) (Han al., 2022).
Performance results are presented in Table 8, 9, and 10. The results show that MOIRAI-MOE achieves the best overall performance across all 6 metrics, highlighting its superiority modeling point forecast accuracy, distribution accuracy and distribution coverage.
Table Summary of datasets used in the zero-shot forccasting evaluations.
Dataset
Domain
Frequency
Prediction Length
Rolling Evaluations
Electricity (Trindade, 2015)
Energy
24
7
Solar (Lai et al., 2018)
Energy
24
7
Turkey Power
Energy
H
24
7
ETTI (Zhou et al., 2021)
Energy
30
3
ETT2 (Zhou al., 2021)
Energy
D
30
3
Istanbul Traffic
Transport
24
7
M-DENSE (Jiang et al., 2023)
Transport
30
3
Walmart (Walmart Competition Admin, 2014)
Sales
W
8
4
Jena Weather (Wu al., 2021)
Nature
1OT
144
7
BizITObs-LD2 (Palaskar et al., 2024)
Web/CloudOps
5T
48
20
12

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table Full zero-shot performance comparisons measured by CRPS and MASE. * indicates the non-zero-hoot datasets, The Avg column is computed by normalizing each dataset's MAE by seasonal naive, then applying the geometric mean. Two Avg values shown: that averages all, and another
excludes Electricity and Solar. Best results are red. Second best results are in blue. Power:
Turkey Power. Traffic: Istanbul Traffic. Weather: Jena Weather. BizITObs: BizITObs-L2C.
Method
Metric
Elletricity
Solar
Power
ETTI
ETT2
Traffic
MDENSE
Walmart
Weather
BizITObs
Avg (all)
Avg (non-leak)
CRPS
0.070
0,512
0.085
0,515
0.205
0,257
0.294
0,151
0.068
0,262
1.000
1.000
Seasonal Naive
MASE
1,203
1.778
1.137
1,236
0.986
CRPS
0.048
0.420
0,046
1.056
0.130
0.110
0.091
0,077
0.054
0.124
0.611
0.604
TIDE
MASE
0.706
1,265
0.904
6,898
2.189
0,618
0.911
0.814
0.832
0,450
0.931
0.934
CRPS
0.052
0,518
0.054
0,304
0.131
0.112
0.070
0.082
0.059
0.074
0.549
0.490
PatchTST
MASE
0.753
1,607
1.680
2.168
0.653
0.732
0.266
0.753
CRPS
0.057
0.056
0.344
0.129
0.105
0.053
0.077
0.540
0.483
TTranfoommer
MASE
0.875
1.342
1.076
2.393
1.841
0,581
0.727
0,761
0.623
0.271
0.767
0.708
CRPS
0.083
0.535
0.072
0,344
0.188
0.237
0.108
0,137
0.079
0.095
0.780
0.714
DLinnar
MASE
0.984
1.257
1.325
1.606
3.194
1.016
0.914
1.115
0.925
0,282
0.938
0.906
CRPS
0.075
0.417
0.122
0,210
0.102
0.758
0.697
TIM
MASE
0.802
1.255*
0.898
1.934
1.547
0.901
1.195
1.477
0,506
0.308
0.831
0.798
CRPS
0.084
0.573
0.066
0,345
0.135
0.182
0.152
0,151
0.092
0.120
0.797
0.726
Timer
MASE
0.967
1,344
1.006
1.697
1.754
0.770
1.196
1,219
0.655
0.376
0.871
0.820
CRPS
0.045
0,280
0.080
0.439
TimesFM
MASE
0.655*
1.391
0.851
1.700
1.644
0.678
0,702
0,735
0.440
0.310
0.689
0.640
Chronoss
CRPS
0.043*
0.389
0.038
0,360
0.097
0.124
0.087
0.079
0.089
0.087
0.543
0.513
MASE
0.629
1.193*
0.717
1.799
1.431
0.622
0.834
0,849
0.606
0.301
0.694
0.661
Chronoss
CRPS
0
0.341
0.387
0.109
0.084
0.471
MASE
0.617
1.002*
0,722
1.898
1.265
0.553
0,712
0.849
0,583
0.301
0.656
0.631
Chronos,
CRPS
0.041
0.339
0.038
0.404
0.091
0.117
0.075
0.073
0.062
0,084
0.500
0.473
MASE
0.615
0.987
0.702
1.959
1,270
0.597
0.724
0,788
0.601
0.310
0.660
0.638
CRPS
0.072
0.048
0.275
0.101
0.173
0.084
0,103
0.081
0.578
0.507
MOIRAIS
MASE
0.981
1,465
0,948
1.701
1.417
0.990
0,836
1.048
0,521
0.301
0.798
0.726
CRPS
0.055
0.419
0.040
0,301
0.095
0.116
0.104
0,093
0.041
0.078
0.520
0.467
MOIRAIS
MASE
0.792
1,292
0.888
1.736
1.314
0.644
1.101
0,964
0.487
0.291
0.736
0.685
CRPS
0,050
0,286
0.094
0.112
0.095
0.051
0.079
0.467
MOIRALL
MASE
0.751
1.237
0.870
1.750
1.436
0.631
0.987
1.007
0,515
0.285
0.729
0.685
CRPS
0.051
0.230
0.044
0,392
0.125
0.152
0.099
0,100
0.070
0.112
0.583
0.586
Time-MoFs
MASE
0.587
0.535*
0.800
1.823
1.672
0.672
0.846
0.833
0.558
0.343
0.662
0.695
CRPS
0
0.386
0.172
0.058
0.111
0.576
Time-MoB_
MASE
0.581
0.689
0,790
1.773
1.878
0.762
0.759
0.817
0.524
0.337
0.678
0.695
MOIRAI-MOE,
CRPS
0.046
0.429
0.036
0,288
0.093
0.108
0.071
0.090
0.056
0.081
0.497
0.450
MASE
0.719
1,222
0.737
1.750
1.248
0.563
0.746
0.927
0.476
0.298
0.670
0.620
CRPS
0.296
0.100
0.079
0.439
MOIRAI-MOE,
MASE
0.638
1.161
0.725
1.748
1.247
0.510
0.721
0,918
0.509
0.290
0.651
0.611
13

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table Full zero-shot performance comparisons measured by MSE and MAE. Note that the MSE and MAE values are relatively large compared to those reported in previous studies like PatchTST, primarily because we compute the loss using raw time series values rather than normalized ones. This approach can more accurately reflect the forecasting accuracy gap. Best results are red.
Second best results are in blue. Power: Turkey Power. Traffic: Istanbul Traffic. Weather: Jena Weather. BizITObs: BizITObs-L2C.
Metric
Eloctriclty
Solar
Power
ETT2
Traffic
MDDNNK
Walmart
Weather
BizITObs
Avg (all)
Aug (Bon-Jeak))
Scusonal Naive
MAE
MSE
1299429.16 16
166 20
1293.24
15:77
492.60
1398196.83
67996.63
122878.95
211.56
203.32
8.72
39929.67
118:38
32876006.66
2637.45
2197.23
174.31
1.000
1000
1,000
1.000
iTranfformer
MSF
1264494 38
1183,57
968959 56
55330 57
178757.02 02
41.77
9005 39
10022819.00
1885.01
20.55
0.508
04455
MAE
165.89
17.61
399,09
17083
278.31
1.85
51.06
1560.68
10.65
2,66
0.741
0.678
Mol F-DI inear
MAE
MSE
1901617.97
197.06
1098.56
16.47
1071490. /6
420.57
39026.35 37
130.99
195287.19 19
328.28
153.71
8.48
13016.78
02.43
26832019.08
2395.50
1619,90
12.81
21.57
2.75
0.857
0.656
0.508
0.575
MSE
2432897.66
66
884.33
647289.67
56256.46
116203.30
114.79
18425.62
39297380.00
1122.55
0.625
0.538
TTM
MAR
179.56
16.46
341.96
158.85
213.61
7.53
%:44
3360.79
8.88
2.97
0.833
0.784
MSE
2205084.30 30
962.26
6876600.25
39235 36
129063.67
75.23
19675.60
28410540.00
1873.8
27.21
0.613
0:527
MAE
200.62
17.57
$70.53
131.31
235.27
6.42
87.72
2646.92
13,65
3.50
0.865
0.804
TimesFM
MSE
1378878 95*
1061.70
384815.80
427789 0Z
169714.41
106.01
10194.73
9494507 86
1317,09
23.23
0.475
0.401
137.57
18.07
277.91
138.12
215.61
49.78
1484.68
0.672
0.612
Chronoss
MSE
1251170.49*
1405, 10
418195.72
60157.02
112472.02
100.62
15377.29
11697271 18
3945,04
23,89
0.587
0.511
MAE
126.25
15.79
275.11
161.23
207.11
5.28
59.26
1693.33
16.90
2.94
0.724
0.691
Chronus
MSE
1147348.35 35
1062.73
400709.337
66320.26
107178.21
80.48
12770.66
15813384,J4 14
1720.53
22.78
0.501
0.439
MAF
121. 600
13.18
265.79
109.60
194.70
4.69
51.58
170K.11
10.28
2.82
0.656
0,628
Chronos
MSE
1073679. 399
1017.98*
362366.3
73974
106362.90
90
13625.07
12389319.84
1874.83
23.61
0.503
0.447
MAE
121.06*
12,86*
277.66 64
177.68
191.07
5.07
53.61
1560.11
11.30
2.89
0.664
0.639
MOIRAIS
MSE
4015423.50
1429,82
757613.06
39481
118636.33
146.24
11041,41
19886286.00
1932.16
0.647
0.498
MAE
219.02
19.19
358.01
133.82
209.68
8.71
58.25
2112.07
10.23
2.90
0.802
0.715
MOIRAIS
MAE
MSE
1734656.25
164.94
1105.95
16.97
477193447
293.74
51793.61
149.15
113071.23
202.89
11.60
17724.71
79.41
1898 1036.00
2046.22
1196.21
7.73
22:44
0.713
0.500
0.650
0.114
MOIRAlt
MAR
MSE
1229872.00
150.66
997.13
16.25
340307.44
262.70
44752.48 48
142.21
106513.38
204.72
101.17
14874.89
66.73
21274060.00
2110.33
1914,39
10.10
21.79
2.77
0.511
0.720
0.669
0.449
MoEa
MSF
1158323 38*
176.27
315704.91
50267 22
114374.42
89.87
11303,31
19934856
1371.87
28.51
0.395
0.408
MAE
120.52
7.07*
254.28
149.31
218.55
5,70
57.43
1742.96
11.35
3.26
0.644
0.663
Time-MoP,
MSE
2200643.75 75
194,84*
350989.61 67
47389 70
121112.59
99.13
9585.73
12876789 32
1264.26
27.34
0.394
0,400
MAE
120.53*
9.06*
262.48
147.11
229.67
6.45
$2.10
1687.00
9.32
3.24
0.650
0,652
MOIRAI-MOFs
MSE
9301-40.33
1113.50
360995.59
45412.81
114669.09
53.05
9426.45
18025986.00
1964.27
0.395
MAE
138.03
16.05
260 82
141.08
194.63
4.78
50.09
1955.77
10.08
2.89
0.668
0.617
MOIRAI-MOEa
MSE
907276.31
1047.63
311227.06
48457.21 21
107284.42
45.83
9740.51
170947664.00
1954.24
22.54
0.434
0.378
MAE
122.27
15.24
251.10
145.50
191.47
4.33
49.73
1919.31
10.31
2.80
0.646
0.605
Table 10. Full zero-shot performance comparisons measured by PICP and QICE. The absence of certain baselines reflects their inability
compute PICP and QICE, as they only support point forecasting. The Avg column is computed by taking the geometric mean across all
datasets. Best results are in red. Power: Turkey Power. Traffic: Istanbul Traffic, Weather: Jena Weather. BizITObs: BizITObs-
Method
Metric
Electricity
Solar
Power
ETTI
ETT2
Trafflc
MDENSE
Walmart
Weather
BizITObs
Avg (all)
PICP
8.653
55.871
21.356
39.286
5.635
21.720
13.000
17.612
43.276
21.042
20.021
Chronoss
QICE
1.694
6.192
3.144
4.885
1.129
3.857
2.189
2.341
5.331
3,480
3.037
PICP
8,201
56.031
23.075
46.111
4.841
18.280
11.333
17.494
44,140
18.021
19,205
Chronosy
QICE
1.785
6.205
3.490
5.697
18869
3.545
2,305
2.298
5.335
2.887
3.218
PICP
7.645
56.379
20.827
43.095
10.079
14.312
8.889
16.530
43.110
17.485
19.015
Chronos,
QICE
1.661
6.243
3,373
5.467
1.746
3.868
1.901
2.216
5.253
2.708
3.082
PICP
1.184
3.436
1.283
1.508
1.984
3.413
0.259
1.908
3.239
3.452
L.666
MOIRAIS
QICE
0.915
2.152
2.786
2.698
2.610
2.769
2.342
1.508
1.433
0.971
1.871
PICP
1.132
3.549
0.470
1.508
3.730
2.884
0.148
1.416
0.267
2.098
1.137
MOIRAIB
QICE
0.422
1.469
1.372
2.575
2.698
3.504
3.140
1.100
1.196
0.997
1.560
PICP
0.032
2.545
0.489
1,984
3.254
1.296
1.778
1.385
0,588
5.223
1.098
MOIRAh.
QICE
0.343
3.506
1,357
2.504
1.834
3.063
1.140
0.827
1.271
2.396
1.520
PICP
0.717
4,150
1.515
0.873
3.254
5.582
0.556
2.957
1.452
1.042
1.678
MOIRAI-MOES
QICE
0.899
1.756
0.979
2.222
2.769
1.952
0.819
1,082
2.157
0,931
1.419
PICP
0.464
0.473
0.172
0.397
4.206
9.550
0.259
3.118
1.528
2.173
1.049
MOIRAI-MOFg
QICE
0.712
1.174
1.703
2.310
2.628
2.734
3.041
1.158
1.601
0.908
1.615
14

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
A.3. Summary of Methods
The following a brief introduction the models used in the evaluation process.
TiDE (Das et al., 2023) encodes the historical data of a time series along with covariates using dense multi-layer perceptrons (MIPss). It then decodes the time series while incorporating future covariates, also utilizing dense MLPs for this process.
PatchTST (Nie et al., 2023) employs Transformer encoders combined with patching and channel independence techniques enhance the performance of time series forecasting.
iTransformer (Liu et al., 2024b) treats independent time series as tokens to effectively capture multivariate correlations through self-attention.
MoLE-DLinear (Ni et al., 2024) trains multiple linear-centric models (i.e., experts) and a router model that weighs and mixes their outputs. In this study, we use the DLinear model as the experts.
LLMTime (Gruver et al., 2023) is a method for time series forecasting that leverages Large Language Models by encoding numerical data as text and generating possible future values through text completions.
Moment (Goswami et al., 2024) refers to a family of open time series foundation models that canhandle different time series analysis tasks. Note that Moment requires fine-tuning for forecast tasks and cannot directly do zero-shot forecast- ing, as stated in the GitHub issue: https: / /g thub. com mttm/mmmenttteee mese eiee-eff foundat oon-mod! moment/issues// http:/2eiissseeeee #issuecomment wowooomeeeentn1nneeeeee 21384788227 Therefore, exclude it in our model comparisons.
TTM (Ekambaram et al., 2024) foundation model based on the light-weight TSMixer architecture, incorporating innovations like adaptive patching, diverse resolution sampling, and resolution prefix tuning.
Timer (Liu et al., 2024c) a decoder-only foundation model, presenting notable few-shot generalization, scalability, and task generality.
TimesFM (Das et al., 2024) is decoder-only time series foundation model that pretrained on a large corpus of time series data, including both real-world and synthetic datasets.
Chronos (Ansari et al., 2024) is an encoder-decoder time series foundation model that uses quantization to convert real numbers into discrete tokens.
MOTRAT (Woo et al., 2024) is a time series foundation model trained on the LOTSA dataset, which contains over 27 billion observations across nine diverse domains.
Time-MoE (Shi et al., 2024) is a concurrent work that applies mixture of experts techniques to time series foundation models.
MOIRAI-MOE is proposed in this study, which is capable of achieving automatic token-level specialization.
A.4. Settings of Methods
Context Length Setting for All Methods In Table 11, we detail the context lengths used for each method in this study, and in their original paper. For full-shot deep learning models, we believe our searching range generally covers the lengths set in their original paper. For foundation models, the choice of input lengths depends on their pretraining strategies. For instance, in the case of TimesFM and Chronos, the input lengths are consistently set to 512 during pretraining. In contrast, for MOIRAI and MOIRAI-MOE, the pretraining algorithm involves randomly sampling a context length in the range [2, 8192]. Thus, searching for the input length on validation set during inference needed.
15

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table 11, Comparison of methods' context lengths: this study versus original papers.
Model
In-Dist. Evaluation (29 datasets)
Zero-Shu Evaluation (10 dalasets)
Original Paper
TIDE
Searching within prediction longths * [2.20]
720
PatchTST
Searching within prediction longths [2.20]
336
iTransformer
Searching within prediction lengths [2,20]
TTM
512
512
Timer
672
672
Time Time-MoE
4,096
(512, 1024, 2048, 3072)
TimesPM
512
512
Chronos
512
512
512
MOIRAl
1000
Searching within range {1000, 2000,
3000, 4000, 5000
Searching within range [1000, 2000. 4000, 5000)
MOIRAI-MOB
1000
Searching within range (1000, 2000, 3000, 4000, 5000)
Searching within range (1000, 2000, 3000, 4000, 5000)
To investigate the potential unfair comparisons arising from relatively shot context lengths of full-shot methods compared to foundation models, additional experiments are conducted: training the state-of-tte --tthee-rttttt specialized models PatchTST and iTransffrmerss using the same context length searching strategy as in MOIRAT and MOTRAI-MOE:
Long Context Lengths for Full-Shot methods
Table 12 presents the results using MSE and MAE. Note that, for clarity, we use Range-1 I to denote the searching range within prediction lengths * [2, 20], which is approximately around the values of [50, 500J. This is the range we used in our paper for specialized models. In addition, we use Range-2 to denote MOIRAI-MOF's strategy: searching within 1000, 2000, 3000, 4000, 5000, According to the results, we observe that from Range-1 to Range-2, the input length searching range increases substantially. However, the average performance of both PatchTST and iTransformer declines significantly across all four metries. For instance, the average MASE of PatchTST increases from 0.808 0.967, and the average MASE of iTrrnnformer rises from 0.767 to 1.013. These results suggest that neither PStchTST nor i'Transfrmmer benefits from very long input lengths; in fact, such lengths negatively impact their performance. And the performance gap between these models and the foundation models becomes even more pronounced.
Table 12, Performance comparison for different context lengths measured by MSE and MAE. Best results are in bold. Power: Turkey Power. Traffic: Istanbul Traffic. Weather: Jena Weather. BizlTObs: BizITObs-L2C.
Method
Context Length
Metrie
Electricity
Solar
Power
ETTI
ETT2
Traffic
MIDNSSE
Walmart
Weather
BirTTOss
Avg (all))
Scnsonal Naive
MSE
1299429.11 16
1293.24
1798196.83
57976.63
122878.95
203.32
39929.67
32876026.66
2197 299722
174.31
L.000
166.20
15.77
192.60
154.98
211.56
118.38
2637.43
10.96
PatcTTST
Range-1
MSE
1534813:00
1125.03
1605878.50
45755.20
167280.13
38.69
9240.59
13749435. 00
1361
18.97
0.511
Range-1
MAE
171.95
17.03
478.86
141,03
291.50
4.51
49.45
1847.65
9.77
2.60
0,740
PaclTTST
Range-2
MSE
1433466.75
973.65
957680.50
249211 448
284311.13
45.14
38315.62
15738720.00
2020.57
15.18
0.717
Range-2
MAE
150.33
15.09
414.32
289.93
346.99
4.96
111.85
1859.13
15.96
2.36
0,880
iTransformer
Range-1
1264494.38
1183.57
968959.56
55320.57
178757,02
9905.39
10922819.00
1885.01
0.508
Rango-1
MAE
165.89
17.61
399.00
170.83
279.21
4.65
$1.06
1560.68
10.65
2.66
0.741
TTranfoormer
Ringe-2
MSE
1856541.38
2587 88
572127.56
54057.63
297574.38
39.58
19981.01
21888656.00
1375.14 14
50,39
0.689
Range-2
MAE
174.00
23,96
350.17
167.26
408.40
4.74
89.69
2501.30
10.11
5.53
0.930
Kange-2
MSE
907276.31
1047 63
311227.06
48487.21
107284..42
45.83
9740.51
170947644000
1984
22.54
0.434
MoIRAI-MoFa
Range-2
122.27
15:24
251.10
145.50
191.47
49.73
1919.31
10.31
Hyperparameter Search for Full-Shot Methods For the three full-shot models used in zero-shot forecasting part, i.e.. TiDE (Das et al., 2023), PatchTST (Nie et al., 2023), and iTransformer (Liu et al., 2024b), we conduct hyperparameter search based on the values specified in Table 13. In addition, we explore the learning rate in the range [1e-6, 1e-3) on a log scale, and set the context length as = m * h, where m is tuned in the range [2, 20], and h is the prediction length. We implement a random search across these parameters over 15 runs and report results based on the best validation CRPS.
Pretraining Details of MOIRAI-MOE All MOIRAI-MOE models are trained on 16 A100 (40G) GPUs using batch size of 1,024 and 6 precision. The small and base model are trained for 50,000 and 250,000 steps LOTSA (Woo et al., 2024), respectively. The patch size P is set to 16 and the masking ratio r for next-token prediction pretraining is 0.3. The corresponding experiments are in Appendix B.3 and B.4. For optimization, we utilize the AdamW optimizer with Ir Ir= 1e-3, weight decay= = 1e-1, 81 = 0.9, 32 = 0.98. We also apply a learning rate scheduler with linear warmup for the first 10,000 steps, followed by cosine annealing.
16

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Table 13. Hyperparameter search values for TiDE, PatchTST, and iTransformer.
Hyperparameter
Values
TDE
hidden dim
{64. 128, 256)
num.encoder.ao eayoeer.layers
[2. 6]
num_decoder.due
[2, 6]
PatcTTST
d_model cam__eender_aaee Jayers
{64, 128, 256)
[2,6]
iTransformer
d_model
[128, 256, 512)
num encoder layers
[2, 4]
B. Additional Results
B.1. Comparison of MOTRAL and MOTRAI-MOE Pretraining Steps
In Figure 7, we present a comparison between MOTRAIs and MOIRAI-MOES in terms of pretraining steps. The results demonstrate that MOIRAI-MOEs outperforms MOIRAIS from the very first evaluation point 25k steps. Furthermore, MOIRAI-MOES at 25k steps achieves better performance than MOIRAIs at 125k steps. This figure highlights the clear advantages of MOIRAI-MOE terms of both model performance and reduced pretraining steps.
Moiral
Aggregated MAE
0.79
Moirai-MoE
0.75
0.71
0.67
0.63
25k
50k
75k
100k
125k
Training Steps
Figure 7. Performance comparison between MOIRAIs and MOIRAI-MOFs across pretraining steps.
B.2. Effects of the Number of MoE Layers
According to recent survey on MoE- based Transformers (Cai et al., 2024), replacing all FFNs with MoE layers is common practice in recent and well-known LLMs, such as Mixtral 8x7B and Qwen .5-MoF. Another popular choice is to replace half the FFN layers with MoE layers. To investigate the efficacy this, we conduct experiments where one out of every two FFN layers was replaced with MoE layers. As shown in Table 14, compared to replacing all FFN layers, this "half" setting reduces pretraining time by 31%. While the downstream performance Monash shows that replacing half of the FFN layers results in 5% performance drop. This demonstrates trade-off between performance and pretraining cost.
Table 14, Effects of the number of MoF layers in MOIRAI-MOF.
Variant
MoE Layers
Total Layers
Activated Params
Total Params
Monash Performance
Pretraining Time
Holl
3
11M 11M
64M 117M
0.68
6.53h
MOIRAI-MOES
9.49h
B.3. Effects of Patch Size
In contrast MOIRAL, which designs multiple input/output projection layers, each associated with specific patch size, MOIRAT-MON utilizes single projection layer with single patch size. In this part, we conduct experiments to examine the impact of different patch size choices. The evaluation results on the Monash benchmark are presented in Figure 8 (left), where the patch size of 16 yields the best performance. Regarding the choice of patch size in relation to performance, the patch size determines the time period encompassed within each token. If the patch size is too large (i.c., 64), the linear projection layer may lack the capacity capture the underlying patterns. Conversely, if the patch size is too small (i.e., 4),
17

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
0.74
0.73
Moirai-MoE-Sman
Moirai-MoE Small
Aggregated MAE
Aggregated MAE
0.72
0.71
0.70
0.69
0.68
0.67
0.66
0.65
0.1
0.2
0.3
0.4
0.5
Patch Size
Masking Ratio
Figure 8. Effects of patch size and masking ratio using MOIRAI-MOES.
the time series token may not contain sufficient semantic information, as highlighted in DLinear. Additionally, patch size affects inference speed; with fixed context window, smaller patch sizes generate more time series tokens, increasing GPU memory usage and ultimately slowing down inference. For instance, using patch size of 4 can take over day to complete all evaluations. Our choice of a patch size of 16 not only delivers strong performance but also maintains reasonable inference speed.
Furthermore, to justify the choice single projection layer over multiple projection layers, we conducted experiments using multiple input/output projection layers within MOIRAI-MOE. The results are presented Table 15. The notation used refers to MOIRAI-MOES with a multi-patch strategy. MOIRAI-MOES-S4 represents MOIRAI-MOÉss with single patch size of 4. The same naming applies other configurations, The results show that S4 and S64 do not perform well S16, however, they significantly outperform MOIRAIs and achieve comparable results to AOOR-A--OES--D These results indicate that the multi-patch strategy does not hold positive significance, since it is comparable to the worst cases of single patch size.
as follows: MOIRAI-MOOs-
Table 15. Effects of single projection layer and multiple projection layers in MOIRAI-MOE.
MOIRAIS
MOIRAI-MOFF--o Mulii
MoIRAI-MoPg-S4
MOIRAL-MOFg-S8
MOIRAI-MoFg-S16
MOIRAI-MOFg-S22
MOIRAt-MOFG-S6i
Monash Performance
0.78
0.72
0.72
0.67
0.65
0.70
0.72
B.4. Effects of Masking Ratio
In this study, we introduce the masking ratio r as a hyperparameter that determines the portion of the entire sequence used solely for robust normalizer calculation, helping to mitigate distribution shift issues. We conduct experiments to assess the effects of different masking ratios, with the evaluation results on the Monash benchmark shown Figure 8 (right). masking ratio of 0.3 delivers the best performance. ratio of 0.1 uses too little data to compute robust normalizer, potentially failing to accurately represent the overall sequence statistics. Conversely, ratio of 0.5 masks half of the data, which may hinder the parallel learning efficiency in next-token prediction pretraining. Therefore, it is crucial to select an appropriate data range that is small enough to avoid excessive masking, yet sufficiently representative for robust normalizer computation.
B.5. Expert Distributions of Different Gating Function
In this part, we present an in-depth comparison of the different gating functions explored in this study. First, we provide additional details on the implementation of the proposed token clustering method. The core idea of this approach is to leverage cluster centroids derived from the token representations of a pretrained model to guide expert allocations. Specifically, we perform inference on our pretraining corpus, LOTSA, using data amount corresponding to 100 epochs. During this process, we extract the self-attention output representations from pretrained MOIRAI model and apply mini- batch k-means clustering continuously update the clusters. The number of clusters is set match the total number of experts. During the pretraining of the MoE model, each token computes the Euclidean distance to each cluster centroid, and these distances are used as token-to-exper affinity scores for expert assignments. Empirical evaluations have demonstrated the effectiveness of this approach compared to randomly learned gating from scratch, indicating that the clustering method better aligns with the inherent distribution of time series representations.
Using the three gating functions explored in this study, i.e., linear projection, linear projection with load balancing, and
18

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
MoE Layer
MoE Layer 4
MoE Layer
0.2
Token (%%
0.0
0.0
0.0
10
30
10
20
30
10
30
Token (%)
0.06
Linear mi Dalance
0.10
0.02
0.05
0.00
10
30
10
20
30
0
10
20
30
0.3
Token (%)
Clusters
0.4
0.2-
0.1-
0.2
0.2
0.0
0.0
0.0
20
30
Expert Index
Expert Index
Expert Index
Figure 9. Visualization of the distribution of expert allocation for MOIRAI-MOES layers 2, 4, and (the last layer) using all data from the Monash benchmark.
token clustering, we present their expert allocation distributions aggregated across all datasets in the Monash benchmark, as illustrated in Figure In terms of selection diversity, we observe the following relationships: Token Clusters (least diverse) Pure Linear Projection (neutral) < Linear Projection with Load Balancing (most diverse). According to their performance results shown in Figure 4, we can establish the following ranking: Token Clusters > Linear Projection with Load Balancing > Pure Linear Projection. Based on all these observations, we offer the following explanation:
In the token clusters approach, the expert selections less diverse because the routing is grounded in pretrained knowledge. The clustering step creates centroids that represent well-structured patterss in the data, and then tokens routed to specific experts that are particularly suited to handle the type of data represented by their corresponding cluster. While this targeted routing reduces diversity, it enhances performance due to the selection of experts based on more meaningful criteria.
The addition of load balancing loss increases the diversity of expert selection by spreading the workload and encouraging the of all experts more evenly. This diversity prevents over-reliance specific experts, potentially improving generalization and performance compared to pure linear projection. However, this approach might be less targeted than clustering, since it still depends on a learned gating function rather than pretrained centroids.
In the pure linear projection method, the gating function is entirely learned from scratch. Without any additional constraints (like load balancing), certain experts might get selected more often than others, leading to neutral level of diversity. Since there is no mechanism to encourage exploration (like load balancing) or specialized routing (like clustering), performance remains lower than the other methods.
B.6. Visualization of Time Series Observations and Expert Allocations
Following the discussion in the main paper, this section investigates the relationship between raw time series observations and their corresponding expert allocations. In Figure 10, the upper subfigure presents Traffic Hourly time series sequence with length of 512. For enhanced visualization, the sequence is segmented using vertical dashed lines, each spanning 16 steps, which is equal to the length of single time series token. The lower subfigure illustrates the expert allocations shallow layers for 32 tokens derived from the 512 observations. The yellow straight line represents the specific experts selected by the token at each position. The alignment of subfigures facilitates an intuitive comparison between the time series trends and the associated expert selections.
The figure includes red square boxes to highlight time series segments exhibiting a downward trend followed by a slight upward pattern. These segments consistently correspond to the activation of two specific experts, as shown in the lower
19

---

Moirai-MoE:: Empowering Time Series Foundation Models with Sparse Mixture of Experts
subfigure. This observation suggests that Moirai-MoF effectively captures time-based structures and demonstrates model specialization at the token level.
Expert Index Observations
13
8
Figure Joint visualization raw time series observations and their corresponding expert selection at shallow layers MOIRAI MOIRAI-MOEs. The upper subfigure depicts the raw time series observations with the x-axis representing time step indices (0 to 511). The lower subfigure shows the expert allocation distributions, where x-axis corresponds to series token indices (0 to 31), and the y-axis represents the indices of the 32 experts in the layer.
C. Limitation
The limitation of this study lies in the efficiency of autoregressive predictions during inference, well-documented challenge for decoder-only architectures. However, inference solutions developed for large language models (LLMs) could help address this issue. For instance, many LLMs leverage quantization techniques (e.g., 8-bit or 4-bit weights) to significantly reduce computational costs while maintaining performance. In future work, we plan to explore model quantization and pruning methods to optimize efficiency. Additionally, we aim to implement key-value (KV) caching techniques to accelerate inference. However, a key challenge lies in our use of instance normalization, which requires recalculating normalization statisties whenever new token generated. This necessity could render the cached hidden states invalid, presenting an obstacle to efficient caching.
D. Visualization
In this section, we visualize the datasets used in the model analyses (NN5 Daily (Figure 11), Traffic Hourly (Figure 12), and Covid Daily Deaths (Figure 13)) to facilitate understanding of the patterns within the time series data.
20

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
NN5 Daily
larget pred
1991-69-01
1997-10-01
1997-42-01
1997-12-11
1038-01-01
1938-02-00 01
1018-19-011
1938-64-11
1098-45-01
target
pred
40
30
1987-09-01 91
1997-30-01
1597-31-00 40
1997-12-81
1998-01-01 01
1998-02-01
1998-03.00 01
1998-04-01 01
1998-05-00 01
target
2007 0997-09-01
1097-10-11
1997-11-01
1997-12-01
1908 01
1998-02-01 01-01
1998 -0 D1
1998-65-41
1918
target
60
www.
40
20
0
1007-09.01
1097-10-01
1997.10 011-311-11
1997-12-11
1098--5-01
1008.00 01
1098 03
1998
Figure 11. Visualization of NN5 Daily data, including both context length and forecast results.
21

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Traffic Hourly
0.4
target pred
2016-12:1
12-8-12-13
2016-12-17
2018-12-21
2016-12-25
2016:12:29
2017 2017-003-01
target
0.14
pred
0.12
0.08
0.04
0.00
2016-12:13
2015-12:17
2016-12.21
2016-12-25
2016-12:29
2017-01-33
target
pred
0.10
2016-32-13
2016-12-17
2016-12:21
2016-12:23
2016-12:29
2017-01-01
target
0.30
predi
0.25
0.10
0.05
2016-12-13
2016-12-17
2016-12-11
2016-12-25
2016-12.29
2017-01-01
Figure 12, Visualization Traffic Hourly data, including both context length and forecast results.
22

---

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
Covid Daily Deaths
target
50
30
10
2020-02-01
2020-03-01
2020-04-01
28260-5-0
2020-06-00 01
2026-07-00
2020-08-00 01
target
pred
1-
2020-02-11
2020-03-01
2020-04-01
2026-5-01
2020-86-01
2029-07-40 .1
2020-08-01
target
4000
2000
20020022-1
2820-03-1
2020 06--0
2028-05011
2020 0200066-22
2820-07-4
2020 0020009.000 01
target
100
pred
50
40
0400
0120-5-11
2020-06-01
2020-07.o1
2020-08-11
Figure 13, Visualization of Covid Daily Deaths, including both context length and forecast results.
23


Reviews:
[{"review_id": "Snecj6TiZn", "reviewer": "Reviewer_h3gH", "summary": "The paper focuses on the pretraining of time series foundation models using large time series corpora. \nThe paper argues that there are significant drawbacks to current approaches that address heterogeneity by grouping time-series based on human-identified features such a frequency. The paper proposes an alternative that involves using a sparse mixture of experts (MoE) that provides an avenue towards automatic specialization. The paper reports results for multiple datasets, demonstrating outperformance of the selected baselines. Experimental analysis is included that explores the operation of the MoE foundation models.\n\n--AFTER REBUTTAL\nI thank the authors for their thoughtful response. I was somewhat surprised that there was no response to my follow-up questions about the routing mechanism and the very sparse utilization of experts. The authors addressed three of my initial concerns, providing good responses. In particular, they emphasized the value of the engineering effort in getting an MoE time-series foundation model to work and they reported variability for some experiments. The authors pointed out that I had misinterpreted the gating mechanism and that it was non-trainable. But understanding this, the follow-up question in my response to the rebuttal was how the non-trainable router differed from a clustering-based hash approach that has been proposed previously. There was no response to this. The authors were claiming this as a significant novel technical contribution of the paper, so the lack of justification is concerning. The other residual concern was how uneven the usage of the experts was. I had concerns about the design approach given that it emerges that only a handful of the experts are being used at higher layers in the architecture. Given that these concerns were not addressed, I will retain my score (weak reject).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[C1] Code repository are corrupted.**\n\nThe issue appears to be related to viewing the files directly through the web interface. We verify that downloading the repository locally resolves the issue.\n\n**[C2] The paper seems to be a relatively straightforward application of existing sparse MoE techniques to MOIRAI, with some careful engineering effort to ensure that the training is successful.**\n\nRegarding the argument \"the token construction process is more of an engineering exercise to make the operation and training efficient; with some careful engineering effort to ensure that the training is successful\". In fact, efficiency is a crucial aspect, particularly when developing Time Series Foundation Models (TSFMs). TSFMs typically have large parameter sizes and require training on large time series corpus, making efficiency critical. Notably, the paper you referenced (Shazeer et al., 2017) uses an entire section (Section 3) to discuss efficiency challenges in training MoE models, including considerations such as batch size, model parallelism, and network bandwidth. These discussions are engineering-focused—arguably even more so than our paper—but does that diminish their importance or exclude them from being considered technical contributions?\n\nRegarding the argument \"the paper seems to be a relatively straightforward application of existing sparse MoE techniques\". While it may sound easy to apply existing MoE techniques to TSFMs, making these techniques effective in practice is challenging. Our work represents a pioneering effort in successfully adapting sparse MoE to TSFMs. How do we define success in this context? An intuitive and primary criterion for success is the downstream model performance. For instance, the concurrent work Time-MoE, can hardly be considered successful, as its performance significantly trails behind Moirai-MoE. Notably, Time-MoE does not even surpass Moirai-Large, as demonstrated in both our zero-shot evaluations and the point forecasting results over 27 datasets reported by the Chronos Team (https://aws.amazon.com/blogs/machine-learning/fast-and-accurate-zero-shot-forecasting-with-chronos-bolt-and-autogluon/). In summary, while a straightforward application of existing MoE techniques to TSFMs may seem easy, achieving SOTA performance is technically non-trivial and depends critically on model design.\n\nFinally, as you acknowledged in the strengths part, we have put significant effort into the behavior of MoE models. This area of investigation has not been explored in the literature, including the Time-MoE work. Thus, this should be recognized as a meaningful contribution of our paper.\n\n**[C3] Cannot detect a significant difference between the proposed gating and the procedure in Shazeer et al, 2017. A more correct claim would be “we introduce a novel procedure for pre-training the gating function in a standard sparse mixture-of-experts model”.**\n\nYour suggested point is reasonable in certain respects; however, we would like to emphasize that the paper you mentioned (Shazeer et al., 2017) employs a linear projection as its gating, which relies on a randomly initialized, trainable weight matrix for expert assignment. In contrast, Moirai-MoE utilizes a gating function that is non-trainable and initialized based on clustering results from a pretrained model, not randomly. We believe this distinction marks a departure from the linear projection gating, effectively constituting a new gating function. We would appreciate further clarification regarding which aspects you consider identical between these two gating functions.\n\n**[C4] No confidence intervals and statistical tests.**\n\nStandard deviations and statistical significance are in Table 1: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view\n\n**[C5] Although the token clusters approach achieves the best performance, these results suggest that the training is not operating correctly.**\n\nThank you for your comment. We do not believe our training is incorrect. Our routing mechanism is based on clustering results derived from a pretrained Moirai model's representations. If certain clusters exhibit similarity, the selection of experts would be naturally constrained, resulting in some experts being underutilized. Thus, the question here pertains to the representation similarity inherent in pretrained TSFMs. Existing research has investigated representation similarities within TSFMs. A recent study (https://arxiv.org/pdf/2409.12915v2) uses Centered Kernel Alignment to measure representation similarity, highlighting clear redundancy in TSFMs such as Chronos and Moirai. Additionally, another study (https://arxiv.org/pdf/2302.11939v2), specifically Section 7, reports that within-layer token similarity tends to increase in deeper Transformer layers. This finding aligns closely with our own observations in Moirai-MoE, where fewer experts are activated due to increased token similarities, particularly in deeper layers.", "claims_and_evidence": "Please see \"Other Strengths and Weaknesses\"", "methods_and_evaluation_criteria": "Please see \"Other Strengths and Weaknesses\"", "theoretical_claims": "Please see \"Other Strengths and Weaknesses\"", "experimental_designs_or_analyses": "Please see \"Other Strengths and Weaknesses\"", "relation_to_broader_scientific_literature": "Please see \"Other Strengths and Weaknesses\"", "essential_references_not_discussed": "Please see \"Other Strengths and Weaknesses\"", "other_strengths_and_weaknesses": "Strengths\n\nS1. The paper proposes to augment a foundational time series model (MOIRAI) with a sparse Mixture-of-experts. The key innovation is a novel strategy to pre-train the experts, which involves clustering tokens according to the attention outputs of an inference model. There is also a carefully executed tokenization strategy. \n\nS2. The paper reports the results of extensive experiments. These demonstrate that the MoE approach achieves a significant performance improvement over the base MOIRAI models, even when there are far fewer active parameters.\n\nS3. The paper conducts a good experimental analysis to provide insights into how the MoE is behaving (e.g., how different frequencies are assigned to different experts at different layers of the model). There are useful visualizations and thorough investigations in the appendices. The paper includes ablation studies as well as studies of the sensitivity to different design choices and parameter settings. \n\nWeaknesses\n\nW1. The technical contributions of the paper are relatively minor. There is some adjustment of the token construction process, but this appears to be more of an engineering exercise to make the operation and training efficient. Beyond this, the primary novel technical contribution is the use of the token clusters during the pre-training of the MoE, with the clustering based on the attention outputs of an inference model. While this is an innovative solution, and a welcome acknowledgement of the challenge of successfully training a sparse MoE, the technical content is presented in 13 lines of text and a single equation. Overall, while there are some innovative contributions, the paper seems to be a relatively straightforward application of existing sparse MoE techniques to MOIRA, with some careful engineering effort to ensure that the training is successful. \n\nW2. The paper claims to introduce “a new expert gating function for accurate expert assignments and improved performance”. This claim does not seem to be supported, as I cannot detect a significant difference between the proposed gating and the procedure in Shazeer et al, 2017. There is novelty in the pretraining based on clusters but this does not imply a new gating function – the claim is misleading. A more correct claim would be “we introduce a novel procedure for pre-training the gating function in a standard sparse mixture-of-experts model”.\n\nW3. The paper does not report variability for any of the experiments and there are no confidence intervals. There are no statistical significance tests.", "other_comments_or_suggestions": "None", "questions_for_authors": "Q1.\tFigures 5 and 6 (and 9 in the appendix) show why expert allocation is important. This paper uses a standard Sparse-MoE setup with all experts having the same structure. If trained and regularised correctly, allocation should be approximately uniform (like Fig. 7 in Mixtral). While it is desirable to have some experts focused on specific frequencies, Figure 6 suggests that many experts are not contributing at all. This is also confirmed by Figure 9 in the Appendix. Although the token clusters approach achieves the best performance, these results suggest that the training is not operating correctly. It suggests some form of representation collapse. Was this investigated by the authors? \n\n\n[R1] Jiang, Albert Q., et al. \"Mixtral of experts.\" arXiv preprint arXiv:2401.04088 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "abzn1SlNtD", "reviewer": "Reviewer_Bkoz", "summary": "This paper introduces a novel foundational model for time series forecasting, building upon the architecture of Moirai. The primary motivation is to address a key limitation of existing time series foundational models, which rely on manually imposed clustering—such as specialized layers for different time series structures (e.g., frequency-based clustering in Moirai). Instead, the authors propose an automated approach using a mixture of experts (MoE) to dynamically detect dataset diversity and automatically cluster heterogeneous patterns within the pretraining data. Additionally, rather than using conventional MoE layers, which are sensitive to initialization, the authors introduce a clustering-based method that enhances robustness. Experimental results on multiple datasets demonstrate the effectiveness of the proposed approach compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**IMPORTANT: Figures 1, 2, 3 and Tables 1, 2 are provided here: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view**\n\n**[C1] Testing on synthetic datasets with clear clustering structures. Assess clustering uniqueness in homogeneous datasets or evaluate whether Moirai-MoE correctly identifies structures in datasets with predefined clusters.**\n\nWe generate synthetic datasets with basic time series patterns to assess the ability of Moirai-MoE in a controllable way. The resulted visualization is in Figure 1.\n\n**[C2] Include zero-shot visualizations, highlight both successes and failures, explaining where Moirai fails and why**\n\nWe conducted additional visualizations comparing Moirai-MoE-Small with Chronos-Small and TimesFM, as shown in Figures 2 and 3. Figure 2 illustrates failure cases for Moirai-MoE—these time series primarily exhibit trend without seasonality. In such cases, both Moirai-MoE-Small and Chronos-Small perform poorly, while TimesFM performs exceptionally well. However, we suspect possible data leakage in TimesFM, as its forecasts align almost perfectly with the future trend, which we believe could be inherently unpredictable. Figure 3 presents success cases where Moirai-MoE outperforms the others. Its forecasts are generally closer to the ground truth, whereas Chronos-Small and TimesFM tend to show larger overestimations or underestimations. Due to word count limitations at the rebuttal stage, we present only two cases here, but we will add more cases and analyses in the appendix of the manuscript.\n\n**[C3] Reporting standard deviations and statistical significance would better demonstrate the stability and reliability of the proposed approach**\n\nStandard deviations and statistical significance are in Tables 1 and 2.\n\n**[C4] Clearly explain how results were collected (e.g., best epochs for validation/test/train, last epoch for in-distribution experiments)**\n\nMoirai-MoE is pretrained for a certain number of epochs on a large time series corpus. After pretraining, the checkpoint saved from the last epoch is used to perform inference on 29 in-distribution datasets. Based on the aggregated performance, we know the optimal number of epochs required to achieve the best in-distribution performance and then we use this checkpoint for zero-shot evaluation.\n\n**[C5] Provide explanation of how \"out-of-distribution\" data is defined and ensure no data leakage occurs.**\n\nWe follow the settings of the well-recognized time series foundation models (TSFMs), such as Moirai, Chronos, and TimesFM, the out-of-distribution data refers to those datasets (including their training, validation, and test sets) that are not included in the pretraining corpus.\n\n**[C6] The contributions are somewhat moderate. The incorporation of MoE is not a significant contribution, as it has already been used in Time-MoE.**\n\nWe would like to argue our position from the following points. First, although Time-MoE has applied MoE to TSFMs, its application has not demonstrated superior effectiveness, as evidenced by its inferior zero-shot forecasting performance. In our zero-shot evaluations, its performance significantly trails behind Moirai-MoE. It does not even surpass Moirai-Large, as demonstrated in both our zero-shot evaluations and the point forecasting results over 27 datasets reported by the Chronos Team (https://aws.amazon.com/blogs/machine-learning/fast-and-accurate-zero-shot-forecasting-with-chronos-bolt-and-autogluon/). Hence, our contribution lies not merely in applying MoE to TSFMs but, importantly, in achieving notably superior performance compared to Time-MoE. Second, we have dedicated considerable effort to exploring and understanding the internal mechanisms of MoE models. This depth of analysis has not been previously addressed in the literature, including the Time-MoE study. Thus, this should be recognized as a meaningful contribution of our paper.\n\n**[C7] Have you considered InfoNCE or DeepCluster for this purpose? Could you comment about time complexity and memory complexity.**\n\nWe appreciate the suggestion regarding the integration of clustering-based losses, such as InfoNCE or DeepCluster, into MoE training. We also find your provided training and inference complexity analysis reasonable. While these methods could indeed offer a more automated approach to structuring representations, their effectiveness might not be guaranteed in practice. Imposing clustering losses during MoE training could introduce optimization challenges and potentially interfere with the primary learning objectives of the model. Our current approach adopts a two-stage process. This separation offers training stability, particularly in the early stages when representations may not be sufficiently structured for effective contrastive learning. Our method can also leverage the inductive bias from the pretrained Moirai model. Nevertheless, we consider your recommendation a valuable and promising direction for future exploration.", "claims_and_evidence": "Key claims of the authors with comments\n\n**Improved Token-Level Clustering**\nMoirai-MoE enables automatic and more efficient token-level clustering compared to Moirai, which relies on human-defined frequency-based clustering. Figure 5 illustrates how Moirai-MoE performs data-driven clustering. However, there is no empirical evidence showing that this clustering is interpretable by humans. The appendix should include more visualizations, highlighting cases where clustering does or does not align with human intuition. Additionally, testing on synthetic datasets with clear clustering structures could help formally validate this approach. For example, experiments could assess clustering uniqueness in homogeneous datasets or evaluate whether Moirai-MoE correctly identifies imposed structures in datasets with predefined clusters. Stronger empirical evidence is needed.\n\n**Improved Expert Gating in MoE**\nMoirai-MoE enhances MoE by introducing a clustering-based expert gating mechanism, addressing the sensitivity issue in traditional MoE architectures. Figure 4 presents an ablation study comparing this gating method to standard linear projection. While the experiments support this claim, further validation is needed. Running multiple trajectories with different initializations could help assess the stability of clustering beyond performance improvements. Additionally, reporting standard deviations would strengthen the claim regarding Moirai-MoE’s robustness.\n\n**State-of-the-Art (SOTA) Performance Across Benchmarks**\nThe authors conduct extensive benchmarking against multiple competitors and datasets. However, incorporating statistical tests, such as reporting standard deviations and statistical significance, would better demonstrate the stability and reliability of the proposed approach.", "methods_and_evaluation_criteria": "**Relevant Benchmark Datasets**\nThe authors primarily use the Monash dataset, which is increasingly recognized in the time series literature, making it a sensible choice for benchmarking.\n\n**Valid Methodological Approach**\nThe proposed methods are well-founded, as they address data heterogeneity—a key challenge in time series foundational models.", "theoretical_claims": "The manuscript does not provide a theoretical guarantee for the proposed method. To enhance its quality, the authors might conduct a theoretical analysis comparing clustering-based MoE to linear mapping. Even in a simplified setting—where each expert is linear—they could analyze how a student MoE approximates a teacher MoE under both architectures (for examples). This would provide deeper insights into the model’s theoretical foundations. But this is more a suggestion than a criticize.", "experimental_designs_or_analyses": "**Experimental Results & Reproducibility**\n\nThe experimental results demonstrate computational efficiency and good performance, but several improvements might improve the manuscript.\n\nA. Statistical Validation\n\n- As previously mentioned, incorporate standard deviations where necessary.\n- Perform statistical tests to confirm the stability of the approach.\n\nB. Reproducibility & Transparency\n\n- Clearly explain how results were collected (e.g., best epochs for validation/test/train, last epoch for in-distribution experiments).\n- Provide a detailed explanation of how \"out-of-distribution\" data is defined and ensure no data leakage occurs.\n- Include algorithmic details (Mainly in algorithm environment) in the manuscript to facilitate reproducibility.\nC. Visualization of Zero-Shot Forecasting\n\n- Include additional visualizations (at least in the appendix) showcasing zero-shot forecasting results even though some are given but not compared with competitors.\n- Specifically, highlight both successes and failures, explaining where Moirai fails and why, as well as where it succeeds (or Moirai-MoE) and the underlying reasons.", "relation_to_broader_scientific_literature": "**MoE in Time Series Foundational Models** – *Weak Contribution*\nThe incorporation of MoE is not a significant contribution, as it has already been used in Time-MoE. While the authors highlight some differences, the novelty remains limited.\n\n**New Expert Gating with Clustering** – *Moderate Contribution*\nThis contribution is moderately novel, as such expert gating has not been previously proposed. However, the lack of theoretical evidence and extensive experimental validation weakens the claim of its stability.", "essential_references_not_discussed": "No essential references to mention", "other_strengths_and_weaknesses": "**Strengths**\n\n- Addressing data heterogeneity is a valuable approach, and the use of MoE appears to be a promising solution.\n- The paper includes extensive experiments across multiple datasets, demonstrating thorough evaluation.\n- The writing is clear, making the paper easy to follow.\n\n**Weaknesses**\n\n- The contributions are somewhat moderate, as several prior works have already explored MoE for time series (e.g., Time-MoE).\n- There is a lack of strong theoretical justification and limited empirical evidence regarding the stability of the expert gating system.", "other_comments_or_suggestions": "All suggestions have been written before", "questions_for_authors": "**On the expert gating**\n\nInstead of using a direct clustering approach for expert gating, introducing a third loss (a clustering loss) in the embedding space could be beneficial. This would encourage natural grouping of similar patterns without requiring an explicit, manually determined clustering step. A contrastive loss (e.g., a soft clustering loss or DeepCluster-like approach) could be used to enforce structure in the learned representation. This avoids the need for a preliminary run of Moirai with a single layer, making the process more automated and adaptive. Have you considered loss functions like InfoNCE or DeepCluster for this purpose?\n\n**Computational Comparison with Moirai**\n\n- Training Complexity:\nMoirai already reduces the sample size by clustering, but if clustering is done implicitly through a loss function rather than manually, the computational overhead may shift from pre-processing to training.\nIf the gating mechanism remains data-dependent, backpropagation complexity might increase due to additional constraints in the loss function.\n- Inference Complexity:\nIf expert selection remains soft, inference might require evaluating multiple experts per sample, increasing computation time.\n\nCould you comment about time complexity and memory complexity of the algorithm and illustrate that empirically.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UNtSkkm2g", "reviewer": "Reviewer_JU2u", "summary": "This paper proposes MOIRAI-MOE, a time series foundation model based on the Mixture of Experts (MoE). It replaces the traditional frequency-grouping strategy with data-driven token-level specialization, thus addressing the pre-training challenges posed by the high heterogeneity of time series data. The model achieves significant performance improvements on 39 datasets by introducing an expert routing mechanism based on the clustering centers of pre-trained dense models and an autoregressive prediction objective.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[C1] While pruning is mentioned as future work, no concrete solution is provided.**\n\nThanks for the comment. A concrete pruning solution is to first evaluate expert usage by tracking gating activations during pretraining. Identify experts with significantly fewer activations (e.g., activated less than 1\\% of total gating decisions) as underutilized. Completely remove these expert modules from the model architecture, thereby reducing GPU memory usage. Lastly, update the gating network accordingly to ensure it routes inputs correctly to the remaining experts.\n\n**[C2] The proposed clustering-based routing relies heavily on pre-trained MOIRAI, which introduces a dependency on the quality of the pre-trained model and dataset (LOTSA), and thus potentially limiting generalization to domains outside the pretraining distribution.**\n\nRegarding the argument about \"potentially limiting generalization to domains outside the pretraining distribution\": This challenge is inherent to all time series foundation models, not solely Moirai-MoE. Therefore, it underscores the importance of assembling a comprehensive and diverse time series pretraining corpus to maximize coverage of potential application domains.\n\nRegarding the argument on the \"dependency on the quality of the pretrained model\": We acknowledge this dependency. Indeed, recent advancements in MoE-based Large Language Models (such as Qwen1.5-MoE) explicitly leverage pretrained dense models to introduce beneficial inductive biases, thereby enhancing model performance. Consequently, we consider this dependency a strategic strength rather than a limitation. Our work aligns with this research direction, with the assumption of a high-quality pretrained model and focusing on effectively leveraging it to achieve further performance gains.\n\n**[C3] MOIRAI-MOE contains significantly more parameters than those of MOIRAI. This increases memory demands during training and deployment. Is it possible to use model compression technology to reduce the parameters of the model?**\n\nThe increase in parameter count of Moirai-MoE does not proportionally affect computational cost during inference because, at any given time, only a subset of the experts are activated. However, model compression techniques can indeed be leveraged to reduce the parameters: (1) By converting weights from full precision (e.g., FP32) to lower precision (e.g., INT8 or FP16), quantization techniques can substantially reduce memory footprint while maintaining accuracy. (2) Techniques such as knowledge distillation can transfer learned representations from a larger Moirai-MoE model to a smaller, compressed model, thus achieving efficiency without a significant loss in performance.\n\nImplementing these compression techniques individually or in combination could mitigate memory concerns associated with Moirai-MoE. We will explore integrating these strategies into our framework and evaluate their impact on model performance and efficiency in subsequent research. Thank you for the suggestion.\n\n**[C4] When MOIRAI-MOE and MOIRAI parameters are equivalent, how does MOIRAI-MOE perform?**\n\nOur current configuration is as follows: Moirai-Base has a total of 91M parameters, while Moirai-MoE-Small has 117M parameters. To address your concern, we introduce a variant of Moirai-MoE-Small, called Moirai-MoE-Small-V2, by reducing the FFN dimension. This variant has a total parameter count of 89M, which is closely aligned with that of Moirai-Base.\n\nWe pretrain Moirai-MoE-Small-V2, evaluate it on the 29 datasets of the Monash benchmark, and compute the aggregated performance. Note that the aggregated MAE of Moirai-Base is 0.71 (according to Figure 3 in the main paper), while Moirai-MoE-Small-V2 achieves an aggregated MAE of 0.67, demonstrating superior performance.\n\n**[C5] MOIRAI-MOE still lags behind fully fine-tuned specialized models like TiDE on some datasets (e.g., ETT1 in Table 8), indicating room for improvement in extreme domain adaptation.**\n\nWe believe there might be some misunderstanding. We carefully compare the performance of TiDE and Moirai-MoE on the ETT1 dataset in Table 8. The CRPS and MASE of TiDE are 1.056 and 6.898, while the CRPS and MASE of Moirai-MoE-Small are 0.288 and 1.750. Since lower values are better, Moirai-MoE-Small is actually significantly better than TiDE on this dataset.\n\nHowever, to address your concern, we do find that on the Walmart dataset, TiDE performs better than Moirai-MoE-Small: TiDE scores 0.077 for CRPS and 0.814 for MASE, while Moirai-MoE-Small scores 0.090 for CRPS and 0.927 for MASE. This is a case where Moirai-MoE requires fine-tuning for better domain adaptation. After fine-tuning Moirai-MoE-Small, the results improve to 0.071 for CRPS and 0.756 for MASE, successfully surpassing TiDE.", "claims_and_evidence": "Claims are supported by in-distribution/zero-shot results in Tables 3, 6, and 8, which show MOIRAI-MOE’s superiority in MAE, CRPS, and other metrics. Ablation studies demonstrate that MoE specialization, not just the decoder objective, drives performance gains.", "methods_and_evaluation_criteria": "The sparse MoE efficiently scales model capacity while maintaining computational efficiency.  Cluster Gating improves load balancing and expert relevance compared to random initialization.\nThe evaluation uses diverse datasets (Monash, zero-shot benchmarks) and metrics (MASE, CRPS), ensuring robustness.", "theoretical_claims": "Theoretical derivations (e.g., MoE layer formulation, gating function) are standard and correctly applied. However, the theoretical grounding (e.g., convergence guarantees) of the cluster-based gating mechanism is not explicitly proven.", "experimental_designs_or_analyses": "This paper includes baseline comparisons of both foundation models (MOIRAI, Chronos) and specialized models (PatchTST). Meanwhile, ablation studies isolate the impact of MoE, gating, and pretraining objectives. Moreover, the efficiency analysis in Table 4 shows comparable inference speeds to MOIRAI despite larger total parameters.", "relation_to_broader_scientific_literature": "Builds on MOIRAI (Woo et al., 2024) by replacing frequency projections with MoE. Extends sparse MoE from language/vision (Fedus et al., 2022) to time series. Compares favorably with concurrent work Time-MoE (Shi et al., 2024) by introducing cluster gating and patch tokenization.", "essential_references_not_discussed": "No critical omissions noted. The paper comprehensively cites relevant works like TimesFM, Chronos, and existing MoE literature.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper proposed a new model architecture, which introduces a Sparse Mixture of Experts (MoE) to dynamically route tokens to specialized experts, overcoming the limitations of human-defined frequency grouping.\n2. The proposed method proposes a novel gating mechanism using cluster centroids from pre-trained dense models, outperforming randomly initialized linear routing.\n3. The proposed method achieves superior performance, which outperforms state-of-the-art baselines (e.g., MOIRAI, Chronos) on 39 datasets, reducing MAE by up to 17% with the same activated parameters and surpassing others with 65× fewer activated parameters.\n\nWeaknesses:\n1. Analysis in Fig. 6 reveals some experts in deeper layers are rarely activated, indicating inefficient parameter usage. While pruning is mentioned as future work, no concrete solution is provided.\n2. The proposed clustering-based routing relies heavily on pre-trained MOIRAI, which introduces a dependency on the quality of the pre-trained model and dataset (LOTSA), and thus potentially limiting generalization to domains outside the pretraining distribution.\n3. MOIRAI-MOE contains significantly more parameters than those of MOIRAI. This increases memory demands during training and deployment. Is it possible to use model compression technology to reduce the parameters of the model?\n4. When MOIRAI-MOE and MOIRAI parameters are equivalent, how does MOIRAI-MOE perform?\n5. While outperforming foundation models, MOIRAI-MOE still lags behind fully fine-tuned specialized models like TiDE on some datasets (e.g., ETT1 in Table 8), indicating room for improvement in extreme domain adaptation.", "other_comments_or_suggestions": "N/A\n\n## update after rebuttal\nWhile the authors have addressed certain technical concerns (e.g., pruning validation and training cost metrics) with additional clarifications, some limitations around generalizability dependencies and methodological novelty remain unresolved. Though the reviewer leans toward maintaining the original score, the paper's demonstrated empirical improvements and structured rebuttal make a case for acceptance, depending on broader reviewer consensus.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIHq9XXtR6", "reviewer": "Reviewer_Bx6G", "summary": "The authors propose an MoE extension to an existing Time series based foundation model MOIRAI. The authors extends the standard Moirai to MoE to reduce the dependency on human-imposed frequency decomposition as it is not a reliable grouping of pre-training data.  They claim that different frequencies can display similar patterns and similar frequencies can display different patterns. Instead the authors use a sparse MoE approach so that the model can learn the groupings. The authors perform an extensive evaluation and show that their approach results in SOTA performance. The paper is well written and provides clear motivation - this paper is a valuable addition to the scientific community.\n\n## update after rebuttal\nAfter considering the response from the authors - my recommendation remains accept. The work is of merit and useful to the ICML community.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[C1] A weakness of this study is around the evaluation of the probabilistic forecasting component of the paper - which I think is more valuable than the deterministic evaluations performed. The results are indeed impressive but a statistical analysis of the probabilistic forecasting performance and characteristics would be a valuable addition to this well motivated study.**\n\nWe provide standard deviations and statistical significance in Table 1 at this link: https://drive.google.com/file/d/1bwJ7dyji_OnSNkYXpA6IOpwnvF6nOZmS/view\n\n**[C2] The authors argue that different frequencies can display similar patterns and similar frequencies can display different patterns - which I agree with. However could some wavelet transformation/layer alleviate this issue within the standard architecture?**\n\nThank you for the insightful comment. Indeed, applying a wavelet transformation in the input space can effectively help identify and distinguish patterns in time series. However, pattern specialization of Moirai-MoE are performed at each Transformer layer, and this allows our model to adaptively capture patterns at various hierarchical abstraction levels throughout the model. This hierarchical and adaptive specialization provides greater flexibility and modeling power compared to a static transformation applied at the input. Nevertheless, your suggestion of integrating wavelet transformations into the input is intriguing, and further experimentation would be valuable to empirically evaluate and compare the performance.\n\n**[C3] No adaptive patching was considered in the method - so how are different resolutions considered if at all?**\n\nThank you for your comment. We would like to clarify that different resolutions are considered equivalent in our context, as we use a uniform patch size. The critical factor in our approach is the pattern or shape of the time series data, and the specialization of the model primarily addresses this aspect of diversity. Additionally, we have already included experiments examining the effects of different patch sizes in the appendix. The results show that the choice of patch size 16 works the best.\n\n\n**[C4] The authors perform mini-batch k-means clustering on the attention outputs to continuously update the cluster centroids at each layer where the cluster number is equal to the number of experts - did you observe any discrepancies doing this such as the number of clusters being significantly different to the number of experts?**\n\nThank you for the insightful question. This question depends on the relative values of the cluster number K and the number of experts E, and we address it in two parts:\n\nCase 1: K smaller than E — This setting is not valid in our framework. Since each cluster corresponds to an expert during MoE training, having fewer clusters than experts would result in multiple experts sharing the same cluster centroid. This leads to repeated expert selections and undermines the intended diversity and specialization of the experts.\n\nCase 2: K larger than E — This is a more interesting scenario. For example, when K=64 and E=32, we face the challenge of mapping the 64 cluster centroids to the 32 experts. A practical and effective strategy we adopt is as follows: for each token (data point), we compute its distance to all 64 centroids, and select the top 32 centroids with the smallest average distances across tokens. These selected centroids are then used during MoE training. Despite having more clusters than experts, this approach still allows us to select the most representative centroids for expert routing.\n\nWe validated this approach experimentally. Results show that this method achieves comparable performance to the standard setting where K equals to E, indicating that even with a larger number of clusters, a careful selection mechanism can yield effective and robust MoE behavior.", "claims_and_evidence": "The authors claim that an MoE approach removes the human-imposed frequency decomposition within the MOIRAI model such that the model itself learns the appropriate token clustering. The claims are well founded and the authors provide clear intuition behind their rationale - as seen in Figure 1. Additionally, the authors clearly demonstrate the SOTA performance of their method which further backs up their approach. The claims stated in the paper are well backed up by the evaluation.", "methods_and_evaluation_criteria": "The proposed method is clearly motivated and the evaluation is comprehensive - 39 datasets and current state of the art time-series FMs.\nThe core idea of MOIRAI-MOE is to exclude human-defined time series groupings while delegating the modeling of diverse time series patterns to the sparsely activated experts in Transformer layers. The authors also investigate existing expert gating functions that generally use a randomly initialized linear layer for expert assignments. They introduce a new expert gating function for accurate expert assignments and improved performance. The author propose automatic time series token-level specialisation, where diverse tokens are handled by different experts, while similar tokens share parameter space, reducing learning complexity. MoE is done by replacing every FFN by an MoE layer consisting of M experts and a Gating function G. The evaluation criteria used is well motivated and comprehensive.", "theoretical_claims": "No theoretical claims were made in the paper", "experimental_designs_or_analyses": "The authors use a comprehensive and appropriate experimental design consisting of relevant datasets and other SOTA time-series foundation models.", "relation_to_broader_scientific_literature": "This paper is of clear interest to time-series forecasting and advancing the current SOTA with respect to time-series Foundation models. The impact of such research is of high significance and impact to the wider scientific community due to the application of time-series forecasting.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "A weakness of this study is around the evaluation of the probabilistic forecasting component of the paper - which I think is more valuable than the deterministic evaluations performed. The results are indeed impressive but a statistical analysis of the probabilistic forecasting performance and characteristics would be a valuable addition to this well motivated study.", "other_comments_or_suggestions": "One typo on line 213: \"Mo\" should be \"MoE\"", "questions_for_authors": "1) The authors argue that different frequencies can display similar patterns and similar frequencies can display different patterns - which I agree with. However could some wavelet transformation/layer not alleviate this issue within the standard architecture?\n\n2) No adaptive patching was considered in the method - so how are different resolutions considered if at all?\n\n3) The authors perform mini-batch k-means clustering on the attention outputs to continuously update the cluster centroids at each layer where the cluster number is equal to the number of experts - did you observe any discrepancies doing this such as the number of clusters being significantly different to the number of experts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes MOIRAI-MOE, an extension of the MOIRAI time series foundation model that incorporates a sparse Mixture of Experts (MoE) architecture. The key motivation is to replace MOIRAI's predefined frequency-based grouping of time series with a data-driven, automated approach where different "experts" (specialized feed-forward networks) handle different types of token patterns. The authors introduce a novel, non-trainable expert gating mechanism based on clustering the attention outputs of a pre-trained dense MOIRAI model, arguing this is more robust than typical trainable linear gating layers sensitive to initialization. Extensive experiments on numerous datasets show MOIRAI-MOE variants achieving state-of-the-art performance compared to the original MOIRAI models and other foundation models like Chronos and TimesFM, often with fewer activated parameters during inference. Initial reviews acknowledged the paper's clarity, the importance of addressing data heterogeneity in TSFMs, and the extensive empirical evaluation. However, concerns were raised regarding the technical novelty (especially compared to prior MoE work like Time-MoE), the justification and novelty of the proposed cluster-based gating function, the lack of reported result variability, and observations of highly sparse expert utilization in deeper layers suggesting potential inefficiency or training issues.

In response, the authors provided detailed clarifications and significant additional experimental results. They defended the technical contribution by highlighting the non-trivial engineering effort required for successful MoE integration in TSFMs (contrasting with Time-MoE's lower performance) and emphasized the novelty of their non-trainable, cluster-initialized gating compared to standard trainable linear or hash-based approaches. They provided results with standard deviations and statistical significance tests, conducted new experiments comparing parameter-matched versions of MOIRAI-MOE and MOIRAI (favoring MOIRAI-MOE), presented results on pruning underutilized experts (showing performance stability), and offered explanations for the sparse expert activation based on representation similarity in deeper layers of pre-trained TSFMs, supported by existing literature. They also addressed minor issues regarding code access, visualizations, and experimental details. The rebuttal satisfied some reviewer concerns, particularly around variability reporting and code access. However, fundamental disagreements persisted regarding the novelty/justification of the gating mechanism versus prior art and the implications of the sparse expert utilization, with some reviewers remaining unconvinced by the provided explanations.
Decision: Accept (poster)