Continual Pre-training of MoEs: How robust is your router?
Authors: Benjamin Thérien
Charles-Étienne Joseph
Zain Sarwar
Ashwinee Panda
Anirban Das
Shi-Xiong Zhang
Stephen Rawls
Sambit Sahu
Eugene Belilovsky
Irina Rish
Venue: Accepted by TMLR ()

Abstract: Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of floating-point operations (FLOPs) per forward pass, MoEs benefit from improved sample efficiency at training time and achieve much stronger performance. Many closed-source and open-source frontier language models have thus adopted an MoE architecture. Naturally, practitioners will want to extend the capabilities of these models with large amounts of newly collected data without completely re-training them. Prior work has shown that a simple combination of replay, learning rate re-warming, and re-decaying can enable the continual pre-training (CPT) of dense decoder-only transformers with minimal performance degradation compared to full re-training. In the case of decoder-only MoE transformers, however, it is unclear how the routing algorithm will impact continual pre-training performance: 1) *do the MoE transformer's routers exacerbate forgetting relative to a dense model?*; 2) *do the routers maintain a balanced load on previous distributions after CPT?*; 3) *are the same strategies applied to dense models sufficient to continually pre-train MoE LLMs?* In what follows, we conduct a large-scale study training a 500M parameter dense transformer and four 500M-active/2B-total parameter MoE transformers, following the Switch Transformer architecture and a granular DeepSeek-inspired architecture. Each model is trained for 600B tokens. Our results establish a surprising robustness to distribution shifts for MoEs using both Sinkhorn-Balanced and Z-and-Aux-loss-balanced routing algorithms, even in MoEs continually pre-trained without replay. Moreover, we show that MoE LLMs maintain their sample efficiency (relative to a FLOP-matched dense model) during CPT and that they can match the performance of a fully re-trained MoE at a fraction of the cost.

Published in Transactions on Machine Learning Research (09/2025)
Continual Pre-training of MoEs: How robust is your router?
Benjamin Thérien 1,2,5
Sennjarnin..eeriennee euueba.uuueueeeeeeeee shartesetienne.josephhhee joppijssepppaaiiaooooee.. outtessiia..ssppptttttttt...o com zsarwar Saroarrrcche Buchicago. edu ashwinee@princeee edu anirban, dasJaacapiialnn shirionn.zhuaeee whang@uppiaaanoneee....o com stephen.rawe rawls@capitaalon com sombit. sahu@capitalon com eugene. belilovsky cosioooskyooooooriiiee wwn.orooooocccooooooooooo ca irina. rish @mella. quebec
Charles-Etiena Etienne Joseph
Zain Sarwar
Ashwinee Panda
Anirban Das
Shi-Xiong Zhang
Stephen Rawls
Sambit Sahu
Eugene Belilovsky2.0
Irina Rish
Université de Montréal, Montréal, Canada
2 Mila Quebec AT Institute, Montréal, Canada
Concordia University, Montréal, Canada University of Chicago, Chicago, USA
Capital One, New York, NY, USA
Reviewed on OpenReview: https: // opwpreviiee net/ pormfo id= dR7C1K71Rs
Abstract
Sparsely-activated Mixture of Experts (MoF) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of poating-point operations (FLOPs) per forward pass, MoEs benefit from improved sample efficiency at training time and achieve much stronger performance. Many closed-source and open-source frontier language models have thus adopted an MoE architecture. Naturally, practitioners will want extend the capabilities of these models with large amounts of newly collected data without completely re-training them. Prior work has shown that simple combination of replay, learning rate re-warming, and re-decaying can enable the continual pre- training (CPT) of dense decodcr-only transformers with minimal performance degradation compared to full re-training. In the case of decoder-only MoE transformers, however, it unclear how the routing algorithm will impact continual pre-training performance: 1) do the MoE transformer's routers exacerbate forgetting relative to dense model?; 2) do the roulers mainlain a balanced load on previous distribulions afler CPT?; 3) are the same strategies applied to dense models sufficient to continually pre-train MoF L.LMs? In what follows, we conduct a large-scale study training a 500M parameter dense transformer and four 500M-active/2B- 2B-tttal parameter MoE transformers, following the Switch Transformer architect ure and a granular DeepSeek-inspired architecture. Each model trained for 600B tokens. Our results establish a surprising robustness to distribution shifts for MoEs using both Sinkhorn-Balancee and Lurd--AA--oss-aalaa continually pre-trained without replay. Moreover, we show that MoE LLMs maintain their sample efficiency (relative to FLOP-matched dense model) during CPT and that they can match the performance of fully re-trained MoE at fraction of the cost.
routing algorithms, even in MoEs
1 Introduction
Sparsely-activated MoE transformers achieve significantly stronger performance than models (e.g., dense models requiring the same amount floating point operations (FLOPs) per forward pass). This is particularly advantagcous today's foundation model lifecycle, where a model spends the majority of its lifetime FLOPs being inferenced. Many closed-source and open-source frontier language models have thus adopted MoE urchitecture (Dai al., 2024; DeepSeck-AT et al., 2024; Jiang al., 2024; Abdin et al., 2024;
dense

---

Published in Transactions on Machine Learning Research (09/2025)
DeepSeek-AI et al., 2025b:a). Given the clear advantages of MoEs over dense transformers, practitioners will certainly want to update MoFs new data as is currently done for dense transformers.
Continual pre-training with replay, learning rate re-warming, and re-decaying has been shown to be a simple but effective solution for updating pre-trained dense autoregressive transformers on large amounts of new data (Tbrahim et al., 2024; Gupta et al., 2023; Parmar et al., 2024) and competitive with full re-training (Ibrahim et al., 2024), while being much cheaper. An open question is whether the same strategies are sufficient to contimually pre-train MoE LLMs? MoE pre-training has been notoriously difficult due to instabilities introduced by the routing algorithm and the need to maintain balanced load across experts (Lepikhin et al., 2021; Shazeer et al., 2017; Fedus et al., 2022; Zoph et al., 2022). During continual pre-training, these challenges may be exacerbated by the distribution shift.
Without proper care, MoE transformers learn greedy routing strategies that overutilize certain experts, leading to poorer downstream performance and poorer accolerator utilization. During MoE pre-training, load balancing strategies are used to prevent such negative outcomes (Fedus et al., 2022; Zoph et al., 2022; Clark et al., 2022; Anthony et al., 2024; Dai et al., 2024). However, the load-balancing algorithms used in SOTA MoE's were not specifically designed for the non-lID data distributions encountered during continual pre-training. Adapting the router's decisions to a new distribution during CPT may compromise the balanced load on previous distributions, potentially leading to exacerbated forgetting and poor accelerator utilization. Avoiding these failure modes is critical for successfully updating MoE' foundation models without the need for full re-training, but the continual pre-training of MoEs has not yet been thoroughly studied in the literature.
In this work, we fill the gap by providing systematic study of MoF continual pre-training. Specifically, we select two popular routing algorithms and two popular MoE architectures used in statc-of-the-art existing work (Dai et al., 2024; Muennighoff et al., 2024; Fedus et al., 2022; Clark et al., 2022) to yield four different MoEs for our study. We then pre-train each MoE language model on 400B tokens of Fen and continually pre-train them on 200B tokens of code data and German web crawl data. Taking the strongest MoE architecture, we compare its performance to full re-training baseline on both datasets. Our contributions can be summarized as follows.
We establish the effect of replay and infinite learning rate (LR) schedules on the forgetting and routing imbalance dynamics of MoE transformer LMs during CPT.
We demonstrate that a Penalty-Balann Balanced (e-g- with and Aux loss) MoF following the DeepSeek architecture can successfully match the performance of a full re-training baseline, at fraction of the cost.
We show that MoEs using either Ponalty-Balanced Balancod or surprisingly robust to distribution shifts in terms of 1) language modeling performance, 2) evaluation benchmarks, and 3) maximum routing imbalance.
routing algorithms are
We provide a comprehensive analysis of how routing decisions change during continual pre-training that provides insight into how MoEs adapt new distributions and forget previous ones.
2 Background
This section provides a concise summary of the relevant background for our study, A more detailed version can be found in Section A of the appendix.
Continual Pre-training of LLMs. Continual pro-training (CPT) extends pro-training to one or more new distributions. Concretely, continual pre-training occurs when a model is trained on a sequence of datasets Do, Di,..., DN with different distributions, N 2 2, and each dataset is sufficiently large (e.g., > 100B tokens in the case of language). Recently, Ibrahim et al. (2024) established that CPT LLMs can match the performance of full re-training by simply repeating the pre-training schedule (cosine annealing) for CPT and replaying previous data. However, if one has control over the initial pre-training, it is possible to further improve CPT by using an infinite learning rate schedule (Janson et al., 2025).
Sparscly-activated MoE transformers differ from their dense counterparts by dynamically routing tokens in sequence to different sets of parameters, called experts. Algorithms for dynamically selecting among
2

---

Published in Transactions on Machine Learning Research (09/2025)
Compute Cost
600
000
Lay ex Per--wiso MRU
Layor-wits wite MRI
Tokens
100
Median
38%
Median
4.15
3.46
200
210
6.0
German
German MoE
Slack
Stack
German
German MoE
Stack Dense
Stack
German
German MoE
Stack
Stack
Dense
Dense
MoE
Donse
MoE
Dense
Dense
MoE
(a)
Compute Cost (4)
(b) Modian MRI, FineWob ())
(c)
Median MRI, Ger. & Stack (L)
AVG Final
Validation Loss
1,0 1at
AVG English
Eval. Performance
60
AVG Downstream
Eval. Performance
30
27.6627.50
Continual cee-training
40
20
Full
Re trainiinn
20
10
7:01 7.44
German
German
Stack Donse
Stack
German German
Stack
Stack
German German
Stack
Stack
Dense
MoË
MoE
Dense
MoE
Dense
MoE
Dense
MoE
Dense
MoE
(d) AVG Validation Loss (L)
(e) AVG English Eval. Acc.
(t)
(f) AVG Task-2 Eval. Acc. (t)
Figure 1: Continually pre-trained (CPT) MoEs match the performance of full re-training across
two dataset transitions: 400B English-> 200B German (40% replay) and 400B English-> 200B Stack (30% replay). We compare the performance of fully re-trained (e.g. trained on the union of 400B English and 200B stack or 200B German) Penalty-Balanced Balanced Top-k MoE and dense baseline, to their CPT counterparts. Despite incurring only a. third of the substantial full-retraining cost, the CPT MoEs match the performance of the fully re-trained models, even achieving improvements median Maximum Routing Imbalance (MRT) in some cases. This shows that MoEs have CPT abilities on par with dense transformers. Note that subfigures (b), (c), and (f) evaluate German and Stack models on different datasets which correspond to their training domain.
experts, known routing algorithms, therefore central to MoEs and may be non-trivially affected by distribution shifts. In this work, we focus studying two prominent Top-k routing algorithms from recent state-of-the-art (SOTA) works: Penalty-Balanced Top-k (PBTk) routing (Shazeer et al., 2017; Dai et al., 2024; Zoph el al., 2022; Fedus et al., 2022) and Sinkhorn-Balanced Top-k (SBTk) routing (Clark et al., 2022; Authony et al., 2024). PBTи methods encourage a balanced load across experts by adding a penalty term (called Aur Loss) to the overall loss. Several recent SOTA works (Dai et al., 2024; Team, 2024) also include a second term, 2-loss, to penalize large-magnitude router logits, which has been shown to promote stability (Zoph et al., 2022). In experiments, combine both losses and refer to the method as PBTK routing. SBT* methods cast the assignment of tokens to experts et al., 2022). The Sinkhorn-knopp algorithm (Knopp & Sinkhorn, 1967) provides an approximate solution to this problem, which can be elficiently computed on GPUs and can be sped up by choosing a favourable initial condition (Anthony et al., 2024). We refer to Sinkhorn routing using the initialization of Anthony et al. (2024) as SBTk this work.
linear assignment problem (Clark
3 Related work
This section provides a review of the most relevant literature, which we complement with more comprehensive review in Section B of the appendix.
Continual Pre-training of Dense Foundation Models Several existing works study continual learning in settings relevant to CPT, finding that self-supervised pre-training benefits from reduced forgetting (Cossu et al., 2022; Davari et al., 2022), that pre-trained models forget less than their randomly initialized counter- parts (Mehta et al., 2023), that forgetting improves as model scale is increased (Ramasesh et al., 2022), and that wider models lend to forget less than deeper models (Mirzadeh et al., 2022). In the context of large-scale CPT of LLMs, Gupta et al. (2023) highlights the importance of re- warming the learning rate when models are pre-trained from a chockpoint that has boen decayed to small learning rate. Following up on their work, Ibrahim et al. (2024) establishes the effectiveness of learning rate ro-warming, LR re-decaying, and replay for large-scale CPT of L.Mss.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Continual Pre-training of MoE LLMs. To the best of our knowledge, only a single work exists exploring the large-scale continual pre-training of MoFs LLMs, while the majority of the literature focuses on upeycling or growing MoEs for continual pre-training. In concurrent pre-print DeepSeek-Conn CoderV2 (DeepSeek-A) et al., 2024), shows that they can continue from a checkpoint the training of a MoE LLM. However, this is only shown for one instance and the analysis of the MoE routing behavior not discussed. Furthermore, there is no comparison to a FL.OP-matched dense model, making it challenging to assess whether the sample efficiency of MoE LLMs is maintained during continual pre-training. Continual pre-training methods for MoEs that are less related to our work generally focus fine-tuning MoE LLMs on small amounts of data (Wang al., 2024c) or growing MoEs (Komatsuzaki et al., 2023; Zhu et al., 2024; Sukhbaatar et al., 2024; Gritsch et al., 2024).
4 Method & Empirical Study
Given our goal of studying the large-scale continual pre-training of MoE LLMs, our main methodological contribution involves identifying practically relevant MoF architectures to study, appropriately combining them with SOTA CPT techniques (e.g. (Ibrahim et al., 2024)), and providing succinct guidelines for continual MoE pre-training derived from our empirical results. In the following section, we will describe the key design choices made when constructing our study w.r.t. MoE architectures, datasets, and CPT techniques and introduce a new metric for mcasuring latency in MoEs. Guidelines for the CPT of MoEs will be outlined in the results section.
4.1 Selected architectures for our study
FLOP-matched Dense Bascline. We select a 24 layer 570M parameter dense decoder-only transformer following the Llama3 architecture (except we use GeLU activations) and using the Llama3 tokenizer (Dubey et al., 2024) (see Sec. F for details).
Granular MoEs. Given the recent popularity and strong performance of DeepSeek MoEs (DeepSeek-Al et al., 2024; Dai et al., 2024; DeepSeek-AI et al., 2025b;a), we include an MoE architecture that activates multiple granular experts and a shared expert. Specifically, each granular MoF has F = 31 total routed experts, K = 3 active experts, and 1 shared expert. This model follows the same Llama3 architecture the dense model described above. Notably, its experts are GEGLU FFNs with an intermediate size that is 1/ the size used in the dense model. We train two granular MoEs utilizing the Penalty-Balanced and Sinkhorn-Balanced Top-k routing algorithms, respectively. We do not drop tokens.
Switch MoEs. Given the historical use of full-sized FFNs in MoEs, our study also includes an architecture similar to (Jiang al., 2024; Fedus et al., 2022) with full-sized experts and no shared expert. We refer to these Switch MoEs and also train two utilizing the Penalty-Balanced and Sinkhorn-Balanced routing algorithms, respectively. Each switch MoE has E = 8 total routed experts, K = 1 active experts, and no shared expert. This model follows the same Llama3 architecture as the dense model described above. Notably, its experts are GEGLU FFNs of the same size as the dense model's FFNs. We do not drop tokens.
4.2 Continual pre- trainin strategy and Datasets
To initially pre-train and subsequently continually pre-train our models, we use three datasets: FineWeb (Penedo et al., 2024), the Stack (Kocetkov et al., 2023), and German Common Crawl (Abadji et al., 2022). We initially pre-train all models on Feb for 400B tokens (task 1) to mimic open and closed source models often pre-trained on large-scale web-scraped English data. Subsequently, we continually pre-train these base models on 200B tokens of Code or German data (task 2) using infinite learning rate schedules and replay (30% & 40% respectively) to mitigate forgetting. We select large amounts of replay for full continual pre-training following previous SOTA work (DeepSeek-AI et al., 2024), but show the effect of modifying the replay percentage in section 5.1. We chose distribution shifts to multilingual and code data as they represent stark distribution shifts from the English pre-training data, while being realistic (c.g., the Llama3 tokenizer is still viable for Whese domains). It should be noted that our total training budget of 600B tokens falls strictly in the overtraining regime with respect to a chinchilla optimal token budget for dense models (Hoffmann et al., 2022). For the Dense bascline, this corresponds to overtraining to roughly 40X the chinchilla optimal recommendation, while for the MoEs, this corresponds to roughly 10X the compute optimal amount for 2B parameter dense model. For comparison, DeepSeekV3 (DeepSeek-Al et al., 2025b) trains to about.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Validation Loss FineWeb
Validation Loss FineWeb
4.00
4.00
Dense Baseline [D]
3.75
3.75
Dense Baseline
SB Switch Mož [D]
3.50
3.50
SB Switch Moï
PB Switch MoE [D)
3.25
3.25
Switch
3.00
3.00
PB Granular MoË [D]
2.75
2.75
PB Gramular MoE
20
40
20
40
Training Tokens (B)
Training Tokens (B)
(a) 0% Replay
(b) 40% Replay
1.7
1.7
Validation Loss German
Validation Loss German
1.6
1.6
1.5
1.5
1.4
1.4
1.3
1.3
1.2
1.2
0
20
40
20
40
Training Tokens (B)
Training Tokens (B)
(c) 0% Replay
(d) 40% Replay
Figure 2: Ablating replay and decay strategy during continual pre-training on German data. We CPT MoË' and a dense baseline from fully-decayed checkpoints (dotted curves, [D]) or non-decayed chockpoint (full curves). The figures report the performance on task 1 (FineWeb) and task (German) while CPT on lask 2. We observe that adaptation Lask similar within an architecture for both checkpoints, that CPT from a non-decayed checkpoint improves forgetting, and that replay mitigates forgetting.
1.1X the Chinchilla optimal ratio for a 671B dense model and the popular Qwen3 series models reaches chinchilla-optmmal training multipliers of 7.66-58.800 6.065 for the MoEs and 54.55-225X for the larger (8B +) dense models. These models are frequently used as starting point for continual pre-training, showing how our 40x chinchilla-optimal for dense and 10x chinchilla-optinal for MoEs represenlative of real application settings.
Compute Equivalent Replay Replaying previous data has been longstanding tool for mitigating catastrophic forgetting (Wang et al., 2024b). In our experiments, we replay previously seen data for this purpose, designating any model using the technique with the suffix "X% Replay'. Here, represents the percentage of samples in given batch that were replayod from the previous distribution. To match compute across different replay budgets, we do not inerease the loken budget when increasing the amount of replay. Instead, decrease the amount of new data seen during CPT.
5

---

Published in Transactions on Machine Learning Research (09/2025)
4.3 Training details
All models in our study (except re-training baselines) were pre-trained for 192, 720 gradient descent steps using batch size of 1024, a sequence length of 2048, the AdamW optimizer, and the Cosine Inf schodule (Ibrahim el al., 2024). For continual pre-training, each model in the main study follows a Cosine Inf schedule resuming from the non-decayed checkpoint, while some models in the ablation section were continually pre-trained from decayed checkpoints following a cosine decay schedule (e.g,, replicating the setting from Ibrahim et al. (2024)). We continually pre-train the models for 95, 370 gradient descent steps using the same batch size and sequence length as during pre-training. Each model was trained across 64 A100 GPUs using data parallelism and zero-1 (Rajbhandari et al., 2020). To accelerate the dropless MoE' forward pass we use the Megablocks kernel (Gale et al., 2023). More details of the exact schedules used for each experiment are provided in section F of the appendix. Additionally, figure 22 illustrates learning rate schedules used for (a) pre-training, (b) continual pro-training, (c) full re-training, and (d) rewarming ablation of Sec. 5.1.
4.4 Maximum Routing Imbalance: A proxy for worst-case latency in MoEs
While performance is one important axis of robustness to distribution shifts, maintaining a balanced load across experts just as important for MoE foundation models. Without a balanced load, MoE transformers inferenced using expert parallelism without token dropping (e.g., as is done for SOTA models (DeepSeek-A) AI et al., 2025b; Zhao et al., 2025)) could be bottlenecked by the speed of single accelerator that receives all the tokens, leading to underutilization of the hardware, lower throughput, and higher costs. To quantitatively assess the effect of distribution shift on load balance, we propose the maximum routing imbalance (MRT): the largest proportion of tokens routed to a single expert in a given MoE layer. Concretely, the MRI at a training itcration and MoE layer j is defined as
EVEB lii E [x())} [B]
MRI((,)) =- max :E(I.....]]
(1)
Where B is set containing all tokens in given batch, L is the indicator function, E' is the number of routed experts, and k is the number of active experts. Since latency increases with computation, and, in an MoE layer, the computation required by a given device increases with the load of experts on that device, then MRI calculated with respect to routing decisions on a distribution is a prozy for the worst case latency of an MoE loyer on the distribution. We will use the MRI throughout the following sections to measure the effect of algorithmic changes to continual pre-training on routing imbalance. In figure we report the maximum MRI across all layers in the MoF (e-g., maX]e([] MR)( where I. is the number of layers) at training iterations immediately preceding and immediatcly after the distribution shift. In figure 4 we report the MRI at cach layer of our MoE transformers before and after continual pre-training.
MRI v.s. Latency While MRI does not report latency, it is faithful behavioural metric that can be used as input to a latency model for estimating the lar. Unlike latency, which will always depend on specific hardware and implementation, MRI is independent of these considerations and is ultimately more comparable across different deployments.
5
Results
5.1 Ablating replay (%) and the checkpoint used for CPT
In the following section, we ablate the replay percentage used during continual pre-training and consider continually pro-training from two distinct checkpoints: a checkpoint that (a) was decayed to Mainn during pre-training (the case for most open-source MoEs) or (b) followed an infinite learning rate schedule and was not decayed (the ideal case achievable when one has control over the pre-training phase). Models in group (a) are continually pre-trained following a linear warmup and cosine decay schedule that rewarms the learning rate to Mmaa before re-decaying it (e.g., as in Ibrahim et al. (2024)), while the models in group (b) are continually pre-trained starting from Teonst following an infinite LR schedule (exact values are provided in Sec. F).
Validation Loss. Figure 2 reports the validation loss for these models during the first 50B tokens of continual pre-training. While we only show the first 50B tokens due to resource constraints, the schedules were set. to
6

---

Published in Transactions on Machine Learning Research (09/2025)
Maximuum
Routing Imbalens
20
+
Unitern Roating
PB Gounular Mall 0% Replay
Begins Ulee 190720)
SB5 Grumsier Med 0% Heploy
Cantinual Fre tcuining
Maximuu
Routing Imbislance (15)
30-
20-
Uniifrrm Bosting
Sis Gransler Mott o% Replay [t]
PS Granuler Mot 5% Huplay [t]
Begine (Ibr 199720)
Continud Prestruiiing
Maximuu
Routing Imbsslance (16)
20-
Clrken Roating
Begios ther 193750
Gruzdar Mal 0% Raplyy
Cruasar Mud Rastar
Greanir Mad (rn Replay
Oreznkr Med 10m Reykay
10-
Gruzdar Mal 0% Rapley
20
10
10
750700
1e0so0
150p00
10tne0
191109
ta1200
100 nm
t90500
taesen
191000
181100
105200
toosa0
100700
teotoo
tarseo
191000
181100
105200
Training Herstion
Training Recation
Training Recation
(a) Non-decayed Granular MaR
(b) Decayed Gramular MoF
(c)
Replay Ablation
Figure 3: Febe -> German CPT checkpoint and replay ablation. We report the median Maximum Ballace Routing Imbalance (MRI) across MoE layers with min/max error bars. Sinkhorn- Balanced (SBTk) MoEs show a slight MRT increase during distribution shift, while PBTk MoEs experience a brief spike before recovering to balanced MRI levels below SBTK, which approach the uniform baseline. The uniform routing bascline (orange line) corresponds to the case where cach expert across all layers reccives the same number of tokens; thus, it represents perfect balance.
decay at 200B tokens, mimicking the start of a longer continual pre-training, Subfigures (a) and (c) show forgetting and adaptation plots using 0% replay, while subfigures (b) and (d) show analogous plots using 40% replay. We observe that as the percentage of replay is increased, the forgetting as measured by W'nneWel validation loss is mitigated, while the adaptation to the downstream datasct harmed. Turning our attention to the checkpoints used, we observe that, for all replay values and all models, using non-decayed checkpoints improves forgetting on the FineWeb without compromising adaptation. These results show that, similar to dense transformers, MoEs can tradeoff forgetting for adaptation with replay and benefit from infinite LR. schedules
Routing Imbalance. Figure 3 (a,b) reports median MRI computed across all transformer blocks of SBTk and PBTk Granular MoEs during CPT with 0% replay, subfigure (c) reports results across different replay percentages and results for switch MoEs are reported in Figures 19 and 18 of the appendix. These figures precisely study the distribution shift by reporting the MRT immediately before and after the transition from English and German data. We observe that SBTk MoEs are consistently robust to the distribution shift, showing only a small increase in MRI across different replay percentages and for decayed and non- decayed checkpoints alike. This is likely attributable to the explicit balancing step in Sinkhorn routing, In contrast, the non-dccayed and decayed PBTK MoE chockpoints go through a period of high routing imbalance immediately following the distribution shift. However, this period is short-lived: the PBTk checkpoints recover to well-balanced MRI levels, better than those of SBTK, within 500 training steps. Subfigure (c) shows that this MRI spike can be mitigated with replay, though the benefit is negligible, as even the no-replay model recovers quickly. These results suggest that although SB is more robust to distribution shifts than PB, this robustness limits the MRI attainable. We hypothesize that the generally higher MRT of PB models may cause an uneven utilization of the MoE's parameters during training. Such a difference in expert utilization during training could possibly explain the differences in performance between PB and SB. Finally, the chaotic phase undergone by PBT6 checkpoints does not last long enough to forego the strong performance of these models.
5.2 Language modeling performance
Having established the benefits of replay and infinite learning rate schedules for continually pre-training MoEs, we now quantitatively verify the efficacy of these techniques by continually pre-training our MoEs on 200B lokens of Code and German Common Crawl data and evaluating their performance relative to two baselines. Specifically, we will compare the performance of the four continually pre-trained MoEs in our study to FLOP-matched dense bascline and a fully re-trained PBTk Granular MoE Bascline (the best-performing architecture in our study). Performance will be measured across 4 axes: validation loss on the pro-training and CPT datasets, English evaluation benchmarks (task 1), German and Code evaluation benchmarks (task
7

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Aggregated benchmark results. MoEs consistently outperform FLOOP-matched dense baselines and exhibit less forgetting w.r.t. validation loss. Compared to the re-training bascline (bluc), MoEs and the dense model match or exceed their performance. These results show MoEs adapt as well as dense models but forget less, likely due to their larger parameter count. All validation losses report log perplexity on a held-out validation set, forgetting (equivalent to backward transfer in Lopez-Paz & Ranzato (2017)) is calculated using validation loss, and downstream tasks report accuracy.
Final Validation Loss (4)
Dowastream Evals,
(*)
Training Tokens
Model
Fine
Stack
German
Forgetting
AVG
English
German
Stack
Dense Baseline
2.881
4.028
3.741
49.84%
23.54%
0.00%
SB Switch MoE
2.711
3.861
3.495
54.14%
25.115%
0.00%
400B Web
PB Swikch. MoB
2,69
3,872
3:451
54.45%
23.37%
0.00%
SH Gramn ar Moli
3,600
3.404
55.71%
22.8335
0.00%
PB Granular MoE.
2.633
3.715
3.370
50.5996
23.40%
0.00%%
Deuse Baseline
1,025
0.059
1.982
49.215
7.37%
4008 FineWeb
2004
SH Switch Mols
2 757
0.944
0.046
1.850
51.76%
9.09%
Stack
PD Switch. MoE
2.749
0.945
0.050
1.847
52.58%
8.22%
(30% Heplay)
SD Gramnaar MaE
2.208
0.925
0.044
1.818
53.5156
7.45%
PB Gramilar MoF.
2.609
0.924
0.046
1.811
53.70%
7.81%
200B Web U 2008 Stack
Deuse Baseline
PB Gramiaa Moll
2 830
398
0.935
1.050
1 1.782
1.958
54.7U%
49.57%
7.44%
3.78%
Deman Heaeline
2.946
1.367
0.066
2.157
48.16%
25.1755
400B FineWeb 2008 Ger-
SB Switch MoE
2.749
1.142
0.039
1.926
51.99%
27.57%
PB Switch. Mob
2.741
1.129
1.035
51.25%
26.5095
(40% Replay)
SB Grannaar ar MalF
2. 2.701
1.118
0.037
1.910
53.35%
28.67%
PD Gramniar McE
2,000
1.099
0.037
1. 1.895
53.61%
27.6556
400B Wen u 2008 German
Dense Basellie
23335
1.390
2.164
48.42%
25.4555
PB Gesaniar MaE
2,089
1.120
1.895
53.94%
27.58%
2), and MRI of final checkpoints. Note that the main conclusions of this section are succinctly summarized in Figure 1.
Validation Loss. Table 1 reports validation loss (c.g., log perplexity on a fixed held-out validation set) results for the main models in our study, while extended results are reported in Table 6 of the appendix. We observe that all MoEs outperform the FLOP-matched dense baseline during pre-training and CPT. Within the MoEs, we observe that PBTk MoEs consistently outperform SBTk MoEs and that Granular MoEs consistently outperform switch MoEs across pre-training and CPT. These findings are consistent with the literature on Granular MoFs (Ludziejewski et al., 2024; Dai et al., 2024), but we believe this is the first time that SBTk routing has been shown to underperform PBTE routing. Since the PBTk Granular MoE achieves the best performance, we use it as our full re-training baseline. Compared to full re-training, our Granular CPT MoEs consistently have higher FineWeb validation loss but achieve lower downstream validation loss with a similar average validation loss. Similar results are found when comparing the CPT dense bascline to its full re-training counterpart. These results demonstrate that the continual learning abilities of MoEs w.r.t. validation loss are on par with dense models for adaptation and are slightly superior in terms of forgetting, likely due to their larger total parameter count.
English Evaluation results. Table presents average accuracy, while Table 4 details per-benchmark results. We select benchmarks where models of our scale (570M active parameters) achieve non-trivial accuracy to maximize signal. Each model is evaluated zero-shot on benchmarks covering Commonsense Reasoning, Reading Comprehension, Scientific Question Answering, and Math: HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2019), PIQA (Bisk et al., 2019), ARC-Fasy, A ARC-Challenge (Clark et al., 2018), SWAG (Zellers et al., 2018), LAMBADA (OpenAI) (Storks et al., 2019), SciQ (Johannes Welbl, 2017), PubMedQA (Jin et al., 2019), and MathQA (Amini et al., 2019) (see Sec. D.1 for details). We find that models trained solely Fib outperform all others, including full re-training baselines. Granular MoEs surpass switch MoEs. CPT models trained on Stack perform similarly to those trained on German. Compared to full re-training, CPT models achieve nearly identical results (within ~ 1%). The dense baseline also matches its full re- re-training counterpart, indicating that MoEs have similar continual learning abilities on pre-training evaluations while benefiting from improved sample efficiency.
German Evaluation results. Table shows average German evaluation performance, while table 5 of the appendix provides per-benchmark breakdown. We use GPT-3 -trn--ttaanltted German versions of
8

---

Published in Transactions on Machine Learning Research (09/2023)
Routing imbslanee (16)
Roating Imbalanos (56
PaweWsb
P3 Gronular MoE
Routing Imbalance
Certias 0% Roplas
Maxemmm
Maximum
22-
Maximum
P3 Cratulis NoE
Stark Beplay PSS Gramilr Mutt"
10-
FlueWeb 17 Demmun
53
P3 dronulrr Mot
10
20
Layer
Layer
Layer
(a) FineWeb
(b) German
(c) Stack
Figure 4: Layer-wise Maximum Routing Imbalance (MRI) for Granular MoEs. We report MRI (eq- 1) on each dataset's 20M-token test set. MRT is consistently lower for Penalty-Balanced MoEs than Sinkhorn-Balanced MoEs. Continual pre-training on FineWeb incurs minimal MRI increase, even with 0% replay. MoEs are most unbalanced with out-of-distribution data (e.g,, non-German models in (b) and non-code models in (c)).
HellaSwag, ARC-Challenge, and TruthfulQA, evaluating each zero-shot (Plüster, 2023). German-trained models outperform English-only ones, and German-trained MoEs surpass the FLOP-matched dense baseline. Among MoEs, modules using the same training tokens perform similarly. CPT MoEs and the full re-training baseline differ by < 1% accuracy, with no clear winner. The dense baseline also performs comparably to full re-training, demonstrating that the continual learning abilitics of MoEs w.r.t. German evaluations are on par with dense models while benefiting from improved sample elficiency.
Code Evaluation results, Table 1 presents average Code evaluation performance, while Table 3 of the appendix provides a pass@k breakdown (k E {1, 10, 50, 100, 150, 200)). Our models are evaluated on Python code-generation lasks from HuaanEval (Chen et al., 2021), as Python is well-represented in our Stack CPT dataset (Table 10). English-trained models can not solve any problem, whereas stack-trained models achieve non-trivial accuracy. Unlike for other performance metrics, CPT switch MoEs slightly outperform their granular counterparts. Compared to full re-training, all CPT MoEs perform marginally better, while the CPT dense model exceeds its baseline by over 3%. We attribute this unexpected improvement to evaluation noise and training variance, given the models' similar validation loss. These results suggest MoEs match dense models in continual learning for code evaluation when accounting for MoEs' improved sample elficiency.
Routing imbalance during and after continual pre-training. Figure 4 shows the layer-wise Maximum Routing Imbalance (MRI) for Granular MoEs across FineWeb (a), German (b), and stack (c), while Figure 16 reports MRI for all MoEs. We include 0% replay baseline for each MoE CPT German to highlight replay's impact on MRI.
In subfigure (a), Penalty-Balanced Balanced MoEs consistently have lower MRI than Sinkhorn-Balancae MoEs across all architectures, and granular MoEs exhibit lower and more stable MRT within archilectures. On FineWeb, continual pre-training causes only a slight MRI increase relative to the pre-trained checkpoint, even for the 0% replay model, except in its first layer. Interestingly, all German-trained MoEs show higher MRI on Feb than their ttack-trained counterparts, with the full re-training baseline, surprisingly, having the highest. This suggests there may be more routing interference between English and German datasets and that continual pre-training may help reduce MRI across distributions, possibly due to the use of Cosinelnf vs. Cosine Annealing schedules for CPT and re-training, respectively.
Granular MoEs also reduce routing imbalance on German (b) and Stack (c) datasots. MoEs become most unbalanced with out-of-distribution data (e.g,, non-German models in (b) and non-code models in (c)). Similar trends hold for Switch MoEs, with an additional finding: high MRI common in early layers of Switch MoEs, independent of baaiiing/testingg distributions, unlike Granular MoEs. These results show that PBTk and SBTk MoEs are robust to distribution shifts w.r.t. MRI and can even outperform re-training baselines, suggesting that continually pre-training MoEs should have no negalive impact on inference latency.
In summary, we find that, across three measures of performance, MoEs continually pre-trained with replay and infinite LR schedules can match the performance of a full re-training baseline and, thus, they have similar CPT abilities to FLOP mlaeehdd dense baseline without any inhibilion from their roulers. Moreover, we show that continually pre-training MoEs has no negative impact MRI compared to re-training.
9

---

Published in Transactions on Machine Learning Research (09/2025)
35.0
17.5
Vocabulary Specializzatonn
Median Co-Activated
$2.5
15.0
30.0
Expert Change
12.5
Continual
10.0
7.5
5.0
2.5
0.0
5
10
15
20
5
10
15
20
Layer
Layer
(a) Co-activated Expert Change
(b) Continual Vocabulary Specialization
100
Router Saturation (%)
Model
Forgetting
60
FineWeb
PB Gramolar Mol
Fib
40% Reglay Moï
German 40% Replay
0.037
German 0% Replay PB Granular MoË
German 0% Replay
1.055
20
Stock 30% Replay PB Granular Moë
Stack 30% Replay
0.046
Layer
(c) Continual Ronter-Saturation
(d) Forgetting of the different MoEs
Figure 5: Layer-wise analysis of ronting changes during CPT. Our goal is to understand how routing decisions change from the pre-trained checkpoints to final checkpoints after continual pre-training. To this end, we analyse changes in routing behaviour from 3 perspectives: which experts tend be activated together (a), the tendency for certain vocabulary tokens to be routed to certain experts (b), and how close routing decisions of the pre-trained checkpoint from CPT checkpoints (c). To provide context these metries, remind the reader of the forgetting (e.g, from table 1) for each model shown in the plots. We observe that the no-replay baseline changes the most in early layers and forgets the most, suggesting that more drastic changes in initial layers may be linked to forgetting.
5.3 Analyzing changes in routing behaviour due to CPT
In the following section, we analyze changes in routing behaviour resulting from continual pre-training. Specifically, we record routing decisions of the MoE checkpoints before and after continual pre-training on 20, 000, 000 tokens of held-out test data, from FineWeb, Stack, and German. To understand how routing decisions change during CPT, we adapt three routing behavior metrics from Muennighoff et al. (2024) the continual pro-training setting: Router Saturation, Vocabulary specialization, and Expert co-activation. We will provide brief descriptions of each in what follows, with formal definitions available in the appendix (Sections D.3.1, D.3.2, and D.3.3).
Router Saturation (RS) is the percentage routing decisions at iteration that match those of the final checkpoint (Muennighoff et al., 2024). We extend this metric multiple training phases for continual pre-training. Figure 5 (c) shows RS between the pre-training and CPT checkpoints for Stack and German Granular PBTk MoEs. RS is lowest in early layers, peaks at layers 2 13, and slightly drops after layer 13. The 0% replay German checkpoint has RS consistently 10 15% lower than the 40% replay checkpoint across all layers. Note that despite adapting well to German, only the no-replay checkpoint suffers significant forgetting on FineWeb. These results suggest that CPT adaptation is most pronounced in layers 0 - 2 and 13 23 and that forgetting has the same pattern but is correlated with lower overall router saturation.
Continual Pouter-Saturatio
Continual Vocabulary Specialization Vocabulary Specialization (VS) quantifies how often a token from dataset. is routed to specific MoF expert relative to its total occurrences (Muennighoff et al., 2024).
10

---

Published in Transactions on Machine Learning Research (09/2025)
By assigning each token in the model's vocabulary to the expert that processes it the most frequently, we can create a one-lo-many mapping between experts and vocabulary entries for an MoF. layer. Then, we can calculate the average VS of each expert by averaging its vocabulary specialization across its assigned tokens. Vocabulary specialization within a layer is calculated by taking the mean average VS across experts in that layer. To compare specialization across model checkpoints, we can re-use the one-lo-many mapping of a previous checkpoint and measure how the specialization w.r.t. this mapping has changed during CPT. Figure 5 shows VS w.r.t. pre-training checkpoints using FineWeb data. VS is notably lower in layers 0-4 after CPT, while there is almost no discernible change in VS for layers 5 - 23. The zero-replay checkpoint exhibits the lowest VS in layers 0-4, correlating with its weaker FineWeb performance and suggesting that excessive VS shifts in early layers may contribute to forgetting.
Co-activated Expert Change Expert co-activation between two experts E, and E, is defined as the ratio of times they are activated together to the total activations of E, over a dataset (Muennighoff et al., 2024). This metric applics only to MoEs with k active experts. A co-activation matrix can be constructed for each ordered expert pair in layer. To compare expert co-activation across model checkpoints, we compute co-activation matrices for all layers of two checkpoints (O(1, ((2) and measure absolute changes by computing statistics of the entries in their clement-wise absolute differences matrix (IC(I) () Figure 5 (a) shows the median co-activation change between the pre- te-irriiiig and CPT checkpoints. Early layers (0-1) exhibit. the largest changes, with a consistent spike at layer 18 for all CPT models and slightly higher median changes in layers 13 23. Among CPT checkpoints, the no-replay variant shows the most significant co-activation shifts. Despite all checkpoints adapting well to new distributions, only the no-replay checkpoint experiences substantial forgetting with co-activation changes in carly (0 2) correlate with higher forgetting.
FineWeb. These findings suggest that adaptation during CPT correlates
2) and later (13 23) MoE layers and that more pronounced changes
In summary, results across all three metrics reveal that routing decisions change most in the early layers of Granular MoE transformers, while changes in other MoE layers are observed for expert co-activation and router saturalion bul not for Vocabulary specialization. Of all models, the no-replay baseline changes the most in early layers and forgets most, suggesting that more drastic changes in initial layers may be linked to forgetting.
6 Conclusion
We have conducted a comprehensive empirical study on the continual pre-training of decoder-only MoE transformer language models. Our large-scale experiments, involving 2B parameter MoEs trained on 600B tokens, demonstrate that both Penalty-Balanced (PBTK) and malkhorn-Balanced (SBTk) routing algorithms exhibit surprising system-level resilience to distribution shifts, maintaining balanced loads as measured by the novel Maximum Routing Imbalance metric. We established that MoEs preserve their sample efficiency advantage over FLOP-matched dense models during CPT and that, when using infinite LR schedules and replay, a Granular PBTk MoEs can match the performance of fully re-trained baselines across German and Code transitions, at a fraction of the computational cost. Finally, we saw that early MoF layers change the most during CPT, suggesting that future work could investigate special treatment of these layers for improved performance. Collectively, our findings establish MoEs as strong continual learners for lext, comparable to dense models, and underscore their potential scalable, adaptable foundation models for language.
Acknowledgments
We acknowledge support from NSERC Discovery Grant RGPIN- 2021-04104 [E.B.], FRQNT New Scholar [E. B.], the Canada CIFAR AT Chair Program [I.R.], the Canada Excellence Research Chairs Program [L.R.), and the FRQNT Doctoral (B2X) scholarship B.T.J. This research was enabled in part by compute resources provided by Mila (mila.quebee). We would also like to thank the GenAI support team at Capital One for their assistance and, in particular, Dhantha Gunarathna. We also thank Akshaj Kumar Veldanda, Andrei Mircea, Hanyang Zhao, and Supriyo Chakraborty for helpful discussions throughout.
11

---

Published in Transactions on Machine Learning Research (09/2025)
References
Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoit Sagot. Towards a cleaner document- oriented multilingual crawled corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pp. 4344 4355. European Language Resources Association, 2022. URL sttps://aclaanthology) wwwpaantttoooooooooooooomo wrggg022........ooo 1rw.-1..e6..
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Picro Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang. Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang. Yue Zhang. Yunan Zhang, and Xiren Zhou Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abss22404.142 2024. doi: 10.48550/ 0044550/ARXTV ARWTV 1404. URL https://dol: http::::/doooo///1///// 4000..8555////. arrivv 2404 14219.
Aida Amini, Saadia Gabriel, Peter Lin, Rik
Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lueas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in Py 9 2023. URL //wp:: www.w.githb.... www.gi...teeeeeeeeeeeeeeeeeeeeee http://leetheaa/gptt-naae
https:
Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models. CoRR, abs/2402.01777. 2024. URL https://doi. org/10 10. 48550/8555i1111 http://vvvv......111. 2402.01774
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical
commonsense in natural language, 2019.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. CoRR, abs/2407.0622 062044, 2024. URL https:///ol org/ 4004448555/0// aarxiv..vvv. www.iv.407...... 2407.06200
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethca Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chanizis, Elizabeth Barnes, Ariel Borbert-Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Tla Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. ar preprint 0rXiv:210700337 2021.
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, Jamos Landon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with distributionnsspecalieed experts. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International
12

---

Published in Transactions on Machine Learning Research (09/2025)
Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. PMLR, 23-29 Jul 2023. URL https://proceeddng/ .ll. http:igg.........rrseeeeeeee chtp//ann22aggg html.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc' Aurelio Ranzalo, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (ods.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 4057-4080 PMI.R, 2022. URL https: / fttps://proceedingg www.eng.........mm.m.m......e..e wwwpp/eeee222 www.k//.aa22a....... html.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challonge, 2018.
Ronan Collobert, Yoshua Bengio, and Samy Bengio. Scaling large learning problems with hard parallel mixtures. Int. J. Pattern Recognit. Artif. Intell., 334-)33493365 2003. URL https://doi.org: org.ooog///o 1otg/10.11 1142/
50218001403002411.
Andrea Cossu, Tinne Tuylelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision, 2022. URL https://arxiv org/abs/ 09357.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedinggs of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thasiand, August 11-16, 2024, pp. 1280- 1297. Association for Computational Linguisties, 2024. URL www..orgg//////// acl- acl-long. 70.
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing repre- sentation forgetting in supervised and unsupervised continual lcarning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 16712 16721, 2022.
Ae,,Seek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. CoRR, abs/2406.1199 2024. URL https://doi... oarg..rr/111 11/110044440 200.48850//rra aero0./rxiv.... 2406. 11931.
DeepSeek-Al, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jiazhoog Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei,
13

---

Published in Transactions on Machine Learning Research (09/2025)
Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Viliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukum Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025a. URL https:/ /tse:://arra /tt///aavv..gaabbaaee 25010 12948.
DeepSeck-AT AT, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Dong, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yilan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijum Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepscek--3 technical report, 2025b. URL https: //arxiv.///iivee/aa org/abs//20/ 19437.
Xianzhi Du, Tom Gunter, Xiang Kong, Mark Tee, Zirui Wang, Aonan Zhang, Nan Du, and Ruoming Pang. Revisiting moe and dense speed-accuracy comparisons for LLM training. CoRR, 0bs/22403.1055 2024. URL
4805008455///a 2405. 15052.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Tourct, Chunyang WI, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Tliaan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Tvan
14

---

Published in Transactions on Machine Learning Research (09/2025)
Evtimov, Jade Copet, Jacwon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.2178. 2024. URL https://doi.on orgg 10. 4w.444/85.i..222 2407. .27883.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:3, 2022. URL http:
//jmlr. ///jjir.oorgpaprr///eees 2ttppaperrs/v23/2--088 htmmm..mm.9...1.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4): 128 135, 1999. ISSN 13646613. doi: 10.0016/113644-66130990002223330 URL
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient Sparse Training with cesture-of-xxpette Proceedings of Machine Learning and Systems, 5, 2023.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mchta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. ar preprint arXiv:2310. 16226, 2023. URL /ttp::://xxxvvvvxaeeeeee wttp.os////....e.eeeee 16226.
Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker, and Ahmet Üstün. Nexus: Specialization meets adaptability for efficiently training mixture of experts, 2024. URL
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Authony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pro-training of large language models: How re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. URL https://openreeview net nett/torum/1drrgPPUlon
Xu Owen He. Mixture of A million experts. CoRR, abs/2407.04153,, 2024. URL https://bbi.org/100 48550/arXiv 2000.4404440 041553
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute--opimmll large language models. CoRR, abs/2203. 255522550011566,.. 2022. URL https://doi.orge
48550/45555/00 2203. 15556.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothóe Lesort, and Irina Rish. Simple and scalable strategics to continually pre-train large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: /ttpss:/opeeee opwnrevieneeeeeee// net//forum?1d=DLmPeeCC.
Robert A. Jacobs, Michael 1. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991. URL https://doi. org/oogg///o 10. 1162/neco.. 1991.3.1.79.
Paul Janson, Vaibhav Singh, Paria Mehrbod, Adam Ibrahim, Irina Rish, Eugene Belilovsky, and Benjamin Thérien. Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre- training, arXiv preprint wwXi::2533008//4 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandcep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibant. Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, //////201.000000 2024. URL. https://doi. org/10 10. 28550/arX4t 2401.04088.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); pp. 2567-2577, 2019.
Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervisod challenge dataset for reading comprehension. In Regina Barzilay and Min-Yer Kan (eds.), Proceedings of the 55th Annual Mecting of the Association for Compulational Linguistics (Volume 1: Long Papers); pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.18653/v1/P17-110 URL https://aclanthhlogg http://cccaaabboooooooo//771111
Paul Knopp and Richard Sinkhorn. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21 (2):3343 - 348, 1967.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Damitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: TB of permissively licensed source code. Trans. Mach. Learn. Res., 2023, 2023. URL https:///ppnnrevier rett/foorummiddxpppadas..
Aran Komatsuzaki, Joan Puigcerver, James Lec-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experte. from dense checkpoints. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreviiee nett/forurPicc-TTtUU00MMaa
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikm,, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, TCLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nes 2021. URL https: /
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang (ods.), Proceedings of the 88th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Evenl, volume 139 mlr.
of Proceedings of Machine Learning Research, pp. 6265 6274, PMLR, 2021. URL http://proceedings. htmml.
Liyuan Liu, Jianfeng Gao, and Weizhu Chen. Sparse backpropagation for moe training. CoRR, abs abs/231 0bss2330.00811, 2023a. URL https://doi org/10 1ttg/10../ 48550/arXiv..... 2310.00811......
Liyuan Liu, Young Jin Kim, Shuohang Wang, Chen Liang, Yelong Shen, Hao Cheng, Xiaodong Liu, Masahiro Tanaka, Xiaoxia Wu, Wenxiang Hu, Vishrav Chaudhary, Zeqi Lin, Chengruidong Zhang, Jilong Xue, Hany Awadalla, Jianfeng Gao, and Weizhu Chen. GRIN: gradient-informed moe. CoRR, abs/2409.15 12136, 2024. URL https: /ttp:////ooo///...// 48550/arXiv. 2409. 12136.
Zeyu Liu, Tim Dettmers, Xi Lin, Veselin Stoyanov, and Xian Li. Towards A unified view of sparse feed-forward network in pretraining large language model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, FMNLP 2023, Singapore, December 6-10, 2023, pp. 15038 15061. Association for Computational Linguistics, 2023b. URL https://doi.owg 10. 18653/v1/2023. .0t.022..mmmlp....mmmmmm..pp emnlp-maie catpppmmin.990...
David Lopez-Paz and Marc' Aurclio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annuua Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6467-6476, 2017. URL http::///pooeeedgg.nnnnurtee www.mm/ps..ccpppp///01..... cctpapprr/21////aaa. 2875277882222277766722777777 Absbractt..r....aa html.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Ila Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, TCLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net 2017. URL https://openrevierr http:////ppaereeeeeeeee/oorrrr/ mwt//toruun1ccsks$88axxx
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Cicbiera, Krystian Król, Tomasz Odrzygózdz, Piotr Sankowski, Marek Cygan, and Scbastian Jaszczur. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview net, 2024. URL https:
Sanket Vaibhav Mchta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. http:///lr..rrppppeee///////////
J. Mach. Learn. Res., 24:214:1 -1::1-00, 2023. URL
html.
Seyed-Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Görür, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Ma- chine Learning Research, pp. 15699 15717. PMLR, 2022. URL httpss/////oceeengs.... mlr press/v162/ mirzadeh22a. html.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experls language models. CoRR, abs/2449.002063 2024. CRL attps://dol org/10. 48550/arXiv. .200000
Ashwinee Panda, Vatsal Baherwani, Zain Sarwar, Benjamin Thérien, Stephen Rawls, Sambit Sahu, Supriyo Chakraborty, and Tom Goldstein. Dense backpropagation improves routing for sparscly-gated mixture- of-experts. In Workshop on Machine Leurning and Compression, NeurIPS 2024, 2024. URL https: // //openreview net met//forumiid-gg86TL..
Jupinder Parmar, Sanjeev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don't retrain: recipe for continued pretraining of language models. CoRR, abs/2407.07266 2024. URL
https://doi.. oropei..rr/1// 48550/8rKKK 2407. 072663.
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data
at scale. CoRR, abs/24 00s//200.115500 2024. URL attps://dol. http:::/ddoooo////////// org/10 0////0.485500 htt0///xxv..2.0..... 17557.
Björn Plister. GermanBenchmark
German Benchmark Datasots, 2023.
URL https://github.com hww.tauu..mmm///e/e//eeee coo/b//////////////////////////e
Jack W. Rae, Katie Millican, Siddhant M. Jayakumar, David Menick, Asya Berglund, Tom Hennigan, Roman Ring, Mandy Korpusik, Matthew Hechtman, Jacob Hilton, John S. Garcia, James Norman, Sasha Borgeaud, Trevor Cai, Jordan Hoffmann, Katarzyna Krawezyk, Arthur Mensch, Thomas Scialom, Eric Alford, Jordan D. L. Ho, Daniel Hesslow, Thomas Gunter, Jason Phang, Beren Millidge, Fan Yang, Marie-Anne Lachaux, Lorrayne de Souza Schmerling, Nat McAleese, Heidy Khlaaf, Simon Osindero, Oriol Vinyals, Karol Hausman, Laurent Sifre, Andrew M. Dai, Geoffrey Irving, Michael C. Mozer, Jeff Dean, and Koray Kavukeuoglu. Scaling language models: Methods, analysis & insights from training gopher. arXiv abs//ba2//1 11446.
preprinl ar Xrx:222112. 222 11446, 2021. URL /ttpp://arxxv...
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations loward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T. Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. URL. https://doi.ore 111.:/00..11///111 1019///41///00.../ 20200.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture--f-experts inference and training to power next-generation Al scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Ballimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. mll. htt.:ggwww..........eeeeemmeee prress/v162/rajjhandaae
18346. PMLR, 2022. URL
html.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on calastrophic lorgetting in neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReviow.net, 2022. URL attps://openrre wttppprrrrrio...otooouuuuu..et net/forum?id ChVS8_yPeEs
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Mare' Aurelio Ranzalo, Alina Beygekiime, Yanu N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 17555-17566, 2021. URL https: //proceedings.nen 220 /tt///////a///eeee00000000002200 -aa.aaataeee0tA-a-aaata.a. html.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial
winograd schema challenge at scale, 2019.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6107-61 6122, 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, TCLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nee net, 2017. URL http:///ppppaeeeeeeeeeeeee/ooaaa wttp:///ooooom//ddi/loooooooo
Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1m 211.4///axxx..224..0774444
dollars. CoRR, hbss/2444.07413 2024. URL https://doi.org
Shane Storks, Qiaozi Gao, and Joyce Yue Chai. Recent advances in natural language inference: A survey of benchmarks, resources, and approaches. arXiv: Computation and Language, 2019. URL https:
2133113350
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, and Xian Li. Branch-train-mix: Mixing expert 1lms into a mixture- of-experts LLM. CoRR, abs/2403.078% 07866, 2024. URL attps://dof http:////oooo/////////// wrgg1..48500//e. 48550/4555.. /ariiv.r/v..... 24033. 07816.
Qwen Team. Qwen1.5-moc. Matching 7b model performance with 1/3 activated parameters', February 2024. htttuuu...bbooo//e/ooooooooooolt
URL https:///wennnn
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. strategy for mixture-of-experts. CoRR, has/2400.15666 2024a. URL Auxiliary-loss-fre load balancing 2408. 15664.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pallern Anal. Mach. Intell., 46(8):5362 -5362-5333, 2024b. URL https:///oi.c org/o0.../// 10. 1109/TPAMI..... Ptt.//TAAM............... 3367329.
Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y. Wu. Let the expert stick to his last: Expert-speccaliized fine-tuning for spanse architectural large language models. CoRR, abs/ 240/00900009 2024c. URL https://doi. org/10. 10.1000...1111 48550/arXiv. 019006.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun fun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Longuage Processing, Brussels, Belgium, October 31- November 4, 2018, pp. 93- 104. Association for Computational Linguisties, 2018.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019.
Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistios. URL https: //aclanthology... org/2022. emnlp-main.271 278.
Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. Doepep: efficient expert-parallel communication library. attps://github.com deepseek-ai/DeepEP, 2025.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y. Zhao, Andrew M. Dai, Zhifong Chen, Quoc V. Le, and James Laudon. Mixture-of-experts with expert choice routing. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Newral Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL /ttp://papers.a nips. cc/paper_ hlles/paper/2022/haatt http://222222/2220.0/00001222210 o Custract-Conferenaae html.
Tong Zhu, Xiaoye Qu, Daize Dong, Jincheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: 2024. URL
Building mixtuure-of-experts from llama with continual pre-training, CoRR,
https://doi.org/ 2at5:/00088555//3333 arr.v./riiiv.. 2206. 26555.........66....
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moc: Designing stable and transferable sparse expert models. CoRR, 2022. URL https:
//arxiv. //arxiv.org/ablllll htt.vv..rrggbbb/22000896.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Contents
1 Introduction
1
2 Background
2
3 Related work
3
4 Method & Empirical Study
4
4.1 Selected architectures for our study
4
4.2
Continual pre-training strategy and Datasets
4
4.3
Training details
4.4
Maximum Routing Imbalance: A proxy for worst-case latency in MoEs
Results
6
5.1
Ablating replay (%) and the checkpoint used for CPT
5.2
Language modeling performance
5.3
Analyzing changes in routing behaviour due to CPT
10
6
Conclusion
11
A Extended Background
21
A.1 Continual pre-training of LLMs
21
A.2 Mixture of experts transformer language models
21
B Extended related work
22
B.1 Mixture of experts language models
22
B.2 Continual pre-training of dense foundation models
23
B.3 Continual pre-training of MoE LLMs.
23
C Training timings
24
D Extended experimental results
25
D.1 Language model evaluation benchmarks
25
D.2 Training and validation loss
28
D.3 Qualitative analysis
................................
31
D.3.1
Continual routing saturation analysis
31 ................................
D.3.2
Continual vocabulary specialization analysis
34
D.3.3
Continual expert co-activation analysis.
38
D.3.4
Continual routing imbalance analysis..
42
D.4 Maximum routing imbalance of MoEs during continual pre-training.
45
20

---

Published in Transactions on Machine Learning Research (09/2025)
E Dataset sizes and sampling proportions
50
F Model hyperparameters
51
A Extended Background
The following section is complementary to section 2 of the main manuscript, providing additional background for our paper.
A.1 Continual pre-training of LLMs
Contimual pre-training extends pre-training to multiple new distributions. Concretely, continual pre-training occurs when a models is trained on a sequence of datasets Do, Di...., DN with different distributions, N 2 2, and each dataset sufficiently large (e.g., > 100B tokens in the case of language) (Ibrahim et al., 2024). Note that the large data scale, here, distinguishes this setting from supervised fine-tuning or instruction tuning where the amount of data is much smaller. Typical application settings of continual pre-training are adapting existing pre-trained models on newly available data or enhancing their capabilities in a specific domain. We will now discuss well-established techniques for continually pre-training dense transformers.
LR Re-warning and Re-decaying. Many open-source LLMs follow a linear warmup and cosine annealing schedule during pre-training, which reaches a large maximum learning rale, 1mmar early on in training and subsequently decays the learning rate to a small minimum value, Tmmá (Hoffmann et al., 2022; Loshchilov & Hutter, 2017; Rae et al., 2021). Naively continuing training at 1mmn Mmaz either leads to too little adapiation or too much forgetting. Instead, Tbrahim et al. (2024) show that Re-warming and Re-decaying the learning rate during CPT is critical for strong continual learning performance.
Infinite LR schedules. While Re-warming and Re-decaying the learning rate following a cosine decay schedule was found to be a good solution when starting from a fully decayed checkpoint, Ibrahim et al. (2024) remark that this stratogy incurs forgetting, due the large LR increase, even when continually pre-training on the same distribution. To circumvent this, the authors propose infinite learning rate schedules that allow for smooth transition in learning rate between continual learning phases and are not bound to fixed number of training steps. These techniques were subsequently confirmed to work well in settings with multiple distribution shifts (Janson et al., 2025).
Replay. Replaying previous data has been a longstanding tool for mitigating catastrophic forgetting (Wang et al., 2024b). In our experiments, we replay previously seen data for this purpose, designating any model using the technique with the suffix " %X% Replay'. Here, X represents the percentage of samples in a given batch that were replayed from the previous distribution. To match compute across different replay budgets, we do not increase the token budget when increasing the amount of replay. Instead, we decrease the amount of new data seen during CPT.
A.2 Mixture of experts transformer language models
Sparsely-activated MoE transformers differ from their dense counterparts by dynamically routing tokens in sequence, X E R to different experts (-)) as opposed to single FFN. Here S is the sequence length, H is the transformer's ' hidden dimension, j indexes over the transformer's blocks, and N is the number of experts per block. This is often referred to as an MoE layer (Shazeer et al., 2017). Typically, these layers are used in place of Feed Forward Networks (FFN) in each transformer block (Fedus al., 2022; Dai et al., 2024), however, recent works (Shen et al., 2024; Zhang et al., 2022) have also replaced the query and output matrices of multi-head self-attention layers with MoE layers. In what follows, we exclusively study MoE transformers that replace FFNs at each block with MoE layers, similarly to state-of-the-art recent work (Dai et al., 2024; Team, 2024; Muennighoff et al., 2024; DeepSock-AT et al., 2025b;a). Morcover, we also study the recent trend of using more granular experts and shared experts (Dai et al., 2024; Team, 2024; MuennighofT et. al., 2024; He, 2024; Liu et al., 2023b; Ludziejewski et al., 2024; Rajbhandari et al., 2022; DeepSeek-A Al et al., 2025b;a).
Algorithms for dynamically selecting among experts, known as routing algorithms (Roller et al., 2021; Shazeer et al., 2017; Zoph et al., 2022; Clark et al., 2022; Lewis et al., 2021), are central to MoFs. A key consideration
21

---

Published in Transactions on Machine Learning Research (09/2025)
for token-choice (we do not consider expert-choice as it is incompatible with autoregressive generation) routing algorithms is achieving a balanced load across experts in a given layer. Without enforcing a balanced load, the router may collapse to only choosing a single or a few experts, leading to poor parameter utilization and higher latency proportional to the load of the most burdened expert (Zhou et al., 2022).
In this work, we focus on studying two prominent Top-k routing algorithms from recent state-of-the-art works, which we refer to as Penalty-Balanced Top-k (PBTk) routing (Shazeer al., 2017; Dai et al., 2024; Zoph et al., 2022; Fedus et al., 2022) and Sinkhorn-Bllaneed Top-k (SBTk) routing (Clark et al., 2022; Anthony et al., 2024). Both algorithms define the router R(x) Wz E + I to be a simple linear projection to the space of experts. Expert probabilities are computed by applying softmax to the router's output: p(r) softmax( (B(())))) Where SB()) is the Sinkhorn load balancing function in the case of SBTk routing or the identity otherwise.
Ranked by p(z), the top-k experts are selected for each token, where h is a hyperparameter selected before training. For a single token, the output of MoE layer j, SMOR, is computed as follows:
fMOE, (x) = (F) + EiEN, Pi(x) (F)
Eier(e)Pi(e Pi(2)
Where [x(x)= = {il (x() E Top elements of p(z)} and SFFN is a shared expert (Rajbhandari et al., 2022; Dai et al., 2024) if one is used or the identity otherwise. While they both route tokens to the top-k experts, the PBTk and SBTk routing algorithms differ in how they balance the load across experts.
Penalty-Balanced Top-k Routing. PBTk methods in the literature (Shazeer et al., 2017; Fedus al., 2022; Zoph et al., 2022; Dai et al., 2024) add penalty terms to the overall loss to encourage a balanced load across experts. The auziliary loss has become the most popular such penalty and is used in conjunction with the æ-loss in several recent stale-of-the-arl MoEs (Dai et al., 2024; Team, 2024). Brielly, the auxiliary loss is minimized when the router assigns an equal proportion of tokens in a given batch to each expert in a given block, while the z-loss penalizes Barge-mannitude router logits. The latter has been shown to promote numerical stability in larger models (Zoph et al., 2022). Given their combination in recent SOTA MoF LLMs (Zoph et al., 2022; Dai et al., 2024), we exclusively study MoEs that combine both auziliary loss and 2-loss, referring to them as PBT6 MoEs.
Sinkhorn-Balanced Top-k Routing. SBTk routing casts the assignment of tokens to experts as linear assignment problem which corresponds to a well-studied problem in optimal transport, namely 'the regularized Kantorovich problem of optimal transport' (Clark et al., 2022). The Sinkhoru-knopp algorithm (Knopp & Sinkhorn, 1967) provides an approximate solution to this problem, which can be efficiently computed on GPUs In practice, this corresponds to adjusting routing probabilities (e.g, according to SB(-), see (Clark et al., 2022) section B.2.1 for details) such that relatively balanced load is obtained without deviating too much from greedy Top-k routing.
B Extended related work
The following section is complementary to section of the main manuscript, providing a more comprehensive summary of the related work.
B.1 Mixture of experts language models
Mixture of experts language models have a long history with fundamental ideas dating back several decades (Collobert et al., 2003; Jacobs et al., 1991). More recently, in the context of large-scale lan- guage modeling, the miixture-ofeexperts layer (Shazeer et al., 2017) was introduced to substantially increase the capacity of an LSTM language model with little detriment to efficicncy. The authors also introduced a load-balancing penalty to encourage even utilization of experts. Subsequently, Fedus et al. (2022) refined the penalty, renamed auziliary loss, which has become a central component of modern MoEs. Follow-up works have focused on massively scaling up MoE LLMs, improving the routing algorithms of these models, improving the quality of the router's gradient estimate, and making architectural improvements to these models. Lepikhin et al. (2021) introduce the MoF layer into the transformer architecture, using two activated
22

---

Published in Transactions on Machine Learning Research (09/2025)
experts (thought to be necessary for nontrivial router gradients) and scaling to umprecedented 600B parameter scale. Subsequently, Fedus et al. (2022) introduced the Switch Transformer, showing that it is possible to scale MoE' beyond 1T parameters despite training them with only a single active expert.
Other works have focused on developing novel routing algorithms. Lewis et al. (2021) cast routing as a linear assignment problem and leverage Hungarian matching in their routing algorithm. Clark et al. (2022) use Sinkhorn's algorithm to approximately solve the assigument problem on GPUs, resulting in a faster algorithm. Anthony et al. (2024) introduce a favorable initial condition to improve the convergence of the iterative Sinkhorn solver, further reducing the cost of Sinkhorn routing. Roller et al. (2021) introduces deterministic routing algorithm based on hash layers. Zoph et al. (2022) introduces loss penalty to promote stability in large-scale MoE routing. Wang et al. (2024a) introduces the first learned routing mechanism that uses neither an entropy regularizer nor an assignment- based approach to balance expert utilization in token-choice routing. Zhou et al. (2022) introduce Expert Choice Routing, a routing paradigm where each expert receives a balanced load and the routing algorithm decides which tokens to send to each of the experts; while it obtains strong performance and automatical ically achieves a balanced load, ECR is incompatible with autoregressive generation so we don't consider it in this work. Other works propose methods to better approximate the full router gradient (Panda et al., 2024; Liu et al., 2024; 2023a).
Finally, a recent, trend of using MoE experts with finer-grained intermediate sizes has shown notable performance gains when compared to using the full intermediate FFN size, as was originally done (Shazeer et al., 2017; Fedus et al., 2022; Lepikhin et al., 2021). Liu et al. (2023b) first observed that utilizing smaller expert layers improves perplexity. Subsequently, researchers have explored scaling laws for fine-graine ine-grained MoEs at small scale (Ludziejewski et al., 2024), pre-trained and released SOTA MoEs that leverage the fine-grained expert architecture (Dai et al., 2024; Team, 2024; Muennighoff et al., 2024), and pushed the idea of thinner experts to its limit, exploring MoEs with millions of experts (He, 2024). While we have reviewed the most relevant works to ours, there are many more works that we have not had the chance to mention here. We refer the avid reader to a recent and comprehensive survey of the area (Cai et al., 2024).
B.2 Continual pre-training of dense foundation models
Continual pre-training of foundation models has the same objectives as continual learning (French, 1999), except that it is applied to large-scale pre-training tasks, which are mainly superveperv..dd. Several existing works study continual learning in settings relevant to continual pre-training- They find that self-supervised pre-training benefits from reduced forgetting (Cossu et al., 2022: Davari et al., 2022), that pre-trained models forget. less than their randomly initialized counterparts (Mehta et al., 2023), that forgetting improves model scale is increased (Ramasesh et al., 2022), and that wider models tend to forget less than deeper models (Mirzadch et al., 2022). In the context of LLM fine-tuning, (Scialom et al., 2022) shows that little replay is needed to prevent forgetting when fine-tuning on small amounts of instruction-tunie context of large-scale (with respect to data) continual pre-training for LLMs, Gupta et al. (2023) highlights the importance of rewarming the learning rate when models are pre-trained from a checkpoint that has been decayed to a small learning rate. Following up on their work, Ibrahim et al. (2024) establish the effectiveness of learning rate re-warming, LRR re-decaying, and replay for large-scale continual pre-training of LLMs. Concurrently, Garg et al. (2023) establishes the performance of the same techniques for CLIP models. Shortly thereafter, Parmar et al. (2024) scale continual pre-training for dense decoder-only transformers further, showing that a 15B parameter model pre-trained for 8T tokens can be effectively pre-trained on 1T tokens of incoming data.
data. In the
B.3 Continual pre-training of MoE LLMs.
To the best of our knowledge, only single work exists exploring the large-scale continual pre-training of MoEs LLMs, while the majority of the literature focuses on upcycling or growing MoEs for continual pre-training.
In a concurrent pre-print DeepSeck-Coden V2 (DeepSeck-AT et al., 2024), shows that they can continue from a checkpoint the training of a MoE LLM. However, this only shown for one instance and the analysis of the MoE routing behavior is not discussed. Furthermore, there is comparison to FL.OP-matched dense
23

---

Published in Transactions on Machine Learning Research (09/2025)
model, making it challenging to assess whether the sample efficiency of MoE LLMs is maintained during continual pre-training.
Continual pre-training methods for MoEs that are less related to our work generally focus on fine-tuning MoE LLMs on small amounts of data (Wang et al., 2024c) or growing MoEs (Komatsuzaki et al., 2023; Zhu et al., 2024; Sukhbaatar et al., 2024; Gritsch et al., 2024; Chen et al., 2023). Wang et al. (2024c) study MoF-specific techniques for parameter-effiient fine-tuning (PEFT). Zhu et al. (2024) proposes technique create an MoE by splitting the FFNs of an existing dense transformer and subscquently continually pre-training it. Sukhbaatar et al. (2024) proposes to continually pre-train dense LLM multiple different datasets, gather the FFN layers from different continually pre-trained models to form MoE layers, merge the parameter tensors other than FFN layers, and subsequently continually pre-train the menged model to learn routing in the MoE part. Gritsch et al. (2024) propose a similar method to train new expert layers that uses domain embeddings from a pro-trained embedding model as the identifier for a domain's experts, allowing the domain embeddings to provide an inductive bias that can help with adding new exports. While these methods allow for improving the capabilities of MoEs with new data, they focus on first upcycling dense models, whereas we focus on updating MoEs pre-trained from scratch.
C Training timings
In the interest of completeness, we provide training timings for each model architecture in our study. We would like to preface this section with the following disclaimer: all the step times that we report our study are specific to our code and the libraries that we use, but are not reflective of the best performance achievable. Timings are subject to vary based on model size, interconnect speed, training precision, accelerator used, implementation, etc. With this in mind, in table 2, we report the mean and standard deviation timing in milliseconds (ms) of different operations across 1000 training steps on 64 A100 GPUs. Specifically, we time the forward pass, backward pass, optimizer step, and dataloading time. An aggregate time is reported in the last column. We also report the MFU achieved in TFLOPs and report the throughput in samples per second. All our experiments use code from the GPT-NeoX library (Audonian et al., 2023) and leverage the megablox grouped GEMM kernel' (Gale et al., 2023). We observe that, in our specific implementation, all MoEs take approximately twice as long per step as the dense model, that granular MoEs have slower forward and backward times compared to Switch MoEs, and that Sinkhorn-balanced MoEs have slower forward and backward limes compared to Penally-balanced MoEs. This reveals some non-negligible downsides (e.g. reflected in step times) to training MoEs compared to a FLOP-matched dense model, including: higher memory, longer optimizer step, and higher communication costs. Despite these drawbacks, Du et al. (2024) find that the performance benefits of MoEs still exceed those of dense models, even when accounting for the additional overhead when training MoEs.
Table 2: Time of different operations during pre-training for each model architecture in our study. We report the mean and standard deviation timing in milliseconds (ms) of different operations across 1000 training steps on 64 A100 GPUs. We use a global batch size of 1024 and a sequence length of 2048. Specifically, we time the forward pass, backward pass, optimizer step, and dataloading time. An aggregate time is reported in the last column. We also report the MFC achieved in TFLOPs and report the throughput in samples per second. We observe that, in our specific implementation, all MoEs take approximately twice as long per step as the dense model, that granular MoEs have slower forward and backward times compared to Switch MoEs, and that Sinkhorn-balanced MoEs have slower forward and backward times compared to Penalty-balanced MoEs.
Model
Forward
Backward
Optimizer
Data
Samples/Sooe
MFU(TFLOPS)
Total Time(ms)
Dense Baseline
318.02 1 1.86
518.60 1 17.73
39.48 1: 5.55
3.72 1 0.44
1156.88 1 52.63
111.32 1: 5.06
879.83
PB Switch MoE
119.36 = 11.97
963.53 + 1.06
101.66 + 0.69
2.12
- 0.33
671.93 + 21.63
86.21 + 3.16
1516.97
SB Switch MoE
494.28 = 11.35
1012.38
+ 6.87
101.58
: 0.70
2.59
: 1.36
631.14
: 33.90
80.97
+ 4.35
1610.83
PB Grauular MoE.
485.05 1 13.66
1091.38
3.68
100.66
1 3.08
2.42
1 0.33
606.85
21.53
77.86
1 2.76
1679.50
SB Granular MoF.
541.62 7.14
5.72 1144.85
100.27
1.47
2.37
0.33
569.72 19.27
73.09
1789.10
24

---

Published in Transactions on Machine Learning Research (09/2025)
D Extended experimental results
In the following section, we provide extended experimental results from the paper in non-summarized format to enhance the reproducibility of our manuscript and allow the reader to dive into whichever details may most interest them.
D.1 Language model evaluation benchmarks
We evaluate the language models in our study on English, Code, and German evaluation tasks. Please note that our goal is not to achiove SOTA performance on these benchmarks; none our models have been aligned or fine-tuned to improve performance. Instead, we seek to evaluate their performance within the context of our controlled scientific study. Given the scale our language models (at most 570M active parameters), we carefully select evaluation tasks that show non-trivial evaluation results; that is, choose tasks for which the models in our suite achieve above random chance accuracy.
Selected English evaluation tasks:
Commonsense Reasoning (0-shot): HellaSwag (Zellers al., 2019), Winogrande (Sakaguchi et al., 2019), PIQA (Bisk et al., 2019), ARC-Easy, ARC-Challenge (Clark et al., 2018), SWAG (Zellers et al., 2018)
Reading Comprehension (0-shot): LAMBADA (OpenAI) (Storks et al., 2019)
Scientific Question Answering (O-shot): SciQ (Johannes Welbl, 2017), PubMedQA (Jin et al.,
2019)
Math (0-shot): MathQA (Amini al., 2019)
Selected German evaluation tasks translated from the corresponding English language tasks using the GPT 3.5 APT (Plüster, 2023).
Commonsense Reasoning (0-shot): KellaSwag-DE (Zellers et al., 2019), ARC-Challenge-DD (Clark et al., 2018)
Reading Comprehension (O-shot): TruthfulQA-DE (Joshi et al., 2017)
Code evaluation tasks
Python: Human Eval (pass@1-200)
Tables 4, 3, and report the performance models in our study on English, Code, and German evaluation benchmarks.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Human Eval after pre-training Fin and continual pre-training on Stack. We report the percentage of problems for which at least one generated solution passes all tests. We observe that all English-only models generate only incorrect solutions, while the models continually pre-trained on code and the full ro-training basclines achieve non-trivial accuracy. Intcrestingly, the SB Switch MoE performs best of all across all pass thresholds. However, given the generally poor performance of the models overall, we attribute differences within dataset type random chance.
Training Tokens
Model
pass 1
pass@10
pass050
pass 100
pass 150
pass@200
Mean
Dense Baseline
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
SB Switch MoE
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
400B FineWeb
PB Switch MoE
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
SB Granular MoE
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
PB Gramular MoF
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
Dense Baseline
0.21%
1.94%
6.87%
9.96%
1188%
13.41%
7.37%
SB Switch MoE
0.24%
2.19%
8.06%
12.15%
14.85%
17.07%
9.09%
400B FineWeb 200B Stack
PB Switch MoE
0.21%
1.93%
7.28%
11.10%
13.56%
15.24%
8.22%
SB Granular MoE
0.18%
1.69%
6.50%
10.01%
12.30%
14.02%
7.45%
PB Gramular MoE
0.16%
1.51%
6.30%
10.40%
13.24%
15.24%
7.81%
Dense Baseline
0.14%
1.20%
3.57%
4.99%
5.98%
6.71%
3,76%
400B FineWeb J 200B Stack
PB Gramular MoE
0.20%
1.82%
6.84%
10.21%
12.13%
13.41%
7.44%
Table 4: Languge models evaluation benchmarks after pre-training (English Web Data), continual pre-training (Code & German Web Data), and full re-training. We report accuracy for all selected benchmarks. We observe that all MoFs and the dense baselines maintain similar relative performance before and after the distribution shift, showing that MoE LLMs' continual pre-training dynamics are similar to dense models with respect to forgetting on evaluation tasks. When comparing the mean evaluation performance of PB granular MoE to the full re- re-training baseline, we observe that the final performance is matched or very nearly matched with a substantially smaller computational cost.
Traleiing Thilos
Model
ARCSCC
ARC-E
HollaBang
LAMIADA OAT
MathqAA
PuhAiiiq.
S:QQ
SWAG
WhreGGradde
Mron
23.355
51.237
38.25%
97 77.00%
25.325
31.885
51.097
K2.70%
47.35%
35.05%
SB Switch MaB
26.7895
45.50%
03.40%
24.05%
71.10%
03.0075
81.90%
$0.3155
10.57%
14.
Jon Fiuetil (Anmnald))
PR Switch
75%5
16.10%
51.91%
865
11.37%
55-1995
67:30%
SUVET5
57.00%
56 15%
Geumiar
25.119%
45.10%
20%
26.00%
5432%
49.30%
$1.15%
60.075
05.7176
Donec Boeelue 28.3505
02.02.55
30.125
028
56 107
70.00%
02.3 5005
01.90%
47.00%5
54.10%
40 18%
24.12/5
41.15%
125
24.525
75.125
K3.3670
0.32%%
anB Fuelish (Nou Amnoiad)
Swcteds 25.541%
41.55%
25.15f%
75.35fl
45.86%
56.277
McB
74.00%
68.645
[4
Dvrec Bonelie
25.575
51.4335
36 375
05.70 7088
56.705 705
70.18%
45 53.55
80.70
17.575
54.59%
L8 10%
WMB Faselids 20B8 Gerzun (40%% Reylas]
SB Swotch Mal
FB Switdd Miak
24.835
25.345
00.003
43.55%
42.25%
00 4076
52
24 24.22%
25.075
13.25%
13.00%
43.40%
07-4095
54.20%
K3.3030
45.25%
45.35%
154155
533155
(1 2555
51 50.0
Sis Grareter
4305
11.375
150%
105%
73.11%
53:
01.000
19.89%
(0 40%
10
PB Gennaka McE
27.055%
44.56%
04.4570
24.00%
73.56%
53,4355
65.70%
45.70%
07.228
JMB FaseWids J 20B Genmua CC
Deree Eowebuse
PB Gronaie Mol
23, 23-25
27.6095
00.055
45,355
38.75%
20.
1777
25, 107
25, 11%
701155
73.507
05. 23.2005
26. 20-55
N0. 90.
13.1%
40.5555
52315%
10.10%
(3 941
10 1154
MaE
56.55%
01.
FaeeWid 200B Stock Reylay)
PB Switcds Madk
25,2555
00
78.107
86
45/0755
52.05%
52
SG Geammbrr Malt
25.34%
19.1993
41.37%
24, 35%
15.005
03.0395
09.9633
40.005
124 13.815
PB Grandie Molt
25. 4305
00 07.35
11 11.5555
51 9118
55. 2075
72.78%
33.8095
88 05.00055
69-6255
13 TIA
FlaeWed 200t Stack
PB Geatnibe Malt
Buclac
27.8255
6776773
13.1
56.385
25, 0055
15.335
50.60%
No. 40%%
50.0555
54 11/4
40.07%
26

---

Published in Transactions on Machine Learning Research (09/202))
Table 5: German Language models evaluation benchmarks after pre-training (English Web Data), continual pre-training (Code & German Web Data), and full re-training. We report accuracy for all selected benchmarks. We observe that all MoEs and the dense baseline improve performance German after continual pre-training. When comparing the full re-training baselines, observe that the average performance of our continually pre-trained models are on par.
Training Tokens
Model
Arc-C DE
Hellaswag DE
TruthfulQA DE (MC1) Mean
Dense Baseline
18.52%
26.78%
25.34%
23.54%
SB Switch MoE
18.60%
26.87%
23.87%
23.11%
400B FineWeb
PB Switch MoE
18.94%
26.56%
24.60%
23.37%
SB Granular MoE
18.34%
26.78%
23.38%
22.83%
PB Granular MoE
18.43%
27.05%
24.72%
23.40%
Dense Baseline
19.28%
32.53%
23.99%
25.27%
400B FineWeb 200B German CC
SB Switch MoE
21.76%
35.74%
25.21%
27.57%
PB Switch MoE
20.73%
35.77%
23.01%
26.50%
SB Granular MoE
22.61%
36.90%
26.19%
28. 28.57%
PB Crauular MoE
22.70%
37 37.23%
23.01%
27.65%
Dense Baseline
20.05%
31.45%
2185%
25.45%
400B FineWeb 200B German CC
PB Granular MoE
21.33%
35.74%
25.70%
27.59%
27

---

Published in Transactions on Machine Learning Research (09/2025)
D.2 Training and validation loss
In the following sections, we present validation loss curves during and after pre-training and continual pre-training for all models in our study, Specifically, we report final validation loss in Table 6 and validation curves during training across figures 6, 7, and 8.
Table 6: Final validation loss of MoEs and dense model after pre-training (English Web Data) and continual pre-training (Code & German Web Data). As expected, we observe that all MoE transformers outperform the dense baselinc during pro-training and continual pertaining with respect to validation loss. Moreover, we observe that MoEs forget marginally less than their dense counterparts. Together, these results show the continual learning abilities of MoEs are on par with dense models in terms of adaptation and are slightly superior in terms of forgetting, possibly due to their larger total parameter count.
Final Validation Loss
Training Tokens
Model
Fine Web
Stack
German
Forgetting AVG
Dense Baseline
2.881
4.028
3.741
SB Switch MoE
2.711
3.861
3.495
100B FineWeb (non-annealed)
PB Switch MoE
2,099
3.872
3.451
SB Granular MoE
2.604
3,690
3.404
Cramuler MoE
2.653
3.715
3.370
Dense Baseline
2.825
4.028
3.741
SB Switch MoE
2,640
3,801
3.495
400B FineWeb (annealed)
PB Switch MoE
2.628
3.872
3.451
SB Cranular MoE
2.595
3.690
3.404
PB Cranular MoE
2.582
3,715
3.370
Donse Baseline
2.939
1.026
0.059
1.982
SB Switch MoE
2.757
0.944
0.046
1.850
400B FineWeb - 200B Stack 30% Replay
PB Switch MoE
2.749
0.945
0.050
1.847
SB Granular MoE
2,708
0.925
0.044
1.816
PB Granular MoE
2.699
0.924
0.046
1.811
Dense Baseline Union
2.866
1.050
1.958
400B FineWeb 200B Stack
PB Granular MoE Union
2.630
0.935
1.782
Deuse Baseline
4.028
1,279
1.399
2.654
SB Switch MoE
3.810
1.180
400B FineWeb 2008 German 0% Replay
PB Switch MoE
3.782
1.059
1.152
2,420
SB Granular MoE
3.701
1.038
1,071
2.369
PB Granuler MoE
3.685
1.028
1.055
2.356
Dense Baseline
2.946
1.367
0.066
2.157
SB Switch MoE
2.749
1.142
0.039
1.946
400B FineWeb - 200B German 40% Replay
PB Switch MoE
2.741
1.129
0.042
L.935
Granular MoE
2.701
Cramuler MoE
2,600
0.037
Dense Baseline Union
2.938
1.390
2.161
100B FineWeb 200B German
PB Granular MoE Union
2.609
1.120
1.895
28

---

Published in Transactions on Machine Learning Research (09/2025)
Validation Loss FineWeb
4.00
Validation Loss German
Replay
1.8
Buselza 10% Suplay
Danse Baseline 40% Replay
5B Switch Mot: 07 Bapbyy
5B Switch MoE 10% Replay
3.50
1.6-
5B Switch Nall 40% Replar
3.25
Beeplay
Repbyy
1.4
5B Griuler Mell Replay
3.00
SB Grazular MaE 10% Reglay
5B Granulac MrE 40% Reglay
P9 Granular Mad Raplay
2.75
1.2-
Renuler Mell 10s
Beepaa
20
40
10
40
50
Training Tokens (B)
Training Tokens (B)
(a) FineWeb Validation Loss
(b) Cerman Validation Loss
Figure 6: Ponalty-Balanced (PB) and Sinkhorn-Balanced (SB) Top-k MoEs behave similarly to the FLOP-matched Dense baseline when being continually pre-trained with varying amounts replay. We continually pre-train MoEs and dense baseline using varying amounts replay: 0% (dotted curves), 10% (dashed curves), and 40% (full curves). We observe that replay substantially reduces forgetting for all models while slightly harming adaptation; that is, the effect of replay is the same for MoEs for dense models.
Validation Loss FineWeb
3.4
3.2
3.0
Dense Baseline
SB Switch MoE
2.8
PB Switch MoE
SB Granular MoE
2.6
PB Granular MoE
50
100
150
200
250
300
350
400
Training Tokens (B)
(a) Fine Web Training Loss
Figure 7: Validation loss during initial pre-training on Fine with Infinite LR schedules. We report decay and constant phases to completion. We observe that all MoE transformers stably decrease validation loss throughout pre-training, with MoEs improving over the dense model as expected. Interestingly, the PBTk MoEs shows incremental improvement over SBTk.
29

---

125 175 Published in Transactions on Machine Learning Research (09/202))
3.05
1.5
Dense Baseline
Validation Loss Fine Web
3.00
Validation Loss Stack
1.4
SB Switch MoE
PB Switch MoE
1.3-
SB Granular MoE
2.90
PB Granular MoE
2.85
2.80
2.75
2.70
0.9
2.65
0.8
2.60
50
100
150
175
150
175
200
Training Tokens (B)
Training Tokens (B)
(a) FineWeb Validation Loss
(b) Stack Validation Loss
3.05
1.7
Validation Loss Fine Web
3.00
Validation Loss German
1.6
2.95
2.90
1.5
2.85
1.4
2.80
1.3
2.75
2.70
2.65
2.60
25
50
75
100
125
150
175
200
50
200
Training Tokens (B)
Training Tokens (B)
(c) FineWeb Validation Loss
(d) German Validation Loss
Figure Validation Loss on CPT and PT datasets during CPT. Subfigures (a) and (c) report FineWeb validation loss, while subfigures (b) and (d) report Stack and German validation loss respectively for models trained those datasets. We observe that all MoFs maintain their sample efficiency after the distribution shift, reaching a lower loss in many fewer iterations than the FLoP-matched dense baseline.
30

---

Published in Transactions on Machine Learning Research (09/2025)
D.3 Qualitative analysis
In the following section, present new metrics for analyzing the routing decisions of MoEs in our study and interpret how they change during continual pre-training, To accomplish this, we take checkpoints before and after continual pro-training and record their routing decisions, loss, routing imbalance, and a number of different metrics on 20M tokens of FineWeb test data (pre-training dataset), 20M tokens German test data (continual pre-training dataset), and 20M tokens of Stack test data (continual pre-training dataset). Our results can be grouped into four main categories: 1) routing saturation analysis, 2) vocabulary specialization analysis, 3) expert co-activation analysis, and 4) routing imbalance analysis.
D.3.1 Continual routing saturation analysis
We adapt the analysis of router saturation from Muennighoff al. (2024) the continual setting. Note that will directly reproduce and slightly modify some lines from Muennighoff et al. (2024)'s definition of router saturation below for clarity and ease of passing from one paper's notation to the other. Concretely, we define continual Continual Router Saturation as:
c n 7
(Ta)
Continual Router Saturation(t,h.j)- 1
(2)
N
where:
Th and Tj: The tasks being considered when selecting checkpoints. Note that h S j. In our case, j E {0, 1, 2} , with To designating the pre-training task (FineWeb) and Ti, T2 designating German and Stack continual pre-training lasks, respectively. While our experiments only consider one transition, in general, there may be many more.
N: The total number of tokens in the dataset.
k: The number of experts activated per input token.
eTT The set of ls experts activated for the ith token at the final checkpoint of the hth task.
e(T); The set of l experts activated for the ith token at the final checkpoint of the jth task.
NE The number of common experts activated for the ith token between the final checkpoints laken from the hth task and jth task.
Figures and 10 consider h and j E {0, 1} for Gramular (31 routed experts, active, 1 shared) and Switch (8 routed experts, active) MoEs respectively, thus comparing the checkpoint before continual pre-training with the checkpoint obtained afterward. The subfigures layers for PBTı MoEs, while the subfigures on the left report the same for switch SBTk MoEs. Each row reports router saturation on a different dataset. We make the following observations:
the right report router saturation across model
(1) the first few layers are consistently among those with the lowest router saturation,
(2) router saturation is lower for checkpoints trained on a given task when is measured with respect to tokens from h, and
(3) the router saturation of models testod on their continual pro-training dataset seems to consistently decrease with small slope as the layers index increases.
Observation (1) suggests that the carly layers may undergo the most change during continual pre-training. Note that the trend of the first few layers having low router saturation is especially pronounced in subfigures (a) and (b), suggesting that most of the forgetting seen during continual pre-training may occur in the early layers. Observation (2) shows that MoEs change their routing decisions more to the distribution they are being trained on in the case of German and Stack, which is intuitive. Observation (3) suggests that layers closer to the final layer of the MoE must. change more adapt the new distribution.
31

---

Published in Transactions on Machine Learning Research (09/2025)
100 4
100
Router Saturation (%)
Router Saturation (%)
60
FineWeb
60
FineWeb
PB Granular MoE
SR Granular MoE
German 40% Replay Granular MoE
German 40% Replay Sn Granular MoE
40
40
German 0% Replay Cranular MoE
German 0% Replay Granular MoE
Stack 30% Replay PB Granular MoË
Stack 30% Replay SB Granular MoE
0
10
15
20
10
15
20
Layer
Layer
(a) FineWeb, Granular MoE
(b) FineWeb, Granular MoB
100
100
Router Saturation (%)
Router Saturation (%)
80
80
40
40
20
20
5
Layer
Layer
(c) German, PBTk Granular MoE
(d) German, SBTk Granular MoE
100
100
Router Saturation (%)
Router Saturation (%)
80
80
60
60
20
20
0
5
10
15
20
0
5
10
15
20
Layer
Layer
(c) Stack, PBTk Granular MoE
(I) Stack, SBTk Cranular MoE
Figure Router saturation at the beginning of continual pre-training for Granular MoEs. Subfigures (a,c,e) report layer-wise router saturation for PBTk MoEs, while subfigures (b,d,f) report router saturation for SBTk MoEs. (a) and (b) measure routing saturation with respect to test tokens from FineWeb, (c) and (d) measure router saturation with respect to test tokens from German, and (e) and (f) measure router saturation with respect to test tokens from Stack. We observe few trends: 1) the first few layers are consistently among those with the lower router saturation, 2) router saturation is consistently lower for checkpoints CPT on the testing distribution showing that these checkpoitns adapt more to that distribution, 3) the router sat uratiooo models lested on their inual pre-training datasel seems to consistently decrease with small slope as the layers index increases, and 4) the no-replay checkpoint consistently has lower router saturation than its 40% replay counterpart.
32

---

Published in Transactions on Machine Learning Research (09/2025)
100
100
Router Saturation (%)
Router Saturation (%)
60
FineWeb
60
FineWeb
PB Switch MoE.
SB Switch MoE
German 40% Replay PB Switch MoF.
German 40% Replay SB Switch MoF
40
40
German 0% Replay Switch MoE
German 0% Replay Switch MoE
Stack 30% Replay PB Switch McE
Stack 30% Replay SB Switch MoE
10
15
20
0
10
15
20
Layer
Layer
(a) Feb, PBTA Switch MoE
(b) FineWeb, SBT& Switch MoE
100
100
Router Saturation (%)
Router Saturation (%)
80
80
40
40
20
20
5
Layer
Layer
(c) German, PBTk Switch MoE
(d) German, SBTk Switch MoE
100
100
Router Saturation (%)
Router Saturation (%)
80
80
60
60
20
20
0
5
10
15
20
5
10
15
20
Layer
Layer
(c) Stack, PBTk Switch MoE
(0) Stack, SBTk Switch MoB
Figure 10: Router saturation the beginning of continual pre-training for Switch MoEs. Subfigures (a,c,e) report layer-wise router saturation for PBTk MoEs, while subfigures (b,d,f) report router saturation for SBTk MoEs. (a) and (b) measure routing saturation with respect to test tokens from FineWeb, (c) and (d) measure router saturation with respect to test tokens from German, and (e) and (f) measure router saturation with respect to test tokens from Stack. We observe few trends: 1) the first few layers are consistently among those with the lower router saturation, 2) router saturation is consistently lower for checkpoints CPT on the testing distribution showing that these checkpoitns adapt more to that distribution, 3) the router sat uratiooo models lested on their inual pre-training datasel seems to consistently decrease with small slope as the layers index increases, and 4) the no-replay checkpoint consistently has lower router saturation than its 40% replay counterpart.
33

---

Published in Transactions on Machine Learning Research (09/2025)
D.3.2 Continual vocabulary specialization analysis
We adapt the analysis of vocabulary specialization from Muennighoff et al. (2024) to the continual sctting. Note that. we will directly reproduce and slightly modily some lines from Muennighoff (2024)'s definition of vocabulary specialization below for clarity and ease of passing from one paper's notation to the other. Concretely, define Vocabulary Specialization
i...B.
Vocabulary Specializationlj, Es,
(3)
N₁.0
where:
Eg The ith expert in an MoF layer.
j: task index specifying which final checkpoint to use (e.g., specifying the final checkpoint after task 1, task 2,...).
The token ID being analyzed.
k; The number of experts considered (we use k=3 for Granular MoE's and k=1 for switch MoEis).
ja.n, ; The number of times input data is routed to E, for when using the final checkpoint of task j.
Ny.; The total number times input data routed across experts for x and the final checkpoint of task j.
Vocabulary Specialization can, therefore, be calculated for each expert at every layer of the model and for each token in the model's vocabulary. By assigning each token in the vocabulary to the expert that processes the most frequently, we can then create a one-to-many many mapping between experts and vocabulary entries for each layer of the MoE. Then, we can calculate the average vocabulary specialization of each expert by averaging over its assigned tokens and averaging across experts to measure specialization within a layer. To compare specialization across model checkpoints, we re-use the one-lo-many mapping of previous checkpoint and measure how the specialization with respect to this mapping has changed during continual pre-training. Concretely, the continual vocabulary specialization (CVS) for an MoE layer I can be defined as follows:
cvs(j.h) h) = Σ Vocabulary Specialization(h, Eay.. =EV
(4)
NE
Újj.z arg max {Vocabulary Specialzaatio((.EE..])) <E[Na]
(5)
Ng: The number of experts in MoE layer
V: The set of tokens in the model's vocabulary (we use the Llama3 tokenizer).
h: A task index specifying the checkpoint from which compute the mapping.
A task index specifying final checkpoint that is used to compute the continual vocabulary specialization.
Note that the dataset of tokens used to compute the CVS is omitled for simplicity. However, the specialization of experts will depend on the distribution of the tokens because the same input token may be routed to different exports depending on the context within which it lives and the context will change depending on the distribution. For instance, the hidden representation of the word "for" in an English language corpus and code corpus may differ wildly.
34

---

Published in Transactions on Machine Learning Research (09/2025)
Figures 11 and 12 report the CVS of Gramular and Switch MoEs, respectively. For each plot, the one-to-many mapping, Oj.z is created from the checkpoint pre-trained on FineWeb (e.g., the checkpoint we start continual pre-training from). All specializations are computed with respect to the input token. When evaluated on FineWeb, we observe across all architectures and balancing strategics that the first few layers for continually pre-trained models have lower continual vocabulary specialization than the pre-trained checkpoint, whereas subsequent layers have vocabulary specialization that closely matches that of the pre-trained chockpoint. This is even the case for the model that uses 0% replay, suggesting that MoEs learn routing policies during pre-training that are relatively unaffected by continual pre-training. When evaluated on German and stack, we observe that all MoEs continually pre-trained on those datasets have lower vocabulary specialization than models not trained on those distributions, showing their adaptation. On German, the zero-replay model has the smallest. CVS. We hypothesize that this is the case because these models adapt. the most to the German distribution and happen to learn new routing patterns, distinct from the ones used on FineWeb. Contrasting the results observed in subfigures (a) and (b) across Figures and 12 other subfigures, we observe that the vocabulary specialization while it changes across all layers for the data during continual pre-training, even for the model that does not utilize any replay. Contrasting this with the stronger performance of the no-replay model on German and its poorer performance on FineWeb, the superior adaptation to German is correlated to the change in vocabulary specialization throughout the model while the poorer performance on the previous distribution is correlated with larger changes vocabulary specialization in the first few layers.
the pre-training datasel only changes for the first few layers,
35

---

Published in Transactions on Machine Learning Research (09/2025)
Vocabulary Specialization
Vocabulary Specialization
35-
30
Continual
Continual
25-
FineWeh
FineWob Granular MoE
PB Granular MoE
15
German 40% Replay PB Granular MoE
German 40% Replay SB Granular MoE
10-
German 0% Replay
10-
German 0% Replay SB Granular MoF
PB Granular MoE
Stack 30% Replay PB Cranular MoE
Stack 30% Replay SB Cranular MoE
10
15
20
10
15
20
Layer
Layer
(a) PRTk Granular MoF, FineWeb
(b) SRTk Gramular MoF, FineWeb
40
Vocabulary Specialization
30
Vocabulary Specialization
35
25
Continual
Continual
20
20-
15-
15-
10
10-
0
10
15
20
0
10
15
20
Layer
Layer
(c) PBTK Gramular MoE, German
(d) SBTk Granular MOE, German
Vocabulary Specialization
Vocabulary Specialization
355
25
30-
Continual
20
Continual
25
15.
10
10
0
10
15
20
10
15
20
Layer
Layer
(e) PBTk Granular MoE, Stack
(f) SBTk Granular MoE, Stack
Figure 11: Continual Vocabulary Specialization for Granular MoEs. We report CVS for each MoE layer in the MoEs when testing models on FineWeb, German, and Stack. We observe that early layers deviate most from the checkpoint after pre-training, while later layers in the continually pre-trained MoEs nearly match the vocabulary specialization of their chockpoints after the first phase pre-training. This is even the case for the checkpoint that does not replay previous data, suggesting that vocabulary specialization for pre-training data is mostly determined during the initial pre-training phase.
36

---

Published in Transactions on Machine Learning Research (09/2025)
Vocabulary Specialization
Continual Vocabulary Specialization
20
Continual
15
15-
FineWoh
FineWeh
PB Switch MoE
SB Switch MoE
German 40% Replay PB Switch MoË
10-
German 40% Replay SB Switch MoË
Gorman 0% Replay PB Switch MoE
Garman 0% Replay SB Switch Mo
5
Stack 30% Replay PB Switch MoE
Stack 30% Replay SB Switch MoE
10
15
20
10
15
20
Layer
Layer
(a) PBTK Switch MoE, Fin
(b) SBTk Switch MoE, FineWeb
Vocabulary Specialization
20.0
Vocabulary Specialization
20.0
17.5
17.5
15.0
15.0
Continual
Continual
12.5
12.5
10.0
10.0
5.0
5.0
0.0
10
15
20
10
15
20
Layer
Layer
(c) PBTk Switch MoE, German
(d) SBTk Switch MoE, German
Vocabulary Specialization
Vocabulary Specialization
20
Continual
Continual
5
5.
10
15
20
10
15
20
Layer
Layer
(c) PBTK Switch MoE, Stack
(6) SBTk Switch MoE, Stack
Figure 12: Continual Vocabulary Specialization for Switch MoEs. We report CVS for each MoE layer in the MoEs when testing models on FineWeb, German, and Stack. We observe that early layers deviate most from the checkpoint after pre-training, while later layers in the continually pre-trained MoEs nearly match the vocabulary specialization of their checkpoints after the first phase pre-training. This is even the case for the checkpoint that docs not replay previous data, suggesting that vocabulary specialization for pre-training data is mostly determined during the initial pre-training phase.
37

---

Published in Transactions on Machine Learning Research (09/2025)
D.3.3 Continual expert co- activation analysis
We adapt the analysis of expert co-activation from Muennighoft et al. (2024) to the continual setting. Note that we will directly reproduce and slightly modily some lines from Muennighoff et al. (2024)'s definition of expert co-activation below for clarity and ease of passing from one paper's notation the other. Concretely, we define Expert Co Co-activation
NE..E,
Expert co-activation(E, E,) NE.
(6)
where:
Eg The first expert.
Ej: The second expert.
Ng, The number of times experts Eg and Fg are activated together.
Ne: The total number of times expert Eg is activated.
The co-activation matrix C for any layer in the MoE can, therefore, be created by setting Ct. define the co-activation difference follows:
Expert co-activation(E;, Ej). Then,
Co-activation Difference(p, q) = |C -C())
Where | is the coordinate-wise absolute value function. Each coordinate i, of the co-activation difference measures the change in expert co-activation for experts i,j between final MoE checkpoints after tasks p and g, respectively. Taking statistics the entries the co-activation difference matrix allows us to measure how expert co-activation changes globally at each layer during continual pre-training.
In Figure 13, report the median of the coordinates of the co-activation difference matrix between cach continually pre-trained Granular MoE in our study (the switch MoEs only activate a single expert so they have no co-activation) and its checkpoint after the initial pre-training phase. We observe that when evaluated on the FineWeb test sct, the Penalty-Balanced MoEs have the largest median differences overall and that they are most pronounced in the first two layers and layer 18. When evaluated on the German lest set, we observe that the models continually pre-trained on German have the largest median differences and that the tendency for BBlannce Mol's to have large differences is maintained. When evaluated on the Stack test set, similar trends are observed.
In Figures 14 and 15, we visualize subset of the full expert co-activation matrices for Penalty-Balanced and Balanced MoEs, respectively. Specifically, we show expert co-activations for the 16 experts with the largest co-activation values. The left-most plots show the co-activation matrix of the checkpoint continually pre-trained on FineWeb without decaying, the middle plots show the co-activation matrix of the chockpoint after continually pro-training on Stack, and the rightmost figures show the co-activation difference matrix. Subfigure (a) shows layer 0, (b) shows layer 11, and (c) shows the final layer. We observe that the so-activation difference is the largest for layer 0 for both Penalty- -alanced and Sinkhorn-Balanced MoEs. Notably, for the Sinkhorn-Balanced granular MoE's pre-trained checkpoint, most of the co-activation weight is placed on the expert 15. However, this strong weighting on expert 15 is attenuated during continual pre-training. In contrast, the co-activations are more dispersed in the Penalty-Balanced MoF. For layers 11 and 23, there minimal change between pre- teeinininng and continual pre- axa-training.
38

---

Published in Transactions on Machine Learning Research (09/202))
12
Median Co-Acivvated
17.5
TireWiwe
Median Co- Ao-Acctvatddd
Expert: (96)
Median Co- Activaleedd
15.8
Expert Change
Expert Change
12.5
70 Fin muiee Melt
German 0% Ruplay Gemmuine
10.0
Stock 50% Seglay
7.5
0.0
20
Layer
Layer
Layer
(a) PBTk, FineWeb
(b) PBTk, German
(c) PBTK, Stack
3.5
Median Co-Activatad
Median co-Activated
Expert Change (%)
SU Geueular MaE
Median Co-Activated
Expert Change (%6
30
Garman 40% Begley
Expert Change
Heppoy SB Gronular MaE
200
3.3
1.5
0.5
0.0
15
Layer
Layer
Layer
(d) SBTK, FibeWeb
(c) SBTk, German
(I) SBTk, Stack
Figure 13: Layer-wise Median Router Co-activation Difference for Granular MoEs.
We report the
median of the coordinates of the co-activation difference matrix between each model in the legend and its corresponding pre-trained checkpoint. We observe that on Web, the Penalty-Balancee MoEs have the largest median differences overall and that they are most pronounced in the first two layers and layer 18. On German, observe that the models continually pre-trained on German have the largest median differences and that the tendency for Penalty-Balann MoEs to have large differences is maintained. On Stack, similar trends are observed.
39

---

Published in Transactions on Machine Learning Research (09/2025)
FineWeb
Stack PB Granular MoE (x2)
Absolute Difference
Granular MoE (x1)
X1 X2l
11
30
13
13
14
15
15
15
18
18
18
20
20
20
21
22
23
23
25
26
26
26
-
28
22
22
50
82
Layer
FineWeb Granular MoE (x1)
Stack PB Granular MoE (x2)
Absolute Difference
[1 -X2
11
12
15
15
15
19
19
19
20
21
21
21
22
23
20
24
24
25
26
26
26
10
30
30
30
E2
22
06
EZ
996
62
06
(b) Layer 11
FineWeb Granular MoE (X₂)
Stack
Absolute Difference
PB Granular MoE (x₂)
[1 -X2l
70
13
13
14
15
17
17
18
20
20
20
40
22
22
22
23
24
25
25
26
28
28
28
30
30
30
32
82
06
(c) Layer
Figure 14: FineWel Router Co-activation Matrix for PB Granular MoE Continually Pre-trained on Stack. The left-most plots show the Co-activation matrix of the checkpoint contimually pro-trained on Web without decaying, the middle plots show the Co-activation matrix of the checkpoint after continually pre- tretriainin on Stack, and the rightmost figures show the co-activation difference matrix. Subfigure (a) shows layer 0, (b) shows layer 11, and (c) shows the final layer. We observe that the co-activation difference the largest layer O, while the other layers change minimally.
40

---

Published in Transactions on Machine Learning Research (09/2025)
FineWeb Granular MoE (x1)
Stack SB Granular MoE (X²)
Absolute Difference
[X] -X2
1
15
15
17
19
19
22
22
22
25
25
25
26
27
10
28
28
28
30
30
30
22
96
88
22
02
06
(a) Layer
FineWeb Cranular MoE (X1)
Stack SB Granular MoE (X2)
Absolute Difference
X1
X21
70
1
2
3
14
50
15
19
19
21
21
21
23
23
23
24
20
25
26
30
30
30
22
06
61
22
12
52
22
12
22
22
08
(b) Layer
FineWeb Granular MoE (X1)
Stack
Absolute Difference
SB Granular MoE (X²)
X1 - X2l
2
60
8
19
19
19
30
23
23
23
26
27
27
28
28
29
30
30
30
11
22
22
02
30
(c) Layer
Figure 15: Feb Router Co-activation Matrix for SB Granular MoE Continually Pre-trained Stack. The left-most plots show the Co-activation matrix of the checkpoint continually pre-trained FineWeb without decaying, the middle plots show the Co-activation matrix of the checkpoint after continually pre-training on Stack, and the most figures show the co-activation difference matrix. Subfigure (a) shows layer O, (b) shows layer 11, and (c) shows the final layer. We observe that the co-activation difference the largest in layer 0, with most of the weight placed on expert We observe that this strong weighting on expert 15 is attenuated during continual pre-training. For other layers, there is minimal change in co-activation between pre-training and continual pre-training.
41

---

Published in Transactions on Machine Learning Research (09/2025)
D.3.4 Continual routing imbalance analysis
While performance is one important axis of robustness to distribution shifts, maintaining a balanced load across experts just as important for MoE foundation models. Without a balanced load, MoE transformers inferenced using expert parallelism without token dropping (e.g,, as is done for SOTA models (DeepSeek- AI et al., 2025b; Zhao et al., 2025)) could be bottlenccked by the speed of single accelerator that reccives all the Lokens, leading to underutilization of the hardware, lower throughput, and higher costs. To quantitatively assess the effect of distribution shift on load balance, we propose the maximum routing imbalance (MRI): the largest proportion of tokens routed to single expert in a given MoE layer. Concretely, the MRI at a
training iteration and MoE layer j is defined as
EeEn 1 (x)) B
MRI(t,)) = max sE[1.....]]
(8)
Where B is set containing all tokens in given batch, 1 is the indicator function, E is the number of routed experts, and k is the number of active experts. Since latency increases with compulation, and, in MoE layer, the compulation requèred by given device increases with the load of experts on that device, then MRT calculated with respect to routing decisions on a distribution is prozy for the worst case latency of an MoE loyer on the distribution. We will use the MRI throughout the following sections to measure the effect of algorithmic changes to continual pre-training on routing imbalance.
In Figures 16 and 17, we set to be the final iteration of training for each model during pre-training and continual pre-training where it is applicable. The figures plot the layer identity the x-axis and the MRI on the y-axis. The left column plots report the MRT of PBTk MoEs, while the right column plots report MRT for SBTK MoEs. Figures 16 shows Granular MoEs, while Figure 17 shows switch MoEs. For Granular MoEs, we observe that the MRI for Penalty-Balanced MoEs is consistently lower than for Sinkhorn-Balanced MoEs, that little increase in MRI on FineWeb is incurred during continual pre trainiann even for the 0% replay model, and that MoEs bocome most unbalanced when seeing out-of-distribution data (e.g., see non-german models in (b) and non-code models in (c)). For Switch MoEs, we observe the MRI for Penalty-Balanced MoEs is similarly consistently lower than for Sinkhorn-Balanee MoFs, that similar to Granular MoEs little increase in MRI on FineWeb is incurred during continual pre- pre-training, even for the 0% replay model, that switch MoEs become most unbalanced when seeing (c,d) and non-code models in (e.0), and that high MRI is prevalent in early layers independent of the training and testing distributions used, unlike for Granular MoEs. Contrasting these differences with the superior language modeling performance of granular MoEs, one could hypothesize that the unstable MRI observed in early layers for switch models that is not present in Granular MoEs may be cause of the performance difference.
data (c.g., see non-german models in
42

---

Published in Transactions on Machine Learning Research (09/2025)
Maximum Routing imbalans (%)
PB Graular MoE
17.5
$3 Granular MoE
PB Grundlar MuE
Routing mabaacce
53 Gramuler Molt
FB Grandlar MoE
Maximum
12.5
Geeman 0% (teplay 83 Crazular Mol
PB Gramdlar Mal
10.0
Stack 30% Beplay Grazular
Wob
7.5
PBI Gramdar Mall
PlneWob I Stack
5.0
PR Gramlar Mal.
10
15
20
0.0
Layer
Layer
(a)
PRTK Gramular MoF, FineWeb
(b) SBTk Gramilar MoF, FineWeb
Routing Imbalance (%)
30
Routing Imbalance (%)
25
Maximum
Maximum
20
15
15-
10
5-
10
20
15
Layer
Layer
(c) PBT* Gramular MoE, German
(d) SBTk Granular MoE, German
Routing Imbalance (%)
25
Routing imbalance
Maximum
Maximum
15.
10
10
15
20
10
15
20
Layer
Layer
(o) PBTk Granular MoE, Stack
(f) SBTk Granular MoE, Stack
Figure 16: Layer-wise Maximum Routing Imbalance (MRI) for Granular MoEs. We report the MRI for each layer in the MoE percentage of all routing decisions made on given dataset's 20M token test set Fineb,, (c,d)German, and (c.f) Stack). The left column PBTk MoEs, while the left column reports results for SBTk MoEs. We observe that the MRI for Penalty-Balancee MoEs is consistently lower than for comparable Sinkhorn-Balance MoEs, that little increase in MRI on FineWeb is incurred during continual pre-training, even for the 0% replay model (except for its first layer), and that MoEs become most unbalanced when seeing out-of-distribution data (e.g., see non-german models in (e,f) and non-code models in (c,d)).
43

---

Published in Transactions on Machine Learning Research (09/2025)
Routing Imbalance (%)
Routing Imbalance (%)
Finelteb
Switch
Garman 406 Suplay
50
German 40% Replay
Switch
Maximum
60-
German Reploy
Maximum
40
German Reginy
Beplay
5B Beglay
20-
30
15
Layer
Layer
PBTK Switch MOE, FineWeb
(b) SBTK Switch MoE, FibeWeb
Routing Imbalance (%)
Routing laballnnce (%)
80
M
AM
Maximum
Maximum
20
10:
15
Layer
Layer
PBTk Switch MoE, German
(d) SBTk Switch MoE, German
Routing Imbalance (%))
Routing Imbalance (%)
MMW
50.
Maximum
Maximum
40. 30
so- 40-
10
20
15
Layer
Layer
(c) PBTK Switch MoE, Stack
(f) Switch MoE, Stack
Figure 17: Layer-wise Maximum Routing Imbalance (MRI) for Switch MoEs. We report the MRI for each layer in the MoE percentage of all routing decisions made given dataset's 20M token test set ((a.b) FineWeb, (c,d) German, and (e.f) Stack). The left column PBTk MoEs, while the left column reports results for SBTk MoEs. We observe that the MRI for Penalty-Balanced MoFs is consistently lower than for comparable Baakhorn-Ballnneee MoEs, that little increase in MRI on FineWeb incurred during continual pre-training, even for the 0% replay model (except for its first layer), that MoEs become most. unbalanced when seeing out-of-distribution data (e.g., see non-german models in (e.f) and non-code models in (c,d)), and that high MRI is prevalent in early layers of Switch MoEs independent of the training and lesting distributions used.
44

---

Published in Transactions on Machine Learning Research (09/2025)
D.4 Maximum routing imbalance of MoEs during continual pre-training.
In the following section, we report on the maximum routing imbalance of MoEs during CPT. Specifically, Figures 18 and 19 report routing immediately before and after the distribution shift, while Figures 20 and 21 report MRI during training for Granular and switch MoEs.
Routing imbalance during training By sparsely activating their weight matrices, MoEs experience performance benefits over FLOP-matched dense models. However, this comes at the cost of increased latency when the model's forward pass is bottlenecked by the latency of single expert. This can become a problem if router at any layer of the MoF chooses to dispatch majority of the token load Therefore, we can estimate the impact of continual pre-training on MoE latency by tracking the worst load imbalance at each layer of the MoE, which is the definition of the maximum routing imbalance equation 1.
particular expert.
In figures 20, and 21, we plot the MRI throughout pre-training (FineWeb) and continual pre-training (German CC) for Switch and Granular MoEs, respectively. Subfigure (a) show the training time and inference time MRI for PB MoEs, while subfigure (b) shows the training time MRI for SB MoEs and subfigure (c) show the inference time MRI for SB Možs. We distinguish between Sinkhorn Balanced training and inference because the Sinkhorn Balancing algorithm is incompatible with autoregressive generation, so in subfigure (c) show MRI for SB models without the balancing step (e.g,, what would be used during auloregressive generation).
For all MoEs, early layers (0-6) seem to have the largest MRI. For Switch and Granular MoËs alike, we observe that SB routing follows a very similar pattern all thronghout pro-training. During the continual pre-training phase, we observe that this pattern changes slightly, actually becoming more balanced throughout continual pre-training for both inference time and training time routing imbalance. Turning our attention the PB MoEs, we observe that Switch MoE's suffer from much greater routing imbalance than their dense Granular counterparts in early layers. However, for most layers of the PB switch MoE and all layers of the PB Granular MoF, the MRT quickly reaches a smaller value than their SB counterparts during pre-training and continual pre-training showing that PB MoEs are also robust to distribution shifts, but that the Granular MoE architecture favorable for continual pre-training
In summary, both PB and SB Top-k routing algorithms are robust distribution shifts, with PB initially being more perturbed by the distribution shifts,, but recovering quickly to a better balance then SB. These resulls demonstrate that using infovile LR scheduuess and replay is enough to continuallyy pre-train MoE LLMs without incurring large increase in MRI.
45

---

Published in Transactions on Machine Learning Research (09/202))
30
Continual Pre-training
70
Continual Pre-training
Routing Imbalance (%)
Begins (Iter 190720)
Routing Imbalance (%)
Regins (Iter 190720)
25
Uniform Routing
60
Unitorm Routing
PB Granular MoE Replay
Switch MoE 0% Replay
Maximum
20
PB Granular
Replay
Maximum
50-
Switch Replay
PB Granular MoE 40% Replay
Switch MoE 40% Replay
40
10
20
190600
190700
190800
190900
191000
191100
191200
190600
190700
190800
190900
191000
191100
191200
Training Iteration
Training Iteration
(a) Cranular Penalty Balanced MoE
(b) Switch Penalty Balanced MoE
Routing Imbalance (%)
8
Continual Pre-training
Begins (Tter 190720)
Routing Imbalance (%))
Continual Pre-training
Begins (Iter 190720)
Uniform Routing
Uniform Routing
SB Gramular MoE 0% Replay
SB Switch MoE 0% Replay
Maximum
SB Gramular MoE 10% Replay
Maximum
22
SB Switch MoE 10% Replay
SB Cramular MoE 40% Replay
2.0
SB Switch MoE 40% Replay
5
18
16-
14
3
12
190600
190700
190800
190900
191000
191100
191200
190600
190700
190800
190900
191000
191100
191200
Training Iteration
Training Iteration
(c) Granular Sinkhorn Balanced MoE
(d) Switch Sinkhorn Balanced MoE
(PB) Top-k MoEs show little change in training-tine maximum routing imbalance as a result of adjusting the replay percentage. We report the modian MRI observed across MoE layers with min and max error bars shortly before and following the distribution shift when continually pre-training the MoEs on German CC. We observe that independent of the replay percentage used, the MoEs recover pre-training level median MRT within 1000 iterations of continual pre-training. However, replay does mitigate the increase in MRI caused by the distribution shift to small extent in PB MoEs. In contrast, the SB MoEs are quite robust to the distribution shift and their routing patterns seem to be invariant to the replay percentage used.
Figure 18: Barkhorn-Ballncee (SB) and
46

---

Published in Transactions on Machine Learning Research (09/202))
Maximum Routing Imbalance
Continual Pre-tzaining
0.7
Continual Pre-training
Begins (Ttrr 190720)
Routing Imbalance (%6
Begins (Jber 190720)
25-
+ SB Gramular MoE 0% Replay
Uniform Souting
0.6
SB Switch MoE 0% Replay
Uniform Routing
Maximum
20-
1 PB Granular MoE 0% Replay
0.5
PB Switch MoE 0% Replay
15
0.3
30
0.2
0.1
190600
190700
190800
190900
191000
191100
191200
190600
190700
190800
190900
191000
191100
191200
Training Beeation
Training Iteration
(a) Non-decayed Granular MoE
Non-decayed Switch MoE
Maximum Routing Imbalance
0.8
Continual Pre- trainininng
Routing imbalance (%)
30
Continual Pr
Begins (Iter 190720)
0.7
Begins (Tter 190720)
Uniform Routing
25
SB Granular MoE 0% Replay [D]
Unidorm Routing
0.6
SB
Replay [D]
Maximum
20-
Granular Mal 0% Replay (D)
0.5
FB Switch McE 0% Replay [D]
15
10
0.3
0.2
0.1
190000
190700
190800
190900
191000
191100
191200
190600
190700
190800
190900
191000
191100
191200
Training Iteration
Training Iteration
(c) Decayed Cranular MoE
(d) Decayed Switch MoE
Figure 19: Decayed Renalty-Balanced (PB) Top-k MoEs have slightly higher MRI during distributions shifts than their non-decayed counterparts. We report the median MRI observed across MoE layers with min and max error bars shortly before and following the distribution shift when continually pre-training the MoEs on German CC. We observe that all SB MoE keep stable MRI throughout the distribution shift, showing that they are mostly unaffected. In contrast, the PB checkpoints suffer from strong routing imbalance after the distribution shift but recover quickly, with the decayed checkpoints reaching a marginally higher MRI.
47

---

Published in Transactions on Machine Learning Research (09/2025)
0.060
Layer
Layer
Maximum Routing Imbalance
Layer
Layer
0.055
Layer 2
Layer
Tayer
Layer
0,050
Layer 4
Layer
Layer 6
Layer
0.045
Layer
Layer
Layer 8
Layer 21
0.040
Layer
Layer
Layer
Layer
Layer
Routing
0.035
Layer
Continual Pre-training
Begins (iter 1907 40)
0.030
50000
100000
150000
200000
250000
300000
Training Iteration
Inference time & train time MRI. PB Granular MoE
0.25
Layar
Layer 13
Madmum Routing imbalance
Layer
Layer
0.20
Layer 2
Layer 15
Layer
Layer
Layer
Layer 17
0.15
Layer
Layer 18
Layar
Layer 20
0.10
Layer
Layer
Layer
Layer 22
Layer
Layer
Layer 11
Uniform Rouling
Layer 12
Continual Pre tre-training
Begins ([ter 190740)
0.00
50000
100000
150000
200000
250000
300000
Training Iteration
Inference time MRI, SB Granular MoE
0.25
Layer
Layer
Maximm Routing imbalance
Layer
Layer
0.20
Layer 2
Layer 15
Layar 4
Layer 17
0.15
Layer
Layer
Layar
Layer 19
Layer
Layer
0.10
Layer
Layer 21
Layer
Layer 22
Layer
Layer
0.05
Layer 11
Uniform Routing
Layer
Continual -ob-trraiiingg
Begins (Iber
0.00
50000
100000
150000
200000
250000
300000
Training Iteration
(c) Train time MRI, SB Granular MoE
Figure 20: Training time and inference time MRI for Granular MoEs throughout pre-training and continual pre-training. We show layer-wise maximum routing imbalance during pre-training and continual pre-training. While Penalty-Balanced MoE's have the same routing dynamics at training and inference time, Sinkhorn balancing is incompatible with auloregressive generation, so we show both inference-time and train-time MRI for SB models. We observe that early layers in the MoE consistently have the largest MRI for both PB and SB MoEs, the MRI of PB MoEs is much better behaved, and after the distribution shift, the MRI of SB models becomes more stable.
48

---

Published in Transactions on Machine Learning Research (09/2025)
0.7
Layer
Layer
0.6
Layer
Layer 14
Layer 2
Layer 15
0.5
Layer 4
Layer 17
Routing
0.4
Layer
Layer
Layer 19
Layer 18
Layer
Layer 20
0.3
Layer 9
Layer 22
0.2
Layer
Layer
Uniform Routing
Layer 12
Continual Pre-tratming
0.1
Begias (lter 190740)
50000
100000
150000
200000
250000
300000
Training lteration
(a) inference time & train time MRI, PB Switch MoE
0.7
Layer
Layer 13
Maximum Roubing Imbalance
0.6
Layer
Layer 2
Layer 15
Layer
Layer
0.5
Layer 17
Layer
Layer 18
0.4
Layer
Layer
Layer 20
0.3
Layer
Layer 21
Layer 22
0.2
Lager 10
Layer 23
Layer
Boating
Layer 12
Continual Pre-tratning
0.1
Begins (Iter 190740)
50000
100000
150000
200000
250000
300000
Training Iteration
bb)
inference time MRI, SB Switch MoE
0.7
Layer
Layer 13
0.6
Layer
Layer 14
Layer
Layer 3
Layer 16
0.5
Layer
Layer 17
Layor 5
Layer 18
Maximum Routing
0.4
Layer
Layer 19
Layer
0.3
Layer 8
Layer 21
Layer
Layer 22
0.2
Layer 23
Layer 11
Unitorm Roating
Continual Pre-tralning
0.1
Begins (lter
50000
100000
150000
200000
250000
300000
Training lteration
(c) Train time MRI, SB Switch MoB
Figure 21: Training time and inference time MRI for Switch MoEs throughout pre-training and continual pre-training. We show layer-wise maximum routing imbalance during pre-training and continual pre-training. While Penalty-Balancee MoEs have the same routing dynamics at training and inference time, Sinkhorn balancing is incompatible with autoregressive generation, so we show both inference-time and train-time MRI for SB models. We observe that early layers in the MoB consistently have the largest MRI for both PB and SB MoEs, the MRI of PB MoEs is much better behaved, and after the distribution shift, the MRI of SB models becomes more stable.
49

---

Published in Transactions on Machine Learning Research (09/2025)
E Dataset sizes and sampling proportions
In the following section, we report the training tokens used, training dataset sizes, and sampling proportions. Specifically, Table 7 reports the amount of data and its exact composition used for different pre-training phases. Tables 8, 9, and 10 report the amount of training tokens and sampling proportions used for FineWeb, Germand, and Stack, respectively.
Table 7: Pre-training and Continual Pre-training Tokens. We report the training tokens for all different model training configurations in this paper. During continual pre- pre-training, each batch contains a proportion of replay tokens from the pre-training dataset and new tokens from the continual pre-training dataset.
Phase
Training Tokens
New Tokens
Replay Tokens
Pre-training
400B FineWeb
400B
400B FineWeb 200B Stack 30% Replay
140B
60B
Continual Pre-training
400B FineWeb + 200B German 40% Replay
120B
80B
400B FineWeb - 200B German 0% Replay
200B
Table 8: Fin CC: Train, Val, and Test dataset sizes used in experiments. For the purposes of our study, create a more manageable subset of FineWeb by subsampling each Common Crawl dump within Fin into smaller subsets. We then sample proportional to the sizes of each subset. We report the full size of the subset we sample from during training (note, we only train on 400B tokens of this subset), ). The exact sizes and sampling proportions each split are committed as they span more than a page, but can be made available upon request.
Source
Train Tokens (B)
Test Tokens (B)
Val. Tokens (B) Sampling Weight
Web CC
2916.650
26.442
26.426
1.000
Table 9: German CC: Train, Val, and Test dataset sizes used in our experiments.
Source
Train Tokens (B) Test Tokens (B)
Val. Tokens (B)
Sampling Weight
German CC
169.291
0.489
0.491
1.000
Table 10: Stack: Train, Val, and Test dataset sizes used in our experiments.
Source
Training Tokens (B)
Test Tokens (B)
Val. Tokens (B)
Sampling Weights
YAML
9.039
0.613
0.609
0.017
Java
19.730
0.587
0.587
0.174
C
17.988
0.594
0.597
0.159
Markdown
21.699
0.477
0.474
0.017
PHP
16.660
0.450
0.447
0.146
C#
9.245
0.552
0.553
0.084
JSON
120.669
0.709
0.695
0.017
TypeScript
6.892
0.418
0,414
0.063
C++
13.998
0.538
0.539
0.124
Python
15.898
0.458
0.457
0.200
Total
251.819
5.396
5.372
1.000
50

---

Published in Transactions on Machine Learning Research (09/2025)
F Model hyperparameters
The following section outlines the hyperparameters used to train the MoEs and dense transformers in our study. Specifically, Table 11 reports the hyperparameters of the schedules and Table 12 reports the model hyperparameters. We also show an example infinite learning rate schedule in Figure 22.
0.00630
0.00030
0.00025
0.00025
0,00020
0.000200
Learning
Learning
0.00015
0.00015
0.00010
0.00010
0
50
100
150
200
250
300
350
400
25
50
75
100
125
150
175
200
Training Tokens (B)
Training Tokens (B)
(a) Pre-training
(b) Continual Pre-training
0.00630
0.00030
0.00025
0.00025
Learning Rate
raing Rate
0,00020
0. 000000
0.00015
0.00015
0.00010
0.00010
0.00005
0.00005
100
200
300
400
500
600
0
10
30
40
50
Training Tokens (B)
Training Tokens (B)
(c) Full Re-training
(d) Ablation (Sec. 5.1)
Illustrated Learning rate schedules used for (a) pre-training, (b) continual pre-training, (c) full re-training, and the (d) rewarming ablation of Sec. 5.1. The exact hyperparameters of these schedules are reported in table 11.
Figure 22:
51

---

Published in Transactions on Machine Learning Research (09/2025)
Table II: Hyperparameters of LR schedules. All models used the same LR schedule hyperparam- eters. We refer the readers Ibrahim et al. (2024) section 7.2 for more thorough explanation of these schedules.
Table 12: Hyperparameters of our Moes and
Dense Transformer.
Description
Value
MoE Transformers Common
Description
Value
Active Parameters
571, 148, 288 2, 025, 236, 480
Parameters
Pre-training
Non-Embeddiing Parameters
1, 893, 902, 336
Schedule Type
Cosinelnf
MoE SM-FFN
Total Iterations
192720
Max learning rate (1mas)
3- 3-10-4
Shared Experts
Min learning rate (7min)
3 30-5
Active Experts
3
Constant learning rate (Nconst)
1.65 10"
Routed Experts
31
Warmup percent (Twarmap)
Total Experts
32
Cooldown iters percent (a)
70
FFN Intermediate Size
704
Constant iters percent (Tann)
0.10
MoE R-FFN
Continual Pre-training
Shared Experts
0
Schedule Type
Cosinelnf
Active Experts
1
Total Iterations
95370
Routed Experts
8
Max learning rate (Mman)
3 3-10-1
Min learning rate (Tmin)
3-10-
Total Experts
8
Constant learning rate (constt)
1.65 10-4
FFN Intermediate Size
2816
Warmup percent (T (Twarmap)
1
Top-k
Cooldown iters percent (Tea)
Z-loss Coeff.
0.001
Constant iters percent (T ann)
80
AUX-llss Coeff.
0.01
Full re-training
Sinkhorn
Schedule Type
Cosine Annealing
Tolerance
0.01
Total Iterations
288090
Max learning rate (Mmaz)
10-1
Dense Transformer
Min learning rate (fmin)
3 10-5
Parameters
571, 148, 288
Warmup percent (Twarmap))
Non- Non-Eubedding Parameters
814, 144
Continual Pre-training Ablation (Section 5.1)
Num attention heads
16
Schedule Type
Cosine Anncaling
Common
Total Iterations
95370
Max learning rate (Mmaa)
3 10
Num layers
24
Min learning rate (1min)
10-5
Hidden size
1024
Warmup percent (T (Twarmap)
FEN Hidden size
2816
FFN Type
GeGLU
Optimizer
AdamW
,1,02
0.9, 0.95
Batch size
1024
Sequence length
2048
Hidden activation
GeLU
Weight decay
0.1
Gradient clipping
1.0
Decay
Cosine
Positional embedding
Rotary
GPTT-J-Residul
True
Weight tying
False
Vocab Size
128000
Rotary PCT
0.25
52


Reviews:
[{"review_id": "8eskiCMs59", "reviewer": "Reviewer_Ghyb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates how Mixture of Expert (MoE) transformer LLMs behave during continued pre-training (CPT) when exposed to distribution shifts. The assumption is that CPT might be challenging for MoEs because the router might not be able to deal with distribution shifts well - in terms of performance and efficiency (load balancing, etc). The authors train a ~500M parameter dense model and isoFLOP MoE models of different variants on FineWeb and do CPT on German language and code. They present insightful ablations, analyses and clear practical guidelines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, but I believe two points are not quite clear currently. \n1. The models are very overtrained at 40x Chinchilla optimal for the dense one. It's not clear that the results will translate to compute optimal regimes (which is what's most relevant for practical applications). \n2. All comparisons between MoE and Dense are biased towards the dense model. The comparison is MoE vs. FLOP-matched Dense which is not fair (see [Du et al. 2024](https://arxiv.org/abs/2405.15052v2)) as the MoE will be slower due to higher memory and communication requirements. See also point 2 in requested changes below", "audience_yn": "Yes", "audience_explanation": "I believe that this work presents a good foundation for future work to build on. Readers will be interested in learning how CPT works for MoE models and what new tradeoffs MoEs present in this setting. The authors nicely point to future directions of inquiry.", "requested_changes": "1. Regarding point 1 above, please mention the dense model size in the abstract in addition to the number of tokens which is already mentioned. Right now the text mentions \"large-scale (>2B parameter switch and DeepSeek MoE LLMs trained for 600B tokens)\" but this is misleading because DeepSeek MoE models are not actually used, and the meaning of \">2B parameter switch\" is not clear from the context.\n2. It would be very insightful to add the time per training/inference step for the reader to see a full comparison of efficiency for dense vs. MoE models. This would be a much better proxy for efficiency than MRI. At the very least it would be beneficial to report latency measurements for the different models that are investigated.\n\nMinor:\n1. MRI is a only a proxy for worst-case latency, not latency (change title of section 4.4)\n2. Citation brackets need fixing, use `Authors (2024)` vs. `(Authors, 2024)` correctly.\n3. Typo in title of Appendix B.2\n4. Please clarify the explanation of the Vocabulary Specialization in Section 5.3: \"...calculate the average VS of each expert by averaging over its assigned tokens and...\" - what do you average over tokens? Surely not the token ID. Do you take the count of tokens per expert? \n5. The plots in Figure 22 are slightly confusing with the text in Section 4.1. Which of the plots show the CosineInf schedule? Does setting (b) in 5.1 start with a warmup or not? Please clarify.", "broader_impact_concerns": "No concerns.", "additional_comments": "Additional questions:\n1. Why is Stack 0% missing from Figure 5? \n2. When using replay, does that increase the total CPT token budget? E.g. using 0% replay with 200B tokens, will using 40% replay use 280B tokens?\n3. Why use different replay percentages for German (40%) vs Stack (30%)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "d89kHN3kC7", "reviewer": "Reviewer_V6J6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies continual pre-training (CPT) for Mixture-of-Experts (MoE) LLMs and asks whether routers harm robustness under distribution shift. The authors consider four models (Granular vs Switch MoE; Penalty-Balanced Top-k (PBTk) VS Sinkhorn-Balanced Top-k (SBTk)), and pre-train them on 400B FineWeb tokens, then continue on 200B tokens of German or The Stack with replay and (usually) an infinite-LR schedule from a non-decayed checkpoint.\n\nIn Section 4, the paper provides an ablation study; increasing replay reduces forgetting on the original domain but hurts adaptation to the new one (which is quite trivial as replay decreases the amount of tokens from new distribution), and starting CPT from a non-decayed checkpoint (infinite LR) improves forgetting. Importantly, they introduce Maximum Routing Imbalance (MRI), the maximum share of tokens sent to any single expert in a layer, and use it as a proxy for worst-case MoE latency. In Section 5.1, the authors observe that SBTk is very stable and shows only a small MRI bump, while PBTk is unstable right after the shift, but quickly recovers to a lower MRI than SBTk. In Section 5.2, all MoEs beat the FLOP-matched dense baseline on validation loss during both pre-training and CPT; within MoEs, PBTk consistently outperforms SBTk and Granular outperforms Switch. Also, CPT MoEs match the full re-training baseline on downstream evaluations, indicating that routers do not block CPT and that MoEs keep their sample-efficiency benefit. In Section 5.3, the authors analyze how routing changes: the authors adapt Router Saturation, Vocabulary Specialization, and Expert Co-activation to the CPT setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe most of the claims are supported by clear experimental evidence, but a few seem insufficiently justified.\n\nIn section 5.1 (routing imbalance), the authors claim that PBTk shows a large imbalance at the very beginning of CPT but becomes more balanced than SBTk as training continues, and that this could be a factor behind PBTk’s better validation loss in Figure 2. However, I do not see a clear connection between the balancing metric (MRI) and validation loss, so this claim feels insufficiently justified.\n\nAlso, there is a confusing point regarding the MRI-related plots in this paper. In Figure 1, the MRI scale appears to range from about 3 to 6, whereas in Figure 3 it stays below 0.4, and in Figure 4 it is presented as a percentage. A detailed explanation is needed to clarify exactly what values or normalizations were used in each figure.", "audience_yn": "Yes", "audience_explanation": "MoE LLMs are a major trend right now (e.g., DeepSeek, Kimi K2). I think that decoder-only MoE Transformers differ in important ways from standard dense models, so checking whether classic CPT techniques still work, and how routing behaves under shift is a nontrivial and important question. While the paper does not introduce a new method or a striking new phenomenon, it provides a detailed systematic understanding of existing practices: the benefits of replay and starting from non-decayed checkpoint, and a clear comparison of routing (PBTk vs. SBTk). In that sense, I consider the findings valuable, and TMLR's audience would be interested in this paper as well.", "requested_changes": "The topic, claims, and direction are good, but the paper feels incomplete at the current stage in terms of presentation. Here I list few of them:\n\n### Typos/Mistakes\n\n- Figure 1: “400B English → 200B German” appears twice; one of them should be “→ 200B Stack.”\n- p.4: “FineWeb Penedo et al. (2024)” → “FineWeb (Penedo et al., 2024)”\n- p.4: “chinchilla optional” → “chinchilla optimal”\n- p.6: “disribution” → “distribution”\n- p.10: “models.Our” → “models. Our”\n- p.22: “Cotinual” → “Continual”\n- Unify capitalization style for appendix subtitles.\n\n### Lack of Description / Presentation Gaps\n- (Figure 1) I may have missed it, but I don’t see which exact setting the results come from, especially the replay ratio. Please specify.\n- The main text under-explains what is deferred to the appendix. For example, in Section 4 when describing the method, add explicit pointers like “see Appendix E for full hyperparameters.”\n- (Figure 3) (a) and (b) use purple = SBTk and black = PBTk, but panel (c) swaps them.", "broader_impact_concerns": "The reviewer does not identify any ethical concerns.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "VM6jgn0Bp0", "reviewer": "Reviewer_HdyQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a large-scale empirical study of CPT for MoE LLMs, comparing different MoE architectures and two routing strategies. The study also introduces Maximum Routing Imbalance as an evaluation of latency and load balance. The authors claim that MoEs maintain their sample-efficiency advantage during CPT, and are robust to distribution shift. They conclude that ghranular PBTk MoEs can match fully re-trained baselines at a fraction of the compute cost.\n\n### Strengths\n- The experimental design can be considered comprehensive, covering different routing algorithms and architectures with useful ablation studies. The results and analysis are clear and, from my perspective, reasonable.\n- The topic covered by this article could be interesting and inspiring to researchers who are diving deep into MoE LLM fine-tuning design.\n\n### Weaknesses\n- The experiments are built upon a 570M model, which is somewhat small. Whether the conclusion can be extended to larger models is questionable.\n- Due to the size of the model, down-stream evaluations are conducted on easy tasks. \n- MRI is a proxy rather than direct latency measurements; no study of token-dropping or alternative routing beyond Top-k.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. I think under the introduced setup, the claims are well-supported.", "audience_yn": "Yes", "audience_explanation": "Yes. I think people will be interested in learning how MoE LLMs behave during fine-tuning.", "requested_changes": "Although not necessary, I hope that the authors could include experiments on larger models, harder tasks, and probably model fine-tuning with quantization and more efficient approaches such as LoRA. \n\nIn addition, the authors should pay more attention to the paper format. For example, letter capitalization, citation format within the parentheses, etc.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)