Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
Authors: Houyi Li
Ka Man Lo
Shijie Xuyang
Ziqi Wang
Wenzhen Zheng
Haocheng Zhang
Zhao Li
Shuigeng Zhou
Xiangyu Zhang
Daxin Jiang
Venue: ICLR 2026 Oral (2026)
Keywords: Large language models (LLM)
Pre-training
Mixture-of-Experts (MoE)
Primary area: foundation or frontier models, including LLMs

Abstract: Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints — that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All code and models will be released publicly.

Published as a conference paper at ICLR 2026
MIXTURE-OF-EXPERTS CAN SURPASS DENSE LLMs UNDER STRICTLY EQUAL RESOURCE
Houyi Li
Ka Man Lo kemanphoebee@maill com
hyli220m. fudan. edu.on
Shijie Xuyang ys ysjxu24dm. fudan, edua..edd.comm
Ziqi Wang
wzql 01011142577mmli/e 1amail edu.
Wenzhen Zheng zhengwenzhen ac.
Haocheng Zhang'
hczhang258m Fudan edu. en
Zhao Li lzjoey@gmail, com
Shuigeng Zhou
sgzhou8 I fudan. edu.
Xiangyu Zhang?
Daxin Jiang? djiang@stepfun.com
robert. zahangestepfun.coom
FFudan University StepFun
University of Science and Technology of China
*Zhejiang University
ABSTRACT
Mfxture-of-Experts (MoE) language models dramatically expand model capac- ity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource straints - that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its signifi- cant practical value and potential. In this paper, we propose novel perspective and methodological framework to study this question thoroughly. First, we com- prehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data re- source. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All model checkpoints are publicly available..
INTRODUCTION
In recent years, Large Language Models (LLMs) built on the Transformer architecture (Vaswani stall 2017) have achieved strong results on a range NLP tasks (Radford et all, 2018: Achiam atall 2023 Touvron et all & [Bai atll 2023). Meanwhile, there has been growing interest in using Mfxture-of-Eaiee Experts (MoE) layers (Shazeer
((17Z) to expand model capacity while keeping the training cost reasonable all 2022: Zoph all 2022: Rajbhandar al, 2022). Recent open-
source initiatives have explored MoE-based LLMs (Dai et all 2024; Jiang et all, 2024, Wei et all
2022 Xue all, 2024; DeepSeek-Al et all 202A)) yet many widely adopted open-source models
"Corresponding author.
https://hueq"
httpa//eee////aaaaaoo//e//eeeeee http://mmmmeuupp..pee dense

---

Published as a conference paper at ICLR 2026
such LLaMA (Touvron et at all 2022),, and Qwen (Yang et all 2024: Qwen et all 2022),, continue to utilize dense architectures, posing open question of whether MoE LLMs can outperform their dense counterparts.
2023ath), 5). DeepSeek's first-gene mest-generation models (DeepSeck- all
Current comparisons MoE and dense LLMs often simplify the analysis to either a data-cenn dlta-ccnttiic or compute-centric perspective. The data-centric view, which keeps total training tokens constant for both MoE and dense models, praises MoE for its reduced activated parameter count per token (i.e., per-token compute cost) and potential for aggressive parameter scaling. For example, DeepSeek- MoE (Dai et all 2024)) reports a 16B-parameter MoE model (with 2.5B activated parameters) that achieves performance on par with 7B dense model under the same data budget, thus suggest- ing 2.5x "parameter-effciinnyy advantage. The compute-centric view, which fixes total training compute, examines the impact of MoE sparsity (i.e., the ratio of activated to total parameters) on per- formance. Under certain sparse configurations, total parameters can swell to nearly x00x those of dense baseline, but at the cost of requiring more training data and managing high memory overhead.
However, neither perspectives fully addresses the complex interplay of critical resource constraints in large-scale model development: the finite nature of training data volume (D), training com- pute (C), and model size (N), which affects both memory and inference throughput. In particular, MoE models typically encounter bandwidth bottlenecks during inference, as all experts reside in GPU high-bandwidth memory and must be moved into shared memory, making parameter count a key runtime cost factor beyond FLOPs. These interdependencies complicate the conclusion of the absolute superiority of MoE dense architectures. Intuitively, dense model with equivalent total parameters should have an advantage by fully utilizing its capacity. This often leads studies to favor MoE for scaling model size rather than direct comparisons at the same parameter count, thus overlooking the real-world resource constraints in large-scale training and deployment.
In this work, we introduce novel perspective aimed providing a more convincing resolution to this debate by posing the following question:
Can Mixture-of Experts surpass dense LLMs under equal total parameter, com- pute, and data constraints?
To reach a definitive conclusion on this issue, we draw insights from a unified parameterization framework for model architecture ($ ( and propose a three-step experimental methodology. First, we search for optimized architecture design to ensure each model candidate achieves its (near- ) optimal performance ($ ED. Second, we explore the optimal activation rate based on this optimized model architecture, with the total parameters N and compute budget C' fixed ($ E.. Third, we present a data reuse strategy to address the additional data demand of MoE models, thereby equating data resource D ($ () We also analyze the efficacy of this framework on downstream tasks ($ Íb.
Our findings, derived from extensive experiments and systematic evaluation under the proposed strict N/C/D parity with dense models, provide strong evidence that MoE architectures with opti- mized backbones and activation rates can indeed achieve superior performance over dense models on both upstream and downstream tasks. This implies that any observed performance gains can be attributed solely to architectural advantages, rather than disparities in parameter count or compute budget. Moreover, this challenges conventional wisdom and paves the way for resource-efficient yet powerful architectures in the next generation of large-scale NLP systems.
Our main contributions are: 1) We demonstrate, for the first time, that under fixed total parameters (N) and fixed compute budget (C), an MoE LLM can surpass its dense counterpart with careful architecture design (see Fig. II 2F). 2) Our experiments reveal the existence of a stable "optimal AR" region that consistently maximizes performance across varying N (see Fig. I. D). 3) We intro- duce a practical data reuse strategy that offsets MoE's s additional data needs, enabling robust gains over dense models without substantially increasing unique training data D (see Fig. 22 ()).
2 RELATED WORK
2.1 MoE LANGUAGE MODELS
Building upon MoE (Shazeer et all 2012). GShard (Lepikhin et al) 2021) facilitates model paral- lelism across devices for massive MoE models. With the advent of Transformers (Vaswani et all
2

---

Published as a conference paper at ICLR 2026
2010, the integration of MoE into the Transformer framework has become a popular model archi- tecture and achieved state-of-the-ar performance. As an early attempt, Switch Transformer (Fedus et all 2022) proposes top-1 gating to simplify MoE architecture and alleviate communication over- head. More recent Transformer-based MoE L.LMs include (Xue et all, 2024; Tiang et all, 2024: Wi etall 2024; Wei et all 2024; DeepSeek-Al et all (024B). The MoE architecture is briefty reviewed in Appendix
2.2 ANALYSES OF MOE SPARSITY
Several studies investigated the impact of varying the number MoE experts and adjusting granular- ity, both of which factors related sparsity. Through ablation studies, DeepSeekMoE (Dai et all 2022)) observes finer granularity results in improvement on overall model performance, and acquires ratio between shared and routed experts that yields slightly better Pile loss. Zoph all (2022) sum- marized the results of several MoE works and indicated that the gain of increasing sparsity quickly diminishes when the number of experts is greater than 256 very sparse model.
Table 1: Notation.
Symbol
Definition
Symbol
Definition
Dataset size tokens.
S
Sequence length.
M
Compute (w/o embedding) token in FLOPs.
Number attention heads.
Dm
Model hidden dimension.
Total training compute in FLOPs, Le., M
Dan
FFN hidden dimension.
Dn
Dimension of attention head.
N
Number of non-vocabulary parameters.
De
Expert hidden dimension.
N.
Number of activated parameters.
D2
Shared expert hidden dimension.
Activation rate, i..., No/N.
E
Number of experts.
Number MoE layers.
K
Number of chosen experts.
La
Number of dense layers.
Activated FFN-to-model ratio MoE lay- ers, i.e. (D, DD)DDD,
Number of total layers, i.e, Le + La-
FFN expansion ratio, i.e. Pm/D...
Total FFN-to- ratio MoE layers, i.e., (Di, ED.)/D..
Model aspect ratio, ie.. Pa/L.
7
Sequence-to-we tiith ratio, i.e., 8/D,.
From a methodological perspective, our comparisons are conducted in a sufficiently trained regime (with D/N 20 for key models) and under fixed-IV setting motivated by deployment memory constraints, which differs from scaling-law sweeps that often rely on undertrained large models at limited compute. We provide
detailed discussion in Appendix F
Concurrently with our work, Ludziejewski et all (20255) found that a sufficiently large MoEs trained with more tokens outperforms dense model with the same total parameters. We further show MoE superiority even at smaller sizes and address the additional data demand via reuse. Abnar et all (2025) studied the scaling law for optimal MoE sparsity. However, their models (up to N 30B) were trained with C = 1e20, much smaller budget compared to the approximately 9x and 30x compute used for our 2B and 7B models to ensure adequate D/N. This likely resulted in undertrained models, potentially affecting their conclusions. Moreover, our study first optimizes the MoE architecture to isolate the effect of different activation rates on performance. Detailed differences between our work and this previous study are discussed in Appendix B
3 EXPERIMENTAL METHODOLOGY
We begin by introducing a unified parameterization framework for model architecture, establishing solid foundation. Then, derive key insights from this parameterization, which informs comprehensive three-step experimental methodology. Finally, we detail the experimental setup used consistently across all subsequent experiments. Our notation is summarized in Table III
3

---

Published as a conference paper at ICLR 2026
3.1 ARCHITECTURE PARAMETERIZATION
To enable a comprehensive and general comparison of dense and MoE-based LLM architectures under realistic deployment constraints, introduce unified parameterization framework that ex- plicitly accounts for both model parameters and per-token compute cost.
Dense model parameterization. For dense model, we approximate the number of non- embedding parameters N and the per-token forward-pass computation cost M as follows:
R (4 + 3a) = (4+ 3a)
(1)
M ~ 2N + 4D. 2N + 4
(2)
2N 27/(4++30)
(3)
where a Dm/Da., y S/D.. and Ç = -D//.. Here, we omit the LayerNorm parameter count as negligible. Inference cost training cost be approximated by C z M x D C 3M D, respectively (based on the standard empirical observation that training typically takes about three times the forward pass for backward computation).
MoE model parameterization. In many real-world settings, only a subset of Transformer layers are replaced with MoE layers. The approximations for total vonabuaayl parameters N, activated parameters N₂, and per-token computation cost M can be expressed as
N ~ (4 D Le + (4 + 1+33a)
(4)
N. = (4+ 38)D (4 3a)
(5)
M 4D 2r, + 452-,T,3
(6)
where j (Die +D.)/DD and 8 = (Dx DD.)/D.. We again omit the parameters and FLOPs of the gating network (router) they comparatively small, and the activation rate (AR) is denoted as Ta. For the simple and common case where all L layers are MoE layers (i.e., La = 0), we have
ra = N = (1 3.0) 34).
M 2r, + L
(8)
Nr + 27/((4 3.)).
(9)
3.2 KEY OBSERVATIONS AND METHODOLOGICAL CONSIDERATIONS
Based on the above parameterization, we highlight the following insights that guide our experiments:
High structural degrees of freedom in MoE. Compared to a dense model (whose shape is almost uniquely determined by L, Dm, and a),, an MoE model has many more design choices: the number of MoE layers (Le), expert-related dimensions (e.g., K, E, De. Die), and so forth. Even with La 0, the final shape depends on μl and 3 in addition to Ç and ra- Exhaustively searching all combinations prohibitively expensive. Therefore, greedy strategy should be adopted.
Activation rate тa is the primary factor. At the same total parameter count N, the ratio per- token FLOPs between MoE model and dense model is primarily determined by the activation rate ra- More specifically, My is the pok-token cost of the dense baseline (with Ç, a fixed), then the pure MoE model's compute, normalized by Ma. roughly behaves as (the union of Equ. B and D)
4 + 3a + 27d
R = ral
(10)
3/ +
where aa and "Im denote S/D, for dense and MoE layers, respectively. As strongly correlates with once the shape hyperparameters (C, a. 3) chosen, R, grows monotonically with Ta-
The trade-off among N, C, and D. Once (the total parameters) is chosen and we fix near- optimal shapes for dense and MoE models, the total training compute for the MoE model can be approximated by C 3 R. M4 D, where Mj is the per-token cost of the dense baseline with the same total parameter count, and R. is the fraction token (relative to the dense baseline). If we want to keep the same total compute C for both MoE and dense models, the MoE model will need R. times more training tokens.
which the MoE model reduces compute per
4

---

Published as a conference paper at ICLR 2026
3.3 THREE- STEP EXPERIMENTAL METHODOLOGY
Motivated by the aforementioned observations, we propose three-step experimental methodology which is both comprehensive and fair, so as to achieve the new perspective motivated in $ I that enables a more conclusive comparison of MoE and dense LLMs under equal resource constraints. The methodology is outlined as follows: 1) Greedy architecture determination. First, determine the macro-level layer composition (i.e, the MoE-to-dense layer ratio Le vs. La and related choices such as shared experts). Second, determine the micro-level MoE design within each MoE layer (e.g., top- K routing and parameter allocation among routed/shared experts), which is largely orthogonal to the global model shape. Finally, select the near-optimal shape hyperparameters (e.g., S, a for dense and Ç, 3 for MoE) for fair comparisons a fixed N. 2) Activation rate analysis under fixed N and C. With the optimal MoE architecture chosen in the previous step, and the optimal dense LLM shape proposed by Kaplan et al. (2020), we compare MoE models versus a dense baseline of the same size, ensuring the total training compute C' is matched. Since C' must be the same, the MoE model typically receives up to R. times more tokens (initially considering repeated or augmented data). 3) Data reuse strategy. To ensure a truly fair comparison the same unique data budget D, we develop a data reuse strategy that offsets MoE's additional data requirement. This enables evaluation under strictly equal N, C, and D.
3.4 COMMON EXPERIMENTAL SETUP
Optimal hyperparameters. MoE training is sensitive to the learning rate (1)) and batch size (B) (Ha etall 2024). Even minor architectural changes, such as variations in E, can lead to different optimal hyperparameters. To address this, we train all our models using the optimal n and B based on the hyperparameter scaling laws proposed (Li (Lietall 2025). Specifically, ei all (2025)) found that the optimal y and B follow power laws and depend only on N and D. Since these scaling laws are applicable both dense and MoE models and are robust across various pretraining data distributions, we apply them determining and B for each of our experiments.
Dense baseline tuning. To ensure a strictly fair comparison, we also tuned the dense baselines by searching for near-optimal structural ratios (e.g., aspect ratio ( and FFN expansion L, Dm: Dtia with given N), guided by the scaling-law recommendations in Li et all (20255;; the resulting dense configurations used throughout the paper are summarized in Table IT.
equivalently
Others. We use internal, high-quality training and validation datasets composed primarily of diverse web text and specific domains such mathematics and code. The training and validation sets have different distributions, requiring the evaluated models to demonstrate strong generalization capabilities. Our models incorporate RMSNorm (Zhang & Sennrictt, 201) for pre-normalizatio, ALiBi (Press all, 2022) positional encoding for multi-head attention, and the SwiGLU (Shazeei 2020) activation function for both feed-forward networks (FFNs) and MoE experts. The training procedures used consistently across all experiments are outlined in Table A in Appendix IEI We employ cross-entropy loss (C) as the training metric and bits-per-character (BPC) as the validation metric.
OPTIMIZED MOE ARCHITECTURE
Building on the insights discussed above, we systematically examine the following model compo- nents the order outlined below: 1) Distribution of MoE and dense layers. 2) Gate score normal- ization. 3) Parameter allocation within MoE. 4) Exploration of optimal structural hyperparameters. Each component incorporates previous conclusions into its experimental settings.
MoE and dense layers arrangement. This part examines how to arrange the distribution between MoE and dense layers. We consider three layer arrangement schemes: every layer is an MoE layer (full), one dense layer followed by MoE layers (1dense), and interleaved MoE and dense layers (interleave). We additionally include shared experts (SE) for some of our experiments.
Table 5 in Appendix E presents the experimental settings and results. The conclusions lows: 1) 1densetSE performs the best, possibly because the dense layer contributes to more stable training. 2) The ratio of shared expert size to total expert size Da/(D + KP.)) has minimal
fol-
5

---

Published as a conference paper at ICLR 2026
2B Denis, 9.1x20 FLOPs, D 058
5475
1295
1.6621 FLOPs,
5:12
0.495
0.490
Data -tokars
5118
20 sedes. D-1140
4E80
22
3290
20 8.0%%
2238
28 fy-10.31
1508
70
1478
70
1288
20 5115
1149
10 0.475
0.1
0.2
0.3
0.4
0.5
0.6
Total Compute C
Activation rate
(a) Fixed (solid) or va (dashed)
(b) Fixed
Figure I: Performance of N 2B models trained with varying data sizes D and activation rates Ta- (a) With a fixed D, performance gain exhibits non-linear dependence on training budget C. Conversely, with fixed ra. increasing D leads to linear performance gain. These findings indicate an optimal activation rate, r 20%, that is consistent across various D values when constant. (b) With fixed training compute C, the optimal activation rate r 20% be clearly seen.
impact model performance. Therefore, we continue using the Idense+SE configuration and set Die
KDD.
Gate score normalization. The results of normalizing gate scores of chosen experts are in Table E in Appendix E Although the addition of normalization does not show an obvious difference in performance loss, it tends to reduce the average balance loss Lisllnce Since normalization requires K > to avoid zero gradient, we opt not to normalize given that some our experiments have K=1.
Top-K setting. In this part, discuss the allocation of parameters within MoE layers, focusing the top-K setting. Expert granularity is adjusted by varying K and D. while keeping their product constant. We conduct three groups of experiments with various r; and the results in Appendix El Note that within each experiment group, the product K D. is not strictly maintained due to compatibility with other hyperparameters. We observe that both overly large K and the K=l setting are generally suboptimal across the three groups. Therefore, we avoid using large K and avoid setting K=1 in our main experiments whenever possible.
given in Table
Model shape ratios. As discussed in 3.2 the shape hyperparameters include three ratios: Ç, a, and B8. We a 2.77 (Touvron et all 20236) and explore the optimal and μ. from which I can be derived. As illustrated in Fig. g in Appendix E. although performance fluctuates wildly given value ( or jr, there is an overall upward trend with increasing Dm for Ç and downward trend for Following the observed trend, we set Ç ss 88 and μ RJ 22 for the subsequent experiments.
OPTIMAL ACTIVATION RATE
In this section, we analyze how the performance of MoE LLMs varies with different activation rates (AR) using model backbones optimized based on the conclusions in $ and examine whether MoE models can outperform dense models. Note that a concurrent study (Abnar et all 2025) suggests that the optimal sparsity of MoE depends on model capacity. However, our findings indicate that, with optimized backbones, the optimal AR remains consistent across models of different sizes. We first detail our experimental setup and results, followed by a further discussion on the conclusions.
Setup. We built series of MoE models with non-vocabulary parameters N x 2B and N RJ 7B, but varying activation rates Ta from 8.7% to 58%. Noteworthy, the model backbones were built upon the findings in $ El as detailed in Table E. M I and [[2 in Appendix E Each model was trained on a proportional subset our dataset, ensuring D/N 2 20 (Hoimann et all 2022) for sufficient training.

---

Published as a conference paper at ICLR 2026
7B MoE series, D-1308
NoEs, uniaue data
fgi 8.9% - 53%
0.468
75 NLE, 2 2.221 FLOPS 0-688
0-0.5%
0.466
0.464
0.462
0.460
Date -inoens
0.458
6268
$138
0.456
6448
3808
2503
0.454
2209
1308
10a
0.3 0.4
0.5
Compute
Activation rate f
(a) Fixed D
(b) Fixed C' and reusing data
Figure 2: Performance of N as models trained with varying data sizes D and activation rate Ta- The optimal activation rate, laa 20%, align with the findings for the 2B models (Figure I].. Additionally, compared to training on the unique dataset, the strict data reuse scheme shows only a slight performance reduction, while the loose scheme often yields better performance.
5.1 OPTIMAL AR POINT
Focusing on the 2B models trained on the same data size D = 114B as shown by the green solid line Figure Inl we observe that the performance gain depends non-linearly on the training budget C. Specifically, the gain is more significant within relatively low range of r,, Starting from points on this curve and fixing the corresponding ra values, increasing D results in linearly diminishing BPC, as indicated by the dashed lines in Figure Ial These results confirm the existence of an optimal AR point, r that remains consistent regardless of /) when is unchanged. When plotting the results from fixed training compute perspective (C Co 9.1c20) in Figure II clearly observe that the optimal AR point is approximately ra x 20%.
5.2 COMPARISON WITH DENSE MODELS
To compare with dense models, we trained two dense models (Table I in Appendix E with N 2B parameters and training budgets C1 = Co = 9.1c20 and C2 = 1.64e21 R 201- The second model (C2) is included for comparison to account for the typically lower Model FLOPSs Utilization (MFU) in MoE training. This reduced MFU arises from load balancing and expert parallelism mechanisms that limit large-block matrix computations. As illustrated in Figure M.. MoB models outperform their C₁ dense counterparts when r, falls within a specific range (approximately 15% to 48% for 2B models). For instance, the MoE model with the optimal AR point r 20% achieves a BPC value that is 0,0064 lower than its C1 dense counterpart and only 0.0049 higher than the O2 dense model. This demonstrates the existence of an optimal activation rate region R where MoF models with ra E R2 can outperform their dense counterparts under the same training budget C and approach the performance of dense models with double the compute. However, the performance gains of MoE models rely on substantial increase in data, e,g. 4.6x larger data size at ra = T = 20%. To mitigate this additional data requirement, we explore a data reuse strategy in E
5.3 CONSISTENCY OF OPTIMAL AR
As illustrated in Fig. R an optimal AR point r also exists for 7B models. Surprisingly, r"" remains consistent for both 2B and 7B models at approximately 20%. suggesting that r is independent of model size. This finding contradicts established studies on MoE sparsity (Abnar et all 2025), which proposes that optimal sparsity (defined (E K)/E) is directly proportional to model size. Nevertheless, our experiments were conducted with strictly controlled variables using optimized backbones, leading us believe that our conclusions are both reliable and scalable (see Appendix B

---

Published as a conference paper at ICLR 2026
Average
Knowledge
Reasoning
-tokens
5.95
1.00
0.66
4438
0.44
3168
0.53
2218
Asziowilon tale
2:30
128 Actastion
411
Act Actxsllin
5.2
Average
Knowledge
Ressoning
7B
0.25
7B MoEs
strict data reuse
0.24
D 68B
7B Derse
D 1308
017
Actvvtion rore Fa
Activatio 1776 F
Actiiaiion 1ate F
Comprehensive
Math
Code
0.00
042
034
0.35
0.000
540
0.75
2.2
0.70
5.9
0.99
0.99
Activasion mane Fg
Activation 1978 F
Actiationn 188e
Figure 3: Downstream performance of 7B models: pre-trained (top) and SFT-ed (middle and bot- tom) versions. Across all benchmark types, MoE models with ra 20% outperform dense model trained with twice the compute, aligning with upstream observations that the optimal AR is 20%.
for more detailed discussion). To further validate the possible universality of our findings, conducted experiments on x 3B models and achieved similar results (see Fig. 8 in Appendix E).
Expert specialization is another significant potential of MoEs in addition remarkable scalability, where each expert focuses on learning specific features or patterns within the data. However, this at- tribute has not yet been clearly observed even in state-of-the-art MoE LLMs (Lo all 20224 Zhang atall 2024)), and effective approaches to achieve it remain under-explored. Based on our observation that MoEs outperform dense models when ra E Rt we conjecture relationship between the opti- mal AR region and the degree of expert specialization. Specifically: 1) When the activation rate is too low (r,, < 10%). the model lacks sufficient parameters to store knowledge effectively. When the activation rate is relatively high (r, > 50%), more experts are typically activated, which may lead to weaker specialization. An activation rate within the optimal region R, likely facilitates a higher degree of expert specialization, thereby enhancing the MoE model's performance compared to its dense counterpart. We leave further analysis of this potential relationship for future work.
DATA REUSE STRATEGY
As discussed in $ b MoEs outperform their dense counterparts but require additional data. To eliminate this increased data demand, we investigate data reusability by training models for multiple epochs using a fixed, smaller dataset size Đ. We extract a sub-dataset of size î from the original training dataset. At the beginning of each epoch after the first, the data are shuffled.
Setup. We explore two distinct schemes, termed the sirict and the loose data reuse schemes.
For the strict scheme, our aim to ensure that both MoE and dense models are trained under com- pletely equal conditions with respect to N, D, and C. Given fixed D) the number of training epochs (ranging from 1.7 to 8.3 in our 3B and 7B model experiments) increases as Ta decreases (hence decreasing M) to maintain the compute budget C. The experimental settings are detailed in Table DEL IS IG in Appendix E. which are mostly the same as those in $ 5,2 except for the training
data used. Specifically, we set Đ = 65B and 114B for the 3B models, and D = 68B for the 7B models, corresponding to the data used for training the dense models.
8

---

Published as a conference paper at ICLR 2026
Table Accuracy of 7B SFT-ed models across different benchmarks.
Dense baseline
MoE: w/ optimal AR
Activation rate
20.07
20.07
Pretrain info
Compute
5,45c21
86c21
2.86c21
Data reuse
CMMLU Greenl MI74))
31.23
32.11
Knowlodge
MMLU (Hendrycks MEMBE
31.26
32.92
24.57
MMddU-Redux (Lacma ctall MIRE!
28.90
30.93
23.73
MMLU-Pro (Wang
14.12
13.59
13.59
(F RID
32.32
Reasoning
LiveBeach (Wih (ERE))
16.82
18.15
16,76
MUSR (Sprague et all
MTA))
35.98
35.58
48.94
Comprehensive
AGlEvval (Zhung dial K1P4
20.89
22.07
21.02
(Sizgun W76
58.02
GAOKAO-Math24 Zhang stall MIFE)
9.92
15.70
9.09
Math
GSM8K
13.34
15.54
APPS (Hlemlyckks et M22
7.35
6.80
8,18
DS. 1000 (La
5.70
6,90
4,60
Code
HvmanEval (Chon WIAD
22.56
21.34
21.95
LoctCode Cotentonn
1.49
1.67
LiveCodeBench ( Innrr mn
K1M
4.21
4.63
3.37
For the loose scheme, we relax the constraint of identical D by fixing the number of training epochs to 2 for all Tas hence D = 0.5D. where the exact value of D corresponds to the specific We conduct experiments on 7B models and the experimental settings are in Table in Appendix E.
Results. The performance under the strict scheme is illustrated by the blue dashed lines in Fig. 2F and in Appendix E. Reusing data / only marginally diminishes performance compared to training on the unique dataset for a single epoch, and MoE models continue to outperform dense baselines. Moreover, increasing ò further narrows the performance gap. The similarity in curve shapes indi- cates that the optimal activation rate r"" remains unchanged. These findings address the primary question posed at the beginning of this paper: fixturr--ffExpertt can surpass dense LLMs un- der equal total parameters, compute, and data constraints, provided that the backbones are optimized and ra E Rg. We further discuss the reuse-vs-unique trade-offs below.
Discussion. Prior works have explored the effectiveness of multi-epoch training for dense and MoE models. Muennigholl et al. (2I2E) developed a scaling law that accounts for the number of repeated tokens and found negligible loss for repeating up epochs compared training unique data, whereas Hernandez et all (2022) showed degradation for dense models. Xue et all (2023) noticed no significant gain for MoEs with repeated training when high-quality data insufficient. We empha- size that our goal here is not to claim that multi-epoch training is generally better for MoEs; rather, we examine whether MoEs can still surpass dense models when the unique-token budget fixed. Concretely, under the loose scheme (green dashed line), for each ra we keep the consumed-token budget D fixed and compare (i) 2-epoch run subset of size D = 0.5D (thus processing D tokens with reuse) and (ii) 1-epoch run that consumes D tokens without reuse (i.e., D unique tokens), where both sampled from the same data recipe/distribution. We find that the 2-epoch reuse setting can match, and sometimes slightly improve over, the 1-epoch unique-toker baseline several suboptimal ra points; however, for the 7B models, the most important optimal point (ra s 20%) it does not exceed the 1-epoch unique-token baseline. In all case for the 7B models, using more than two epochs (multi-epoch) consistently degrades performance. For the 3B models under the strict reuse setting, at a fixed ra using larger unique-token budget (e.g., D 114B) con- sistently outperforms smaller one (e.g., 5 65B), aligning with the intuition that more unique data is better even under multi-epoch training (see Fig. E and Tables [HHH in Appendix (E). More- over, increasing the unique-token budget from 144B (trained for 2-3 epochs) to 309B (trained for I epoch) yields only a marginal improvement under the same loken-consumption budget (Fig. D) Moreover, under a fixed consumed-token budget (D 309B), increasing the unique-token budget from D = 114B (trained for ~2-3 epochs with reuse) to 309B (trained for epoch without reuse) yields only marginal improvement (Fig. 5). Overall, these results suggest that MoE models may

---

Published as a conference paper at ICLR 2026
tolerate mild repetition (around two epochs) under fixed token-consumption budget, but additional repetition becomes harmful.
7 ANALYSIS OF DOWNSTREAM PERFORMANCE
To assess whether the optimal ARs generalize to downstream tasks, we conduct SFT on our 7B pre-trained models (trained w/ and w/o strict data reuse) and evaluate both the pre-trained models total number 29 benchmarks (see Fig. B and Tab. 2D, including categories such as reasonime ing and knowledge. The comprehensive list of benchmarks can be found in Appendix D For all SFT trainings, we use a fixed data size D, and thus varying C' across models with different Ta. The 7B dense model trained with /wice the compute is included for comparison.
and SFT-ed models
MoE rs. Dense at ra ra" For both PT and SFT models, MoEs outperform their dense equiv- alents across all benchmark types when the optimal activation rate (rī*)) is 20%, highlighting the very possible universality of the optimal AR point across different training phases and data domains. Furthermore, ri" remains unchanged during SFT, even with varying C, suggesting that the SFT data size may have an upper limit for per- formance improvement, provided the PT model is adequately trained. Additionally, the average performance at гa * ra is consistently inferior to that of dense models, underscoring the critical role of the optimal AR point.
20%. This result aligns with upstream findings that
MoE vs. Dense at ra < <r Dense models outperform MoEs across all domains for PT models. After SFT, MoEs overtake dense models comp rehensh i and knowledge tasks. However, notable performance gap remains in math h, highlighting the PT stage's importance for math ability.
MoE vs. Dense at ra r Compared the dense models, MoËs perform better on knowledeg but worse on reasoning for PT models, and usually slightly underperform after SFT.
Sparser Denser. For PT models, denser MoEs (r, r outperform match the performance of sparser MoEs (ra < r"")) across all domains, consistent with Figure 2R, regardless of data reuse. When training on unique data, sparser MoEs perform better on knowle edwg. Notably, denser MoE performance significantly degrades with data reuse, especially for SFT.
Impact data reuse. For both PT and SFT models, data reuse has little impact on reason ing but causes significant degradation in knowledgy performance. Surprisingly, at Ta = r the SFT-ed MoEs trained with data reuse outperform both MoEs and dense models trained on unique data. This implies that model can master reasoning skills (rather than merely memorizing information (HH eLai, 2IZE!) with relatively small dataset (Muennighol) el al 2025 Wang el af, 2025) and further enhance its capabilities through multiple training epochs.
CONCLUSION AND FUTURE WORKS
In this paper, we propose a three-step experimental methodology to investigate whether MoEs can surpass their dense counterparts under the same constraints total parameters, compute, and data. By optimizing the architecture, identifying the optimal activation rate region, and reusing data, we arrive at a positive answer to this question. Future work will explore how optimal activation rates enhance model capabilities and whether similar conclusions hold for other training methods like upcycling (Komatsuzzki et all 2023) and MoEfication (Zhang et all, 2022). We hope this work offers valuable insights for the architectural design of next-generation models.
LIMITATIONS
The limitations of this work include: 1) Hindering by the high computational cost, we did not train
models larger than 7B. 2) As described in $ gl we focus mainly the impact of several main components of MoEs, but fix the rest to narrow the scale of experiments. 3) Exploration other elements can provide further comprehensive guidance for the architectural design of MoEs.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501. 12370, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936.
2019. URL http::tt/s/sttt http://caaaattoooooo hn www.thooogyy..ooo www.oog......oooooooooo 1-119-1300 3007
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Tristan Coignion, Clément Quinton, and Romain Rouvoy. A performance study of Ilm tlm-generated code on leetcode. In Proceedings of the 28th International Conference on Evaluation and Assess- ment in Software Engineering, pp. 79-89 090-89,20 2024.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.060N 2024.
DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, et al. Deepseek 1lm: Scaling open-source language models with longtermism, 2024a. URL https://arxiv.
brg/aba/2401.00 02954
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, et al. Deepseek-v3 technical report, 2024b. URL
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368 0668-2378,12 2019. URL https://aclanttooo
0TG/N19-1246/
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (3(10)):139, 2022.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He. Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are done with mmlu? arXiv preprint arXiv:2406.04/27 2024.
11

---

Published as a conference paper at ICLR 2026
Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upeycling large language models into mixture of experts. arXiv preprint arXi::2410.077: 2024.
Dan Hendryeks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace Hc, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.099 202
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021b.
Danny Hermandez, Tom Brown, Tom Conerly. Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodde Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCan- dlish. Scaling laws and interpretability of learning from repeated data, 2022. URL https:
//axxxv.orcaab///2. 10487.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In Advances in Neural Information Processing Sys- tems, volume 35, pp. 30016-30030, 2022.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International conference on machine learning, pp. 4411-4421 PMIR, 2020.
Yi Hu, Xiaojuan Tang, Haotong Yang. and Muhan Zhang. Case-based or rule-based. How do transformers do the math? In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL ptEtts:://ppnreete net/ htte/oormm11d=4Vqrreea
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 27Xiv:2403.07974, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.04008 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture- of-experts from dense checkpoints. In International Conference on Learning Representations, 2023. URL https::/
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wes Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Process- ing. pp. 784,794, Copenhagen, Denmark, September 2017. Association for Computational Lin- guisties. doi: 20.18653/v1/D11
URL https: /ac. antho ory.org/D11-110///
12

---

Published as a conference paper at ICLR 2026
Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong. Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang. and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference Machine Learning, pp. 18319-18345. PMLR, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL
net/ feet/forum?0 diggge /XHTMYD.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. Find- ings of the Association for Computational Linguistics ACL 2024, pp. 11595-11620, 2024. URL
httt:://aalaanhhoooooyyoo////// otg.::/..............ooooaaaaaa.
findings-acL
Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang. Shijie Xuyang. Yuantao Fan. Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i, step law - optimal hyperparameter scaling law in large language model pretraining. arXiv preprint arXiv:2503.04715. 2025.
Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-experts in large language models. arXiv preprint arXiv:2406. 18219, 2024.
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Maciej Stefaniak, Michal Krutul, Jan Malašnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Mitoš, and Sebastian Jaszczur. Joint moe scaling laws: Mixture of experts can be memory efficient, 2025. URL https://arxiv
org/abs 2502. 05172.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling lata-constrained language models. Advances in Neural Information Processing Systems, 01550388-53770 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoerr Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint 1rXXi:2501.19390 2025.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations, 2022. URL attps: hptp///peeeeeeeeeeeeee/ee /tt.tt//eeeeeeeeeeeeee..tttt/t/s aat.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024. URL https://aas /twpiv..rivvv.... www./os///aaaoooooooo.o
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. URL https://con... cdp..pen...pp.ooo.ooooooe com/
Susearch-covvrr/lannee wwteeeuuussuuueiiieeeeeee laaeed//aagageeee understanding
paper.par
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moc: Advancing mixture-of-experts inference and training power next-generation scale, 2022. URL nttps://arxxv.orq/
aps fos/2201,.00 05596.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. 2020.
13

---

Published as a conference paper at ICLR 2026
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqu: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP))JCNIPP), pp. 4463-4473, 2019. URL tttps:
7/aclantho ogy. ooy.orr//19-14551
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate, mixture-of-experts layer. arXiv preprint arXiv: 1701106538 2017.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thougle with multistep soft reasoning. arXiv preprint arXiv:2310.16044 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-beneh tasks and whether chain-of-thought can solve them. Findings of the Association for Com- putational Linguistics: ACL 2023, pp. 13003-13051, 2023. URL /tttps://cclanthe
org/2023 Windingss aclaassaacl...
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open founda- tion and fine-tuned chat models. arXiv preprint drXiv:2307.09288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 2017.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2071, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. Skywork-moes A deep dive into training techniques for mixture- of-experts language models. arXiv preprint 21t.::220000066633 2024.
Johannes Welbl, Nelson Fiu,, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 1 707.06209. 2017.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- limited I1m benchmark. arXiv preprint 24ti:22006.131144 2024.
Shaohua WU, Jiangang Luo, Xi Chen, Lingjun Li, Xudong Zhao, Tong Yu, Chao Wang, Yue Wang, Fei Wang, Weixu Qiao, et al. Yuan 2.0-m32: Mixture of experts with attention router. arXiv preprint arXiv:2405.1790
2024.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762-4772, 2020. URL https: / antho wtw..oo......................... wwwpyoorg/22...0 col ing.giimaaminnmmmmaaama ing-main 419/.
14

---

Published as a conference paper at ICLR 2026
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeett not to repeat: Insights from scaling Ilm under token-crisis, 2023. URL /xt.s://ax.....//////..
13230
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL /tt.::///axxvv.o////////////////
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 3444-3453, 2019. doi: 10.18653/ 1472. URL
https://aclantho.. wrg./1y.-pp. 117271
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305. 12474, 2023.
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse of arXiv preprint arXiv:2407.09590, 2024.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures experts. In Findings of the Association for Com- putational Linguistics: ACI.202) pp. 877-890, 2022. htt.h:://///.m a/ac lnntho www.oooy.oo... org/
2022. F iindingg-acc....
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:230.006364 2023.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08806, 2022.
APPENDIX
A BACKGROUND: MIXTURE-OF-EXPERTT
The MoE architecture primarily consists of gate and several experts. Typically, the gate g(.) is com- posed of a linear layer W, followed by a Softmax and a Top-K operation, and the experts { follow the standard FEN structure. The computation of an MoE block then be formulated as
15

---

Published as a conference paper at ICLR 2026
follows:
8,(2) = Softmax, (Wf,))
(11)
T(x) (s))( K);
(12)
(s((z) if T(x),
9:(x))
to otherwise,
(13)
y Σ ge(w)) Ex(2))
(14)
where sx(x: denotes the gate score for the i-th expert. Unless otherwise specified, use the above non-normalized Top-K gating in this paper (i.e., we do not renormalize the K selected scores to sum to 1). For completeness, the commonly used Top-K normalization is
gi(x)
(e(x)) E JET 25(x)
(15)
otherwise.
We also adopt the standard auxiliary load- had-balancing loss (used throughout our experiments) to en- courage uniform expert utilization. Given minibatch B, define
f
Ie
T())..
(16)
B
DEB
P:
(x)),
(17)
B
and compute
Libalnnce
F
S.P
Luuat LCE + M
(18)
B EXTENDED ANALYSIS ON RELATED WORK
We notice a concurrent work (Abnar all 20Z5) studied scaling law for optimal MoE sparsity. We highlight the differences between our work and theirs as follows:
Formulation: We define "sparsity" as the activation rate Ta N./N, which is more general definition than that proposed by Abnar et all (2025), namely the ratio of inactive experts the total number of experts, (E - KK/E.
Methodology: Given that the activation rate тa does not depend on the underlying model architecture, we can thus easily take into consideration other components such shared expert and build all our models upon the optimized architecture proposed in $ gl This ensures the observed performance differences solely attribute to the varying activation rates.
Sufficient training: Our main comparisons operate in a sufficiently trained regime (often beyond the compute-optimal point). For example, the 2B MoE models in Table g have D/N ranging from 53 to 252, and the 7B MoE models in Table [ have P/N ranging from 20 to 93, while our dense baselines satisfy D/N 2 20 (Table IT), aligning with the common compute-optimal guideline (Hoffmann et all 2(22)) This perspective is practically impor- tant since industrial model design often cares about the best achievable performance at a fixed-IV budget (e.g., training a 7B model on trillions of tokens), and it also enables more reliable downstream SFT comparisons that would be less meaningful with undertrained checkpoints.
Perspective under limited resources: Scaling-law studies often adopt broad sweeps that trade depth for breadth under limited compute, which can lead to undertrained large-model settings (e.g., Abnar all (2025) uses C' 1e20 per run for sweep up to 30B, and Ludziejewski et all (2025) uses at most 80B tokens). contrast, our 7B study allocates substantially more compute per setting (e.g., multiple runs with C z 2.86e21 C s 5.45e21; Table [2 and Table I3), prioritizing a high-D/N regime to more deeply investigate the fixed-N & fixed-C' question motivated by deployment memory constraints.
16

---

Published as a conference paper at ICLR 2026
Table 3: Pretraining data recipe compared with the LLaMA-I recipe.
DataSet Class
Our Recipe
Our Duta Set Detail
LLaMA-1 Recipe
LLaMA-1
Datu Set Detail
Recipe Difl'
WebDara-en
79.53%
CC (English)
82%
67% CC 15% C4 (English)
-2.47%
Code
4.62%
The Stack
4.50%
Github-Big Query multi-lingual book3, Guienbery class name
+0. 12%
Wikipodia
5.06%
1.69%, en: 0.13%, others: 3.24%
4.50%
+0.56%
Book
5.18%
open source English books
4.50%
+0.68%
arXiv
Stncknxchange
2.21%
class name
2.00%
as class name
+0.21%
Conclusion: We discover optimal activation rate that appears to be independent of model sizes, whereas Abbar all (2025) find that the optimal sparsity increases with model size.
Our conclusion regarding a consistent optimal activation rate contradicts the findings of Abnar et all (2025). While we believe our findings are reliable, given that our experiments conducted with strictly controlled variables using optimized backbones and sufficient training data, we acknowledge the possibility that the optimal ra might slightly shift for model sizes significantly beyond our studied range (i.e., N >> 7B). Nevertheless, contend that the optimal Ta be considered consistent within a certain range model sizes, in contrast to the significant changes reported by AAhnr el all (2025))
C PRETRAINING DATA RECIPE
For reproducibility, provide the mixture ratios of our pretraining corpus and comparison with the LLaMA-1 recipe (Lonvron et all (о2ΟT). Our recipe is intentionally close to a Issttle-ttlle mixture, and the corresponding data sources have public counterparts.
D COMPREHENSIVE LIST OF BENCHMARKS
To assess whether the optimal ARs generalize to downstream tasks, conduct SFT 7B pre-trained models (trained w/ and w/o strict data reuse) and evaluate both the pre-trained models and SFT-ed models on total number of 29 benchmarks (Figure B). The comprehensive list of benchmarks is provided here.
For pre-trained models, evaluate
Knowledge: BBH (Suzgun et all 2023). PIQA (Bisk et all 201%), SCIQ (Well et all 20IZ), SIQA (Sap et all 2019)
Reasoning: ARC (Clark all 2013. BoolQ Clark all 201), CLUE et 2020
DROP (Dua et all 2019). HellaSwag Zellers all 2019), NaturalQA (Kwiatkowski et all 2019) RACE etll 2017), WinoGrande (Sakaguchr all 2020), XTREME etll
2020)
For SFT-ed models, we evaluate on:
Comprehensive: AGIEVAL (Zhong et al, 2025), BBH
Knowledge: CMMLU (L all 2024), MMLU (Hendrycks et al, 20216), MMLU-
Redux (Gema et all 2024), MMLU-Pro Pro (Wang ct all 2024)
Reasoning: DROP, LiveBench (White et all 20244) MuSR (Sprague et all, 2023
Math: GAOKAO-Math24 (Zhhang al. 2025), GSM8K (Cobbe all 2021)
Code: APPS (Hendrycks et all 2021a), 1000 (1 ata 2023), HumanEval (Chennettal
2021), LeetCode (Coignion et all 2024)), LiveCodeBench (Jain et all 2024)
E MORE EXPERIMENTAL RESULTS
17

---

Published as a conference paper at ICLR 2026
1,700
1.555
Dm
1.400
640
Loss
768
096
1024
1408
1.455
Figure 4: Results for model shape ratios and μ.. An overall upward trend is observed in Dm increases, while y exhibits a downward trend with increasing Dm-
Table 4: Common training recipe.
Hyperparameter
Setting
Vocab
65536
Optimizer
Adam
Weight decay
0.1
Gradient clipping norm
1.0
IR Scheduler
Cosine
Warmup iters
clip(0.01 Iters, 200, 2000)
Min LR
le-5
Table 5: Experimental settings and results of MoE layer arrangement and shared expert. Hyperpa- rameters shared by all experiments: Dm 1408, Dmn 3904, Norm True.
N
N,
M
D2
I.
E.
D.c
Scheme
C
Conclusion
2.00
346M
8.7788
22
64
16
35
2
800
1600
full+SE
1.6813
346M
8.77c8
16
1600
interleave+SE 1.6697
performs
2.028
346M
8.77c8
22
64
16
70
4
800
0
inLerleave
2.15B
366M
6.63e9
11
128
16
85
5
352
1760
Sd 1,8700
Sdense+SIi 1.8737
1denseeSB performs the hest
2.15B
367M
6.63e9
11
128
16
70
4
800
0
interleave
interleave
2,15B
9,31e8
17
Idenae
2.15B
368M
9,31e8
22
64
17
36
800
800
Idense+SI
1.6712
(De + KD)) impacts little
9.31e8
1600
Se
Table 6: Experimental settings and results of gate score normalization. Hyperparameters shared by all experiments: Scheme idense, L 17, Dm 1408, Dua 3904, 22, Dr 64.
N
N,
(%)
M
E
K
1)
Die
Norm
C.
Puiuue
2,15B
368M
17.08
9.31c8
35
800
1600
Y
1.6726
1.355
2,15B
368M
17.08
9.31c8
35
800
1600
N
1.6712
1.452
2,15B
368M
17.08
9.31e8
37
800
0
Y
1.6752
1.409
2,15B
368M
17.08
9.31e8
37
800
0
N
1.6750
1.440
Table 7: Experimental settings and results of top-K setting. Hyperparameters shared by all exper- iments: Scheme = dense, I = 16, Dm 1408, Dmn = 3904, H = 11, Dr = 128, Norm =
False.
N
N,
(96
M
E
K
D.
D.:
2.15B
591M
27.47
8.00g9
8
1
3528
3528
2.0470
2,15B
591M
27.40
8.00e9
11
320
3520
2,0338
2.15B
949M
44.00
1.0e10
2 3176
6352
1.9996
2.15B
948M
1.0le10
288
2.0266
2.15R
1.24B
57.57
1.19e10
3
2888
8664
2.0156
2.11B
1.228
$7.68
1.18c10
88
33
256
8448
2.0235
18

---

Published as a conference paper at ICLR 2026
Table 8: Experimental settings and results of model shape ratios. Hyperparameters shared by all experiments: Scheme Idense, S 16384, Dn 128.
N
N,
D
Dm
L
H
E
K
D.
Du,
E
2.15e9
3.67e8
640
1774
34
50
4
608
2432
51.30
20.39
1.694
2.15e9
3.69c8
640
1774
37
5
38
736
2208
47.15
18.78
1 1,696
2.14c9
3.69c8
1774
49
1536
35.20
L,695
2.14e9
3.67ch
768
2129
20
99
8
448
3584
62.42
41.42
1.693
3,6988
2484
124
62.21
1.699
2.15c9
3.68c8
896
2484
20
91
416
2912
45.50
48,16
I 1.687
2.13e9
3,67c8
2484
24
54
576
2304
37.29
39,96
1
2.15e9
3.69c8
896
2484
34
61
352
1408
25.54
28.15
1,680
2.16c9
3 3,68c8
896
2484
37
47
416
1248
23.21
25,89
1 1,682
2.14g9
2839
28
80
23.91
L6811
2.16c9
3.69c8
1024
2839
49
49
2
256
512
12.75
22.33
1.693
3194
79
2.14c9
3 3,68c8
1152
3194
34
64
3
256
768
14.89
35,91
1,679
2.15e9
3,69c8
1280
3549
28
10
113
160
800
14.75
48.41
1.675
2.15e9
3.70c8
1408
3904
24
46
416
832
14,18
62.22
1.678
2.13c9
3.68c8
1536
4258
12
12
65
4
576
2304
25,88
140.64
L.685
4258
20,00
1.674
2.15e9
3.66c8
1536
4258
20
12
95
4
224
896
14.44
81.84
1.681
2.16c9
3 6.688
1792
4968
128
960
14.25
129.00
2,14c9
3,67c8
1920
5323
12
15
71
3
416
1248
16.03
175.55
1 1,699
Table Experimental settings and results of optimal ARs for MoE. models with N 2.15B and
fixed Hyperparameters shared by all experiments: I = 16,S 2048, D. 1408, Dmn 3904, 11, Dr = 128, 88.
N₂
(%)
M
D
C
D D/N
E
K
Da
D&
B
# Iters
BPC
1. 1888
8.74
1.6899
L.4e11
1.92:20
89
1
352
2.01c-3
672
82833
0.5235
1 88e8
8.74
1.6869
1.68111
2.83220
78
89
1
352
352
2.26e-3
832
98771
0.5155
1.888
3.67e1l
6.16e20
2.87e-3
133187
88c8
8.74
1.68c9
5.4tell
9.10c20
252
89
352
352
3.24e-3
1728
152927
0.5048
2.33c8
10.81
1.95c9
1.1411
2.22:20
88
2
352
2.01c-3
672
82833
0,5136
2,33e8
10.81
1.95e9
L.62e11
3.16c20
75
88
2
352
704
2.24e-3
896
88444
0.5084
1,95e9
2.31el1
4.50e20
3.49e.3
110149
2.33c8
10.81
1.95e9
3.29e11
6.41e20
153
88
2
352
704
2.78e-3
1280
125427
0.5002
2.33c8
10.81
1,95c9
4.68c11
9.12c20
218
352
3.100-3
1600
[42822
0.4967
4.11c8
19.11
3.02c9
L.14111
3,44c20
53
84
6
352
2112
2.01c-3
672
82833
0,5013
4.11e8
3.02c9
1.46e11
4.42+20
2.17e-3
93015
0.4971
4.11e8
19.11
3.02e9
1888e11
5.67e20
87
84
6
352
2112
2.34e-3
960
95469
0.4953
4.1
1
3.02c9
2.41el1
7.27c20
112
84
2112
2.526-3
1024
114870
0.4909
4.1
19.11
3.02e9
3.09c11
9.34x20
144
84
6
352
2112
2.73e.3
1280
117950
0.4872
34.95
5.06e9
1.14e11
5.77c20
15
2.01e-3
82833
0.4963
7.52c8
34.95
5.06e9
1.80111
9.13e20
84
84
15
320
4800
2.31e-3
896
98256
0.4892
1 1.09c9
50.79
1e9
1.14
8.10=20
84
26
288
7488
2.01e:3
672
82833
0.4950
1.0999
50.79
7.1
1.28c11
9.13c20
60
84
26
288
7488
2.080-3
704
89125
0.4933
Table 10: Experimental settings and results of optimal ARs for MoE models N 2.15B with fixed C. Hyperparameters shared by all experiments: = 16, S = 2048, D.. 1408, Dan = 3904, = 11, Dr 128, 88. The green row corresponds to the MoE model with the lowest BPC on the validation set.
N,
(.6%)
D)
D/N
K
D
Duc
1
# Iters
BPC
1.8888
8.74
5.41e11
22.50)
352
352
3.24e.3
152927
0.5048
2.3368
10.82
1.96c9
4.68c11
9.19-20
218
22.50
88
2
352
704
3.10c-3
1600
142822
0.4967
3.24c8
15.04
2.50c9
3.75e11
9.38:20
174
22.50
86
352
1408
2.89c-3
1344
136378
0.4966
3.68c8
17.11
2.77c9
3.39e11
9,38:20
158
22.50
85
352
1760
2.80c-3
1296
127756
0.4874
3.8988
18.06
2.89c9
3.25el1
9.38c20
151
22.50
93
320
1920
2.76c.3
1280
123862
0.4871
4.1
19.12
3.03e9
3.09e11
9.38=20
144
22.50
84
352
2112
2.73c-3
1280
11955
0,4872
4.29e8
19.94
3.13e9
2.99e11
9.38c20
139
22.50
92
320
2240
2.70e.3
1248
117177
0.4857
5.90e8
27.46
4.10c9
2.23e11
9.15c20
104
22.55
3528
3528
2.46c-3
1024
106335
0.4907
7.52c8
34.96
5.08c9
1.8011
9.16:20
84
22.50
84
15
320
4800
2.31o-3
896
98256
0.4822
9.48c8
44.11
1.47e11
3176
6352
2.16c-3
93460
0.4899
1.09e9
50.80
7.12c9
1.29111
9.15x20
60
22.50
84
26
288
7488
2.08e-3
704
89125
0.4933
1.24c9
57.73
1.14e11
9.13x20
8664
2.006e-3
82833
19

---

Published as a conference paper at ICLR 2026
Table 11: Experimental settings and results of optimal ARs for MoE models with N 6.52B with
fixed D. Hyperparameters shared by all experiments: L = 24, 2048, Dm 2048, Din
5464, H -1,,616661 Dn 128, < 85.3.
N₂
(%)
M
D
C
D/N
E
K
D:
D2c
B
Tters
BPC
7.26c8
11.15
5.59c9
1.30e11
7.25c20
82
512
1024
4.74c-4
640
98816
0,4808
8.70e8
13.36
6.46c9
1.30e11
8.37e20
19.90
81
3
512
1536
4.74c-1
640
98816
0.4763
1.0299
1.30e11
9.49.20
19,90
512
1.3e9
20.03
9.07c9
1.30e11
1.17221
19.88
78
512
3072
4.74c
640
988
0.4703
1.7099
26.11
1.15e10
1.30e11
1.48221
19,90
86
448
4480
4.74o-4
640
0.4681
3.4709
53.30
2.2le10
1.30e11
2.86221
19.90
84
28
384
10752
4.74c-1
640
98816
0.4664
Table 12: Experimental settings and results of optimal ARs for MoE models with N = 6.52B with
fixed C. Hyperparameters shared by all experiments: L 24, 2048, Dm 2048, Dmn
5464, H = 16, Dn 128, Ç = 85.3. The green row corresponds to the MoE model with the lowest
BPC on the validation set.
N,
(%6%
M
D
D/N
E
K
De
D.c
B
# Iters
BPC
5.8588
8.97
4.73e9
6.05e11
2.86221
92.88
21.00
83
1
512
512
7.62e
1512
195502
0.4665
1.30c8
11.19
5.59c9
5.11el1
2.86221
78.47
21.00
2
512
1024
7.23e-4
1360
183630
0.4624
8,74c8
13.41
6,46c9
4,43c11
2.86c21
67.95
21.00
81
3
512
1536
6.92c44
1232
175482
0,4580
1.0299
15.63
7.33c9
3.90e11
2.86e21
59.89
21.00
4
512
6.64e-4
1152
165447
0.4571
1,31g9
20.07
9,07c9
3.16e11
2.86:21
48.50
21.00
6
512
3072
6,23g-4
1040
148410
0,4543
1.71e9
26.18
1.15e10
2.50a11
2.86e21
38.32
21.00
10
448
4480
5.80e-4
960
127035
0.4580
1.96e9
30.07
1.30e10
2.21e11
2.86e21
33.83
21.00
12
448
5376
5.57e
800
134597
0.4588
3.48c9
53.38
222e10
1.30e11
2.86221
19.87
21.00
28
384
10752
4.74c-4
640
98816
0.4670
Table 13: Experimental settings and results of optimal ARs for 2B, 3B, and 7B dense baselines.
N
M
D
P/x
L
H
Du
Dm
Iters
BPC
2.15e9
1.44e10
6,50e10
9,36e20
30.23
28
17
2176
8848
8.44c.4
320
99182
0.4921
2.15c9
1.4c100
1.14e11
1.64c21
53.02
28
2176
1.00a-3
124032
0,4808
3,29c9
2.24e10
6,26e10
1.40211
19.03
44
19
2432
7008
1.23e-3
448
68253
0.4835
3,29c9
1.25el1
2.80221
6.48c9
4.21e10
6.80e10
2.86e21
10.49
32
32
4096
11008
3.89e44
432
76813
0.4736
6.48c9
4.21e10
1,30c11
5.45c21
20.00
32
4096
11008
4.76e-4
96816
0.4594
Table 14: Experimental settings and results of data reusing (Ó 68B) for MoE models with N
6.52B with fixed C. Hyperparameters shared by all experiments: L
24, 2048, D.. =
2048, Dm, = 5464, H 16, Dr = 128, 85.3. The green row corresponds to the MoE model
with the lowest BPC on the validation set.
N
(%)
M
Epoch
D/N
1
De
D2
17
BPC
7.30g8
11.19
5.59c9
5.11e11
7.52
2.86:21
78.47
21.00
82
2
512
1024
7.23c-4
1344
185816
0.4656
8.7488
6.4629
4.43e11
2.86c21
6.92c-4
175482
0.4618
1.029
15.63
7.33e9
3.90e11
5.74
2.86e21
59.89
21.00
80
4
512
2048
6.64e 4
1152
165447
0.4601
1.1
9.07c9
3.16
2.86e21
3072
6,23e-4
150729
0.4590
1.71e9
26.18
1.15e10
2.50e11
3.67
2.86221
38.32
21.00
86
448
4480
5.80e
960
127035
0.4597
1.9699
30.07
1,30e10
2.21e11
3,24
2.86e21
33.83
21.00
84
12
448
5376
5.57e-4
792
135956
0.4603
Table 15: Experimental settings and results of strict data reuse (Ó = 65B) for MoE models with
N 3.29B with fixed C. Hyperparameters shared by all experiments: L 24, 2048, Dm
1408, Dum = 3904, H = 11, Dn = 128. The green row corresponds to the MoE model with the lowest BPC on the validation set.
N.
rx(%)
1>
Epoch
M
C
D/N
E
K
De
1)u
F3
# Iters
RPC
2.78c8
8.46
5.41e11
8.33
2.51e9
1.36221
164.62
89
352
352
3.24e-3
1728
152927
0.4916
344788
10.54
4.68c11
7.20
2.92c9
1.36221
142.36
88
2
352
704
3.10c-3
1600
142822
0.4841
4.83e8
14.70
3.75e11
5.78
3.74c9
114.19
86
2.89e-3
1344
0.4774
6.2088
18.83
3.09e11
4.76
4.56e9
1.1x21
94.06
84
6
352
2112
2.73e.3
1280
117950
0.4757
8.9388
27.12
2.23e11
3.43
6.19c9
1.38:21
67.74
8
3528
2.465e-3
1024
106335
0.4794
L.1499
34.75
1.80c11
2.77
7.69c9
1.39221
54.85
84
15
320
4800
2.31e-3
896
98256
0.4786
1.449
43.77
1.77e11
2.26
9.48e9
1.39x21
44.52
8
2
3176
6352
2.169e-3
768
93460
0.4799
1.6629
50.63
1.29e11
1.98
1.08<10
1.39x21
39.09
84
26
288
7488
2.08e:3
704
89125
0,4830
1.8999
57.40
1.14e11
1.75
1.22e10
1.39221
34.61
8
3
2888
8664
2.006e-3
672
82833
0.4823
20

---

Published as a conference paper at ICLR 2026
38 MoEs, 1.4e21 FLOPs, 658
38 MoËs, 1.4e21 FLOPs, Ó 1148
0.490
3B MoE. 1.4e21 FLOPs, unique data
3B Dense, 1.4e21 FLOPs, D 65B
3B Dense, 2.8e21 FLOPs, D 1308
0.485
BPC
0.480
Data tokkens
541B
0.475
4688
3098
223B
1808
0.470
1478
1288
114B
0.1
0.2
0.3
0.4
0.5
Activation rate
Figure 5: Performance of N 3B models trained with varying data sizes D and activation rate Тę- The optimal activation rate, r = 20%. aligns with the findings for the 2B models (Figure M).. Additionally, compared to training on the unique dataset, the data reuse scheme shows only slight performance reduction. To save computational costs, only one model trained on unique data is included for reference.
Table 16: Experimental settings and results of data reuse (D 114B) for MoE models with N 3.29B with fixed C. Hyperparameters shared by all experiments: L 24, S 2048, Dm
1408, Dma = 3904, H = 11, Da 128. The green row corresponds to the MoE model with the
lowest BPC on the validation set.
N.
(2(E)
D
Epoch
M
C
D/N
E
K
De
D2
B
# Iters
BPC
2.78e8
8.46
5.41e11
4.75
2.51e9
1.36e21
164.62
89
1
352
352
3.24e-3
1728
152927
0.4896
347788
10.54
4.68e11
4.11
2.92e9
1.36221
142.36
88
352
704
3.10e:3
1600
142822
0.4820
4.83e8
14.70
3.75e11
3.29
3.74c9
1.40211
114.19
86
4
352
1408
2.89e.3
1344
136378
0.4760
6.20s8
18.83
3.09c11
2.71
4.56c9
141c21
93.93
84
6
352
2112
2.73c-3
1280
117950
0.4747
8.93c8
2.465e-3
1.14g9
34.75
1.80e11
1.58
7.89e9
1.39x21
54.85
84
15
320
4800
2.31e:3
896
98256
0.4778
1.449
2.169e-3
1.6699
50.63
1.29e11
1.13
1.08c10
1.39.21
39.09
84
26
288
7488
2.08c-3
720
87144
0.4825
1.899
57.40
1.14e11
1.00
1.2e100
1.39x21
34.61
3
2888
8664
2.006e-3
672
82833
0.4816
Table 17: Experimental settings and results loose data reuse MoE models with N 6.52B
with fixed C. Hyperparameters shared by all experiments: L 24, S 2048, D.. 2048, Dii
5464, H = 16, Dn = 128. The green row corresponds to the MoE model with the lowest BPC on the validation set.
N.
M
C
D/N
E
K
D.
Di.
B
# Itors
BPC
7.30e8
11.19
2.56el1
5.59c9
2.86c21
78.47
82
512
1024
7.23e-4
1344
185816
0.4591
8.74c8
13:41
2.21e11
6.46c9
2.86c21
67.93
512
1536
6.92c-4
1232
175482
0.4557
7.33c9
2.86221
512
6.64e-4
1152
165447
0.4550
1.3199
20.07
1.58c11
9.07c9
2.87e21
48.50
78
6
512
3072
6.23e44
1024
150729
0.4549
1.7c9
26.18
1.25c11
L.15100
2.86211
38.32
448
4480
5.80c-
960
127035
0.4570
1.9699
30.07
1.10c11
1.30100
2.86221
33.83
84
12
448
5376
5.57e-4
792
135956
0.4583
21

---

Published as a conference paper at ICLR 2026
F THE USE OF LARGE LANGUAGE MODELS: AN EXPLANATION
Only a small fraction of complex paragraphs are written with the assistance and modification of ChatGPT. For instance, we provide the prompt: "I am writing an academic conference paper in the field of computer science. Please help me polish the wording of this paragraph, organize the sentences, and express them in a more academic way." After that the outputs are rigorously reviewed to ensure accuracy and appropriateness.
22


Reviews:
[{"review_id": "2KL6raTuxT", "reviewer": "Reviewer_r8nz", "summary": "This paper investigates whether MoE models can outperform dense models at the same param and compute budget. The authors performed comprehensive experiments to determine the optimal activation rate, and showed MoE model outperforming their dense counterparts in controlled condition.", "questions": "- Same as weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This work is done in a very principled way. The authors carefully planned for the steps to determine the design decision, and tested the main question with numerous experiments.\n- The result leads to a surprising and potentially impactful conclusion, MoE can outperform dense model at the same parameter scale.", "weaknesses": "- The results on some datatsets, e.g. GSM8k, MMLU, is surprisingly low, when compared to other open sourced models at 7B scale, e.g. LLAMA2. If the quality of fineweb edu is not too bad, have the models have been sufficiently trained? Have the evaluation followed best practices?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkCa01DqkS", "reviewer": "Reviewer_NUMD", "summary": "The paper shows that MoE can surpass dense Transformers using the same resources for training. A similar result has been shown in one of the recent papers cited in the Related Work section, where the models were matched in total parameters and training compute. In addition to this setup, in Section 6 the submission authors consider an additional modification: to match the number of training tokens used, they train the MoE model for multiple epochs (so the MoE dataset size is still technically larger, but the number of _unique_ tokens seen in training is matched). Furthermore, the authors present a set of ablations on optimally setting the MoE training hyperparameters.", "questions": "1. Question about optimal sparsity. In the methodology for deriving optimal sparsity, the authors fix the model size and compute budget, and perform a grid across sparsity levels. To keep the compute constant, the authors **modify the dataset size**. Therefore, the observed optimum may just be an effect of seeing **the optimal token to parameter ratio** for the given compute budget. To actually determine the optimal sparsity level for a given FLOPs budget, the paper should contain a two-dimensional sweep over both token-to-param ratios **and** sparsity levels. Are the authors able to provide such experiments or explain/clarify? This is an important technical concern regarding methodology and paper conclusions.\n2. Based on the literature, scaling laws coefficients can change depending on the data distribution [3]. The submission authors set lr and batch size based on hyperparameter scaling laws from [4]. While [4] concludes that their results are robust across data distributions, did the authors perform an ablation on a small scale to make sure these optimal hyperparameters are unchanged? If yes, such plot should be placed in the paper. \n3. Question about gate score normalization. Based on Table 5, the authors conclude that normalized/not normalized router logits result in a similar loss. However, In the setups the Top-K used is relatively small (2 or 4). Intuitively, with more granular models, where there are more experts activated (like in Table 6: top-K=33, experts=88), the importance of gate normalization could be more pronounced. Did the authors consider such setup? E.g. comparing the model from the last row of Table 6, but trained with/without gate score normalization.\n\n[3] Brandfonbrener et al., Loss-to-Loss Prediction: Scaling Laws for All Datasets\n\n[4] Li et al., Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Large scale of experiments.\n2. Evaluation both based on BPC and on downstream tasks (these two types of evaluation can behave differently for MoE and dense Transformers [2]).\n3. Evaluation both in pretraining and SFT.\n4. The authors declare releasing all code and models.\n5. A considerable set of ablations and studies on setting the MoE hyperparameters.\n\n[2] Jelassi et al., Mixture of Parrots: Experts improve memorization more than reasoning", "weaknesses": "1. Reproducibility: the authors do not disclose the dataset used in the paper, which limits reproducibility.\n2. Please also see Questions to the authors, where I placed my technical concerns/questions regarding the methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSe4aORdfe", "reviewer": "Reviewer_9SQb", "summary": "The paper investigates the performance of MoE with respect to dense models.\nFirst, an ablation studies are conducted to search for the best architecture.\nThen, the paper finds that there is an optimal activation rate at 20%", "questions": "1. Could you explain in more detail how you used hyperparameter scaling laws proposed in (Li et al., 2025) (line 208)? Did you re-fit the scaling laws with your data? I could not find the scaling law parameters in the Appendix or elsewhere.\n2. Ablation studies in [2409.02060] show that shared experts are not effective. Do you have any idea what leads to this different conclusion?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Experiments are conducted at a fairly large scale, 7B model.\n2. A thorough investigation on the model architecture is performed. \n3. Comparing MoE and dense models by repeating data used to train MoE is new and is a good practice for future work which compare MoE with dense models.", "weaknesses": "1. Some of the architecture search study is not very convincing. For example, it is mentioned that larger K hurts performance and hence K=1 is chosen, but in Table 6 K=1 has worse performance. Also, only K=1 and K=11 are tested with but not more fine-grained values like K=4,8 etc. Hence, it is not obvious K=1 is the best choice.\n2. Some of the design choices are quite unconventional, e.g., one dense layer followed by MoE layers and K=1, while popular public models like Qwen, Mixtral use full MoE layers and K larger than 1. This could be due to the fact that the specific greedy architecture search done by the paper leads to such a design choice, and hence the result may not be generalizable well.\n3. Many previous MoE studies [2501.12370,2502.05172,2502.03009] provide scaling laws which are concrete guidelines for scaling MoEs. No such studies are performed, which may limit the usefulness of the paper.\n4. As a result, this paper sounds like a whitepaper that merely reports experimental results instead of providing findings that are useful to the community and in a more scientific way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Hzi5BkNE4", "reviewer": "Reviewer_K2nX", "summary": "The authors do a big sweep of MoE hyperparameters under equal resource constraints (parameter count, training data, and data budget fixed). They find the optimal shape for their model, and then they show that, surprisingly, they can outperform the dense baseline under these constraints.", "questions": "- The authors use a lot of single-letter parameterization. Some of these lack a clear conceptual meaning in the paper. However, for lots of them, there is one: for example, alpha is the expansion factor for the MLP, zeta is the aspect ratio of the model, etc. Explaining this would greatly improve the paper's readability.\n- What does it mean to \"First, decide the MoE to dense layer ratio (Le vs. Ld) to focus on MoE’s internal benefits.\"? Specifically, what is focusing on internal benefits?\n- What is the non-normalized gate?\n- What is the exact architecture? What regularization loss is used? What transformer variant? What activation, what norm, what attention, what positional encoding, etc?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Many trained models, extreme compute budget\n- Almost every important hyperparameter checked\n- Their model outperforms the dense baseline", "weaknesses": "- The authors find many surprising results that contradict the literature and intuition. One of these is that K=1 outperforms K>1. While I am not aware of other papers with exactly the same fixed budgets, there are papers examining the effect of K under fixed compute budget, e.g [1]. This finding also contradicts my personal experience. Another surprising point is the multi-epoch training efficiency, also mentioned in the Discussion paragraph from L 426-434. The authors should either discuss or experimentally demonstrate an analysis that explains why this might be the case.\n- Clarity: see questions below.\n- The dataset and the code don't appear to be public, and a lot of implementation details are missing from the paper. For example what is the non-normalized gate? Sigmoid?\n- It's unclear how well the dense baseline is tuned compared to the super well-tuned MoE models.\n\n[1] Zoph et al, 2022: ST-MoE: Designing Stable and Transferable Sparse Expert Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)