Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
Authors: Changxin Tian
Kunlong Chen
Jia Liu
Ziqi Liu
Zhiqiang Zhang
JUN ZHOU
Venue: ICLR 2026 Poster (2026)
Keywords: Scaling Laws
Mixture-of-Experts
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We conduct a large-scale empirical study, training over 300 models up to 28B parameters, to systematically investigate the relationship between MoE architectural configurations and efficiency.

Abstract: Mixture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from computational cost. However, this decoupling creates a critical challenge: predicting the model capacity of a given MoE configurations (e.g., expert activation ratio and granularity) remains an unresolved problem. To address this gap, we introduce Efficiency Leverage (EL), a metric quantifying the computational advantage of an MoE model over a dense equivalent. We conduct a large-scale empirical study, training over 300 models up to 28B parameters, to systematically investigate the relationship between MoE architectural configurations and EL. Our findings reveal that EL is primarily driven by the expert activation ratio and the total compute budget, both following predictable power laws, while expert granularity acts as a non-linear modulator with a clear optimal range. We integrate these discoveries into a unified scaling law that accurately predicts the EL of an MoE architecture based on its configuration. 
To validate our derived scaling laws, we designed and trained MoE-mini, a model with only 0.85B active parameters, alongside a 6.1B dense model for comparison. When trained on an identical 1T high-quality token dataset, MoE-mini matched the performance of the 6.1B dense model while consuming over 7x fewer computational resources, thereby confirming the accuracy of our scaling laws.
This work provides a principled and empirically-grounded foundation for the scaling of efficient MoE models.

Published as a conference paper at ICLR 2026
TOWARDS GREATER LEVERAGE: SCALING LAWS FOR EFFICIENT MoE LANGUAGE MODELS
Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang Jun Zhou Ling Team, Ant Group (t (tianchangxin.c tcx, 1 ingyaoo. zzq) @entggruppcco.m com
ABSTRACT
ofxture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from com- putational cost. However, this decoupling creates a critical challenge: predicting the model capacity of a given MoE configurations (e.g., expert activation ratio and granularity) remains an unresolved problem. To address this gap, we introduce Efficiency Leverage (EL), metric quantifying the computational advantage of an MoE model over a dense equivalent. We conduct a large-scale empirical study, training over 300 models up to 28B parameters, systematically investigate the relationship between MoE architectural configurations and EL. Our findings re- veal that EL is primarily driven by the expert activation ratio and the total compute budget, both following predictable power laws, while expert granularity acts as a non-linear modulator with a clear optimal range. We integrate these discoveries into unified scaling law that accurately predicts the EL of an MoE architecture based on its configuration. To validate trained MoE-mine i, model with only 0.85B active parameters, alongside a 6. IB dense model for comparison. When trained on an identical 1T high-quality token dataset, McE -mini matched the performance of the 6. IB dense model while con- suming over 7x fewer computational resources, thereby confirming accuracy of our scaling laws. This work provides principled and empirically grrouud foundation for the scaling of efficient MoE models.
derived scaling laws, we designed and
INTRODUCTION
Mixture-of-Experts (MoE) models (Shazeer al. 2017 Jiang al. 2024: DeepSeek-Al 2024 have emerged as a leading paradigm for constructing large language models (LLMs) (Zhao et al.
Scol
Lary Dons
1.0
14
2.0
2.9
3.8
5.2
6.5
7.7
8.6
12
22
8.3
Bencoency Leveroge
Granuanity
-B
23
9.4
1.4
1.0
15
23
33
1.3
9.5
11.8
13.2
12
18
15
23
33
1.2
9.1
11.4
13.0
100
PLOIP
Connse
100.0 100.0%
50.0%
25.00
12
12.5%
3,1%
1.8%
0.8%
0.4%
Cmne
Act. Ratio
(a) Definition of Efficiency Leverage (EL)
(b) Estimated EL 1e22 FLOPs
Figure 1: Illustaaioon of the Efficiency Leverage (EL) metric for MoE architecture versus dense architecture and its estimated values using Eq. for le22 FLOPs.
"Corresponding author

---

Published as a conference paper at ICLR 2026
2023), primarily due to its remarkable computational efficiency (Clark et al. 2022). By leveraging sparse activation, MoE models can dramatically increase their total parameter count without pro- portionally increasing the computational cost (FLOPs). For instance, DeepSeekMoE (Deepseek-A] et al. 2024), with 16 billion total parameters, activates only 2.8 billion per token, yet achieves per- formance comparable a 7-billion-parameterr dense model, showcasing parameter efficiency gain of approximately 2.5x. However, the decoupling of computational cost from the total parameter count in MoE introduces new challenge in assessing model's capacity, Specifically, neither the total nor the activated parameter count alone serves as reliable proxy for performance of MoE models. Consequently, predicting the effective capacity of specific MoE architecture and setting realistic performance expectations before pre-training remains a critical and unresolved problem. While scaling laws are fundamental for predicting language model performance, their application to MoE models remains fragmented. Prior work has largely studied architectural factors like spar- sity or granularity in isolation (Clark et al., 2022 Ludziejewski et al., 2024 Abbar et al., [2255] Ludziejewski et al. 2025). This leaves a critical question unanswered: how do these factors collec- fively determine an MoE"s s true computational advantage over a standard dense model?
To address this gap, introduce Efficiency Leverage (EL), metric that quantifies an MoE's computational advantage over a dense counterpart. As illustrated in Figure Ia at compute budget CMOR, we define EL as the ratio of training compute budgets a dense model Dense requires match the performance (e.g., identical loss) of an MoE model XMaE: EL D'DDessii CMoE) Cats This definition provides a powerful and intuitive benchmark for MoE architectural comparison: an EL of 5, for example, means an MoE architecture matches the performance a dense model trained with five times the compute budget. Consequently, for fixed compute budget, higher EL directly translates to greater efficiency, enabling larger and more capable models.
-ress.
To build predictive framework for EL., our study follows a three-stage methodology. First, establish fair training conditions by deriving scaling laws for hyperparameters and data allocation in preliminary experiments. Second, we systematically isolate the impact of core architectural dimensions (such as activation ratio, granularity, and shared experts) on EL. Finally, we synthesize these findings into unified scaling law that accurately predicts an MoE configuration's 's EL, offer- ing a practical guide for designing next gextt-eenration efficient models. Applying this methodology, we trained over 300 MoE models up to 28B parameters, using total of 680k 18800eqqivalent GPU- hours. This large-scale effort led us to identify several core principles for optimizing the efficiency of MoE models. Our key findings are:
1. Activation ratio as the primary driver of efficiency. The expert activation ratio emerges as the primary determinant of EL. We observe stable power-law relationship: EL increases as the activation ratio decreases (i.e., as sparsity increases). This reveals that sparsely acti- vated pathways yield consistent and predictable gains in computational efficiency.
2. Expert granularity as non-linear modulator. Superimposed on this primary trend, expert granularity introduces a log-polynomial adjustment to EL. This effect is independent of the total compute budget and implies an optimal range for expert size. Our experiments, loss, identify this optimum to be between 8 and 12.
which utilize a standard
3. Amplifying effect of the compute budget. Crucially, the EL of given MoE architecture is not static; it scales with the training compute budget, also following a power law. This finding underscores the advantage of MoE models in large-scale pre-training, where their efficiency gains become increasingly significant as computational resources expand.
4. Secondary impact of other architectural factors. Other design choices, such as shared experts or the specific arrangement of MoE layers, have a secondary impact on EL,, as they typically possess broadly applicable, near-optimal settings that require minimal tuning.
Synthesizing these findings, we derive unified scaling law for EL. This law integrates the effects of compute budget, activation ratio, and expert granularity, providing a predictive framework to guide efficient MoE design. As a practical demonstration, Figure 1b visualizes the predicted EL landscape under a 1e22 FLOPs budget, highlighting optimal architectural regions.
According to our derived scaling law for EL, predict that an MoE model with 3.1 % activa- tion ratio and a granularity of 12 should achieve an efficiency leverage of over 7x at this compute scale. To validate this prediction, we designed and trained MoR-mine minn (17.5B total, 0.85B active
2

---

Published as a conference paper at ICLR 2026
parameters) against 6.1B dense counterpart 1-trillion-token dataset. The results confirmed our hypothesis: MoE-mini achieved a lower final training loss and slightly outperformed the dense model across downstream tasks. This outcome empirically validates our law's prediction of > 7x efficiency gain. These findings establish our scaling law as a solid theoretical and empirical founda- tion for designing future large-scale, efficient MoE models.
PRELIMINARY
2.1
MIXTURE-OF- TRANSFORMERS.
Total and Active Parameters. We distinguish between a model's total parameters (N), which include all weights (including all experts), and its active parameters (Na). which comprise only the non-expert weights and the subset of experts activated for a given token.
Routable and Shared Experts. An MoE layer contains two types of experts: E routable experts, from which a gate dynamically selects Es per token, and E, shared experts, which are consistently activated for all tokens to process common knowledge.
Activation Ratio and Sharing Ratio. We characterize the expert configuration with ratios that quantify utilization. The Activation Ratio (.A), defined as A = (Eg + E.)/(E + E.), measures the overall sparsity of the MoE layer. The Sharing Ratio (S), defined as (./(E, E.), represents the proportion of activated experts that are shared.
Granularity of Experts. While traditionally the expert dimension (dexpen) was tied to the FFN intermediate size (e.g., 4dmode)) recent work decouples them to explore finer-grained experts. We define Expert Granularity (G) as = 2d dexpen to systematically analyze expert size. A higher G indicates a shift towards more, smaller experts for fixed parameter budget, departing from the conventional practice where despent was tied to the FFN's intermediate dimension (e.g., 4elmustel)
Model Scale in Computation. Following prior work et al. 2024), define model scale (M) as the non-embedding FLOPs per token. This metric provides a fair basis for comparing dense and MoE architectures, it inherently accounts for sparse activation. The total training compute (C) for D tokens is then given by C = M D. We provide the exact calculation for M in Appendix [
2.2 SCALING LAWS FOR MOE OPTIMAL HYPER SEFER--ARAMTERSS
To ensure fair architectural comparisons, we first establish scaling laws for the optimal training hy- perparameters of MoE models. By performing large-scale hyperparameter search over wide range of compute budgets (C), we derived the scaling laws for the optimal learning rate (1799) and batch size (BoPt). Our analysis, detailed in Appendix E.I reveals key distinction from dense mod- els: at larger compute scales, MoE models favor significantly larger batch size and a slightly lower learning rate (Figure 2ap. This phenomenon attributable MoE's sparse backpropagation, where gradients from only subset of tokens in a batch update each expert's parameters. We validated that these derived laws are generalizable across MoE models with varying expert activation ratios (shown in Figure 8). This confirms that our findings provide reliable foundation for exploring diverse MoE architectures under near-optimal training conditions.
2.3 SCALING LAWS FOR MOE OPTIMAL MODEL-DATA ALLOCATION
To achieve compute-optimal training, the allocation of fixed FLOPs budget (C) between model size (M) and data size (D) is critical. We analyze this trade-off for Mixture-of-Experts (MoE) models and compare them against dense models. Our analysis, detailed in Appendix F.2 yields
two key insights: First, consistent with prior work (Bi et al.] 2024:; Hoffmann et al. 2022). the optimal allocation for both MoE and dense models involves splitting the compute budget roughly equally between model and data scaling (i.e., the scaling exponents are close to 0.5). Second, and
Our definition (2d / despen) differs from I Ludziejewski et al. (2024) (Addmdde/ despen) to align with recent models (DeepScck-Al AI 2024 Moonshot-Al 2025). This choice leads to different observed scaling phenomena.
3

---

Published as a conference paper at ICLR 2026
(a) Scaling laws for optimal hyperparameters
(b) Scaling laws for optimal model-data allocation
Figure 2: Scaling laws for optimal hyperparameters and optimal model-data allocation. Blue and red lines represent the fitted laws for MoE and dense models, respectively, derived the same training dataset. Gray circles are the experimental data points used for fitting.
more crucially, at any given compute budget, the optimal MoE model is computationally smaller but trained more data than its optimal dense counterpart (Figure This suggests that MoEs possess greater capacity per parameter, enabling them to effectively leverage larger datasets with smaller model sizes. This finding is particularly significant for training in data-rich but compute-limited scenarios, as it highlights a path toward greater efficiency. The above scaling laws for optimal hyperparameters and optimal model data allocation provide principled basis for model and data selection in our subsequent experiments.
3 EFFICIENCY LEVERAGE: METRIC FOR QUANTIFYING MoE COMPUTE-EFFICIENS
To quantify the computational advantage of MoEs, we introduce core metric: Efficiency Leverage (EL). Intuitively, EL measures how many more FLOPs a corresponding dense architecture requires to achieve the same performance as MoE architecture.
Formally, the EL is defined as the ratio the compute budgets required for a dense and an MoE architecture achieve the same target loss value, L" at the same dataset. Let Lx(C) be the optimal loss scaling function for an architecture X, representing the best achievable loss for a given compute budget C. Consistent with prior work (Kaplan et al. 2020 Henighan al. 2020 Achiam al., 2023), this function is typically modeled as a power law, e.g., (x( axC* +bx. The compute required to reach a target loss L* is therefore given by the inverse function, Cx(L") = L (L*). The EL is then formally expressed
EL(AMoF X L*) Cxocase (L")
CxsLB (T+*)
In our work, we define the target loss L as the loss achieved by the MoE model at its own compute budget, CMoE (i.e., - Lxmoe (OMoE)). This practical choice simplifies the EL function of the MoE architecture, the dense architecture, and the MoE model's 's compute budget:
Xpouse (L (CMoE))
CDense
EL D'Dusss; CMOE)
CMoB
CMoE
where DDnse is the compute budget required for the dense model to match the MoE's loss, obtained by inverting the dense model's 's loss scaling curve.
Our primary goal is to build a predictive model for EL based on key MoB architectural choices. We focus on three critical dimensions that govern MoE capacity: the Activation Ratio (A), Expert Granularity (G), and Shared Expert Ratio (S). Other factors, like the arrangement of MoE layers, have a secondary impact (detailed Appendix F.4 To achieve this, we conduct systematic ablation studies, varying one architectural dimension
time across a range of compute budgets (3 18 to 3 x 1026 FLOPs). Crucially, to ensure fair comparison, our methodology is guided by the preliminary findings in Sections 2.2 and 2.3 For each experiment, we use our derived scaling laws to set the comparable suboptimal dodel/data allocation and training hyperparameters. This rigorous protocol ensures that every architecture is evaluated near its peak potential, yielding robust and reliable results. Full experimental details are Appendix D The following sections first analyze each factor's impact EL individually, then synthesize these findings into unified scaling law.

---

Published as a conference paper at ICLR 2026
4 SCALING LAWS FOR EFFICIENT MOE ARCHITECTURE
To achieve greater leverage, we first conduct an extensive empirical study on the architectural con- figurations of MoE and derive unified scaling laws for efficient MoE architectures.
4.1 EMPIRICAL STUDY ON THE INTERPLAY BETWEEN Loss AND MoE ARCHITECTURE
Our investigation focuses several critical architectural factors: the expert activation ratio (A), expert granularity (G), and sharing ratio (S). For each architectural dimension, we vary it system- atically while holding other factors and the model scale M constant. To ensure fair comparison, all models are trained following the training hyperparameters derived from our scaling laws (Sec- tion 2). Guided by the scaling laws for optimal model-data allocation (Section 2.3 we train each model on over three times its optimal number of tokens. This was done to simulate the overtrained
state commonly observed in real-world scenarios. A detailed analysis and a complete list of trained models are provided in Appendix F and Appendix | respectively.
Expert Activation Ratio (4), We first investigate the activation ratio (A), which governs model sparsity. By varying the total number of experts while keeping the number of activated experts fixed, our IsoFL.OPs experiments reveal a clear power-law relationship: for any given computational budget and any given model scale, training loss monotonically decreases with the activation ratio (Figure Baj. This trend holds consistently down to the lowest ratio tested, 1/128 (088)), demon- strating that greater sparsity yields higher parameter efficiency without an observable turning point. Moreover, this efficiency advantage is amplified at larger training scales, confirming that sparser models are increasingly beneficial high-computation regimes. See Appendix F. for details.
Expert Granularity (G). Next, analyzed expert granularity (G), which defines the trade-off
between employing numerous small experts versus fewer large ones. Our experiments reveal a distinct U-shaped relationship between granularity and training loss, demonstrating the existence of an optimal point that maximizes performance per FLOP (Figure 3b). This optimum proved to be remarkably stable across different compute budget (e.g., G=12 in our tests). This suggests that while overly coarse- coarse-grained experts fail to effective specialization (Deepseek-A Al et al. (024)), excessively fine-grained experts is also often suboptimal. Crucially, we find that routing quality a key factor, as poor load balancing shifts the optimal point toward coarser granularities (details in Appendix F.2
Shared Expert Ratio (S). Our analysis of the shared expert ratio (S) reveals a U-shaped perfor- mance curve, where small but non-zero ratio minimizes training loss (Figure [cp. Furthermore, we identify a subtle scaling trend: the optimal S decreases as the compute budget grows. This leads to a practical heuristic for lcage-scale training (e.g.. > 10 FLOPs): a "one shared expert" design. rep- resenting the minimal effective non-zero ratio, is the most efficient choice (details in Appendix F.3
Other Architectural Factors. We further analyzed two design dimensions to enhance MoE effi ciency: layer arrangement and compute allocation between attention and FFN. We found that incor- porating dense layers in the early stages of MoE has minor impact on efficiency but helps mitigate routing imbalances and reduces overall parameters. For compute allocation, allocating 30%-40% of FLOPs to the attention mechanism achieves optimal or near-optimal performance, with minor impact outside this range. Detailed results are available in Appendix F.4
4.2 SCALING LAWS FOR MoE EFFICIENCY LEVERAGE
Based on the empirical study in Section[4.1 shared experts and other design factors have a secondary impact on EL, as they typically have robust, near-optimal settings. Therefore, we focus on deriving parametric scaling law for EL as a function of activation ratio A, granularity G, and FLOPs C.
4.2.1 UNIVARIATE SCALING LAWS FOR EFFICIENCY LEVERAGE
To systematically analyze each core architectural dimension, vary it while holding the others and the total compute budget (i.e., FLOPs per token, M) constant. This controlled approach is essential, as full combinatorial exploration would be prohibitively complex and unaffordable.
5

---

Published as a conference paper at ICLR 2026
2.12
180
Stared
(a) IsoFLOPs curves for varying A (b) IsoFLOPs curves for varying G (c) IsoFLOPs curves for varying S
Figure 3: Impact of MoE architectural choices on performance. (a) Activation Ratio (A): At fixed compute budget, loss monotonically decreases with a lower activation ratio. The advantage of sparsity is magnified at scale. (b) Expert Granularity (G): A U-shaped relationship between granularity and loss reveals an optimal point (marked by orange stars) that maximizes efficiency. (c) Shared Expert Ratio (S): A U-shaped loss curve shows that a low, non-zero S is optimal.
Our procedure for deriving the scaling law for each architectural dimension follows three stages, detailed in Algorithm []] First, we generate dataset of "(compute, loss)' pairs by training
suite of MoE models (Tables T and their dense counterparts. Second, we fit these points to loss scaling curves for each architectural setting. From these curves, we compute the EL for various MoE architectures and FLOPs budgets, as illustrated in Figures 10b 1lb and [nd] Finally. we collect the resulting EL values from different settings and use them to derive the univariate scaling laws for activation ratio A, granularity G, and FLOPs C, presented in Figure
Interaction of Efficiency Leverage and Activation Ratio. Our preceding analysis identifies the activation ratio (A) as the primary factor influencing EL. As illustrated in Figure 4al reducing the activation ratio (i.e., increasing sparsity) consistently yields substantial efficiency gains, following similar power-law relationship across all FLOPs budgets. This leads us to hypothesize: for a given FLOPs budget and granularity, there exists a power-law dependence between EL and activation ratio.
log (Lc = a log Å, i.e. ELC,o(A) = AAa,
1
where
À A + (1/Astart 1/Amazz + Amax
where Á a saturating transformation of A. as defined in Clark et al. (202)) and we set the lower bound of meaningful activation ratio O. Clearly, when A = 1. have EL 1. indicating that the EL of the dense model is 1, which satisfies the dense equivalence. We fit Eq. to the data for each compute budget, and the resulting predictions (dotted lines in Figure 4a) align well with our observations. Notably, the fitted exponent a is not constant. It increases as A decreases, indicating a diminishing benefit from increased sparsity, consistent with prior work (Clark et al.) 2022]. Furthermore, 0 also increases with the compute budget C. suggesting greater leverage for larger models. We will analyze the relationship between FLOPs and EL in the following paragraph.
Interaction of Efficiency Leverage and Expert Granularity. As previously observed, an optimal expert granularity exists that maximizes the EL. Thus, we hypothesize that for fixed FLOPs budget C' and activation ratio A, the relationship between EL and G follows a log- tog-polynomial pattern:
log ELC.A(G) =ag + by (log G (log G + cG)),
where ag is the -ndelettt base EL. representing the theoretical value when granular- ity is 1. by controls the strength of the curvature the relationship between EL and granularity, reflecting the sensitivity of the model architecture to changes in expert granularity. eg determines the position of the optimal granularity that maximizes EL. We fit Eq 2 to each FLOPs budget and plot the predictions for varying granularity dotted lines in the Figure 4b As shown, the curves under different FLOPs budgets are highly similar (i.e., with similar values of by and cq), indicating that the impact of expert granularity MoE efficiency is consistent across various compute budgets.

---

Published as a conference paper at ICLR 2026
An 6%
Au6 3%,
18422
A- 100%
1.00 1.50
Gromubrity
(a) Scaling with Activation Ratio
(b) Scaling with Granularity
(c) Scaling with Compute Budget
Figure 4: Scaling behavior of efficiency leverage (EL.) (a) With fixed granularity (G 2), EL follows power law with respect to activation ratio A across all tested compute budgets (C). (b) With a fixed activation ratio (A = 3.1%). EL's scaling with granularity conforms to a log-polynomial law across all compute budgets. (c) With both activation ratio (A) and granularity (G) held constant, EL scales with compute according to a standard power law. Points below 3e20 FLOPs represent experimental data, while points beyond this threshold are predictions extrapolated from scaling law.
Interaction of Efficiency Leverage and Compute Budget. Based the analysis presented in Section 44. and Section 4.2.1 I] we observe that efficiency advantage MoE increases as computational budget grows. To formalize the relationship between the FLOPs budget and EL, we assume standard power-law pattern as follows:
log (C),o(C) ac log cC, i.e. EL),c(C) exp(ce) C
where ac reflects the scaling capability of MoE efficiency with respect to the compute budget under given configurations A and G. We collect the values of the EL corresponding to different model architectures under the granularity setting of 2, and fit Eq. to each architectures. The predictions for varying granularity are plotted as dotted lines in the Figure The results indicate that all tested MoE architectures show trend of higher EL as the FLOPs budget increases, demonstrating the potential of MoE large-scale pre-training.
Our choice for each univariate scaling law is justified by a goodness-of-fit comparison against sim- pler alternatives. The specifics of the comparative analysis are presented Appendix N
4.2.2 JOINT SCALING LAW FOR EFFICIENCY LEVERAGE
Based the preceding observations and univariate scaling laws, identify three key insights:
The activation ratio (or sparsity) is the primary driver of MoE efficiency, establishing foundational power-law relationship.
Building upon this power law, expert granularity imposes a non-linear adjustment that op- erates independently of the compute budget.
Furthermore, the efficiency advantage of MoE over dense models is amplified by the com- pute budget C through the power-law pattern.
To unify these interconnected effects, we propose the following joint scaling law for EL:
BI(A,G,C)
(4)
where a = a + d'log C' is the compute-dependent exponent that captures the primary power-law relationship between EL and FLOPs ratio. The term represents the base scaling exponent at reference compute budget, while is a positive constant that quantifies how the EL is amplified by larger compute budget C. The parameters /3 and y model the non-linear impact of granularity G. This quadratic form in log G directly reflects the log-polynomial pattern observed our initial analysis, capturing the existence of an optimal granularity.
4.2.3 FIT AND VALIDATION
To validate the proposed scaling law for EL, we fit Eq. Ausing Huber loss and the BFGS optimization algorithm (Hoftmann et al.) 2022). We use data points with an EL factor below training.
7

---

Published as a conference paper at ICLR 2026
while those reserved validation set. As depicted in Figure [] the resulting model achieves R of 0.9858 and demonstrates strong predictive power. This is evidenced by low RMSE both the training set (0.2169 over 200 points) and the validation set (0.5275 over 24 points). The quality of the fit further corroborated by the residuals, which are ap- proximately normally distributed and centered at zero (mean = -mean=-0.0273, std. = 0.28033. The fit- ted coefficients and more detailed goodness- of-fit analysis can be found in Appendix G and Appendix N.4 respectively.
enticetcccc Levergee
Figure 5: Validation of the Scaling Laws for Ef- ficiency Leverage. We fit Eq, 4 to the data points with efficiency leverage of less than 6, using the remaining points as the validation set.
The alignment between the scaling law and both the training data and validation set pro- vides strong empirical support for the proposed relationship. More importantly, the scaling law exhibits remarkable extrapolation capabilities, as accurately models performance trends for high-leverage validation points outside the training range. These results confirm that Eq, efffctivvly captures the underlying interaction between MoE archi- tecture and EL.
Furthermore, we select le22 FLOPs compute budget, and apply our fitted scaling laws to predict EL across various MoE configurations. As shown in Figure our analysis predicts that EL exceeding 7x can be achieved at budget of le22 FLOPs with an activation ratio of 3. 3.1% and a granularity of 12. This claim is experimentally validated in the following section.
MoE-MINI MORE EFFICIENT MOE LANGUAGE MODEL
To validate the scaling laws derived in Section we designed a new MoE model, MoE-no---i- mini,, con- figured with architectural parameters predicted be highly efficient. It features total of 17.5B parameters but only 0.85B active parameters, achieved through a granularity of G = 12 and a low activation ratio of A 3.4%. Referring to Figure [ at the 1e22 FLOPs compute budget, we hy- pothesize that MoE MoE-mini achieves more than 7x in compute-efficienc leverage over a comparable dense model. Concurrently,
train traditional dense model with 6.1 billion parameters (named "Dense-S 5 1B") for comparison. This section presents detailed analysis of the performance dif- ferences between MoE-mini and the conventional dense model Dense-6. 1B, highlighting that the active parameter count, training costs, and downstream inference costs of Dense-68 6. 1B are more than seven times those of MoF-mi ni. The architectures of Mo given in Table I while ther detailed architectures and training setting provided in Appendix D
and Dense-6. 6. B are
Table 1: Detailed Architectures of MoE-mini and Dense- 6. 1B for Comparison.
Model
Thayers
dmodel
dyfn
despert
Thhads
Thk bead
E
Eg
E,
N
Na
Dense-6. 1B
28
4096
14336
32
8
6.11B
6.1
McE-mini (A0.8B)
20
2048
5120
384
16
4
384
12
1
17.5B
0.85B
5.1 TRAINING DYNAMICS
The Dynamic of Training Loss The training loss curves for MoE-mini and Dense-6. 6. 1B, shown in Figure 6al illustrate a clear difference in their convergence behavior. The dense model exhibits faster convergence during the early training phases, indicating an aptitude for rapid initial learning. In contrast, MoE-mini loss decreases more gradually at the start. However, over the full course of training, MoE minemin steadily improves and ultimately achieves performance level com- parable to that of the dense model, highlighting its ability to reach high performance with sufficient training. Focusing cluding stage, the performance gap between McE McE-mini and Dense-6 6 1B narrows to a negligible difference of about 0.01 in loss value. This confirms that MoE-mi -ii ní can nearly match the dense
the final 100 billion tokens of training provides further insight. In this
8

---

Published as a conference paper at ICLR 2026
300
Entrrieer Nocriec
1.00
(a) Dynamic of Training Loss.
(b) Dynamic of Benchmarks.
Figure 6: Dynamic of Training Loss (left) and Benchmarks (right).
model's effectiveness while operating with significantly fewer computational resources. Crucially, this near-equal performance underscores MoE-minn'ss ability to deliver over 7x gains in training efficiency, making it highly cost-effective and powerful alternative for large-scale pre-training.
The Dynamic of Benchmarks Throughout training, MoE-mini and Dense-6. 1B demon- strated remarkably synchronous performance gains on standard benchmarks, shown in Figure 6b The data reveals clear and consistent trend: the two models improved almost synchronously. At no point during training did one model show decisive or lasting advantage over the other. This lockstep progression continued until the end of the training cycle, where they posted nearly identi- cal final scores on the evaluation leaderboard. This synchronous dynamic and convergent outcome suggest a fundamental parity in their learning efficiency and final performance ceiling under our experimental conditions.
5.2 EVALUATION
Evaluation Benchmarks To provide holistic assessment our model's capabilities, we evaluate it on a diverse suite of downstream benchmarks. These tasks are grouped into five key categories: General Knowledge and Reasoning, Language Understanding. Professional Knowledge, Math, and Code. A detailed list of all benchmarks used in each category is provided in Appendix
Evaluation Results The comparative evaluation, summarized in Table reveals that McE-mi n achieves a superior overall score of 45.5, outperforming Dense-6. 1B's 44.0. This result demon- strates that McE-mine tute only about 13% of its competitor's during inference, it strikes an exceptional balance between performance and efficiency. Beyond the overall average, MoE-mini demonstrates consistent ad- vantages across most key domains, including reasoning, language understanding, code generation, superiority is particularly pronounced tasks requiring high cod-
achieves a "small yet powerful" feat: while its activated parameters consti-
and advanced mathematics. ing proficiency and deep contextual understanding. While there are minor variations specific benchmarks, the general trend confirms its strong potential in solving complex problems. This result validates that MoE-no MoE-mini achieves an impressive 7x efficiency leverage, delivering perfor- mance comparable to a B. dense model that uses over 7 times the active parameters. detailed, comparison is provided in Appendix
Table Performance comparison MoE-mini (17B-A0.8B) and Dense-66 1B.
Model
General//easoning
Professional
Language
Code
Math
Overall Avg.
Dense-6. 1B
55.8
44.0
69.2
36.9
32.9
44.0
MoE-mini (A0.8B)
56.2
44.7
71.6
39.8
34.7
45.5

---

Published as a conference paper at ICLR 2026
6 RELATED WORK AND DISCUSSION
We provide a broader survey of related work in Appendix B and compare our findings with key prior studies in Appendix[[] Our work formulates scaling laws in terms of Efficiency Leverage (EL., diverging from prior loss-centric studies. This EL-based approach offers a more direct and practical framework for understanding MoE efficiency for two key reasons: 1) EL directly quantifies an MoE's compute advantage. Unlike absolute loss, which is dataset-specific and hard to interpret, EL provides generalizable architectural insight. 2) EL can Simplify Model Selection. Instead of fitting multiple complex loss functions, practitioners can use our scaling laws to directly compare the efficiency of different MoE configurations. This dramatically simplifies architectural design choices. In short, while traditional laws predict what the loss will be. formulation quantifies how much more efficient an MoE architecture is, offering actionable design guidance.
LIMITATIONS AND FUTURE WORK
Our study has four primary limitations, which also point to valuable directions for future research.
First, following standard practice (Clark et al. 2022 Kaplan et al. 2020 Hoffmann et al. 2022), we measure computational cost in theoretical FLOPs. This aagnost-- metric overlooks practical wall-clock effects (communication, memory, kernel efficiency, parallelization). Our work thus establishes a theoretical upper bound on efficiency, a necessary first step before optimizing for real-world costs. Second, to make a systematic analysis feasible, we assume that MoE architectural factors are independent. This allowed us to pragmatically study each factor isolation and syn- thesize the results into a unified law. However, this approach may overlook interaction effects that could unlock further optimizations. Third, due to resource constraints, we applied a single hyper- parameter scaling law to all MoE models, regardless of their sparsity. While effective, developing a sparsity- aware hyperparameter law that tailors settings to each model is a promising avenue for future work. Fourth, our scaling laws foeus compute budget rather than its allocation between training data and model size. Establishing a Chinchilla-tike scaling law in term model size and dataset size for MoEs' efficiency to guide this trade-off is an important next step.
Despite these limitations, our findings confirm the significant potential of MoE models, which pro- vide clear path toward more capable and efficient models in terms of theoretical compute cost.
8 CONCLUSION
In this work, we introduce Efficiency Leverage (EL), a metric quantifying an MoE model's 's computa- tional advantage over a dense counterpart, to analyze how architectural choices govern performance. Our large-scale study of over 300 models reveals that MoE efficiency follows predictable principles: EL scales as a power-law with activation ratio and compute budget, while expert granularity has non-linear effect with distinct optimal range. Other factors, like shared experts, have ondary impact. We unified these principles into a single scaling law that accurately predicts MoE efficiency. To validate it, we designed 0.85B activated parameter MoE model which, predicted, achieved over 7x efficiency leverage, confirming our law's 's robust predictive power. For future work, our framework can be extended in several key directions: (I) Incorporating memory constraints and communication overhead into the EL framework, particularly for distributed training scenar- ios where these factors dominate practical efficiency. (2) Developing a unified metric that balances training compute with inference latency, enabling end-to-end efficient architecture co-design. We hope this work inspires continued innovation in MoE. architectures toward greater leverage.
USE OF LARGE LANGUAGE MODELS
During the preparation of this work, used LLMs (e.g., GPT-5 and Gemini-2.5-pp -pro) to assist with editing and polishing the manuscript for clarity and readability. Furthermore, the plotting code for the figures presented in this paper was generated with the assistance of these models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-cxperts language models. arXiv preprint 250iv: 12370, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv2303, 08774, 2023.
Joshua Ainslie, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv::305. 1.005.11245 13245, 2023.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models fill in the middle. CoRR, doi: 1044550ARXXW.2207.422. 48550/arXiv 2227.14255
hos/2207.14255, 2022.
URL hww.t:tt////// //dii http://oooooogo..oooo/o
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richard- son, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315, 2021. htthttps: //tpxvaa.vvvvv/aa///// httg/abs//nt 2ww.. 03315
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek IIm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0259 102954, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020. The Thirty-Secone Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artifi-
cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020. doi: 10.1609/AAALV334005. 024105.6239
URL doi . 1609////a11.0.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374 2021. URL httl:tttss//aaataa org/ abs/ 03374
Yangyi Chen, Binxuan Huang, Yifan Gao. Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance 1lms. arXiv preprint arXiv:2410.08527, 2024.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057- 4086. PMLR, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14416. 2021. URL https://erxins /ttps://arxivv.or//abb///////// wtt. 14168

---

Published as a conference paper at ICLR 2026
DeepSeek-AL. Deepscek-v3 technical report, 2024. http:://aw....../...////o/// 19437
Deepseck-AL, Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Ji- ashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06006 2024.
Samir Yitzhak Gadre, Georgios Smymiss, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mereat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08544 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:://////. F Ffpg52swvg
httpt::/oppeeeeeeeeeeee http///ppeeeiieeeeeee/eeeeeeeewe net forum?id
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021a. URL htttttts///oeeeeeeeeee
formm ceerrmmmiidddeee d7k7 jmI 3Gm)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1. NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. URL htwppppppp..cppppe////011.. pttp://ppppp///////////
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, el. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, al. Predicting emergent abilities with infinite resolution eval- uation. arXiv preprint arXiv:2310.03262. 2023.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06399 2024.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi multi-discipline chinese evaluation suite for foundation models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 files/paper/
16. 2023, 2023. URL http: /ttt:://papprs.. wwwps.cc/ppaee Abssttttad--bbbatttttacaaacaa -at and
Benchmarks, html
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance in machine translation. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Selar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL httphhttss::/pppeeeeet
net htt.rr/eeeeeeeeettoooooooooooooo Foo.mmmid=chfuuC3111
Albert Q QJiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 040088 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 003361, 2020.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Se- bastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 785- 794. Association for Computational Linguistics, 2017. doi: 10.18653/11/117-0083 URL http:::/doi..rg//// http:://ll..ccy....1855////////
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding chi- nese. In Lun Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associa- tion for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. (1260-11285 Association for Computational Linguistics, 2024. doi: 10. 1865INN//024FFIIDIIGGSSS ACL.671. URL
Findings-acl. 671
Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part opttiaa hy- perparameter scaling law in large language model pretraining. arXiv preprint arXiv:2503.04715, 2025.
Seng Pei Liew, Takuya Kato, and Sho Takase. Scaling laws for upcycling mixture-of-experts lan- guage models. arXiv preprint NXiv:2502.00009, 2025.
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang. Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and ap- plication proficiency of Ilms with a hicrarchical mathematics benchmark. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting. August 1 11-16, 2024. pp. 1884-6915 Associ- ation for Computational Linguisties, 2024. doi: 18.13/ 1AA.S////222..NIINNNAAAALLL https://doi. orgg /10. 186118663//11 2016553vv//// wll.ll/22222 /224 firdiiiginss.... acl 411
URL
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://
@penreview.nee forum iid=iiddi 1id=1qvx610Cuu
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X10 2017.
13

---

Published as a conference paper at ICLR 2026
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždż, Piotr Sankowski, et al. Scaling laws for fine-graine grained mixture of experts. In Forty-first International Conference Machine Learning, 2024.
Jan Ludziejewski, Maciej Pióro, Jakub Krajewski, Maciej Stefaniak, Michat Krutul, Jan Matašnicki, Marek Cygan, Piotr Sankowski, Kamil Adamczewski, Piotr Mitos, et al. Joint moe scaling laws: Mixture of experts can be memory efficient. arXiv preprint 20X10 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' aunichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381 23911 Association for Computational Linguistics, 2018. doi: 10.18653/V1/D11 11260
URL http::tps://.m //tpi. www.i.orgg100 105.1011/////1/1111100
Moonshot-Al. Kimi k2: Open agentic intelligence, 2025. URL htttps:/moonshotte
github. ic/Kimi K2/
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Jacob Morrison, Sewon Min, Wei- jia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-ccperts language models. arXiv preprint arXiv:2409.02061 2024.
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varsh- ney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. arXiv preprint arXiv:2406.17J60 2024.
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation bench- mark for cross-lingual natural language generalization. In Nicoletta Calzolari, Min-Yer Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 8383-8394, ELRA and ICCL, 2024. URL https: http:/aaaannoooooooooo /
wrg//02g//.0. www.-meee......... 735
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-leve. google-proof q&a benchmark. CORR, abs/2311. 2023. doi: 10.48550/RXXV.22111 12022. URL https: wtt..o/./111..5.0000.
12022
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405. 2024.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023. URL htttts::/ppeervv.tt wwt.eeewwwweeeeeeeeeeeee.e..e.ee ftt//formm? d=d-d==VXxxxx22
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimall Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.0044 2023.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.072 2015.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-experr layer. arXiv preprini arXiv:/701 70170051138888 2017.
Freda Shi, Mirac Suzgun, Markus Freitag. Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual chain-of-thooe -fthouggtt reasoners. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
ew net. forum? d= FR3 TXp
14

---

Published as a conference paper at ICLR 2026
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, al. Hunyuan-large: An open- source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:241 I. 0.2265, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether
solve them. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics; ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13003-13051 Association for Computational Linguistics, 2023. doi: 10. 10. 10..8653/v1/202233 findinggg acl. 824
ACL.88 URL https: //o11 .rg/
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqar question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: 2018.
Ning Tao, Anthony Ventresque, Vivek Nallur, and Takfarinas Saber. Enhancing program synthesis with large language models using many-objective grammar-guided genetic programming. Algo- rithms, 17(7):287, 2024. doi: 10.3390/AA17770287 URL pttps://dci.ooe a17070287
Siqi Wang, Zhengyu Chen, Bei Li, Keqing He, Min Zhang, and Jingang Wang. Scaling laws across model architectures: A comparative analysis of dense and moe models in large language models. arXiv preprint arXiv:2410.05661, 2024a.
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He. Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang. Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and chal- lenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference
Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL http: /tctp://paper..ee wipperrs..ippi.oomm cc/ www.aais...pppaatiiiiee http:ppper_fifeeeeee fitperrrriiles/pap///22000 Dww.aaattattsaeae _ndddd____d______
Ben.hmarks brack.htnm
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. CMATH: can your language model pass chinese elementary school math test? CoRR, abs/2306. 16636, 2023. doi: 10.48550/ARXIV. 2306.16636. URL httpts:///ott.
org/10...o.ooooooooo 48550/ arxiv. 2306 16636
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4791- 4800. Association for Computational Linguistics, 2019. doi: 10. D0.18653/V1/P19-1422 URL http:::/abi.ooog11..1118888 wtt.:///....1133111111
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, 122.242055.2244,,. 2023. doi: 10.48550/ARXV..2005.. 12474. URL https:://do/ 1tt.b/.......1.....///..../.. 2305.
12474
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. et al. A survey of large language models. arXiv preprint 3xX.x:2333311222 1(2), 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foun- dation models. In Kevin Duh, Helena Gómez-Adorno and Steven Bethard (eds.), Find- ings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico,
15

---

Published as a conference paper at ICLR 2026
June 16-21, 2024, pp. 10. SIN.NIINNNNNNNCL114999 URL 2024.
Association for Computational Linguistics, 2024. doi:
149
Barret Zoph, Irwan Bello, Sameer Kumar. Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022.
A NOTATION
To aid readabilit,, provide list of key symbols used throughout this paper.
Table 3: Notation.
Symbol
Description
Number of routable experts.
E.
Number of activated experts.
E.
Number of shared experts.
Number non-vocabulary parameters.
Na
Number of activated parameters.
dmodel
Model hidden dimension.
desport
Expert hidden dimension.
Total training compute FLOPs
M
Compute (w/o embedding) per token in FLOPs.
D
Dataset size in tokens.
A
Activation ratio, ie.. (Fa F../((F F.). Granularity of experts, i.e., 2dddes:/deqper
Shared expert ratio, i.e., Ex/ + E.)
B RELATED WORK
B.1 SCALING LAWS FOR LANGUAGE MODELS
Scaling laws provide framework for understanding and predicting the performance of language models under varying conditions. Kaplan et al. (2020) laid the foundation by demonstrating that model performance adheres predictable power-law relationships involving model size, dataset size, and compute budget. Building on this, Hoffmann et al. (2022) introduced the Chinchilla scaling laws, highlighting the importance of balancing model size and training data volume for compute- optimal training. They showed that scaling model size without a corresponding increase in data leads to diminishing performance gains. [Sardana et al. (2023) advanced this understanding by corporating inference costs into compute-optimal frameworks, proposing strategies for optimizing performance under fixed inference constraints. Additionally, [Bi et al. (2024) emphasized the crit- ical role of data quality, demonstrating that higher-quality datasets enable more efficient scaling, particularly with larger models. Recent advancements have applied these scaling laws to various specialized areas. For example, hyperparameter optimization has been explored in the context of scaling laws (Bi et al. 2024 Li et al.) 2025), while Gadre et al. (2024) investigated the phenomena model performance. Furthermore, scaling laws have been alyzed for their impact on downstream task performance across range of applications (Chen et al. 2024 Ruan al.] 2024 [sik et al. 20255: Hu al.) (2023] Grattafiori al. [2024] Li et al.) (025). underscoring their adaptability and relevance in addressing both theoretical and practical challenges language modeling.
of over-training and its implications
B.2 SCALING LAWS FOR MIXTURE-OF-EXPEERTS (MoE)
ofceurr--o-Exppeee Experts (MoE) models (Shazeer al. 2017 Lepikhin al. 2020 have emerged as a powerful architecture for language modeling, primarily due to their ability to decouple computa- tional cost from parameter count. Recent research has further explored optimizations within the MoE
Mixture-of
16

---

Published as a conference paper at ICLR 2026
paradigm. For instance, DeepSeekMoE (Deepseek-AL et al. 2024 investigated the impact of fine- grained expert settings on model performance. proposing a novel design that incorporates shared experts and hybrid structure combining dense layers with MoE layers. Complementing this, ZZoph et al. (2022) highlighted that the performance gains from increased sparsity diminish significantly once the number of experts exceeds 256, suggesting practical limit for highly sparse models. With the widespread adoption of the MoE architecture, the scaling laws governing MoE models have been extensively studied. Early work by Clark et al. (2022) examined scaling by varying model size and the number of experts on fixed dataset, concluding that routed models offer efficiency advantages only up to a certain scale. This analysis was subsequently extended by [Ludziejewski et al. (2024), who incorporated variable dataset sizes and explored the effects of expert granularity. Additionally, Wang et al. (024a) investigated the transferability and discrepancies of scaling laws between dense models and MoE models. [Abnar et al. (2025) advanced this line of inquiry by deriv- ing scaling laws for optimal sparsity, explicitly considering the interplay between training FLOPs and model size. They also analyzed the relationship between pretraining loss and downstream task performance, noting distinct behaviors between MoE and dense models on certain tasks. More re- cently, Ludziejewski et al.] (2025) derived joint scaling laws applicable to both dense Transformers and MoE models, demonstrating that MoE architectures can outperform dense counterparts even under constraints of memory usage or total parameter count. [iew et al. (2025) derive empirical scaling laws for upcycling LLMs to MoE models, relating performance to both dataset size and architectural choices.
C COMPARISON WITH PREVIOUS WORKS.
Comparison with Clark et al. 2002).. Clark et al. (022)) used fixed dataset and concluded that the efficiency of MoE models over dense models diminishes beyond a certain scale. In contrast, our results (Figure 10) demonstrate that MoE models consistently more compute-effiiiet across all scales we tested. The discrepancy may lie in their experimental design: using a fixed dataset. As our scaling laws establish (Section 2.3). MoE models require proportionally more training data than dense models for compute-optimal training. A fixed dataset therefore systematically under-trains MoEs, leading to an unfair comparison and flawed conclusions. Our convergence curves (Figure (aa) and findings from Ludziejewski et al. (2024) confirm this: MoEs, despite a slower start, eventually surpass dense models. Unlike prior work, we follow scaling laws to allocate resources, dynamically scaling training tokens with compute. This ensures the fairness and reliability of our comparison.
Comparison with Ludziejewski et al. (2024). Our findings on expert granularity differ from Ludziejewski et al. 2024) in two key ways. First, we find log-polynomial relationship suggesting an optimal granularity, not their reported monotonic trend where finer is always better. Second, our MoE's efficiency loss (EL) is typically under 10x, substantially lower than their reported ; 100x "Rel- ative FLOPs to train equivalent Transformer". These discrepancies stem from three core differences in experimental design: (1) Granularity definition: Our definition (G = 2d 2dmodet/dexpaa) dexper). aligned with leading models (DeepSeek-A] 2024 Moonshot-Al] 2025). uses experts half the size of theirs at the same nominal granularity. This allows us to test truly finer spectrum. (2) Hyperparameter strate- gies: We optimize hyperparameters for each compute budget, unlike their fixed-setting approach, which is crucial for fair comparison as optimal settings vary with scale (Section 2.2 (3) Base MoE architectures: Our MoE uses a denser activation ratio (1/32 vs. their sparser 1/64). Their inherently more efficient baseline may inflate their reported gains. In summary, our differing conclusions arise from exploring finer granularity spectrum under fairer, optimized training conditions.
Comparison with Abnar et al. (2025). While our findings align with Abnar et al. (0225) on the principle that larger, sparser models perform better under fixed compute budget, our work extends theirs in two crucial ways. First, methodologically, we optimize training hyperparameters and sys- tematically analyze architectural factors like expert granularity, uncovering its log-polynomia effect on performance. Second, and more importantly, our primary contribution is the derivation of a novel scaling law for the efficiency leverage of MoE models over their dense counterparts, rather than for loss. This law's key advantage is its independence from specific datasets. It directly quantifies the relationship between MoE architecture and relative efficiency, yielding more generalizable and actionable principles for model design.
17

---

Published as a conference paper at ICLR 2026
Comparison with Ludziejewski et al. (2025). Our work and Ludziejewski et al. (2025) are com- plementary, as we investigate different aspects of MoE scaling laws. We focus on optimizing ar- chitectural parameters (i.e., granularity, activation ratio) within a fixed compute budget and model scale. They, in contrast, determine the optimal allocation between model size and data volume under both compute and memory constraints. While also explored model-data allocation, our analysis was intentionally limited. Its purpose was not to derive a comprehensive allocation strategy, but rather to establish that MoE and dense models have fundamentally different resource needs. This foundational insight was critical, justifying our approach of providing ample, near-optimal training budgets ensure fair and reliable comparison across all models in our main experiments.
Reconciling Findings Shared Expert Effectiveness with O et al. 2024). Contrary to the findings of Muennighoff et al. (2024) with OLMoE, our scaling law analysis sug- gests shared expert generally beneficial. We attribute this discrepancy primarily to our broader scope of analysis and distinct model architecture. While the OLMoE conclusion stems from a single data point, ours derived from a trend across numerous models and scales. This broader perspec- tive reveals that although specific configurations in our study perform best without shared expert (e.g., M-2e9, Figure 3c) aelaaiggg with OLMoE's observation- the dominant trend favors its use. Furthermore, our 256-expert architecture features significantly higher sparsity and finer granularity than OLMoE's, key structural difference that, along with varying training parameters, can alter its impact. Therefore, we conclude that while its benefit is context-dependent, a shared expert is robust choice from a general scaling perspective.
D EXPERIMENTAL SETUP
Architecture and Tokenizer We adopt Grouped Query Attention (GQA) (Ainslic et al. 2023) architecture based on the standard decoder-only Transformer, consisting of an embedding layer, multiple alternating layers of attention mechanisms and feed-forward networks, and final de- embedding layer. Additionally, we use the BPE (Byte-Pair Encoding) algorithm (Sennrich et al.) 2015) and RoPE (Rotary Positional Embedding) (Su al. 2024) to handle positional information. The vocabulary size is 126,464, and the sequence length is 4,096.
Expert Routing Strategy In our MoE layers, routing network assigns each token's hidden state by to the top-Na experts. This is achieved by generating gating scores 9r = Softmax (Wg ht), where W, is a learnable matrix. The final output is weighted sum of the selected experts' outputs: = EieTop ) Dt.i Eh,),,, where E, is the i-th expert in total N experts. To ensure balanced expert utilization and stable training, we incorporate two standard auxiliary losses: a load balancing loss (Lepikhin al.) 2020)) (coefficient of 0.01) encourage uniform token distribution, and router z-loss (Zoph et al. 2022) (coefficient of 0.001) to regularize the magnitude of the gating logits.
Optimizer and Scheduler tion with a standard deviation 0.006 and optimized using the Adam optimizer (Loshchilov & 81 0.9 and 83 0.95, with 0.1 weight
The parameters of experimental models are initialized from a distribu-
Hutter 2017). The optimizer's hyperparameters are decay applied. The learning rate schedule employs a WSD wwarmup-stable-deeee
strategy (Hu) al. 2024): the first 1% of training steps use linear warm-up, followed by exponential decay that reduces the learning rate to 10% of its peak value.
Pre-training Data The training data is soureed from a large-scale multilingual corpus, primarily covering English and Chinese, while also including various other languages. This corpus encom- passes web text, mathematical materials, programming scripts. published literature, and diverse textual content. To validate model performance, we extracted 2T-token subset from this corpus for training. In Table we present the composition of the training datasets for all experiments. Unless otherwise specified, this configuration is used throughout.
18

---

Published as a conference paper at ICLR 2026
Table 4; Pre-training data composition.
Type
Web
Books
Wiki
Academic
Code
News
Social
Domain
SFT
Math
Exam
Ratio
46.0%
5.0%
4.0%
6.0%
25.0%
0.1%
1.9%
1.0%
4.0%
6.0%
1.0%
Other Training Configurations Our implementation built Megatron-LM and employs hybrid parallel strategy combining Expert Parallelism (EP), Tensor Parallelism (TP), and Pipeline Parallelism (PP). We utilized b6 precision for all forward and backward passes maximize throughput, while maintaining float32 for the master weights and optimizer states to ensure numer- ical stability. The representative parallelism configurations in our experiments are as follows.
E
DETAILED PRELIMINARY EXPERIMENTS
E.
SCALING LAWS FOR MOE OPTIMAL HYPER- PARAMEERS
The performance of MoE model is sensitive to its hyperparameters. To ensure that our subsequent architectural comparisons are reliable, it is crucial to evaluate each configuration under its optimal hyperparameter settings. Therefore, we first conduct a preliminary study to establish the scaling laws for optimal MoE hyperparameters. Previous research (Bi et al.) 2024) has established that the optimal hyperparameters primarily function of the total computational budget. Accordingly, we performed a hyperparameter search across a compute range of 3e17 to 3e20 FLOPs, using a Sarmuu-SStabl-DDa
(WSD) learning rate schedule (Hu et al. 2024). We trained multiple models, varying both learning rate and batch size, which were sampled from log-base-2 grid. Specifically, the exponents for the learning rate ranged from -11 to -9.0, and for the batch size, from 18 to 21. To make this analysis tractable, we initially fixed the MoE configuration to one with 64 experts, of which are activated per token, plus an additional shared expert (resulting in an activation ratio A 7.8% and granularity G 2). Detailed settings of the experimental models are available the Appendix D We then verified that the conclusions from this configuration generalize across different activation ratios.
Figure illustrates the fitting process. To ensure robustness, we identify "near-optimal" configu- rations those achieving loss within 0.25% of the minimum for given compute budget. After removing outliers, we fitted the optimal batch size, BTP, and learning rate, 17tHH, against the compute budget C. The resulting scaling laws reveal clear trends: ddp increases and rySPi decreases with larger C. The final formulas obtained from the fitting process are as follows:
17 1.1576 0
(5)
Bap = 0.0694 C..364
key finding emerges when comparing these laws to those of dense models. As shown in Figure MoE models favor significantly larger batch size and a slightly lower learning rate large compute scales. This phenomenon is attributable to MoE's sparsity: during backpropagation, each expert's s parameters updated using only subset of the tokens in batch, whereas dense parameters receive gradients from the entire batch (Sun et al. 2024).
To validate the generalizability of these laws, we conduct experiments MoE models with varying activation ratios. We used the derived laws to predict optimal hyperparameters at a compute budget of 3e20 FLOPs, after fitting them on data up to 1e20 FLOPs. As shown Figure g the predicted optimal regions effectively capture the best-performing hyperparameters for activation ratios from 4.7% 10.9%, demonstrating that the laws can be applied MoE models within this range of activation rates. This confirms that our hyperparameter scaling laws provide a reliable foundation for exploring diverse MoE architectures under fair and near-optimal training conditions.
E.2 SCALING LAWS FOR MOE OPTIMAL MODEL-DATA ALLOCATION
To determine optimal allocation between model size and data size, we analyze loss trajectories across FLOPs budgets from hyperparameter scaling experiments. By identifying the (M, D) combination that yields the minimum loss for fixed FLOPP budget, we derive optimal allocation strategies for
19

---

Published as a conference paper at ICLR 2026
(Denc) 0,00566--00 ga00 (NEE)
7- 1-11176--- OMEE)
Learning Rate
220
Stze
EennEEmeddinng Training
Non-Emibedding Training
(a) Learning rate scaling curve
(b) Batch scaling curve
Figure 7: Scaling laws for optimal hyperparameters. Blue and red lines represent the fitted laws for MoE and dense models, respectively, derived on the same training dataset. Gray circles are the experimental data points used for fitting.
0.9005
0.003
03991
014077
0.801
20-
2.000
2.49
2,0500
20
2,528
2002
2035
2,050
200
2.015
2,09
2,10
2.35
2.05
2.027
Gistina
2 054
2.0425
Bandh
* Prsicaton
Miniand Wv
2,005
Patimna
2005
2.00
2.0403
200
2,025
2.05
212-
2.000
2.045
2 .33
20
2,038
2.045
2,020
207
2,024
2,000
2,003
2,000
2.0025
L995
Loaniigg Haus
Loa Rate
Learnng Rate
(a) 10.9%
(b) A 7.8%
(c) 4.7%
Figure 8: Validation of MoE hyperparameters scaling laws across different activation ratios (4). "Near-optimal refers to hyperparameters achieving a loss within 0.25% of the optimal ones.
20

---

Published as a conference paper at ICLR 2026
specific MoE configurations activating 4 of 64 experts and additional shared expert (A 7.8%, G = 2). Crucially, MoE capacity exhibits strong dependence on activation ratio. Thus, this analysis aims to deepen our understanding of MoE architectures and to provide general guidance for model selection in subsequent experiments. The problem can be formally defined as:
" arg min ((,,D;,CAGGGS) s.t. - M,D
(6)
The resulting scaling laws for the optimal model size (M and data size (Der) are presented in Figure | and summarized in Table For comparison, we derive the same laws for dense models. Our analysis yields two key insights:
1.
The optimal allocation coefficients for different architectures are similar and close to 0,5. This aligns with findings from previous studies (Bi et al., 2024 Hoffmann et al. 2022). indicating that for compute-optimal training, the budget should be split roughly equally between increasing model size and data volume.
2. Crucially, any given compute budget, the optimal MoE model is computationally smaller (lower MP but trained on more data (larger DePi) than its optimal dense counterpart. This suggests that MoEs possess greater capacity, enabling them to support larger train- ing datasets with smaller model sizes. In real-world scenarios where data is abundant but computational resources are limited, this is significant for improving efficiency.
While practical training strategies may deviate from this compute-optimal allocation, these scaling laws provide crucial reference. They offer principled basis for determining the necessary amount of training data for a given model to approach convergence, designing informative ablation studies, and ultimately, developing more efficient MoE architectures.
Table 5: Scaling law parameters for compute-optimal allocation of model scale (MoP) and data size (Dopt) for MoE and dense models identical datasets.
Optimal Model Scale (M
Optimal Data Size (D
Dense MoE
M 0.0655 54.522
t 15.2582 C D 5.2232 4..9
M 0.1915 50.5095
#ie coRos (534 (Denel E1995 25300 slot)
D 13.255 (Denel
2255 glue
10a
10
117
Enbbectinnggg Thaining 7.0%
Man- www.mbbeiiinggg Tranning 7.0%
(a) Optimal Model Scale (M Scaling
(b) Optimal Data Size (DoH) Scaling
Figure 9: Scaling laws for optimal model scale (M/Pt and data size (Dopt) on identical datasets. For given budget, MoE models (blue) optimally allocate more resources to data and fewer to model size compared to dense models (red).
F DETAILED EXPERIMENTAL ANALYSIS OF MoE ARCHITECTURE
F. OPTIMAL EXPERT ACTIVATION RATTO
We begin by investigating the activation ratio (A), critical factor governing MoE efficiency. Our experimental design isolates the effect of A by holding the computational cost per token (M)
21

---

Published as a conference paper at ICLR 2026
stant. This is achieved by fixing the number of activated experts and their granularity, while varying the total number of experts the pool from 2 to 256. This setup allows us to explore wide range of activation ratios (from 0.8% to 100%, where 100% represents a dense model) without altering the forward pass Fl.Ps. The optimization problem for given compute budget C thus:
1 arg min L(A;
The IsoFLOPs curves, presented in Figure 10a reveal a clear and consistent trend. Across all tested FLOPs budgets (from le18 to 3e20), loss monotonically decreases with activation ratio, following power-law pattern. For all configurations, the lowest tested ratio of 0.8% consistently yields the minimum loss. This finding suggests core principle: for a fixed computational cost, greater model sparsity (i.e., lower activation ratio) leads to higher parameter efficiency.
To quantify this efficiency improvement, we fit series of loss scaling curves at different activation ratios. Based on these curves, compute the efficiency leverage for different activation ratios and FLOPs budgets, as illustrated Figure TOb] The results reveal two key trends. First, for a fixed FLOPs budget, the EL consistently increases the activation ratio decreases, indicating that sparse activation can always enhance computational efficiency. Second, for a fixed activation ratio, the EL grows with the computational budget, demonstrating that the MoE advantage amplified at larger scales. These findings confirm that reducing the activation ratio yields substantial efficiency gains, and these benefits are magnified in large-scale, high-computation regimes.
3,00
2.0
26
311
15a
32
1.70
1271
100
10"
1071
2271
Activation Batio
PLOPS
PLOP%
Activation Rabi
(a) IsoFLOPs curves for varying A
(b) Loss and EL scaling curve over varying A.
Figure 10: Impact of the Activation Ratio on Loss and Efficiency. (a) any fixed compute budget (each colored line), lower activation ratios yield lower loss. The orange stars mark the optimal loss point. (b) Loss EL scaling curves illustrate that increases with both higher compute budgets and lower activation ratios, showing that MoE advantages are magnified at scale.
Key Takeaway
Monotonic Relationship Between Efficiency and Activation Ratio. For a fixed computational cost, model performance consistently improves the activation ratio decreases. This indicates a direct, monotonic relationship between sparsity and effi- ciency.
Efficiency Gains Amplify with Scale. The efficiency advantage MoB models (their EL. grows with the total training budget. This highlights their suitability for large-scale training, where their benefits become even more significant.
F.2 OPTIMAL GRANULARITY OF EXPERTS
The granularity of experts is critical factor in the efficiency of MoE. While prior works (Ludziejew- ski et al.) 2024 Deepseek-Al Al et al., 2024) suggests that finer-grained experts improve performance, the optimal balance remains open question. To investigate the influence of expert granularity MoE efficiency, for a fixed model size and activation ratio A, we vary the expert granularity from 2 to 16 by increasing the total number of experts from 64 to 512 while proportionally decreasing the
22

---

Published as a conference paper at ICLR 2026
size of each expert keep computational cost (FLOPs) per token constant. This creates spectrum of models from coarse- coarse-graines grained (fewer, larger experts) to line-grained (more, smaller experts). By training these models and comparing their final training losses, identify the granularity that yields the best performance for a given FLOPs budget. This problem is formalized as:
arg min C(C;C,
(
where GoD is the optimal granularity that minimizes the training loss L under a fixed FLOPs budget C, model size M, activation ratio A, and shared expert ratio As shown in Figure Tla our experiments across a range of FLOPs budgets (10'6 to 1020) reveal distinct trend. For any given budget, as we increase expert granularity, the training loss first decreases and then, after reaching minimum, begins to increase. This demonstrates the existence of an optimal expert granularity that maximizes computational efficiency of MoE. To further analyze this relationship. we loss scaling curves for different granularities (Figure Ilb.. quantifying their impact on EL.
Our study yields two primary insights: First, for fixed FLOPSs budget, the training loss follows U-shaped (polynomial) relationship with respect expert granularity, which confirms optimal point for maximizing model performance per FLOP. This finding contrasts with the conclusions of Ludziejewski et al. (2024), and we detail the reasons for this discrepancy in Section C Second, across different FLOPSs budget, the optimal granularity remains within stable range (around 12 in experiments), offering reliable heuristic for model design. Furthermore,
find that routing balance significantly impacts the choice of optimal granularity. Poor routing balance shifts the opti- mal point towards coarser granularities and degrades overall model performance (see Appendix F.4 for details). This suggests that improving routing mechanisms could unlock the potential of even more fine- -eaiiin MoEs, marking a promising direction for future work.
5029 3:39 3,08 vain
3.00
2.75
57T
2,90
32.54
2.55
1344
2.00
20-55
1300
1.7
2010
1.5
2225
3125 2107
1.5
Gransfarity
Gramdarity by
(a) 1soFLOPs
varying G.
(b) Loss and efficiency leverage scaling curve varying G.
Figure 11: Impact of the Experts Granularity G on Loss and Efficiency. (a) IsoFLOPs curves reveal U-shaped (polynomial) relationship between expert granularity and training loss. Orange stars mark the optimal granularity for each FLOPs budget. MoE efficiency improves as FLOPs increase and expert granularity approaches the optimal range.
(b) Loss and EL scaling curves show that
Key Takeaway 2
Existence of Optimal Expert Granularity. For a fixed FLOPs budget and model scale, training loss exhibits U-shaped (polynomial) relationship with expert granu- larity, indicating optimum that maximizes efficiency.
Stable Range of Optimal Expert Granularity. The optimal granularity (e.g.. around 12 in our experiments) is stable across a wide range of FLOPs budgets. However, poor routing balance shifts this optimum toward coarser granularity.
The Impact of Routing Balance on the Optimal Expert Granularity. To investigate how rout- ing quality influences the optimal expert granularity, induce a state of routing imbalance. This is achieved by setting the coefficient of load balancing loss to 0.001, setup known to cause load im- balance. In this setting, train MoE models with a varying expert granularity while maintaining
23

---

Published as a conference paper at ICLR 2026
constant total parameter count. As shown in Figure 12 our results reveal that coarser expert gran- ularity becomes optimal under such imbalanced routing. Specifically, the IsoFLOPs curves (Fig- ure 12ap demonstrate that models with coarser granularity (G 6, 8) achieve lower loss for given computational budget. This trend is consistently observed the loss scaling curves (Figure T2b). This phenomenon indicates that when the routing mechanism becomes performance bottleneck, a fine-grained architecture with numerous specialized experts is counterproductive. The weakened router cannot distribute tokens effectively, nullifying the benefits of specialization. Consequently, the model benefits more from a coarser-gue -raisedd design with fewer, more generalized experts, as this simplifies the routing task and mitigates the detrimental effects of the load imbalance.
3003
3,009
1007 Sn12
30
3.00
2000
2719
28
2.53
2.30
1591
2.2
1384
1300
1337 2206
U.S. 200
59
3231
1.70
1271
109
101
124
1090
1071
330
100
Poded Sox LN
PLOPS
P.0P%
Granulerty
(a) IsoFLOPs curves over different granu- larity.
(b) Loss scaling curves under over different granularity.
Figure 12: Impact of Expert Granularity on Loss Under Weakened Routing Balance.
Summary of Expert Granularity Analysis. Our analysis of expert granularity reveals a trade-off' between specialization and load balancing. On one hand, prior work has shown that finer-grained partitioning enhances expert specialization (Deepseek-AL et al. 2024). On the other hand, our ex- periments demonstrate that increasing granularity leads to greater load imbalance. We quantify this imbalance using the coefficient of variation (CV) of the expert loads, where higher value in- dicates a greater imbalance. As detailed Table [5| increasing the number of experts (E) directly correlates with higher CV. These results were obtained while holding the balancing loss coefficient constant, in order to isolate the effect of granularity. This controlled setting highlights the inherent challenge of routing tokens to a larger set of smaller experts, thereby distinguishing the benefits of specialization from the practical difficulties utilization.
F.3 OPTIMAL SHARED EXPERT RATIO
Shared experts are always active to capture common knowledge (Deepseek-A) Al et al. 2024). To determine the optimal proportion of shared experts, we designed a series of experiment to isolate the impact the shared expert ratio S. We fix the total model size M, the activation ratio A, and the total number of active experts (E, + F.a. We then systematically vary S by substituting routed experts (Fe)) with shared experts (F.), exploring configurations from fully specialized (S 0%) to highly shared (S = 83.3%). This allows to identify the optimal ratio that minimizes training loss for a given computational budget. The problem is formalized as:
S = arg min L(S; C,M,
5
where Sopt is the optimal shared expert that minimizes the training loss L under fixed FLOPs budget C, model size M, activation ratio A, and granularity G. Our experiments, as depicted in Figure 13a reveal a U-shaped relationship between the shared expert ratio and training loss. The minimum loss is generally achieved at a relatively low shared expert ratio, while having no shared experts (S 0%) usually results in suboptimal performance. Furthermore, we observe subtle trend where the optimal sharing ratio appears to scale with the compute budget. This is supported by our empirical scaling law (EL.) analysis in Figure 13b which shows that lower FLOPPs budgets
24

---

Published as a conference paper at ICLR 2026
( 10a0) benefit from slightly higher sharing ratio (S 16.7%), whereas larger budgets (> 1020) achieve greater efficiency with a lower ratio (S = 8.3%).
Since large-scale pre-training runs typically exceed -0a0 FLOPs, this suggests practical heuristic: the optimal design choice is to use the lowest possible zen-zero sharing ratio. Assuming the dimen- sions of shared and regular experts are equal, this can be heuristically implemented by setting the number shared experts to one.
3.30
200
2300
180
625
Elft
130
1.2
35a
32"
1000
1271
1.70
10'
10"
10
32a0
3000
2011
300
330
Shared Expert
FLOP%
(a) IsoFLOPs curves over varying G. (b) Loss and efficiency leverage scaling curve over varying G.
Figure 13: Impact of the Shared Ratio S on Loss and Efficiency. (a) Loss curves demonstrate that a low, non-zero sharing ratio minimizes training loss, outperforming both no shared experts (S 0%) and highly shared configurations.. (b) EL analysis reveal that the optimal sharing ratio
is higher (S 7%) for smaller FLOPs (< 1020) and decreases to = 8.3% for larger FLOPs (> 10at)
Key Takeaway 3
Optimal Sharing Ratio Exhibits Subtle Scaling Trend. We identify a subtle scal- ing trend between the optimal shared expert ratio and the compute budget: the ideal ratio decreases the compute budget increases.
"One Shared Expert" Rule for Large-Scale Training. For large-scale pre- pre-training with uniformly sized experts, the optimal design hcuristic is to employ a single shared expert. This configuration establishes the minimal non-zero sharing ratio.
F.4 OTHER CONFIGURATION OF MoE ARCHITECTURE
Arrangement of MoF and Dense Layers To ensure balanced routing in the early layers, main- stream MoE models typically replace all FFNs except for the first few layers with MoE layers. We investigate the impact of this design decision the efficiency of MoE models. To ensure mean- ingful exploration space, we extend all models in our experiments to 60 layers and set the first 1, 2, or 3 layers as dense layers sequentially. The dimension of these dense layers is set match the total dimension of the activated experts in the corresponding MoE layers, ensuring the overall computa- tional cost (FLOPs/hoken) remains constant. This design allows the proportion of dense layers on MoE efficiency. The experimental results, presented in Figure [4a and 14b] reveal the following key findings: From model performance perspective, replacing the hirst few layers with dense layers has minor impact. Using a dense proportion of zero as the baseline, we estimated the efficiency leverage for each configuration. Within FL.OPs budget of up to x 10 FLOPs, the efficiency leverage remains close to 1. This indicates that configuring the ini- tial layers as dense offers negligible efficiency improvement. However, this adjustment effectively reduces the total number of parameters in the model and mitigates routing imbalances in the early layers. Thus, despite its limited efficiency gains, this remains a valuable design optimization. 2) Fur- ther investigation into the optimal proportion of dense layers under varying computational budgets reveals a trend: as FLOPs budgets increase, the optimal dense proportion also grows. For example, in our experiments, when the compute budget is 1 x 1018 FLOPS, the optimal dense proportion is
isolate and study the effect of
25

---

Published as a conference paper at ICLR 2026
zero. As the compute budget increases to 3 x 10 FLOPs, the optimal dense layer proportion shifts to approximately 2/60 or 3/60.
315
151
Diideroy Leverge laye's
125
2.69
Layars
Sod MM
Propartion
Emicseery Leaeern
24
Ailnnion Proportion
eilt
FLOP%
FLOP
1977
Attention FLOPs Proportion
Figure 14: Impact of Dense Layers Proportion and Compute Budget Allocation between Attention and FFN, (a.b) Replacing the first few layers with dense layers shows minor impact model per- formance. As computational budgets increase, the optimal proportion of dense layers also gradually rises. (c,d) Modifying the attention FLOPs ratio within broad range (20%-50%) has negligible influence on model performance, demonstrating the robustness of this configuration.
Compute Resource Allocation between Attention and FFN As two core components of the Transformer model, the attention mechanism (Attention) and FFN account for the majority of the model's computational load. To this end, we explore the impact of computational allocation between the attention mechanism and the FFN on the efficiency of the MoE model. Specifically, we construct series of models with fixed model scale M but varying compute budgets by increasing the hidden layer size of the attention module while reducing the hidden layer size of each expert in the MoE. We then observe the performance changes of these models under different computational allocations and evaluate their scaling trends. The experimental results are illustrated in Figure 14c and [4d] revealing the following key findings: When the attention FLOPs ratio is between 30% and 40%, it represents a relatively stable and reliable configuration. Models tend to achieve optimal or near- optimal performance within this range. This configuration consistent with the default settings of mainstream open-source MoE models. 2) Adjusting the attention FLOPss ratio within a broader range (20%-50% has minor impact on model performance, As shown in Figure| [4d] the loss scaling curves and efficiency leverage of these models are nearly identical. Since the attention mechanism generally has a higher computational density (i.e., FLOPs-per LLOPs-per-parameter compared to the FFN, increasing the attention FLOPs ratio while keeping the overall model size constant reduces the total number of model parameters, resulting in higher knowledge density. However, this also implies potentially higher downstream inference costs.
26

---

Published as a conference paper at ICLR 2026
Key Takeaway
Introducing Dense Layers a Valuable Design Optimization. Incorporating dense layers in the early stages of MoE has minor impact efficiency but helps mitigate routing imbalances and reduces overall parameters. The optimal proportion dense layers increases with higher FLOPs budgets, though it offers limited efficiency gains.
Robustness of Compute Budget Allocation between Attention and FFN Allocat- ing 30%-40% of FLOPs to the attention mechanism achieves optimal or near-optimal performance, with minor impact outside this range. Increasing attention FLOPs pro- portion enhances knowledge density but reduces downstream inference efficiency.
G VALUES OF THE FITTED COEFFICIENTS.
To validate the proposed scaling law for EL, we fit Eq 4 using Huber loss and the BFGS optimization algorithm (Hoffmann et al. 2.22). We use data points with an EL factor below 6 for training, while those are reserved as a validation set. The values are presented in Appendix
Table 6: Values of the Fitted Coefficients.
a
d
7
3
Astart
Amuz
1.23
-7.61e-2
1.67e-2
-1.17e-1
1.63e-2
5.28c+16
H DETAILED RESULTS OF MOE-MINI EVALUATION
Evaluation Benchmarks To evaluate performance, consider diverse suite of downstream tasks designed to provide a holistic assessment of model capabilities. These tasks are grouped into
several categories, such (a) General Knowledge/Reasoning (e.g.. ARC (Bhakthavatsalam
al.
2021
AGIEval (Zong et al., 2024), OpenBookQA (Mihaylov et al. 2018). BBH (Suzgun et al.
2023
ProntoQA (Saparov & He| 2003). PIQA (Bisk et al. 2020)) HellaSwag (Zellers
2019
Multi-LogiEval (Patel et al. 2024h) (b) Language Understanding (e.g.. RACE (Lai et al.
2017
(c) Professional Knowledge (e.g., MMLU (Hendryeks al. (021a). CMMLU (Li al.
2024
MMLU-Pro /Wang et al. 2024b). GPQA (Rein et 20.3). C-Eval (Huang et al. (202)). Common-
senseQA (Talmor al.) 2018)) (d) Math (e.g, GSM8K (Cobbe al.] 2021). MATH (Hendrycks)
et al., 2021b), GAOKAO (Zhang et al. 2023), 2aakaa20233MMth-thi MGSM (Shi et al. 2023).
CMATH (Wei et al.) 2023). MathBench (Liu al.) 2024), Minerva-Math (Lewkowyez et al.)
2022). CN-Middle School 24) (e) Code (e.g., Humaneval (Chen et al. (2021). HumanEval-on (Peng
ct al. 2024), HumanEval-plus (Liu et al.) 2023), HumanEval-FIM (Bavarian al. 2022), Live-
Bench (Jain et al. 2025). MBPP (Tao al. 2024), MBPP-Plus LLu 2023). CruxEval (Gu
ct al. 2024)
Evaluation Results The comparative evaluation in Table 7 reveals that MoE-mine achieves an average score of 45.5, surpassing Dense-6 6. 1B's 44.0. This result compellingly demonstrates that McE -immmnni accomplishes a "small yet powerful" feat with significantly lower inference costs, its activated parameters amount to only about 13% of its competitor's s, striking exceptional balance between performance and efficiency.
Upon closer examination of performance across specific dimensions, MoE-mini advantages are both comprehensive and focused. In general knowledge and reasoning tasks, exhibits notable advantages in open-ended question answering tasks such as OpenBookQA and complex logical reasoning benchmarks like Multi-LogiEval. This trend continues in specialized knowledge domains, where MoE-mia ni delivers better results comprehensive academic benchmarks like MMLU and MMLU-Pro. Its superiority is particularly evident in language understanding tasks, as consistently outperforms its competitor in the RACE series of reading comprehension tests, showcasing stronger contextual understanding capabilities. In tasks requiring high coding proficiency, McE-mini stands significantly, especially in the HumanEval-PPan benchmark, which measures code robustness,
27

---

Published as a conference paper at ICLR 2026
Table 7: Detailed performance comparison of MoE MooEmmini (17B-A0.8B) and Dense
Metric
Dense-6 1B
MoE-mini (A0.8B)
ARC-challenge
59.7
57.0
ARC-easy
78.0
78.7
AGIEval
33.4
34.9
OpenBookQA
68.6
75.2
General Knowledge
BBH
48.0
35.7
/Reasoning
ProntoQA
16.5
19.5
Mulii-Logiival
55.6
61.3
HellaSwag
65.6
PIQA
76.6
77.2
Average
55.8
56.2
MMLU
51.1
53.1
MMLU-Pro
21.7
24.0
CMMLU
50.7
51.9
Professional
C-Eval
52.5
51.1
Knowledge
CommonsenseQQA
63.6
60.6
GPQA
24.8
27.3
Average
44.0
44.7
Language
RACE-middle
73.4
75.6
RACE-high
65.0
67.6
Understanding
Average
69.2
71.6
HumanEval
31.7
35.4
HumanEval-cn
34.2
32.3
HumanEval-PP
35,4
51.8
HumanEval-FIM
62.8
Code
MBPP
41.0
44.6
MBPP-Plus
50.0
51.6
LiveCodeBench
7.5
7.4
CruxEval
32.9
34.1
Average
36.9
39.8
GSM8K
59.2
58.0
MATH
23.7
29.8
CMATH
60.5
62.9
MGSM-zh
35.6
CN-Middle School 24
41.6
42.6
Math
Minerva- Math
3.3
2.9
MathBench
27.5
28.6
Gaokao2023-Mathe
33.1
GAOKAO-Math24
12.1
17.6
Average
32.9
Overall Average
44.0
45.5
achieving an impressive lead of over 16 points. Similarly, in mathematical reasoning, while slightly lagging in basic arithmetic tasks like GSM8K, it excels in challenging benchmarks such as MATH and GAOKAOO-Math2
demonstrating strong potential in solving complex problems. Collectively, McE-mins n achieves 1.5-point overall advantage, validating its parameter-efffiient MoE design. not only drastically reduces inference costs through sparse activation but, more critically, its "expert networks" seem to enable higher performance ceilings in key areas such language understanding, code generation, and advanced reasoning.
Pre-training Evaluation of MoE-mini We present a detailed evaluation of MoE-min's training process. Figure 15 provides comprehensive comparison across datasets and categories, outlined the main experiments in Section 5.2 The results show that MoE-mini achieves comparable performance Dense-6 6. 1B the majority of datasets.
28

---

Published as a conference paper at ICLR 2026
Deron
1050
owvi
corricturra..r
agloval
CPAA
teloonng
A00 0000
90
10:0
400
2000
1830
9olen131
Toles
10.0
-70
18.00
operall tunanieall.c
fumaread plus
niteop00 2015
Mic 10100
Hot
Paace (eds
(stav1s)
soar 66
Idde
Brocodacacch
Crudkaal
pordds
201
4.1 NO
Coriee 6.18
Deron K10
200
200
R05
300
181
201
10:0
65
1000
411
1230
regim
math
gasica20033en
10085
PC
Dates 24
Muti Lay
Mcc
200
400
200
03
2000
200
800
000
200
080
30
1000
200
60
2000
Figure 15: Overall and category-wise performance comparison between Mož-mini (17B-A0.8B) and Dense-66 1B.
29

---

Published as a conference paper at ICLR 2026
I ESTIMATING FLOPS
To analyze the efficiency of our models, we quantify the computational cost in terms of total training
Floating Point Operations (FLOPs). Following standard practice (Kaplan et al. 2020). we estimate the total training FLOPs as approximately three times the cost of single forward pass (Ctrain & Crua). The forward pass FLOPs are the sum computations from the attention and feed-forward network (FFN) layers, plus final logit projection.
For model with hidden size dmndel: batch size B, and sequence length 8, the cost of the attention block per layer, Canu. which includes Grouped-Quen Attention (GQA) (Ainslie et al. 2023) and all projections, is approximately:
1
2 Rtp///go
Cart R 2Bsdmood
+ 144Bs2dmode
(10)
where ni, and ne are the number of attention and key-value heads, respectively. The FFN cost varies by layer type. A dense layer with intermediate size dit requires Cdense Bdsdddddd FLOPs. A MoE layer activating Ea experts, each with size dexpent, requires:
Cmoe fin 6Bsdmodel (Ee dexpent
(11)
If shared expert of size datared is used, its cost, 4Bsdmode is added. For model with L layers (of which the first Leesse are dense) and a vocabulary of size the total forward FLOPs are:
Crwd = Σ(Cana + Cttn.s) + BBddmed
(12)
1=1
where Cän,i is the FFN cost for the i-th layer, which can be either Cdense fin or Cmoe fin-
J LIST OF EXPERIMENTAL MODELS
The detailed configurations for all experiments conducted in this study are presented in Tables (activation ratio). Tables (expert granularity), Tables I0 (shared experts), Tables (layer arrange- ment), and Tables 12 (compute allocation between attention and FFNs).
K METHODOLOGY FOR CALCULATING EFFECTIVE LEVERAGE (EL)
Our methodology for obtaining EL datapoints does not involve training a unique dense model to match the loss of each individual MoE run. Instead, we employ a more systematic and scalable approach based on modeling the loss-compute scaling behavior for each model family. For a specific MoE architecture (c.g., for a fixed activation ratio and granularity), as show in Algorithm the process of obtaining data points for EL is as follows:
1. Collecting (compute budget, optimal loss) Data: We first train a suite of MoE models (see Tables to [2] and a corresponding suite of dense counterparts. For a given MoE configuration, its dense counterpart is defined as standard Transformer architecture, equivalent to an MoE model with a 100% activation rate. As shown Table 8 the model architecture with an activation rate of 1.0 serves as the dense counterpart for all other MoE configurations in our study. All models are trained on the same dataset with the same recipe, for up to x 10 FLOPs. This process generates a set of (compute, optimal loss) data points {(C, {)} for both the specific MoE architecture XMoE and dense architecture X'eeese This is illustrated in Figure 10a Figure
1la and Figure [3a
Fitting Loss Scaling Curves: We then fit separate loss scaling functions, Lxmae (.) and
LXpame ()), to the collected data for the specific MoE and dense architecture, respectively. We use standard power-law form, Lx(C)= = ax.. 8 + bx, consistent with prior work (Kaplan et al. 2020
Henighan et al.) 2020 Achiam al. 2023). This process yields smooth loss scaling functions that can predict the architecture 's optimal loss at any given compute budget C. These fitted curves are
shown in the left panels of Figure 10b Figure TTb. and Figure T3h
30

---

Published as a conference paper at ICLR 2026
3. Computing EL via Interpolated Loss Matching For MoE model compute budget CMOE, we first use its fitted curve to calculate the predicted loss: L LXS.R (Cmo)). Next, we use the dense model's loss curve, Li (C),, to find the compute required to achieve the same loss by solving Dense L Xpense (C*). Finally, we compute the EL defined: EL C /CMaE- These EL values are shown in the right panels Figure Tob Figure l1b and Figure 13b This methodol- ogy based loss scaling curves allows us systematically evaluate EL across continuous range of compute budgets.
Algorithm 1 Calculating Efficiency Leverage (EL)
Require: Pdense PMaE: Sets of (compute, loss) data points from dense and MoE training runs. Require: CMoB: The target MoE compute budget for which to calculate EL.
Ensure: EL: The calculated Efficiency Leverage value.
D Part 1: Fit Loss Scaling Functions from Data 2: for each model family X E (Dense, MoË} do
1:
3: Fit continuous loss scaling function (x(C) to the corresponding data points Px- b TTpiccllly uses parametric model like a power law, e.g., L(C)) = aC-b +
4:
end for
5:
Calculate EL via Loss Matching > Calculate the target loss achieved by the MoE model. 7: if' L is not attainable by the dense model (e.g., below its loss floor) then
D Part 2:
6: C <- TxxE (OMOE)
8: return undefined
9: end if
10: Solve for Cxima such that Dienss (CDemse) L*
This is done by inverting the parametric function or using a numerical root-finder.
11: EL f- CDass
>> The final EL is the ratio of compute budgets.
Chue 12: return EL
L COMPUTATIONAL RESOURCES
Our study utilized total of approximately 680,000 equivalent H800 GPU-hours. The allocation of this computational budget is detailed below:
A total of 360,000 hours were used for preliminary experiments. This phase involved extensive hyperparameter tuning and explorations into optimal model and data allocation.
The main architectural scaling experiments, which form the core contribution of this work, required 200,000 hours.
Final validation runs, including the complete training of our 16-billion parameter MoE and dense models on IT tokens, consumed the remaining 120,000 hours.
This substantial investment underpins the reliability and scale of our empirical findings.
M IMPACT OF THE NUMBER ATTENTION HEADS
To investigate the impact of the number of attention heads (Nhead), we conducted a series of supple- mentary experiments. For each model size, we systematically varied Thead while keeping all other hyperparameters constant, as detailed in Table 14 The results, visualized in Figure [16] reveal that model performance is not sensitive to a single specific number of heads. Instead, we identified a range of "near-optimal" values, defined as configurations achieving final validation loss within 0.5% of the minimum observed loss for that model size. Based on this observation, and in line with common practice in scaling law studies (Hoffmann et al. 2022 Ludziejewski et al. 2024), scale Theaa proportionally with the model dimension (dmode)) in our main experiments (see Tables 8 Bo 12). This approach ensures robust and fair comparison across different model scales.
31

---

Published as a conference paper at ICLR 2026
2.5
Model Dimension (dmodd))
204
2,4
*
512
640
788
2.3
+ 1024
2.2
2.1
2.0
+
1.9
1.8
10
15
25
30
Number Heads (Thead)
Figure 16: Impact of the Number of Attention Heads. Red stars denote "near-optimal" configura- tions, defined as those achieving final loss within 0.5% of the minimum for each model size.
N GOODNESS-OF-FIT ANALYSIS FOR SCALING LAWS
Our methodology is empirical, aligning with established practices in scaling law research (Clark al. 2022, Hoffmann et al. 2022 Ludziejewski et al. 2024; Abnar et al.] 2025 Besiroglu et al. 2024 For each component of the scaling law, first identified the underlying trend through data visualization. We then selected functional form that best captures this trend and rigorously validated our choice by comparing its goodness-of-fit (e.g., R against simpler alternatives.
N.1 SATURATING TRANSFORM FOR ACTIVATION RATIO (A)
We empirically observed a trend of diminishing returns when decreasing the activation ratio (A); that the performance gains from smaller ratio lessen as approaches zero. saturating trans- formation, which has been successfully employed to model similar phenomena (Clark et al. 2022), well-suited capture this effect. To validate this choice, we compared the of our proposed form (Eq. [ against standard power-law model. As shown in Table 16 the saturating transformation achieves a significantly higher R-squared value, confirming its superior fit the data.
N.2 LOG-POLYNOMIAL FUNCTION FOR EXPERT GRANULARITY (G)
Our analysis revealed U-shaped relationship between expert granularity (G) and model perfor- mance, where the EL first increases and then decreases as grows. A quadratic polynomial function is standard and parsimonious choice for modeling such non-monoto trends and is widely used in prior work (Ludziejewski et al. 2024 Abbar et al., 2025). We confirmed its suitability by comparing its fit against standard power-law and saturating transform. The results in Table demonstratt that the log-polynomial form provides the most accurate fit to our empirical observations.
N.3 POWER LAW FOR COMPUTE BUDGET (C)
The selection of power law for the compute budget (C) was directly motivated by our empirical data. As visualized in Figure 4c| the log-log plot of Effective Loss versus training FLOPs exhibits distinct linear relationship. This linearity the hallmark of power-law dependency, making the natural and most appropriate functional form for this component of our model.
N.4 JOINT SCALING LAW FOR EFFICIENCY LEVERAGE
Finally, the joint scaling law (Eq. 4 an empirical composite model designed to synthesize our individual findings. Its structure is not arbitrary but reflects the observed interactions: The term for Activation Ratio (A) serves as the primary driver of efficiency. This is then modulated by the inde- pendent, non-linear adjustment from Granularity (G). Finally, the entire efficiency gain is amplified
32

---

Published as a conference paper at ICLR 2026
by the Compute Budget (C) through the overarching power-law pattem. This composite structure provides the most comprehensive explanation of the joint effects. Its high accuracy empirically validated in Figure 5 achieving R-squared of 0.9858. The model demonstrates strong predic- tive power with low RMSE on both the training set (0.2169 over 200 points) and the validation set (0.5275 over 24 points). We further analyzed the residuals of the fitted scaling law, shown in Figure 17 The residuals are approximately normally distributed and centered at zero (mean -0.0273, std. dev. 0.2803), which confirms the high quality of the fit.
Mean: -0.027
0.2803
50
Frequency
30
Residual (Observed Predicted)
Figure 17: Plot of residuals of our estimated joint scaling law.
Table 8: Experimental configurations for the expert activation ratio analysis. Within each group, the number of activated experts (E, 2) is fixed, while the total number of experts (E) is varied to study the effect of the activation ratio.
Piayera
dmantei
dexpert
Theads
Piky hend
E
?
Max traningg FLOPs
384
320
8
2
12.48.16..32664.288..2
1.52e-3
98
2c18
512
512
2
[2.488.16.32.641288.55
1.31e-3
6e18
640
640
12.488.16.32,66,122..2.
1.11e-3
2e19
14
768
768
4
22.48.16.32.64.1222222
9.5e-4
342
6c19
16
1024
1024
16
4
1.4.8.16.32.64.11222555
8.1e-4
531
2c20
22
1280
1280
20
4
[2.488116.32.441228..2
7.0c-1
795
6c20
33

---

Published as a conference paper at ICLR 2026
Table 9: Experimental configurations for the expert granularity analysis. Within each group, the base model architecture is fixed while the MoE configuration (total experts E, activated experts E.. shared experts Eg, and expert dimension dexpea) is varied to study the effect of granularity.
Ilagers
dmode!
Thaeads
E
Ea
despert
7
Max training FLOPs
64
384
128
192
192
6
128
8
384
98
1.52e-3
2c18
256
8
4
96
384
64
512
48
64
512
128
256
192
170
8
512
8
147
1.31c-3
6c18
256
128
384
85
512
64
64
640
128
320
192
213
10
640
10
228
1.11c-3
2c19
256
160
384
12
106
512
80
64
768
128
384
192
256
14
768
12
342
9.5e-
6e19
256
192
384
12
128
512
96
64
1024
128
512
192
341
16
1024
16
531
8.1e-4
2c20
256
256
384
12
170
512
16
128
64
1280
128
640
192
426
22
1280
20
795
7.0c-4
6c20
256
320
384
12
213
512
16
160
34

---

Published as a conference paper at ICLR 2026
Table 10: Experimental configurations for the shared expert ratio analysis. Within each group, we fix total number of experts (E = 256) and the total number of activated pathways (E, + E, = 12), while varying the ratio between specialized experts (E,) and shared experts (E.) to study its impact on performance.
nlayers
dmodet
Thneads
Ea
E,
dexpen
B
1
Max training FLOPs
256
2
10
96
256
4
256
6
6
96
8
384
98
1.52e-3
2c18
256
8
4
96
256
11
1
96
256
12
96
256
2
10
128
256
4
128
256
6
128
8
512
147
1.31e-3
6e18
256
8
4
128
256
11
128
256
12
128
256
2
10
160
256
4
8
160
6
160
10
640
10
228
1.11e-3
2c19
256
8
4
160
256
11
160
256
12
0
160
256
2
10
192
256
4
8
192
6
192
14
768
12
342
9.5e-4
6e19
256
8
192
256
11
2
192
256
12
0
192
256
2
10
256
256
4
8
256
256
6
256
16
1024
16
531
8.1e-4
2c20
8
256
256
11
256
256
12
0
256
256
2
10
320
256
4
8
320
256
6
320
22
1280
20
795
7.0c-4
6c20
256
320
256
11
320
256
12
0
320
35

---

Published as a conference paper at ICLR 2026
Table 11: Experimental configurations for the arrangement of MoE and dense layers analysis. Within each group, the total number of layers is fixed at 60, while the mix of dense layers (Tidense layers) and MoE layers (Tmoe layers) is varied to study the impact of their ratio and placement on performance.
Rragero Iagers
Rooorfoyers
dmnodel dife
Tizeeds
E
Fr E,
Bexeen
B
Max training FLOPSs
384
1280
384
1.52c-3
2c18
2
58
3
57
0
60
59
512
2048
1
512 147
1.31e-3
6c18
2
60
640
2560
10
64
2
1
640
228
1.1le-3
2e19
3
57
59
2
58
768
3072
1
768
342
9.5c-
6c19
3
57
0
60
59
60
1024
4096
64
1024
531
8.1e4
2c20
60
1280
5120
20
64
2
1
1280
795
7.0c-1
6c20
3
57
36

---

Published as a conference paper at ICLR 2026
Table 12: Experimental configurations for analyzing the compute allocation between attention and FFNs. Within each group, the core MoE structure is held constant, while we systematically vary the model's hidden dimension (dmodel) and the expert dimension (dexpen) explore the optimal trade-off in compute allocation between the attention mechanism and the FFN experts.
layers
dmodel
despert
Theads
head
Eg
Eg
1
Max training FLOPSs
450
2
1.52e-3
2e18
368
380
2
64
2
1.52e-3
96
2e18
8
384
320
2
64
1
2
1.52e-3
96
2c18
8
400
260
2
64
2
1,52e-3
96
2c18
416
208
2
64
1.52e-3
96
2c18
1.31e-3
6e18
8
512
512
2
64
2
1.31e-3
160
6e18
8
544
410
2
64
1.31e-3
160
6e18
560
364
2
64
1.31e-3
160
6c18
320
1.31e-3
160
6c18
600
766
10
2
1.11e-3
224
2c19
640
640
10
2
64
1.11e-3
224
2c19
10
680
528
10
2
64
1.11e-3
224
2e19
700
476
64
1.11e-3
224
2c19
1.1le-3
2e19
696
988
12
64
2
9.5e-3
320
6e19
14
768
768
12
4
64
2
9.5e-3
320
6e19
14
816
642
12
4
64
9.5c-3
320
6c19
840
584
64
9.5c-
320
6c19
888
474
9.5c-3
320
6c19
896
1378
16
64
8.1e-3
512
2e20
16
1024
1024
16
4
64
2
8.1e-3
512
2e20
1088
876
4
64
8.1e-3
512
2e20
1152
8.1e-3
512
2c20
1184
680
64
8.1e-3
512
2c20
22
1120
1686
20
4
64
2
7.0e-3
768
6e20
22
1280
1280
20
4
64
2
7.0e-3
768
6c20
22
1360
1110
20
64
7.0e-3
768
6c20
1440
7.0e-3
6c20
22
1520
816
20
64
2
7.0e-3
768
6e20
Table 13: Representative parallelism configurations in experiments.
Model
Plaayes
clmodel
dita
dexpert
E
E.
EP
TP
PP
Experimental Model Example
8
384
96
256
8
1
Experimental Model Example
8
512
128
256
8
1
Experimental Model Example
10
640
160
256
8
1
Experimental Model Example
768
256
8
Experimental Model Example
1024
256
256
8
1
Experimental Model Example
22
1280
320
256
8
2
Dense- 6, Mož-mini
4096
14336
1
20
2048
5120
384
384
12
8
1
Table 14: Experimental configurations for analyzing the impact of the number of attention heads (Nnent). Within each experimental group, only Thead is varied while other parameters remain fixed.
Miagers
dmodel
desport
Tcheads
head
E
E.
E,
")
B
training FLOPs
8 384
320
12,4,6,8,12]
2
64
2
1.52e-3
98
2e18
8
512
512
12,4
2
64
2
1.31e-3
147
6e18
10
640
640
14.8,10.16.201
2
64
2
1.1le-3
228
2e19
14
768
768
12,
64
2
9,5e-4
342
6e19
16
1024
1024
16,2,16,24.31
4
64
2
8.1e-4
531
2e20
37

---

Published as a conference paper at ICLR 2026
Table 15: Impact granularity on load balancing. Finer granularity (more experts) increases the coefficient of variation (CV), indicating greater load imbalance.
Blagers
dmodel
Thcadd
E
E
Es
dexpert
B
Training FLOPs
CV
14
768
12
192
6
3
384
342
9.5e-4
6e19
0.033
14
768
12
256
8
192
342
9.5e-4
6e19
0.052
14
768
12
384
12
6
128
342
9.5e-4
6c19
0,061
14
768
12
512
16
96
342
9.5e-4
6c19
0.093
Table Goodness-of-th comparison for the functional form relating EL and Activation Ratio (A).
The saturating transformation provides a better fit.
Functional Form
R
Power-law with saturating transformation (ours, Eq. 1 Standard power-law (i.e., log (A) x log A)
0.9915
0.9772
Table 17: Gfodness-of- fit comparison for the functional form relating Effective Loss (EL) and Ex-
pert Granularity (G). The log-polynomial function best captures the U-shaped trend.
Functional Form
R-squared
Log-polynomial (ours, Eq.
0.9575
Standard power-law (i.c., log EL(G) x log G)
0.8276
Power-law with saturating transformation
0.9432
38


Reviews:
[{"review_id": "RBOxLr1OY5", "reviewer": "Reviewer_k9rc", "summary": "This paper studies the effectiveness of training MoE versus dense models from the compute perspective.\n\nFirst, a detailed ablation study is performed to search for the best architecture for training an MoE, considering shared experts, granularity, sparsity.\nThen, scaling laws of the leverage (gain of performance wrt dense model) are established by considering the above factors independently.\nFinally, the model is trained at the scale of around 20B total-1B active MoE to validate the performances.", "questions": "- How many GPU hours are used in the study?\n- What kind of software (and techniques, like parallelisms) is used for training?\n- What could be the reason for the difference wrt Olmoe in terms of the effectiveness of shared expert? In the Olmoe paper ([2409.02060]), it is shown that using shared expert is not effective.\n- Using different number of head/ kv heads affects the number of model parameters. How does changing the kv head affect the results?\n  - In table 8 it seems like the number kv head changes with size. What is the justification for it?\n- Since both model size (N) and dataset size (D) are varied in the experiments, I wonder if the scaling law could be described in terms of N and D, which could be more expressive than using only compute (which is roughly 6*N*D), i.e., something similar to the chinchila-like scaling law?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The study is of large scale (20B moe and 1T tokens) and detailed ablation studies justify their choices of architecture.\n- A principled approach (scaling law) is used to tune the LR and batch size, further solidifying the argument for the proposed leverage scaling law.\n- While the optimal configuration is quite impossible due to many hyperparameters, this paper still provides valuable and principled insights into how to scale the MoE.", "weaknesses": "- The experimental design for analyzing the Activation Ratio (A) (Section F.1, Table 8) appears structurally limited as it holds the active computational cost nearly constant.\n  - The study varies the Activation Ratio (sparsity) exclusively by changing the total number of routable experts (E) while fixing the number of activated experts (Ea=2) and shared experts (Es=1). This approach means the study explores *only* the effect of increasing total parameters without changing the active expert.\n- Formulating the scaling law in terms of the efficiency leverage is not very useful. This means that a user would have to first compute the dense model scaling law, before applying it to the proposed scaling law to figure out the MoE scaling behavior. A direct formula relating only the MoE configuration to the loss (eg [2501.12370]) would be more useful.\n- Codes are not available (or a description of it if possible).\n  - Training configuration is lacking as well (e.g., what kind of parallelism techniques is used etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mLAYuwgE9S", "reviewer": "Reviewer_7T91", "summary": "This paper introduces 'Efficiency Leverage' (EL) to quantify the computational advantage of MoE models over dense models . It comprehensively investigates the scaling laws for MoE models, analyzing key factors such as expert granularity and activation ratio (sparsity). The study is supported by extensive data and figures and validates its findings by training a high-performing MoE model (MoE-mini) that confirms the predicted efficiency gains. The work provides a practical guide for MoE model design.", "questions": "1. The derivation of optimal hyperparameters in Section 2.2 scales them as a function of total compute $C$, adapting a methodology from dense models. However, the paper notes that MoE backpropagation is different, as experts see only a subset of tokens . Why should the optimal batch size and learning rate scale only with total $C$, rather than also depending on MoE-specific factors (like activation ratio) that directly influence gradient statistics for each expert?\n\n2. Could the authors provide more details on the fitting process for the scaling laws (Eq. 2, 3, 5)? Specifically, what was the rationale for choosing the log-polynomial form for granularity ? Were simpler functional forms tested, and how did their fit quality compare?\n\n3. Regarding the finding that EL improves as activation ratio decreases (Key Takeaway 1) : Do the authors expect this trend to break down at a certain point when considering practical implementation overheads (e.g., wall-clock time) instead of just theoretical FLOPs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work's originality stems from its formulation of \"Efficiency Leverage\" (EL), a clear metric to quantify MoE computational advantage . It uses this concept to build a unified scaling law for EL, connecting it to the compute budget, activation ratio, and granularity.\n\n2. The empirical quality is high, supported by over 300 trained models. A key strength is the preliminary work in Section 2, which derives MoE-specific scaling laws for optimal hyperparameters and data allocation . This step ensures a fair comparison, addressing a methodological flaw in prior studies.\n\n3. The paper is clearly structured, and its claims are well-supported by data. The final validation, MoE-mini, successfully confirmed the law's >7x efficiency prediction . This demonstrates the work's significance as a practical guide for designing future efficient MoE models.", "weaknesses": "1. The conclusion that efficiency monotonically increases with sparsity (Key Takeaway 1) is based on a theoretical FLOPs model. This omits the practical wall-clock costs of routing, communication (e.g., all-to-all), and memory bandwidth for loading many distinct expert weights, which can become bottlenecks at high sparsity.\n\n2. The functional forms for the scaling laws (Eq. 2, 3, 4, and 5) are presented without strong justification. It is not clear why these specific complex forms (e.g., log-polynomial for $G$, saturating transform for $A$) were chosen over simpler alternatives, or how the goodness-of-fit was evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFiOC4UjjN", "reviewer": "Reviewer_WZ6e", "summary": "The paper introduces Efficiency Leverage (EL), a metric designed to compare the training efficiency of sparse Mixture of Experts (MoE) versus dense Transformer models. The authors' primary contribution is developing a scaling law that expresses EL as a function of model architecture and compute budget and validating it on a moderately large computational scale.\n\nThe methodology proceeds in three phases:\n\n1. **Establishing a testbed**: The authors develop predictors for optimal training hyperparameters, which allows them to reduce bias in the subsequent scaling law analysis.\n\n2. **Optimal model/data allocation study**: They investigate how to optimally allocate model size and dataset size given a fixed compute budget, separately for MoE and dense architectures.\n\n3. **Scaling law formulation**: Using 1. and 2., they derive and fit a joint scaling law across three variables: model sparsity, expert granularity, and total training budget (measured in FLOPs). They study marginal scaling behavior with respect to each variable individually before formulating the joint relationship.\n\nThe authors validate their scaling law in two ways. First, they present plots showing fit quality on a held-out validation set, though without goodness-of-fit metrics. Second, and more importantly, they conduct two large-scale training runs—one dense model and one sparse MoE model—trained on identical tokens using compute budgets significantly larger than those used to fit the scaling law. The validation succeeds: the models achieve comparable performance, yet the dense model requires more than 7× the active parameters, confirming the scaling law predictions (though I need explanation from the authors on the details).\n\nAdditionally, the paper analyzes how architectural choices affect scaling behavior, including the number of shared experts, expert granularity, and the proportion of attention versus MLP FLOPs.", "questions": "# Questions for Authors\n\n1. **What is the precise definition of EL?** Can you provide a complete mathematical definition that reconciles the formal definition in lines 186-188 with the simpler characterizations used throughout the paper (e.g., \"computational advantage\" or \"performs like a dense model with N times the active parameters\")? Under what assumptions does EL become independent of compute budget?\n\n2. **How did you generate the (MoE, dense) run pairs for EL datapoints?** What is the exact methodology for pairing runs? Do you train dense models specifically to match the loss of each MoE run, or do you use some form of interpolation/prediction?\n\n3. **Do you drop tokens at any point?** Please clarify whether token dropping occurs and if any expert capacity mechanism is implemented during training, evaluation, or both.\n\n4. **What is the quality of fit for your scaling laws?** The plots show that test performance appears worse than train performance, but there is no discussion of fit quality. Did you perform a detailed analysis similar to, for example, Besiroglu et al. (2024)? Specifically:\n   - What are the RMSE values for train and test?\n   - How does fit quality vary across different dimensions (model size, granularity, sparsity)?\n   - What do the marginal distributions look like?\n   - Are there systematic patterns in the residuals?\n5. **How was load balancing implemented?** Was it run locally, or were the routing statistics all-reduced across e.g. the expert parallel group or globally? Please provide implementation details.\n6. **What is your intuition behind granularity performing worse beyond a certain point?** The paper states that \"excessively fine-grained experts suffer from insufficient capacity, while overly coarse-grained ones fail to achieve effective specialization.\" What evidence or analysis supports this claim? I was only able to find an anlysis on model quality (training perplexity), but without any analysis on specialization or otherwise.\n---\n\n## References\n\nBesiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). Chinchilla Scaling: A replication attempt. arXiv preprint arXiv:2404.10102.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Conceptual contribution:**\n\nEL has the potential to be a useful metric that simplifies efficiency comparisons between MoE and dense architectures. However, the metric needs to be precisely defined, and its usage throughout the paper must remain consistent with that definition, with any claims properly justified.\n\n**Empirical validation:**\n\nThe validation experiments successfully corroborate the paper's claims. The authors train models at significantly larger scales than those used to fit the scaling law, and the predictions hold. This is a strong empirical result.\n\n**Methodology:**\n\n1. **Structured experimental foundation**: The authors take a systematic approach to building the foundation for EL scaling law experiments. They follow existing literature to develop two key components:\n   - A system for choosing optimal hyperparameters with respect to compute budget. While they apply insights from only one MoE configuration to other architectures, they validate this transfer \"in the vicinity\" of 16× sparse MoE. The dense architecture is treated separately, so all comparisons are fair with regards to choice of core training hyperparameters.\n   - Scaling laws for efficient budget allocation, once again done separately for dense and Moe models.\n\n2. **Comprehensive architectural analysis**: Before deriving EL scaling laws, the authors systematically study key architectural factors including expert activation ratio, expert granularity, and shared expert ratio. This provides a solid foundation for understanding how these design choices affect the scaling behavior.", "weaknesses": "**Major Issue: Imprecise definition of EL and its inconsistent use throughout the paper, lack of clarity in experimental setup**\n\nThe definition of Efficiency Leverage (lines 186-188) lacks precision, and its usage throughout the paper creates confusion for readers. Several issues arise:\n\n1. **Unclear functional form**: The formal definition of EL makes it a function of three arguments: MoE architecture, MoE compute budget, and dense architecture. However, the authors' usage throughout the text suggests a different understanding. Consider these statements:\n\n   - Lines 16-17: \"a metric quantifying the computational advantage of an MoE model over a dense equivalent\"\n   - Lines 25-27: \"When trained on an identical 1T high-quality token dataset, MoE-mini matched the performance of the 6.1B dense model while consuming over 7× fewer computational resources, thereby confirming the accuracy of our scaling laws\"\n   - Lines 70-72: \"an EL of 5, for example, means the MoE architecture performs like a dense model with five times the active parameters for a similar compute budget\"\n\n   These statements imply EL is an architecture-level property, independent of compute budget. Yet even under favorable assumptions (e.g., Chinchilla-like scaling), given the current definition, EL remains a function of compute budget and the chosen dense architecture.\n\n2. **Missing justification**: Lines 185-189 make claims that do not clearly follow from the stated definition. To improve this, the authors should provide a complete derivation with explicit assumptions and literature references showing how EL reduces to the simpler form.\n\n3. **Section 4 methodology unclear**: \n   - How were the datapoints for EL obtained? What is the exact procedure for training dense models to reach the precise training loss of a given MoE run?\n   - Figure 4 shows both fitted curves and datapoints, but their nature is ambiguous. Are these points from actual experiments or predictions? Figure 3 shows models up to ~3×10²⁰ FLOPs, yet Figure 4 plots extend to at least 1×10²² FLOPs (acknowledging that 4c is explicitly an extrapolation). If the points are predictions, this is misleading—the notation is not clearly specified, and the marked points suggest real experiments. If they are actual experiments, then the final validation run at 1×10²² FLOPs is not truly an extrapolation beyond the fitting range, which diminishes the claimed strength of the scaling law validation.\n\n**Technical issues:**\n\n- In the EL definition, the variable $C_{target}$ is never used and should be removed.\n- The use of ε → 0 in the constraint is redundant. As ε approaches zero, this becomes an equality constraint. Why not write it as an equality directly?\n- Sections 4.2.1 and 4.2.2. are concerned with fitting scaling laws for each variable separately, keeping the other two fixed, and fitting the joint form, respectively. Thus, the name of 4.2.1 should rather be called either marginal or univariate scaling laws. Separable means its already a joint function (so, of all variables), but it's of a special form functional form.\n**Unsupported claim:**\n\n>Lines [249-251]: \"It represents a critical balance: [...] while overly coarse-grained ones fail to achieve effective specialization.\"\n\nThis claim about expert granularity is stated as fact but lacks supporting analysis or evidence in the paper. The only thing that is supported is that increased granularity deteriorates model quality from some point on - though, judging by the analysis in Figure 13, it is possible that simply having the balancing loss be a function of granularity (more experts -> more balancing needed) will make granularity once again effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)