Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Authors: Samira Abnar
Harshay Shah
Dan Busbridge
Alaaeldin El-Nouby
Joshua M. Susskind
Vimal Thilak
Venue: ICML 2025 poster (2025)
Keywords: Mixture-of-Experts
Scaling Laws
Primary area: deep_learning->large_language_models
TL;DR: This paper presents an empirical investigation into the trade-off between parameter count and FLOPs per token in scaling Mixture-of-Experts (MoE) language models.

Abstract: Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the compute per example. While scaling typically involves increasing both, the precise interplay between these factors and their combined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of-Expert models (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the ratio of non-active to total parameters, affects model performance in terms of both pretraining and downstream performance. We find that under different constraints (e.g. parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance.These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Samira Abnar
Harshay Shah 2
Dan Busbridge
Alaaeldin El-Nouby'
Joshua M Susskind
Vimal Thilak"
Abstract
stream task performance (Devlin al., 2019; Brown et al., 2020; BIG-bench authors, 2023) and unlocks emergent ca- pabilities (Wei et al., 2022a).
Scaling the capacity of language models has con- sistently proven to be reliable approach for im- proving performance and unlocking new capa- bilities. Capacity can be primarily defined by two dimensions: the number of model parame- ters and the compute per example. While scal- ing typically involves increasing both, the precise interplay between these factors and their com- bined contribution to overall capacity remains not fully understood. We explore this relationship in the context of sparse Mixture-of- of-Experts (MoEs), which allow scaling the number of parameters without proportionally increasing the FLOPs per example. We investigate how varying the sparsity level, i.e., the fraction of inactive parameters, im- pacts model's performance during pretraining and downstream few-shot evaluation. We find that un- der different constraints (e.g., parameter size and total training compute), there is an optimal level of sparsity that improves both training efficiency and model performance. These results provide a better understanding of the impact of sparsity in scaling laws for MoEs and complement existing works in this area, offering insights for designing more efficient architectures.
A recurring notion in these studies is that model capacity is well quantified by the total number of model parame- ters. However, the number of parameters is not the only means increase model capacity. Compute per example (i.e., a fixed-sized input), measured in FLoating OPerations (FLOPs), also plays significant role (Clark al., 2022). In fact, several mechanisms (Shazeer et al., 2017; Dehghani et al., 2019; Wei et al., 2022b; Goyal et al., 2024; Csord' as et al., 2024) allow for independent variation of the number of parameters or FLOPs per example within a model. For in- stance, Sparse Mixture-of-Expert (MoE) models (Shazeer et al., 2017) introduce "FLOP-free parameters" by lever- aging sparsity, where only a subset of expert modules is activated for each input.
When studying scaling laws for specific classes of models, e.g., vanilla transformers, the total number of parameters can serve as a reasonable relative estimator of FLOPs per example. Therefore, using the number of parameters as a measure of model capacity in scaling law studies is appro- priate. In scenarios or for architectures where the number of parameters and FLOPs per example are not directly linked, it is essential to jointly consider the effects of these variables on scaling model capacity (Clark et al., 2022). We therefore ask
1. Introduction
"Can we draw scaling laws for the optimal trade-off' between parameter count and FLOPs per example?"
Empirical scaling laws for language model pretraining (Ka- plan et al., 2020; Hoffmann et al., 2022; OpenAI, 2023; 2024; Gemini Team et al., 2024; Henighan et al., 2020; Clark et al., 2022; Yun et al., 2024; Ludziejewski et al., 2024) have demonstrated that proportionally increasing model capac- ity, along with data and total compute budget, consistently decreases pretraining loss (i.e., perplexity), improves down-
To address this question, we study sparse Mixture-of-Exxpe Transformers (MoEs) (Shazeer et al., 2017: Lepikhin et al., 2021: Fedus et al., 2022: Zoph et al., 2022: Muennighoff et al., 2024) in the context of language modeling. Exist- ing scaling law studies for MoEs, investigate the role of variables like number and granularity (Ludziejewski et al., 2024) of experts, underlying dense model size and inference compute in predicting the performance the models under different conditions such as training or inference compute optimality (Du al., 2021; Clark al., 2022; Yun et al.,
"Core Contributors I 'Apple "MIT. Correspondence Samira Abnar <abnar @ app>, Vimal Thilak <vthilak @aalaakaappp.ccoo apple.com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixturre-ff-Expert Language Models
Sparsity
Loss C
Sparsity
Loss
0%
50%
98%
2.2
2.9
0%
50%
98%
2.1
2.8
2.8
2.8
2.7 4
4
2.6
0
2.5
2.4
2.3
2.2
204M aee 0.00
MoE Sparsity S
MoE Sparsity
268
18
800M
485M
800M
485M
294M
178M
109M
BBM
Total Parameters
Active Parameters N Wattive
(a) IsoFLOP surface over sparsity and total parameters
(b) IsoFLOP surface over sparsity and active parameters
Figure 1. IsoFLOP surface over observed prefraining loss L, model size (in ferms of total N and active parameters Na), and sparsity S. We fit a polynomial function mapping N (or N.). S. and their interaction to L, using empirical data. For both fits the MSE loss for predicting loss held out 0.0001. These results indicate that for fixed compute budget, increasing model sparsity leads to a reduction in pretraining loss. When considering optimal model size, we observe opposite trends for total parameters (N) (Figure a) versus active parameters (Na) (Figure b). (See Figure in Appendix D.1 for results with different total compute budgets C.)
2024; Ludziejewski et al., 2024). In this paper, we focus on the interaction between FLOPs per example and total parameter count, and their impact on model performance in MoEs, through large-scale empirical study.
formance and the relationship between upstream and downstream performance is not impacted by the spar- sity level. However, on downstream tasks that pre- sumably require more "reasoning", we observe that for models with the same perplexity on the pretraining data distribution, sparser models, i.c., models with fewer active parameters, perform worse.
We define sparsity the ratio of inactive experts to the total number of experts, which controls the ratio of the total number parameters to FLOPs per example in MoEs. We evaluate loss and downstream metrics for different sparsities, model sizes, and compute budgets. Through qualitative and quantitative analysis derive scaling laws which disentan- gle total parameters FLOPs per example in MoEs, we can estimate the optimal sparsity level under the setting where both total training FLOPs and total number of parameters are given and fixed. Generally, we find that:
Our results, in line with findings from previous relevant studies (Ludziejewski et al., 2024; He, 2024) scaling laws for MoEs, show increasing sparsity level leads to better performance and efficiency during pretraining. Consider- ing the various methods increase compute per example during inference adaptively conditioned on task or example complexity, we conclude that approaches like MoEs, which reduce the unit compute cost (i.e., FLOPs per token) by increasing the sparsity level, hold significant promise given their potential to enhance efficiency in both pretraining and inference.
During pretraining, increasing model's capacity by adding more parameters yields greater benefits than in- creasing FLOPs per example. We observe that the size of compute-optimal models increases as we increase the training budget (measured terms of total FLOPs) while the active number of parameters, hence FLOPS per example, decrease for compute-optimal models.
2. The Interplay between Model Parameters and Sparsity in MoEs
During inference, FLOPs per example seem to play more important role!. For many tasks, upstream performance is good predictor of downstream per-
Is there an optimal trade-off between parameter count and FLOPs per example MoEs under the setting where the training compute budget (i.e., total training FLOPs) is fixed?
'A relevant discussion here is the recent trend of increasing test- time compute, e.g., OpenAl ol model (OpenAI, 2024), achieved
by generating tokens way for introducing parameter- free-FLOPs.
2

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixturre-of-Epperts Language Models
(a) Optimal MoE Sparsity S*
(b) Optimal Total Parameters N*
[c) Optimal Active Parameters N
Optimal MoE Sparsky
Optimal total parameters
2.7
Optimal active parameters
2.
3.0
2.8
2.6
Setraning Loss
Pretraining Loos
2.7
2.8
0.66
Pretraining
2.5
2.6
2.4
2.4
2.3
22
0%
63%
86%
95%
99%
176M
46
268
178M
2949
485M
200M
1B
MoE Sparsity S
Total Parameters N
Active Parameters No
500M
2B Total Parameters N
24%
71%
95%
24%
48% MoE Sparsity 5
95%
MoE Sparsity S
Figure 2. IsoFLOP slices along Sparsity and Model Size (C = 1e20). We use fitted isoFL.P surfaces (Section 2) to analyze how sparsity S and model size N impact the loss L for a fixed compute budget. We identify optimal points by (a) fixing N and varying S, (b) fixing S and varying N and (c) fixing S and varying active parameters Na. Observe that (a) the optimal sparsity S inereases with increasing model size N and converges to while (b) and (c) show that the optimal model size N and active parameter count Na increase and decrcase respectively with increasing sparsity levels. (sec Figure 9 in Appendix D.I for other total training compute budgets.)
Intuitively, under infinite data setting, scaling model capac- ity along with the training compute budget leads perfor- mance improvements. Previous scaling law studies suggest that, conditioned on training compute budget measured in FLOPs denoted by C. the optimal number of parameters, (C), exhibits a power-law relationship with C (Hoff- mann et al., 2022):
into two parts:
1. "How does the spursity level impact the scaling laws of the relationship between N and for optimal models?" To address this question in $2.1, we fix S and vary N, studying how optimal N and Na change for different values of S:
(C) arg min C(N;C) & C
(1)
N = arg min L(N:C,))
(3)
N
Our goal is to study how to optimally trade-off FLOPs per example and total parameters in MoEs. In MoEs the balance between parameters and FLOPs can be expressed through the sparsity level, S. We define S as the ratio of non-active to total number of experts, i.e., S = E-K, where E is the total number of experts and K is the number of selected experts per token. We can vary the sparsity level by either changing the number of active experts K or total number of experts E. Essentially, for models with the same N, the model with higher 5 will have fewer active parameters N.. resulting in fewer FLOPs per example. For more details on the notations and experimental settings see Appendix A and Appendix B.
2. "Is there an optimal balance between total number of parameters and the sparsity level under fixed training- compute budget?" address this question in $2.2, we fix N and vary S, studying how optimal S changes across different values of N:
S arg min C(S:
(4)
g
As the first step, considering fixed training compute budget C, we fit 3D surface, referred the IsoFLOP surface, in Figure 1a, using a polynomial function, following approach II of Hoffmann et al. (2022). Compared to Hoffmann et al. (2022) we include the sparsity variable and fit a single 3d IsoFLOP surface across all data points, rather than fitting separate 2d IsoFLOP curves for fixed sparsity levels or model sizes. We conducted a grid search determine the optimal polynomial degree for N, S, and the interaction term N x S, finding that degree of (2, 2, 2) resulted in the lowest cross-validation error. Both N and S are in log space (see Appendix B for more details).
(N", S*) = arg min S: N,5
To simplify the problem of understanding the joint role of N and S in predicting L, we break the problem, Equation 2,
"Sparsity level determines the number of active parameters given the total number of parameters and we use the active num- her of parameters a proxy for FLOPs per example, as 6NN, D provides good estimate of the total FLOP count for MoEs; see Ap- pendix C for details.
As seen in Figure la, the IsoFLOP surface plot is parabolic along model size, suggesting that the findings of Hoffmann
3

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Centuure-of-Expeert Language Models
et al. (2022) extend MoEs across different sparsity lev- els, i.e., L(N; C,S) is parabolic, with its optimal solution located at the turning point. When considering the total number of parameters N, the optimal value increases as the sparsity level increases, while for the active numher of parameters Na the optimal value decreases with the sparsity level. This indicates that by increasing the sparsity level the training compute optimal models are larger but have fewer FLOPs per example, i.e., lower inference cost. Moreover, along sparsity, the pretraining loss decreases monotonically, indicating that, for the same compute budget, sparser mod- els achieve better pretraining performance. We observe the same pattern across different training compute budgets (See Appendix D.I). To better understand and explain these ob- servations, we examine slices of the IsoFLOP surface along the axes of S and N separately in $2.1 and $2.2, respec- tively.
change as we vary N?
To address this, we look into slices of the IsoFLOP surface along the sparsity dimension. As we can see in Figure 2a, for fixed training compute budget and fixed model size (SS N, C)) exhibits a parabolic profile, reaching its opti- mum value at the vertex where S = S", is noteworthy that for a given total training compute, there is threshold value No. for the total number of parameters, where for larger models, models with N > Net. increasing sparsity always has positive impact, i.e., optimal sparsity level approaches 1.0. More accurately. for a fixed compute budget the opti- mal sparsity level increases with model size and converges to 1 as the model size grows (see Figure 4 in $D.2 in the Appendix for more details). Note that the optimal model, here is not the largest model, i.e., there is a compute optimal model size in terms of total parameters even after sparsity is introduced, and increasing total number of parameters would lead to under-training if training compute budget is fixed.
2.1. Optimal Model Size for Fixed Sparsity Level
Here examine how sparsity influences scaling laws gov- erning the relationship between N, Na and C for training- compute optimal models, i.e. how does N and No, for a given C, S (Equation 3), change as we increase S? Look- ing at slices of the IsoFLOP surface along the model size dimension, in Figure 2b and Figure 2c, we observe how the IsoFLOP curves shift along loss and model size. Consider- ing the training-compute optimal model, for a fixed compute increase sparsity. Furthermore,
These results highlight the importance of balancing the num- ber of parameters with FLOPs per example in MoEs. Intu- itively, when the total number of parameters is small, higher sparsity results in fewer active parameters, and thus fewer FLOPs per example. This reduction in FLOPs per example may lead to inefficiencies during both training and inference. Conversely, when the total number of parameters is large, for a reasonable amount of FLOPs per example, a fixed compute budget may not allow sufficient training on enough tokens to make use of the model's 's additional capacity.
budget, loss decreases while sparser models have larger N compared to denser models, as seen in Figure 2b, they have a smaller active parameter count Na; hence, fewer FLOPs per example. In- tuitively, more parameters in total increase the capacity of the sparser models to fit the data, while fewer number of active parametes, hence fewer FLOPs per example, allow the model to be trained with more tokens, i.e., higher D, for the same training compute budget.
3. Impact of Training Compute Budget on the Interaction between Model Parameters and Sparsity
Does increasing compute budget impact the interaction be- tween the parameters and FLOPs per example in MoEs and how they contribute to model's capacity? In other words, does the recipe for optimally increasing model capacity, i.e., optimal sparsity level for MoEs change as we scale up the total training compute?
2.2. Optimal Sparsity Level for Fixed Model Size
In this section we aim to understand the dynamics between the total number of parameters and FLOPs per example in MoEs. In Section 2.1 we are considering the case where there is no bound on the total number of parameters. In this case, we observe that under fixed training compute budget in terms of FLOPPs, it is better to train sparser models with higher total number of parameters. However in practical scenarios it is reasonable assume that there would be some bounds the memory and hence the total number of parameters of a model. This leads us to a fundamental question: Is there an optimal balance between the total number of parameters and and FL.OPs per example under a fixed training-compute budget? Thus, we investigate the optimal sparsity level when total number of parameters is fixed. Specifically, we ask: Given N and C, How does S
To answer this question. in Figure 3 we illustrate the trends for changing the total number of parameters, N the num- ber of active parameters, NV., and the loss, L", with sparsity level across different compute budgets.
Figure 3c shows that the optimal sparsity level approaches I across all compute budgets used in our experiments. There is significant difference observed in the slope of the loss vs sparsity curves across different training compute budgets used in our experiments. This observation suggests that there is no diminishing effect of sparsity on the pretraining loss as we increase training compute budget, i.e., if there is no constraint on the model size, sparsity improves the
4

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for cexture-offExperrt Language Models
(a) Effect Total FLOPs Con Optimal N(S)
Effect Total FLOPs Optmal NJ(5)
(0) Effest of Total FLOPs C Lons
Optimaal Todn Parumelerss N'
Optimal Agive Parameters M/
Optanal Prozafing Loge
165
84
16.
4050
17MM
66%
0%
40%
62%
60%
40%
MdE. Spansityy S
MrE Sparsily
MoE Sparsit 5
Compusa Sudger (n FLOPS)
16.00
26:00
1821
Figure 3. Effect of compute budget on model size, number of active parameters and loss with sparsity. Across all compute budgets, we observe that (a) the optimal model size N increases with sparsity, (b) the optimal number of active parameters No decreases with sparsity, and (c) the loss L decreases with sparsity.
Effect of Total FLOPs C on (N)
We use downstream tasks from the evaluation suite in 1 11m- foundry" for benchmarking our pretrained mod- cls, specifically in an in-context few-shot learning setup. This setup focuses on evaluating a model's s ability to learn and adapt to new tasks with limited examples. The down- stream task are devided into four pre-defined categories namely: language understanding, world knowledge, reading comprehension, and symbolic reasoning to help us systemat- ically test whether the downstream vs upstream performance trend remains the same or is different as we vary sparsity values.
Optimal MoE Sparsity 5 (NN) C)
48%
25%
65%
0%.
53V
1900
365M
718M
32
1.00
200
Total Parameters N
We observe from Figure 5a (language understanding), Fig- ure 5c (commonsense reasoning). and Figure 5d (world knowledge) that, in an in-context few-shot learning setting, there is a strong correlation between upstream (pretraining) loss and downstream performance (error) across all these tasks. For these tasks, downstream performance in the few- shot setting predictable based on upstream performance, regardless of the sparsity level. This indicates that, in the context of these tasks, the optimal sparsity level follows the same trend as the optimal sparsity observed during pretrain- ing. However, Figure 5b (reading comprehension) shows an example of a task where models with higher sparsity transfer more poorly compared to denser models. This decrease in the transfer performance of sparser models on these tasks may be due to the lower inference-time compute in sparser models compared to their denser counterparts for a similar pretraining loss. Further analysis is needed to verify this intuition.
Compute Budget C
3e19 Be19 FLOPs
1x20 FLOPs
3e20 FLOPs
1e21 FLOPs
Figure 4. Effect of training budget C' and total parameters N on MoE sparsity. Optimal MoE sparsity S changes with respect to the total number of parameters N and the training budget C. The r-axis represents the total parameters N on a logarithmic scale, and the y-axis shows the optimal MoE sparsity S*.
performance of the model across all training budgets.
In Figure 3a and Figure 3b,, we see a consistent trend of increasing N and decreasing N, for compute optimal mod- els as sparsity level increases across all training compute budgets. Moreover, as can be seen in Figure 4, when model size in terms of total number of parameters is fixed, optimal sparsity level decreases with training compute budget while increases with model size as discussed in Section 2.2.
If fewer FLOPs per example are the reason behind the worse transfer performance in sparser models, this effect might di- minish with a larger total training compute budget, as the op- timal active number of parameters increases. Moreover, one can use approaches like chain-of-thought reasoning (Wei et al., 2022b) to independently increase FLOPs per example
4. Effect of MoE Sparsity on Downstream Task Performance
In this section, we study how sparsity affects the relationship between upstream and downstream performance of MoEs. In other words, does sparsity impact the relative gains from improvements in pretraining tasks on downstream tasks?
GGithub repository: httpp::///. tighthub.... com / /mosa cm1 Lim foundry
5

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for cesture-of-Experre Language Models
(a) Language Understanding
(b) Reading Comprehension SQUAD ([D-shot)
(c) Common Sense Reasoning
(d) World Knowledge
Lambada OpenAl (D (0=shot)
PIQA (10:shot)
BIG-bench QA Wikidata (10-shot)
0.75
0,425
0.70
Downstream Taisk Emor
0.65
Downstream Task Enor
0.9
Downstream Tazk Enor
0.400
Downstream Sa Enor
0.8
0.375
0.60
0.7
0.350
0.55
0.325
0.6
0.50
0.300
0.5
0.45
0.6
0.275
0.4
Random chance: 0.75
Random chance: 0.99
Random
Random chance: 0.75
0.40
0.250
0.3
22
2.4
26
2,8
3,0
3.2
22 2.4 2.6 2.8
3.0 3.2
222
24
2.6 28
3.0
3.2
2.2
24
2.6 2,8 3.0 3.2
(Upstream) Pretraining Loss L
(Upstream) Pretraining Loss
(Upstream) Pretrairing Loss L
(Upstream)) Pretraining Loss L
Sparsity: 0% (Derse)
Sparsity: 25%
Sparsity: 50%
Sparsity: 75%
Sparsity: 90%
Sparsity: 95%
Figure 5. Effect of sparsity on downstream vs upstream performance. Downstream error shows a tight relationship with pretraining ("upstreamm)) loss across downstream tasks across all sparsity levels.
during inference time.
measures this relationship, where a larger 3 implies greater benefit from increased data. The constant represents an asymptotic minimum for the loss, as both model size and dataset size approach infinity.
In Appendix E, we explore whether increasing inference- time compute via Chain-of- Shain-of-Thhughh Thought (CoT) prompting can improve the performance of MoEs on tasks that require more reasoning. Our experiments indicate that MoEs benefit more from this increased compute compared to dense models with a similar number of active parameters. This suggests that dynamic compute allocation during inference may be crucial for MoEs to perform well on complex reasoning tasks.
For dense models with fixed total training FLOPs, C, the parameters N and D are interrelated through the equation for estimating FLOPs per example, given as = 6NN D for transformers. However, in MoEs (Mixture of Experts models), this relationship involves the active number of pa- rameters N₂ rather than the total parameter count N. Thus, D and Na define the total training FLOPs rather than D and N. Given the analysis conducted in $2, we know that if the total number of parameters N is fixed, the optimal sparsity level, i.c., active number of parameters would depend on N. Motivated by this observation, we suggest the following parametric form that includes a multiplicative interaction between N and S or No to predict the loss:
While our results may indicate that there may be ad- ditional benefit obtained via sparsity in MoEs beyond the efficiency gains for pretraining, we caution the reader that this suggestion may be an artifact of the scale of our exper- iments. In the end, since, as shown in $2, sparser models are more efficient both in terms of training and inference cost (when measured in terms of theoretical FL.OPs), we can reach better pretraining performance with higher spar- sity levels at lower cost, which can translate to better downstream performance.
a b
+ d
L(N, = Na +
+e D + (1 (1-S)A (1 5) Nv
5. Incorporating Sparsity into Scaling Laws
(6) The term (1 - S) in the above equation provides a rough estimate of the percentage of active parameters. If the expo- nent for the multiplicative terms is the same then that term provides approximate estimate of the number of active parameters.
The scaling laws proposed by Kaplan et al. (2020) provide framework for predicting loss in dense models by estab- lishing power-law relationship between loss L, number of parameters N and dataset size D, where N and D interact linearly. Formally, the relationship is given by:
By incorporating sparsity into the scaling law equation, we can eliminate the need for parameters specific to MoEs, such as the total and active number of experts. As demonstrated by Frantar et al. (2024), this formulation also holds for other sparsity mechanisms, such as weight sparsity, where individual neural network connections are pruned.
L(N,D) = a b D8
te
(5)
Na
Here, the term N captures the inverse relationship between model size and loss, where an increase in model size N leads to reduction in loss. The exponent quantifies the rate of this decrease; a larger a suggests a steeper reduction in loss with increasing model size. Similarly, the term 3 indicates the impact of dataset size D on loss, with larger datasets contributing to lower loss values. The exponent /
We use the recipe described by Hoffmann et al. (2022) and use the L-BFGS algorithm to fit the coefficients in equa- tion 6 using a Huber loss with = 10 3 Optimal coef- ficient values were determined through a grid search (see
6

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Expert Language Models
Table 2 for search values). The results of data fitting and validation are shown in Figure 6. The estimated values are shown in Table 3 in Appendix F.
While the trend of increasing active number of pa- rameters is similar across all training compute bud- gets; the optimal active number of parameters de- crease more rapidly with sparsity as the training compute budget increases (see $3).
6. Discussion
Effect of Sparsity on Downstream Performance: For most downstream tasks, models with similar pretraining perplexity have similar downstream task performance re- gardless of sparsity. For reading comprehension tasks (e.g., CoQA (Reddy et al., 2019), SQuAD (Rajpurkar et al., 2018)), denser models perform better, poten- tially due to their higher inference- compute than perplexity matched sparse model. Strategies to increase inference time compute dynamically (Wei et al., 2022b; Goyal et al., 2024) may address this gap.
Our findings amplify the findings of Ludziejewski et al. (2024) and further justify the effort to work toward MoEs with experts larger in number and smaller in size (He, 2024). For downstream tasks which their performance is pre- dictable given the pretraining loss (i.e., perplexity), sparsity potentially provides efficiency gains both during pretraining and inference.
Here a summary of our observations as discussed in Sec- tions 2 to 5
Parametric Scaling Law: We propose parametric form for scaling laws that accounts for sparsity. The model coef- ficients are estimated using the empirical data obtained by training compute-optima models. An interesting obser- vation from Appendix is that the exponent for sparsity term is negative which is consistent with our intuition that sparser models lead to lower perplexity.
Larger, Sparser Models Perform Better under a Fixed When memory and communication overheads are disregarded, increasing sparsity while pro- portionally expanding the total number of parameters con- sistently leads to a lower pretraining loss, even when constrained by a fixed training compute budget (see $ 2).
Compute Budget:
Optimal Sparsity for Fixed Model Size: For any given number of parameters and under a fixed training compute budget, model performance as function of sparsity ex- hibits a parabolic pattern, reaching its peak at an optimal sparsity level (see $2.2). Specifically, the optimal sparsity level:
6.1. I. Limitations
In our analysis, similar to other scaling law studies (Kaplan et al., 2020; Hoffmann et al., 2022), we have measured the costs for both training and inference exclusively in terms of FLOPs. While there may be discrepancies between actual computational costs and theoretical FLOPs due to hardware specifications, infrastructure, and implementation details, it is reasonable to abstract away from these factors when comparing similar models under fixed conditions. However, important aspect not accounted for in this study is the cost associated with memory usage and communication overhead, which could potentially increase as we raise the sparsity level. Incorporating these factors is challenging because they are highly dependent on the hardware used. To address this limitation to some extent, in Section 2.2 we investigate the optimal sparsity level under the setting where total number of parameters is fixed.
Increases with the total number of parameters ap- proaching 1.0 for larger models. i.e., if a model is relatively small for a given training compute budget, sparsifying it more than a threshold will hurt its per- formance. On the other hand, if a model relatively large for a given compute budget, further sparsifying it helps as it leads to increase in the number of to- kens the model is trained on under the given training budget constraints (see $2.2).
Increases across all model sizes as the training com- pute budget increases (see $D.1 and $D.2).
Effect of Sparsity on Scaling Laws for Optimal Model Size: For any specific sparsity level, performance of the models as a function of their size exhibits parabolic be- havior under a fixed training compute budget. i.e., the model reaches its optimal performance at a vertex, that indicates optimal model size. Under these conditions:
Despite the limitation with using an approximate method to quantify FLOPs, our findings highlight the importance of investing in methods to enhance the efficiency of sparse Mixture-of-Exxe -xffexxettt models. By increasing model capacity through additional parameters while minimizing per-unit computation costs, these models have the potential to im- prove both efficiency and performance. The availability of GPUs with larger memory, for e.g., the recently intro- duced H200 GPU chip with 141 GB of memory as well as improving the efficiency of training and deployment pipelines (NeMo Authors, 2025) suggest that there is sig- nificant interest in developing efficient implementations for
The optimal active number of parameters decreases as the sparsity level increases, leading to smaller FLOPs per example and more efficient inference even though the total number of parameters increases (see 1).
7

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- -fptuurr--f-Exeets Language Models
Phrere
2.4
4C0-000
2.3
400.200
500.00
Predict Losg
292.900
200.308
156.000
2.2
100000
24
Observed
(a) on data used to estimate coefficients.
(b) Validating scaling law on held-out datasct.
Figure Scaling law fit data obtained from training compute-optinal models. Figure 6(a) shows fit the data used to estimate the coefficients for equation 6, while Figure 6(b) validates these coefficients on held-out dataset. All data points with S = 0.98 were excluded from the fitting process for sulpll-sample validation. The dashed lines represent equal loss values.
MoEs.
7.2. Scaling Laws for MoEs
Mixture-of-Expert -f-eertsss (MoE) models (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022; DeepSeek-Al. 2025) have emerged
7. Related Work
powerful architecture for language modeling, primarily because they decouple computational cost from parameter count. This separation between param- eters and FI FLOPs per token in MoF architectures calls for scaling laws that can accurately factor in the contributions of both.
7.1. Scaling Laws for Language Models
Scaling laws have proven to be powerful framework for understanding and predicting the performance of language models. Existing studies, such as Kaplan et al. (2020) and Hoffmann et al. (2022), reveal that power-law rela- tionships govern model performance as function factors like model size, data size, and compute budget, offering predictable performance improvements with increased re- sources.
Previous research on the scaling behavior of MoE models has established foundational scaling laws, incorporating factors such as total parameter count, the number of experts, and the granularity of these experts (Clark al., 2022; Ludziejewski et al., 2024; Wang et al., 2024). However, these studies typically assume a fixed configuration for other critical variables influencing FL.OPs per token, such as the number of active experts per input. In contrast, we propose generalized scaling law that considers variables like active parameter count and sparsity level, thereby expanding the applicability of MoE scaling laws.
Hoffmann et al. (2022) emphasizes the critical balance be- tween model size and the number of training tokens when the training compute budget is fixed, showing that scaling the model without corresponding data increases can lead to suboptimal performance. Additionally, DeepSeek-AN (2024) explores more nuanced scaling behaviors by incorporating data quality, demonstrating that higher-quality data allows for more efficient scaling, and thus, larger portion of the compute budget should be allocated to increasing model size.
A common theme in the literature suggests that training sparser models-achieved by increasing the number of smaller experts-offet significant gains in efficiency for both pretraining and inference phases. Through compre- hensive large-scale study, we provide empirical evidence for this, analyzing the impact of sparsity level efficiency and defining optimal configurations.
Recent work extends scaling law analysis to specialized contexts, including over-training (Gadre et al., 2024). down- stream task performance, and multilingual or multi-modal settings, where scaling laws provide valuable insights and can be adapted to address specific challenges.
Supporting this. Du et al. (2021) demonstrates GLaM's superior efficiency and performance compared to GPT-3, showing that MoE architectures can achieve high perfor- mance with significantly lower computational and energy
8

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for cesture-offExpertt Language Models
costs. Further insights are offered by Clark al. (2022), who analyze scaling behaviors across various MoE routing techniques. While their study finds that MoEs generally outperform dense models, it also notes diminishing bene- fits as base model sizes grow. Ludziejewski et al. (2024) challenge this conclusion, attributing the diminished returns partly to the fixed number of training tokens across models and constant expert sizes, By introducing "granularity" and adjusting training durations, they demonstrate that MoEs can outperform dense models across any compute budget, debunking the notion of diminishing returns for MoEs with adaptive expert configurations. More recently, Jelassi et al. (2024) finds that, on downstream tasks, MoEs scale effi- ciently with the number of experts (i.e., increasing sparsity) on memorization tasks, but their reasoning capabilities satu- rate and lag behind dense models on tasks requiring complex reasoning when compared based on total number of param- eters.
per example, higher number of parameters, and lower pre- training loss simultaneously. In other words, in the context of MoEs, if there are no constraints on the total number of parameters, increasing the capacity of the model through parameter count seem to be the optimal strategy if lower pretraining loss is the main goal. On the other hand, when comparing how well the pretraining performance transfers to various downstream tasks, denser models exhibit better transfer performance certain types of task that potentially rely on deeper processing of the input vs the knowledge stored in the parameters of the model. This potentially sig- nals the importance of the role of FLOPs per example in increasing the capacity of the model during inference. Our experiments demonstrate that MoEs use Chain-of-Thoughh prompting more effectively than dense models, achieving better performance when allocated additional computational resources during inference. This observation reveals an in- teresting direction to improve the performance efficiency of MoEs at inference time.
Another approach by He (2024) explores the benefits of training MoEs with larger numbers of smaller experts rather than the conventional setup of fewer, larger experts. They introduce Parameter Efficient Expert Retrieval (PEER), novel routing mechanism designed tackle the computa- tional and optimization challenges that arise when handling a high number of experts, thus enabling efficient scaling of MoE models.
Future work will focus on determining the optimal balance between FLOPs per example and parameter count, with an emphasis on conducting in-depth analyses of model perfor- mance across diverse downstream tasks. A key direction will involve exploring strategies to balance parameter allo- cation and computational demands to minimize inference costs. Developing scaling law studies to identify optimal approaches for achieving efficiency and performance during inference represents a critical area for further investigation.
Lastly, Yun et al. (2024) draws attention to the increased inference costs associated with scaling MoEs by adding experts. While additional experts may not substantially affect training costs, they can inflate inference costs, thereby diminishing deployment efficiency. To address this, the study proposes an over-trained budget allocation strategy, optimizing MoE models for both performance and efficiency in deployment.
Another important avenue will be to examine how the find- ings on the role of sparsity in MoEs generalize to archi- tectures approaches that employ different mechanisms for independently adjusting FLOPs per example and the number of trainable parameters. Additionally, an intriguing direction for future exploration is the study of scaling behav- iors in models that enable negative sparsity values through parameter sharing.
8. Conclusion
In this paper, we investigated the optimal trade-off between parameters and compute per example for maximizing model capacity. Our findings indicate that sparsity, as knob that controls FLOPs per example in MoEs, is a powerful mecha- nism for optimizing model performance under constrained training compute budgets. By balancing the total number of parameters, compute, and sparsity, Moßs can be scaled more effectiivlyy. These insights provide valuable guidance for scaling language models, especially for MoEs, where the trade-offs between parameters and FLOPs must be carefully managed.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
A Acknowledgmentss
The authors would like to thank Vaishaal Shankar, Far- tash Faghri, Skyler Seto, Mustafa Shukor, Amitis Shidani, David Grangier, Etai Littwin. Alexander Toshev and Prec- tum Nakkiran for their insightful discussions, feedback and technical support that significantly contributed to the devel- opment of this paper.
MoEs were originally introduced to allow increasing model capacity without a significant increase in inference cost. Our experiments show that under fixed total training compute budget increasing sparsity in MoEs leads to smaller FLOPs
9

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Centture-of-xxppere Language Models
References
universal transformers. 2024. URL https:/ /topss://ap/ /api....... http://ppp........e.eeee//m sww.ntmmaniii.. cichonlr.cchoaar.
ArXiv, abs/2405.16039,
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan,
org Coo.ooaD:
Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L.,, Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C.. Lu, K. Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang,
DeepSeek-AL. Deepseek LLM: Scaling open-source lan- guage models with longtermism. ArXiv, abs/2401.02995 2024. URL https://app. httatee/aaallsemmttiiiccc. org//oorpusD::26618833 www.sti
A., Yang, H., Yang, J., Yang. S., Yao. Y.. Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprini arXiv: 16609, 2023.
DeepSeek-Al. Deepseek-rl: Incentivizing reasoning ca- pability in llms via reinforcement learning. https:// github. com/ hwtp/eeeppeeeaaeeeeeeee /eeepeeeeeeeellaeeeee uary 2025. Accessed: 2025-01-21.
BIG-bench authors. Beyond the imitation game: Quantify- ing and extrapolating the capabilities of language mod- cls. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL https:/ /ttp:///eeeeeeeee.eeeee net/forue wtt////rummddiieeeem wiwiidduuyyyysSS11.
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Kaiser, L. Universal transformers. In In- ternational Conference on Learning Representations, 2019. URL htths:/ htpp///ppeeeeeeeeeeeeeeeeeeeeeep net/ forum?? i dd=HyzdRiRYY,
/
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, I.., He, H., Leahy, C., McDonell, K., Phang, J. et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 2022.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pp. 4171-4186, Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: (0.118653/1//111-11000 URL https://aclarttool/ wtt.:/gg......oggg......
Brown, T,, Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.. Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL cc
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph. B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstemm K. S., Duke, T., Dixon, L., Zhang, Le, Q. V., Wu, Y., Chen, Z. and Cui, C. Glam: Efficient scaling of language models with mixture-of-experts. ArXiv, abs/21 12.22200.5 2021. URL https://apt semanticscholas org Btts/CooppSII::4512111
http:://pppcceeenggg.eeeeeeee 145 (5577odebbcca1967111118888888883 pdf.
fi /ttppppeerflllleeppppp////////11
Paper.
Clark, A., Casas, D. d. I., Guy, A., Mensch, A., Paganini, M.. Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., Driessche, G. d., Rutherford, E., Henni- gan, T., Johnson, M. Millican, K., Cassirer, A., Jones, C., Buchattkkaa,, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Rae, J., Elsen, E., Kavukcuoglu, K., and Simonyan, K. Unified scaling laws for routed language models. In Proceedings of the 39th International Confer- ence on Machine Learning. PMLR, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Amy Yan and, A. F., and et al. The llama herd of models. arXiv preprint arXiv: 2407.21783, 2024.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: scaling trillion parameter models with simple and ef- ficient sparsity. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1 14168, 2021.
Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D., and Evci, U. Scaling laws for sparsely foun- dation models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / openrevieww net forum? d= 9K22NKYIP.
Csord'as as, R., Irie, K., Schmidhuber, J., Potts, C., and Manning, C. D. Moeut: -fxeureoof-experts
10

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Exprrtssss-sesssssssr Language Models
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J. Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jit- sev, J.. Dimakis, A. G., Ilharco, G., Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muennighoff, N., and Schmidt, L. Language models scale reliably with over-training and on downstream tasks. CoRR, abs/2403.08540, 2024. URL https: pttttp::/doi.orree org/10. 48550/ 2wt.xiii..40.... 08540.
memorization more than reasoning. arXiv preprint arXiv:2410 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL pestps://arx/a org/pdf/2001. 11.361p......
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (GS] Scal- ing giant models with conditional computation and auto- matic sharding. In International Conference Learning Representations, 2021. URL https: oppp/eveen.eeeeeeeeeee net / ht//foomm/// www.:mmmiiiiwwiieeee 7XHTmbb.
Gale, T. Narayanan, D., Young, C., and Zaharia, M. MegaBlocks: Efficient Sparse Training with Mixture-of- Experts. Proceedings of Machine Learning and Systems, 5, 2023.
Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.- B., Yu, J., Sorieut, R., Schalkwyk, J., Dai, A. M., Hauth,
Li, Q.. Cui, L., Zhao, X., Kong, L., and Bi, W. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. arXiv preprint wrXiv:2402.19255, 2024.
A. et al. Gemini: A family of highly capable multi- org/
modal models, 2024. URL abs 221 118005
Ludziejewski, J., Krajewski, J., Adamezewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóidž, T., Sankowski, P., Cygan, M., and Jaszezur, S. Scaling laws for fine-grained mixture of experts. In ICLR 2024 Workshop on Mathematical and Empirical Un- derstanding of Foundation Models, 2024. URL https: / //openrevi ew net. forum? ?
Goyal, S., Ji. Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. Think before you speak: Train- ing language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. URL https: wpwpe/eeevvveeeeeeeee n forum? id pcC4CRCC..
d= d rr
He, X. O. Mixture of a million experts. arXiv preprint 240:02407.0413,, 2024.
Mirzadeh, I., Alizadeh, K., Shahrokhi. H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 2024.
Henighan, T., Kaplan, J.. Katz, M., Chen, M., Hesse, C.,, Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ry- der, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive genera- tive modeling. arXiv preprint arXiv: Arxiv2010.117001, 2020.
Muennighoff, N., Soldaini, L.. Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert, N., Gu, Y., Arora, S., Bhagia, A., Schwenk, D., Wadden, D., Wettig, A., Hui, B., Dettmers, T., Kiela, D., Farhadi, A., Smith, N. A., Koh, P. W., Singh, A., and Hajishirzi, H. Olmoe: Open mixture- language models, 2024. URL https:::: arxiv. org. abs 200.904000000 02060.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S. Simonyan, K., Elsen, F.,, Vinyals, O., Rae, J., and Sifre, L. An empirical analysis of compute-optimal large language model training. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022. URL htttpp:://prccceddings... neurips. cc /te/pppee_flllees/peerr///////1/ dle2fafff6s888770331111eeee pdf.
NeMo Authors. Nemo: toolkit for conversational ai and large language models. https://githhn com/ NVTD AADDIAANMMMM 2025.
OpenAl. Gpt-4 technical report. PREPRINT, 2023.
OpenAL. Openai ol system card. arXiv preprint arXiv: 2472. 2024.
Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for SQuAD. In Con Gureyych. I. and Miyao, Y. (eds.), Proceedings of the 56th Anmuaa Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Compu- tational Linguistics. doi: 10. 10.18653//1/P18-22222 URL wwwp:ogg.orr...881
Paper-
Jelassi, S., Mohri, C., Brandfonbbeeerr, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y.. Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixtuure-ffEEpeert Language Models
Reddy, S., Chen, D., and Manning, C. D. CoQA: A conversational question answering challenge. Transac- tions of the Association for Computational Linguistics, 7:249-266, 2019. doi: 10.1162/tacl 00266. URL https:/ acl antho wtgg/19----0 9-1066
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Slack, D., Lyu, Q., et al. A careful ex- amination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332, 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural layer.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. ST-MoE: designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.0800 08906, 2022.
networks: The sparsely-gated mixture- In International Conference on Learning Representations, 2017. URL https: http:///opppeeeeeeeeeeeee htwp:rrviieeeee//eoeeeeeee forum? id=B1ckMDqlg
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset. ht https: //ithbb. com//togettarrcommtttee////eee a jamaa- Data, April 2023. Accessed: YYYY-MM-DD.
Wang. S., Chen, Z., Li, B., He, K., Zhang, M., and Wang, Scaling laws across model architectures: A compara- tive analysis dense and MoE models large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference Empiri- cal Methods in Natural Language Processing, pp. 5583- 5595, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1118655/11/2024 mainp- URL https: http:///claathhoogggo org/ 2024.202.000 emnlp main.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL https:// openreview.co net forttt/orm?eaa (tloorrmmmdyyysssssdsaaaaa Sur- vey Certification.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022b. URL https: / htt.aapss/ppeeeevee..ccco www.oeeeee..nee..eeeeee.. net./
Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 14322, 2023.
Yun, L., Zhuang, Y. Fu, Y., Xing, E. P., and Zhang, H. To- ward inference-optimal mixture-of-exper large language models. arXiv preprint arXiv:2404. 022552, 2024.
12

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Expretsssssssssssssss Language Models
A. Preliminaries
A.I. Notation and Terminology
To aid readability, we provide a list of key symbols used throughout this paper.
Symbol
Description
N
Total number of model parameters
N
Active number of model parameters
s
Sparsity level (ratio of non-active to total experts)
S
Optimal sparsity level
L
Pretraining Loss (Categorical Cross-Entropyy
L
Optimal pretraining loss
C N
Total training compute budget (in FLOOPs)
Optimal total number parameters
No
Optimal active number of parameters Expansion factor (number of experts per MoE layer)
E
K
Number of selected experts per token Granularity of experts (size relative to base MLP)
G
Dataset size (number of training tokens)
8, N, a, d,
Coefficients in the parametric scaling law equation
In this paper, we use the term "compute" in a general sense refer computational cost. Unless otherwise specified, "compute" and "FLOPs" (Floating Point Operations) are used interchangeably to quantify this cost.
A.2. Mixture-of- (MoE) Transformers
Mixture-of -fpprttss Transformers modify the standard transformer architecture by introducing in the MLP layer. In this design, the experts are MLP (Multi-Layer Perceptron) modules that follow the attention mechanism and are seleetively activated for each token. A gating mechanism determines which MI. experts are most relevant for each token, ensuring that only a subset of experts (top-k) k) is active at any given time, while the rest remain inactive. Below, we provide the notations used throughout the paper for various terms related to training MoEs.
Total and Active Parameters: MoEs, we distinguish between total and active parameters, denoted by N and Na. respectively. The total parameter count, N, includes all parameters of the network, encompassing both the experts and the rest of the architecture. The active parameter count, N.. refers the parameters associated with the active portion of the experts, along with the rest of the network that is always utilized.
Top-k Expert Selection: In MoEs, the gating mechanism assigns tokens subset of experts using top-k selection process, where & denotes the number of experts activated for each token. The gate computes a relevance score for each expert, and the top experts with the highest scores are selected and activated. This selective activation limits the computational overhead by ensuring that only a fraction the experts are used per token.
Expansion Factor and Granularity: The expansion factor, typically denoted by E, represents the increase in model capacity due the inclusion of multiple experts, measured multiplicative factor relative to the base dense model. The granularity, G, determines the size of each expert relative the size of the MLP module in the base dense model. The total number of experts in the model is given by E x G, where E scales the capacity and G controls the level of granularity.
In general, sparsity is defined as the ratio of inactive to total parameters. However, in the context of MoEs, we focus on the sparsity of the MLP modules specifically. Therefore, we define the sparsity level as the ratio of inactive to total experts, given by:
Sparsity (S):
5 number of non-active experts
(7)
number of total experts
13

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for ceatureeof-Experty Language Models
This definition provides interpretable measure of sparsity but cannot be directly used to calculate the active parameter count N₂ due to the contribution of other parameters in the model that remain unsparsified.
14

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Exprrtsssxsssssssssss Language Models
B. Experimental Setup
We train and evaluate auto-regressive sparse Mixture-of-Experts (MoE) language models of varying sizes and configurations on subsets of the RedPajamaVI dataset (Together Computer, 2023). The key variables we explore in our experiments are total model parameters N, training compute budget C, and the MoE sparsity S.
Pre-training data. Our models pre-trained subsets of the RedPajamaV] dataset' (Together Computer, 2023), which attempts to replicate the LLaMA pre-training data recipe and comprises 1.2 trillion tokens from sources such as Common Crawl, C4, GitHub, and Wikipedia. In all our experiments, the effective dataset size is adjusted based on the training compute budget C and the model size N. We tokenize the data using the GPT-NeoX tokenizer (Black et al., 2022), which has a vocabulary size of 50, 432 tokens.
Model and tokenizer. We use auto-regressive transformer-basee MoE language models in order to study compute- parameter trade-offs by varying MoE sparsity. We use the Megablocks library (Gale et al., 2023) to train dropless MoEs in which the routing mechanism ensures that all tokens are efficiently routed without being dropped due to routing capacity constraints.
Optimizer and scheduler. We optimize our models using the scale-free Adam optimizer' with variable learning rate, a weight decay of x 10 and fixed Adam-specific parameters 3 = (0.9, 0.95) and E = 1 x 10-8, 80-8, We use a learning rate scheduler consisting of linear warm-up phase followed by a cosine decay. The warm-up phase increases the learning rate from 0 to the base learning rate fraction of the total training steps (selected from {0.1, 0.05, 0.02)). After warm-up, the learning rate decays following cosine schedule for the remaining training steps.
Fitting IsoFLOP surfaces. Recall that in Section 2, we fit isoFLOP surfaces to predict pretraining loss L as polynomial function of model size N and MoE sparsity S for fixed training budget C. The polynomial function takes the form
SN =
N
S +
C(NN Sy +
(8)
t=1
where N No and $ log(1 -)-we find that applying log transformations improves the fit of the resulting 3, 4}, found that the best fit was obtained for = B = y 2, i.e., a quadratic polynomial over Ñ and S. We evaluate the fitted IsoFLOP surfaces in Figure by (a) re-running the fitting procedure k = 100 times on randomly subsampled data and (b) evaluating the Pearson correlation between the true and predicted pretraining loss values on a set of held-out data points.
IsoFLOP surface. Through a grid search over the polynomial coefficients a1 022 at E (0, 1,
Hyperparameters. We followed established best practices to train MoEs that included carefully searching over important hyperparameters like learning rate, weight decay, warm up schedule. Furthermore, we used a load balancing loss, router-Z loss stabilize training and QK-normalization stabilize training. We fix subset of hyperparameters for which changing values in preliminary experiments (a) did not significantly improve pre-training loss, (b) the optimal value remained the same across several model configurations, or (c) in order to reduce the search space (i.e., limited compute resources). Specifically, we first opted to use z-router loss (Zoph et al., 2022) and gl-normalization (Wortsman et al., 2023) in order to stabilize training for large MoEs. Second, we fixed MoE router jitter noise to 0, as it did not improve performance. We also fixed our batch size to 1024 for all model sizes.
We swept over hyperparameters that, when adjusted, (a) significantly improved pre-training loss and (b) the optimal values varied across different model configurations. We increase the MoE sparsity by decreasing the number of active experts and/or increasing the number of total experts. We also varied the MoE granularity (Ludziejewski et al., 2024), MoE load balancing regularizer, Adam learning rate, and linear warm-up steps (fraction) in order to improve pre-training loss. The table below summarizes our hyperparameter sweeps:
HGitHub repository: htcps:://tth:hhbb http:/hhhw...ccoooooee///rreette
Data
$ "Scale-free Adam: https:: htt///aaaaaaapppppae soan--spggthu....
ing/
15

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Exper Experts Language Models
Table Hyperparameter configurations and search spaces
Hyperparameter
Configuration
Search Space
Sparsity Level
Tuned
{0, 25, 50, 75, 90, 95, 98 ) %
Number of Total Experts
Tuned
Adjusted depending on sparsity Adjusted depending on sparsity
Number of Active Experts
Tuned
Granularity
Tuned
[1.2)
Learning Rate
Tuned
[0.003, 0.002, 0.001]
Load Balancing Factor
Tuned
(0.02, 0.05)
Warm-up Steps
Tuned
{2, 5, 103%
Batch Size
Constant
1024
Jitter Noise
Constant
z-Loss
Constant
z-Router Loss
Constant
0.001
QK Norm
Constant
Applied
is also noteworthy that, in this paper, we have prioritized training compute-optimal models, in contrast to many published results on large language models (LLMs), which often rely on over-trained models. As a result, the performance of the models we use for the analysis in this paper is not directly comparable to those of other studies, where they overtrain smaller language models, to reduce the cost of inference relative to training.
16

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for ceatuure-ofExxpertt Language Models
C. Estimating Mixture-of-Expers (MoE) FLOPs
Similar to prior work on scaling laws (e.g., Kaplan et al. (2020); Hoffmann et al. (2022); Ludziejewski et al. (2024)), we use theoretical FLOP estimates as proxies for training and inference costs of language models. In this section, we (a) outline our methodology for estimating FLOPs for MoEs and (b) show that the proposed estimator closely approximates empirical FLOPs of large-scale MoEs.
Setup and notation. Consider an MoE model with Rayyers MoE layers, each with an embedding dimension of dmodel- We denote the number of total experts and active experts in each MoE layer by Extal and Excive respectively. Follow- ing Ludziejewski et al. (2024), we let G denote the MoE granularity, which defaults to and controls the size of each expert relative to the size of (eed-forward layer in an equivalent dense transformer. In order to change sparsity in a more granular manner, we treat the number of active experts as an independent variable that does not scale with granularity G. In our experiments, we use a vocabulary size noyocab 50, 432, context length Rexx of 2048, and GLU modules (Gated Linear Units) (Shazeer et al., 2017) over feed-forward modules as the architecture of choice for MoE experts. We also set the (a) hidden dimension of each GLU expert dmm to 4 dmmodll and (b) instantiate MoEs where the number of attention heads Thaads times the dimensionality for each head dnend equals dmoodet i.c., Theads Chen
dmodel-
Estimating module-specific FLOPs. To estimate the FLOPs of a given MoE model, we first individually estimate the FLOPs per token incurred by a forward and backward pass through every module in MoEs. Then, we aggregate these estimates to obtain the final estimator for the FLOPs per token incurred by a forward and backward pass through the model.
Like in prior work on scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), we take a two-step approach estimate module-specific FLOPs. Given module, first estimate the number of parameters in the module and then scale this with appropriate constant corresponding to the number of add-multiply operations per parameter through forward and backward pass of the given module. We also omit non-leading terms such as non-linearities biases, and layer normalization in our estimation. We estimate the FLOPs per token for attention modules, MoE routers, MoE experts, and the final un-embedding layer as follows:
1. Attention module. We estimate the FLOPs incurred via the QKV (and final) projections, attention logits, and attention values of all heads in a multi-head attention module as follows.
QKV (and fina)) projections. These projections involve 4 Oamodel7Phaad Cheads mone parameters. Following Ka- plan et al. (2020), we use the multiplicative constant C = 6 to account for the add-multiply operations per parameter in a forward and backward pass through linear modules, resulting in a PL.OP--pe-tokke estimate of
4-C dindes-
Attention logits. The FLOPs required to compute the attention logits for all nax tokens equals C n dmoseel FLOPs, making the FLOP-per-token estimate equal to C naxdmmodll-
Attention values. The computation of attention values requires a per-token weighted sum over naxx dmudel- dimensional vectors, making the estimate C' nexxdmodel
2. MoE module. Given an MoE layer, we estimate the FLOPS incurred by its router and all experts separately.
The MoE routing linearly maps a dimute-dimennionall token embedding to a diwa--iienioon logit vector, which is subsequently used to map the token to Enetive active experts. Following Ludziejewski et al. (2024), we use a multiplicative constant R 14 that accounts for the udd-multiply-roon route operations per router parameter. The resulting FLOP estimate equals R dmustel Frotal
Router:
Experts. Each MoE experts corresponds to a GLU module (Shazeer et al., 2017) with dm = 4- dmoctl- Since there are Factive active experts with granularity G, each involving three linear projections, this results a FL.OP estimate of 1/c 3- Eastive C demoteedan 120//- Eaciiv dmodel-
3. Un-embedding layer. The un-embedding linear layer maps the final d embedding of a token to Dl -iman--dimesioonall logits, making the FL.OPs-per-oken C Tevecaddmmseel
Estimating MoE FLOPs. We can aggregate the module-level FLOP estimates described above to estimate the FLOPs per token required for a single forward and backward pass through a given MoE model as follows:
Mayer ( (4Cd 20dmmdel + 12C /c F Frecive Prndel + Rdlmwset Fronat) + CTooocath
17

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Experts Language Models
When / is small, which is typically the in practice, the FLOPs induced by MoE routing can be ignored as they contribute negligibly the estimator. This allows simplify the estimator to:
2netx
12
7lyocab
MoE FLOPs per token == C' -layerr
4
+
+
dynodel
G
dmodel Vayyrss
Evaluating 6N, D as a FLoes-per-token estimator in MoE Models For standard dense transformers, the FLOPs are often estimated 6ND (Kaplan et al., 2020; Hoffmann et al., 2022). Given that /) is fixed and not adjusted dynamically, N can serve as a relative estimator of FLOPs per token for dense transformer models.
To adapt the 6NN D estimator for MoE models, replace N with Na (the active number of parameters) the number of parameters used in every forward and backward pass. In Figure 7, we evaluate the accuracy of the 6NN estimator by plotting the ratio between the MoE FLOPs estimator described in Equation 9 and 6.N, as function of model size N and a fixed context length D = 2048. The results show that, across all sparsity levels, the ratio remains close to one, and the gap between the two estimators decreases as model size N increases.
Sparsity Level
$=0.0
1.14
5=0.25
5=0.5
1.12
5=0.75
MoE FLOPs/ 6N D
$=0.9
S=0.95
1.10
1.08
1.06
1.04
1.02
100M
1B
10B
100B
1T
Total Number of Parameters (N)
Figure 7. Accuracy of 6N, FLOPs Estimator for MoEs. Ratio the MoE FLOPs estimator (Equation 9) to the 6N, estimator as function of the total number of parameters, for fixed context length of D 2048, used in our experiments.
18

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for ceaturre-ff-xpertt Language Models
D. Additional Analysis
D.I. Interplay between parameters and FLOPs per example
Recall that in Section 2, we showed that isoFLOP curves were predictive of pretraining loss for different parameter counts and sparsity levels. In this section, we show similar results with additional training compute budgets.
1. In Figure 8, first show that IsoFLOP surfaces mapping model size N and sparsity level S pre-training loss I. are predictive in a similar way for all training compute budgets that we consider, ranging from 3e19 to le21 FLOPs.
2. In Figure 9, we analyze the fitted IsoFLOP surfaces (one for each training budget) and find that the (a) effect of model size N on optimal MoE sparsity S and (b) the effect of MoE sparsity S on the optimal total and active parameters, N* and No, is similar for all training budgets.
D.2. Effect of training budget and model size on optimal MoE sparsity
Recall that in Section 3, we demonstrated how the relationship between optimal total parameters Nv optimal active parameters N*a, and optimal pretraining loss predictably changes as a function of sparsity S and training budget C. In this section,
the fitted isoFLOP surfaces analyze how the optimal MoE sparsity S* changes function of total parameters N and training budget C, shown in Figure 4. Our main findings are:
Across all training budgets (ranging from 3e19 to le21 FLOPs), increasing the total parameters N leads to an increase in the optimal sparsity level S*
For a fixed model size (i.e., total parameters N), increasing the training budget C generally reduces the optimal sparsity level S*
The relationship between model size N and optimal S* is not linear. For smaller models (up to about 500- 10th parameters), the optimal sparsity remains at 0 (i.e., dense) for most compute budgets.
D.3. Effect of sparsity on downstream task performance
In Section 4, we analyzed the relationship between upstream pre-training loss and downstream task performance across different MoE sparsity levels. We found that language understanding and world knowledge tasks generally showed a strong correlation between upstream and downstream performance, while reading comprehension tasks seemed favor denser models to some extent.
In this section, we provide additional plots for a broader range of tasks within each category to further support our findings. We consider the following tasks:
Common Sense Reasoning: PIQA, CommonSenseQA, OpenBookQA, COPA
Language Understanding: LAMBADA, HellaSwag, Winograd, Winogrande
Reading Comprehension: SQuAD, CoQA, BoolQ
World Knowledge: TruthfulQA, ARC-Fasy, ARC-Challenge
Figure 10 shows the relationship between upstream pre-training loss and downstream task performance for these additional tasks. Each row corresponds to task category and each subplot represents different task, with points colored according to MoE sparsity S. The z-axis represents the upstream pre-training loss, while the y-axis shows the downstream task performance metric (usually accuracy or error rate). These results supplement our main findings from Section 4:
We observe consistent trends across tasks within each category, with language understanding and world knowledge tasks showing strong correlations between upstream and downstream performance regardless of sparsity.
Reading comprehension tasks continue to show slight advantage for denser models, while common reasoning tasks (which can be considered part of the symbolic problem-solving category) show more varied relationships between upstream and downstream performance.
19

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Expee Experts Language Models
Nooo (Budjec FLOPH)
Ground
(0) os (Budges FLOPS)
Prudkted
5-00%
iverieeeet
Tesa becoppttenll
Tetal
15
26
20
12
Crount Pro-iuming Loea
Grouns Gronsseuhh Protureggoor
Ie/leslls Surfece (Budger: 1620 FLOPO
(1) Gimasttmshh Prad Hed-oal
(2) lo op Surface (Budge 3400 FLDPH)
(3) Grount-mutn Precktted Heebod Data
MBc + 6267)
V22- 0003
Pred and Pro Wo0
opes
25%
Sendly
5-88%
Gronn in Provratiing
[2] Gourddihh Prodided Loas HulkeurData
2.0
Spanig
Groandoohh Pre-zeiiinggoo cco
Figure 8. IsoFLOP surfaces over total parameters N, MoË sparsity S, and pretraining loss L for different compute budgets. The rows correspond IsoFLOP surface fitted using models trained with budget of 3e19, 6e19, 1e20, 3e20, and 1e21. The subplots the left visualize IsoFLOP surfaces mapping total parameters N and sparsity level S to pretraining loss L. The subplots on the right correlate the ground-truth pretraining loss with the estimated pretraining loss on held-out data. Taken together, these results show that isoFLOP surfaces are accurate proxies for understanding how model size and MoE sparsity jointly impact pretraining loss.
20

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Mixture-of-Experrs Language Models
Socel
14 Carpus Oytiaa fow 31M
jo Ca pute Cyltel You Prseetien of
Hy Campus Dytioal Faliv
Scrppi Ooree miL Saaniy's 10p
IoCassi Daina Kareitaee N/
20 Conoal Ciliaal Passees
corcid Ootina Fourneur
Sea Ieveor
Figure 9. Optimal MoE configurations predictably change with training compute budget. Each row corresponds am analysis of how optimal MoE sparsity S", total parameters N and active parameters N change for a given training budget. The subplots on the left show that (a) increasing the training budget increases the model size N (denoted with black dots) with the minimum pretraining loss and (b) for models smaller than a threshold (which increases with training budget), dense models (i.c. 0% sparsity) fare better than sparse MoEs. The subplots in the second and third panel show that (a) increasing MoE sparsity increases the optimal total parameters N and decreases the optimal active parameters NN. In both cases, for a fixed sparsity level, increasing the budget shifts increases the optimal total and active parameters.
D.4. Comparing IsoFLOP Surface Analysis with Independent 2d IsoFL.PPs
Recall that in Section 2, we used IsoFLOP surfaces that predict pre-training loss across varying parameter counts and sparsity levels to understand how optimal sparsity and optimal model size depend on each other.
In this section, we evaluate whether these findings remain consistent when we do not rely on fitted IsoFLOP surfaces. Specifically, similar to Approach II in Hoffmann al. (2022), we directly fit univariate quadratic functions that map model size N to pre-training loss T., independently for each sparsity level and training compute budget. We then assess these univariate fits to determine whether our findings in Section 2 hold.
In Figure 12, each row shows how the optimal total and active parameters change function of MoE sparsity for fixed training budgets. As in our findings from Section 2 (Figure 2), increasing sparsity increases the optimal total parameters while decreasing the optimal active parameters. Moreover, larger compute budgets still result in higher optimal total and active parameters, regardless of the sparsity level.
Furthermore, in Figure 11, we observe that across all training compute budgets, increasing sparsity reduces the optimal pre-training loss. This is consistent with the trends identified in Section (Figure 3), thereby validating our earlier results.
E. Does Chain-of-Thought prompting benefit sparse MoEs more than dense models?
In Section 4, we observed that dense models fare marginally better than sparse MoEs on reading comprehension tasks, potentially due to the higher inference-time compute of a dense model than a perplexity-matched sparse MoF. Then, in Section 6, we hypothesized that alternative strategies to increase inference-time compute may reduce the gap between sparse MoEs and dense models on such tasks. In this section, we test this hypothesis by leveraging a "length-controlled" variant of sew-shot Chain-of Thought (CoT) prompting to indirectly control inference- time compute. We then use this to study the effect of inference-tine compute on downstream task performance of dense and MoE models.
21

---

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Cintturr-oo-xxxperr Language Models
Co Common Sanse Reasoning
PIDA
CommonSense shot)
OpenBook DA (Q-shot)
COPA shot)
0.53
Diersaeim Tax Eioer
Downnteeee TAX Enet
Downomomm Ereeror
0.315
0.00
0.525
0.275
0.250
Bandom ChAnCe 0.50
Random chance 0.80
0.52
Ra Random chancel 0.75
2.30
Random chance 0.50
24
22
3.2
3,
22 24
(Upstr ramm tngrrinningooss
(lostreami Pretrannngg Loss
(Upstresm) Pretraining loss
(Upstream) Pretraining Loss
Sparsity 0% (anss))
Sparsing 25%
Spars 53%
Soarsity: 73%
Sparsity: 90%
Sparsity: 95%
Language Understanding
[a] LAMBADA (0- shot!
(b) HelaSwag (30-shot)
[c) Winegred (0-shot)
(d) Winagrande (0-shot)
2.75
0.2
0:20
Downatuam bask SI10
0.0
5.65
2.42
0.50
0.40
0.7
2.82
2:35
0.55
2.53
0.4
5,45
0.42
Random chance 1.00
Random
Random charce
Bandom chance
2.2
222
22
24
22
3. 32
21
24
26
22
30
21
24
24
20
32
22
24 24
22 32
(Uipstram)) Pretraiiiig Lowa
(Upstr wam] Prefneining Lows
[Upstream] Pretraining Lone
(Upstream) Pretneining
Speniity 0% (Denss)
Sparsity: 25%
Spansity 50%
Spariity 75%
Sparsity: 90%
Sparsiy: 95%
Reading Comprehension
(a) SQUAD shots
(b) COQA (0-shot)
(cl BoolQ (10
Devisrreem Tank Error
0 Taakk Extt
0.40
Dam Talk Entr
3.85
0.50
3.80
6.
0.40
375
370
0.00
Bandom chance: 0.99
3.63
Bandom chance: 0.99
0.35
Bandom chance 0.50
(ipstream) Pretraining
oppttream Pretrsiring Loss
@pptrremm Pretrairing
Sparsity 0% (Densa)
Sparsit
Sp Sparsity 50%
Sparsity: 75%
Sparsity 90%
Sparsity: 95%
World Knowledge
BIG-Bench Wikidts (10-shot)
ARC-Easy (10-shot)
(c) Challenl (10-shot)
5.30
2.55
5.75
Downstream Talk
0.7
Task
0.55
3.72
0.6
0.50
0,70
0.5
Random chance:
Random chance:
Random chance: 0.75
5:35
3.54
0.3
22
2.4
25 28
22
2.4
25
22 2.4 25
(Upstream) Pretraining L
[Upstream]] Pretrairing Loss
[[pstremm) Pretraeing Loss
(Dense)
Sparsity: 25%
Sparsity:
Sparsity:
95%
Figure 10. Downstream task performance vs. upstream pre-training loss. Fach subplot shows the relationship hetween upstream pre-training loss (x-axis) and downstream task performance (y-axis) for a specific task, Similar to our results in Section 4, we find that the MoB sparsity level does not change the relationship between upstream pre-training loss and downstream task performance.
22

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Experts Language Models
Effect of MoE Sparsity Pretraining Loss
Training Compute budget (in FLOPs)
2.7
3c19
Optimal Pretraining lass
2.5
1e20
3e20
1e21
2.3
2.2
62%
56%
95% 95%
McE Sparaity 5
Figure/ 11. Effect of MoE sparsity on pretraining loss across different training compute budgets. As sparsity increases, the validation loss decreases for all compute budgets, with larger budgets (darker lines) achicving lower losses at cach sparsity level. This trend consistent with the findings from Section 3, demonstrating that increasing sparsity reduces the optimal pretraining loss across all compute budgets.
Experiment setup. evaluate Qwenl.5 models (Bai et al., 2023) on the GSM8k dataset (Cobbe et al., 2021) to study the effect of few-shot CoT prompting (Wei et al., 2022b) on downstream task performance. We look at the effect of increasing inference-ting compute (via CoT prompting) on GSM8k performance of dense models with sizes ranging from 0.5B to 14B and BB sparse MoE. We also use 10 fixed examples from the GSM8k train split fhw-shot examples for all runs.
Length-controlle CoT prompting enables control over inference-time compute. Given an instruction-tuned model and problem from the GSM8k dataset, we control the inference-time compute of the model by controlling the number of tokens generated to output the final answer to the given problem. To do so, we observe that providing instructions via system prompts (with few-shot CoT prompting) does not effectively control the number of generated tokens and, as result, inference-time compute. We also observe that the average number of tokens in the few-shot GSM8k answers (provided in-context) strongly influences the number of tokens generated by the model solve the given GSM8K question. Therefore, similar work designing GSMSk variants to analyze language modeling phenomena (Mirzadeh et al., 2024; Li et al., 2024; Zhang et al., 2024), we prompt an instruction-tunee model-LLlmma- experiment-4e rewrite GSM8k answers in approximately & E {5..... 100} words. Then, we use the paraphrased GSM8k examples as few-shot examples to indirectly control the number generated tokens. As shown in Figure 13(a), this approach enables systematic control over the length of paraphrased answers. The strong correlation (p = 0.88) between few-shot and generated answer lengths in Figure 13(b) validates that our approach effectively modulates inference-ting time compute.
(Dubey et al., 2024) in our
23

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Exper Experts Language Models
Compute budgel: 3e Active Paramelers
Compute budgel: 3e118 Total Parameters
38
Scorsiy (sparsormaaser)
Sparsiy (spamserdatter))
0% (N: 550 2MG
95% (N: 238 0M)
Lote (volistion)
0% (N) 520, MM)
90% (N: 1.78)
25% (N: 343, DM4)
95% (N: 189, 4M))
25% (N 4:18
95% 2.78)
(N: 5M))
(N: 101.9M)
50%
90%
5.18)
75% (N: 334, BM)
75% (N. 624 BM)
3.8
100000
276
6.00
1.49
3.24
4.00
Adtiv Parameters
Total Parameters
Compute budgel: 6er 19| Active Paramelers
Compute budgell 6er18 Total Paramelers
Sporsiyy (spaser-ddarke))
MiE Sparsity (camer-darkar)
3.0
25% (N. 468 SM)
0%% (N: 509 2M)
90% (N: 2M)
95% (N: 282 963)
LOOS Ioltooo
(N 50R2M) 25% (Ni 584 7M)
95%
90%
(N: 9.9
5M)
(4: 2M)
50% 779,
90%
2.0
75% (N: 4084488800 GM)
% (N: 1 (.)
100.0V
1.45
275
4.08
10.0
1,40
2.73
4.00
Activ Parmeeters
Total Parmmeeers
Compute budget: 1e+ Active Parameters
Compute budget: 1e+ Total Parameters
3.4
Searsity (sparser-darker)
MaE Sparsity boamerddakar
0% (N: 9MG
90% (N BM)
(N 884 OM) 28% (Nt U36 MM) 50% (NL 1:1B)
90%
25% IN: 656, 6M)
95% (N: 338 4M)
80% (N: 5.08]
95%
2.0
75%
9M)
75% (N: 1.85)
100/07
2:29
XM
2:78
9:48
Activn Paremeters
Total Panemaiers
Compute budget: 2e| Active Parameters
Compute budget: 20+ Total Parameters
(pparsendarienn
McE Sparsity (spamerndation)
Loss (ell3atoo
25% (N: 1.48)
0% 1.38)
95% (N: R08 294)
4M)
Lote (olidatoon
25% (N: 1.88)
IN: 1.38)
98% (N: 8.68)
90% 6.68)
(N:1.3))
453.7M4
50% (N:
(N:
(N. 1,08)
75% (N:
100.00
5
633
KLCM
5.40
5.00
Adtve Parameters
Total Parameters
Compute budget: 1c-21| | Active Parameters
Compute budget: 1c+21 Total Parameters
3.0
McE Sparaily (arkker-
MaE Sporsily (ppenersdarke)) (N:
2.90)
Lo05 MOOPOo)
3.0
25% IN: 2:58)
80% (N: 08
25% (N: 3.28)
90% (N: 12.1B)
50% 248)
95% (N: 1.38
80% 4.48)
95% (N: 21
19
Parameters
Total Parameters
Figure 12. Effect of MoE sparsity optimal total and active parameters across different training compute budgets. Each row shows the change in total and active parameters as a function of sparsity level for fixed training budgets. Increasing sparsity lcads to an increase in the optimal total parameters while reducing the optimal active parameters, consistent with our findings in Section 2 (Figure 2). Larger training compute budgets result in higher optimal (total and active) parameters across all sparsity levels.
24

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Exper Experts Language Models
GSMBK variants with paraphrased answers
Few-shot answer length Generated answer longth Inference-time compuna
Number words specitted In parsahraaing prompt
Longth or anawers gererataa by the modol
50
103
50
0.02
Correlation
0.02
20
50
100
120
165
30
40
50
Number of words paraphrased answers
Length of paraphrased answers used row-shot CoT prompt
Figure 13. Varying inference-time compute via length-controlled Chain-of-Thought Thought prompting, We can control inference- time compute (number generated tokens) via Chain-of Though prompting in two steps: (a) generating paraphrased GSM8K answers varying lengths (5-100 words) and (b) using paraphrased answers as few-shot examples in CoT prompts to influence the length of generated answers determine the model's output length (p 0.88 correlation). The systematic shift answer length distributions in subplot (a) and the linear relationship between few-shot answer length and generated answer lenth (p 0.88 correlation) in (b) validate that our prompting approach effectively modulates inference-time compute, enabling controlled studies its impact on model performance.
Effect of length-controlled CoT downstream task performance. We investigate how inference-time compute affects GSM8k performance across Qwen1.5 models using 10-shot length-controlled CoT prompting, varying target answer lengths k from 1 to roughly 70 words on average. As shown in Figure 14, when we increase inference-time compute indirectly through longer generated answers (z-axis), we observe a roughly linear improvement in GSM8k performance (y-axis) across all model sizes. The linear fits for each model also indicate a pattern through their slopes m: larger dense models benefit more from additional inference-ting tintennce-timeeeeee compute. This effect is particularly striking when comparing models at the extremes (i.e., Qwen 5-0.5B versus Qwenl..5-1BB)
To analyze whether inference-time compute affects dense and sparse MoE models differently, we examine how the MoE
"performance" slope (i.e., accuracy gain per generated word) varies with model size. While the cannot be directly compared to dense models due to different active parameter counts, we can account for this by plotting m against the number of active parameters for both architectures. As shown in Figure 15, when controlling for active parameter count, the MoE model exhibits a higher slope than would be expected from interpolating between dense models. This suggests that MoE models benefit more from dynamically increased inference time compute compared to dense models with equivalent active parameters.
25

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of- Expeetts Language Models
Effect of 10-shot lengthcco CoT on GSM8k performance
Model and slope
Qwen 0.5B (dense), 0.001
0.6
Qwen 1.88 (dense), m 0.0044
Qwen 14B (dense), 0.010
0.5
Accuracy
Qwen 4B (dense), m = 0.006
Qwen 5x2.7B (MoE), 0.008
Qwen 7B (dense), m 0.009
0.3
0.2
0.2
Number of words generated (proxy for inference-time compute)
Figure 14. Effect of length-controlled CoT prompting on GSM8k performance across model scales. We evaluate the relationship between inference-time compute (controlled via answer length) and GSM8k accuracy for dense Qwen1.5 models (0.5B-14B parameters) and 5x2.7B sparse MoE. For all models, increased inference-time compute improves accuracy roughly linearly, with slopes m indicating marginal effect. Larger dense models show steeper slopes, demonstrating they benefit more from additional inference-time compute-for example, the 14B model's ' performance improves from 20% to 70% as answer length increases, while the 0.5B model improves only from 1% to 5%.
Marginal effect of sierence-time compute model size
(accuracy gain per generated aead word)
0.010
Dense Qwenl.5 models
Qwen1.5-5x2.11
0.008
0.006
Performanch
0.002
12
14
Number of active parameters (B)
Figure Sparse MoEs benefit more from increased inference-time compute than dense models. We plot the marginal effect inference-time compute (accuracy gain per generated word) against model size for dense Qwenl.5 models and 5x2.7B sparse Qwenl.5 MoE. When controlling for number of active parameters, the MoE model (orange) shows a higher performance slope than would be expected from interpolating between dense models (bluc), suggesting that sparse MoEs benefit more from dynamically increased inference-time compute via strategies like CoT prompting.
26

---

Parameters FLOPs: Scaling Laws for Optimal Sparsity for cexture-of-Experry Language Models
F. Incorporating Sparsity into Scaling Laws
Table 2 shows the parameters used to initialize L-BFGS used to the proposed parametric scaling law given in Equation Table 3 shows the estimated parameters for the parameteric model. We use a held out dataset that consists of data points for models with sparsity value S 0.98 validate the performance of the estimated model coefficients. The mean squared error and the Huber loss error the dataset used fit the model is 0.00056 and 0.0036 respectively and 0.0058 and 0.0011 respectively on the out-of-sample validation set. The quality of the fit measured via the Rt metric is 99% on fitting data and 68% on the held out validation dataset.
Table 2. Initial values used estimate coefficients in Equation 6.
Coefficients
Initial Values
log(a). log(b), log(c), log(d)
0., 20]
at, 99, 7
[0, 0.,5..5, 0.5, 0.75, 1.25) [-1, 0,5, 0.5, 1] 1.5
X, log(e)
Table 3. Estimated values for coefficients in Equation 6.
Estimate Coefficient
0.5962
B
0.3954
10.1666
0.1603
0.1595
16612.50
5455.67
0.4598
17.26
0.94
27


Reviews:
[{"review_id": "re1vnVn42o", "reviewer": "Reviewer_r3sS", "summary": "The paper provides empirical scaling laws for MoE-based LLMs. The experimental setup is simply training MoE-LLMs on the RedPajama dataset and then evaluating by comparing the eval loss. With this setup the authors find:\n\n1. For every sparsity level and FLOPs budget, there seems to be a unique optimal model size (fig. 1 and 2)\n2. The optimal model sparsity increases as a function of total model size (fig. 4)\n3. Downstream performance scale reliably w.r.t. validation loss irrespective of the model sparsity (fig. 5)\n\n## update after rebuttal\n\nI will keep the current score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Weaknesses (W):\n\n> The novelty is lacking. There have already been many scaling papers for MoEs and it's not clear what is new here. The paper shows that for a given FLOPs budget and sparsity there is an optimal model size. This has been known for dense models for a long time, and one would expect it to hold for MoE models too.\n\nW1: We thank the reviewer for raising this point about prior work on dense models that relates optimal model size for a given FLOPs budget. Our intention with this work is to shed  light on how model size and sparsity jointly interact for compute-optimal models. While the reviewer correctly points out that one could hypothesize that MoEs should scale in an analogous way to dense models based on prior studies by setting sparsity to a fixed value, it is unclear how one might set this sparsity value optimally. We further study the practical case of fixed model size, which can be particularly important in on-device deployment scenarios, and suggest a parametric form for scaling laws that the reviewer highlighted as a strength of the paper. \nIn general, our study suggest that when designing a new architecture, or hardware, we should keep in mind that allowing the model to have more parameters vs more FLOPs per example would be more efficient as long as the goal is to do well on the pre-training task and DS tasks that are well correlated with the pre-training performance of the model.\n\n> The evaluation of the fits is lacking. The authors evaluate that with MSE, it's better to use a scale-invariant metric like R^2.\n\nW2: The R^2 values are 99% on fitting data  and 68% on the held out extrapolation dataset (sparsity = 98%). We thank the reviewer for raising this point. We will update our paper with the above results.\n\n> It is not clear if there are any new lessons for practitioners. Everyone knows that MoEs work well and that balancing training duration and model size is needed.\n\nW3: The findings from this study suggest that MoEs don’t always work well unless sparsity values are set carefully. Conceretely, we find that optimal sparsity values grows with model size. Additionally, we find that MoEs transfer performance depends on the nature of the downstream task (knowledge vs reasoning) which is consistent with the observations made by in concurrent work by Jelassi et al.[1]. These are valuable findings to both practitioners & researchers that want to build efficient MoEs.\n\n[1] Jelassi et al. Mixture of Parrots: Experts improve memorization more than reasoning.  October 2024 / ICLR 2025\n\n### Response to Questions (Q)\n\n> Can you provide R^2 metrics for the fits and evaluate them on extrapolation\nQ1: We provide R^2 metrics in our response to W2 above.\n\n> Why do you use “scale-free Adam optimizer”? It is non-standard.\nQ2: We thank the reviewer for carefully reading our paper and appendix. We used AdamW described in Loshchilov and Hutter (https://arxiv.org/abs/1711.05101). We will clarify this detail in the final version.\n\n> Regarding related work you write “However, these studies typically assume a fixed configuration for other critical variables influencing FLOPs per token, such as the number of active experts per input.” — could you give more details on exactly what previous studies cover, and how it differs from what you cover?\nQ3: We highlight the important papers to answer the reviewer’s question\n- Clark et al. assume a fixed size dataset of 130 billion tokens to derive scaling laws. \n- Ludziejewski & Krajewski et al. conduct their experimetns at a much smaller scale compared to our work and focus on varying granularity while we also consider number of active experts per input and using expert-choice routing while we use token-choice routing  \n\nWe have discussed related work more extensively in Appendix A. We will gladly discuss additional works that the reviewer may want us to do so.\n We thank the reviewer for asking this clarification and are happy to provide further information.\n\n> Fig 1 says “These results indicate that for a fixed compute budget, increasing model sparsity leads to a reduction in pretraining loss” — this seems to not be true e.g. in figure 4. Please clarify this.\n\nQ4: Figure 4 studies the case where the model size is a constraint and shows how sparsity value changes for a given model size. If there is no bound on the total number of parameters then optimal sparsity level approaches 1. OTOH Figure 1 shows that for a given compute budget, optimal models with higher sparsity have a larger parameter count and smaller active parameter count. We thank the reviewer for this question and will adjust the captions to make things more clear in our final version.\n\nWe thank the reviewer for carefully reading our paper and the appendix! We will follow this suggestion and promise to update the draft in the final version of our paper. If our rebuttal is satisfactory, we ask the reviewer to consider raising their score.", "claims_and_evidence": "The claims themselves are reasonably well supported, but I'm not sure they are very novel.", "methods_and_evaluation_criteria": "Yes, the methodology to evaluate the LLMs is sound. The scaling laws fits are evaluated by MSE which is not a good metric. It's better to use a scale-invariant metric like R^2.", "theoretical_claims": "Na", "experimental_designs_or_analyses": "Checked the experimental setup, seems reasonable.", "relation_to_broader_scientific_literature": "na", "essential_references_not_discussed": "No references missing AFAIK, but more details needed.", "other_strengths_and_weaknesses": "Pros:\n\n1. The paper is well-written. \n2. Scaling laws are impactful.\n\n\nCons:\n\n1. The novelty is lacking. There have already been many scaling papers for MoEs and it's not clear what is new here. The paper shows that for a given FLOPs budget and sparsity there is an optimal model size. This has been known for dense models for a long time, and one would expect it to hold for MoE models too.\n\n2. The evaluation of the fits is lacking. The authors evaluate that with MSE, it's better to use a scale-invariant metric like R^2. \n\n3. It is not clear if there are any new lessons for practitioners. Everyone knows that MoEs work well and that balancing training duration and model size is needed.", "other_comments_or_suggestions": "1. Please give some more details on the experimental setup in the main paper. Currently it's hidden in the appendix.", "questions_for_authors": "1. Can you provide R^2 metrics for the fits and evaluate them on extrapolation?\n2. Why do you use “scale-free Adam optimizer”? It is non-standard.\n3. Regarding related work you write “However, these studies typically assume a fixed configuration for other critical variables influencing FLOPs per token, such as the number of active experts per input.”  -- could you give more details on exactly what previous studies cover, and how it differs from what you cover?\n4. Fig 1 says “These results indicate that for a fixed compute budget, increasing model sparsity leads to a reduction in pretraining loss” — this seems to not be true e.g. in figure 4. Please clarify this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPGGt4BEvl", "reviewer": "Reviewer_uTJa", "summary": "This paper investigates the relationship between the number of model parameters and the compute per example, measured in Floating Point Operations (FLOPs), in the context of sparse Mixture-of-Experts (MoE) language models. The authors aim to understand how varying the sparsity level—defined as the fraction of inactive experts—affects model performance during pretraining and downstream tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and thorough review and for their support. \n\n> Essential References Not Discussed\n\nThe reference pointed to by the reviewer discusses techniques to sparsify MoEs after training whereas we discuss optimal sparsity during pretraining and its implications on downstream tasks. We will discuss the reference mentioned by the reviewer[1] in the related works section in the updated version of our paper as that can help the reader navigate the vast literature on MoEs. We thank the reviewer for bringing this reference to our attention.\n\n### Response to Questions (Q)\n\n> Apart from the training loss, any scaling laws about some benchmark performance?\n\nQ1: We thank the reviewer for this question. Our experiments suggest that transferring performance from pretraining to few-shot downstream (DS) tasks depends on the nature of the task. We find this observation intriguing and believe that there is more work to be done to uncover the reasons behind this behavior, offering one hypothesis that tasks that can benefit strongly from more inference-time compute can be hampered by increasing sparsity naively. \nMore generally, our trained models may not be suited for additional downstream evaluations since we do not post-train (RLHF, instruction finetuning) our models. We agree with the reviewer that the question of how sparsity affects DS evaluations but this work work is out of the scope of this paper but is an excellent topic for future work.\n\n\n> Is there any difference between training moe from scratch of continue pretrain from smaller dense model?\n\nWe chose the most typical setup to study sparsity behavior in MoEs that uses multiple experts in the feed-forward network (FFN) layer.  Understanding the behaviors of pretraining starting from a smaller dense checkpoint, for e.g. MoEfication [1, 2] while interesting is outside the scope of this work. We can discuss this as future work though.\n\n[1] Shang et al. MoEfication: Transformer Feed-forward Layers are Mixtures of Experts 2022\n[2] Szatkowski et al. Exploiting Activation Sparsity with Dense to Dynamic-k  Mixture-of-Experts Conversion 2024\n\nWe once again thank the reviewer for their review and support. We look forward to engage further with the reviewer if there are any additional questions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No proofs in this paper", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper further explores the scaling of MOE structured models.", "essential_references_not_discussed": "One possible discussion about the sparsity in the expert level in ACL2024.\n[1]Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models", "other_strengths_and_weaknesses": "1. The paper is well written, I really enjoy reading the paper. This paper is interesting and important. The findings of this paper is important for future MOE LLM training and architecture designs.\n2. The experiments are comprehensive and inspiring.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Apart from the training loss, any scaling laws about some benchmark performance?\n2. Is there any difference between training moe from scratch of continue pretrain from smaller dense model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eOT10bGdN", "reviewer": "Reviewer_6Xtz", "summary": "This paper explores parameter-FLOP trade-offs in sparse MoE LLMs. The author finds that:\n1. Increasing sparsity during pretraining improves efficiency and performance under a fixed compute budget.\n2. More parameters benefit pretraining, while FLOPs are crucial for inference, especially for reasoning tasks.\n3. Optimal sparsity increases with size and compute, approaching full sparsity for large models.\n4. Downstream performance correlates with pretraining loss, but denser models excel in tasks like reading comprehension.\n5. The authors design a new scaling law incorporating MoE sparsity, guiding efficient design.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and thorough review and for their support. We thank the reviewer for pointing out the comprehensive nature of our empirical work and presentation and respond to the question(s) raised by them below:\n\n### Response to Weaknesses\n\n> Sparsity improves deployment efficiency in production. It would make the paper stronger to discuss how the sparsity results can lead to real-world benefits beyond FLOPs where memory and communication costs matter. Also, it would make the paper stronger if connections could be made to other methods introducing sparsity to LLM, such as activation sparsity.\n\nWe thank the reviewer for bringing up activation sparsity. This is an interesting question as activation sparsity in language modeling is an active area of research [1, 2, 3]. We chose to focus on the most typical setup for inducing sparsity in MoEs, focusing mostly on model size and compute cost, and deferred studying other forms of sparsity that may have their own tradeoffs to future work. We once again thank the reviewer for bringing this interesting question to our attention.\n\n[1] Mirzadeh et al. Relu strikes back: Exploiting activation sparsity in large language models 2023\n[2] Szatkowski et al. Exploiting Activation Sparsity with Dense to Dynamic-k  Mixture-of-Experts Conversion 2024\n[3] Liu et al. TRAINING-FREE ACTIVATION SPARSITY IN LARGE LANGUAGE MODELS 2025\n\n### Other comments \n\n> Line 82: Sentence is unfinished.\n> Line 217: Duplicated \"and\".\n\nWe thank the reviewer for spotting these typos in our draft. We will fix these errors and carefully proofread to ensure we catch and fix other errors in the final version of our paper. We once again thank the reviewer for their review and support. We look forward to engage further with the reviewer if there are any additional questions.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing empirical results.", "methods_and_evaluation_criteria": "Yes, the evaluation setup and results are standard for developing new scaling laws.", "theoretical_claims": "The paper is mainly empirical. The derivation of scaling laws was presented clearly.", "experimental_designs_or_analyses": "Yes, I checked the experiment designs.", "relation_to_broader_scientific_literature": "This work extends existing scaling laws to MoE settings.", "essential_references_not_discussed": "It would make the paper stronger if the authors could connect to ideas to improve deployment efficiency beyond FLOPs.", "other_strengths_and_weaknesses": "Strengths:\n1. The authors provide a comprehensive empirical analysis spanning multiple compute budgets and tasks, which provides new insights for MoE design.\n2. The empirical analysis offers clear visualizations that effectively convey the trade-offs between total parameters, active parameters, and compute.\n\nWeaknesses:\n1. Sparsity improves deployment efficiency in production. It would make the paper stronger to discuss how the sparsity results can lead to real-world benefits beyond FLOPs where memory and communication costs matter. Also, it would make the paper stronger if connections could be made to other methods introducing sparsity to LLM, such as activation sparsity.", "other_comments_or_suggestions": "Line 82: Sentence is unfinished.\n\nLine 217: Duplicated \"and\".", "questions_for_authors": "See the above section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2NoNaiJGb", "reviewer": "Reviewer_TF5J", "summary": "The paper investigates the optimal trade-off between the total number of parameters and FLOPs per example in Mixture-of-Experts (MoE) models under a fixed pretraining FLOPs budget. Experimental results show that increasing total model parameters (i.e., increasing sparsity and reducing active parameters per input) leads to lower pretraining loss for a given compute budget. The paper also studies the relevant impact on the downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time reviewing our paper. We find it encouraging that the reviewer finds our work valuable and relevant as we systematically study optimal model size and FLOPS-per-token in MoEs. We address the concerns below. Please note that we partially quote or paraphrase reviewer's comments due to space constraints:\n\n> One limitation is that the experiments focus primarily on theoretical FLOP estimates, without a thorough quantitative analysis of memory and communication overheads ...\n\nWe have discussed the limitations of using theoretical FLOPs estimates in Section 6.1. We agree that it would be very valuable to benchmark different hardware profiles and leave that as future work.\n\n> The paper focuses on a specific MoE implementation, which may limit the generalizability of the findings ...\n\nWe chose the most typical setup for MoEs in our work so we opted to use token-based routing. Examining this choice in detail would indeed be very valuable which we leave as future work \n\n> The stated research goal is to investigate the optimal trade-off between total parameter count and FLOPs per sample, but the scaling law primarily focuses on total parameters and sparsity ...\n\nThis is a fair point, and deserves an explanation. While our goal is to understand the relationship between model size and FLOPS-per-example, we study this using a surrogate control knob: increasing the sparsity in MoEs decreases the number of active parameters, which in turn decreases the FLOPs-per-example under settings where compute can take advantage of sparsity. We will make this clear in the paper. \n\n> The paper's findings suggest a monotonic relationship where increasing sparsity, increasing total parameters, and decreasing active parameters all lead to better performance. This raises questions about whether this represents a true \"trade-off\" as described in the paper's title ...\n\nThis is an excellent point raised by the reviewer. Our study finds that as we scale MoE model size, we need to scale up the sparsity value as well. This finding does suggest a tradeoff: we can increase model size but need to reduce the number of active parameters (via sparsity) to gain benefit.  We hope that our work encourages practitioners to invest in building infrastructure for training very large scale MoEs and provides some guidelines on what sparsity value and model sizes to use for a given compute budget. However, given the scale used in our study, we acknowledge that there maybe limitations that will be uncovered when we run models at very large scale, i.e., extrapolating to model sizes significantly larger than what we have used in our study, for which we have not been able to verify how scaling behaves due to compute limitations.\n \n\n> The paper does not provide sufficient discussions on relationship between pre-training loss and downstream performance in both the main paper and the appendix. Figure 10 suggests this relationship varies significantly across tasks and sparsity levels. A more thorough analysis ...\n\nThis is another excellent point raised by the reviewer. We were able to conduct some downstream analyses the results of which suggest some task types transfer better than others which is an intriguing finding in our paper, and offered an initial hypothesis that increased test-time compute demands of some tasks may conflict with increasing sparsity (which otherwise improves pre-training loss). A thorough investigation as well as studying test-time interventions would be very interesting but is left as future work.\n\n\n> MoE models are known to present unique training challenges ...\n\nThis is a good point raised by the reviewer. We will add to the discussion in our paper that special care needs to be placed on MoE training to ensure that sub-optimal results are not due to poor optimization. We followed established best practices to train MoE that included carefully searching over important hyperparameters like learning rate, weight decay, warm up schedule. Furthermore, we used a load balancing loss, router-Z loss to stabilize training and QK-normalization to stabilize training. All of these details are noted in the appendix of our paper. We acknowledge that training a model at scale (size and sparsity) larger than what is shown in the paper is an involved task.\n\n\n> Presentation issues\n\nWe will fix these in the next revision of our paper.\n\n> Line 137 (left): ... To clarify, does the model always use a fixed K for all tokens?\nYes We use dropless token-based routing\n \n> I emphasize that the paper has the potential to be a very strong paper given the systematic empirical study. I am looking forward to an improved version of the paper.\n\nWe appreciate the reviewer’s comment. We hope our rebuttal above addresses reviewer concerns, and promise to update the paper to reflect the reviewer’s thoughtful feedback. If this is satisfactory, we ask the reviewer to consider raising their score.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No proofs in this paper.", "experimental_designs_or_analyses": "Experimental designs are sound, but there are a few limitations: \n\n- One limitation is that the experiments focus primarily on theoretical FLOP estimates, without a thorough quantitative analysis of memory and communication overheads. Those factors are critical for MoE models with large numbers of parameters.\n\n- The paper focuses on a specific MoE implementation, which may limit the generalizability of the findings. Comparing different routing mechanisms or expert configurations would provide valuable insights into whether the observed relationships between sparsity, parameters, and performance hold across different MoE variants. That being said, I'm fine with the current paper using one specific MoE implementation as an initial study.", "relation_to_broader_scientific_literature": "This paper aligns well with important research topics in both scaling law research and MoE model development, targeting an important open question about optimal sparsity in large-scale language models. The contributions are still valuable for understanding the scaling of MoE models.", "essential_references_not_discussed": "I'm not aware of such related works.", "other_strengths_and_weaknesses": "## Strengths\n- Timely empirical study on an important research topic.\n- Systematic experiments on a wide range of settings.\n- I particularly like the analysis of optimal sparsity for a given model size. In my opinion, it represents one of the most practically valuable contributions, since it takes real-world constraints like memory and communication requirements into account. The paper would be strengthened by expanding this section and highlighting these real-world implementation considerations more prominently.\n\n## Weaknesses\nMy overall impression is that while the paper presents extensive experimental results, these findings could have benefited from more structured presentation with clearer takeaway messages. To elaborate,\n\n- The stated research goal is to investigate the optimal trade-off between total parameter count and FLOPs per sample, but the scaling law primarily focuses on total parameters and sparsity. Although sparsity and FLOPs per sample are relevant, this **inconsistency** can make the reader confusing.\n- The paper's findings suggest a monotonic relationship where increasing sparsity, increasing total parameters, and decreasing active parameters all lead to better performance. This raises questions about **whether this represents a true \"trade-off\"** as described in the paper's title. The practical implications of this finding could be more thoroughly discussed - is the recommendation to use increasingly large, extremely sparse models? What are the practical limits to this approach?\n\n- The paper does not provide sufficient discussions on relationship between pre-training loss and downstream performance in both the main paper and the appendix. Figure 10 suggests this relationship varies significantly across tasks and sparsity levels. A more thorough analysis of when and why sparse models transfer differently to downstream tasks compared to dense models would enhance the paper's contributions.\n- MoE models are known to present unique training challenges compared to dense models. The paper may benefit from including analysis of how the optimal sparsity levels might be influenced by considerations of training robustness and convergence.\n\nI emphasize that the paper has the potential to be a very strong paper given the systematic empirical study. I am looking forward to an improved version of the paper.", "other_comments_or_suggestions": "## Presentation issues:\n- Line 45 (right): missing a space between the reference and \"of\".\n- Figure 1b: I recommend to use $N_a$ to denote the number of active parameters in this figure so that it looks consistent to others.\n- Line 81 (right) perform worse o. $\\to$ perform worse.\n- Line 325 (left): demonstrates  $\\to$ demonstrate\n- Line 403 (left): supports $\\to$ support", "questions_for_authors": "- Line 137 (left): _\"$K$ is the number of selected experts per token\"_: To clarify, does the model always use a fixed $K$ for all tokens?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studied the trade-off between sparsity and performance in MoE models under different constraints. The author found that given A fix parameter size or total training compute, there is an optimal level of sparsity that improves both training efficiency and model performance. 

Reviewers generally agree that the research topic is important, and this paper is well-written and the experiment setting is comprehensive.
Decision: Accept (poster)