Scaling Laws for Upcycling Mixture-of-Experts Language Models
Authors: Seng Pei Liew
Takuya Kato
Sho Takase
Venue: ICML 2025 poster (2025)
Keywords: language modeling
mixture of experts
scaling law
upcycling
Primary area: deep_learning->large_language_models
TL;DR: Exploring scaling laws for upcycling dense language models to MoE, revealing key trade-offs and guidelines for efficient training.

Abstract: Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters.
 There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling), and training computationally efficient models like mixture-of-experts (MoE).
 In this paper, we study the upcycling of LLMs to MoE models, of which the scaling behavior remains underexplored.
 Through extensive experiments, we identify empirical scaling laws that describe how performance depends on dataset size and model configuration.
 Particularly, we show that, while scaling these factors improves performance, there is a novel interaction term between the dense and upcycled training dataset that limits the efficiency of upcycling at large computational budgets.  
 Based on these findings, we provide guidance to scale upcycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.

Scaling Laws for Upcycling Mixture-of-Experts Language Models
Seng Pei Liew!
Takuya Kato
Sho Takase'
Abstract
Dai al., 2024; Yang et al., 2024).
Pretraining large language models (LLMs) assource-intensve, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such com- putational demands: reusing smaller models to train larger ones (upcycling), and training compu- tationally efficient models like mixture-of seteeeoff-eppttt-se (MoE). In this paper, we study the upcycling of LLMs to MoE models, of which the scaling be- havior remains underexplored. Through extensive experiments, identify empirical scaling laws that describe how performance depends dataset size and model configuration. Particularly, we show that, while scaling these factors improves performance, there is novel interaction term be- tween the dense and upcycled training dataset that limits the efficiency of upcycling at large com- putational budgets. Based on these findings, we provide guidance scale upeycling, and establish conditions under which upcycling outperforms from-scratch trainings within budget constraints.
One direction to further accelerate training convergence is leveraging smaller pretrained models to guide the training of larger Mož models. Komatsuzaki et al. proposed upcycling, which reuses the dense checkpoint to continued pretrain the upcycled MoE. The MoE expected to specialize and optimize routing more rapidly by leveraging the pretrained dense model weights.
Despite the promise of efficient MoE training via upcycling, the effectiveness and limitations of this techmique remain unclear. While some (Wei et al., 2024; He et al., 2024) have already adopted to training large-scale MoE models, Muennighoff et al. (2024) reported negative results where upcycling can slow down training convergence. We believe these seemingly contradictory conclusions are due to insuffi- cient comprehensive studies and assessments. There is also lack of guidance on how and when to upcycle, hampering wider adoption of this technique.
In this paper, we seek to better understand large language models' (LLLMs) upeyeling to MoE models via a series of controlled experiments, spanning up to few hundred billion (B) training tokens and models up to 7B total parameters. Specifically, we uncover precise power-law scalings for the language modeling performance (cross-entrops loss) with respect to training dataset size (for both dense and upcy- cled MoE training), and the model configuration, including the total number of parameters (model size). Building on these results, we provide a framework for assessing when upcyeling offers advantages over from-scratch training and how performance gains depend dataset size and model configuration.
1. Introduction
Large-scale neural network architectures, such as dense transformers (Vaswani et al., 2017), have seen remarkable success across a wide range of tasks, particularly achiev- ing human-level capabilities in natural language process- ing (Achiam al., 2023). However, they often demand an enormous amount of computation, imposing challenges of computational efficiency and scalability. Sparse mod- (MoE) architectures (Shazeer et al., 2017; Lepikhin et al.) have emerged as an alternative achieving better peficicnccy-erfommance trade-off via partial activation (routing) of neural parameters when processing the input. Even so, MoE models still require substantial compute power reach its full potential (Wei et al., 2024;
Main results. The major technical findings of this paper are summarized below. Let D1, D2 be the number of tokens used to train the dense model and upcycled MoE respec- tively. Denote the cross-entropy test loss of the upcycled MoE by L. We find that the upcyeled MoE satisfies the following relation for a wide range of model configuration:
els like mixture-of-
'SB Intuitions, Tokyo, Japan. Correspondence to: Seng Pei Liew <xengpei.ieww @sbiinuutiiionssc..p.>....
L = AD, to D tos D: + E
(1)
where a;'s (i = 1, 2,3) are positive scaling exponents, and A, E are constants independent of D1, Dr.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Moreover, the empirical performance has a power-law seal-

---

Scaling Laws for Upeyeling MoE
3.2
From-scratch MoE more efficient
Additional tokens (B)
3.1
32
311
Numerical
3.0
2.0
Fit
2.8
1.7
2.8
Upcycling more efficient
0.
0.7
-24
R. R.acive
-2.7
100
10a
10a
10a
Sparitty
0.0
Dense model size (B)
Figure 1. Left: Upeycling improves with sparsity and the number of active parameters. We find that upcycling to MoE which sparser and has more active parameters improves performance. The z-axis shows the value of cross-entropy loss. See Section 4.2 for details. Right: Efficiency of upeycling diminishes with sunk cost and model size. For all additional token budgets for upeycling Mixtral-like MoE above the curve(s), training MoE from scratch is more efficient, whereas for all token budgets below it, upcycling is more efficient. Shown are the numerical (blue) and analytical (red) solutions of Equation 13. See Section for details.
ing with respect to the model size as well as its sparsity (Equation 11). These allow to quantify the effectiveness and shortcomings of upcycling. Particularly, our empirical results suggest that:
D: dataset size in token
N: non-embedding model size
A, B, F: scaling factors of the power law, independent of the variable under consideration
E: irreducible loss of the power law, independent of
Increasing Dj (sunk cost) reduces the initial losses of the upcycled MoE but results in slower training progress with D2 (upcycled MoE tokens).
tee variable under consideration
a, B, y: scaling exponents the power law, indepen- dent of the variable under consideration
Upcycled MoE benefits from increased sparsity and active parameters without noticeable trade-offs. See
2. Preliminaries
the left panel in Figure
2.1. Model details
We propose a joint scaling law of dataset and model sizes for Mixtral-like MoE (Jiang et al., 2024) (Equa- tion 12). While we find that the advantage of upeyeling diminishes with increasing sunk cost and dense model size (Nı), upcycling remains effective when D2 are limited.
Dense model. Our dense models decoder-only trans- formers pretrained with autoregressive language model- ing objective. The architecture is most similar to Llama2 models (Touvron et al., 2023), incorporating advances such as SwiGLU (Shazeer, 2020) and rotary position embedding (Su et al., 2024). We use the Llama tokenizer of vocabulary size 32,000.
Particularly, upcycling is beneficial when D2 remains below a certain threshold, where the pretrained dense model can still accelerate convergence. However, be- yond this threshold, corm-scratch training of MoE be- comes more efficient. The threshold D is
Mixture-of- The MoE in consideration is the same as our dense model, but with all MLP blocks replaced by multiple blocks (experts) with the same configuration (Fedus et al., 2022). A router consisting of a single-layer MLP outputs the routing probability of the tokens the experts.
0.7+0.04 lo7
Ni 109
D" 2 4
tokens
(2)
The model configuration has two key parameters: Rexperts representing the number of experts, and RrTpk. which spec- ifies how many of the is routed to at each layer. The output of the experts is linearly combined and passed to the next layer.
experts each token
See the right panel of Figure 1.
Notations. We summarize main notations used in the paper.
The MoE and corresponding dense model with model size Ndenses consisting of nexpert experts, denoted with
cross-entropy loss in
2

---

Scaling Laws for Upeycling MoE
prefix "Mexpert" ", e.g., 8x where the dense correspon- dent is of 1B in model size. We refer the number of non-embedding model parameters (that is, total number of parameters minus the number of embedding and language modeling head parameters) used for computation per token as the number of active parameters.
3.1. Setup
Dense models in consideration. We train a suite of dense models with model sizes 15 million (M), 44M, O.1B, 0.2B, 0.5B and IB. The model configuration (number of layers, Muyer. hidden dimension dmodel. and MLP hidden dimen- sion, dmip) used in this paper is summarized in Table 3 of Appendix A.
Upeyeling. The upcyeling scenario assumes that one is given a pretrained dense model and would like to train an MoE with the same configuration but replacing the MLPs with the MoE modules ( (Komatsuzaki et al.). By replicating the dense model's MLP weights nepertt times to form the experts, the knowledge from the dense model can be reused to accelerate the training of the MoE compared to training the MoE from scratch (from random parameter initializa- tion). Other modules' weights are copied from the dense counterparts directly, with the router's initial weights ran- domized. See Figure 2 for an illustration of upeycling. We employ this technique for our study. See Appendix A.6 for other details, including alternative upcycling methods.
We denote N1 by the dense model's 's total number of non- embedding parameters, N2 by the MoE's total number of non-embedding active parameters, Nuotal by the MoE's total number of non-embedding parameters. They are given by (ignoring subleading contribution proportional to dmodel)
N1 = (4 + 12) moddel 16Mayer
L model (4)
N2 & (4 + 12n Mayer emnded
(5)
Ntotal & (4 1e2expert Mayer
(6)
where the attention and MLP module contribute 4d 4df and 12d parameters respectively per layer.
Learning rate schedule. We are interested in training models with different numbers of training token budget. Previous work (Hoffmann et al., 2022) used the cosine learning rate (LR) schedule and trained separate models for each number of training token budget, which is resource- consuming. We instead employ the warmup-stable-deen. (WSD) learning rate schedule (Bi et al., 2024; Hu et al., 2024), which requires only a single model sweep with a sufficiently large number of training tokens. As the LR is constant for the majority of learning, the saved interme- diate model checkpoints can be reused to emulate differ- ent numbers of training token budget: the checkpoints are continued-pretrainng reusing the optimizer states from the constant LR stage with a new, shorter LR schedule. This reduces substantially the number of new training runs re- quired. In Appendix A.4, we show that the performance of dense and MoE training is on par with the usual cosine LR schedule. The maximum LR value is tuned individually for each model size and for both dense and MoE training.
2.2. Power-law Ansatz
There is an extensive literature showing that the loss of train- ing deep learning models has a simple power-law behavior: -= A x + E, for single variable X, including model size and dataset size (Hestness et al., 2017; 2019; Rosenfeld et al.; Henighan et al., 2020)'. We use this ansatz in this
work.
Hoffmann et al. (2022) have shown that when training dense transformer, the cross-entropy loss is well-described by the following "Chinchilla" scaling law:
== AD a BN +E
The first and second terms quantify the limitation of learn- ing due to limited dataset and model sizes respectively. The scaling exponents, o, 33 control how fast the loss decreases with respect to dataset and model sizes respectively. E is a constant: it is the irreducible loss representing the inherent entropy of text. All parameters are be fitted with experi- mental observations. We also assume Equation 3 for models trained from scratch.
Dataset. We use training dataset derived from the Com- monCrawl portion of Slimpajama-DC: (Shen et al., 2023), containing 368B tokens in total. The test loss calculated from the default validation set (0.3B tokens) defined therein. In Appendix B, we train models on two different datasets (Japanese language and source code datasets) to show that the scaling behavior generalizes across datasets.
3. Experimental Design and Results
In this Section, we describe the design of our experiments, before presenting some of the experimental results. Ad- ditional details, including ablation studies are available in Appendix A.
3.2. Upcycled Training
Upcycled training is performed by first initializing the MoE with the dense pretrained models as mentioned before. Sub- sequently, we train the upcycled MoE with standard cross- entropy loss augmented by an auxiliary load-balancing loss with coefficient 10-3 to minimize expert's imbalance of
'The actual form we assume is L. ( + F such that the loss is finite at the limit X + 0. However, since the values of X we consider often much larger than I (10' more), approximate as = A + for notational convenience.
3

---

Scaling Laws for Upeycling MoE
Dense model (Liama etc.) of size N
Upcycled MoË (Mixtral etc.)
Ougput
Trained tokens
Output Trained for Dj tokens
Add
Add
MLP (multi-layer perceptron)
Duplicate
MLP
MLP
TooK
Rourer
MoE module
Norm
Norm
Add
Add
Copy
Attention
Attention
Norm
K.Rurm
Input
Input
Figure 2. Upcycling and factors affecting MoE's performance. Upeycling involves initiating the weights of the MoE (activating TTTpp experts per token) by reusing the weights (duplicating the weights of MLPs Nexport times) of an existing dense transformer of size Nj that has been trained for Dj tokens. The (upcycled) MoE is further trained for D2 tokens. Language modeling performance improves when scaling these factors. We study and develop formulac (scaling laws) consisting these factors to predict the empirical performance.
activation. See Appendix A.7 for ablation of the coefficient.
01=9:23,02-9.23
D1-9923.02-11155
To this end, obtained three sets of results in total, of which we denote by: dense training (dense model training from scratch), MoE training (MoE training from scratch), upcycled training (MoE training for D2 tokens where the dense model pretrained for D, tokens has been reused).
D1=9.25.02-2.11
2.9
D1-9.73,07-4.61
D1=1.15.D2-9223
D1=1.15.02=1.15
01-1.15.02-2.31
D1-1.15002-4611
2.7
3.3. Training results
28
We show some of the resulting test loss curves of our upey- cling experiments in Figure 3 (see model evaluation with standard benchmarks in Appendix A.9). Some observations can be made from the plot:
D2, Tokens trained (B)
Figure 3. Loss curves of upcycling. Intermediate test losses of the 8x0.1B MoE (2 experts activated per token) trained for a variety of total number of tokens, D2, when upcycled from dense model pretrained with various numbers training tokens (D2) in B.
(a) upcycling from a dense model with more pretrained tokens leads to lower initial losses.
(b) The overtrained dense model is, the smaller the rate of the final loss change of the upcycled MoE becomes (the exponent becomes smaller).
4. Scaling Laws
Our ultimate goal is to understand the performance of upey- cling with respect to various factors illustrated in Figure 2. However, simultaneously varying all these variables in- vestigate the scaling behavior computationally prohibitive. We separately investigate two aspects closely related to up- cycling: dataset sizes (Section 4.1) and model configuration (Section 4.2). In the next Section, we further study the joint scaling behavior of dataset and model sizes with a predetermined MoE architecture.
The reason for (a) is function preservation. As all experts inherit the same weights from the dense model, the output of the MoE module is preserved from the dense model irrespective of the routing at initialization, and therefore the loss preserved as well. For (b), our intuitive explanation is as follows. The duplicated experts' weights already close to the optimal ones when they are upcycled from overtrained dense model. It is then harder for the experts to diversify and specialize at the MoE training stage to further lower the loss. As we will reveal soon, our proposed scaling law captures these phenomena.
4.1. Scaling Law for Dataset Sizes
We fix the model size while varying D,, to study the scaling behavior with respect to these variables. To determine the functional form of the scaling law, I(D,, D2), we require

---

Scaling Laws for Upeycling MoEE
it satisfy certain properties:
From Figure 4, we observe that the scaling exponent in Equation 7 decreases as D1 increases. To quantify this relationship, we model the scaling exponent as a function of D1. denoted as -a( D1). A scatter plot of -a(D1) reveals the following logarithmic relationship:
Requirement 1. L(D1, D2) follows the power law with respect to D2, as shown empirically in Figure 4. This aligns with the power-law ansatz, treating upeycling as analogous to standard MoE training with dataset size D2, initialized with dense parameters rather than random weights:
-i(Di) - 7 log D1 + E..
(10)
L(DI, D2) Ln, = AD2" +E
(7)
which fits the data well, as shown in the same Figure. Sub- stituting this expression to Equation 7, we obtain a term of the form D2 --= log D. directly supporting the interac- tion term in the multiplicative scaling law. This shows that the interaction term is not merely an artifact of theoretical derivation but is essential to explain observed empirical trends. Additionally, we show that the multiplicative depen- dence, A x D also arises from empirical observations, as detailed in Appendix C.
Requirement 2. As D2 - O, the loss should reduce to the power-law scaling behavior of the dense counterpart with respect to Dj, consistent with the function- preeeeving initialization of upcycling (see Section 3.3):
Lo D2) = ADT" + +E
We investigate the following functional forms of power law satisfying these requirements:
Multiplicative:
4.1.1. I. INTERPRETATIONS
L(D,, D2) = ADT (1 + D2) -ax+ag log Dr + E
Several quantitative and qualitative observations can be made from the scaling law of dataset sizes. From our fit (Table 8 in Appendix C), we notice trend 02 Z 01 > a3. This means that while increasing either of D1 and D2 helps improve performance, as upcycled training has a slightly larger exponent (a2), increasing D2 helps train faster.
2 AD D22 02g log Di + F
(8)
Additive:
I(D1,, D2) = AD
01 + F(1 + D2) tag log Dr
+ E
~ ADT + FD2 FD log D.
+ E
(9)
Upcycled MoE has better head start (effective scaling factor is smaller). Fixing D1, we see that the effective scaling factor for D, is AD Increasing D2 lowers the effective scaling factor, and hence the loss of upeycling. Indeed, fixing D2, we see that the model performs better with increasing D₁ in Figure 3.
Both forms include an interaction term (og log Di) in the scaling exponent, capturing the interplay between D, and
D2-
Empirical comparisons of functional forms. We empiri- cally compare these functional forms, including the special case where Dj and D2 have no interaction. The optimiza- tion uses the Huber loss (8 = 10-3) and the BFGS algo- rithm, fitting the logarithm of the loss via the LogSumExp trick applied to the RHS of Equations 8 and 9. The leave- one-out root mean square error (RMS) serves as the fit metric.
Upcycled MoE trains slower with larger sunk cost (ef- fective scaling exponent is smaller). Again fixing D1, we see that the effective scaling exponent with respect to D₂ is 022 0z3 log D1. This means that the larger the sunk cost (D1) is, the loss decreases more slowly with D2, indicating diminishing returns from increasing D2 at higher Dj values, agreeing with Figure 4's results.
The fit is performed on O.IB dense model upcycled to MoE architectures with Rexpert {4, 8) and TTop [1,2}, , trained on 5 x 5 grid of D₁, Dz. The fitting results are shown in Table 1, where we can see that the multiplicative functional form (with non-zero interaction) achieves consistently the lowest leave-one-out RMS error across the experimented MoE architectures. Henceforth, we adopt Equation 8 in our scaling laws.
(Peespert..Toppk)
(4.1)
(4,2)
(8.1)
(8,2)
Mul.
0.0111
0.0081
0.0105
0.0031
Mul. (ars = 0) Add.
0.0169
0.0085
0.0180
0.0095
0.0165
0.00843
0.0167
0.0093
Add.(ag = 0)
0.0196
0.0117
0.0430
0.0113
Table 1. Multiplicative scaling law with interaction consistently achieves lowest error: Leave-one-out RMS error for fitting the loss for MoFs upeycled from dense O.IB model, with functional forms of Equations 8, 9, and specific cases with ag = O. The first and second number in the bracket of the first row indicates the MoE architecture's 's parameter, Nexpert and Topp respectively.
Multiplicative scaling law from empirical observations. To strengthen our proposal for Equation 8, we demonstrate through a bottom-up approach that the multiplicative nature of the scaling law, including the interaction term, arises
naturally from empirical data.
5

---

Scaling Laws for Upeycling MoE
De cokens, Dj
0:50
Derse takens,
1:79
1:30
Dense takens, Dt
6:20
13.00
1.88
2:30
4,00
18.3
2.50
5.20
2.2
2.3
((D2)) AD7" +E
24
8x15M
8x0.18
8x18
23
2.55
Upsydled lokers,
Upsydied toters, Dj 151
Uppyciedd botenns D2 IDI
0.05
40.09
00.0
-000
-d1Du
0.09
0.06
8x15M
8X0.1B
8x1B
-0(D
D1 +
.0.12
40.09
3.25
0.50
1.00
1.50
2:00
50
Da (B)
D:
Figure 4. Top: D2 has power-law scaling. We show scaling behavior of upcycled training tokens (D2) for different values of dense tokens (D1). Bottom: Interaction term explains decreasing exponents. The fitted exponents in the upper plots used to fit Equation 10 as a function of D1, and are shown to agree well with the functional form.
4.2. Scaling Law for Model Configuration
the following: dense model of size {15M, 44M, O. 0.2B), Rexpert {4, 8, 16, 32} and RTopk = {1, 2, 2, The scat- ter plot is shown in the left panel of Figure 1 (for better readability, see the 2D slices of the plot with fixed spar- sity in Figure 13). To this end, we find that the following functional form provides the best fit to the data:
Previous Subsection mainly concerns with when to upey- cle, i.e., the training tokens used when upcycling. Here, we study how to upcycle the dense model, i.e., the model configuration of the MoE.
L(P, AP- FN So +E
(11)
3.2
Predicted Loss
See Appendix D.1 for details.
3.0
We obtain the following values for the exponents: 81 1.87 B2 0.34. See Figure 5 for the fits. Unlike dataset sizes, due to computational constraints, we are unable to generate more data points to make a more precise estimate extending the range of model configuration. Nevertheless, we do not observe performance trade-offs when scaling sparsity and active parameters within the range of our ination, in contrast to the scaling law of dataset sizes. Our work suggests that: Performance of upcycling improves with sparsity and active parameters without noticable trade-offs.
2.8
2.6
2.6
2.8
3.0
3.2
Actual Loss
Figure 5. Fits of scaling law for model configuration.
To do this, we define sparsity, P = Niotal/N and con- sider the dependency of the performance on sparsity and active parameter N2. of which the variables capture scaling behavior with respect model parameters and architectural details (Nexpert. and (TTpp). See Equations 4 to 6.
5. Joint Scaling Law
In this Section, show that the upcycled MoE follows joint scaling law with respect to dataset size and model size, provided the MoE architecture is fixed. While increasing sparsity and active parameters generally improves perfor- mance as shown before, compute and memory constraints impose practical upper limits. Thus, while scaling MoE
Using these variables, we can derive and validate the ap- propriate functional form for the scaling law, similar to the analysis performed in Section 4.1. We fit the functional form by running experiments fixing D1 D2 = 2.3B, varying
6

---

Scaling Laws for Upeycling MoE
configurations much as possible is theoretically benefi- cial, practitioners should choose architecture that aligns with their available compute, memory, and expected train- ing token budget. For our study, we adopt widely used MoE configuration (Rexpert = 8, TeTopk = 2), which has been implemented in several large-scale, publicly available models, including Mixtral-8x7B and 8x22B (Jiang et al., 2024), and Frrashina2-8x70--
344
Fited Villdated
Predicted Loss
2.8
2.6
2.4
To establish the scaling law with respect to model size, it is sufficient to consider the dependency solely on N1 among other definitions (Equations 4, 5, 6), as they are equivalent up to a multiplicative constant, e.g., / = 1.75. To determine the scaling law, we require the functional form to satisfy the following, analogous to previous Requirements:
2.0
2.2
2.6
3.2
Actual Loss
Figure 6.
Fits of the joint upcycling scaling law.
A
B
a//1
/
Requirement LD.,D, (Ni) Dr + ED,.Da
Dense
8.83
12.3
0.088
0.116
Requirement 4. Lo D2. N1) ADT +
MoE
32.0
7.05
0.161
0.080
BN 8 + E, i.e., approaching Chinchilla scaling law. The straightforward functional form that satisfies these require- ments is as follows 3.
Upcycled
16.3
8.53
0.043
0.085
7.98c-4
0.112
Table 2. Fitted parameters for joint scaling laws. Note that for MoE, we fit the parameters with variable Ni (instead of Nu) for comparison conveniences. The irreducible loss applicable all laws, E, is fitted to be 0.165.
L(Di, ADT D2 t2 log Di + BNN + E (12)
Fitting, Towards this end, we fit three separate scaling laws, corresponding to dense, MoE, and upcycled trainings, incor- porating dataset and model sizes (the first two are fitted with the functional form of Equation 3). We use the same fitting procedure presented in Section 4.2. As the irreducible loss E is universal irrespective of scaling laws under considera- tion, we first estimate E from joint fit of dense and MoE scaling laws, then fix it when fitting the upcycling scaling law to reduce overfitting and enhance extrapolation ability.
found that upcycling retains an advantage up to 120% of the sunk cost (D2). For instance. to match the performance of an upcycled MoE trained with an additional 0.4 trillion (T) tokens after 2T dense tokens, training an MoE from scratch would require 2.4T tokens effens--efectively saving 2T tokens in upcycled training. Conversely, Muennighoff et al. (2024) reported that training from scratch requires less than 100% of the sunk cost under different settings, indicating that it could be more efficient. These seemingly contradictory results suggest that upcycling efficiency depends on both sunk cost and model size.
In Figure 6, we show that the fitted result of upcycling scal- ing law extrapolates well, achieving validation RMS error of 0.015. The fitted parameters using all data for all three scaling laws are summarized in Table 2. We consider other fitting possibilities and present more results in Appendix E. For the rest of this Section, we consider the application and implication of the joint scaling law.
To investigate this, we define D' the number of tokens required for training from scratch to match the performance of an upcycled MoE with the same sunk cost, i.e.,
Upe (D1 = D', MoF LN
(13)
5.1. Training MoE from Scratch versus Upcycling
Since the above equation involves non- non-integer polynomial exponents, we solve it numerically and approximate the solution analytically (Equation 2). Figure shows that D* decreases with increasing model size, with D* equal to 4B tokens for an 8x model. When D2 S D*, the required DMoE catch up more than 100% of Dí: upcycling remains more efficient than training from scratch. However, for D2 Z D*, the efficiency reverses, favoring training from scratch. Our findings indicate that: Upcycling is more efficient than training an MoE from scratch for lower sunk cost and training token budget, but its efficiency diminishes as the sunk cost or model size increases.
Past studies (Komatsuzaki et al.; Muennighoff et al., 2024) have explored the efficiency of training an MoE from scratch compared to upcycling a dense model. Komatsuzaki et al.
Phttps: http://gggggggggccooooo wttp://ee....ooioiiiiii....cieat wttsssiiiiiiiiiio
sarashina2 70b
*One could extend this with Equation 11 create a truly unified scaling law encompassing all model configurations. However, we refrain from doing so in this work. as reliably fitting such scaling law with many independent variables would require significantly extensive experimentation beyond current computational capabilities.
7

---

Scaling Laws for Upeycling MoE
5.2. Compute Allocation and Ooppule-Optimal Upeycling
than twice the budget used for dense training. In contrast, we have presented a more general guideline.
Given fixed total floating-port point operation (FLOPs) budget C, we analyze how to optimally allocate compute between dense training and upcycled training. Specifically, we solve:
Scaling laws. Power-law scaling appears in a variety of natural and -aman--made phenomena. Scaling studies that are perhaps closest to our work are those considering two- stage training, e.g., transfer learning, fine-tuning, and model growth (Mikami et al., 2022; Zhang et al.; Du et al., 2024). Prior work on transfer learning has proposed scaling laws us- ing multiplicative additive forms involving D1 (pretrain- ing data) and D2 (fine-tuning data) (Mikami et al., 2022; Zhang et al.). A notable similar phenomenon in transfer learning is ossification, where pretraining can hurt fine- tuning performance (Hernandez et al., 2021). However, to our knowledge, no prior work incorporates an interaction term (ag in Equation 1) to capture such effects. Although our experiments specifically focus on upcyeling into MoE models, we believe that the core insights, such as the interac- tion between dense and upcycled training budgets, are useful for formulating two-stage training regimes more broadly, including transfer learning as mentioned above, and poten- tially model growth.
min I(DI, D2, N)) s.I. FLOPs(DI. Ni) = C P1.
We find that for a given compute budget without any pre- trained models, training a compute-optimal dense or MoE from scratch outperforms the two-stage dense-to-upcycled MoE approach (see Figure 16 in Appendix F22). This sug- gests: If no pretrained model exists, direct training is preferable to upeyeling for optimal performance.
In scenarios where pretrained dense model is already avail- able, we can determine the compute- optimall scaling of MoE upcycling. The compute cost of upeycled training is approx- imated as C2 = 62 (Kaplan et al., 2020). Optimizing the compute leads to the scaling relations (see Appendix F.2 for derivation):
Reff
D x C2 "aff
NTPi X C2 Peff
(14)
In the context of MoE, Clark al. (2022) studied how different architcctural choices affect MoE's scaling, while Krajewski et al. (2024) investigated fine-grained experts' scaling behavior. The latter work fits joint scaling law with respect to dataset and model sizes, but we find differ- ences in the obtained parameters. It is likely due to several differences in methodology: the largest dense model they experimented with was smaller (85M), and they did not tune the LR for each model size. Nevertheless, our find- ings that MoE scales better than its dense counterpart with sufficiently large computational budget do agree with theirs.
where Geff == 02 az log Dj. Notably, as D1 increases, Ocff decreases, meaning larger pretrained models require disproportionately more tokens for efficient upeycling.
For instance, applying this to the Llama2 models (7B, 13B, 70B), which were trained on 2T tokens (Touvron et al., 2023), we estimate that compute-optimal upcycling follows the scaling D, x N11.8 indicating that larger models require nearly quadratic increases in upcycling data.
Overall, we find that upcycling is inefficient relative from- scratch trainings when considering compute optimality.
7. Discussion and Conclusion
6. Related Work
In this paper, we have presented compelling evidence that MoE upcycling follows novel scaling laws with dataset and model configuration, revealing trade-offs due to interactions between dataset sizes.
Mixture-of-Experts, While interests in developing open
MoE LLMs are relatively recent (Hu et al., 2024; Yang et al., 2024; Dai et al., 2024; Liu et al., 2024; Sun et al., 2024), the use of MoE in deep learning can be traced early 2010s (Eigen al., 2013; Bengio al., 2013). See Cai et al. (2024) for detailed survey of modern MoE models.
Given the complexity of upcycling, deriving a unified scal- ing law that allows for simultancous optimization of dataset size and model configuration is beyond our current compu- tational reach. However, our findings still provide practical guidance, as the joint scaling law (Equation 12) enables ac- curate predictions of upeycling performance once the MoE architecture is fixed. In practice, we recommend selecting the largest viable MoE size and sparsity within compute and memory limits while balancing this choice against the expected training token budget. Once the architecture is set, Equation 12 can guide dataset/model scaling, demonstrating reliable extrapolation of upeyeling performance.
Upcycling, Leveraging pretrained models to expedite the training of larger dense models is also known as model growth (Chen et al., 2015). In the context of training MoEs reusing dense pretrained models, Komatsuzaki al. are the first studying such a scenario with encode-decoder models. There are studies (Hu et al., 2024; Yang al., 2024; Lo et al., 2024; Wei et al., 2024; Muennighoff et al., 2024; He et al., 2024) offering insights into upcycling decoder-only transformers. However, a systematic investigation has been lacking. Wei et al. (2024) made only a rough recommenda- tion: use upeyeling when the budget for training is smaller
While our empirical formulae successfully capture the ob-
8

---

Scaling Laws for Upeycling MoE
served scaling behavior, the underlying mechanism, partieu- larly the interaction term, remains theoretically unexplained. To our knowledge, formal justification for such a term is lacking in the literature (Paquette et al.). Future work could explore it through, e.g., simplified models (Hutter, 2021; Maloney et al., 2022; Lin et al.; Bordelon et al., 2024; Bahri et al., 2024; Paquette et al.). Additionally, extending our scaling laws to alternative MoE architectures (shared ex- perts and fine-grained experts (Dai et al., 2024; Krajewski et al., 2024)), modalities (vision transformer (Zhai et al., 2022)), and other multi-stage training paradigms (as dis- cussed in Section 6) would help assess their generality and implications.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., al. Piqa: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 34, pp. 7432-7439, 2020.
Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. In Proceedings of the 41st International Conference on Machine Learning, pp. 4345- 4382, 2024.
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., and Huang, J. A survey on mixture of experts. arXiv preprint arXiv:2407.06204, 2024.
Chen, T., Goodfellow, I., and Shlens, J. Net2net: Accel- erating learning via knowledge transfer. arXiv preprint arXiv: 151.055511.05411,, 2015.
Software and Data
The source code and data (cross-entropy losses) for analyses of the paper is available at
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W. Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
https: / ctt.ccco//aaoooo////// wtt.//ooot/siiiieeeeeeee wtssssiiiiiinss.... sparse upcyccinu scalings laws.
Impact Statement
This paper works toward the goal of advancing the field of machine learning and language modeling, with an emphasis on scaling. There are many potential societal consequences of our work, none which we feel must be specifically high- lighted here.
Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoe, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed lan- guage models. pp. 4057-4085 PMLR, 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 05457, 2018.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 3XXi:23033088744 2023.
Computer, T. Redpajama: open dataset for train- ing large language models, October 2023. URL http:://github..ommmmgggege www.ub.com/toooo RedPajama-DD -aaaaaama-Daa....
Bahri, Y. Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27)e23111877 1877812 2024.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts lan- guage models. arXiv preprint arXiv:2401 10000666666 2024.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3453 2013.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 35:16344-1635, 2022.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. Deepseck Ilm: Scaling open-source language models with longtermism. arXiv preprint 2rXiv:2401.02990 2024.
Du, W., Luo, T., Qiu, Z., Huang. Z., Shen, Y., Cheng, R., Guo, Y., and Fu, J. Stacking your transformers. A closer look at model growth for efficient Ilm pre-training. arXiv preprint arXiv:2405. 15319, 2024.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. pp. 2397-2430. PMLR, 2023.
Eigen, D., Ranzato, M., and Sutskever, I. Learning fac- tored representations in a deep mixture of experts. arXiv preprint 13/2.4334 2013.
9

---

Scaling Laws for Upeycling MoE
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23( 23(10))11-39 2022.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.40088 2024.
Gao, I.., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le No A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001 083661, 2020.
J., Reynolds, L., Schoelkopf, H.. Skowron, A.. Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https::/ : /enodo. org/ records// 12608602.
Kocetkov, D., Li, R., Jia, L., Mou, C., Jernite, Y., Mitchell, M., Ferrandis, M., Hughes, S., Wolf, T., Bahdanau, D., et al. The stack: tb of permissively licensed source code. Transactions on Machine Learning Research.
Hägele, A., Bakouch, E.,, Kosson, A., Von Werra, L.,, Jaggi, M., et al. Scaling laws and compute-optinal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232- 2024.
Komatsuzaki, A., Puigcerver, J., Lee- J., Ruiz, C. R., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., and Houlsby, N. Sparse upcycling: Training mixture-of- experts from dense checkpoints.
He, E., Khattar, A., Prenger, R., Korthikanti, V., Yan, Z., Liu, T., Fan, S., Aithal, A., Shoeybi, M., and Catanzaro, B. Upeycling large language models into mixture of experts. arXiv preprint arXiv:240..75224 2024.
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Ander- sch, M., Shoeybi, M., and Catanzaro, B. Reducing acti- vation recomputation in large transformer models. Pro- ceedings of Machine Learning and Systems, 5:341-353, 2023.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Krajewski, J., Ludziejewski, J., Adamezewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóždú, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.0787/ 2024.
Hernandez, D., Kaplan, J., Henighan, T., and McCan- dlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Le Scao, T. Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., Raffel, C., Sanh, V.. Shen, S.. Sutawika, L., Tae, J., Yong, Z. X., Launay, J., and Beltagy, I. What language model to train if you have one million GPU hours? pp. 765-782, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/202.. madings-emmllp URL https: / // a aalanthologgyyy org/ org/2022.0 2022. fi nd ngs emn p. 54.
Hestness, J., Narang, S., Ardalani, N., Diamos, Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 00049, 2017.
Hestness, J., Ardalani, N., and Diamos, G. Beyond human- level accuracy: Computational challenges in deep learn- ing. pp. 1-14, 2019.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T.. Rutherford, E., de Las Casas, D.,. Hendricks, L. A., Welbl, J., Clark, A., al. Training compute- optimal large language models. Proceedings of the 36th International Conference on Neural Information Process- ing Systems, pp.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Gshard: Scaling giant models with conditional computation and automatic sharding.
2022.
Lin, L., Wu, J., Kakade, S. M., Bartlett, P., and Lee, J. D. Scaling laws in linear regression: Compute, parameters, and data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Hu, S., Tu, Y., Han, X., He. C., Cui, G., Long. X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint NNi:22404.06395, 2024.
Liu, J., Cui, L., Liu, H., Huang. D., Wang, Y., and Zhang, Y. Logiqa: a challenge dataset for machine reading compre- hension with logical reasoning. pp. 3622-3628, 2021.
Hutter, M. Learning curve theory. arXiv preprint arXiv:2102.04074 2021.
10

---

Scaling Laws for Upeycling MoE
Liu, L.., Kim, Y.J., Wang, S., Liang, C., Shen, Y., Cheng, H., Liu, X., Tanaka, M., Wu, X., Hu, W., et Grin: Gradient- informed moe. arXiv preprint arXiv: 2111:2409..1166, 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. 2017.
Lo, K. M., Huang, Z., Qiu, Z.,, Wang, Z., and Fu, J. A closer look into mixture-of-experts in large language models. arXiv preprini arXiv:2406. 18219, 2024.
Shen, Z., Tao, T., Ma, L., Neiswanger, W., Liu, Z., Wang. H., Tan, B., Hestness, J., Vassilieva, N., Soboleva, D., et al. Slimpajama-do: Understanding data combinations for Ilm training. arXiv preprint arXiv:2309. 108/8, 2023.
Loshchilov, I., Hutter, F., al. Fixing weight decay regu- larization in adam. arXiv preprint arXiv:1711.05101.5 2017.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv: 1909:v/909.000000 2019.
Maloney, A., Roberts, D. A., and Sully, J. A solvable model of neural scaling laws. arXiv preprint arXiv:2210. 2022.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B to- ken cleaned and deduplicated version of RedPajama, June 2023. URL https: hgtp:///gaga/ccceeeee../oooa/ co/ datasets/cerebrat
Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y. Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for syn2real transfer: How much is your pre-training effective? pp. 477-492. Springer, 2022.
imP aaaaaaaaaaaaa21111 627B.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50377 2023.
Su, J., Ahmed, M.. Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127066 127063, 2024.
Sun, X., Chen, Y., Huang, Y., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., et al. dunyuan-large An open-source moe model with 52 billion activated pa- rameters by tencent. arXiv preprint arXiv:2411.02266 2024.
Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O.,, Lambert, N., et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02660, 2024.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q..
Bernrrdi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction re- quiring broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1525-1534 2016.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint urXiv:2307.09288. 2023.
Paquette, E.,, Paquette, C., Xiao, L., and Pennington, J. 4+
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
phases of compute-optimal neural scaling laws. The eight Annual Conference on Neural Information Processing Systems.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Infor- mation Processing Systems, 37:100535-100577 2024.
Wei, T., Zhu, B., Zhao, L., Cheng, C., Li, B., Lüi, W., Cheng, P., Zhang, J., Zhang, X., Zeng. I.., et al. Skywork-moe: A deep dive into training techniques for mixture-of-experts language models. arXiv preprint arXiv:2406.06563. 2024.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales.
Welbl, J., Liu, N. F,, and Gardner, M. Crowdsourcing multiple choice science questions. CANUT2017, pp. 94, 2017.
Sakaguchi, K., Bras, R. L.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Yang, A., Yang. B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F,, al. Qwen2 technical report. arXiv preprint arXiv:2407.1067/ 2024.
Shazeer, N. Glu variants improve transformer. arXiv preprint arXi:22002.05202 2020.
Zeng, A., Liu, X., Du, Z., Wang. Z., Lai, H., Ding, M., Yang, 7., Xu, Y., Zheng, W., Xia, X., et al. Glm-1300

---

Scaling Laws for Upeycling MoE
An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. pp. 22104-1211, 2022.
Zhang. B., Liu, Z., Cherry, C., and Firat, O. When scal- ing meets I1m finetuning: The effect of data, model and finetuning method.
Zhang, P.. Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401002385, 2024.
12

---

Scaling Laws for Upeycling MoE
A. More on Architecture and Experimental Design
A.I. Megatron-LM configuration
Infrastructure. Our experiments are performed on multiple nodes, each consisting of 8 NVIDIA H100 80 GB GPUs, interconnected via InfiniBand HDR. The software we use for training the Megatron-LM library (Shoeybi et al., 2019).
We and modify the Megatron-L.M (core v0.8.0) library for our experiments". Models are trained with data type 6 Except for the largest MoE we train (8x1B), which has tensor parallelism configured to be 2. all models are trained with data and sequence parallelisms only (Korthikanti et al., 2023). Other optimization libraries used include FlashAttention (Dao et al., 2022) and TransformerEngine See the example scripts provided on Github".
A.2. Model configuration
Let us elaborate more on our architectural choices. The intermediate hidden dimension size, dMLP. is set to be modell We do not implement bias in the linear layers. We also do not use techniques geared for treating training instabilities (which we did not encounter in our study), such as 7-loss or QK normalization. Efficiency-mootivtted implementations like grouped query attention are not considered as well for simplicity. The number of attention head is chosen to increase with model size following practices in the LLM literature. Other designs of the architecture follow Llama2's closely (Touvron et al., 2023). See Table for the model configurations. They are selected such that the ratio Tlayer// lies in the range 32 to 64, as in Kaplan et al. (2020). We use the smaller models for ablation studies.
A.3. MoE configuration
Let us describe the routing mechanism within the MoE module studied in this work. Denote the number of experts by nexpert. the number of activated experts by TTppkk and the output of expert by Oexp,i. At each layer, the output tokens of the attention layer are passed to router, which consists of single-layer perceptron with weight W, responsible for calculating (z), G2(z),... G Greppnn (z)), where
G(z)= Softmax(TopK(W z))
(15)
The TopK operation ensures that only nexpert experts activated for each token, resulting in sparse computation. The output of the MoE module, OMoE, is the weighted expert outputs which can be written as follows:
nexpori
OMoE
Σ Gs(x))esp..())
i=1
Note that this is also known as token-choice algorithm. Furthermore, we do not use the token-dropping mechanism as in Fedus et al. (2022). We also do not study MoE variants such as shared experts and fine-grained experts (Dai et al., 2024; Krajewski et al., 2024), as upcycling these variants is not straightforward.
Let us move to discussing the hoad-balancing loss. It has the form (Fedus et al., 2022):
4n
Rexparta
T
Laux
Σ
Σ
P3
Qi
where 17 is the coefficient for the auxiliary loss, the number of tokens, PI. the router output probability for token j be assigned token and Q; is the number of token assigned to expert i. We ablate the coefficient in Appendix A.7.
A.4. Ablation of learning rate schedules
Here, compare the performances of using WSD and the commonly used learning rate (LR) cosine schedules. Dense model and MoE used in our ablation are 0. B. and 8x44m respectively, with training configuration given in Table 6. We can see from Figure that both schedules yield similar performances.
'https: ://g thub. com NV WWW.I//////gatoo--.Il.. 'ttpps: : / /i www.tttw.www.....o http://WWWIAAA/aaaaffffrrrrrrrgn
dense wtt.a//...g...........occooo/ota
MoE example:
13

---

Scaling Laws for Upeyeling MoE
Dense, cosine
Dense, WSD
MoE, cosine
2.5
McE,
2.0
1000
1.5
1.0
0.5
cosine
1.75
Training
training tokens
Figure 7. Comparing WSD and cosine schedules of learning rate. Left: we see that different schedules cause little differences between the losses, for both dense MoE training. Right: the learning rate schedules use shown.
A.5. Training configuration
The common setup training shown in Table 4, and the model-dependent setup (warmup iteration, standard deviation of the normal distribution for initializing weights, maximum iteration run, battch size, tuned LR) is shown in Table As described in the main text, set to be roughly the same the total model size (Porian al., 2024). Linear decay 10% of the maximum LR value is used in the last stage of the schedule, with the length set to be around 10% of the training length, following Hägele et al. (2024).
the WSD schedule for training. The number of warmup steps of the WSD LR schedule is
saarithmically-spaace intermediate checkpoints are saved and used to emulate different numbers of training token budget. We also increase both the training length and batch size with model size following common practices without performing precise tuning.
Model
7laaer
dmodel
Thead
denss
Nttota (8 experts) MoE
15M
9
320
4
14,751,680
92,189,120
44M
12
480
8
44,248,800
276,538,088
0.1B
15
640
8
98,323,400
614,496,640
0.2B
21
832
8
232,623,040
1,453,845,952
0.5B
26
1120
16
521,889,766
3,261,732.320
1B
30
1504
16
1,085,859,424
6,786,500,704
Table 3. Dense models used in study and their parametric details. Note that dMLP, is set be 4dmodel- In the last column, show the total non-embedding model parameters of the corresponding MoE with experts.
Configuration
Details
Context length
1,024
Embedding tying
False
Optimizer
AdamW (Loshchilov et al., 2017)
Adam 31
0.9
Adam Bg
0.95
Adam E Weight decay
1e-8
0.1
Gradient clipping
1.0
Table 4. Training configuration used throughout the paper.
14

---

Scaling Laws for Upeycling MoE
Model
warmup iter.
init. size
Max iter.
batch size
LR (8x)
15M
200
0.035
17,600
128
8e-3 (2e-3)
44M
200
0.029
17,600
256
4e-3 (2e-3)
0.1B
200
0.025
17,600
512
4e-3 (2e-3)
0.2B
400
0.022
35,200
512
2e-3 (2e-3)
0.5B
800
0.019
70,400
512
4e-4 (4e-4)
1B
800
0.016
70,400
1024
4e-4 (4e-4)
Table Model-dependent training configuration. "init. size" refers standard deviation the normal distribution used for initializing the weights. "Max iter." refers to the maximum iteration run on the model. The MoE counterpart uses the same configuration except learning rate (last column).
Configuration
Details
Batch size
512
train iter.
4,000
Warmup iter.
200
Auxiliary loss coeff.
10-3
Table 6. Training configuration for ablation studies.
A.6. Upcycled training's configuration
We initialize the router weights from normal distribution with zero mean and variance of 2/ 2/ddmodel (Le Scao et al., 2022: Chowdhery al., 2023; Zeng et al., 2022) (the same initialization is used for from-scratch trainings).
Regarding the learning rate (LR) of upcycled training, there are several choices: using the LR at the end of dense training with a constant LR schedule, i.c., treating upcycled training as a kind of fine tuning; using the LR of dense/MoE training (Komatsuzaki et al.; He et al., 2024). We consider these three choices without retuning the LR. With the other training settings are set to be the same as the one used in training the dense models, including the use of the WSD LR schedule, we find that using the MoF. LR leads to better performance. See Figure 8.
As side note, observe that the loss for the constant LR schedule decreases monotonically, while the loss increases initially for other cases; there is rewarming stage when using the WSD LR schedules, which also be observed in Figure 3.
Let us further comment on alternative upcycling methods (parameter reinitialization) that could potentially further accelerate training. Adding some form noises to the dense MLP weights, or modifying partially the weights would intuitively help upcycled training generalize faster. Our preliminary experiments (adding Gaussian noise the expert weights of the upcycled model; partially randomizing the MLP weights; and substituting the MLP weights with their low-rank couterparts randomly encourage expert divergence) however did not any advantages of doing Note that this observation is also consistent with previous negative reports (Komatsuzaki et al.; Wei et al., 2024: Muennighoff al., 2024). Henceforth, we simply copy the weights directly from the dense model to perform upcycled training.
A.7. Ablation of auxiliary coefficients for load-balancing loss
The auxiliary loss affects the badd-balancing loss (smaller values meaning more balanced expert usage) well as the cross-entropy loss. We plot these losses in Figure 9 for training 8x0.1B MoE.
We see that, setting the coefficient to be too small leads to imbalance in expert usage, while setting the coefficient to be too large interferes with the cross-entropy loss. We set to around 10 which gives the right balance. We did not make finer tuning of the coefficients and adopted 10-3 in our experiments.
A.8. Ablation of dataset repetition for upcycling
In our experiments, we have used the same dataset for training both dense and upcycled models, following practices (He al., 2024). As an ablation, we split our dataset to two non-overlapping portions, and perform the two-stage training with
15

---

Scaling Laws for Upeyeling MoE
3.00
Constont LR
2.95
Dense LR (te-3)
2.90
2.75
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Tokens trained (B)
Figure 8. Ablation of L.R when upcyeling 8x0.1B MoE. We compare the performance of upcycled training (from a dense model trained for 2B tokens) using constant LR 10-4, LR the dense training), I.R used for dense training, and L.R used MoE training. We find that the latter works the best. The training setup follows Table
cooft=1
1.6
coeff=1
coeff=le-3
coeff=1e-3
coeff-1e-6
1.5
coeff-le
Lowet-balle Buct-ballncing Loss
1.4
LO55
1.3
1.2
1.0
0.50
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.0
0.5
1.0
1.5
2.0
Tikens traiindd (8)
Takens Urained (B)
Figure Ablating auxiliary coefficients. Left: Cross-entropy losses, where can seen that auxiliary coefficient of performs worst. Right: Load-balancing losses, where the larger the coefficient is, the smaller the load-halancing loss becomes. Setting the coefficient to he 10 gives the right balance between these two losses.
the distinct datasets. In Figure 10, we find that the difference in performance is very small. This also aligns with previous investigation on data repetition, where it is shown that there is a little difference in performance up to 4 times of repetition (Muennighoff et al., 2023).
A.9. Evaluation with standard benchmarks and comparison with other existing models
We compare the performance of our trained IB models against existing models with similar sizes, Pythia (Biderman et al., 2023) and TinyLlama (Zhang et al., 2024), based on standard natural language processing benchmarks, ARC (Clark et al., 2018), lambada (Paperno et al., 2016), logiqa (Liu al., 2021), 1), piqa (Bisk et al., 2020), seiq (Welbl et al., 2017), and winogrande (Sakaguchi et al., 2021).
Table 7 shows the results. First,
that the dense model perform similarly the open models, indicating that models have been trained correctly. Second, the total tokens used for dense and upcycled training are around 100B, similar to those under comparison), also indicating that upcycling has progressed correctly and improved the scores.
that the upcycled model achieves overall the best performance (note that
A.10. Estimated Total GPU hours
Instead of reporting the actual runtimes on our cluster, which varied in our experiments due to many factors affecting the cluster (number of available nodes, congestion, etc.), we give a theoretical estimate of total GPU hours used for obtaining the joint scaling law, which involves rumning the largest tested model with most training tokens in this paper.
The estimate is follows. We calculate the FL.OPs for training the dense, MoF, upcycled MoE models with maximum
16

---

Scaling Laws for Upeycling MoE
2.925
repetition
2:900
Repeated
2.850
2.525
2,800
2.75
2.750
2.725
Figure 10. Ablation of data repetition when upeycling an 8x0.1B MoE. We do not observe notable difference in the loss. The training
setup follows Tahle 6.
Models
Pythia-1B
VinyLlama-l.I
1B
Upeycled 8x1B
Datasets
Pile
Slimpajama & Starcoder
Slimpajama 74B
Slimpajama
Tokens
100B
103B
37B
ARC-c
25.59
24.32
27.65
30.12
ARC-e
47.26
44.91
52.10
56.14
lambada
53.52
45,08
49.72
logiqa
29.49
26.11
27.65
piqa
69.31
67.30
65.89
67.19
seiq
77.3
78.10
82.00
winogrande
51.22
53.28
54.93
57.77
Avg.
50.53
49.98
52.94
Table Benchmarks' performance comparison across models. Reported scores are accuracies (normalized by byte length whenever applicable). The first two columns scores of existing models. The last two columns are evaluation results of models trained in this work. The upcycled 8x1B model is upeycled from the 1B model. Our models are evaluated with the LM Evaluation Harness v0.4.0 library (Gao et al., 2024).
iterations using the 6ND approximation, ignoring the additional FLOPs required to continued pretrain models with shorter iterations (as can reuse the intermediate checkpoints). We further assume that the per-second TFLOPs of the GPU is 400, and is the same for both dense and MoE models ?. We obtain,
Dense model: 6.14 x 10 FLOPs
MoE: 1.08 x 18 FLOPs
Upeycled MoE: 5.38 18 FLOPs
The total GPU hours henceforth approximately 4,900. We note that the exact total GPU hours used are larger as we have run various additional studies as detailed in the paper. We further note that using the cosine learning rate schedule would cost about twice more GPU hours when varying the number of training token budget.
B. Generalization across Other Datasets
We show that the scaling behavior studied in the main text generalizes across datasets in use. Aside from Slimpajama (Computer, 2023; Soboleva et al., 2023; Shen al., 2023) used in the main text, test with two additional datasets: Japanese portion of the CommonCrawl corpus, and the Stack code dataset (Kocetkov al.). We use tokenizer of enlarged
https:: /tt::/gitti tttpt/tttttttt//////////t/.g / NVIDIA/ A WADDDI/MMaaatt. Megatren LM/bi ob/maaimiii. wttp:o/a/..mmeeeeeeeeeeeeoeeeee www.iiimgatonn//eeeeeeeeeeeeee ctte://cre/tteeeee cocor/trranse tta. /moe RRADME. md. Note that MoE requires more GPU memory to store its total model parameters, inducing overhead that may slow down training.
17

---

Scaling Laws for Upeycling MoE
vocabulary size of 102,400 when training with the former dataset We upcycle 0. model with the training configuration set to the same as those used for the experiments presented in the main text. 'he results are presented in Figure 11, where the scaling behaves similarly to those presented in Figure 4, suggesting that our scaling law for dataset is independent of dataset in use. We however do not further study scaling law for model configuration and the joint scaling law due to computational constraints.
further observe:
1. The Japanese dataset has higher validation loss (harder task), while the code dataset results in lower loss, making it a relatively easier task in terms of cross-entropy loss.
2. Interestingly, the Japanese dataset is harder to saturate with increasing D1 meaning upcycled training remains effective. In contrast, the code dataset saturates more quickly. making upcycling less beneficial.
The more fine-graing tineggraieed question, e.g., how mixtures of pretraining data impact respective downstream performances, is beyond the scope of our current work, and we leave it for future study.
Dense tokens.
Dense tokens, D2
1.28
1.28
1.78
1.78
2.38
2.38
4.68
4.68
10
11
Loo5
5
13
23
3.4
37
8x0.1B
8x0.1B
2.3
12
1.2
12
Upcycled tokers. Dj IB)
Upcycled tokens. Dy (B)
Figure 11. Scaling behavior generalizes across datasets used. Left: Japanese dataset. Right: Code dataset.
C. More Results on the Scaling Law for Dataset Sizes
C.1. Fitting across architectures
Figure shows the fitting of the scaling law for dataset sizes across architectures (fixing dense model size to 0.
(Nexpert./Toppk)
(4,1)
(4.2)
(8,1)
(8,2)
Q1
0.28
0.19
0.29
0.18
Qg
0.29
0.20
0.30
0.20
0.3
0.01
0.008
0.01
0.008
Table 8, Fitted exponents of the scaling law for dataset sizes across architectures.
C.2. More validation of Equation
We first make an observation from Figure 4 that the slope of the fitted lines, that is, the scaling exponent as Equation decreasing with D,. Let us consider the scaling factors A fitted in Figure 4. The scatter plot of A's in Figure 12 leads us to deduce that the following relation:
A DT"
(16)
htttps:/huuginggacce.. wtt.:/iiiiiiiiiiioooiooiioiiiitt wtw.:o/tttittttoaaaasasaaaaaaaaa
18

---

Scaling Laws for Upeycling MoE
Indeed, find that the above equation fits well in the Figure. Substituting Equation 7, we obtain the RHS term of the form D " "D20 indicating that Equation 1 arises not only from first principles as in the main text, but from empirical observations as well.
Dense tokens, D:
0.28
22-
Dense tokens, Di
1.28
Dense takens, Dt Meb
B-45
1.75
12 80
0.60
2.20
2.2
1.c8
6.
20
4.20
L00
2.-
2:3
2.8
L(D2) AD +
2.-
8X15M
8x0.18
8x1B
0.5
2.1
10
2.53
22
Upsyclau Luiuro, D,
Upaycied Lokero, Dj
Ubpyccied boleene, Dy 1DI
1.455
A(D2) x D
1.52
A(D2) DT"
A(D D
1:31
ADD1
8x15M
A(O.3
8x0.18
A(Ds)
8x18
1.470
1440
1:20
1.45
1.450
114
D:
Figure 12. Top: Fitting plots same as those in Figure 4. Bottom: Fitting scaling factor has power-law behavior. The fitted scaling factors are shown fit well with Equation 16.
C.3. Bilinear form
We would also like to note that by taking the logarithm of the first term of Equation 8, we have a log Di + 02 log D2 + ag log D2 log D2, which is bilinear in terms of log Dj and log D2 (that is, linear with each variable separately), indicating that the multiplicative term with interaction is a natural generalization of the single-variable power law.
D. More Results on the Scaling Law for Model Configuration
D.1. Fitting the Scaling Law for Model Configuration
Recall that we wish to understand the cross-entropy loss as a function of sparsity, defined as P P= Ntotal and the number of active parameters, N2. Starting from the power-law ansatz, we require that the loss satisfies
L(P,N2) = Lp(N2) = AN, 91 + E
and
L(P.Nz) LN, AP- 82 E
This is reasonable as P. when fixing N2. is the total number of model parameters, which we expect to satisfy the power-law ansatz. Analogously, N2 corresponds to the number of dense model parameters, which should satisfy the power-law ansatz
as well.
We then consider, as before, both additive and multiplicative functional forms, with and without interaction, satisfying the above requirements, and evaluate them using leave-one-our RMS error:
Multiplicative:
L(P,Ng = AP- + N2) Ba log + E
R AP- 52 log P E
(17)
19

---

Scaling Laws for Upeycling MoE
Additive:
((P,N2) - AP- + F(1 (1++N) N2) (22-222 -2- -2) log
+ E
AP-9: + FN2 B2+ log P +
(18)
We obtain RMS eттоrS of 0.0414, 0.0351, 0.0341 and 0.0322 for multiplicative (with interaction), multiplicative (without interaction), additive (with interaction), and additive (without interaction) cases, respectively. That is, the additional form without interaction functional form provides the best to the data.
D.2. Alternative Visualization of the Scaling Law for Model Configuration
We show the 2D slices (with fixed sparsity) of the model configuration scatter plot as in the left panel of Figure 1, in Figure 13.
E. More Results on the Joint Scaling Law
E.1. Validating Requirement 3
We show in Figure 14 that the upeycled model empirically satisfies power-law scaling with respect to N1, i.e. Requirement 3, repeated below for convenience.
Requirement 3. Dn (Ni) = BI + ED
E.2. Another functional form satisfying Requirements 3 and 4
For convenience purposes, let us repeat Requirement 4:
Requirement 4. li,,, D2, Mi) ADT + BNT* + E, i.e., approaching Chinchilla scaling law.
There is another functional form satisfying Requirement: and 4:
B
LD,1, = D +
D N N82-00 log Dr +
(19)
Fitting the above equation however yields negative exponents, which are, empirically, an 0.10, 02 -0.07, -0.08, 03 -0.008. We reject the hypothesis of this functional form the loss is predicted increase with datasct/model sizes, violating the power-law ansatz well empirical observation.
E.3. Why fitting scaling laws separately?
We provide reasoning why fit three scaling laws separately, instead of fitting an unified scaling law which holds for all three dense, MoE from-scratch, and upcycled training scenarios.
The scaling law for MoE trained from scratch is expected to have parameters different from those for upcycled training (that is, they are not equal to each other at D1 - 0), because upeyeled MoEs are initialized with identical MLP/expert weights from the dense models, while MoEs trained from-scratch have different initial (random) MLP/expert weights. Thus, we expect them to scale differently.
Similarly, we expect the scaling law for dense model has parameters different from those for upeycled training the limit D2 + 0. As be seen from Figure 3, the upeycled training undergoes rewarming phase in the beginning, where the loss increases initially before decreasing. This causes deviation from the dense model scaling law at D2 - O, although we still expect it to correlate with the dense model scaling law, i.e., Requirement 2 should hold based on function preservation and (empirical) observations that rewarming does not completely de-correlate the loss behaviors.
Finally, our preliminary investigations also show that unified scaling law does not fit well. We consequently consider them separately. Moreover, we note that there can be other possibilities of functional forms that we do not explore further.
20

---

Scaling Laws for Upeycling MoE
F. More Implication of the Joint Scaling Law
F.I From-scratch training vs Upcycling
We have shown in the main text that upcycling is only effective when the sunk cost or the model size is small. We further visualize this by plotting the losses for model sizes IB, 7B and 70B, fixing D1 various values, with respect to MoE and D2 for from-scratch and upcycled training respectively in Figure 15.
F.2. More on compute optimality upcycling
As mentioned the main text, while training the Mixtral-like MoE from scratch shown to more efficient in the long run, requires more compute (1.75 times larger than dense training). There is compute-performance trade-off between these stages of training.
We show in Figure 16 that the optimization results in Section 5.2 that training from scratch can always be considered to be more performant.
F.3. Derivation of compute-optinal formula
As in the main text, we want to scale N2, D2 optimally, No Drrt given a FL.PPs budget and fixing D1, while minimizing the loss L, which we write as LD, (D2, N1). This is equivalent to solving the following:
a
Lp, (D2,CC/11.52))
0,
aD
D==Dope
Ln, (C2/10.5N1,
ON1
N₁ N1=
where we have used N2 1.75N, and C2 62. Solving the above equations leads
D2T = G
10.5
C2
Nopt G
10.5
where
A OeffOeeff
1/(aut +3)
G
B3
B
0 ==
Oeff + 8
Ocff
b =
+ :
Aeff AD an
Off = 02
0z log D1
We can henceforth relate Drpt and Nopt via
Dopt G (G/ o/b &
8/op aa/az-aship
and
Nopt = G-1 DG b/a
x (Dore) (a2-ay log Di)18
21

---

Scaling Laws for Upeycling MoE
Sparsity = 1.3
Sparsity
Sparsity
1.05
3.0
31
2.95
230
2.8
29-
2.80
2.8
2.75
2.0
2.70
2.7
0.10
6:10 activee
Sparety = 3.5
Sparsity 37
Sparsity 6
32
2.50
2.05
1000
2.85
2.75
2.70
8.50
0.15 NLactive (8)
6.1c
K_achiv
M_ M_active (8)
Sparsity 6.9
Sparsity T.4
Sparsity 12.2
2.5
2.55
2.55
31
2.50
1033
30
238
2.9
2.75
28
605
0.10
0.15
0.20
0.25
0.30
0.55
0.00
03
02
0.3
0.4
0.5
0.8
0.7
006
cic
0.1
0.20
Nuactive
R,active
M_octive (t)
Figure 13. Scaling behavior with respect to model configuration. Shown are plots of the cross-entropy loss with respect to active parameter N₂ fixing sparsity P.
22

---

Scaling Laws for Upeycling MoE
D1w4:68, 02w4.69
125
Lo, (N1) E = BN
30
No (B)
Figure 14. Upeycled model has power-law behavior with respect to Ni.
Dugs sunk cast
COSE
Upcycled (02-14c<11)
Upcycled (.0=-.0--09)
22
Mlsce SUTI CAN
Uocycles (D1-2.0e+09)
Uscycleo (01-1.42112)
Upcyckel (D1=2.06 112)
Uscycled (D1-1.0e133
Duee sunk cost
Scratch (2#=1.000)
2.0
Suratch
2.0
1.0
Drace
[01-4,7e:00)
8x7B
Decx suitrggg
1.4
8x708
165
1630
1ell
1e12
169
1610
1ell
1e12
1630
2601
3632
Dy/Dre:
Dy/Drue
D2/Dwu
Figure 15. Token budget of from-scratch MoE training and when it catches up with upcycled MoE's performance. We compare loss-versus-token plots of from-scratch and upcycled MoE trainings at various dense training budgets (sunk costs) and model sizes. Upcycling is considered to be efficient only when DMØE sunk cost. We observe that the efficiency of upeycling diminishes with sunk cost and model size.
5.0
Upcycled training
Dense training
4.5
MoE training
4.0
Loss
3.5
3.0
2.5
1016
1017
1018 FLOPS
1019
1020
Figure Compute-optimal training. Upcycled training performs worse even when D1, D2, N₂ are allocated optimally, compared compute-optimal training of dense MoE models.
23


Reviews:
[{"review_id": "7Q3UDl6ZOT", "reviewer": "Reviewer_wa23", "summary": "This paper investigates the scaling laws for upcycling pretrained dense language models (LLMs) into sparse Mixture-of-Experts (MoE) architectures. By conducting extensive experiments with models up to 1B dense and 7B MoE, the authors identified scaling laws which describe the relationship between the cross-entropy loss with dataset size and model configuration. This paper also indicate that upcycling is more efficient than training from scratch under certain conditions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and positive evaluation of our work.\nWe especially appreciate the **recognition of our empirical approach, the soundness of our experimental design, and the validation of our scaling law derivation in Section 4.1**. \nWe also thank the reviewer for acknowledging the practical value of techniques such as the WSD learning rate schedule. \nThe comments regarding the justification of Equation 11 and the desire for more concrete experiments around compute optimality are very helpful, and we address them in detail below.\n# Comment on Equation 11\nWe acknowledge that a derivation and ablation study of Equation 11 were lacking, an oversight on our part. We appreciate the opportunity to address it.\n- During the rebuttal period, we conducted a principled analysis to derive and validate the appropriate functional form similar to what is done in section 4.1.\n- Recall that we wish to understand the cross-entropy loss as a function of sparsity, defined as $P= N_{\\rm total}/N_2$ and the number of active parameters, $N_2$.\n- Starting from the power-law ansatz, we require that the loss satisfies $L(P,N_2)= L_{P}(N_2) = A N_2^{-\\beta_1} + E$ and $L(P,N_2)= L_{N_2}(P) = A P^{-\\beta_2} + E$. This is reasonable as $P$, when fixing $N_2$, is the total number of model parameters, which we expect to satisfy the power-law ansatz. Analogously, $N_2$ corresponds to the number of dense model parameters, which should satisfy the power-law ansatz as well.\n- We then consider, as before, both additive and multiplicative functional forms, with and without interaction, satisfying the above requirements, and evaluate them using leave-one-out RMS error. We obtain \n\n| multiplicative (with interaction)           | multiplicative (without interaction)         | additive (with interaction)         | additive (without interaction)       |\n|------------------|----------------|------------------|----------------|\n| 0.0414 | 0.0351 | 0.0341 | 0.0322 |\n\ni.e., **additional form without interaction** provides the best fit to the data. \nWe will revise the paper to include the corrected form, along with a derivation and ablation comparison of alternative models.\n\n# Comment on more experiments\n- We agree that additional experiments on compute optimality could further strengthen the claims in Section 5.2. However, this would require sweeping over many configurations with comparable FLOP budgets, which is not feasible under our current computational constraints. Such experiments would require multiple times the total GPU budget we already spent (estimated in Appendix A.10), amounting to over 10,000 USD even under conservative assumptions (e.g., 1 USD per GPU hour).\n- Moreover, while compute optimality is important, it is not the primary focus of this work. Our main goal is to understand how upcycling performance depends on data and model sizes. That said, our formulation naturally supports compute analysis, and we use it in Section 5.2 to identify regimes where upcycling becomes less compute-efficient than training from scratch. Note that our findings are robust and extrapolatable: in **Figure 6**, we show that scaling behavior observed in sub-0.5B parameter models can **reliably predict** the performance of larger models (up to 1B parameters).\n- Our approach is also in line with prior work, e.g., Krajewski et al. (2024) derived a data–model size scaling law and use it to predict FLOP-optimal behavior, rather than performing exhaustive FLOP-based sweeps, emphasizing interpretable scaling relationships under realistic compute budgets.\n\nWe will further correct typos and figure presentation as suggested in the revised version.\n\nWe hope these address your concerns. Please let us know if further clarification would be helpful.", "claims_and_evidence": "1.\tIn section 4.1, the derivation of scaling law for dataset sizes is convincing. The authors firstly set some priori requirements for the functional form of the scaling law (Requirement 1 and 2, which are reasonable), and then empirically fit the functional form with experiment results.\n2.\tIn section 4.2, the hypothesized functional form (Equation 11) requires stronger justification. It is not clear why the relationship between the loss and model configuration should follow the form of Equation 11. The paper does not provide theoretical motivation or ablation studies comparing alternative formulations.\n3.\tIn section 5.2, the authors claimed that larger pretrained models require disproportionately more tokens for efficient upcycling, and that upcycling is inefficient relative to from-scratch trainings when considering compute optimality. These claims are supported by the derived scaling law. However, concrete experiments would significantly strengthen these claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate with the problem of understanding upcycling efficiency and scaling laws for MoE models. For example, using WSD LR schedule can reduce computation overhead, and its reliability is verified in the appendix.", "theoretical_claims": "This work adopts an empirical approach, which aligns with the practical approaches of empirical scaling law research. Some  theoretical analysis seem reasonbale.", "experimental_designs_or_analyses": "The experimental designs and analysies are largely sound for the studied scope (models up to 1B dense / 7B MoE).", "relation_to_broader_scientific_literature": "This paper appropriately cites relative works on MoE architecture, scaling law, and upcycling.", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "Strengths:\n1.\tFirst systematic study of upcycling scaling law for MoE models.\n2.\tThis work provides practical insights for MoE model training.\n3.\tCode is available.\nWeaknesses:\n1.\tSome claims may need empirical support. See item 3 in Claims And Evidence.", "other_comments_or_suggestions": "1.\tEquation 12: correct left-hand side to L(D1, D2, N1).\n2.\tFigure 1: replace the 3D plot with 2D slices (fixed sparsity/parameters) for better readability.", "questions_for_authors": "See weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NxG9KgPu3", "reviewer": "Reviewer_j1Qr", "summary": "This work investigates the scaling behavior of upcycling dense LLMs into mixture-of-experts architectures. Through extensive experiments, the authors design and fit scaling laws that describe how language modeling performance depends on dataset size and MoE configuration, including sparsity and number of experts. They find that while upcycling can reduce initial losses and accelerate convergence, its efficiency diminishes as the size of the dense pretrained model and upcycled dataset size increase. Beyond a certain computational budget, from-scratch MoE training becomes more effective than upcycling. The study provides a quantitative framework for evaluating when and how to adopt upcycling and offers guidance on scaling dataset size and model configuration for efficient pretraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We are especially grateful for the **recognition of our experimental design, the quality of the fitted scaling laws, and the clarity of our empirical methodology**. \nWe also appreciate the constructive suggestions. \nLet us clarify some of the questions and comments raised in the following.\n\n# Question on initial parameterization\nWe appreciate the reviewer’s insightful observation on whether modifying initialization can improve performance, and indeed we have studied this and mentioned it in **Appendix A.6**. Let us elaborate on it further:\n- To investigate whether this effect could be mitigated, we experimented with several initialization modifications for the MoE experts. These included (i) adding Gaussian noise to the expert weights of the upcycled model, (ii) partially randomizing the MLP weights, and (iii) applying low-rank approximations to encourage expert divergence. However, across all these approaches, we observed no significant improvement in training dynamics or final performance.\n- These results are consistent with findings from prior work, including Komatsuzaki et al (2022), He et al (2022), and Muennighoff et al (2023), who similarly explored various re-initialization or weight-perturbation strategies in MoE settings, with limited success. Overall, our experiments support the conclusion that the vanilla upcycling strategy (without additional modification) is the most effective approach currently known.\n\n# Comment on Figure 3\nThe reason some training curves begin from intermediate number of tokens (e.g., around 4.1 B tokens) stems from our use of the Warmup-Stable-Decay (WSD) learning rate schedule combined with a checkpoint reuse strategy for efficiency (explained in **Section 3.1**). \n- Specifically, for runs with smaller $D_2$ budgets (e.g., $D_2$ = 4.61B), we initialize from an intermediate checkpoint of a longer run ($D_2$ = 9.23B) and continue pretraining from that point to reach the desired total token count, without needing to retrain from scratch for every token budget.\n- Therefore, the curve that appears to start around 4.1B tokens means that it has already undergone 4.1B tokens of training along the “main branch”, and the red line shown corresponds to additional continued pretraining to reach a total of 4.61B tokens.\n\n# Comment on different text domains\nThank you for the insightful suggestion. We agree that understanding how scaling laws vary across text domains is an important direction. In fact, we already have some results on this in **Appendix B**, where we replicate our experiments on two additional datasets, Japanese and code, to test the generality of our findings. Let us elaborate:\n- We observe that the scaling relations introduced in the main text hold consistently, regardless of dataset.\n- The Japanese dataset has higher validation loss (harder task), while the code dataset results in lower loss, making it a relatively easier task in terms of cross-entropy loss.\n- Interestingly, the Japanese dataset is harder to saturate with increasing $D_1$, meaning upcycled training remains effective. In contrast, the code dataset saturates more quickly, making upcycling less beneficial.\n\nThis is a promising evidence that the core scaling behavior we identify is robust across diverse domains.\nHowever, the more fine-grained question of how mixtures of pretraining data impact respective downstream performances is an orthogonal direction.\n While interesting, it is beyond the scope of our current work.\nFor readers interested in downstream impacts, we do include results in **Appendix A.9 and Table 7**, where we evaluate downstream performance on various tasks for models trained with SlimPajama.\n\n# On analyzing the threshold\n- In **Section 5.1**, we define the threshold $D^*$ as the token count at which training a model from scratch matches the performance of an upcycled MoE with the same total token budget using derived scaling laws with fixed model configuration. We solve this equation numerically and also provide an analytic approximation (Equation 2) to aid practical use.\n- We also note in the text that covering all possible configurations and settings would require **exponentially more compute**, which is infeasible in academic environments. As such, we focus on a widely used MoE configuration (Mixtral) to make the analysis tractable.\n- Finally, we offer guidance on how to apply this threshold in practice, highlighting the need to balance model size, compute, and token budgets in the beginning of **Section 5 and Section 7**. Once the configuration is set, one can run the scaling law experiments and use the procedure mentioned above to get the threshold.\nWe will revise the text to make these points clearer.\n\nWe hope these addresses your concern. Please let us know if further clarification would be helpful.", "claims_and_evidence": "The claims in this work, specifically the empirically determined scaling laws, are backed up by good experimental evidence.", "methods_and_evaluation_criteria": "The use of a Llama-like architecture makes sense for evaluating scaling laws. As does the use of SlimPajama, a dataset composed of a number of commonly used subdomains.", "theoretical_claims": "I’m not an expert in scaling laws, but the functional forms of the scaling laws they fit for both dense, and MoE models makes sense at a high level.", "experimental_designs_or_analyses": "I think the experimental design (train dense models at various sizes with varying token budgets, then upcycle to MoEs with varying token budget and various number of experts and sparsity) is very appropriate.", "relation_to_broader_scientific_literature": "The experiments in this work are interesting, but also very narrow. The authors implement only the most basic form of upcycling. If the authors explored other methods of initializing the upcycled MoE, this work could be of interest to a wider community", "essential_references_not_discussed": "None that I am familiar with.", "other_strengths_and_weaknesses": "Strengths:\n- This work provides extensive experimentation and demonstrates the quality of their fitted scaling laws for multiple settings\n- The exploration of multiple functional forms makes the results more sound\n\nWeaknesses:\n- While this work discusses a threshold at which it is better to train an MoE from scratch, there is no clear guidance on how to predict or calculate this threshold in practice for new settings. This area could benefit from more exploration and a deeper analysis", "other_comments_or_suggestions": "- I think Figure 3 can be made more clear. At the moment, it appears that only 2 of the lines (D2 = 9.23) have the full training curve. For example, presumably, the red line (D1=9.23, D2=4.61) has been trained for 4.61 billion tokens after upcycling, so why does the curve only start around 4.1 billion?\n- It would be very interesting to see how the scaling laws of different text domains are similar/different. Since the SlimPajama dataset is composed of a number of domains, this should theoretically be possible with your dataset. In particular, I think it would be very interesting to see how the mixture of pre-training data impacts the downstream performance on each individual domain", "questions_for_authors": "- This finding “Increasing D1 (sunk cost) reduces the initial losses of the upcycled MoE but results in slower training progress with D2 (upcycled MoE tokens).” sounds like the initial parameterization of the upcycled MoE is already in a local minima. Are there possible techniques that you may consider to account for this finding? For example, would adjusting the initialization of each expert in the upcycled model allow the experts to diverge quicker and improve the scaling law?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eh09KM1bF2", "reviewer": "Reviewer_byMA", "summary": "This paper studies the computationally efficient training of large language models (LLMs) through upcycling where smaller pretrained dense models are utilized as initial checkpoints to train larger Mixture-of-Experts (MoE) models. Given that training large-scale language models from scratch demands considerable computational resources, the authors explore empirical scaling laws related to upcycling dense checkpoints into MoE models which remains relatively unexplored. Through experiments involving models of varying sizes (up to 7B parameters) and datasets containing hundreds of billions of tokens, the authors establish empirical scaling laws that characterize how model performance (measured by cross-entropy loss) depends on factors such as dataset size, number of tokens used in dense pretraining (D1), tokens used in upcycled MoE training (D2), model size, and model sparsity (ratio of total parameters to actively-used parameters). More specifically, a novel multiplicative scaling law is proposed, which describes the MoE model’s cross-entropy loss as a function of both the dense pretrained tokens (D1) and additional MoE training tokens (D2). This scaling law incorporates a previously unidentified interaction term between the dense and MoE datasets, highlighting diminishing efficiency of upcycling as the initial dense training budget (the sunk cost) increases. The authors also identify that the performance of upcycled MoE models consistently improves with increased sparsity and number of active parameters. The paper offers explicit guidance, suggesting scenarios under which upcycling is beneficial or not. For instance, they derive a threshold token count (D*) as a function of dense model size, providing a practical rule to decide whether upcycling is computationally advantageous.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We especially appreciate the **recognition of its novelty and relevance, as well as the clarity of our presentation**.\nWe also appreciate the constructive suggestions. \nLet us clarify some of the questions and comments raised in the following.\n# Question on Relevance to transfer learning and other scaling laws\nWe have briefly discussed connections to transfer learning (and cited relevant two-stage training methods like model growth and pretraining/SFT) in **Section 7**, but we agree that this deserves a more in-depth discussion, which we describe below.\n- Prior work (e.g., Mikami et al 2022) on transfer learning has proposed scaling laws using multiplicative or additive forms involving $D_1$ (pretraining data) and $D_2$ (fine-tuning data).\n- However, as far as we know, there has been no attempt to incorporate an interaction term such as ours, namely $D_1\\log D_2$. We believe this interaction is **novel and meaningful**: it is both empirically validated and theoretically motivated by our derivation in **Section 4.1** (see also **Appendix C.3**).\n- While our focus is on upcycling MoE models, we expect the interaction term to also be applicable in transfer learning settings. For example, transfer learning literature has observed *ossification* effects (Hernandez et al 2021), where models pretrained on large datasets can become resistant to adaptation during fine-tuning. \n- The diminishing returns we observe from increasing $D_1$ align with this notion. Our interaction term $D_1 \\log D_2$ captures precisely such diminishing gains and may offer a useful framework for modeling ossification quantitatively in transfer learning scaling laws (and other two-stage training methods).\n  \nWe will revise the discussion to emphasize this connection.\n\n# Comment on Limited Generalizability to Larger Models\n- We acknowledge the concern about generalizability to larger models; however, we note that this limitation is common across empirical studies of scaling laws, including highly influential ones.\n- Training and evaluating models at the 70B+ scale requires orders of magnitude more computational resources which is currently infeasible in an academic setting. We estimate this to require 5,000 times more FLOPs (70x larger model and 70x more tokens) than what we have run (see Appendix A.10), amounting to an additional cost of 25 million USD (even assuming 1 USD per GPU hour) .\n- Despite this, we provide empirical evidence supporting the robustness of our findings: in **Figure 6**, we show that scaling behavior observed in sub-0.5B parameter models can **reliably predict** the performance of larger models (up to 1B parameters).\n- This suggests that the scaling laws identified are stable across model sizes, and we expect the trends to extend to even larger models, consistent with patterns observed in prior work (e.g., Chinchilla paper). \n\n# Comment on theoretical explanation\n- While we indeed have not included formal theoretical proofs, we emphasize that the proposed scaling law is not purely empirical. Its functional form is guided by well-motivated principles, as detailed in **Section 4.1**, and was **noted by two other Reviewers to be a reasonable theoretical contribution to empirical scaling law studies**. \n- Our work also follows the tradition of impactful scaling law work (e.g., OpenAI, Chinchilla paper) that focused on empirical observations and practical utility, without extensive theoretical justification.\n- That being said, we agree that theoretical understanding is important, and although it is not the primary focus of this paper, we provide related references in **Section 7**.\n\n# Comment on Limitation to upcycled models\nWe respectfully disagree with the characterization that our work is limited in scope due to its focus on upcycled MoE models.\n- In practice, both MoE architectures and upcycling strategies have become central components of modern LLM development. Recent state-of-the-art models, including DeepSeek, Qwen, Skywork-MoE, and Mixtral, adopt MoE architectures while also leveraging dense-to-sparse upcycling. This growing trend highlights that upcycling is not just a research curiosity but a practical and widely adopted technique in the current LLM landscape. Our findings are therefore both timely and relevant, offering insight into how/when upcycling can be efficient.\n- Although our experiments specifically focus on upcycling into MoE models, the core insights, such as the interaction between dense and upcycled training budgets, are useful for formulating two-stage training regimes more broadly, including transfer learning as mentioned above, and potentially model growth and pretraining-SFT framework as cited in the main text.\n\nWe hope these address your concern. Please let us know if further clarification would be helpful.", "claims_and_evidence": "Yes, overall, the claims made in the submission are supported by experimental evidence. Empirical scaling laws for the interaction between dense pretraining tokens (D1_1​), upcycled MoE tokens (D_2​), and model configurations are convincingly supported by experimental results.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense.", "theoretical_claims": "NA, the paper does not provide explicit theoretical proofs for its claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and valid.", "relation_to_broader_scientific_literature": "The scaling laws derived from this work can impact mixture of experts (MOE) models in diverse domains. However, it is likely limited to only upcycled MOE models.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n+ The paper addresses a relatively under-explored topic: the scaling laws specific to \"upcycling\" dense pretrained models into Mixture-of-Experts (MoE) architectures.\n+ The writing is clear, well-organized and easy to follow.\n+ The provided scaling laws and guidelines offer practical insights for training large models efficiently, which is highly relevant to the community given the computational expense of training language models.\n\nWeaknesses: \n- Limited Generalizability to Larger Models: The experiments are limited to models of size up to 7B parameters, it remains unclear if the results extend to significantly larger MoE models (e.g., 70B+ parameters).\n- Lack of Theoretical Explanation\n- Scope seems limited to only upcycled MoE models.", "other_comments_or_suggestions": "Discussing the similarities and differences between the scaling laws in these different settings can make the contribution more impactful and of interest to a wider audience.\n\nFigure 1 (left): it is not easy to see the cross-entropy values, it might be clearer to include a color scale for these values. \nWhile scaling laws for upcycling MOE is novelty, there are previous work on related application such as transfer learning.", "questions_for_authors": "How do the scaling laws here compare with other scaling laws in related setting such as transfer learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: This paper studies scaling laws for upcycling dense models into MoE models. With extensive experiments, the paper present scaling laws between dataset size and model configuration. Based on the observations, the paper propose that from scratch training could be more efficient in a certain condition.

Overall, the reviewers acknowledge the soundness of the experimental design to find the empirical scaling laws. The reviewers also appreciate the insights from the empirical experiments. On the other hand, the reviewers questioned about weak theoretical support and limited generalizability with limited model sizes. The authors provided more detailed theoretical analyses in the rebuttal to address those issues. Also, the authors clarified the limitation of compute resource to further scale up the experiments.

All three reviewers gave weak accepts in the review process and confirmed the rating after the rebuttal phase. Therefore, I would give a weak accept as the paper provides meaningful insights to the community.
Decision: Accept (poster)