Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Authors: Taishi Nakamura
Takuya Akiba
Kazuki Fujii
Yusuke Oda
Rio Yokota
Jun Suzuki
Venue: ICLR 2025 Poster (2025)
Keywords: mixture of experts
large language models
continual pre-training
Primary area: foundation or frontier models, including LLMs

Abstract: The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to a dense model of equivalent capacity. Upcycling is an approach that initializes and trains an MoE model using a pre-trained dense model. While upcycling leads to initial performance gains, the training progresses slower than when trained from scratch, leading to suboptimal performance in the long term. We propose Drop-Upcycling - a method that effectively addresses this problem. Drop-Upcycling combines two seemingly contradictory approaches: utilizing the knowledge of pre-trained dense models while statistically re-initializing some parts of the weights. This approach strategically promotes expert specialization, significantly enhancing the MoE model's efficiency in knowledge acquisition. 
Extensive large-scale experiments demonstrate that Drop-Upcycling significantly outperforms previous MoE construction methods in the long term, specifically when training on hundreds of billions of tokens or more.
As a result, our MoE model with 5.9B active parameters achieves comparable performance to a 13B dense model in the same model family, while requiring approximately 1/4 of the training FLOPs.
All experimental resources, including source code, training data, model checkpoints and logs, are publicly available to promote reproducibility and future research on MoE.

Published as a conference paper at ICLR 2025
DROP-UPCYCLING: TRAINING SPARSE MIXTURE OF EXPERTS WITH PARTIAL RE-INITIALIZATION
Taishi Nakamura 1,2.3, Takuya Akiba", Kazuki Fujii', Yusuke Oda?,
Rio Yokota 1.3 Jun $.2uki4.5.3 Institute of Science Tokyo, "Skana AI, 'NII LLMC, "Tohoku University, 'RIKEN taishi@rio scrc. wir.isctt aar.isct.ac. jp, jun. wwwwkittoooou...ooomo
ABSTRACT
The Mixture of Experts (MoE) architecture reduces the training and inference cost significantly compared to dense model of equivalent capacity. Upeycling is approach that initializes and trains an MoE model using a pre-trained dense model. While upcycling leads initial performance gains, the training progresses slower than when trained from scratch, leading to suboptimal performance in the long term. We propose Drop- Spcyelin a method that effectively addresses this prob- lem. Prop-Uppyccle ing the knowledge of pre-trained dense models while statistically re-initializing some parts of the weights. This approach strategically promotes expert specializa- tion, significantly enhancing the MoE model's efficiency in knowledge acquisi- tion. Extensive large-scale experiments demonstrate that Drop- Upeycling signifi- cantly outperforms previous MoF. construction methods in the long term, specif- ically when training hundreds of billions of tokens more. As result, MoE model with 5.9B active parameters achieves comparable performance to a 13B dense model in the same model family, while requiring approximately 1/4 of the training FLOPs. All experimental resources, including source code, training data, model checkpoints and logs, are publicly available promote reproducibil- ity and future research on MoE.
combines two seemingly contradictory approaches: utiliz-
Weights
Data
Sttasets/llmmjp-coppppss
Code
Logs
INTRODUCTION
Large-scale language models (LLMs) have achieved remarkable results across various natural lan- guage processing applications (Brown et al. 2020 Wei al. [022 Ouyang et 2022 OpenAl 2024). This success largely depends on scaling the number of model parameters, the amount of train- ing data, and computational resources (Kaplan et al. 2020 Hoffmann al., 2022). which leads to substantial training and inference costs of LLMs. Building and deploying high- -egnhprrrorancee models also require enormous resources, posing significant barrier for many researchers and practitioners.
The Mixture of Experts (MoF) architecture has emerged a promising approach to address the escalating resource demands of L.LMs. MoE introduces multiple experts into some parts of the network, but only subset is activated any given time, allowing the model to achieve superior performance with reduced training and inference costs (Shazeer et al. 2017 Lepikhin et al. 2021 Fedus al. 2021). In fact, cutting-edge industry models like Gemini 1.5 (Team et al. 2024) and GPT-4 (based on unofficial reports) (OpenAl 2024) have adopted MoE, suggesting its effectiveness.
We refer to transformer-baste based LLMs without MoE dense models and those incorporating MoE as MoE models. Upeycling (Komatsuzaki et al., 2023) an approach that initializes and trains an MoE model using pre-trained dense model, which aims transfer learned knowledge for better

---

Published as a conference paper at ICLR 2025
Original dense model
W
W
Expert
replication
gtt;
S
(3) Diversity
re re-initiaiization
Add
W
gite
WHST
dows
W
W gata
COMBs
W
down
Figure I: Overview the Drop-Upcycling method. The key difference from the naive Upcycling is Diversity re- re-initialization, introduced in Section
initial performance. However, naive Upcycling copies the feedforward network (FFN) layers during initialization, which makes difficult to achieve expert specialization. This disadvantage prevents effective utilization of the MoE models' full capacity, resulting slower convergence over long training periods. Thus, there exists a trade-off between the short-term cost savings from knowledge transfer and the long-term convergence efficiency through expert specialization.
In this paper, we propose Drop-Upcycling briefly illustrated in Figure Drop-Upeyclin works by selectively re-initializing the parameters of the expert FFNs when expanding dense model into an MoE model. The method is carefully de- signed to promote expert specialization while preserving the knowledge pre-trained dense models. Specifically, common indices are randomly sampled along the intermediate dimension of the FFNs, and the weights are dropped either column-wiss wise or row-wise, depending on the weight matrix types. The dropped parameters then re-initialized using the statisties of those weights.
method that effectively addresses this trade-off, as
Extensive large-scale experiments demonstrate that Drop-Upcycling nearly resolves the trade-off between the two aforementioned challenges and significantly outperforms previous MoE model construction methods such as training from scratch and naïve Upcycling. By leveraging pre-trained dense models, -peycllngg can start training from better initial state than training from scratch, reducing training costs. On the other hand, Drop-Upeycling avoids the convergence slowdowns ob- served with naive Upcycling. Specifically, in our extensive long-term training experiments, Drop- Upeycling maintained a learning curve slope similar to that of training from scratch, consistently staying ahead. This success is attributed to effective expert specialization. As result, we con- structed an MoE model with 5.9B active parameters that performs on par with a 13B dense model from the same model family, while requiring only approximately 1/4 of the training FLOPS.
This research is fully open, transparent, and accessible to all. With over 200,000 GPU hours of ex- perimental results, conducted on NVIDIA H100 GPUs, all training data, source code, configuration files, model checkpoints, and training logs used in this study are publicly available. By providing this comprehensive resource, we aim to promote further advancements in this line of research.
Our technical contributions are summarized as follows:
We propose Drop-Upeycling, novel method for constructing MoE models that effectively balance knowledge transfer and expert specialization by selectively re-initializing parame- of expert FFNs when expanding dense model into an MoE model.
Extensive large-scale experiments demonstrate that Drop-Upeycling consistently outper- forms previous MoE construction methods in long-term training scenarios.
All aspects of this research publicly available. This includes the MoE model with 5.9B active parameters that performs comparably to 13B dense model in the same model family while requiring only about 1/4 of the training FLOOP.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 MIXTURE OF EXPERTS
The concept of Mixture of Experts (MoE) was introduced about three decades ago (Jacobs et al,
1991 Jordan & Jacobs 1994). Since then, the idea of using sparsely-gated MoE building block within neural network layers (Eigen et al.] 2014; Shazeer et al. (017) has evolved and has been incorporated into wranffommr-buseee language models (Lepikhin et 2021|22211000 Fedus et al. 22021). For a detailed overview of MoE, please refer to recent survey papers (Cai et al. (024). Sparsely-gated MoE is currently the most common approach for building large-scale sparsely- aparsely-activated models. In this paper, we focus on sparsely-gates MoE (also referred to as sparse MoE or sparsely-activare MoE), and unless otherwise specified, the term MoE refers
There are various designs of MoE layers and ways to integrate them into transformer-basee LLMs. For example, in addition the standard token-centric routing, expert-centric routing has also been proposed (Zhou et al. 2022). To incorporate common knowledge, it has been suggested to introduce shared experts that are always activated (Dai et al.) (024). To simplify the discussion, we assume the most standard top-k token choice routing as the MoE layer and decoder-only transformer-basee LLM that uses MoE layers only in the FFNs as the MoE model. These are common design choices for recent MoE-based LLMs. such as Mixtral (Jing al.] (2024). Skywork-MoE (Wei et al.] (022)) Phi-3,5-MoE (Abdin et al. (024). and Grok- Specifically, these models use experts (Mixtral and Grok-1) or 16 experts (Skywork and Phi-3.5-MoE), with the top-2 experts being activated per input token. Our experiments also use top-2 routing with 8 experts per layer, as this setup aligns with those practical configurations. These facts indicate that Drop-Upeycling can be applied to most variations of MoE models. See Section 3. for technical details of MoE.
2.2 MOE MODEL INITIALIZATION
As with conventional neural networks, MoE models can be initialized randomly and trained from scratch. However, reduce training costs, leveraging existing pre-trained dense models has become a standard approach. Below, we introduce a few methods for achieving this.
Upcycling (Komatsuzaki al. 2023) leverages the weights of a pre-trained dense model for initial- izing an MoE model by initializing the experts in the MoE layer as replicas of the FFN layers in the dense model. The main advantage of Upcycling that it boosts the model's initial performance. However, as our experiments show, MoE models initialized with Upcycling tend to have much slower convergence, leading to suboptimal performance when trained for longer durations.
Franch-Train-MMi (BTX) (Sukhbaatar et al. 2024) is technique where pre-trained dense model replicated and fine-tuned on different datasets to produce multiple distinct expert dense models. These experts are then integrated into an MoF model, followed by additional training to optimize the routers. While this method appears to ensure expert specialization by design. Jiang et al. (2024) has highlighted that the diversity achieved in this way differs from that required for MoE layer experts, leading to suboptimal performance as a result. Our experiments also show that BTX suffers from suboptimal convergence similar to those observed in Upeycling.
Concurrent with our work, the Qwen2 technical report (Yang et al. 2024) briefly suggests the use of methodology possibly related to Drop- Upeyclingg in training Qwen2-MoF. Due the report's brevity and ambiguity, it is unclear their method exactly matches ours. Our paper offers a valuable technical contribution even if the methods are similar. The potential application of Drop-Upcycling in an advanced, indestry-developed model like Qwen2-MoE that underscores the importance of fur- ther open investigation into this approach. We acknowledge the Qwen2 authors for sharing insights through their technical report.
3 METHOD
In this section, explain the Drop-Upeycling method. Drop-Upcycling initializes MoE model by utilizing a traitdd dense model and consists of three steps:
https://x.ai/bllgg//e/
3

---

can be applied to Published as a conference paper at ICLR 2025
1. Expert Replication:
The weights of the dense model are copied to create the MoE model. All layers, except for the FFN layers, are copied directly from the dense model. The FFN layers replaced with MoE layers, and the original FFN weights are copied to all experts within these MoE. layers.
2. Diversity Re-initialization: In each MoE layer, a subset of the expert parameters is ran- domly selected and re- re-initialized using the original statistical information. This promotes diversity among the experts while partially retaining the knowledge of the original model, which facilitates expert specialization during subsequent training.
3. Continued Training: After initialization, the MoE model is trained using the standard next-token prediction loss. Optionally, load-balancing loss, commonly applied in MoE training, can also be incorporated.
In the following, we explain the expert initialization and diversity injection processes.
3.1 SWIGLU AND MOE LAYERS
We provide brief overview of the Moß architecture. First, we review the feedforward network (FFN) layer transformers. The SwiGLU activation function (Shazeer) (020),, now standard in state-of-the-art LLMs like LLaMA (Touvron et al.) 2023) and Mixtral (Jiang et al.] (024). will be used for explanation here. However, it should noted that transformers with any activation function. The FFN layer with SwiGLU is defined as follows:
SwiGLU(x) (Swish(x" W gate x W up W down
Here, x Rd represents the input vector and @ denotes the Hadamard product. Each FFN layer contains the following three weight matrices: gate, Wup E Rdsx xd,, and down E RRdxxd, The dimensions da and d are referred to as the hidden size and intermediate size, respectively.
When MoE is introduced into transformer, each FFN layer replaced with an MoE layer, while the rest of the architecture remains unchanged. Let assume we use experts and Top-k gating, An MoE layer comprises router and n expert FFNs, The router has a weight matrix W rowter E Rd xn The i-th expert FFN is denoted as SwiGLU (x), which, like a standard FFN layer, consists of three weight matrices. These weights denoted w(() w(f) and (r() The output y of the MoE layer is computed as follows:
gate
down
y Σ g(x); (x)
where g(x); is the i-th element of the output g(x E IR" of the Top-k routing function, defined as:
g(x) Softmax(Top-k(xx W Wrouter)).
(3)
Since h < is typically the standard setting, only the top-k selected experts out of are computed. Therefore, the MoE layer is sparsely activated, meaning that only subset of the parameters is involved in the computation. The number of parameters engaged in the computation for given input is referred to as the active parameters of the MoE model. This value widely used as an approximation for the computational cost it correlates well with the cost of both training and inference. For non-MoE models, the total number of parameters corresponds to the active parameters as all parameters are involved in every computation.
3.2 EXPERT REPLICATION
Following (Komatsuzaki et al. 2023). 3, we first construct a Transformer with MoE layers by repli- cating the weights from a pre-trained Transformer with standard FFN layers. As explained earlier, the architecture remains identical except the FFN layers, so we simply copy the weights of all non- FFN components. Each FFN layer needs to be replaced with an MoE layer, and the new MoE layers are constructed as follows: The router weights W Wreater are initialized randomly. For the
experts, the weights from the original FFN are copied, such that ( ( down
W gate: w Wup and
W down-
4

---

Published as a conference paper at ICLR 2025
Is
R
W
H, o²
N p
W
Partial re-initialization
1 I Is
Maintained
Initialization of expert weights. Columns (rows) are selected according a set of randomly selected indices of the intermediate layer S, then all elements of them are re-initialized with the normal distribution. Other columns (rows) are maintained.
Figure 2:
Drop-Upeycling can also be applied to fine-grained experts and shared experts See Appendix for details.
Dai et al. 2024
3.2.1 DIVERSITY RE-INITIALIZATIN
Diversity re-initialization is the key step in Drop-Upcycling. This process is carefully designed to balance between knowledge retention and expert diversification. In particular, it is crucial to drop original weights along the intermediate dimension of the FFN layer based on shared indices across all three weight matrices. Specifically, the following operation is applied to every expert FFN in every MoE layer.
Step 1: Column-wise Sampling. We sample indices from the set of integers from to interme- diate size d,, namely, Id, 2.. .dy, to create a set of partial indices S. A hyperparameter r (0 < < 1) controls the intensity re-initialization, determining the proportion r used for sampling. That is, S S Id, and |S| |rdy]-
Step 2: Statistics Calculation. We calculate the mean and standard deviation of the matrices of the weights corresponding to the selected indices S. Specifically, we compute the mean and variance (Mup Tup). (Mgate o Fgaee). ). and (Jowwn: doowa) from the values obtained only from the non-zero columns of Is in the products Is @ W gates Is @ W up. and Is @ WT down. respectively, where Is is the indicator matrix whose values are in the i-th column for i E and otherwise.
Step 3: Partial Re-Initialization. Finally, using the calculated statistics, we perform partial re- initialization of the three weight matrices W gate Wmp. and W down, obtaining W gate. W Wup- up- and W down- For the selected indices, the weights are dropped and re-initialized randomly, while for the unselected indices, the original weights are retained.
Let Raype be matrix whose values are sampled from the W(Hoype (diy)) distribution, where type is one of the gate, up, or down, i.e., type {gate, up, down]. We then obtain W type by using the following equation:
type Is @ Raype (1 Is)
④ Wtype:
type:
(4)
where we consider that the matrices, W type Raypes W type are all transposed if type down.
Figure 2 iillustrates how we generate a single expert weight matrix from the original dense weights.
3.2.2 THEORETICAL CHARACTERISTICC
Applying the re-initialization strategy explained above, the initial MoB model obtained by Drop- Upcycling has the following characteristics:
1. Parameter sharing among experts: since each expert retains the original representations with a ratio (1-r), with routing where & experts are selected, approximately (1 (1-r)* of representations are preserved.
5

---

Published as a conference paper at ICLR 2025
2. Characteristics of initial feedforward layers: Consider the output of an MoE layer with parameter re- re-initialization ratio>> r:
y FFN common (x) + Σ gx), FeFNetaiee,,
retained; (x) FFFNoommonn common (x) + FFN diverse, (x)] (5)
where FFN common represents the output from parameters that are common to all selected A experts (the proportion of such parameters is approximately (1 - r)* due to each expert independently preserving ratio - r) of original parameters), FFN reNteetieed; is expert i's output using uniquely retained original parameters (ratio (1 r)), and FFNdiverse: is the out- put using reinitialized parameters (ratio r). The estimation error in the number of common
parameters has magnitude O A detailed derivation is provided in Appendix C.5
4 EXPERIMENTAL SETUP
We conducted experiments to demonstrate the effectiveness of Drop- Uppyclingg described in Sec- tion 3 To clarify our model configurations, we introduce a notation where, for example, "8x152M" denotes an MoE model with eight experts and whose base dense model size is 152M.
We selected the Llama Touvron et al. 2023) and Mixtral (Jiang et al. 2024) architectures for dense and MoE models, respectively, for our experiments. We employed experts and the dropless (Gale et al. 2023) token choice top-2 routing (Shazeer et al. 2017) for the MoE. Detailed descriptions of the model configurations are provided Appendix A.3
We evaluated four different methods to build MoE models, namely, training from scratch, naive Upcycling (Komatsuzaki et al.] (023). Random Noise Upcycling (Komatsuzaki et al. 2023) and Branch-Train-Min (Sukhbaatar et al. (0244) to compare the performance with Drop- Upcycling. Moreover, we also evaluated dense models to provide a reference of the typical performance of LLMs in our configuration and illustrate the performance gains of MoE models. We initialized all parameters of dense models using a Gaussian distribution A (0, 0.02). The dense models are also used as the seed models of MoE models, except when train MoE models from scratch. When training MoE models from scratch, we used the same initialization method as the dense models, that is, (O0, 0.02). In Random Noise Upeycling, Drawing from (Muennighoff et al. 2024). initialize by copying the dense model parameters and then add Gaussian noise ((, 0.02) to 50% of the weights in each FFN layer. In Branch-Train-MMix
first obtained three distinct expert dense models by further training a seed dense model with 100B extra tokens of either Japanese, English, or code. Then, we used the four dense models (the seed dense model and three expert dense models) to initialize the parameters of an MoE model. Specifically, we averaged all parameters in the four dense models except the FFN layers and duplicated the FFN layers in each model twice to build eight MoE experts. Note that this method invol extra training steps with 300B more tokens compared to the other MoE construction methods.
Unless otherwise stated, dense models were trained on IT tokens, and MoE models were trained on 500B tokens. Our training data was obtained from publicly available data. We describe the detailed statistics of the training datasets in Appendix B. We followed the typical training configurations used Llama to train dense models and Mixtral for MoE models. Details of the
used are described in Appendix A.4 Moreover, the implementation and the computational envi- ronment used in our experiments are described in Appendix [A.2]
We conducted comprehensive evaluation using wide range of tasks in Japanese and English. We used 12 evaluation datasets that can be categorized into seven types. The details of the evaluation datasets and metrics are described in Appendix B.2
RESULTS AND DISCUSSION
In this section, we address the following questions through experiments: Is Drop-Upeycling superior to existing MoE construction methods, and does Drop-Upeycla resolve the issue of slower con- vergence? (Section Does it perform well even in large-scale settings? (Section What is the
impact of the re-initialization ratio r? (Section 5.3) How are the experts specialized? (Section 5.4

---

Published as a conference paper at ICLR 2025
8x152M Trakning Loss (Lower is better)
8x1.5B Training Loss (Lower better)
8x3.78 Training Loss (Lower better)
2,50
1.97
2.12
Entropy
Cross Entropy
Cross Entroyy
2.88
1.83
1.38
400
500
200
1.31
100
230
300
400
Pracessed Takens (Bilions)
Processed Takons (Bilions)
Processed Takens (Bilions)
8x1 52M Downstream Task Score (Higher is botter)
8x1.5B Downstream Task Score
8x3.7B Downstream Task Score
(Higher is better)
(Higher better)
41
45
Average Score
Average Score
33
Average
37
29-
35-
100
2000
360
400
500
25
100
200
300
400
500
29
100
200
300
400
500
Processed Tokens (Bilions)
Processed Takens (Bilions)
Processed Tokens (Bilions)
From Scraich
nsive Upsyeling
Drop-Upeyccing (rm 0.5)
Branch-TrammXX
Random Upcycling (r-
Drop-Upeycing 1.0)
Figure 3: Comparison of learning curves for different MoE construction methods. The top and bottom rows illustrate the changes in training loss and downstream task scores during training, respectively, In both metrics, the proposed method, Drop-Upcycling with r = 0.5, achieves the best performance, gaining initial knowledge transfer while avoiding convergence slowdown.
5.1 METHOD COMPARISON
First, we compare Drop-Upeycling with existing methods using small (8x152M) to medium (8x1.5B) scale settings. The left two columns of Figure FigureBillustrace the learning curves under these settings. The top and bottom rows illustrate the changes in training loss and downstream task scores during training, respectively. Note that in LLM pretraining, training loss serves reliable per- formance indicator since the risk overfitting is low. The performance on downstream tasks is represented by the average score across 12 tasks, which is commonly used the overall evaluation metric. A detailed breakdown will be discussed later in conjunction with Table
Figure shows that Drop-Upeycling at 0.5 (green) is significantly more efficient compared to other methods. The top row shows the training loss, while the bottom row displays the evaluation scores using downstream tasks. In both metrics and for both model sizes, Drop- Prop-Upcycling becomes the clear winner after some training. Notably, the slope of the learning curve, which indicates con- vergence rate, is superior. Furthermore, it can be observed that the slope of the learning curve is consistent with the case of training from scratch, suggesting that Prop-Uppycche resolves the cru- cial challenge of balancing knowledge transfer and expert specialization Upeycling. For further analysis on expert specialization, see Section 5.4
Among existing methods, naïve Upcycling exhibited the slowest loss reduction rate and improve- ment in task scores. Branch- Branch-Trran--ii Train-Mix. which starts MoE training after each expert has been trained for 100B steps on different domains such as Japanese, English, and code, initially shows an advan- tage over naïve Upcycling due to this favorable initialization. However, its long-term learning pace on par with naive Upcycling, and it is ultimately overtaken by study, we evaluated setting 1.0 in Drop- Prop-upcylling, in addition to the standard configuration involves random initialization of all FFNs while reusing weights for embeddings and self-attention layers. This configuration might seem inefficient at first glance. Nevertheless,
As an ablation 0.5. This

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of evaluation results between models with different initialization. Train- ing from scratch (FS), Seaach-Train--e (BTX), naive Upeycling (NU), Random Noise Upcycling (RNU) and Prop-Upcyclingg (DU) are compared. BTX requires additional 300B tokens to obtain specialized dense models before MoE construction. Bold letters indicate the highest score within each model size.
Model
Truining
Individual Scores
tecture
Archl-
MoE
Init
Tokens
(x10"))
JEM HQA
NILC
JSQ
Sum
E-AJ
WMT
J-EE
QA
TQA
HS
SQ v2
EN
BBH
Avg
Dense 152M
MoE 8 8x15MM:
Dense
1,0088
1.59
7.9
10.6
0.5
28.6
2.0
60.6
11.5
13.3
MoE
FS
500B
0.91
25.2
13.6
19.4
1.8
0.9
0.4
16.5
2.6
31.2
12.9
64.4
10.7
16.6
MoB
800B"
1.39
26.6
2.7
32.5
5.3
65.0
15.9
18.5
McE
NU
500B
0.91
28.2
16.2
24.4
3.5
3.0
1.1
18.2
5.8
31.9
4.5
63.5
14.7
17.9
(+=0.5)
29.4
2.3
McE
DU (+:0.5)
500B
0.91
32.2
18.0
30.6
3.7
4.7
2.3
16.8
6.1
32.5
6.2
64.2
19.1
19.7
(r=1.))
Dense 1,5B
MoE Ix1.B:
Dense
1,000B
11.76
49.6
42.5
48.1
11.3
16.8
8.5
22.2
23.8
42.9
16.2
82.5
25.1
32.5
McE
500B
9.05
48.3
45.4
59.1
7.5
16.6
6.9
26.4
31.5
47.3
15.0
83.7
25.9
34,5
MoE
BTX
800B
12.58
44.3
51.8
69.4
11.9
22.4
12.5
27.8
39.2
49.7
18.7
86.4
28.9
38.6
MoF:
NLI
500R
9.05
50.4
61.7
12.4
10.5
26.8
36.2
47.7
19.0
85.0
27.2
37.4
MoE
RNU (r=0.5)
500B
9.05
53.6
50.5
71.2
12.3
22.3
11.7
26.4
40.0
49.9
19.1
84,9
27.5
39.1
MoF:
DU (55)
500R
9.05
51.1
52.3
72.5
13.7
22.5
12.5
30,6
41.3
50.4
21.2
86.2
29.1
40.3
MoE
DU (r=1.0)
500B
9.05
52.1
50.9
68.8
12.3
21.9
12.4
25.0
39.1
49.7
20.6
86.0
27.9
38.9
Table 2: Comparison between dense and MoE with large-scale configuration. Drop- Drop-Upcycling (DU) works well even x scale. The MoE model with Drop-Upcycling outperforms dense models trained with higher computational costs, demonstrating the effectiveness of Crop-Upeycling.
Model
Training
Individual Seares
Architecture
MoE
Init
Aci Params Total Params
Tokens
(x10'))
FLOPs
JEM HQA
NILL
JSQ
Sum
XL.
WMT
E-J
WMT
J-E
OB QA
TQA
HIS
SQ
XW:
EN
BBH
Avg
Dense 3.78
3.78 / 3:7B
1,000B
2.70
44.5
47.2
78.8
12.8
21.4
15.4
25.0
33.8
47.3
23.7
85.9
28.7
38.7
Moll 3.7.7B
3.98 188
500B
1.98
53.5
50.8
69.5
10.4
20.6
13.9
29.0
45.8
51.1
21.1
87.1
28.1
40.1
Moll 8xc DU (y-0.5
3,98 188
5008
1.98
47.5
57.0
82.2
16.3
25.0
19.0
31.2
53.6
54.4
26.3
88.5
32.2
44.
Dense
13B/ 13B
505B
7.43
47.6
58.3
85.2
14.1
24.5
18.3
31.4
48.6
53.1
29.3
88.3
35.2
44.5
large-scale experiments reveal that even such seemingly naive baseline outperform naive Up- cycling in certain scenarios. For additional analysis on the impact of the value, refer to Section 5.3
Table provides a comparison of the final downstream task performance for models trained with various methods under these 8x 152M and 1.5B settings. Model numbers refer to the leftmost column of this table. This table also includes the dense models used for upeycling. Specifically, Model is the dense model used to initialize Models 3-7, and Model is used to initialize Mod- els 10-14. The proposed method, Drop- Moo-Uppycling (DU) with r 0.5, consistently demonstrates superior performance across these model scales.
5.2 SCALING TO 8x3.7B
To further evaluate the effectiveness of Drop-Upcycling in larger-scale settings and to build prac- tical MoE model, we conducted experiments with an 8x3.7B configuration. Due to computational resource constraints, experiments under the 8.3.7B setting were limited to training from scratch and Drop-Upeycling with r 0.5.
The rightmost column of Figure iillustrates the learning curves under this configuration. Similar to the 8x 152M and 8x 1.5B settings, Drop-Upeyclie significantly outperforms training from scratch. There is initial gain in performance due to the improved initialization, and expert diversification allows the training to progress as efficiently as in the case training from scratch, ensuring that Drop-Upcycling never gets overtaken.
Table compares the models' final downstream task performance. Model numbers refer to the left- most column of this table. Model is dense model used a base model for the Upeycling. Models
8

---

Published as a conference paper at ICLR 2025
8x1 M5152M Training Loss (Lower is better)
8=152M
Downstream Task Score (Higher better)
8x1.5B Training Loss (Lower better)
8x1.5B Downstream Task Score (Higher is better)
18.30
2.06
2.83
18.75
2.01
33
Entropy
Average Score
Cross Entropy
Average Score
2.78
15.50
1.97
29
2.73
14.25
1.83
25
2.59
50
100
150
13.00
50
100
150
1.89
50
100
150
21
50
100
150
Processed Tokens (Bilioon))
Processed Tokens (Bilitons)
Processed Tokens (Biliion))
Processed Tokens (Bilions)
nalve Upcycciin
Drop-Uppyllingg (r- 0.1)
Drop-Upcycling (r- 0.25)
Drop-Upcycling 0.5)
Drop-Upcycling
Drop-Upcycling
Figure 4: Impact of re-initialization ratio r. The training loss and downstream task score over the total number of tokens processed during training on 8x152M (left two figures) and 8x1.5B (right two figures) settings are illustrated. Even with different r values, Drop- Prop-Upeycling robustly outperforms naive Upcycling. and 0.5 appears be the most effective ratio.
2 and 3 are MoEs built using naïve Upcycling and Drop-Upcycling, respectively, demonstrating the superiority of Drop-Upcycling. In addition, two different baseline dense models, Models and 5, are included in the table. Model 4 is 13B dense model. Our 8 8.3.BB MoE architecture has fewer active parameters than this 13B model, leading lower training and inference costs. Nevertheless, the 8.3.7B MoE model using Drop-Upcycling achieves better performance upon completion of training. Model model with Drop-Upeycling surpasses this dense model indicates that rather than continuously in- vesting resources into training dense models, might a superior option to convert them to MoE models through Drop-Upcycling and continue training certain point in the process.
3.7B dense model trained with 2.IT tokens. The fact that our 8x3.7B MoE
5.3 ANALYSIS 1: RE-INITIALIZAITON RATIO
We conducted a study to investigate the impact of the re-initialization ratio r in Drop-Upcycling. Tigureeiillusrratt the effects of different re-initialization rates 0.0 (naive Upeyeling), 0.1, 0.25, 0.5, 0.75, and 1.0 on models of sizes 8x152M and 8x1.5B. Each model was trained to 150B tokens, during which we monitored the training loss and the progression of the average downstream task scores.
The experimental results revealed similar trends across both model sizes. In terms of long-term performance, re-initialization ratio of 0.5 yielded the best results for both models, maintaining superiority both training loss and average task scores. An interesting pattern emerged regarding the influence of the re-initialization ratio. With lower re-initialization rates, particularly at 0.0 (naïve Upcycling), the models struggled to significantly improve beyond the performance of the original pre-trained models. While re- re-initializatin rates of 0.1 and 0.25 showed promising performance in the early stages of training, they were eventually surpassed by the 0.5 re-initialization rate as training progressed. These observations suggest that increasing the re-initialization ratio helps the models escape local optima, enabling more effective learning. However, excessively high re-initialization rates of 0.75 or 1.0 appeared to hinder the effective knowledge transfer from the pre-trained dense models. This phenomenon highlights an important trade-off concerning the MoE initialization: balance must be struck between knowledge transfer and effective expert specialization. Drop- Upeycling with r = 0.5 is robust and practical method that ideally balances these two aspects.
5.4 ANALYSIS 2: EXPERT SPECIALIZATION
We analyze expert routing patterns to examine how Prop-Upeycling facilitates expert specialization. We apply the methodologies of Jiang al. (2024) and Muennighoff et al. (2024) to 8x1.5B MoE models trained with different methods. This analysis investigates how data from different domains routed to various experts. As input data from different domains, the validation sets from

---

Published as a conference paper at ICLR 2025
8x1.5B From Scratch
8x1.5B naive Upeycling
8x1.9B Drop-Uppyceling (r 0.5)
0.75
Blll
0.75
Rouling Probability
0.28
0.225
8..m
0.28
Expert
Expert
English Wildpedia
Japanese MC4
Wikipodaa
Stack
Figure 5: Comparison expert routing patterns across different MoE construction methods. Drop-Upeycling exhibits more balanced expert utilization than naïve Upeycling. Results shown for layers 0 (first), 8, 16, and 23 (last); see Appendix C.2 for results on all layers.
Japanese and English Wikipedia; the validation set of the Japanese MC4 dataset (as split by the authors; see [.L.-jpp 2024), originally introduced by Raffel al. (2019); The Stack (Kocetkov al. 2023) ; and the English C4 dataset (Muennighoff al. 2024).
In Figure we observe that naïve Upeycling with global load balancing results in a highly imbal- anced routing pattern, where the majority of experts were underutilized not utilized at all, with only two experts being always selected across layers. While laser-wise load balancing mitigate such expert collapse, performance between these two strategies (see Appendix C.3). In contrast, both the model trained from seratch and the one enhanced with Drop-Upeycling (with
found significant differences in the training loss trajectories model
0.5) exhibit domain-specialized routing patterns regardless of the load balancing strategy. The routing patterns reveal that certain experts specialize processing specific types data, such as Japanese text, English text, or code snippets, as evident from the distinct expert selection probabilities corresponding to each dataset.
These findings suggest that Drop- -pcyelpgyll promotes effective expert specialization independently of the load balancing strategy, which likely contributes to the improved performance observed in our experiments. For detailed routing patterns across all 24 layers and further analysis of load balancing strategies, see Appendix and
CONCLUSION
a novel method for efficiently constructing Mixture of Experts (MoE) models from pre-trained dense models. Selectively re-initializing parameters of effectively balances knowledge transfer and expert
In this paper, we introduced
expert feedforward networks, specialization, addressing the key challenges in MoE model development.
Our extensive large-scale experiments demonstrated that Drop-Upeyeling, significantly outperforms previous MoE construction methods. As result, we achieved an MoE model with 5.9B active parameters that matches the performance of 13B dense model from the same model family while requiring only about 1/4 of the training FLOPs.
By making all aspects of our research publicly available-include data, code, configurations, checkpoints, and logs we aim to promote transparency and facilitate further advancements ef- ficient LLM training. We believe that uoop-upyylinnoofeers practical solution to reduce resource barriers in deploying performerormance LLMs, contributing to broader accessibility and innovation in Al research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The authors would like to thank Masanori Suganuma and Kou Misaki for providing valuable dis- cussions and feedback during the preparation of this manuscript. This work was supported by the "R&D Hub Aimed at Ensuring Transparency and Reliability of Generative Al Models" project of the Ministry of Education, Culture, Sports, Science and Technology. This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.
AUTHOR CONTRIBUTIONS
Taishi Nakamura initiated the project, designed the method, and carried out the experiments. Takuya Akiba co-designed the experiments and formulated the overall research strategy. Kazuki Fujii im- plemented the training codebase used for the experiments. Yusuke Oda handled the training of the dense models. Jun Suzuki and Rio Yokota provided guidance and oversight throughout the project. All authors contributed the writing and approved the final manuscript.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, and Harkirat Behl et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint urXiv:2404.14219, 2024.
Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, and et al. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of WMT, pp.
1-55, 2020.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda et al. Askell. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Weilin Cai, Juyong Jiang. Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. CoRR, 0bbs/2407.06244. 2024.
Damai Dai, Chengqi Deng. Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li. Wangding Zeng, Xingkai Yu, and Y. et al. Wu. DeepSeekMoF: Towards ultimate expert specialization in mixtu language models. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 1280-1297. Association for Computational Linguistics, 2024.
Tri Dao. FlashAttention Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations, 2024.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
William Fedus, Barret Zoph, and Noam M. Shazeer. Switch transformers: Scaling to trillion param- eter models with simple and efficient sparsity. Mach. Learn. Res., 10:1-920:39 2021.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient Sparse Training with Mixture-of- Expersss Proceedings of Machine Learning and Systems. 5, 2023.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, ... and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics (ACL), pp. 4693-4703, 2021.
Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan. Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upcycling large language models into mixture of experts. arXiv preprint 4rX1v:2410.07524, 2024.

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, and Aidan et al. Clark. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, 2022.
Ai Ishii, Naoya Inoue, and Satoshi Sekine. Construction of a Japanese multi-hop QA dataset for QA systems capable of explaining the rationale (ML & BE nj Bl ts. H (740)) tz &
0) H t iit 1 its y - 2 te y h M ] (in Japanese). In the 29th Annual Meeting of Japanese Association for Natural Language Processing (NI.P2023), pp. 2088-2093, 2023.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural 3():79-8, 1991.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, and Florian Bressand et al. Mixtral of experts. arXiv preprint Stt::2401.04088. 2024.
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural Comput., 6(2:1811221, 1994.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1601-1611. Association for Computational Linguistics, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.083. 2020.
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. The stack: 3 TB of permissively licensed source code. Transactions Machine Learning Research, 2023.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-off-eppert. from dense checkpoints. In International Conference on Learning Repre- sentations, 2023.
Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. JGLUE: Japanese general language understanding evaluation. Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2957-2966, 2022.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS Hardd Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- lutions, 2021.
LLM-jp- Llm-jp: A cross-organizational project for the research and development of fully open japanese 1lms. arXiv preprint arXiv:2407.03963 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? A new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391. Association for Computational Linguistics, 2018.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, and Nathan Lambert et al. Olmoe: Open mixture-of-expprrs language models. arXiv preprint arXiv:2409.0200 0224000000000,.... 2024.
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2024.
12

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Gray et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 784-789, 2018.
Satoshi Sekine. Development of question answering system focused on an encyclopedia [ ス5 & [M [ (in Japanese). In the 9th Annual Meeting of Japanese Association for Natural Language Processing (NLP2003), pp. 637-640, 2003.
$ & $ &
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05000 2020.
Noam Shazeer, * AAzalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-experr layer. In International Conference on Learning Representations, 2017.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, and et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15725- 15788. Association for Computational Linguistics, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wer Li, Wen tau Yih, Jason E Weston, and Xian Li. Branch-train-mixx Mixing expert LLMs into a mixture-of-experts LLM. In Conference on Language Modeling, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Compu- tational Linguisties: ACL 2023, pp. 13051. Association for Computational Linguistics, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, and Shibo Wang et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computa- tional Linguistics (ACL), pp. 3534-3546, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, and Faisal Azhar et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2025
Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lüi, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, and Liang Zeng et al. Skywork-mos moe: A deep dive into training techniques for mixture-of-experts language models. arXiv preprint arXiv:2406.06563, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, and Fei Huang et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800. Association for Computational Linguistics, 2019.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, 2019.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y. Zhao, Andrew M. Dai, Zhifeng Chen, Quoe V. Lc, and James Laudon. Mixture-of-expert with expert choice routing. In Advances in Neural Information Processing Systems, 2022.
14

---

Published as a conference paper at ICLR 2025
Table 3: Detailed FL.OPs Breakdown for Transformer Models (Single Forward Pass)
Component
FLOPs
Embeddings
2suda
Attention (per layer)
Key and value projections
Asdadan Rg 28d4 dan 28 dgna
Query projections
Key Query logits
Attention matrix computation
2s dema 2sdan..dh
Softmax @ value reductions
FFN (SwiGLU, per layer)
Dense model
Asd + 22d dp Th ne (Asdad 2xd
MoE. model
Final Logits
2sd
Total (Dense)
embeddings +m (attentioo + ffnDemse) )+ logits embeddings +-n (attention ffi logits
Total (MoF)
A EXPERIMENTAL SETUP DETAILS
A.1 FLOPS CALCULATION
Table presents the method for calculating FLOPs (floating point operations) for the forward path in transformer components. The variables used are as follows: (sequence length), d, (hidden size), (vocabulary size), d (FFN intermediate size), ny (number of layers), ni (number of attention heads), ny (number of query groups), dę (attention head dimension), and ne (number of selected experts per token). For matrix multiplication Amx X Xxxn 2m X X n FLOPs are required in the forward pass (the factor of 2 accounts for both multiplication and addition operations). The table displays the main FLOPs contributors for the forward path only. It should be noted that the computational costs for sigmoid and Hadamard product within SwiGLU calculations, MoE gate computations, and RMS Norm calculations are considered negligible and thus omitted from this analysis. While not shown in the table, backward propagation typically requires approximately twice the FLOPS of forward propagation.
A.2 IMPLEMENTATION AND TRAINING ENVIRONMENT
For our experiments with MoE models and the training of the 1.5B Dense model, we utilized the TSUBAME 4.0 supercomputer at the Global Scientific Information and Computing Center, Institute of Science Tokyo. This environment equipped with NVIDIA H100 SXM5 94GB GPUs, with each node housing 4 H100 GPUs. Inter-node communication is facilitated by InfiniBand NDR200 interconnects. The training of our largest model, the 8x3.7B model, employed 16 nodes (totaling 64 GPUs). For the training of the 1.2MM and 3.7B Dense models, we leveraged the high-performance computing nodes (PHY) provided by Sakura Internet. This setup features NVIDIA H100 80GB GPUs, with each node containing 8 H100 GPUs. The network interface is equipped with four 400Gb RoCEv2-compatible NICs and two 25Gb NICs. The training of our largest Dense model (3.7B parameters) utilized maximum of 32 nodes (totaling 256 GPUs).
For implementation, we used Megatron for Dense model training, and moe-recipes for MoE model training. Additionally, Flash Attention 2 (Dao 2024) utilized to improve computational efficiency and reduce memory usage. All the training processes were conducted using 6 precision.
15

---

Published as a conference paper at ICLR 2025
Table 4: Model Configuration Details
Model
Act Params/ /
Layers
modeel
dr
Attn Heads
KV Heads
Vocab
Total Params
Size
Dense 152M
152M / 152M
12
512
2,048
8
99,574
Dense 1.5B
1.5B 1.5B
24
2,048
7,168
16
8
48,586
Dense 3.7B
3.7B 3.7B
28
3,072
8,192
24
99,574
Dense 13B
13B 13B
40
5,120
13,824
40
40
99,574
MoE 8x152M
190M 417M
12
2,048
8
99,574
MoE 8x1.5B
2.6B 8.9B
24
2,048
7,168
16
8
48,586
MoE 8x3.7B
5.9B / 18B
28
3,072
8,192
24
24
99,574
A.3 MODEL CONFIGURATIONS
As described in Section we selected the Llama (Touvron al. 2023) and Mixtral (Jiang al.) 2024) architectures for dense and MoE models, respectively, for our experiments. Both architec- tures are based on the Transformer (Vaswani al. 2017) with several improvements, including RMSNorm (Zhang & Sennrich (211)). SwiGLU (Shazeer (020). and rotary position embeddings (RoPE) (Su et al., 2024). The notable difference Mixtral (MoE) from Llama (dense) is that all feedforward network (FFN) layers are replaced by sparsely gated MoE layers. Table shows the details of the model configuration.
A.4 MODEL TRAINING CONFIGURATIONS
As shared settings for training all models, we adopted the following hyperparameters: AdamW optimizer [Loshchilov & Hutter 2019) with 31 0.9, 32 0.95. and e = 10 sequence length of 4096, weight decay of 0.1, and gradient clipping of 1.0. The global batch size was set to 1024 for the 1.5B, 3.7B and 13B models, and 512 for the 152M model.
We used cosine decay for learning rate scheduling. For Dense models, the maximum learning rate x 10-5 over 1,000B tokens for the 1.5B model, and 2,072B tokens for the 152M, 3.7B and 13B models, with the learning rate remaining constant during the final 2000 steps. For MoE models, the maximum learning rate was set to x 10 -4,4,, and it decayed to 2 x 10 over 500B tokens. Additionally, to prevent instability in training due to unbalanced routing on the MoE models, a load balancing loss was introduced, with the coefficient unified at 0.02 across all MoE models.
was set to x 10-4, and decayed
B DATASETS AND EVALUATION METHODS
B. TRAINING DATASET DETAILS
used the LLM-jp corpus v an open corpus curated by the LLM-jp working group, for training English and Japanese bilingual language models. The corpus consists of 1.7T tokens in English, Japanese, and source code with a small amount of Chinese and Korean tokens. Following the LLM- jp's scheme, some Japanese portion of the corpus upsampled by 2 to obtain 2T training tokens in total.
Table 5 describes the statistics of the corpus subsets that used for training data of the Dense and MoE models our experiments.
Table 6 details the dataset distribution percentages used for training the different model sizes. The 152M, 3.7B, and 13B models share the same data proportions, while the I 1,5 model has slightly different percentages.
pes. Version 1.0.0
16

---

Published as a conference paper at ICLR 2025
Table 5: Statistics of the training dataset.
Language
Subset
#tokens x 10"1
English
Dolma 1.6 (sampled) Soldaini et al. 2024
945.
Wikipedia
4.7
Japanesc
Common Crawl (LLM-jp 2024
381.
Kaken
0.9
NDL WARP HTML. NDL WARP PDF Wikipedia
207.
1.3
Chinese
Wikipedia
0.8
Korean
Wikipedia
0.9
Code
The Stack (Kocetkov et al.
2023)
114.
Table 6: Dataset Distribution Overview (Percentages)
Language
Subset
152M/3.7B/13B
1.5B
Dolma
45.6%
39.7%
English
Wikipedia
0.2%
0.5%
Common Crawl
36.8%
49.5%
Kaken
0.1% %
0.1%
Japanese
NDL WARP HTML NDL WARP PDF
0.1%
11.5%
Wikipedia
0.1% %
0.2%
Chinese
Wikipedia
0.1%
Korean
Wikipedia
0.1%
Code
The Stack
5.5%
10.1%
Total Tokens (B)
2,072
1.000
B.2 EVALUATION DATASETS AND METHODOLOGGESS
Table proviidss detailed information about the evaluations used in our experiments. The evaluation tasks comprise both Japanese and English language assessments. We utilized publicly available evaluation code for our assessments
The evaluation tasks categorized into types, such free-form QA (NILLC (Sekine)
2003).
JEMHQA (Ishii et al., 2203)), ), machine reading comprehension (JSQuAD (urihara et al.
2022
SQuAD2 (Rajpurkar al. 2018)). abstractive summarization (XL-Sum (Hasan et al.
2021
machine translation (WMT T 20 En-Ja, Ja-En (Barrault et al. 20200), question answering (Open-
BookQA (Mihaylov et al. 2018). TriviaQA (Joshi al.] 2017)) common sense reasoning (Hel-
laSwag (Zellers et al. 2019), Xinngrad (Tikhonov & Ryabinin 202T)). and logical reasoning (Big Bench Hard (BBH) (Suzgun et al. 2023)). We used 4-shot prompting for the Free-form QA, machine reading comprchension, machine translation, question answering, and commonsense rea- soning tasks, I-shot prompting for the abstractive summarization task, and 3-shot prompting for the logical reasoning task. Moreover, we also applied the Mhain-of-Thoughe method (Wei et al. 2022) for the logical reasoning task.
https://githubs wwws/owwwwwlloooo 1m/ 11 1ow luation
17

---

Published as a conference paper at ICLR 2025
Table 7: Evaluation Benchmark Details
JEM HQA
NILLC
JSQ
Sum
XL-
WMI
E-+J
WMT
J-
QA
OB
TQA
RS
SQ
V2
XW- RN
BBH
Dataset
JEMHQA
NILL
ISQUAD MRC
XL-Sam
WMIZDD
OBQA TAVIQQ
HellaSwag MRC
SQUADZ MRC
XWIND Commonsense
BBH
Task
QA
QA
Logical
Reasoning
Reasoning
Language
JA
JA
JA
EN 1,000
JA
EN
EN
EN
EN
EN
EN
EN
# Instancess
120
198
4,442
766
9933
300
17,944
10,042
11,873
2,325
6,511
Few-shot
Evaluation
Character
Accuracy
Table 8: Gate Initialization Pattern Comparison for 8x1.5B Models (Training Tokens: 50B)
Initialization
Results
Distribution
JEM
III
JSQ
XI.
J-EE
+J
OBQ
TrQ
SQ2
HeS
XWI
BBH
AVG
1 (o,, 0.02)
46.1
63.6
9.2
15.4
22.4
19.4
41.7
15.6
80.0
25.9
32.1
2 (0, 0.2887)"
50.6
38.6
54.6
9.3
15.5
8.3
20.6
18.4
41.1
14.3
79.8
24.7
31.3
3 U(-0.0346, 0.0346)
49.2
38.9
61.0
9.7
16.0
7.9
23.6
18.9
41.7
15.5
80.9
23.9
32.3
W(-0.5, 0.5)
8.6
15.5
20.6
17.7
41.0
80.0
26.0
30.8
5 1)(
51.5
36.8
55.6
9.0
15.7
7.9
21.6
18.3
41.0
15.3
80.1
25.1
31.5
Ni, d): Normal distribution with mean and standard deviation a.
U(a, b): Uniform distribution over the interval (a, b].
a /1/12 R 0.2887, matches the standard deviation of (4(0, 1).
Corrected from L((-0.0346, 0.0346) to match the standard deviation of 0.02. Bold values indicate the best score for each task.
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
C.I COMPARISON OF GATE INITIALIZATION METHODS
We conducted detailed investigation into the effects of gate initialization on the performance of naïve Upcycling. An ablation study was performed on five different initialization patterns. Table B presents the comparison results of different gate initialization patterns in an 8x1.5B model. Perfor- mance was evaluated after training 50B tokens.
While preliminary experiments had indicated better results with a standard deviation of 0.28, our main experiments revealed that uniform distribution with a standard deviation of 0.02 achieved the highest average performance across tasks. Based on these results, we adopted a uniform distribution (U(-0.03466, 0.0346), as the standard method for gate initialization in this study. It is worth noting that gate initialization may not critical factor in model performance, and any initialization that avoids extreme values such excessively high standard deviations is likely to be sufficient.
C.2 DETAILED ANALYSIS OF EXPERT ROUTING PATTERNS ACROSS LAYERS
For comprehensive view of routing patterns across all layers, we provide detailed plots of expert routing probabilities for all 24 layers, grouped into early, middle, and late stages. These plots offer more granular analysis of how routing behaviors evolve throughout the model depth.
Figures 6 show the expert routing patterns for all 24 layers of the 8x1.5B MoE models trained with different methods, grouped into early (layers 0-7), middle (layers 8-15), and late (layers 16-23) stages. This comprehensive view allows for detailed analysis of how routing patterns evolve across the entire model depth.
These figures illustrate how the routing patterns evolve throughout the model layers, providing in- sights into the specialization and behavior of experts at different depths. Notably, the naïve Up- cycling method does not exhibit clear evidence of bias towards specific domains in any layer. In contrast, proposed method demonstrates domain specialization in multiple layers across the net- work from those closest to the input to those near the output while reusing the parameters of the dense model. This indicates that approach effectively facilitates expert specialization in several
18

---

Published as a conference paper at ICLR 2025
8x1.5B From Scratch
8x1.5B naive Upcycling
8x1.9B Drop-Uppycllig (r 0.5)
0.25
0.70
0.25
0.70
Routing Probablitt
0.28
0.76
0.75
Laye
0.25
11.1
8
Expert
Expert
Expert
English Wikppedi
Japanese
Japanese Wikipdda
Figure 6: Expert routing patterns for early layers (0-7) of the 8x1.5B MoE models.
layers without the need to train from scratch, leveraging the pre-trained dense model to achieve efficient dopain-specific routing throughout significant portions of the network depth.
19

---

Published as a conference paper at ICLR 2025
9x1.5B Upcycling
8x1.5B Drep-Upcycling =
0.25
0.78
Layera
0.25
0.75
Layer
0.25
Routing Probabilty
2.7.7
0.20
0.75
0.75
Layer
0.28
Expert
Expert
Expert
English Wikipedia
Japanese MC4
Japaneeee Wikiped'a
The Stack
Figure Expert routing patterns for middle layers (8- 15) of the 8x1.5B MoE models.
20

---

Published as a conference paper at ICLR 2025
8x1.5B
9x1.5B Upcycling
8x1.5B Drep-Upcycllng (r=
0.25
0.75
0.6
0.22
0.75
18
0.25
Routing Probabilty
0.77
0.22
21
0.75
Layer
0.76
Layer
0.28
Expert ID
Export
Expert
English Wikipedia
Japanede
Japaneeee Wikiped'a
The Stack
Figure 8: Expert routing patterns for late layers (16-23) of the 8x1.5B MoE models.
21

---

Published as a conference paper at ICLR 2025
From Scratch
Branch-Train MIX Training Losa
native Upoyaling Training Losa
Drop-Upeycling (r- 0.5)
Drop Prop-ppeydlig (r- 1.0)
Training Losa
Training Loss
Training Lose
8.00
3.00
3.00
3.00
3.50
2275
2.75
2.75
2.75
2.50
2.50
2.50
2.50
2.50
Crote
2.25
2.25
2.25
2.20
2.26
2.00
2.00
2.00
2:00
2:00
Processed Tokans [Billons)
Procsssod Tokens (Bilioss)
Procossod Tokers (3llions)
Procossed Tokens (Bllkos)
Processed Tokoos (Bllios)
From Scratch (Higher better)
Branch-Traiin MIX (Higher batter)
natre Upoyaling
Drop Upeycllng (r= 0.5] (Higher ballar)
Drop Drop-Upcycling (r= 1.0) [Higher better)
Score (Higher better)
35
35
30
30
30
10g
10g
10g
10g
10
Procasted Tokens (Bl))
Processed Tokens (oonsss)
Proceesed Toknre (111)))))
Processed Tokens ([1lions)
Processed Takens (Billons)
Giotal Load Balance Loss
Layerwite Load Balance
Figure 9: Comparison between global and layer-wise load balancing across different initial- ization methods. Top: Training loss trajectories over 40B tokens. Bottom: Evaluation metrics measured at iterations corresponding to 10B, 20B, 30B, and 40B tokens. Results show comparable performance between global and layer-wise approaches across all methods.
C.3 COMPARING GLOBAL VS. LAYER- WAYER-WISE WISE LOAD BALANCING
In our experiments (Section 5), we applied load balancing loss globally rather than layer-wise. This approach aligns with the implementation in the HuggingFace Transformers library and widely adopted in the community. To analyze the effect of global and layer-wise load balancing, we con- ducted comparative analysis between global and layer-wise load balancing applications across 40B tokens for different initialization methods (From Scratch, Branch-Train-MiX, naive Upeycling, and Drop-Upeye -peycllngg with 0-5 and 1=1.0) in the 8x1.5B setting. As shown in Figure both approaches yield similar training loss trajectories and downstream task performance. These results suggest that the effectiveness of Drop-Upcycling is not significantly affected by whether load balancing loss is applied globally layer-wise.
Figures TO 12 show the routing patterns when applying layer-wise load balancing loss at 40B tokens. The results demonstrate that patterns similar to training from scratch. In contrast, naïve Upcycling shows nearly uniform routing across all layers except the final layer, which aligns with findings reported in inninng et al. (2024). Our proposed Drop- Uppycliin method appears to escape the local optima observed in naive Upcycling, which likely contributes to its improved performance.
(r=0.5) exhibits Nomain-speciallzze routing
The trade-offs between hayer-wise and global load balancing whether to enforce uniform expert utilization through layer-wise application or to allow potential expert collapse with global applica- tion along with broader questions about MoB architecture design (such as varying expert counts per layer) remain as interesting directions for future research.
22

---

Published as a conference paper at ICLR 2025
8x1.5B From Scratch
8x1.5B nalve Upeycling
8x1.5B Drop-Uppcceling (r 0.5)
0.75
0.25
0.75
0.25
0.75
Routing Probabiitt
0.75
Layer'
0.26
0.76
Lyyet
0.75
Layer
Expert
Expert
English WiMppedi
Wikipeda
The Stack
Figure 10: Expert routing patterns for early layers (0-7) under layer-wise load balancing at 40B
tokens
23

---

Published as a conference paper at ICLR 2025
8x1.5B From Scratch
8x1.5B naive Upeycling
8x1.5B Drop-Uppyceling (r 0.5)
0.75
0.25
1m Edht
0.75
0.20
Hnt.
0.78
Routing Probabiltt
240.77
0.76
0.26
0.76
83
0.75
Expert
Expert
English WiMppdia
Wikipeda
The Stack
Figure 11: Expert routing patterns for middle layers (8- under layer-wise load balancing at 40B tokens
24

---

Published as a conference paper at ICLR 2025
8x1.5B From Scratch
8x1.5B naive Upeycling
8x1.5B Drop-Uppcceling (r=
0.5)
0.75
0.25
1ld
0.75
0.20
Ind
0.78
Routing Probabiltty
0.76
1ayer/2
22
0.75
Expert
Expert
English WiMppeia
Wikpedd
The Stack
Figure 12: Expert routing patterns for late layers (16-23) under layer-wise load balancing at 40B
tokens
25

---

Published as a conference paper at ICLR 2025
8x152M
8x1.58
8x3.7B
175
Drop- Upcycliyy Taken Lead
2000
Drop-Upcycling Token Lead
150
150
100
Drop-Upcycling Token
125
100
1.00
75
50
-50
-100
300 400 Processed (Billions)
(Billions)
Tokens (Billions)
From Scratch
Branch-Train-Mik
Naive Upcycling
Figure 13: Convergence catch-up analysis. We compare the relative convergence speed of Drop- Upcycling and baseline methods by examining the number of training tokens required to reach the same loss value. The x-axis represents the number training tokens processed by the base- line method, while the y-axis shows the difference in training tokens needed by Drop-Upeycling to achieve the same loss. Positive values indicate that Drop-Upcycling achieves the loss faster, while negative values suggest the baseline method is ahead.
C.4 CONVERGENCE CATCH- UP ANALYSIS
To examine the selection of methods based on the training budget and to explore potential extrapola- tions of long-term trends beyond the scope of our analysis far, conduct brief relative quanti- tative analysis of the convergence speeds Drop- Prop-Upcycling and baseline methods. In Figure [3] we compare the number of training tokens required to reach the same loss value for coop-Upcylling. and the baseline methods. The plot shows that no significant trend of diminishing advantage for Drop- Upeycling the baseline methods is observed. This indicates that training from scratch would require an impractically large number of tokens to match Drop-Upcycling, making Drop-Upcycling the better choice in practical scenarios.
However, it is important to acknowledge the limitations of this analysis. First, the effect of the learning rate (LR) schedule must be considered. Differences in LR due different step counts could artificially influence the observed trends in convergence advantage. For example, we hypothesize that the widening advantage of Drop-Uper Upcycling observed late in training (after 400B tokens) may not entirely reflect the contribution of Drop-Upeychhe itself but could instead be attributed to the influence LR scheduling. To eliminate the impact LR scheduling, conducting all experiments with constant LR would provide more valid basis for this comparison.
Second, worth noting that ing individual experts before MoE training. In setup, for instance, three expert models were pretrained using 100B tokens each, requiring a total of 300B tokens for dense model training before the MoE training phase. As result, while Branch-Train-Me Train-Mix appears to show initial advantage in the plot, this advantage diminishes when accounting for the total training budget. Thus, in terms of overall efficiency, Branch-Train-Ne Train-Mii offers little to advantage during most of the training process.
utilizes an additional training budget for pretrain-
C.5 DETAILED DERIVATIONS OF THEORETICAL CHARACTERISTICS
Consider the output of MoE layer with parameter re-initialization ratio Let FFN retained, (x) denote the output from expert i's preserved original parameters (ratio (1 - r)) and FFNavvese, (x) denote the output from reinitialized parameters (ratio r). The exact form of MoE output is:
N y
Σ g(x); (FFNeetsined, (x) FFNdiverse, (x))
where 9(x), is the gating function defined in 3 Note that g(x); = for experts not among the top-k selected.
26

---

Published as a conference paper at ICLR 2025
Let Sy denote the set of indices for the le selected experts. We can rewrite the output
y = Σ g(x)
(FFN (x) + FiNdiierse, (x))
iESt
Σ gxx);
|FFNcommon (x) + (FFNNettined, (x) FFN (x)) + FFNdiverse, (x)
KES&
FFNcommon
FFN common (x)
Σ 9(x), i +
Σ
g(x); [FFN (x) FFF (x) + FFNdiverse, (x)
sESN
iESμ
FFN common (x)
Σ g(x): ; retaieed, (x) FFNcommon (x) FFN diverse, (x)]
iE9
N
= FFN common (x) + 9(x);
Σ
[FFNretained, (x) - FFN (x) + FFN (x)
2=1
where the third equality follows from distributing the sum, the fourth equality follows from Eies, gx); 1, and the final equality holds because g(x); for & Sk. Here FFN (x)
represents the output from parameters common to all selected experts.
For each expert, ratio (1-7) r) of parameters are randomly preserved from the original FFN. When k experts are selected, the probability that parameter is preserved in all k experts is (1 - r)*. Therefore, approximately
r)* -dy dimensions have common preserved parameters among selected experts, where d is the intermediate dimension size. Note that beyond these completely common parameters, there may be partial parameter sharing among subsets of the selected experts due to the random preservation process.
To understand the error bound O Vd, consider that for any two experts i, the number over-
lapping parameters follows binomial distribution Bid,, (1 -)) By the Central Limit Theorem, the deviation from the expected value scales with /dy. leading to a relative error of O in the parameter overlap estimation.
C.6 EXTENSIONS TO FINE- GRNAIGRAINED AND SHARED EXPERTS
We discuss the natural extension of Drop-Upeycling to advanced MoE architectures: fine-grained experts and shared experts proposed in DeepSeekMoE (Dai et al. 2024). For an original dense FEN with hidden dimension d, and intermediate size dy. DeepSeekMoE introduces granularity parameter m to split each of N experts into finer segments (each with intermediate size d d//m), where mk experts total number of experts becomes mN with mik nonzero gates, which reduces to mN hig experts and mk k2 gates when using shared experts.
selected by top-mk routing, and ks shared experts process all tokens. The
C.6.1 EXTENSION TO FINE-GRAINED MOE
For simplicity of discussion, we assume d is divisible by m for fine-grained MoE (a realistic as- sumption since m is typically power and dy contains powers of as factors). The output of the MoE layer is expressed as:
m.N
Σ g(()() FFN (x)
When applying Drop-Upeycling to convert from dense FFN layer to fine-grained MoE layer, conduct the following steps:
1. Expert Dimension Sampling. First, randomly sample d,/m dimensions from the original FFN intermediate dimension dy for each expert.
2. Column-win wise Reinitialization Sampling. For each expert's sampled d /m dimensions, se- lect an index set S where [S] [r dy/m] dimensions to be reinitialized.
3. Statistics Calculation. Calculate means and standard deviations (Hun amn): (Higate ggate):
(Jdoow: down for the weight matrices corresponding to the selected indices S.
27

---

Published as a conference paper at ICLR 2025
4. Partial Re-Initializaaion. Initialize each expert's weight matrices according
type = Is @ Raype + (1 Is) o W Wype
(9)
where Reype is sampled from N (Htyee (Trype)
Note that the portion reinitialized by our method needs to be scaled down due to the increased num- ber of activated experts in top-mk routing resulting smaller g(x), While the absolute magnitude information in router outputs might adapt during training, following He et al. (2024), scaling the weights of down and Wap might be beneficial.
C.6.2 COMBINATION WITH SHARED EXPERTS
When using both shared experts and fine- line-grained experts, the output is:
m.
(o)
g())(1-k)) (T)
(10)
Here, shared experts are always active and process dimensions (dh, d f//m= kz), while fine-grained experts each process d /m dimensions.
We initialize fine-grained experts using the method described above. For shared experts, we can either randomly sample d f/m k; dimensions from the dense FFN and directly copy the corre- sponding weights, or apply Drop-Upcycling to those sampled dimensions. We apply weight scaling to both types of experts.
Note that whether shared experts maintain the same functionality as dense remains open research question, and comparing initialization methods for shared experts left for future work.
C.6.3 LIMITATIONS AND FUTURE DIRECTIONS
While we provide basic extensions of our method to fine-grained and shared expert settings, sev- eral important research questions remain unexplored. Our method could serve as a baseline for investigating how knowledge from dense models transfers to these advanced MoE architectures. Specifically, analyzing the transformation process from dense to fine-grained or shared experts could provide valuable insights into how these architectures function and develop specialization. For example, tracking how knowledge is distributed across iieeggrinedd experts during training, understanding what types of information shared experts learn to capture, could deepen our under- standing of these MoE variants. Such analyses could also inform better initialization strategies and architectural choices for future MoE models.
28


Reviews:
[{"review_id": "ba1T6YPGMR", "reviewer": "Reviewer_FeSK", "summary": "This paper presents Drop-Upcycling for MoE training utilizing the knowledge of pre-trained dense models while statistically re-initializing\nsome parts of the weights. The proposed method promotes expert specialization (which is often a bottleneck in traditional upcycling), significantly enhancing the MoE model’s efficiency in knowledge acquisition.", "questions": "I think the identified problem is important but I’d like to rate the current submission below the acceptance threshold (included towards reject) due to limited technical contributions, mainly similarity with existing upcycling methods like the one used in Qwen2.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow.\n- The idea of drop-upcycling is interesting although not new (see weaknesses section).\n- Extensive experiments and release of all the relevant materials for reproducibility.", "weaknesses": "- How is the proposed upcycling method different from the upcycling used in Qwen2: https://arxiv.org/pdf/2407.10671? Qwen2 shuffles parameters along the intermediate dimension to promote diversity which I think is very similar to the current method. They also randomly initialize 50% of the parameters. I would suggest the authors to discuss the differences with experiments.\n- What are the benefits of upcycling over training from scratch? I think changing architectural components are often difficult as upcycling is often constrained by the dense model architecture.\n- Will the current method going to be scalable to very large MoEs, say 100B+ parameter models? What additional challenges author think to encounter in training large MoEs with upcycling? \n- Can upcycling going to help in continual learning? E.g., can we add new experts to an existing MoE using dropupcycling?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMX5baiAnU", "reviewer": "Reviewer_u1Mr", "summary": "This paper proposes \"Drop-Upcycling\",  a method that enables an improved initialization scheme for expert parameters based on the weights of a pre-trained dense model. The approach addresses the lack of diversity in expert weights, a limitation of the standard Upcycling method where expert weights are simply cloned from the FFN in the dense model. \n\nDrop-Upcycling randomly samples indices from the intermediate dimension of the FFNs. The weights corresponding to these indices from $W_{up}$, $W_{gate}$, and $W_{down}$ matrices are dropped and reinitialized using the statistics of the dropped weights. The parameter $ r $ controls the proportion of indices sampled, with $ r = 1 $ representing full random initialization and $ r = 0 $ corresponding to naive upcycling.\n\nThe authors performed thorough experiments with an overall 200k plus GPU hours. A key insight is that the advantages of the approach is mainly in long-term training scenarios (>100B tokens).", "questions": "- As the authors highlighted, the Qwen1.5-2.7B-MoE model seems to follow a similar initialization strategy as Drop-Upcycling. It would be interesting to see how this initialization strategy could be extended for fine-grained MoEs with Granularity >1. Do the authors have thought on how they could extend their method accordingly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The empirical evaluation section and the number of large-scale training conducted are impressive.\n- The proposed method is very simple and the parameter $r$ intuitively controls the trade-off between knowledge retention from the dense network and incorporating diversity.\n- The authors implement their drop-upcycling on different parameter scale regimes from ~400M all the way to 18B parameters.\n- The authors provide the source code which is great for reproducibility", "weaknesses": "- Although the empirical evaluation is thorough and the results are promising, the paper has rather limited novelty. The paper would have become stronger if the authors could handle more challenging cases appearing in recent MoE literature such as fine-grained MoEs, or experimenting with a shared expert design. It would be beneficial to report the result of some other intuitive baselines such as adding random noise to the channels of the upcycled experts.\n\n- The way the load-balancing loss is applied seems ineffective. Could you elaborate why the loss was not applied layer-wise as that should prevent expert collapse. Even the model trained with Drop-upcycling shows half of the experts in layer 23 are not used by any of the domains.\n\n- The authors mention that the proposed method outperforms other upcycling approaches in long-term training scenarios. It seems in the very long-term training setting, training from scratch may outperform all methods and in the very short-term training (e.g. <50B tokens) the Branch-Train-Mix upcycling achieves the lowest loss. It would be best if the authors could explain this angle better and also study effect of model parameter size. For example at r=0.5, it takes around 70B Tokens to surpass Branch-Train-Mix for the $8\\times 152M$ model, while ~130B tokens are required for surpassing it for the $8\\times 1.5B$ Tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lfCRcvd8u", "reviewer": "Reviewer_jqcD", "summary": "This paper introduces Drop-Upcycling, a novel method for initializing Mixture of Experts (MoE) models from pre-trained dense models. The key innovation is selectively re-initializing parameters of expert feedforward networks while preserving some knowledge from the pre-trained model. Through extensive experiments, the authors demonstrate that Drop-Upcycling outperforms existing initialization methods like naive Upcycling and Branch-Train-MiX, achieving better performance with reduced training costs. Their best MoE model with 5.9B active parameters matches the performance of a 13B dense model while requiring only 1/4 of the training FLOPs.", "questions": "1. As a suggestion, the authors could consider expanding the theoretical analysis section with formal proofs or empirical validation of the approximations used. This would strengthen the paper's theoretical foundations.\n2. It would be valuable to include experiments or discussions on how Drop-Upcycling performs with different MoE architectures or configurations beyond the standard setup used in the paper.\n3. Despite the Qwen2 report being brief, the authors could provide a little more detailed analysis comparing their method with Qwen2-MoE,  highlighting any technical differences and their implications. This would be important given that the authors themselves point out the similarity in the two papers. In any case, I appreciate the \"open investigation\" into this kind of approach.\n4. It would be interesting to investigate whether using different re-initialization ratios for different layers (e.g., lower ratios in early layers and higher ratios in later layers) could improve performance. The authors could conduct experiments with layer-wise adaptive ratios or provide an analysis justifying why a uniform ratio is sufficient.\n5. It would also be great to see an analysis of the computational overhead introduced by the re-initialization process compared to naive Upcycling.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a clear technical contribution with Drop-Upcycling that addresses a fundamental challenge in MoE training - balancing knowledge transfer and expert specialization. This is demonstrated through comprehensive empirical results.\n2. The experimental evaluation is thorough and well-designed, including ablation studies on re-initialization ratios, detailed analysis of expert routing patterns, and large-scale experiments across multiple model sizes (152M to 3.7B).\n3. The work demonstrates significant practical impact by achieving comparable performance to larger dense models with substantially reduced computational costs, which is particularly relevant given the increasing focus on efficient training of large language models.", "weaknesses": "1. While the paper shows Drop-Upcycling works well for decoder-only transformers with MoE in FFN layers, there's limited discussion or analysis of how the method might generalize to other architectures (e.g., encoder-decoder models) or different MoE configurations (e.g., attention-based MoE layers).\n2. The theoretical analysis in Section 3.2.2 is relatively brief and could benefit from a bit more substance. I understand the space constraints, but it would be great if the authors could include some formal verification or empirical validation for the approximations used in equation (5).\n3. The paper uses a fixed re-initialization ratio (r=0.5) across all layers and experts. Given that different layers in transformer models are known to learn different types of features, a more adaptive approach to setting the re-initialization ratio based on layer position or expert functionality might yield better results. The authors don't explore this possibility or justify why a uniform ratio is optimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gCP2BzIS5B", "reviewer": "Reviewer_QL39", "summary": "This paper presents Drop-upcycling, which aims to boost the effectiveness of upcycling method by selectively re-initializing parameters to achieve the trade-off between expert diversity and knowledge transfers. Besides, this work provides a detailed introduction of employing upcycling for the initialization of MoE models, with good readability. The experiments are designed using two dense models such as Llama and Mixtral, and this work includes extensive experimental evaluation across seven types of tasks from both Japanese and English.", "questions": "- 1) In the caption of Figure 2, please add the definition of $S$ to ensure self-containment.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- 1) This paper is well-written and the Method Section is well-organized, thoroughly demonstrating the authors' understanding of the upcycling methods for MoE models. After reading the Method Section, readers gain valuable insights and can quickly follow the authors' motivation.\n\n- 2) The Drop-upcycling proposed in this paper to enhance the specialization of experts in MoE models is very straightforward, and it includes a hyperparameter that can control the degree of expert specialization and knowledge transfer.\n\n- 3) The experiments in this work are comprehensive, and the Experiment Section provides relatively analyses of the observed phenomena.", "weaknesses": "- 1) In Table 1, the training costs of different MoE initialization methods vary. It is recommended to add a comparison of training times for different models to demonstrate the advantages of your proposed method.\n\n- 2) In line 428, the author uses Model 1, Models 2 and 3, etc., to describe the performance comparison of different baseline models. However, it is unclear which model each number represents. It is recommended to add a \"Baseline Models\" section to introduce the names and methods of different baseline models. The same issue also appears in line 411.\n\n- 3) There are many ways to enhance the specialization of each expert in MoE models. For example, the Branch-Train-MiX (BTX) [Ref A] ensures expert diversity and specialization by feeding each expert different data distributions. In contrast, this work proposes a hyperparameter $r$ to control the initialization density of the FFN layer to ensure specialization. What are the advantages of this approach compared to the BTX method? Please give a brief discussion during the rebuttal period.\n\n[Ref A] Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)