Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
Authors: Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou
Accepted at: acl25

Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

Xue Zhang1,2*, Yunlong Liang3, Fandong Meng3, Songming Zhang1,2, Yufeng Chen1,2†, Jinan Xu1,2, Jie Zhou3
1Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education
2School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China
3Pattern Recognition Center, WeChat AI, Tencent Inc, China
{zhang_xue, smzhang22, chenyf, jaxu}@bjtu.edu.cn

Abstract

Continually expanding new languages for existing large language models (LLMs) is a promising yet challenging approach to building powerful multilingual LLMs. The biggest challenge is to make the model continuously learn new languages while preserving the proficient ability of old languages. To achieve this, recent work utilizes the Mixture-of-Experts (MoE) architecture to expand new languages by adding new experts and avoid catastrophic forgetting of old languages by routing corresponding tokens to the original model backbone (old experts). Although intuitive, this kind of method is parameter-costly when expanding new languages and still inevitably impacts the performance of old languages. To address these limitations, we analyze the language characteristics of different layers in LLMs and propose a layer-wise expert allocation algorithm (**LayerMoE**) to determine the appropriate number of new experts for each layer. Specifically, we find different layers in LLMs exhibit different representation similarities between languages and then utilize the similarity as the indicator to allocate experts for each layer, *i.e.*, the higher similarity, the fewer experts. Additionally, to further mitigate the forgetting of old languages, we add a classifier in front of the router network on the layers with higher similarity to guide the routing of old language tokens. Experimental results show that our method outperforms the previous state-of-the-art baseline with 60% fewer experts in the single-expansion setting and with 33.3% fewer experts in the lifelong-expansion setting, demonstrating the effectiveness of our method.

1 Introduction

Although existing large language models (LLMs) have exhibited remarkable ability in high-resource languages, their performance in multilingual scenarios is still limited (Lai et al., 2023b; Zhang et al., 2024c; Huang et al., 2025). To enhance the multilingual ability of LLMs, continually expanding new languages for existing LLMs is a sustainable approach, which does not require excessive computation or data resources than re-training from scratch (Huang et al., 2024). During this process, the key challenge is updating the model with new language data while avoiding catastrophic forgetting of the ability in originally proficient languages.

As a promising solution, MoE-LPR (Zhou et al., 2024), a two-stage training strategy, utilizes the Mixture-of-Experts (MoE) architecture to achieve expansion of new languages and seeks to preserve the ability of the original languages by targeted routing. Specifically, at the first stage, it adds and trains several new experts beyond the dense LLM to learn new languages. Then, at the second stage, it tries to recover the ability of old languages by language priors routing training. Although this method shows the potential of MoE on balancing the capability of newly expanded languages and the original languages, there are still two problems. For one thing, the expansion of MoE-LPR is inefficient and leads to high resource usage after expansion (*e.g.*, the original Qwen1.5-1.8B has to be 3.2× larger to accommodate three additional languages). For another, although MoE-LPR designs the language priors routing to recover the abilities of the original languages, the corresponding performance of old languages still declines noticeably.

Towards these limitations, we first analyze the similarity between hidden states in different languages and find that some layers exhibit high similarity. On this basis, we propose a layer-wise expert allocation algorithm (**LayerMoE**) to determine the appropriate number of new experts for each layer, with the insight that higher representation similarity requires fewer experts. Additionally, we speculate that high similarity between old and new languages
in some layers may confuse the router network, causing the forgetting of old languages. To further mitigate the forgetting, we add a classifier in front of the router network on the layers exhibiting higher similarity to guide the routing of tokens in old languages.

We conduct experiments under two settings: (1) single-expansion, where the model is enhanced once by incorporating a new language group, and (2) lifelong-expansion, where the model undergoes sequential adaptation to multiple new language groups. The corresponding experimental results on four benchmarks demonstrate the effectiveness of our LayerMoE. Specifically, in the single-expansion setting (please refer to Table 1), our method achieves superior performance with 60% fewer experts compared to the previous state-of-the-art method, i.e., better expansion of new languages, and less forgetting of old languages. In the lifelong-expansion setting (please refer to Table 2), our method also outperforms baselines, requiring 33.3% fewer experts. These results indicate that our expert allocation algorithm offers a resource-efficient solution for continuous language learning, and the classifier added in front of the router network further prevents the forgetting of old languages.

In summary, the major contributions of this paper are as follows\footnote{The code is publicly available at https://github.com/XZhang00/LayerMoE.}:

• We devise a layer-wise expert allocation algorithm to assign the appropriate number of new experts for each layer, which is parameter-efficient and achieves better expansion performance of new languages.

• We add a classifier in front of the router network on the layers that exhibit higher similarity, further mitigating the forgetting of old languages.

• Experimental results in the single-expansion and lifelong-expansion settings demonstrate the effectiveness and the efficiency of our LayerMoE.

2 Background

In this section, we first introduce the two-stage expansion method MoE-LPR (Zhou et al., 2024) and discuss its limitations.

Stage-1: Continual pre-training with MoE. The dense model is upcycled to an MoE model by adding new feed-forward networks (FFNs) and router networks. Formally, the MoE model includes \( N \) experts and a linear router network \( W_r \in \mathbb{R}^{h \times N} \) for each layer, where \( h \) is the dimension of hidden states after the Attention module. During the forward process, the top-\( K \) experts \( \mathcal{I} \) are dynamically selected by the router according to the router scores:

\[
G(\mathbf{x}) = \text{Softmax}(\mathbf{x} \cdot W_r),
\]

\[
\mathcal{I} = \{i \mid G_i(\mathbf{x}) \in \text{TopK}(G(\mathbf{x}), K)\},
\]

where \( \mathbf{x} \in \mathbb{R}^h \) is the input token hidden state vector of the router network, \( G(\mathbf{x}) \in \mathbb{R}^N \) represents the router scores of each expert, and \( K \) is the number of the activated experts (\( K = 2 \)). Then we normalize the weights of the selected experts and weighted sum their outputs to obtain the final representation:

\[
\mathbf{y} = \sum_{i \in \mathcal{I}} \frac{G_i(\mathbf{x})}{\sum_{j \in \mathcal{I}} G_j(\mathbf{x})} E_i(\mathbf{x}) + \mathbf{x},
\]

where \( G_i(\mathbf{x}) \) and \( E_i(\mathbf{x}) \) denote the router score and the output of the \( i \)-th expert respectively.

In stage-1, the parameters of the original dense model are frozen to avoid catastrophic forgetting and the newly added experts and routers are updated with the data of expanded languages. The training objective includes the next token prediction loss \( \mathcal{L}_{NTP} \) and the load balance loss \( \mathcal{L}_{balance} \) (Fedus et al., 2022). Given the MoE model \( \mathcal{M} \) with the parameters of the newly added experts \( \theta_e \) and the routers \( \theta_r \), the \( \mathcal{L}_{NTP} \) of one input sequence \( t \) is \( \mathcal{L}_{NTP}(\theta_e, \theta_r) = -\sum_i^{|t|} \log p_{\mathcal{M}}(t_i \mid t_{<i}) \). The \( \mathcal{L}_{balance} \) is an expert-level load balance loss (please refer to Appendix A for the detailed introduction) to mitigate the risk of routing collapse. The final training loss of stage-1 is:

\[
\mathcal{L}_{stage1}(\theta_e, \theta_r) = \mathcal{L}_{NTP} + \alpha \mathcal{L}_{balance},
\]

where \( \alpha \) is a hyper-parameter that controls the weight of the load balance loss (\( \alpha = 0.01 \)).

Stage-2: Review with LPR. In this stage, only the routers are updated with the mixed data of the old languages and the new languages. The language priors routing (LPR) loss is designed to route the old language tokens to the original frozen experts (i.e., the expert 0 in each layer). Formally, given a batch tokens \( \mathbb{T} \) including the old language
tokens set \( \mathbb{T}_{old} \) and the indicator function \( F(t) \), the LPR loss \( \mathcal{L}_{LPR} \) is defined as:

\[
\mathcal{L}_{LPR}(\theta_r) = - \sum_{t \in \mathbb{T}} F(t) \log G_0(t),
\]

\[
F(t) = \left\{ \begin{array}{ll}
1, & \text{if } t \in \mathbb{T}_{old}; \\
0, & \text{if } t \notin \mathbb{T}_{old};
\end{array} \right.
\]

where \( G_0(t) \) is the routing score of the expert 0. The final training loss of stage-2 is:

\[
\mathcal{L}_{stage2}(\theta_r) = \mathcal{L}_{NTP} + \beta \mathcal{L}_{LPR},
\]

where \( \beta \) is a hyper-parameter that controls the weight of the LPR loss (\( \beta = 0.1 \)).

Discussions about MoE-LPR. During stage-1, MoE-LPR directly adds uniformly distributed experts for each layer, i.e., each layer adds the same number of new experts. This expansion paradigm may lead to inefficient expansion and high resource usage, especially in lifelong learning settings. Inspired by the findings on language-specific and language-agnostic characteristics of different layers (Chen et al., 2023; Zhang et al., 2024b), we raise a question: do all layers need the same number of new experts? One intuition is that decreasing the number of experts in some layers may not hurt the overall performance, which motivates us to design an expert allocation algorithm. During stage-2, although the LPR loss is used to guide the routing of old language tokens, the performance of the old languages still declines noticeably. This suggests that the routing rules in some layers may be difficult for the router network to learn.

3 Methodology

To address the two limitations of MoE-LPR, we first analyze the similarity of hidden states between different languages across different layers. Then we introduce our layer-wise expert allocation algorithm based on the similarity. Lastly, we introduce the classifier ahead of the router network in the layers with high similarity to control the routing of the old languages.

3.1 Similarity Analysis

Considering the simplistic structure of the routing network, which essentially performs a linear transformation, the routing results are typically dependent on the input of the router network, i.e., the Hidden States obtained after the Attention module (HSAs). Therefore, we analyze the similarity of HSAs in different languages to investigate the potential differences of languages at different layers.

Given two languages \( \ell_1 \) and \( \ell_2 \), we randomly sample many examples for each language and perform forward propagation to obtain HSAs of each token at different layers. Then we randomly select \( Q \) tokens for each language and take their HSAs to build the candidate set \( \mathbf{Q}_{\ell_1} \) and \( \mathbf{Q}_{\ell_2} \) for calculating the similarity of these selected tokens in different languages. For example in layer \( i \), the similarity of \( \mathbf{Q}_{\ell_1}^i \) and \( \mathbf{Q}_{\ell_2}^i \) are obtained by calculating the cosine similarity of any two HSA vectors:

\[
\operatorname{sim}(\mathbf{Q}_{\ell_1}^i, \mathbf{Q}_{\ell_2}^i) = \frac{1}{Q^2} \sum_{u \in \mathbf{Q}_{\ell_1}^i} \sum_{v \in \mathbf{Q}_{\ell_2}^i} \cos(\mathbf{u}, \mathbf{v}),
\]

where \( \cos(\cdot) \) denotes the cosine similarity, and \( \mathbf{u}/\mathbf{v} \in \mathbb{R}^h \). These HSAs of randomly selected tokens contain different contextual information (Ethayarajh, 2019), which simulates the actual input of the router network.

As a preliminary study, we calculate the similarity of HSAs between some languages in different layers. We select Qwen1.5-1.8B (Team, 2024) as the base model and choose four languages including English (en), Spanish (es), Bengali (bn), and Greek (el). For each language \( \ell \), we sample examples from CulturalX (Nguyen et al., 2024a) and then randomly select \( Q = 100000 \) tokens to obtain the candidate set \( \mathbf{Q}_\ell \). Then we follow Eq.(8) to calculate the similarity between four language pairs, including en&en, en&es, en&bn, and bn&hi. The results in Figure 1 show that the similarity in different layers exhibits significant differences, with higher similarity in the middle and last few layers, and lower similarity in the 0~4 layers and 17~21 layers. en&en (the red line) displays the highest similarity across all layers among the four language pairs, while the other 3 language pairs show similar

![The similarity of HSAs in different languages across all layers within Qwen1.5-1.8B.](page_1012_186_388_282.png)

Figure 1: The similarity of HSAs in different languages across all layers within Qwen1.5-1.8B.
Figure 2: The procedure of our LayerMoE. In Step 1, we first calculate the indicated similarity for each layer. In Step 2, we allocate experts based on the similarity. The layers with lower similarity are allocated more experts, while those with higher similarity are assigned fewer experts. Additionally, we add a classifier in front of the router in the layers with higher similarity to guide the routing of old language tokens.

trends with en&en.

According to the similarity difference across all layers, we speculate that the higher similarity indicates that HSAs capture more language-agnostic information, while the lower similarity suggests more language-specific characteristics, which aligns with previous studies (Zhang et al., 2024b; Tang et al., 2024) that have observed the existence of language-agnostic/specific neurons within LLMs. We hypothesize that a layer with higher similarity needs fewer new experts when expanding new languages, as the model can extract language-agnostic representations in this layer. Conversely, lower similarity implies that more experts are necessary to accommodate language-specific features for the new languages. On this basis, we propose our LayerMoE.

3.2 Layer-wise Expert Allocation

Figure 2 shows the procedure of our method. When expanding new languages, we first calculate the similarity between new&new and new&old languages for each layer. Then we allocate appropriate experts for each layer according to the similarity.

Step 1: Calculate Similarity. Given the dense model \( \mathcal{M}_0 \), and its proficient language group \( L_{old} \) and current expanded language group \( L_{new} \), we first randomly select \( Q \) tokens to build the candidate token set \( Q_{old} \) and \( Q_{new} \). Then we calculate the similarity of \( Q_{new}\&Q_{old} \) and \( Q_{new}\&Q_{new} \), and compute their average as the indicated similarity of each layer. For example in layer \( i \), the indicated similarity \( S^i \) is calculated by:

\[
S^{i}_{new\&old} = \frac{\sum_j |L_{new}| \sum_k |L_{old}| \operatorname{sim}(Q_{ij}^i, Q_{lk}^k)}{|L_{new}||L_{old}|},
\]
\[
S^{i}_{new\&new} = \frac{\sum_j |L_{new}| \sum_{k,k\neq j} |L_{new}| \operatorname{sim}(Q_{ij}^i, Q_{lj}^j)}{2|L_{new}|(|L_{new}|-1)},
\]
\[
S^i = \frac{S^{i}_{new\&old} + S^{i}_{new\&new}}{2},
\]

where \( \operatorname{sim}(\cdot, \cdot) \) follows Eq.(8).

Step 2: Allocate Experts. Given the dense model with \( m \) layers, we obtain the indicated similarity vector \( S = [S^1, S^2, \ldots, S^m] \) after Step 1. Then numbers of experts across \( m \) layers \( N = [N^1, N^2, \ldots, N^m] \) is calculated as follows:

\[
N^i = \left\lceil \left( \frac{(S^i)^{-1}}{\sum_{i=1}^m (S^i)^{-1}} \right) \times \delta \right\rceil,
\]

where \( \delta \) is the target sum number of all experts such that \( \sum_{i=1}^m N^i = \delta \), and \( \lceil \cdot \rceil \) denotes round up to the
nearest integer. As a result, fewer experts will be allocated to the layers with higher similarity, while more experts to the layers with lower similarity. After allocating experts to each layer, we conduct the post-training (stage-1) using the data of new languages to obtain the MoE model \( \mathcal{M}_{stage1} \).

3.3 Routing Classification Network
During stage-2, the key objective is \( \mathcal{L}_{LPR} \) that trains the router network to route the tokens in old languages to old experts. To reduce the confusion of the routers on old languages and new languages, we add a routing classification network \( W_c \) in front of the router network to judge whether the current token belongs to old languages, where \( W_c \in \mathbb{R}^{h \times 2} \). After obtaining the model \( \mathcal{M}_{stage1} \), we first calculate the similarities \( S_{new\&old} = [S^1_{new\&old}; S^2_{new\&old}; \ldots; S^m_{new\&old}] \) between new and old languages following Eq.(9) for each layer. Then we add the classifier to the top-K layers according to \( \{i|S^i_{new\&old} \in \text{TopK}(S_{new\&old}, K)\} \). The training objective of \( W_c \) is:

\[
\mathcal{L}_{CLS}(\theta_c) = - \sum_{t \in \mathbb{T}} F(t) \log_{\text{softmax}}(\mathbf{x} \cdot W_c),
\]

where the indicator function \( F(t) \) is the same as Eq.(6), \( \mathbf{x} \) is the hidden state of the token t, and \( \theta_c \) denotes the parameters of the classifiers. Therefore, the final training loss of stage-2 is:

\[
\mathcal{L}_{stage2}(\theta_r, \theta_c) = \mathcal{L}_{NTP} + \beta \mathcal{L}_{LPR} + \gamma \mathcal{L}_{CLS},
\]

where \( \beta \) and \( \gamma \) are hyper-parameters that control the weights of \( \mathcal{L}_{LPR} \) and \( \mathcal{L}_{CLS} \).

During inference, the tokens classified to old languages are directly routed to the old expert, while others classified to new languages are normally routed following Eq.(3):

\[
y = \left\{
\begin{array}{ll}
E_0(\mathbf{x}) + \mathbf{x}, & cls(\mathbf{x}) = 0, \\
\sum_{i \in I} \frac{G_i(\mathbf{x})}{\sum_{j \in I} G_j(\mathbf{x})} E_i(\mathbf{x}) + \mathbf{x}, & cls(\mathbf{x}) = 1,
\end{array}
\right.
\]

where \( cls(\mathbf{x}) = \argmax(\mathbf{x} \cdot W_r) \), “0/1” denotes the current token belonging to “old/new” languages, and \( E_0 \) represents the old expert.

4 Experiments

4.1 Experimental Setup
We conduct the experiments under single-expansion and lifelong-expansion settings. The single-expansion setting means expanding one group of new languages to enhance the original model once, while the lifelong-expansion setting means sequentially adapting the model on multiple groups of new languages.

Model and Languages. Following MoE-LPR (Zhou et al., 2024), we select Qwen1.5-1.8B (Team, 2024) as our base model, which has a powerful multilingual tokenizer and lower computation overhead for easily upcycling. Based on the ability of Qwen1.5-1.8B, we select three languages with high performance as the old language group (G0) to observe the catastrophic forgetting phenomenon: English (en), Chinese (zh), and Spanish (es). Additionally, we choose six low-resource languages\(^2\) in which Qwen1.5-1.8B performs poorly as the expanded languages: Greek (el), Hungarian (hu), Turkish (tr), Bengali (bn), Hindi (hi), and Nepali (ne). Specifically, the selection of el, hu, and tr follows MoE-LPR (Zhou et al., 2024), while bn, hi, and ne are selected to observe the performance of different methods on morphologically rich non-Latin scripts, which pose greater modeling challenges and suffer from limited resources. Hence, we divide them into two groups (G1: el, hu, and tr, G2: bn, hi, and ne) for the single-expansion setting (G0→G1, G0→G2, and G0→G1+G2) and the lifelong-expansion setting (G0→G1→G2, and G0→G2→G1).

Training. During stage-1, we train the new experts and the router network with new language data. For each new language, we sample 2 billion tokens from CulturalX (Nguyen et al., 2024a), a substantial multilingual dataset with 6.3 trillion tokens in 167 languages. The detailed training setup follows MoE-LPR (Zhou et al., 2024): the batch size is 512, the sequence length is 1024, the learning rate is 5e-5, and the weight \( \alpha \) of the load balancing loss is 0.01. During stage-2, we mix the data of old languages and new languages to train the router network and the classification network. The top-7 layers with high similarity for the single-expansion settings and top-5 layers for the lifelong-expansion settings are taken according to the best performance to add the routing classification network. We randomly sample 50K examples for each old language and 100K examples for each new language as the training data of stage-2. Following MoE-LPR (Zhou et al., 2024),

\footnotetext{2The performance on six languages is shown in Table 1. More details of each language are listed in Appendix B.1.}
the English/Chinese/Spanish data is sampled from Slimpajam (Soboleva et al., 2023), SkyPile-150B (Wei et al., 2023a), and CulturalX (Nguyen et al., 2024a), respectively. The 100k examples for each new language are sampled from the training data of stage-1. We set the batch size to 768 and the learning rate to 5e-5. The weight of \( \mathcal{L}_{LPR} \) and \( \mathcal{L}_{CLS} \) loss is set to 0.1. All experiments are conducted on 8 × A100-40G GPUs.

Evaluation. We evaluate all methods on 4 benchmarks: ARC-Challenge (25-shot) (Clark et al., 2018), MMLU (5-shot) (Hendrycks et al., 2021), HellaSwag (10-shot) (Zellers et al., 2019), and Belebele (5-shot) (Bandarkar et al., 2024). We utilize the lm-evaluation-harness\(^3\) framework (Gao et al., 2024b) to conduct evaluation on their multilingual version. Please refer to Appendix B.2 for the detailed introduction.

4.2 Baselines

MoE-LPR (Zhou et al., 2024) adds the same number of new experts for each layer. Gao et al. proposes MOLA that experts in lower layers are more similar than those in higher layers, i.e., higher layers need more experts. For G0→G1, G0→G2, we conduct three versions of MoE-LPR, adding 2/3/5 experts per layer as 3*24/4*24/6*24 (Qwen1.5-1.8B contains 24 layers). And MoLA (\( \nabla_2 \) 2468) is employed following the best setting, allocating 2/4/6/8 experts for 0~5/6~11/12~17/18~23 layers. For G0→G1 + G2, G0→G1 → G2, and G0→G2→G1, we conduct MoE-LPR (7*24/4*24) as baselines. For a fair comparison, we reproduce all versions of MoE-LPR and MOLA using the same data as our method, i.e., 2 billion tokens for each new language.

Additionally, we directly take five baselines of MoE-LPR (trained with 8 billion tokens for each new language) under G0→G1 for a clear comparison, including LoRA (Hu et al., 2021), Full Fine-tuning, LLaMA-Pro (Wu et al., 2024), MoE, and LoRAMoE (Dou et al., 2024). LoRA (Hu et al., 2021) introduces Low-Rank Adaptation for efficiently fine-tuning large language models. In this paper, the LoRA targets include all linear modules. The LoRA rank is set to 8. Full Fine-tuning is fine-tuning all parameters on the dense model. LLaMA-Pro (Wu et al., 2024) insert new transformer blocks to replace the duplicate layers. Only the newly inserted blocks are trained to store new knowledge during post-pretraining. 12 new layers (the best setting) are inserted into the original dense model. MoE means that only the stage-1 of MoE-LPR is conducted. LoRAMoE (Dou et al., 2024) is a novel framework that combines multiple LoRAs with a router network. The router selects all LoRAs for each token. The LoRA rank is set to 180, and the number of LoRAs is 8 for each module. Due to the poor performance of the 5 baselines, we do not reproduce them and directly take the results trained with 8 billion tokens for each new language from the MoE-LPR paper (Zhou et al., 2024).

![The expert allocation of our method and baselines for each layer under G0→G1, G0→G2.](page_1012_186_377_246.png)

Figure 3: The expert allocation of our method and baselines for each layer under G0→G1, G0→G2.

4.3 Allocation of Experts

After calculating the similarity following Step 1, we assign the number of new experts for each layer according to Eq.(10). For a fair comparison with MoE-LPR (3*24), we set the targeted number of new experts \( \delta \) in Eq.(10) to 72. The allocation results of our LayerMoE and baselines (under G0→G1 and G0→G2) are present in Figure 3. Our method allocates more experts for the shallow and deep layers and less experts for the middle layers. The detailed similarity and expert allocation results of our method under each setting are plotted in Figure 4 of Appendix B.3.

5 Results and Analysis

5.1 Main Results

Single-Expansion. The evaluation results of the single-expansion setting (G0→G1 and G0→G2) are listed in Table 1. The results show that our method surpasses all baselines in “Old-avg” and “New-avg” under the expansion of G1 and G2. Particularly, our method outperforms the previous SOTA method “MoE-LPR (6*24)” with 60% fewer parameter numbers of newly added experts
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Expand-params</th>
    <th colspan="4">ARC-C</th>
    <th colspan="4">MMLU</th>
    <th colspan="4">Hellaswag</th>
    <th colspan="4">Belebele</th>
    <th colspan="4">Old-avg</th>
    <th colspan="4">New-avg</th>
    <th colspan="4">avg</th>
  </tr>
  <tr>
    <th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th><th>Old</th><th>New</th>
  </tr>
  <tr>
    <td colspan="25">Old (G0): en, es, zh → New (G1): el, hu, tr</td>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>/</td><td>33.31</td><td>22.93</td><td>47.23</td><td>30.52</td><td>49.70</td><td>29.27</td><td>55.89</td><td>32.92</td><td>46.53</td><td>28.91</td><td>37.72</td>
  </tr>
  <tr>
    <td>*LoRA</td><td>/</td><td>28.33</td><td>23.89</td><td>37.42</td><td>29.30</td><td>41.48</td><td>29.78</td><td>39.45</td><td>26.93</td><td>36.67</td><td>27.48</td><td>32.07</td>
  </tr>
  <tr>
    <td>*Full Fine-tuning</td><td>/</td><td>31.72</td><td>25.98</td><td>43.51</td><td>33.18</td><td>47.38</td><td>35.28</td><td>45.26</td><td>33.70</td><td>41.98</td><td>32.04</td><td>37.00</td>
  </tr>
  <tr>
    <td>*LLaMA-Pro</td><td>0.6B</td><td>31.77</td><td>24.35</td><td>44.06</td><td>34.02</td><td>48.36</td><td>33.85</td><td>48.78</td><td>31.52</td><td>43.24</td><td>30.94</td><td>37.09</td>
  </tr>
  <tr>
    <td>*MoE</td><td>4B</td><td>32.51</td><td>26.43</td><td>44.16</td><td>35.07</td><td>48.54</td><td>37.01</td><td>45.37</td><td>32.74</td><td>42.65</td><td>32.81</td><td>37.73</td>
  </tr>
  <tr>
    <td>*LoRAMoE</td><td>0.8B</td><td>32.43</td><td>26.63</td><td>45.41</td><td>34.17</td><td>48.61</td><td>37.17</td><td>47.74</td><td>32.81</td><td>43.55</td><td>32.70</td><td>38.12</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>4B</td><td>32.26</td><td>26.55</td><td>46.51</td><td>34.95</td><td>49.75</td><td>37.53</td><td>52.92</td><td>39.04</td><td>45.36</td><td>34.52</td><td>39.94</td>
  </tr>
  <tr>
    <td>MOLA (∇)</td><td>3.2B (↓ 20%)</td><td>32.49</td><td>27.00</td><td>46.38</td><td>33.98</td><td>49.63</td><td>37.03</td><td>53.85</td><td>38.37</td><td>45.59</td><td>34.10</td><td>39.84</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>2.4B (↓ 40%)</td><td>32.29</td><td>26.69</td><td>46.51</td><td>34.11</td><td>49.60</td><td>37.12</td><td>52.33</td><td>37.74</td><td>45.18</td><td>33.92</td><td>39.55</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>1.6B (↓ 60%)</td><td>32.77</td><td>26.43</td><td>46.54</td><td>34.59</td><td>49.63</td><td>37.08</td><td>52.07</td><td>37.15</td><td>45.25</td><td>33.81</td><td>39.53</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>1.6B (↓ 60%)</td><td>33.03</td><td>26.66</td><td>46.12</td><td>34.80</td><td>49.24</td><td>37.01</td><td>54.82</td><td>40.37</td><td>45.80</td><td>34.71</td><td>40.26</td>
  </tr>
  <tr>
    <td colspan="25">Old (G0): en, es, zh → New (G2): bn, hi, ne</td>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>/</td><td>33.31</td><td>23.05</td><td>47.23</td><td>29.54</td><td>49.70</td><td>28.22</td><td>55.89</td><td>28.85</td><td>46.53</td><td>27.41</td><td>36.97</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>4B</td><td>33.06</td><td>23.85</td><td>47.04</td><td>30.62</td><td>49.69</td><td>30.92</td><td>53.11</td><td>33.85</td><td>45.72</td><td>29.81</td><td>37.77</td>
  </tr>
  <tr>
    <td>MOLA (∇)</td><td>3.2B (↓ 20%)</td><td>32.48</td><td>23.67</td><td>47.02</td><td>30.82</td><td>49.73</td><td>30.48</td><td>53.59</td><td>32.15</td><td>45.71</td><td>29.28</td><td>37.49</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>2.4B (↓ 40%)</td><td>33.08</td><td>23.27</td><td>47.21</td><td>30.74</td><td>49.64</td><td>30.67</td><td>52.85</td><td>33.04</td><td>45.70</td><td>29.43</td><td>37.56</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>1.6B (↓ 60%)</td><td>32.77</td><td>23.53</td><td>46.97</td><td>30.72</td><td>49.68</td><td>30.56</td><td>52.59</td><td>33.33</td><td>45.50</td><td>29.53</td><td>37.52</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>1.6B (↓ 60%)</td><td>32.52</td><td>24.02</td><td>47.10</td><td>30.85</td><td>49.46</td><td>30.47</td><td>55.15</td><td>34.11</td><td>46.06</td><td>29.86</td><td>37.96</td>
  </tr>
</table>

Table 1: The results of the single-expansion settings (G0→G1, G0→G2). The results marked with * represent that they are directly taken from the MoE-LPR paper (Zhou et al., 2024), which utilizes 8 billion tokens for each new language during post-pretraining. Other experiments are trained with 2 billion tokens for each new language. “Expand-params” denotes the parameter number of newly added experts and (↓ %) means the reduction percentage of parameters compared to MoE-LPR (6*24). The results in bold mean they are better than the previous SOTA method “MoE-LPR (6*24)”. The detailed results for each language are reported in Table 7 and 8 of Appendix C.

(i.e., from 4B to 1.6B), which suggests that our method is more parameter-efficient. On the contrary, MOLA and MoE-LPR (3*24/4*24) underperform MoE-LPR (6*24) when decreasing the number of expansion parameters. Although other baselines (LoRA, Full Fine-tuning, LLaMA-Pro, MoE, and LoRAMoE) utilize 8B tokens for each new language during post-pretraining, they still perform poorly due to the lack of special designs. Additionally, when keeping the same parameter numbers (1.6B) with “MoE-LPR (3*24)”, our method exhibits a significant improvement over “MoE-LPR (3*24)” referring to “New-avg” (for G1 from 33.81 to 34.71 and for G2 from 29.53 to 29.86). This superiority indicates that allocating experts based on the language similarity in different layers has a positive contribution to the expansion of new languages. Furthermore, the results of “Old-avg” showcase that our method performs the best on preserving the ability of the old languages, with 45.80 for expanding G1 and 46.06 for G2. These results prove that appropriate correction of the router network can effectively mitigate the catastrophic forgetting of old languages.

The expansion performance exhibits differences among different languages according to “New-avg”. When conducting G0→G1, the performance of new languages improves by 5.8 from 28.91 to 34.71, while only improves by 2.45 (from 27.41 to 29.86) on G0→G2. Although the amounts of training tokens (2B) for each new language are the same, the morphologically rich non-Latin languages in G2 (bn, hi, ne) obtain a smaller performance improvement compared to expanding G1. The difference can be attributed to their lower encoding efficiencies (Arnett and Bergen, 2024; Rust et al., 2021; Ahia et al., 2023; Petrov et al., 2023; Ali et al., 2024), i.e., these languages typically employ UTF-8 encoding during the tokenization process, which results in longer encoded sequences. Consequently, the number of effective training tokens is reduced, limiting the potential for performance improvement. These results suggest that more tokens are needed for better expansion of G2. Furthermore, when expanding G1 or G2, there are different degrees of forgetting of old languages. Expanding G1 brings more forgetting (from 46.53 to 45.80) compared to expanding G2 (from 46.53 to 46.06), which may be caused by more token overlaps between G0 and G1 since the writing systems of en/es in G0 and hu/tr in G1 all contain the Latin alphabet (as shown in Table 6 of Appendix B.1).

Lifelong-Expansion. The results of G0→G1→G2 and G0→G2→G1 in Table 2 demonstrate that
<table>
  <tr>
    <th>Methods</th>
    <th>Order</th>
    <th>G1-E-p</th>
    <th>G2-E-p</th>
    <th>All-E-p</th>
    <th>G0-avg</th>
    <th>G1-avg</th>
    <th>G2-avg</th>
    <th>avg</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>/</td>
    <td>46.53</td>
    <td>28.91</td>
    <td>27.41</td>
    <td>34.29</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td>
    <td>G0→G1+G2</td>
    <td>4.8B</td>
    <td>4.8B</td>
    <td>4.8B</td>
    <td>45.15</td>
    <td>34.17</td>
    <td>29.98</td>
    <td>36.43</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td>
    <td>G0→G1+G2</td>
    <td>2.4B</td>
    <td>2.4B</td>
    <td>2.4B (↓ 50.0%)</td>
    <td>45.28</td>
    <td>34.07</td>
    <td>29.55</td>
    <td>36.30</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td>
    <td>G0→G1+G2</td>
    <td>1.6B</td>
    <td>1.6B</td>
    <td><b>1.6B (↓ 66.7%)</b></td>
    <td><b>45.56</b></td>
    <td><b>34.73</b></td>
    <td><b>29.94</b></td>
    <td><b>36.74</b></td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td>
    <td>G0→G1→G2</td>
    <td>2.4B</td>
    <td>2.4B</td>
    <td>4.8B</td>
    <td>45.88</td>
    <td>34.40</td>
    <td>30.26</td>
    <td>36.85</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td>
    <td>G0→G1→G2</td>
    <td>1.6B</td>
    <td>1.6B</td>
    <td><b>3.2B (↓ 33.3%)</b></td>
    <td><b>46.10</b></td>
    <td><b>34.56</b></td>
    <td><b>30.29</b></td>
    <td><b>36.98</b></td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td>
    <td>G0→G2→G1</td>
    <td>2.4B</td>
    <td>2.4B</td>
    <td>4.8B</td>
    <td>45.28</td>
    <td>33.42</td>
    <td>28.39</td>
    <td>35.70</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td>
    <td>G0→G2→G1</td>
    <td>1.6B</td>
    <td>1.6B</td>
    <td><b>3.2B (↓ 33.3%)</b></td>
    <td><b>46.25</b></td>
    <td><b>33.76</b></td>
    <td><b>29.80</b></td>
    <td><b>36.60</b></td>
  </tr>
</table>

Table 2: The results of G0→G1+G2 and the lifelong-expansion settings (G0→G1→G2 and G0→G2→G1). “G1/G2/All-E-p” represents the expansion number of newly added parameters for G1, G2, and the sum, respectively. The detailed results for each language are reported in Table 9 of Appendix C.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="2">Old-avg (↑)</th>
    <th>New-avg (↑)</th>
    <th>avg (↑)</th>
  </tr>
  <tr>
    <th>LayerMoE (G0→G1)</th>
    <th>w/ random</th>
    <th>w/o classifier</th>
    <th>LayerMoE (G0→G2)</th>
    <th>w/ random</th>
    <th>w/o classifier</th>
  </tr>
  <tr>
    <td></td>
    <td>45.80</td>
    <td>45.68 (↓ 0.12)</td>
    <td>45.33 (↓ 0.47)</td>
    <td>34.71</td>
    <td>34.11 (↓ 0.60)</td>
    <td>34.61 (↓ 0.10)</td>
    <td>40.26</td>
    <td>39.90</td>
    <td>39.97</td>
  </tr>
  <tr>
    <td></td>
    <td>46.06</td>
    <td>45.94 (↓ 0.12)</td>
    <td>45.39 (↓ 0.67)</td>
    <td>29.86</td>
    <td>29.15 (↓ 0.71)</td>
    <td>29.53 (↓ 0.33)</td>
    <td>37.96</td>
    <td>37.54</td>
    <td>37.46</td>
  </tr>
</table>

Table 3: The ablation results under G0→G1 and G0→G2. “w/ random” means randomly allocating numbers of new experts for each layer without calculating the similarity. “w/o classifier” denotes no classifier is added in front of the router network.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">Old</th>
    <th colspan="3">New</th>
    <th colspan="3">avg</th>
  </tr>
  <tr>
    <th>top-1</th>
    <th>top-2</th>
    <th>top-3</th>
    <th>top-4</th>
    <th>top-5</th>
    <th>top-6</th>
    <th>top-7</th>
    <th>random-7</th>
    <th>last-7</th>
  </tr>
  <tr>
    <td></td>
    <td>44.90</td>
    <td>44.81</td>
    <td>45.10</td>
    <td>45.41</td>
    <td>45.28</td>
    <td>45.74</td>
    <td>45.38</td>
    <td>45.32</td>
    <td>44.96</td>
  </tr>
  <tr>
    <td></td>
    <td>34.30</td>
    <td>34.69</td>
    <td>34.41</td>
    <td>34.67</td>
    <td>34.66</td>
    <td>34.72</td>
    <td>34.30</td>
    <td>34.60</td>
    <td>34.60</td>
  </tr>
  <tr>
    <td></td>
    <td>39.60</td>
    <td>39.75</td>
    <td>39.76</td>
    <td>40.04</td>
    <td>39.97</td>
    <td>40.23</td>
    <td>39.78</td>
    <td>39.96</td>
    <td>39.78</td>
  </tr>
</table>

Table 4: The results of adding classifiers in different layers under G0→G1. “top-*” denotes adding classifiers to * layers with the highest similarity. “random-7” means randomly selecting 7 layers, while “last-7” represents the 7 layers with the lowest similarity.

our method outperforms MoE-LPR (7*24) with fewer 33.3% new experts in terms of “\( G0/G1/G2\)-avg”, proving the effectiveness of our method in lifelong-expansion setting. Additionally, we surprisingly observe that G0→G1→G2 brings better expansion performance for G2 (“G2-avg”) and less forgetting of old languages (“\( G0\)-avg”) compared to G0→G1+G2. But G0→G2→G1 does not show similar improvement for G1. This difference indicates that the learning order of different languages may have non-trivial influences on the final performance of each language.

5.2 Ablation Study

To verify the effectiveness of our expert allocation algorithm and the added classifier, we conduct the ablation experiments under G0→G1 and G0→G2 settings and report the results in Table 3. These results show that randomly allocating experts for each layer without calculating the similarity significantly decreases the expansion performance (↓ 0.60 in G1, ↓ 0.71 in G2). Additionally, the forgetting of old languages becomes worse (↓ 0.47 in G1, ↓ 0.67 in G2) when no classifiers are added in front of the router network. Overall, these results prove the necessity of our layer-wise expert allocation algorithm and the added classifier.

5.3 Classifiers in Different Layers

The top-K layers with the higher similarity between new and old languages are selected according to \(\{i|S_{new&old}^i \in \text{TopK}(S_{new&old}, K)\}\) to add the classifier. To investigate the best value of \(K\), we conduct the experiments from top-1 to top-11 and top-24 (all layers) under G0→G1 and list the results in Table 4. The results show that as more classifiers are added, the performance of the new language expansion first increases and then decreases, and the performance of the old language preservation increases constantly. The best overall performance peaks on the top-7 setting. Additionally, we also report the results of “random-7” and “last-7” to verify the effectiveness of adding classifiers in layers with higher similarity. Randomly selecting layers or opting for those with lower similarity consistently yields poor performance in preserving the capability of old languages.

5.4 Generalization Study

To evaluate the generalization of our method on different models, we conduct a brief verification of our method on G0→G1 with Llama-3.2-3B
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="2">ARC-C</th>
    <th colspan="2">MMLU</th>
    <th colspan="2">Hellaswag</th>
    <th colspan="2">Belebele</th>
    <th colspan="2">FLORES</th>
    <th colspan="3">Old-avg New-avg avg</th>
  </tr>
  <tr>
    <th>Old</th><th>New</th>
    <th>Old</th><th>New</th>
    <th>Old</th><th>New</th>
    <th>Old</th><th>New</th>
    <th>Old</th><th>New</th>
    <th>Old-avg</th><th>New-avg</th><th>avg</th>
  </tr>
  <tr>
    <td>Llama-3.2-3B</td>
    <td>44.64</td><td>33.39</td>
    <td>49.84</td><td>42.15</td>
    <td>63.56</td><td>43.62</td>
    <td>70.48</td><td>61.37</td>
    <td>84.88</td><td>83.96</td>
    <td>62.68</td><td>52.90</td><td>57.79</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td>
    <td>43.70</td><td>37.58</td>
    <td>49.99</td><td>43.98</td>
    <td>64.17</td><td>51.19</td>
    <td>70.74</td><td>64.41</td>
    <td>84.58</td><td>84.35</td>
    <td>62.64</td><td>56.30</td><td>59.47</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td>
    <td>44.10</td><td>37.64</td>
    <td>49.96</td><td>44.08</td>
    <td>64.07</td><td>51.33</td>
    <td>71.22</td><td>64.63</td>
    <td>84.72</td><td>84.69</td>
    <td>62.81</td><td>56.47</td><td>59.64</td>
  </tr>
</table>

Table 5: The results of Llama-3.2-3B under the single-expansion setting (G0→G1). The detailed results for each language are reported in Table 10 and 11 of Appendix C.

(Grattafiori et al., 2024) and evaluate the performance on 5 benchmarks. Particularly, we add the machine translation task FLORES (Team, 2022) to prove the effectiveness of our method on generation tasks. Specifically, we set the following directions: en → zh, zh → en, en → es, es → en, en → el, el → en, en → hu, hu → en, en → tr, and tr → en. And we utilize COMET (Rei et al., 2020) as the evaluation metric. The results in Table 5 demonstrate the effectiveness of our method across different LLMs and generative tasks, which further proves the practical application value of our method.

6 Related Work

There are two categories (Huang et al., 2024) in expanding new languages to build powerful multilingual LLMs: (1) utilizing the collected multilingual corpus to train the multilingual LLMs from scratch and (2) continually training LLMs on top of existing LLMs only with new language data. The first category generally necessitates meticulous design in terms of the data ratio between different languages (AI et al., 2024; Zhang et al., 2024a; Nguyen et al., 2024b) and the implementation of curriculum learning strategies (Wei et al., 2023b; Üstün et al., 2024) during the training process to ensure a balanced improvement in proficiency across various languages. Training from scratch typically requires excessive computational resources and training durations, which are often challenging to implement and environmentally unfriendly.

The second category is to continually train the original model with new language data to enhance the ability of new languages while avoiding the catastrophic forgetting of the ability in originally proficient languages. MoE-LPR (Zhou et al., 2024) and MoE-CT (Li et al., 2024) upcycle the dense model to the MoE model for improving the multilingual ability of LLMs. Similarly, some methods also have introduced external parameters to improve the ability of the original LLMs and preserve the original capabilities by freezing the old parameters. LLaMA-Pro (Wu et al., 2024) insert new transformer blocks to replace the duplicate layers to further enhance the original LLM. LoRAMoE (Dou et al., 2024) combines multiple low-rank adapters with a router network to achieve better performance on downstream tasks. In this work, we only focus on the multilingual expansion and propose LayerMoE to address the limitations of existing methods, e.g., inefficient expansion, high resource usage, and accumulated forgetfulness of old languages. Additionally, the concurrent work AlphaLoRA (Qing et al., 2024) similar to ours allocates LoRA experts based on Heavy-Tailed Self-Regularization Theory to mitigate redundancy of LoRA experts and improve the general ability of LLMs. Differently, we calculate the representation similarity of different languages in different layers as the indicator to allocate new experts for each layer, which is intuitive and effective for the expansion of new languages.

7 Conclusion

In this work, we propose a layer-wise expert allocation algorithm to assign the appropriate number of new experts for each layer when expanding new languages for existing LLMs. We first calculate the similarity of different languages for each layer as the indicator and allocate more experts to the layers exhibiting lower similarity. Additionally, we add a classifier in front of the router network for the layers with higher similarity to further mitigate the forgetting of old languages. Experimental results demonstrate the effectiveness of our method, which is parameter-efficient and achieves good expansion of new languages and less forgetting of old languages.

Limitations

Although our method achieves better improvements in terms of “Old-avg” and “New-avg” than MoE-LPR (6*24) with 60% fewer experts, the improvements in different benchmarks and languages are inconsistent. The inherent reasons for the inconsistency are not yet clear and need further inves-
tigation. Additionally, we set the total number of new experts according to the baseline “MoE-LPR (3*24)” for a fair comparison rather than exploring the optimal total number. In the future, we will continually solve these limitations for a better expert allocation solution.

Acknowledgments

The research work described in this paper has been supported by the National Nature Science Foundation of China (No. 62476023, 61976016, 62376019, 61976015) and Henan Provincial Science and Technology Research Project (No. 252102210102), and the authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper.

References

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904–9923, Singapore. Association for Computational Linguistics.

01. AI, .; Alex Young, Bei Chen, Chao Li, Chengu Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. Yi: Open foundation models by 01.ai. Preprint, arXiv:2403.04652.

Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Ruttmann, Max Lübbing, Johannes Levelling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. 2024. Tokenizer choice for LLM training: Negligible or crucial? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3907–3924, Mexico City, Mexico. Association for Computational Linguistics.

Catherine Arnett and Benjamin K. Bergen. 2024. Why do language models perform worse for morphologically complex languages? Preprint, arXiv:2411.14198.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 749–775. Association for Computational Linguistics.

Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. Preprint, arXiv:2308.13198.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. Preprint, arXiv:2312.09979.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961.

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiayouan Guo, Jie Yang, and VS Subrahmanian. 2024a. Higher layers need more lora experts. Preprint, arXiv:2402.08562.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024b. A framework for few-shot language model evaluation.

Aaron Grattafori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2024. A survey on large language models with multilingualism: Recent advances and new frontiers. Preprint, arXiv:2405.10936.

Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2025. A survey on large language models with multilingualism: Recent advances and new frontiers. Preprint, arXiv:2405.10936.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023a. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veysheh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023b. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.

Tianhao Li, Shangjie Li, Binbin Xie, Deyi Xiong, and Baosong Yang. 2024. Moe-ct: A novel approach for large language models training with resistance to catastrophic forgetting. Preprint, arXiv:2407.00875.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024a. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL.

Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhijiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024b. Seallms – large language models for southeast asia. Preprint, arXiv:2312.00738.

Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Preprint, arXiv:2305.15425.

Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. 2024. AlphaLoRA: Assigning LoRA experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20511–20523, Miami, Florida, USA. Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.

Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5701–5715, Bangkok, Thailand. Association for Computational Linguistics.

NLLB Team. 2022. No language left behind: Scaling human-centered machine translation.

Qwen Team. 2024. Introducing qwen1.5.

Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemiloke Onilude, Neel Bhandari, Shivalka Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muenmighoff, Marzieh Faddee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics.

Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwai Lu, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. 2023a. Skywork: A more open bilingual foundation model. Preprint, arXiv:2310.19341.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023b. Polylm: An open source polyglot large language model. Preprint, arXiv:2307.06018.

Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. Llama pro: Progressive llama with block expansion. Preprint, arXiv:2401.02415.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Shaolei Zhang, Kehao Zhang, Qingkai Fang, Shoutao Guo, Yan Zhou, Xiaodong Liu, and Yang Feng. 2024a. Bayling 2: A multilingual large language model with efficient language alignment. Preprint, arXiv:2411.16300.

Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2024b. Multilingual knowledge editing with language-agnostic factual neurons. Preprint, arXiv:2406.16416.

Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024c. Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11189–11204, Bangkok, Thailand. Association for Computational Linguistics.

Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. 2024. Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. Preprint, arXiv:2408.11396.
A  Load Balance Loss

The load balance loss (Fedus et al., 2022) generally mitigates the risk of routing collapse by constraining an even distribution of tokens across different experts. Given a batch tokens \( \mathbb{T} \) and \( N \) experts for each layer, the load balance loss \( \mathcal{L}_{balance} \) (Zhou et al., 2024) is calculated following:

\[
\mathcal{L}_{balance} = \sum_{i=1}^{N} f_i P_i,
\] (14)

\[
f_i = \frac{N}{K|\mathbb{T}|} \sum_{t \in \mathbb{T}} \mathbbm{1}\{\text{Token } t \text{ selects expert } i\},
\] (15)

\[
P_i = \frac{1}{|\mathbb{T}|} \sum_{t \in \mathbb{T}} G_i(t),
\] (16)

where \( t \) represents a token, \( G_i(t) \) is the router score of the expert \( i \), and \( \mathbbm{1}(\cdot) \) is a indicator function when the token \( t \) selecting the expert \( i \) equals 1.

B  Experimental Details

B.1  Detailed Introduction of Different Languages

The language families and writing systems of our nine selected languages are listed in Table 6. Specifically, \( bn \), \( hi \), and \( ne \) all belong to the Indo-Aryan branch of the Indo-European family and use non-Latin scripts, making them more challenging to model effectively and less well-resourced compared to Latin-script languages.

<table>
  <tr>
    <th>Languages</th>
    <th>Language Family</th>
    <th>Writing System</th>
  </tr>
  <tr>
    <td>English (<i>en</i>)</td>
    <td>Indo-European (Germanic)</td>
    <td>Latin alphabet (26 letters)</td>
  </tr>
  <tr>
    <td>Chinese (<i>zh</i>)</td>
    <td>Sino-Tibetan (Sinitic)</td>
    <td>Chinese characters</td>
  </tr>
  <tr>
    <td>Spanish (<i>es</i>)</td>
    <td>Indo-European (Romance)</td>
    <td>Latin alphabet (27 letters, including ñ)</td>
  </tr>
  <tr>
    <td>Greek (<i>el</i>)</td>
    <td>Indo-European (Hellenic)</td>
    <td>Greek alphabet (24 letters)</td>
  </tr>
  <tr>
    <td>Hungarian (<i>hu</i>)</td>
    <td>Uralic (Finno-Ugric)</td>
    <td>Latin alphabet with diacritics (40 letters)</td>
  </tr>
  <tr>
    <td>Turkish (<i>tr</i>)</td>
    <td>Turkic</td>
    <td>Latin alphabet (29 letters)</td>
  </tr>
  <tr>
    <td>Bengali (<i>bn</i>)</td>
    <td>Indo-European (Indo-Aryan)</td>
    <td>Bengali script</td>
  </tr>
  <tr>
    <td>Nepali (<i>ne</i>)</td>
    <td>Indo-European (Indo-Aryan)</td>
    <td>Devanagari script</td>
  </tr>
  <tr>
    <td>Hindi (<i>hi</i>)</td>
    <td>Indo-European (Indo-Aryan)</td>
    <td>Devanagari script</td>
  </tr>
</table>

Table 6: The detailed language families and writing systems.

B.2  Evaluation Details

The lm-evaluation-hardness framework has included the multilingual version of ARC-Challenge, MMLU, and HellaSwag translated by Okapi (Lai et al., 2023a). For unsupported languages in new languages, we manually integrate the Turkish version from OpenLLM Turkish leaderboard<sup>4</sup> and the Greek version translated by ILSP<sup>5</sup> following the official guide<sup>6</sup>.

B.3  Similarity and Allocation

Figure 4 presents the detailed similarity and expert allocation of our method for each layer under the five settings.

<sup>4</sup>https://huggingface.co/spaces/malhajar/OpenLLMTurkishLeaderboard
<sup>5</sup>https://huggingface.co/ilsp
<sup>6</sup>https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md
Figure 4: The detailed similarity and expert allocation of our method for each layer of Qwen1.5-1.8B under the five settings.

C  Detailed Results

Qwen1.5-1.8B:  The detailed results for each language under the single-expansion settings (G0→G1, G0→G2) are listed in Table 7 and 8. The detailed results of G0→G1+G2 and the lifelong-expansion settings (G0→G1→G2 and G0→G2→G1) are reported in Table 9.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">ARC-Challenge</th>
    <th colspan="3">MMLU</th>
    <th colspan="3">Hellaswag</th>
    <th colspan="3">Belebele</th>
    <th rowspan="2">G0-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>37.97</td><td>27.86</td><td>34.10</td><td>45.67</td><td>38.81</td><td>57.21</td><td>61.62</td><td>39.35</td><td>48.14</td><td>60.78</td><td>47.44</td><td>59.44</td><td>46.53</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>38.05</td><td>26.84</td><td>31.88</td><td>45.77</td><td>38.26</td><td>55.50</td><td>61.34</td><td>39.41</td><td>48.49</td><td>58.44</td><td>45.89</td><td>54.44</td><td>45.36</td>
  </tr>
  <tr>
    <td>MOLA</td><td>37.88</td><td>26.84</td><td>32.74</td><td>45.81</td><td>38.30</td><td>55.02</td><td>61.36</td><td>39.27</td><td>48.26</td><td>60.11</td><td>47.11</td><td>54.33</td><td>45.59</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>36.69</td><td>27.35</td><td>32.82</td><td>46.09</td><td>38.57</td><td>54.87</td><td>61.17</td><td>39.37</td><td>48.27</td><td>58.33</td><td>44.22</td><td>54.44</td><td>45.18</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>37.46</td><td>27.35</td><td>33.50</td><td>46.00</td><td>38.67</td><td>54.94</td><td>61.20</td><td>39.43</td><td>48.25</td><td>59.44</td><td>44.44</td><td>52.33</td><td>45.25</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>38.57</td><td>26.58</td><td>33.93</td><td>45.81</td><td>37.97</td><td>54.59</td><td>60.90</td><td>38.98</td><td>47.84</td><td>60.56</td><td>46.89</td><td>57.00</td><td>45.80</td>
  </tr>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th>G1-avg</th>
  </tr>
  <tr>
    <th>el</th><th>hu</th><th>tr</th>
    <th>el</th><th>hu</th><th>tr</th>
    <th>el</th><th>hu</th><th>tr</th>
    <th>el</th><th>hu</th><th>tr</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>22.95</td><td>23.20</td><td>22.63</td><td>28.94</td><td>32.25</td><td>30.37</td><td>29.15</td><td>29.63</td><td>29.04</td><td>33.33</td><td>32.00</td><td>33.44</td><td>28.91</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>25.26</td><td>27.14</td><td>27.24</td><td>32.06</td><td>36.30</td><td>36.49</td><td>36.09</td><td>38.63</td><td>37.87</td><td>38.33</td><td>36.56</td><td>42.22</td><td>34.52</td>
  </tr>
  <tr>
    <td>MOLA</td><td>25.94</td><td>27.40</td><td>27.67</td><td>31.43</td><td>35.80</td><td>34.71</td><td>35.85</td><td>37.79</td><td>37.46</td><td>35.22</td><td>39.00</td><td>40.89</td><td>34.10</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>25.34</td><td>27.57</td><td>27.16</td><td>31.58</td><td>35.79</td><td>34.97</td><td>35.70</td><td>38.26</td><td>37.39</td><td>34.44</td><td>40.22</td><td>38.56</td><td>33.92</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>24.32</td><td>27.05</td><td>27.92</td><td>31.90</td><td>36.27</td><td>35.59</td><td>35.55</td><td>38.18</td><td>37.51</td><td>32.00</td><td>39.56</td><td>39.89</td><td>33.81</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>25.34</td><td>26.80</td><td>27.84</td><td>31.96</td><td>36.76</td><td>35.68</td><td>35.61</td><td>38.10</td><td>37.31</td><td>37.44</td><td>41.56</td><td>42.11</td><td>34.71</td>
  </tr>
</table>

Table 7: The detailed results of Qwen1.5-1.8B under the G0→G1 setting for each language.

Llama-3.2-3B:  Figure 5 presents the detailed similarity and expert allocation of our method for each layer of Llama-3.2-3B under the G0→G1 setting. The detailed results for each language under the G0→G1 setting are listed in Table 10 and 11.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">ARC-Challenge</th>
    <th colspan="3">MMLU</th>
    <th colspan="3">Hellaswag</th>
    <th colspan="3">Belebele</th>
    <th rowspan="2">G0-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>37.97</td><td>27.86</td><td>34.10</td><td>45.67</td><td>38.81</td><td>57.21</td><td>61.62</td><td>39.35</td><td>48.14</td><td>60.78</td><td>47.44</td><td>59.44</td><td>46.53</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>37.29</td><td>27.26</td><td>34.62</td><td>46.05</td><td>38.46</td><td>56.60</td><td>61.73</td><td>39.27</td><td>48.07</td><td>59.67</td><td>46.33</td><td>53.33</td><td>45.72</td>
  </tr>
  <tr>
    <td>MOLA</td><td>37.37</td><td>27.26</td><td>32.82</td><td>45.88</td><td>38.71</td><td>56.48</td><td>61.76</td><td>39.09</td><td>48.34</td><td>59.56</td><td>46.22</td><td>55.00</td><td>45.71</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>37.80</td><td>27.18</td><td>34.27</td><td>46.00</td><td>38.88</td><td>56.76</td><td>61.65</td><td>39.03</td><td>48.24</td><td>59.67</td><td>45.22</td><td>53.67</td><td>45.70</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>37.80</td><td>27.09</td><td>33.42</td><td>45.52</td><td>38.62</td><td>56.78</td><td>61.59</td><td>39.13</td><td>48.32</td><td>59.89</td><td>45.33</td><td>52.56</td><td>45.50</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>36.86</td><td>27.44</td><td>33.25</td><td>46.15</td><td>39.05</td><td>56.09</td><td>61.50</td><td>39.11</td><td>47.77</td><td>61.89</td><td>46.67</td><td>56.89</td><td>46.06</td>
  </tr>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">bn</th>
    <th colspan="3">hi</th>
    <th colspan="3">ne</th>
    <th colspan="3">G2-avg</th>
  </tr>
  <tr>
    <th>bn</th><th>hi</th><th>ne</th>
    <th>bn</th><th>hi</th><th>ne</th>
    <th>bn</th><th>hi</th><th>ne</th>
    <th>bn</th><th>hi</th><th>ne</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>22.50</td><td>23.46</td><td>23.18</td><td>29.59</td><td>29.89</td><td>29.13</td><td>28.23</td><td>28.48</td><td>27.95</td><td>29.67</td><td>30.00</td><td>26.89</td><td>27.41</td>
  </tr>
  <tr>
    <td>MoE-LPR (6*24)</td><td>24.64</td><td>25.26</td><td>21.64</td><td>30.29</td><td>30.57</td><td>31.01</td><td>29.90</td><td>32.78</td><td>30.08</td><td>33.22</td><td>35.22</td><td>33.11</td><td>29.81</td>
  </tr>
  <tr>
    <td>MOLA</td><td>24.29</td><td>25.09</td><td>21.64</td><td>31.01</td><td>30.67</td><td>30.77</td><td>29.44</td><td>32.54</td><td>29.46</td><td>31.00</td><td>33.33</td><td>32.11</td><td>29.28</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>23.52</td><td>24.91</td><td>21.39</td><td>30.73</td><td>30.54</td><td>30.94</td><td>30.02</td><td>32.55</td><td>29.45</td><td>32.11</td><td>34.33</td><td>32.67</td><td>29.43</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>24.98</td><td>24.57</td><td>21.04</td><td>31.59</td><td>30.87</td><td>29.70</td><td>29.55</td><td>32.68</td><td>29.44</td><td>32.44</td><td>34.33</td><td>33.22</td><td>29.53</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>24.21</td><td>25.26</td><td>22.58</td><td>31.10</td><td>30.43</td><td>31.01</td><td>29.31</td><td>32.42</td><td>29.67</td><td>33.44</td><td>33.44</td><td>35.44</td><td>29.86</td>
  </tr>
</table>

Table 8: The detailed results of Qwen1.5-1.8B under the G0→G2 setting for each language.

![Bar and line chart showing the number of experts and similarity for each layer of Llama-3.2-3B under the G0→G1 setting.](page_1012_1342_420_312.png)

Figure 5: The detailed similarity and expert allocation of our method for each layer of Llama-3.2-3B under the G0→G1 setting.
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Order</th>
    <th colspan="3">ARC-Challenge</th>
    <th colspan="3">MMLU</th>
    <th colspan="3">Hellaswag</th>
    <th colspan="3">Belebele</th>
    <th rowspan="2">G0-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>/</td><td>37.97</td><td>27.86</td><td>34.10</td><td>45.67</td><td>38.81</td><td>57.21</td><td>61.62</td><td>39.35</td><td>48.14</td><td>60.78</td><td>47.44</td><td>59.44</td><td>46.53</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1+G2</td><td>37.88</td><td>26.92</td><td>33.85</td><td>45.26</td><td>37.61</td><td>54.51</td><td>61.43</td><td>39.29</td><td>48.14</td><td>59.33</td><td>44.56</td><td>53.00</td><td>45.15</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>G0→G1+G2</td><td>37.80</td><td>27.69</td><td>32.82</td><td>45.29</td><td>38.12</td><td>54.39</td><td>61.45</td><td>39.11</td><td>48.20</td><td>59.89</td><td>44.33</td><td>54.22</td><td>45.28</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1+G2</td><td>37.46</td><td>27.01</td><td>34.27</td><td>45.63</td><td>38.14</td><td>54.18</td><td>61.37</td><td>39.08</td><td>47.86</td><td>60.11</td><td>46.22</td><td>55.44</td><td>45.56</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1→G2</td><td>37.54</td><td>26.84</td><td>33.16</td><td>45.71</td><td>38.29</td><td>55.42</td><td>61.43</td><td>39.21</td><td>48.12</td><td>61.22</td><td>47.33</td><td>56.33</td><td>45.88</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1→G2</td><td>37.29</td><td>26.58</td><td>33.93</td><td>45.69</td><td>38.86</td><td>55.40</td><td>61.23</td><td>39.23</td><td>48.05</td><td>62.22</td><td>48.00</td><td>56.67</td><td>46.10</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G2→G1</td><td>38.14</td><td>27.18</td><td>33.42</td><td>46.07</td><td>38.26</td><td>55.28</td><td>61.36</td><td>39.44</td><td>48.26</td><td>57.44</td><td>47.78</td><td>53.67</td><td>45.28</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G2→G1</td><td>38.31</td><td>27.09</td><td>34.70</td><td>46.19</td><td>38.79</td><td>55.85</td><td>61.19</td><td>39.05</td><td>48.05</td><td>61.89</td><td>46.44</td><td>57.44</td><td>46.25</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Order</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th rowspan="2">GI-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>/</td><td>22.95</td><td>23.20</td><td>22.63</td><td>28.94</td><td>32.25</td><td>30.37</td><td>29.15</td><td>29.63</td><td>29.04</td><td>33.33</td><td>32.00</td><td>33.44</td><td>28.91</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1+G2</td><td>26.46</td><td>28.00</td><td>27.92</td><td>30.38</td><td>35.14</td><td>34.64</td><td>36.40</td><td>39.01</td><td>37.92</td><td>35.33</td><td>39.00</td><td>39.78</td><td>34.77</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>G0→G1+G2</td><td>26.20</td><td>27.65</td><td>27.92</td><td>31.01</td><td>35.08</td><td>34.37</td><td>35.60</td><td>38.14</td><td>37.11</td><td>34.11</td><td>41.11</td><td>40.56</td><td>34.07</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1+G2</td><td>25.34</td><td>27.83</td><td>27.92</td><td>31.51</td><td>36.43</td><td>35.32</td><td>35.50</td><td>38.00</td><td>37.27</td><td>36.89</td><td>42.11</td><td>42.67</td><td>34.73</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1→G2</td><td>25.51</td><td>26.20</td><td>27.75</td><td>30.51</td><td>35.29</td><td>34.46</td><td>35.20</td><td>37.92</td><td>37.08</td><td>37.44</td><td>43.44</td><td>42.00</td><td>34.40</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1→G2</td><td>25.34</td><td>25.77</td><td>27.75</td><td>31.94</td><td>36.58</td><td>35.09</td><td>35.17</td><td>37.92</td><td>36.24</td><td>37.34</td><td>43.34</td><td>42.18</td><td>34.56</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G2→G1</td><td>25.34</td><td>26.97</td><td>26.47</td><td>30.76</td><td>34.41</td><td>34.47</td><td>35.11</td><td>37.42</td><td>37.09</td><td>33.67</td><td>38.44</td><td>40.89</td><td>33.42</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G2→G1</td><td>24.57</td><td>26.80</td><td>26.30</td><td>30.86</td><td>35.36</td><td>35.26</td><td>34.54</td><td>37.07</td><td>36.12</td><td>36.44</td><td>39.00</td><td>42.78</td><td>33.76</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Order</th>
    <th colspan="3">bn</th>
    <th colspan="3">hi</th>
    <th colspan="3">ne</th>
    <th colspan="3">bn</th>
    <th colspan="3">hi</th>
    <th colspan="3">ne</th>
    <th rowspan="2">G0-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td><td>/</td><td>22.50</td><td>23.46</td><td>21.89</td><td>29.59</td><td>29.89</td><td>29.13</td><td>28.23</td><td>28.48</td><td>27.95</td><td>29.67</td><td>30.00</td><td>26.89</td><td>27.41</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1+G2</td><td>24.29</td><td>24.91</td><td>20.96</td><td>30.94</td><td>32.26</td><td>31.31</td><td>30.03</td><td>33.24</td><td>30.03</td><td>31.56</td><td>34.00</td><td>36.22</td><td>29.98</td>
  </tr>
  <tr>
    <td>MoE-LPR (4*24)</td><td>G0→G1+G2</td><td>24.38</td><td>25.00</td><td>21.47</td><td>29.64</td><td>31.80</td><td>30.09</td><td>29.59</td><td>32.72</td><td>29.73</td><td>33.11</td><td>34.33</td><td>32.78</td><td>29.55</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1+G2</td><td>23.27</td><td>25.26</td><td>21.47</td><td>31.23</td><td>32.05</td><td>30.60</td><td>29.67</td><td>32.34</td><td>29.66</td><td>32.67</td><td>36.00</td><td>35.00</td><td>29.94</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G1→G2</td><td>23.44</td><td>25.34</td><td>21.81</td><td>31.66</td><td>32.31</td><td>31.41</td><td>29.55</td><td>32.60</td><td>29.54</td><td>34.22</td><td>35.44</td><td>35.78</td><td>30.26</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G1→G2</td><td>23.78</td><td>25.43</td><td>22.33</td><td>31.42</td><td>31.92</td><td>31.11</td><td>29.71</td><td>32.49</td><td>29.88</td><td>34.02</td><td>35.62</td><td>35.82</td><td>30.29</td>
  </tr>
  <tr>
    <td>MoE-LPR (7*24)</td><td>G0→G2→G1</td><td>25.32</td><td>24.40</td><td>21.81</td><td>30.41</td><td>31.89</td><td>29.92</td><td>28.79</td><td>30.73</td><td>29.00</td><td>27.78</td><td>29.67</td><td>31.00</td><td>28.39</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>G0→G2→G1</td><td>24.98</td><td>25.17</td><td>22.67</td><td>31.75</td><td>29.95</td><td>30.41</td><td>29.25</td><td>32.49</td><td>29.46</td><td>32.56</td><td>35.11</td><td>33.78</td><td>29.80</td>
  </tr>
</table>

Table 9: The detailed results of Qwen1.5-1.8B under the G0→G1+G2, G0→G1→G2, and G0→G2→G1 settings for each language.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">ARC-Challenge</th>
    <th colspan="3">MMLU</th>
    <th colspan="3">Hellaswag</th>
    <th colspan="3">Belebele</th>
    <th rowspan="2">G0-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Llama-3.2-3B</td><td>51.11</td><td>43.59</td><td>39.23</td><td>56.42</td><td>49.09</td><td>44.00</td><td>76.32</td><td>61.50</td><td>52.87</td><td>74.22</td><td>68.56</td><td>68.67</td><td>57.13</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>50.00</td><td>42.31</td><td>38.80</td><td>56.25</td><td>49.63</td><td>44.08</td><td>76.35</td><td>62.23</td><td>53.94</td><td>74.89</td><td>69.22</td><td>68.11</td><td>57.15</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>49.91</td><td>42.56</td><td>39.83</td><td>56.05</td><td>49.77</td><td>44.07</td><td>76.28</td><td>62.18</td><td>53.74</td><td>75.33</td><td>69.44</td><td>68.89</td><td>57.34</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th colspan="3">el</th>
    <th colspan="3">hu</th>
    <th colspan="3">tr</th>
    <th rowspan="2">GI-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Llama-3.2-3B</td><td>31.76</td><td>34.85</td><td>33.56</td><td>41.08</td><td>43.01</td><td>42.37</td><td>43.37</td><td>44.65</td><td>42.85</td><td>65.22</td><td>59.67</td><td>59.22</td><td>45.13</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>36.64</td><td>39.21</td><td>36.89</td><td>43.96</td><td>44.51</td><td>43.46</td><td>52.49</td><td>51.33</td><td>49.74</td><td>68.44</td><td>63.00</td><td>61.78</td><td>49.29</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>36.64</td><td>39.73</td><td>36.55</td><td>43.88</td><td>44.86</td><td>43.51</td><td>52.72</td><td>51.66</td><td>49.60</td><td>67.89</td><td>64.22</td><td>61.78</td><td>49.42</td>
  </tr>
</table>

Table 10: The results of Llama-3.2-3B on ARC-Challenge, MMLU, Hellaswag, and Belebele under the G0→G1 setting for each language.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">en→zh</th>
    <th colspan="3">zh→en</th>
    <th colspan="3">en→es</th>
    <th colspan="3">es→en</th>
    <th colspan="3">old-avg</th>
    <th colspan="3">en→el</th>
    <th colspan="3">el→en</th>
    <th colspan="3">en→hu</th>
    <th colspan="3">hu→en</th>
    <th colspan="3">en→tr</th>
    <th colspan="3">tr→en</th>
    <th colspan="3">new-avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
    <th>en</th><th>es</th><th>zh</th>
  </tr>
  <tr>
    <td>Llama-3.2-3B</td><td>84.36</td><td>83.95</td><td>84.79</td><td>86.4</td><td>84.88</td><td>82.47</td><td>85.86</td><td>83.81</td><td>86.15</td><td>78.97</td><td>86.53</td><td>83.96</td>
  </tr>
  <tr>
    <td>MoE-LPR (3*24)</td><td>84.19</td><td>83.19</td><td>84.67</td><td>86.27</td><td>84.58</td><td>84.18</td><td>85.89</td><td>83.24</td><td>85.6</td><td>81.95</td><td>85.26</td><td>84.35</td>
  </tr>
  <tr>
    <td>LayerMoE (Ours)</td><td>84.32</td><td>83.67</td><td>84.63</td><td>86.26</td><td>84.72</td><td>85.14</td><td>86.18</td><td>83.7</td><td>85.65</td><td>82.02</td><td>85.44</td><td>84.69</td>
  </tr>
</table>

Table 11: The detailed results of Llama-3.2-3B on the FLORES benchmark under the G0→G1 setting for each language.

Reviewer paper summaries: This paper proposes a layer-wise expert allocation algorithm to assign an appropriate number of new experts for each layer when expanding existing LLMs with new languages. To this end, two major steps are investigated: (1) calculate the similarity between different languages for each layer, and allocate more experts to the layers showing lower similarity, and (2) add a classifier in front of the router network the layers with higher similarity to improve the forgetting for old languages by the model. This research builds on the existing approach of Zhou et al. (2024), that is extended with the two mentioned steps.
Summary of strengths: - the presented research offers a more efficient (in terms of parameters) approach for adding new languages to existing LLMs. Although the improvments are modest, and not consistent across different experimental setup, the approach is shown to obtain good experimental results.
- the methodology is explained well, and illustrated with insightful illustrations (e.g. Figure 2).
Summary of weaknesses: The related research section is very limited, and should be better placed before discussing the results, as some of the described approaches are used as baselines for the current study. It is also uncommon to have a table with results in the introduction section, it would be better to give a more insightful explanation of the other approaches in the experimental results section, where results are compared between approaches (now the information in Table 1 is mainly explained in the caption of the table, instead of the main text of the paper).

The discusssion of the results focuses mainly on the differences between the presented approach, and the approach it is built on (Zhou et al. (2024). There is only a high-level description of the results, and for instance no discussion on the different languages (shown in the appendix), although although I think it is important to discuss them in more detail in the paper as well. It shows for instance that for Chinese, results are constantly better with the Qwen1.5-1.8B model, and same holds for Spanish, in the G0>G1 setup. Similar observations can be made for other setups.

Comments after Author Discussion: I believe the proposed changes can improve the quality of the paper. It is difficult to judge at this stage, however, how substantial they will be, so I decided to keep my score.
Comments, suggestions and typos: - “declines obviously” > declines noticeably
Overall assessment scores: 3.5
Metareview score: 4
Metareview: This paper introduces LayerMoE, a novel layer-wise expert allocation algorithm designed to expand large language models (LLMs) to support new languages while mitigating catastrophic forgetting of old languages. The method improves upon existing Mixture-of-Experts (MoE) architectures by dynamically allocating new experts based on representation similarity between languages at different layers, thereby reducing parameter costs. Additionally, a classifier is introduced to guide token routing for old languages, further minimizing performance degradation. The proposed approach demonstrates significant improvements over prior methods, achieving comparable or better performance with fewer experts, making it both efficient and effective for multilingual LLM expansion.
Summary of reasons to publish: 1. The proposed method significantly reduces the number of experts required (60% fewer in single-expansion and 33.3% fewer in lifelong-expansion settings) while maintaining or improving performance in both target (new) and source (old) languages. This addresses a critical challenge in multilingual LLM expansion.
2. The layer-wise expert allocation algorithm, which dynamically allocates experts based on representation similarity, is novel and intuitive. This approach provides a scalable and practical solution for expanding LLMs to new languages.
3. The paper is well-written, clearly structured, and easy to follow. It includes extensive experiments that thoroughly demonstrate the effectiveness of the proposed method, supported by detailed results and insightful illustrations.