Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
Authors: Chong Li, Yingzhuo Deng, Jiajun Zhang, Chengqing Zong
Accepted at: acl25

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model

Chong Li, Yingzhuo Deng, Jiajun Zhang, Chengqing Zong*
State Key Laboratory of Multimodal Artificial Intelligence Systems,
Institute of Automation, CAS, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
{lichong2021, dengyingzhuo2024}@ia.ac.cn, {jjzhang, cqzong}@nlpr.ia.ac.cn

Abstract
The curse of multilinguality phenomenon is a fundamental problem of multilingual Large Language Models (LLMs), where the competition between massive languages results in inferior performance. It mainly comes from limited capacity and negative transfer between dissimilar languages. To address this issue, we propose a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages. Specifically, the model is first tuned on monolingual corpus to determine the parameter deviation in each layer and quantify the similarity between languages. Layers with more deviations are extended to mixture-of-experts layers to reduce competition between languages, where one expert module serves one group of similar languages. Experimental results on 18 to 128 languages show that our method reduces the negative transfer between languages and significantly boosts multilingual performance with fewer parameters. Such language group specialization on experts benefits the new language adaptation and reduces the inference on the previous multilingual knowledge learned.1

1 Introduction
After training on the massive multilingual corpus, large language models obtain impressive multilingual abilities, e.g., cross-lingual natural language understanding (Xue et al., 2021) and in-context learning (Lin et al., 2022; Scao et al., 2023; Wei et al., 2023b; Anil et al., 2023; Üstün et al., 2024). However, their performance in medium-to low-resource languages, still lags behind that of high-resource languages (Lai et al., 2023; Asai et al., 2024; Li et al., 2024), and is hindered by the curse of multilinguality phenomenon (Aharoni et al., 2019; Wu and Dredze, 2020). It is found that the limited capacity and negative language transfer mainly contribute to the curse of multilinguality phenomenon (Chang et al., 2024). Thus, our key research problem lies on: How to flexibly augment the capacity of LLM for massive languages?

To address the research problem above, Pfeiffer et al. (2022) fine-tuned a module for each new language to augment parameters. The additional language identification process hinders its general application and affects the inference performance if misclassification. Blevins et al. (2024) trained models for new languages using the multilingual base model as initialization, and assembled them with vanilla models during inference. It largely increases the inference cost and the amount of model parameters which linearly grows with the number of languages involved.

In contrast, we introduce language specialization to the mixture-of-experts structure to scale up the parameters of the model. In particular, monolingual corpus from each language \( x \) is first adopted to tune the model and obtain the layer-wise parameter deviation \( \Delta \theta_l^x \) like Figure 1(a). Layers near the

![A diagram showing two steps: (a) Calculate the parameter deviation for each language, and (b) Dynamically scale up multilingual LLM.](page_1016_370_340_246.png)

Figure 1: (a) We first statisticize layer-wise parameter deviation of the multilingual language model for each language, (b) then dynamically scale up layers with more deviations into mixture-of-experts layers for language groups divided by language similarity.

1Corresponding author.
1Our code and model weights are available at https://github.com/ZNLPL/DMoE
input and output of LLM are often found with more derivation than the others (refer to Figure 10 in Appendix B for more details). We argue that layers with more deviation require more capacity to contain language-specific knowledge, while the other layers can be shared with all languages, like the “concept space” in the multilingual LLM (Wendler et al., 2024). Thus, the former is extended to the mixture-of-experts layer, and the parameter of each expert is tuned by a group of similar languages like Figure 1(b). It aims to precisely exploit parameters during scaling up and keep a similar inference cost for each token. Such designation is also beneficial for extending new languages while reducing the effect on the previously learned languages. Given a new language to adapt, we first determine its similarity between existing language groups, then copy and fine-tune the expert for the most similar language group to achieve a better transferring performance and alleviate catastrophic forgetting. The experimental results on 18 to 128 languages show that our method significantly improves multilingual performance and mitigates the curse of multilinguality phenomenon. The improvement in perplexity reaches 11.4% over the continual pre-training method and even surpasses X-ELM (Blevins et al., 2024) 9.6% with 3.6x fewer parameters on average. In summary, our contributions lie in the following:

• We propose a mixture-of-experts training framework to flexibly group languages and dynamically augment the capacity of multilingual large language models.

• We formalize language grouping into a maximin optimization problem and introduce a token-level language classification loss to specialize mixture-of-experts layers.

• Extensive experiments on 18 to 128 languages demonstrate the effectiveness of our method which largely mitigates the curse of multilinguality phenomenon.

2 Related Works

2.1 Quantify Language Similarity

The LANG2VEC method (Littell et al., 2017) represents languages as typological, geographical, and phylogenetic vectors to calculate the similarity between them and has been widely adopted (Blevins et al., 2024; Chang et al., 2024). However, they rely exclusively on language- or data-intrinsic features, ignoring the characteristics of the downstream models. To address this limitation, prior works have proposed model-specific representations, such as learnable language vectors (Tsvetkov et al., 2016; Östling and Tiedemann, 2017; Johnson et al., 2017) and leveraging hidden states of the model (Malaviya et al., 2017) or gradients of the loss function (Wang and Zhang, 2022) to derive language representations. These model-specific approaches often require training from scratch or incur high computational costs by recalculating similarity during training. In contrast, our method utilizes parameter deviations as language representations, enabling stable similarity estimation through fine-tuning the downstream model on a small dataset in the preparatory phase.

2.2 Mixture of Experts

Since the concept of mixture-of-experts proposed (Jacobs et al., 1991; Jordan and Jacobs, 1994), it has been widely applied to SVM (Collobert et al., 2001), Gaussian process (Tresp, 2000), Dirichlet process (Shahbaba and Neal, 2009), LSTM (Theis and Bethge, 2015; Shazeer et al., 2017), and Transformer (Lepikhin et al., 2021; Roller et al., 2021; Fedus et al., 2022; Dai et al., 2022; MistralAI, 2023; Dai et al., 2024; Zhou et al., 2025). Adding more experts scales up the total capacity of the model while keeping similar inference costs on each token. Previous studies mainly focus on designing a better load-balancing routing strategy (Roller et al., 2021; Fedus et al., 2022; Zhou et al., 2022) and a training method (Sukhbaatar et al., 2024). Our work is similar to X-MOD (Pfeiffer et al., 2022), which trains an adapter module for each language in all layers. The main differences lie in 1) grouping similar languages in each expert to boost cross-lingual transfer rather than allocating one adapter for each language. 2) Text for inference can be directly input to our model without specifying languages which is inflexible and required for X-MOD.

2.3 Multilingual Large Language Model

The pre-training methods of multilingual large language models (Conneau et al., 2020; Lin et al., 2022; Scao et al., 2023; Yang et al., 2023; Wei et al., 2023b; Üstün et al., 2024; Dang et al., 2024; Ming et al., 2024; Ji et al., 2025) are mainly extended from the one for the monolingual corpus (Radford et al., 2018; Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020), and relied on a
Figure 2: The overview of our method to group and scale up the multilingual LLM. (a) Given pre-training languages, we first determine their parameter deviation \( \Delta \theta^x \) on the model, then group similar languages by the similarity of \( \Delta \theta^x \). (b) These layers with higher \( \| \Delta \theta_i^x \| \) are extended to MoE layers, where each expert is tuned with tokens from the corresponding language group to specialize. (c) To adapt to the new language, we copy the multilingual expert from the most similar language group, then only fine-tune the router and expert added.

balanced sampling method to mitigate the performance gap between languages.

To mitigate the curse of multilinguality, Blevins et al. (2024) applied the Branch-Train-Merge method (Li et al., 2022) on the training of multilingual language model, where one model serves for a group of languages, and assembled output of top-m models after language identification during inference. In contrast, our method is motivated by the distribution of parameter deviation during the training of multilingual large language models and strives to scale up the parameter on the language-specific layers. It keeps a similar cost without additional language classification while augmenting the capacity of the multilingual language model during inference.

3 Method

As shown in Figure 2, to train a Dynamic Mixture-of-Experts model (DMoE), we first fine-tune the model on the monolingual corpus to obtain the parameter deviation for each language. Then the parameter deviation is used to cluster similar languages (Section 3.1) and determine layers to extend parameters (Section 3.2). Besides, new languages for adaption are also dynamically assigned to the most similar language cluster to mitigate catastrophic forgetting (Section 3.3).

3.1 Dynamic Language Clustering

The quality of the clustering method is primarily influenced by the choice of similarity metric, making the determination of an appropriate metric central to its effectiveness. We first obtain the parameter deviation of the model by fine-tuning only ten steps, investigated in Appendix B, and take it as a representation of distinctive characteristics for each language. Given the high-dimensional nature of the parameter deviation, we employ cosine similarity as the metric to measure the similarity between languages. To satisfy the clustering process, we define the intra-group language similarity as follows:

\[
\operatorname{Sim}(\theta, G_k) = \min_{x, y \in G_k} \frac{\Delta \theta^x \cdot \Delta \theta^y}{\| \Delta \theta^x \| \| \Delta \theta^y \|}
\]

where \( G_k \) is the \( k \)-th group of languages, \( \Delta \theta^x \) and \( \Delta \theta^y \) are the parameter deviation of language \( x \) and \( y \) respectively on the parameter \( \theta \), and \( \Delta \theta^x = [\Delta \theta_1^x; \Delta \theta_2^x; \cdots; \Delta \theta_N^x] \) is the concatenation of the parameter deviation from all \( N \) layers after fine-tuning on language \( x \).

A higher intra-group similarity indicates that the languages within the group are more similar, resulting in less conflict between them. This reduces the potential for gradient conflicts during the continuous pre-training on different languages, making it more appropriate to share parameters with the same expert. Therefore, we can perform clustering by maximizing the similarity within each group, which can be formalized as follows:

\[
\max_{G_1, G_2, ..., G_K} \sum_{k=1}^K \operatorname{Sim}(\theta, G_k)
\]

However, obtaining the global optimal solution to this problem is NP-Hard. Additionally, the number of languages in each group needs to be balanced to enhance the utilization of experts. To address these challenges, we employ a greedy algorithm. The pseudo-code is provided in Algorithm 1.
Algorithm 1 Balanced Language Clustering

Input: Parameter deviations for different languages \( \Delta \Theta = \{\Delta \theta^1, \Delta \theta^2, \ldots, \Delta \theta^K\} \), Number of groups \( K \)

Output: Language clustering result \( Groups \)

1: Initialize \( Groups = \{ \} \)
2: while \( \Delta \Theta \) is not empty do
3:   Compute the cosine similarity between languages \( (i, j) \) for all \( \Delta \theta^i, \Delta \theta^j \in \Delta \Theta \)
4:   Find the most similar pair of languages \( (i^*, j^*) \)
5:   Merge languages \( i^* \) and \( j^* \) to form a group: \( G = \{i^*, j^*\} \)
6:   Remove \( i^* \) and \( j^* \) from \( \Delta \Theta \)
7:   while \( |G| < \frac{|\Delta \Theta|}{K} \) do
8:     Compute the intra-group similarity (Eq. 1) of \( G \cup \{m\} \) for all \( \Delta \theta^m \in \Delta \Theta \)
9:     Find the group \( G \cup \{m^*\} \) that maximizes the intra-group similarity
10:    Add \( m^* \) to \( G \)
11:    Remove \( m^* \) from \( \Delta \Theta \)
12:   end while
13:   Add group \( G \) to \( Groups \)
14: end while
15: Return: \( Groups \)

3.2 Dynamic MoE Layer Extension

We assume that the layers with large parameter deviations are important and language-specific during fine-tuning, requiring additional capacities to mitigate the conflicts between languages. Thus the top-\( \epsilon \) of dense layers and extended to the mixture-of-experts layers with \( g \) experts, where \( \epsilon \in [0, 1] \) and \( g \in \mathbb{N}^+ \) are hyper-parameters. Each expert is initialized from the parameters of the original dense layer, while the ones of router are randomly initialized. Corpora from the same language group are organized in the same batch and used to fine-tune the parameters of the corresponding expert. We also train the router with the following language group classification loss:

\[
\mathcal{L}_{RC}(\theta) = - \sum_x \sum_{i=1}^M [\log (\mathrm{P}_i(l|x; \theta))]
\]

where \( x \) is a token from the language group \( l \), and \( \mathrm{P}_i(\cdot) \) is the probability estimated by the router at the \( i \)-th MoE layer. Thus the training loss comes to the weighted sum of Causal Language Modeling (CLM) loss and the above language group classification loss:

\[
\mathcal{L}(\theta) = \mathcal{L}_{CLM}(\theta) + \alpha \mathcal{L}_{RC}(\theta)
\]

where \( \alpha \in \mathbb{R}^+ \) is a hyper-parameter.

3.3 Dynamic Language Adaptation

Given new languages to adapt, we introduce a method to augment their capacity while reducing the inference to other languages learned. Specifically, samples from the new language are first input to experts through a hard routing strategy. The multilingual expert with the lowest perplexity is considered the most similar one, which is copied and only fine-tuned for fast adaptation. It is noted that the other part of parameters, like the shared dense layers and the other experts, are frozen to avoid catastrophic forgetting during the new language learning (Winata et al., 2023).

4 Experiments

4.1 Experiments Settings

Large Language Models  We adopt the multilingual Bloom (Scao et al., 2023) and English-centric Gemma (Team et al., 2024) series models in this work.

Corpus  Two multilingual corpora, CulturaX (Nguyen et al., 2024) and MADLAD-400 (Kudugunta et al., 2023), are used in this work. We set the language sampling exponent to 0.3 following mT5 (Xue et al., 2021).

Evaluation Tasks  There are five multilingual tasks, covering natural language inference (Conneau et al., 2018), paraphrase detection (Yang et al., 2019), and multilingual reasoning tasks (Ponti et al., 2020; Lin et al., 2022; Tikhonov and Ryabinin, 2021), selected to evaluate the performance of multilingual LLMs. To reduce the variability of prompt and evaluation method, we choose the default prompt from the language model evaluation harness framework (Gao et al., 2024).

Baselines

• + Pre-train, where the base model continues to pre-train on the same multilingual corpus. It denotes the performance of the vanilla continual multilingual pre-training method.

• X-ELM (Blevins et al., 2024) trains a model for two similar languages, and ensembles outputs from top-m models during inference, where m is set to 2 in this work.
• Branch-Train-Mix (Sukhbaatar et al., 2024) trains models specialized for one domain and merges them to obtain a mixture-of-experts model, which shows significantly better performance than Branch-Train-Merge (Li et al., 2022). We apply our dynamic language clustering results to it, serving as a strong multilingual mixture-of-experts baseline.

To conduct a fair comparison, the total training token amount is the same for all methods. Hyperparameters are reported in Appendix A.

4.2 Results on 18 Languages

We first conduct experiments on 18 languages from 9 language families. Figure 3 illustrates the pairwise language similarity calculated by the \( \Delta \theta \) of BLOOM\(_{560M}\), and more details of other models refer to Appendix C.1. It mostly exhibits some linguistic characteristics. For example, Tamil (ta) and Telugu (te), which both come from the Dravidian language family, show a similar trend among languages and have higher similarity than the other languages. Based on the pair-wise similarity, languages are divided into multiple groups by Algorithm 1, and results are reported in Table 1. Appendix C.1 reports the results of other language models investigated.

![The cosine similarity between 18 languages for BLOOM_{560M}.](page_172_1042_1097_312.png)

Figure 3: The cosine similarity between 18 languages for BLOOM\(_{560M}\).

After language clustering, we continue pre-training on 18 languages under a 6.5B tokens amount budget from CulturaX. The perplexity results of models across different parameter amounts are shown in Table 2. We can find that scaling up model parameters brings better language modeling performance compared with the continued pre-training method ("+ Pre-train"). DMoE obtains the highest average improvement on perplexity (+11.4% over "+ Pre-train") than the other two strong baseline methods (+0.8% and +2.2% respectively) and requires the least additional parameters. It is noted that DMoE with 9 experts outperforms X-ELM (Blevins et al., 2024) using 3.6x less parameters. The improvement mostly comes from unseen languages like German (+17.9%) and low-resource languages like Urdu (+13.6%). Appendix C.2 shows similar results of Qwen2.5 base models.

Figure 4 illustrates the trend of perplexity improvement over the continual pre-training baseline using DMoE across 18 languages. It can be found that languages with higher perplexity benefit more from our method. Moreover, with more language groups divided, DMoE shows better multilingual language modeling performance.

<table>
  <tr>
    <th></th>
    <th>ar</th>
    <th>ur</th>
    <th>bn</th>
    <th>it</th>
    <th>de</th>
    <th>nl</th>
    <th>ta</th>
    <th>te</th>
    <th>hi</th>
    <th>id</th>
    <th>fr</th>
    <th>vi</th>
    <th>ru</th>
    <th>uk</th>
    <th>th</th>
    <th>ko</th>
    <th>ja</th>
    <th>zh</th>
  </tr>
  <tr>
    <th>9G</th>
    <td>1</td><td>2</td><td>2</td><td>3</td><td>3</td><td>4</td><td>5</td><td>6</td><td>6</td><td>7</td><td>8</td><td>8</td><td>9</td><td>9</td><td>9</td><td>9</td><td>9</td><td>9</td>
  </tr>
  <tr>
    <th>6G</th>
    <td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>3</td><td>3</td><td>3</td><td>4</td><td>4</td><td>5</td><td>5</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td><td>6</td>
  </tr>
  <tr>
    <th>6G<sup>(L)</sup></th>
    <td>1</td><td>2</td><td>3</td><td>4</td><td>3</td><td>5</td><td>5</td><td>5</td><td>6</td><td>6</td><td>4</td><td>4</td><td>6</td><td>2</td><td>2</td><td>3</td><td>3</td><td>3</td>
  </tr>
  <tr>
    <th>6G<sup>(R)</sup></th>
    <td>2</td><td>5</td><td>4</td><td>6</td><td>6</td><td>2</td><td>3</td><td>3</td><td>4</td><td>6</td><td>3</td><td>5</td><td>4</td><td>2</td><td>5</td><td>1</td><td>1</td><td>1</td>
  </tr>
  <tr>
    <th>3G</th>
    <td>1</td><td>1</td><td>2</td><td>2</td><td>1</td><td>3</td><td>1</td><td>1</td><td>2</td><td>2</td><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td>
  </tr>
  <tr>
    <th>2G</th>
    <td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td>
  </tr>
</table>

Table 1: The grouping results of BLOOM\(_{560M}\), where "2G" denotes the result that divides into 2 groups. "6G\(^{(L)}\)" and "6G\(^{(R)}\)" indicate the LANG2VEC and random language clustering results, respectively.

Transfer language similarities. To evaluate the effectiveness of dynamic language clustering (Section 3.1), we replace the 6-group dividing into the LANG2VEC (Littell et al., 2017) and random grouping results in Table 1. The "w/ Random Cluster" row reports the language modeling result on BLOOM\(_{560M}\), which is worse than the DMoE
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#Param.</th>
    <th colspan="6">High</th>
    <th colspan="6">Medium</th>
    <th colspan="6">Low</th>
  </tr>
  <tr>
    <th>ar</th><th>de</th><th>fr</th><th>it</th><th>ja</th><th>nl</th><th>ru</th><th>zh</th><th>bn</th><th>hi</th><th>id</th><th>ko</th><th>th</th><th>uk</th><th>vi</th><th>ta</th><th>te</th><th>ur</th><th>Avg↓</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>560M</td>
    <td>56.7</td><td>126.4</td><td>37.4</td><td>85.3</td><td>55.7</td><td>129.5</td><td>31.3</td><td>59.0</td><td>42.3</td><td>32.7</td><td>47.4</td><td>25.7</td><td>14.0</td><td>39.7</td><td>27.9</td><td>64.2</td><td>88.4</td><td>60.0</td><td>56.9</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>560M</td>
    <td>39.0</td><td>27.6</td><td>22.0</td><td>17.8</td><td>15.0</td><td>16.6</td><td>8.2</td><td>36.4</td><td>26.5</td><td>20.9</td><td>32.6</td><td>8.4</td><td>4.4</td><td>6.8</td><td>18.5</td><td>31.0</td><td>26.9</td><td>29.8</td><td>21.6</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>5.03B</td>
    <td>35.2</td><td>34.6</td><td>21.3</td><td>17.3</td><td>18.0</td><td>21.5</td><td>9.8</td><td>37.9</td><td>25.1</td><td>19.5</td><td>28.8</td><td>9.3</td><td>4.4</td><td>8.5</td><td>18.9</td><td>26.9</td><td>23.0</td><td>27.5</td><td>21.5</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>1.57B</td>
    <td>39.4</td><td>25.8</td><td>21.6</td><td>17.1</td><td>14.7</td><td>15.3</td><td>8.0</td><td>35.8</td><td>26.4</td><td>21.0</td><td>32.1</td><td>8.1</td><td>4.1</td><td>6.6</td><td>18.1</td><td>31.4</td><td>27.0</td><td>30.0</td><td>21.2</td>
  </tr>
  <tr>
    <td>DMoE (2 Groups)</td>
    <td>635M</td>
    <td>37.9</td><td>25.0</td><td>21.5</td><td>17.1</td><td>15.0</td><td>14.5</td><td>7.4</td><td>34.4</td><td>25.4</td><td>20.5</td><td>31.3</td><td>7.9</td><td>3.8</td><td>5.9</td><td>18.4</td><td>30.3</td><td>26.7</td><td>28.6</td><td>20.6</td>
  </tr>
  <tr>
    <td>DMoE (3 Groups)</td>
    <td>710M</td>
    <td>37.0</td><td>25.3</td><td>21.8</td><td>17.2</td><td>14.4</td><td>14.4</td><td>7.1</td><td>35.0</td><td>24.6</td><td>19.9</td><td>30.4</td><td>7.7</td><td>3.7</td><td>5.6</td><td>18.4</td><td>29.3</td><td>26.3</td><td>27.4</td><td>20.3</td>
  </tr>
  <tr>
    <td>DMoE (6 Groups)</td>
    <td>937M</td>
    <td>36.2</td><td>23.5</td><td>20.9</td><td>16.0</td><td>13.8</td><td>13.6</td><td>7.3</td><td>34.1</td><td>23.9</td><td>19.1</td><td>30.7</td><td>7.4</td><td>3.9</td><td>5.9</td><td>17.6</td><td>27.9</td><td>23.4</td><td>26.8</td><td>19.5</td>
  </tr>
  <tr>
    <td>w/ Gemma Clusters</td>
    <td>937M</td>
    <td>36.7</td><td>23.5</td><td>21.4</td><td>16.1</td><td>13.8</td><td>13.6</td><td>7.3</td><td>34.1</td><td>23.7</td><td>19.7</td><td>31.8</td><td>7.4</td><td>3.6</td><td>5.6</td><td>17.7</td><td>27.8</td><td>23.2</td><td>27.1</td><td>19.7</td>
  </tr>
  <tr>
    <td>w/LANG2VEC Clusters</td>
    <td>937M</td>
    <td>37.6</td><td>25.2</td><td>20.9</td><td>15.8</td><td>13.4</td><td>13.3</td><td>7.6</td><td>34.2</td><td>25.0</td><td>19.3</td><td>30.3</td><td>7.2</td><td>3.5</td><td>6.2</td><td>17.7</td><td>28.0</td><td>23.6</td><td>28.1</td><td>19.8</td>
  </tr>
  <tr>
    <td>w/ Random Clusters</td>
    <td>937M</td>
    <td>38.5</td><td>25.7</td><td>21.5</td><td>17.0</td><td>14.6</td><td>15.2</td><td>7.9</td><td>35.3</td><td>25.7</td><td>20.6</td><td>31.8</td><td>8.1</td><td>4.1</td><td>6.5</td><td>18.1</td><td>30.6</td><td>26.1</td><td>29.4</td><td>20.9</td>
  </tr>
  <tr>
    <td>w/o Class. Loss</td>
    <td>937M</td>
    <td>36.5</td><td>24.2</td><td>21.2</td><td>16.4</td><td>14.4</td><td>13.9</td><td>7.2</td><td>34.8</td><td>24.6</td><td>19.7</td><td>30.9</td><td>7.7</td><td>3.8</td><td>5.8</td><td>18.2</td><td>28.4</td><td>24.1</td><td>27.6</td><td>20.0</td>
  </tr>
  <tr>
    <td>DMoE (9 Groups)</td>
    <td>1.16B</td>
    <td>36.9</td><td>23.1</td><td>21.1</td><td>14.7</td><td>14.0</td><td>13.3</td><td>7.5</td><td>34.6</td><td>25.1</td><td>19.6</td><td>29.7</td><td>7.2</td><td>3.6</td><td>6.1</td><td>17.9</td><td>27.6</td><td>23.0</td><td>27.0</td><td>19.5</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1T</sub></td>
    <td>1.72B</td>
    <td>41.2</td><td>63.0</td><td>24.3</td><td>44.0</td><td>35.2</td><td>63.0</td><td>20.1</td><td>40.1</td><td>127.0</td><td>22.6</td><td>32.7</td><td>17.4</td><td>9.8</td><td>23.9</td><td>19.2</td><td>140.1</td><td>43.6</td><td>36.3</td><td>33.5</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>1.72B</td>
    <td>25.1</td><td>21.0</td><td>15.3</td><td>15.5</td><td>11.7</td><td>17.4</td><td>7.0</td><td>23.4</td><td>16.8</td><td>15.3</td><td>23.0</td><td>7.8</td><td>4.3</td><td>8.5</td><td>13.1</td><td>21.6</td><td>21.1</td><td>22.6</td><td>16.1</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>15.50B</td>
    <td>24.9</td><td>20.2</td><td>15.0</td><td>12.1</td><td>12.1</td><td>12.3</td><td>7.1</td><td>24.3</td><td>17.4</td><td>14.4</td><td>21.0</td><td>7.5</td><td>3.8</td><td>6.0</td><td>12.9</td><td>19.3</td><td>16.6</td><td>19.4</td><td>14.8</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>5.75B</td>
    <td>25.4</td><td>18.7</td><td>16.0</td><td>15.2</td><td>11.3</td><td>13.6</td><td>6.0</td><td>23.6</td><td>17.6</td><td>15.3</td><td>24.1</td><td>7.4</td><td>3.7</td><td>7.0</td><td>13.4</td><td>23.1</td><td>21.1</td><td>20.5</td><td>15.7</td>
  </tr>
  <tr>
    <td>DMoE (3 Groups)</td>
    <td>2.33B</td>
    <td>26.3</td><td>17.5</td><td>16.0</td><td>12.5</td><td>11.0</td><td>10.5</td><td>5.7</td><td>25.2</td><td>17.5</td><td>15.1</td><td>22.4</td><td>6.3</td><td>3.2</td><td>4.5</td><td>13.8</td><td>21.1</td><td>19.6</td><td>19.9</td><td>14.9</td>
  </tr>
  <tr>
    <td>DMoE (6 Groups)</td>
    <td>3.22B</td>
    <td>26.1</td><td>17.2</td><td>15.8</td><td>12.4</td><td>10.7</td><td>10.2</td><td>5.8</td><td>24.9</td><td>17.4</td><td>14.5</td><td>22.8</td><td>6.1</td><td>3.3</td><td>4.6</td><td>13.3</td><td>20.3</td><td>17.5</td><td>19.6</td><td>14.6</td>
  </tr>
  <tr>
    <td>DMoE (9 Groups)</td>
    <td>4.14B</td>
    <td>26.6</td><td>16.8</td><td>16.0</td><td>11.5</td><td>10.8</td><td>10.0</td><td>5.9</td><td>25.1</td><td>17.8</td><td>14.8</td><td>22.2</td><td>5.9</td><td>3.1</td><td>4.8</td><td>13.3</td><td>20.1</td><td>17.0</td><td>19.8</td><td>14.5</td>
  </tr>
  <tr>
    <td>Gemma<sub>2B</sub></td>
    <td>2.51B</td>
    <td>54.8</td><td>12.5</td><td>23.6</td><td>13.4</td><td>11.1</td><td>11.4</td><td>5.1</td><td>69.1</td><td>68.9</td><td>44.2</td><td>45.5</td><td>5.6</td><td>2.8</td><td>4.0</td><td>19.8</td><td>70.5</td><td>62.8</td><td>54.2</td><td>32.2</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>2.51B</td>
    <td>28.8</td><td>10.1</td><td>17.1</td><td>9.3</td><td>7.3</td><td>6.5</td><td>4.1</td><td>29.4</td><td>21.7</td><td>14.9</td><td>21.9</td><td>3.8</td><td>2.3</td><td>3.1</td><td>13.0</td><td>20.4</td><td>17.0</td><td>20.4</td><td>14.0</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>22.56B</td>
    <td>30.2</td><td>11.1</td><td>19.3</td><td>10.2</td><td>8.0</td><td>7.3</td><td>4.3</td><td>34.6</td><td>22.7</td><td>15.3</td><td>23.9</td><td>3.9</td><td>2.3</td><td>3.3</td><td>14.1</td><td>19.9</td><td>16.5</td><td>22.8</td><td>15.0</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>11.57B</td>
    <td>27.7</td><td>10.7</td><td>18.2</td><td>9.7</td><td>7.5</td><td>6.8</td><td>4.2</td><td>30.6</td><td>18.5</td><td>14.4</td><td>23.8</td><td>3.9</td><td>2.3</td><td>3.2</td><td>13.0</td><td>17.4</td><td>14.5</td><td>19.3</td><td>13.7</td>
  </tr>
  <tr>
    <td>DMoE (9 Groups)</td>
    <td>6.53B</td>
    <td>24.8</td><td>9.9</td><td>16.8</td><td>9.1</td><td>7.0</td><td>6.3</td><td>4.0</td><td>27.8</td><td>17.6</td><td>12.9</td><td>19.6</td><td>3.6</td><td>2.2</td><td>3.1</td><td>12.2</td><td>15.3</td><td>12.8</td><td>17.3</td><td>12.4</td>
  </tr>
</table>

Table 2: The normalized perplexity on the valid split of CulturaX. The perplexity is normalized to the vocabulary of Bloom following Wei et al. (2023a). † denotes the language unseen in the pre-training of BLOOM. “High”, “Medium”, and “Low” indicates the available amount of linguistic resources. The best and second results are denoted in bold and underlined, correspondingly.

model (+1.4 PPL on average). We argue that the poor result arises from the negative transfer between dissimilar languages, especially deteriorating the performance of low-resource languages like Urdu (+2.6 PPL). And LANG2VEC brings an inferior result, +0.3 PPL on average, comparing our model-specific method. It demonstrates that better language clustering results can bring better cross-lingual transfer to the low-resource languages. The language clustering result of Gemma<sub>2B</sub> is further applied on BLOOM<sub>560M</sub> to investigate the transferability of our method. It is interesting to find that BLOOM<sub>560M</sub> with Gemma clusters is slightly worse than the vanilla model in Table 2. Although our method shows some transferability, we still recommend using language similarity classification based on its parameter derivation for better results.

Trade-off between learning and forgetting.
When new languages come for multilingual models to adapt, it is better to achieve fast adaptation while alleviating the catastrophic forgetting of languages learned (Sun et al., 2020; Zhao et al., 2022; Wu et al., 2024). We adopt 4 unseen languages: Belarusian (be), Malayalam (ml), Marathi (mr), and Serbian (sr) to evaluate the performance of models. As shown in Table 3, the dense model suffers a catastrophic forgetting of the 18 languages learned after Language Adaptation Pre-Training (LAPT), especially on the medium and low resource languages (+2.0 PPL). In contrast, the proposed Dynamic Language Adaptation (DLA) method (Section 3.3) for DMoE achieves better adaptation results on new languages, and mitigates the catastrophic forgetting of the languages learned (only +0.7 PPL). It benefits from language-specific expert design and fine-tuning method, which provides a better module for initialization and reduces the inference to the modules learned.

![The average perplexity of DMoE across 18 languages under different ε using BLOOM560M, where ε = 1 denotes all layers are extended to MoE layers.](page_1092_1442_388_246.png)

Figure 5: The average perplexity of DMoE across 18 languages under different ε using BLOOM560M, where ε = 1 denotes all layers are extended to MoE layers.
Figure 6: The router distribution of top-1 expert for texts in different languages. (a) DMoE trained with randomly initialized router. (b) DMoE trained with language classification loss. Refer to Appendix C.3 for more details.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">New Languages</th>
    <th colspan="3">Old Languages</th>
  </tr>
  <tr>
    <th>be</th>
    <th>ml</th>
    <th>mr</th>
    <th>sr</th>
    <th>High</th>
    <th>Medium</th>
    <th>Low</th>
  </tr>
  <tr>
    <td>Gemma$_{2B}$</td>
    <td>10.0</td>
    <td>7.1</td>
    <td>11.4</td>
    <td>12.5</td>
    <td>26.9<sub>±9.4</sub></td>
    <td>15.1<sub>±7.9</sub></td>
    <td>10.5<sub>±4.3</sub></td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>9.3</td>
    <td>11.7</td>
    <td>11.2</td>
    <td>17.1</td>
    <td>15.1<sub>±4.4</sub></td>
    <td>8.5<sub>±3.7</sub></td>
    <td>5.4<sub>±1.7</sub></td>
  </tr>
  <tr>
    <td>w/ LAPT</td>
    <td>6.4</td>
    <td>4.3</td>
    <td>6.1</td>
    <td>8.3</td>
    <td>16.9<sub>±4.6</sub></td>
    <td>10.6<sub>±3.8</sub></td>
    <td>7.3<sub>±3.1</sub></td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>8.9</td>
    <td>10.0</td>
    <td>11.8</td>
    <td>17.2</td>
    <td>14.4<sub>±4.2</sub></td>
    <td>7.7<sub>±3.4</sub></td>
    <td>4.8<sub>±1.6</sub></td>
  </tr>
  <tr>
    <td>w/ DLA</td>
    <td><b>6.2</b></td>
    <td><b>4.0</b></td>
    <td><b>5.4</b></td>
    <td><b>8.2</b></td>
    <td><b>15.0<sub>±4.2</sub></b></td>
    <td><b>8.5<sub>±3.2</sub></b></td>
    <td><b>5.4<sub>±1.8</sub></b></td>
  </tr>
</table>

Table 3: The perplexity after adding new languages.

Ablation study We first modify the hyper-parameter \( \epsilon \) to determine the effect of augmenting the layer with higher parameter deviation. Figure 5 shows that scaling up layers with higher derivation is much better than the random augmentation baseline when \( \epsilon \) is less than 0.5. To balance the parameter amount and performance, we set \( \epsilon \) to 0.4 in this work.

The router classification loss is removed to quantify its contribution. As shown in the “w./ Class. Loss” row of Table 2, the perplexity increases by 0.5 on average. Figure 6 illustrates the statistics of token distribution assigned to the top-1 expert. It can be found that the bottom layer like the first layer does not show language specification without router classification loss (Figure 6(a)). Tokens are mostly assigned to the expert tuned in the same language with router classification loss as expected (Figure 6(b)).

4.3 Extend to 128 Languages

In this section, we further scale up the number of languages from 18 to 128 and increase the amount of pre-training tokens to 17.7B. Following previous findings, the number of language groups is set to 16, and refer to Table 8 in Appendix C.1 for more details of language dividing.

Table 4 reports the perplexity of 20 languages across different resources and the average result of 128 languages. It can be found that DMoE significantly mitigates the curse of multilinguality phenomenon and outperforms Branch-Train-Mix 1.1 PPL on average across 128 languages. The improvement mostly comes from unseen languages and low-resource languages, which reach 2.1 PPL on average for the five extremely low-resource languages in Table 4. The eighty languages with Latin script improved by 2.9 PPL over the continual pretraining model on average, while the other non-Latin languages improved by only 0.7 PPL.

We calculate the improvement across language families and find that the trend is similar where our method outperforms the baseline methods. It
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="10">High</th>
    <th colspan="10">Medium</th>
    <th colspan="10">Low</th>
    <th colspan="10">Extremely-Low</th>
    <th colspan="10">ALL 128L</th>
  </tr>
  <tr>
    <th>ar</th><th>de</th><th>en</th><th>it†</th><th>ja</th><th>hi</th><th>id</th><th>th†</th><th>uk†</th><th>vi†</th><th>kk†</th><th>mn†</th><th>my†</th><th>te</th><th>ur</th><th>br†</th><th>pa†</th><th>sw</th><th>ug†</th><th>zu</th><th>Arg†</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td><td>42.4</td><td>111.3</td><td>68.8</td><td>42.5</td><td>55.5</td><td>30.6</td><td>41.3</td><td>13.7</td><td>44.5</td><td>21.8</td><td>29.0</td><td>31.2</td><td>6.1</td><td>91.4</td><td>72.5</td><td>261.7</td><td>131.6</td><td>224.9</td><td>44.5</td><td>1278.9</td><td>154.4±157.0</td>
  </tr>
  <tr>
    <td>+ Pre-train</td><td>34.4</td><td>23.8</td><td>44.6</td><td>20.5</td><td>12.9</td><td>20.5</td><td>26.9</td><td>3.6</td><td>7.3</td><td>15.1</td><td>5.7</td><td>6.8</td><td>2.8</td><td>30.3</td><td>37.6</td><td>40.2</td><td>32.5</td><td>45.5</td><td>9.2</td><td>36.2</td><td>20.7±12.8</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td><td>32.5</td><td>21.0</td><td>40.3</td><td>17.1</td><td>12.1</td><td>20.0</td><td>26.1</td><td>3.5</td><td>6.6</td><td>14.6</td><td>5.4</td><td>6.3</td><td>2.8</td><td>31.6</td><td>37.1</td><td>35.4</td><td>34.1</td><td>43.5</td><td>9.0</td><td>31.2</td><td>19.2±12.2</td>
  </tr>
  <tr>
    <td>DMoE</td><td>32.1</td><td>19.5</td><td>40.8</td><td>16.3</td><td>11.3</td><td>19.8</td><td>25.8</td><td>3.4</td><td>6.4</td><td>14.7</td><td>5.2</td><td>6.2</td><td>2.7</td><td>29.3</td><td>35.2</td><td>31.4</td><td>30.5</td><td>39.7</td><td>8.1</td><td>28.3</td><td>17.7±11.2</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td><td>30.4</td><td>56.4</td><td>45.3</td><td>44.9</td><td>35.0</td><td>20.8</td><td>27.8</td><td>9.7</td><td>26.7</td><td>15.3</td><td>19.1</td><td>21.1</td><td>4.4</td><td>46.8</td><td>44.0</td><td>113.2</td><td>61.7</td><td>80.3</td><td>28.4</td><td>266.8</td><td>71.9±52.1</td>
  </tr>
  <tr>
    <td>+ Pre-train</td><td>22.9</td><td>15.9</td><td>30.3</td><td>14.1</td><td>9.7</td><td>14.7</td><td>19.3</td><td>3.1</td><td>5.4</td><td>11.3</td><td>4.5</td><td>5.4</td><td>2.5</td><td>22.3</td><td>26.8</td><td>27.9</td><td>23.9</td><td>30.8</td><td>7.4</td><td>26.0</td><td>14.9±8.7</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td><td>21.1</td><td>15.3</td><td>29.7</td><td>13.1</td><td>9.4</td><td>14.1</td><td>18.6</td><td>3.1</td><td>5.3</td><td>10.9</td><td>4.5</td><td>5.4</td><td>2.6</td><td>22.4</td><td>25.7</td><td>26.2</td><td>24.2</td><td>29.8</td><td>7.5</td><td>24.1</td><td>14.3±8.4</td>
  </tr>
  <tr>
    <td>DMoE</td><td>22.7</td><td>14.9</td><td>31.3</td><td>12.8</td><td>8.9</td><td>14.7</td><td>19.1</td><td>3.0</td><td>5.1</td><td>11.3</td><td>4.3</td><td>5.1</td><td>2.5</td><td>22.1</td><td>26.0</td><td>23.9</td><td>23.2</td><td>29.2</td><td>6.8</td><td>22.8</td><td>13.7±8.1</td>
  </tr>
</table>

Table 4: The perplexity of 20 languages on the valid split of MADLAD-400 (Kudugunta et al., 2023). Refer to Table 10 to 17 in Appendix C.4 for all results of 128 languages. † denotes the language unseen in the pre-training of BLOOM. “High”(>1%), “Medium”(>0.1%), “Low”(<0.01%), and “Extremely-Low”(<=0.01%) indicates the available amount of linguistic resources on the CommonCrawl following Lai et al. (2023).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">Zero-shot Results</th>
    <th colspan="5">Few-shot Results</th>
  </tr>
  <tr>
    <th>XNLI</th><th>PAWS-X</th><th>XCOPA</th><th>XStoryCloze</th><th>XWinograd</th>
    <th>XNLI</th><th>PAWS-X</th><th>XCOPA</th><th>XStoryCloze</th><th>XWinograd</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td><td>36.2±3.3</td><td>51.5±1.6</td><td>53.9±4.1</td><td>53.5±3.5</td><td>53.7±4.0</td><td>34.4±2.4</td><td>51.1±1.2</td><td>53.4±3.5</td><td>52.6±3.5</td><td>53.3±4.2</td>
  </tr>
  <tr>
    <td>+ Pre-train</td><td>37.1±3.5</td><td>52.9±2.4</td><td>53.6±2.9</td><td>54.8±2.6</td><td>54.9±4.1</td><td>34.7±2.6</td><td>51.6±1.1</td><td>53.8±2.4</td><td>52.7±2.8</td><td>53.8±5.1</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td><td>37.2±4.1</td><td>53.1±2.8</td><td>54.1±2.7</td><td>53.8±2.8</td><td>54.4±3.6</td><td>35.3±2.6</td><td>51.4±1.1</td><td>53.9±3.2</td><td>53.1±2.8</td><td>54.2±4.3</td>
  </tr>
  <tr>
    <td>DMoE</td><td>37.5±4.5</td><td>53.2±1.7</td><td>54.1±2.7</td><td>54.1±2.8</td><td>55.1±4.3</td><td>35.7±2.5</td><td>52.2±1.4</td><td>54.7±2.6</td><td>53.4±2.7</td><td>55.1±4.6</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td><td>39.2±5.4</td><td>53.9±1.6</td><td>55.1±5.7</td><td>56.0±4.7</td><td>55.1±5.2</td><td>37.1±4.5</td><td>50.5±1.1</td><td>55.2±5.8</td><td>55.5±5.2</td><td>55.5±5.2</td>
  </tr>
  <tr>
    <td>+ Pre-train</td><td>39.2±4.9</td><td>53.7±1.5</td><td>55.0±3.9</td><td>56.4±3.6</td><td>55.5±4.6</td><td>37.1±3.2</td><td>51.8±1.7</td><td>55.3±4.5</td><td>56.0±3.6</td><td>55.8±4.6</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td><td>39.1±5.1</td><td>53.5±1.6</td><td>55.6±3.6</td><td>56.4±3.8</td><td>56.4±4.7</td><td>38.6±3.2</td><td>51.3±1.4</td><td>55.9±4.3</td><td>56.2±4.0</td><td>55.8±4.4</td>
  </tr>
  <tr>
    <td>DMoE</td><td>39.8±4.8</td><td>54.1±1.2</td><td>56.0±4.1</td><td>56.6±3.8</td><td>56.4±5.1</td><td>37.5±2.9</td><td>52.2±2.4</td><td>55.7±3.8</td><td>56.1±3.4</td><td>56.4±5.1</td>
  </tr>
</table>

Table 5: The in-context learning results of models after training on 128 languages. The number of demonstration samples in the “Few-shot” setting is set to four in this work. Table 18 to 22 report results of all languages.

is interesting to find that languages belonging to the Niger-Congo family, which only take up 0.4GB in the pre-training corpus of BLOOM, benefit the most from our method (Figure 7).

![Bar chart showing average perplexity of BLOOM_560M across language families after training on the 128 languages.](page_1012_1342_388_246.png)

Figure 7: The average perplexity of BLOOM<sub>560M</sub> across language families after training on the 128 languages.

In-context learning results on five multilingual datasets are shown in Table 5. Appendix C.4 reports the results of all languages on these tasks. Similar to the language modeling results, DMoE also boosts the in-context learning performance and outperforms baseline models across two parameter amounts under zero-shot and few-shot settings.

The performance on the multilingual reasoning task XWinograd benefits most from our method, which improves 1.6% on average over the base model. It further demonstrates the effectiveness of our method in improving multilingual large language models.

5 Conclusion and Future Work

In this paper, we propose a method to mitigate the curse of multilinguality by augmenting parameters and boosting cross-lingual transfer. Multilingual large language models trained with our method achieve better language modeling and in-context learning performance than the continued pre-trained dense model and other scaling methods. These language-specialized experts make it easier to learn new languages and keep multilingual knowledge learned.

The specialization of experts can be further improved in the future, e.g., a shared expert learning general knowledge and other experts specializing in language-related knowledge. Designing a method to determine the language similarity with less cost or calculate better language clustering results is another direction. We hope this work can moti-
vate more studies on the curse of multilinguality phenomenon and put forward the development of multilingual language models.

Limitations

The first limitation lies in the additional computation to fine-tune and determine the parameter derivation for each language, which will linearly increase with the number of languages involved and the parameter amount of the model. Transferring the language similarity calculated from the small model into the larger model is a promising method to save computation.

The coverage of training and evaluation languages is another limitation. For example, languages from the Trans-New Guinea language family are not involved in this work. It is mainly due to the constrain of languages provided by the multilingual corpora and datasets used.

Our method relies on dynamic grouping languages and scaling parameters, which brings a higher training cost than the dense model. Due to the limited computation budget, the parameter amount of LLMs investigated in this work is less than 22.6B, and the token amount in training is restrained at 17.7B.

Acknowledgements

We would like to thank Junhong Wu and the anonymous reviewers for their helpful discussions and valuable comments. The research work was supported by the Natural Science Foundation of China (No. 62336008) and the Strategic Priority Research Program of Chinese Academy of Sciences (No. XDA04080400).

References

Roe Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, and et al. 2023. Palm 2 technical report. Preprint, arXiv:2305.10403.

Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. BUFFET: Benchmarking large language models for few-shot cross-lingual transfer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1771–1800, Mexico City, Mexico. Association for Computational Linguistics.

Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, and Luke Zettlemoyer. 2024. Breaking the curse of multilinguality with cross-lingual expert language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10822–10837, Miami, Florida, USA. Association for Computational Linguistics.

Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Ben Bergen. 2024. When is multilinguality a curse? language modeling for 250 high- and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4074–4096, Miami, Florida, USA. Association for Computational Linguistics.

Ronan Collobert, Samy Bengio, and Yoshua Bengio. 2001. A parallel mixture of svms for very large scale problems. In Advances in Neural Information Processing Systems, volume 14. MIT Press.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Alexis Conneau, Rutÿ Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. Preprint, arXiv:2401.06066.

Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Stable-MoE: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7085–7095, Dublin, Ireland. Association for Computational Linguistics.
John Dang, Shivalka Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraaat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Aycr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach, Haonan Li, Kyle McDonell, Niklas Muenmighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79–87.

Shaoxing Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyan O’Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and Barry Haddow. 2025. Emma-500: Enhancing massively multilingual adaptation of large language models. Preprint, arXiv:2409.17892.

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.

Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214.

Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A multilingual and document-level large audited dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. (GS)hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.

Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. Improving in-context learning of multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8058–8076, Mexico City, Mexico. Association for Computational Linguistics.

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. Preprint, arXiv:2208.03306.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL
and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8–14, Valencia, Spain. Association for Computational Linguistics.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Chaitanya Malaviya, Graham Neubig, and Patrick Litell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2529–2535, Copenhagen, Denmark. Association for Computational Linguistics.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed precision training. In International Conference on Learning Representations.

Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-llm: Bridging languages via massive multilingual training for cross-lingual enhancement. Preprint, arXiv:2412.04003.

MistralAI. 2023. Mixtral of experts. Mistral AI news.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huy Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL.

Robert Östling and Jörg Tiedemann. 2017. Continuous multilinguality with language vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 644–649, Valencia, Spain. Association for Computational Linguistics.

Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479–3495, Seattle, United States. Association for Computational Linguistics.

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI blog.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery.

Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E Weston. 2021. Hash layers for large sparse models. In Advances in Neural Information Processing Systems.

BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, and Alexandra Sasha Luccioni et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.

Babak Shahbaba and Radford Neal. 2009. Nonlinear models using dirichlet process mixtures. Journal of Machine Learning Research, 10(8).

Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations.

Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang-Wen Li, Wen tau Yih, Jason E Weston, and Xian Li. 2024. Branch-train-mix: Mixing expert LLMs into a mixture-of-experts LLM. In First Conference on Language Modeling.

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2020. Distill and replay for continual language learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3569–3579, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak,
Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and Pouya Tafti et al. 2024. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295.

Lucas Theis and Matthias Bethge. 2015. Generative image modeling using spatial Istsns. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.

Alexey Tikhonov and Max Ryabinin. 2021. It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534–3546, Online. Association for Computational Linguistics.

Volker Tresp. 2000. Mixtures of gaussian processes. In Advances in Neural Information Processing Systems, volume 13. MIT Press.

Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1357–1366, San Diego, California. Association for Computational Linguistics.

Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’ouza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muenninghoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics.

Qian Wang and Jiajun Zhang. 2022. Parameter differentiation based multilingual neural machine translation. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelfth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 11440–11448. AAAI Press.

Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwai Lu, Rui Hu, et al. 2023a. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341.

Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023b. Polylm: An open source polyglot large language model. Preprint, arXiv:2307.06018.

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, Thailand. Association for Computational Linguistics.

Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023. Overcoming catastrophic forgetting in massively multilingual continual learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 768–777. Toronto, Canada. Association for Computational Linguistics.

Junhong Wu, Yuchen Liu, and Chengqing Zong. 2024. F-MALLOC: Feed-forward memory allocation for continual learning in neural machine translation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7180–7192, Mexico City, Mexico. Association for Computational Linguistics.

Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bitranslate: Augmenting large language models with multilingual translation capability over 100 languages. Preprint, arXiv:2305.18098.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.

Yang Zhao, Junnan Zhu, Lu Xiang, Jiajun Zhang, Yu Zhou, Feifei Zhai, and Chengqing Zong. 2022. Life-long learning for multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv:2212.02800.
Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. 2025. Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. In AAAI-25. Sponsored by the Association for the Advancement of Artificial Intelligence. February 25 - March 4, 2025, Philadelphia, PA, USA, pages 26092–26100. AAAI Press.

Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. 2022. Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems, volume 35, pages 7103–7114. Curran Associates, Inc.

A Hyper-parameters

Following Scao et al. (2023), the global batch size is set to 512 samples with 2048 tokens during the language adaptation pre-training stage. AdamW optimizer (Loshchilov and Hutter, 2019) with \( \beta_1 = 0.9 \) and \( \beta_2 = 0.999 \) is used in this work. We empirically set the learning rate to 2e-5, adopt bf16 mixed precision training (Micikevicius et al., 2018) and ZeRO-3 (Rasley et al., 2020) to save GPU memory cost. And the \( \alpha \) in the loss function (Equation 4) is empirically set to 1.28. The model learns language-specialized experts and routers in the dynamic MoE layer extension stage, and is trained normally after that. All MoE models adopt top-2 routing during inference in this work.

B Language Delta

Figure 8 illustrates the cosine similarity of the parameter deviation during fine-tuning. It can be found that the deviation is relatively small after 10 tuning steps, and the cosine similarity of the one at the 10th step between the parameter deviation at the 40th step is higher than 80% for all languages. The language similarity matrices are similar using the parameter deviation at different steps (Figure 9). Therefore, we only fine-tune 10 steps to determine the parameter derivation of models for each language.

Figure 10 shows the distribution of parameter deviation \( \| \Delta \theta^x \| \) across layers of BLOOM_{560M} for 18 languages. It is interesting to find that layers near the embedding or output layer often have a relatively high parameter derivation \( \| \Delta \theta^x \| \).

C Additional Results

C.1 Language Similarity and Grouping

Figure 11 and 12 illustrates the language similarity matrix calculated by all layers and the last 3 layers of the parameter derivation of BLOOM_{1.7B}. Except for the difference in absolute value, they are very similar in relative trend. Therefore, we adopt the last 3 layers of the parameter derivation to calculate the language similarity by default.

Figure 13 shows pair-wise language similarity matrices of Gemma_{2B}. It can be found that the one of BLOOM_{1.7B} is similar to the one of BLOOM_{560M} (Figure 3), which may arise from the same pre-training corpus used. In contrast, the language similarity matrix of Gemma_{2B} has a higher average similarity value and different patterns between languages. As shown in Figure 14, It is interesting to find that replacing the multilingual corpus from CulturaX to MADLAD-400 results in a similar matrix.

Given the language similarity matrix calculated, we obtain the language grouping results for BLOOM_{1.7B} and Gemma_{2B} in the 18-language experiments using Algorithm 1 (Table 6 and 7). Similar languages like Tamil and Telugu are often grouped in the same language cluster. BLOOM_{560M} and BLOOM_{1.7B} have the same language clustering result under the six and two groups settings.

<table>
  <tr>
    <th></th>
    <th>ar</th>
    <th>ur</th>
    <th>bn</th>
    <th>it</th>
    <th>de</th>
    <th>nl</th>
    <th>ta</th>
    <th>te</th>
    <th>hi</th>
    <th>id</th>
    <th>fr</th>
    <th>vi</th>
    <th>ru</th>
    <th>uk</th>
    <th>th</th>
    <th>ko</th>
    <th>ja</th>
    <th>zh</th>
  </tr>
  <tr>
    <th>9G</th>
    <td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>6</td><td>3</td><td>1</td><td>7</td><td>7</td><td>8</td><td>8</td><td>4</td><td>2</td><td>9</td><td>9</td>
  </tr>
  <tr>
    <th>6G</th>
    <td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>3</td><td>3</td><td>4</td><td>4</td><td>4</td><td>5</td><td>5</td><td>5</td><td>6</td><td>6</td><td>6</td>
  </tr>
  <tr>
    <th>3G</th>
    <td>1</td><td>2</td><td>2</td><td>3</td><td>3</td><td>3</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>3</td><td>3</td><td>1</td>
  </tr>
  <tr>
    <th>2G</th>
    <td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td>
  </tr>
</table>

Table 6: The grouping results of BLOOM_{1.7B}, where '2G' denotes the result that divides into 2 groups.

Table 8 reports the 16 language groups used in the 128 languages experiment, which is calcu-

![The cosine similarity between Δθ^x at the i-th step and the one at the 40th step for each language using BLOOM_{560M}.](page_1012_120_410_312.png)

Figure 8: The cosine similarity between \( \Delta \theta^x \) at the i-th step and the one at the 40th step for each language using BLOOM_{560M}.
Figure 9: The language similarity matrices calculated by the parameter derivation at different fine-tuning steps using BLOOM_{560M}.

<table>
  <tr>
    <th>ar</th><th>ur</th><th>bn</th><th>it</th><th>de</th><th>nl</th><th>ta</th><th>te</th><th>hi</th><th>id</th><th>fr</th><th>vi</th><th>ru</th><th>uk</th><th>th</th><th>ko</th><th>ja</th><th>zh</th>
  </tr>
  <tr>
    <td>9G</td><td>1</td><td>1</td><td>2</td><td>6</td><td>3</td><td>4</td><td>4</td><td>2</td><td>5</td><td>6</td><td>5</td><td>7</td><td>7</td><td>8</td><td>8</td><td>9</td><td>9</td>
  </tr>
  <tr>
    <td>6G</td><td>1</td><td>1</td><td>2</td><td>3</td><td>4</td><td>2</td><td>2</td><td>1</td><td>3</td><td>4</td><td>5</td><td>3</td><td>5</td><td>5</td><td>6</td><td>6</td><td>6</td>
  </tr>
  <tr>
    <td>3G</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>3</td><td>2</td><td>3</td><td>3</td><td>3</td><td>3</td><td>3</td>
  </tr>
  <tr>
    <td>2G</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td>
  </tr>
</table>

Table 7: The grouping results of Gemma_{2B}, where “2G” denotes the result that divides into 2 groups.

lated by the parameter deviation of BLOOM_{1.7B}. BLOOM_{560M} adopts this result to save computation for the similar trend with BLOOM_{1.7B} in the 18-language experiment.

C.2 More base models on 18 Languages

We apply our method to Qwen2.5 base models and report the results in Table 9, which are in line with the previous results in Table 2.

C.3 Token Router Distribution

We statisticize the top-1 expert distribution across the mixture-of-experts layers in Figure 15 and 16. As shown in Figure 15, the language specialization emerges at the last five MoE layers, while MoE layers often show language specialization with router language classification loss (Figure 16).

C.4 Extend to 128 Languages

We report the perplexity of all 128 languages on Table 10 to 16. And in-context learning results on five multilingual datasets are shown in Table 18 to 22.
<table>
  <tr>
    <th>Index</th>
    <th>Languages</th>
    <th>Index</th>
    <th>Languages</th>
  </tr>
  <tr>
    <td>1</td>
    <td>ceb, en, fil, hil, ilo, la, lg, so</td>
    <td>9</td>
    <td>el, grc, hu, os, pl, ro, tr, yi</td>
  </tr>
  <tr>
    <td>2</td>
    <td>fr, it, ny, sn, sw, xh, yo, zu</td>
    <td>10</td>
    <td>bn, gu, kn, ml, mr, pa, ta, te</td>
  </tr>
  <tr>
    <td>3</td>
    <td>am, dv, he, ka, ko, lo, my, ti</td>
    <td>11</td>
    <td>az, br, ckb, fa, ps, sd, ug, ur</td>
  </tr>
  <tr>
    <td>4</td>
    <td>ar, ca, es, eu, hi, id, pt, vi</td>
    <td>12</td>
    <td>ht, ig, jv, mg, ms, sl, su, vec</td>
  </tr>
  <tr>
    <td>5</td>
    <td>cnh, cs, de, ha, kha, lus, nl, uz</td>
    <td>13</td>
    <td>fy, haw, lv, mi, sm, st, tet, to</td>
  </tr>
  <tr>
    <td>6</td>
    <td>kaa, kk, ky, mn, ru, sah, tt, tyv</td>
    <td>14</td>
    <td>co, co, gl, ja, ne, oc, yue, zh</td>
  </tr>
  <tr>
    <td>7</td>
    <td>da, et, fi, fo, gsw, is, no, se</td>
    <td>15</td>
    <td>ee, gd, hmn, lb, mt, om, rm, ts</td>
  </tr>
  <tr>
    <td>8</td>
    <td>av, be, ce, mk, sr, tg, udm, uk</td>
    <td>16</td>
    <td>bo, kbd, kl, km, pap, sa, th, tk</td>
  </tr>
</table>

Table 8: The 16 language groups divided for the 128 languages experiment.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#Param.</th>
    <th colspan="6">High</th>
    <th colspan="6">Medium</th>
    <th colspan="6">Low</th>
    <th rowspan="2">Avg ↓</th>
  </tr>
  <tr>
    <th>ar</th><th>de</th><th>fr</th><th>it</th><th>ja</th><th>nl</th><th>ru</th><th>zh</th><th>bn</th><th>hi</th><th>id</th><th>ko</th><th>th</th><th>uk</th><th>vi</th><th>ta</th><th>te</th><th>ur</th>
  </tr>
  <tr>
    <td>Qwen2.50SB</td>
    <td>494M</td>
    <td>110.8</td><td>147.3</td><td>9.5</td><td>133.8</td><td>19.8</td><td>20.2</td><td>3.7</td><td>36.8</td><td>336.2</td><td>191.4</td><td>85.2</td><td>2.5</td><td>3.4</td><td>5.4</td><td>23.3</td><td>44.7</td><td>2.1e<sup>5</sup></td><td>689.1</td><td>1262.0</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>494M</td>
    <td>75.7</td><td>89.2</td><td>8.3</td><td>75.3</td><td>15.9</td><td>10.9</td><td>3.4</td><td>32.4</td><td>40.6</td><td>32.8</td><td>56.9</td><td>2.4</td><td>2.8</td><td>3.2</td><td>18.7</td><td>9.0</td><td>192.5</td><td>48.8</td><td>39.9</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>444B</td>
    <td>74.1</td><td>87.9</td><td>8.2</td><td>72.0</td><td>15.3</td><td>10.7</td><td>3.4</td><td>30.3</td><td>51.9</td><td>40.2</td><td>55.5</td><td>2.3</td><td>2.8</td><td>3.2</td><td>18.2</td><td>11.9</td><td>404.1</td><td>57.4</td><td>52.8</td>
  </tr>
  <tr>
    <td>DMoE (9 Groups)</td>
    <td>175B</td>
    <td>68.8</td><td>81.1</td><td>8.0</td><td>67.9</td><td>15.3</td><td>9.8</td><td>3.4</td><td>31.6</td><td>33.6</td><td>27.8</td><td>51.8</td><td>2.3</td><td>2.7</td><td>3.0</td><td>17.7</td><td>8.6</td><td>173.0</td><td>38.7</td><td>35.8</td>
  </tr>
  <tr>
    <td>Qwen<sub>SB</sub></td>
    <td>1.54B</td>
    <td>59.4</td><td>70.8</td><td>7.0</td><td>64.4</td><td>13.2</td><td>12.2</td><td>2.0</td><td>25.1</td><td>136.2</td><td>86.4</td><td>45.0</td><td>2.2</td><td>2.8</td><td>3.9</td><td>15.0</td><td>29.6</td><td>4809.7</td><td>246.9</td><td>312.9</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>1.54B</td>
    <td>48.6</td><td>52.7</td><td>6.6</td><td>47.1</td><td>11.9</td><td>8.3</td><td>3.0</td><td>22.5</td><td>26.9</td><td>22.6</td><td>38.0</td><td>2.1</td><td>2.5</td><td>2.8</td><td>13.6</td><td>7.4</td><td>121.0</td><td>32.7</td><td>26.1</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>13.89B</td>
    <td>47.4</td><td>52.1</td><td>6.6</td><td>45.0</td><td>11.5</td><td>8.1</td><td>2.9</td><td>21.6</td><td>31.3</td><td>25.9</td><td>36.4</td><td>2.1</td><td>2.5</td><td>2.8</td><td>13.2</td><td>6.7</td><td>174.2</td><td>35.6</td><td>29.3</td>
  </tr>
  <tr>
    <td>DMoE (9 Groups)</td>
    <td>3.19B</td>
    <td>47.0</td><td>51.7</td><td>6.6</td><td>45.9</td><td>11.8</td><td>8.1</td><td>3.0</td><td>22.4</td><td>25.6</td><td>21.6</td><td>37.4</td><td>2.1</td><td>2.5</td><td>2.7</td><td>13.3</td><td>7.2</td><td>101.4</td><td>30.4</td><td>24.7</td>
  </tr>
</table>

Table 9: The normalized perplexity on the valid split of CulturaX. The perplexity is normalized to the vocabulary of Bloom following Wei et al. (2023a). “High”, “Medium”, and “Low” indicates the available amount of linguistic resources. The best and second results are denoted in bold and underlined, correspondingly.

<table>
  <tr>
    <th>Model</th>
    <th>am</th>
    <th>ar</th>
    <th>av</th>
    <th>az</th>
    <th>be</th>
    <th>bn</th>
    <th>bo</th>
    <th>br</th>
    <th>ca</th>
    <th>ce</th>
    <th>ceb</th>
    <th>ckb</th>
    <th>cnh</th>
    <th>co</th>
    <th>cs</th>
    <th>da</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>13.84</td><td>42.39</td><td>48.53</td><td>187.51</td><td>65.85</td><td>45.23</td><td>7.17</td><td>261.68</td><td>41.64</td><td>51.08</td><td>225.93</td><td>35.40</td><td>445.37</td><td>197.32</td><td>146.79</td><td>84.96</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>4.39</td><td>34.43</td><td>8.27</td><td>11.68</td><td>8.63</td><td>33.11</td><td>3.44</td><td>40.25</td><td>24.85</td><td>11.25</td><td>20.18</td><td>8.97</td><td>32.45</td><td>32.27</td><td>17.03</td><td>10.04</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>4.12</td><td>32.53</td><td>7.53</td><td>10.85</td><td>7.61</td><td>32.62</td><td>3.47</td><td>35.44</td><td>23.59</td><td>9.62</td><td>18.37</td><td>8.46</td><td>28.38</td><td>28.55</td><td>15.43</td><td>8.67</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>4.08</td><td>32.06</td><td>7.18</td><td>9.82</td><td>7.38</td><td>31.69</td><td>3.39</td><td>31.42</td><td>23.61</td><td>9.39</td><td>16.57</td><td>8.07</td><td>27.59</td><td>25.97</td><td>14.30</td><td>7.98</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td>
    <td>8.60</td><td>30.41</td><td>32.15</td><td>72.69</td><td>36.83</td><td>28.67</td><td>5.58</td><td>113.16</td><td>26.55</td><td>34.38</td><td>93.94</td><td>24.93</td><td>188.45</td><td>103.17</td><td>73.24</td><td>42.47</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>3.74</td><td>22.93</td><td>6.40</td><td>8.68</td><td>6.40</td><td>22.04</td><td>3.15</td><td>27.93</td><td>18.00</td><td>8.30</td><td>15.50</td><td>7.51</td><td>24.43</td><td>22.78</td><td>12.06</td><td>6.97</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>3.71</td><td>21.10</td><td>6.25</td><td>8.43</td><td>6.16</td><td>20.82</td><td>3.27</td><td>26.20</td><td>17.12</td><td>7.96</td><td>14.83</td><td>7.22</td><td>23.23</td><td>22.41</td><td>11.82</td><td>6.51</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>3.62</td><td>22.66</td><td>5.94</td><td>7.88</td><td>5.94</td><td>21.89</td><td>3.12</td><td>23.89</td><td>17.88</td><td>7.69</td><td>13.80</td><td>7.07</td><td>21.90</td><td>20.34</td><td>11.15</td><td>6.18</td>
  </tr>
</table>

Table 10: The perplexity of language “am” to “da” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

<table>
  <tr>
    <th>Model</th>
    <th>de</th>
    <th>dv</th>
    <th>ee</th>
    <th>el</th>
    <th>en</th>
    <th>eo</th>
    <th>es</th>
    <th>et</th>
    <th>eu</th>
    <th>fa</th>
    <th>fi</th>
    <th>fil</th>
    <th>fo</th>
    <th>fr</th>
    <th>fy</th>
    <th>gd</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>111.29</td><td>4.96</td><td>300.78</td><td>18.28</td><td>66.79</td><td>266.55</td><td>39.23</td><td>310.09</td><td>80.16</td><td>161.83</td><td>250.93</td><td>234.67</td><td>253.93</td><td>43.25</td><td>285.93</td><td>172.37</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>23.83</td><td>2.10</td><td>20.79</td><td>4.34</td><td>44.58</td><td>31.29</td><td>24.03</td><td>29.32</td><td>39.50</td><td>14.98</td><td>24.25</td><td>17.72</td><td>24.34</td><td>27.21</td><td>24.39</td><td>15.01</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>21.03</td><td>2.11</td><td>19.75</td><td>4.18</td><td>40.29</td><td>26.97</td><td>23.67</td><td>24.47</td><td>37.73</td><td>14.51</td><td>21.26</td><td>15.86</td><td>20.78</td><td>25.53</td><td>20.67</td><td>12.95</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>19.46</td><td>2.05</td><td>15.78</td><td>4.02</td><td>40.82</td><td>22.84</td><td>23.14</td><td>22.67</td><td>36.67</td><td>13.40</td><td>18.91</td><td>14.18</td><td>19.00</td><td>25.97</td><td>18.13</td><td>11.55</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td>
    <td>56.36</td><td>3.49</td><td>121.31</td><td>12.17</td><td>45.33</td><td>126.18</td><td>27.31</td><td>151.08</td><td>40.32</td><td>86.79</td><td>113.01</td><td>98.05</td><td>111.79</td><td>28.28</td><td>143.77</td><td>84.11</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>15.89</td><td>1.97</td><td>35.7</td><td>3.53</td><td>30.27</td><td>21.05</td><td>18.15</td><td>19.85</td><td>28.41</td><td>11.96</td><td>16.76</td><td>13.23</td><td>16.88</td><td>20.03</td><td>16.03</td><td>11.38</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>15.34</td><td>2.00</td><td>14.43</td><td>3.57</td><td>29.69</td><td>20.33</td><td>17.43</td><td>18.64</td><td>24.97</td><td>11.35</td><td>16.27</td><td>12.54</td><td>15.78</td><td>18.89</td><td>14.92</td><td>10.30</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>14.86</td><td>1.94</td><td>13.35</td><td>3.43</td><td>31.34</td><td>17.73</td><td>18.19</td><td>17.50</td><td>25.37</td><td>10.89</td><td>15.07</td><td>11.62</td><td>14.72</td><td>20.01</td><td>13.55</td><td>9.85</td>
  </tr>
</table>

Table 11: The perplexity of language “de” to “gd” on the valid split of MADLAD-400 (Kudugunta et al., 2023).
<table>
  <tr>
    <th>Model</th>
    <th>gl</th>
    <th>grc</th>
    <th>gsw</th>
    <th>gu</th>
    <th>ha</th>
    <th>haw</th>
    <th>he</th>
    <th>hi</th>
    <th>hil</th>
    <th>hmn</th>
    <th>ht</th>
    <th>hu</th>
    <th>id</th>
    <th>ig</th>
    <th>ilo</th>
    <th>is</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>121.12</td>
    <td>19.74</td>
    <td>143.45</td>
    <td>180.03</td>
    <td>559.77</td>
    <td>73.99</td>
    <td>20.93</td>
    <td>30.55</td>
    <td>220.05</td>
    <td>120.38</td>
    <td>391.80</td>
    <td>156.40</td>
    <td>41.35</td>
    <td>262.93</td>
    <td>271.81</td>
    <td>207.29</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>26.66</td>
    <td>6.00</td>
    <td>42.16</td>
    <td>39.27</td>
    <td>27.09</td>
    <td>10.39</td>
    <td>7.01</td>
    <td>20.45</td>
    <td>17.53</td>
    <td>13.11</td>
    <td>33.86</td>
    <td>15.41</td>
    <td>26.93</td>
    <td>28.80</td>
    <td>26.04</td>
    <td>20.08</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>24.84</td>
    <td>5.69</td>
    <td>37.22</td>
    <td>40.53</td>
    <td>24.27</td>
    <td>9.67</td>
    <td>6.71</td>
    <td>19.98</td>
    <td>14.53</td>
    <td>11.94</td>
    <td>30.05</td>
    <td>14.10</td>
    <td>26.10</td>
    <td>26.85</td>
    <td>22.45</td>
    <td>17.36</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>23.70</td>
    <td>5.45</td>
    <td>35.14</td>
    <td>37.51</td>
    <td>22.80</td>
    <td>9.00</td>
    <td>6.44</td>
    <td>19.78</td>
    <td>13.22</td>
    <td>10.78</td>
    <td>26.19</td>
    <td>13.00</td>
    <td>25.81</td>
    <td>23.77</td>
    <td>19.76</td>
    <td>15.98</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>61.19</td>
    <td>12.99</td>
    <td>90.97</td>
    <td>67.49</td>
    <td>191.64</td>
    <td>47.25</td>
    <td>15.00</td>
    <td>20.82</td>
    <td>96.72</td>
    <td>79.33</td>
    <td>169.97</td>
    <td>70.31</td>
    <td>27.80</td>
    <td>92.00</td>
    <td>130.53</td>
    <td>92.95</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>18.81</td>
    <td>4.71</td>
    <td>28.90</td>
    <td>27.69</td>
    <td>19.49</td>
    <td>8.48</td>
    <td>5.58</td>
    <td>14.70</td>
    <td>12.23</td>
    <td>10.47</td>
    <td>23.94</td>
    <td>10.98</td>
    <td>19.25</td>
    <td>20.90</td>
    <td>18.45</td>
    <td>14.12</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>18.04</td>
    <td>4.74</td>
    <td>27.91</td>
    <td>27.50</td>
    <td>18.94</td>
    <td>8.24</td>
    <td>5.55</td>
    <td>14.14</td>
    <td>10.97</td>
    <td>9.79</td>
    <td>21.31</td>
    <td>10.91</td>
    <td>18.65</td>
    <td>19.81</td>
    <td>17.42</td>
    <td>13.36</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>17.84</td>
    <td>4.60</td>
    <td>26.20</td>
    <td>27.18</td>
    <td>17.73</td>
    <td>7.91</td>
    <td>5.41</td>
    <td>14.68</td>
    <td>10.44</td>
    <td>9.48</td>
    <td>20.24</td>
    <td>10.27</td>
    <td>19.08</td>
    <td>18.63</td>
    <td>15.59</td>
    <td>12.44</td>
  </tr>
</table>

Table 12: The perplexity of language “gl” to “is” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

<table>
  <tr>
    <th>Model</th>
    <th>it</th>
    <th>ja</th>
    <th>lv</th>
    <th>ka</th>
    <th>kaa</th>
    <th>kbd</th>
    <th>kha</th>
    <th>kk</th>
    <th>kl</th>
    <th>km</th>
    <th>kn</th>
    <th>ko</th>
    <th>ky</th>
    <th>la</th>
    <th>lb</th>
    <th>lg</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>82.38</td>
    <td>55.55</td>
    <td>277.50</td>
    <td>12.20</td>
    <td>50.53</td>
    <td>30.80</td>
    <td>200.91</td>
    <td>28.98</td>
    <td>231.11</td>
    <td>10.09</td>
    <td>196.03</td>
    <td>24.03</td>
    <td>47.03</td>
    <td>111.79</td>
    <td>261.41</td>
    <td>369.16</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>20.47</td>
    <td>12.86</td>
    <td>36.26</td>
    <td>4.16</td>
    <td>6.56</td>
    <td>7.68</td>
    <td>20.35</td>
    <td>5.71</td>
    <td>16.50</td>
    <td>3.85</td>
    <td>46.08</td>
    <td>7.36</td>
    <td>7.37</td>
    <td>36.19</td>
    <td>25.71</td>
    <td>46.19</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>17.12</td>
    <td>12.12</td>
    <td>34.82</td>
    <td>4.08</td>
    <td>6.16</td>
    <td>7.12</td>
    <td>18.22</td>
    <td>5.38</td>
    <td>14.78</td>
    <td>3.81</td>
    <td>47.55</td>
    <td>7.12</td>
    <td>6.89</td>
    <td>34.72</td>
    <td>21.35</td>
    <td>40.71</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>16.33</td>
    <td>11.34</td>
    <td>30.43</td>
    <td>3.89</td>
    <td>5.95</td>
    <td>6.35</td>
    <td>17.05</td>
    <td>5.20</td>
    <td>13.57</td>
    <td>3.63</td>
    <td>43.80</td>
    <td>6.74</td>
    <td>6.61</td>
    <td>32.04</td>
    <td>18.92</td>
    <td>36.13</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>44.86</td>
    <td>35.01</td>
    <td>149.22</td>
    <td>10.42</td>
    <td>32.12</td>
    <td>20.19</td>
    <td>124.88</td>
    <td>19.14</td>
    <td>120.42</td>
    <td>8.26</td>
    <td>92.29</td>
    <td>16.12</td>
    <td>31.49</td>
    <td>66.75</td>
    <td>132.24</td>
    <td>141.18</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>14.13</td>
    <td>9.67</td>
    <td>26.62</td>
    <td>3.63</td>
    <td>5.12</td>
    <td>5.86</td>
    <td>14.83</td>
    <td>4.52</td>
    <td>12.68</td>
    <td>3.36</td>
    <td>32.93</td>
    <td>6.00</td>
    <td>5.73</td>
    <td>27.12</td>
    <td>16.92</td>
    <td>33.36</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>13.10</td>
    <td>9.39</td>
    <td>25.49</td>
    <td>3.66</td>
    <td>5.14</td>
    <td>5.83</td>
    <td>14.51</td>
    <td>4.50</td>
    <td>12.57</td>
    <td>3.39</td>
    <td>32.53</td>
    <td>5.96</td>
    <td>5.73</td>
    <td>27.40</td>
    <td>15.66</td>
    <td>31.24</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>12.83</td>
    <td>8.92</td>
    <td>23.54</td>
    <td>3.51</td>
    <td>4.86</td>
    <td>5.50</td>
    <td>13.19</td>
    <td>4.31</td>
    <td>11.14</td>
    <td>3.25</td>
    <td>32.03</td>
    <td>5.72</td>
    <td>5.44</td>
    <td>26.05</td>
    <td>14.69</td>
    <td>28.27</td>
  </tr>
</table>

Table 13: The perplexity of language “it” to “lg” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

<table>
  <tr>
    <th>Model</th>
    <th>lo</th>
    <th>lus</th>
    <th>lv</th>
    <th>mg</th>
    <th>mi</th>
    <th>mk</th>
    <th>ml</th>
    <th>mn</th>
    <th>mr</th>
    <th>ms</th>
    <th>mt</th>
    <th>my</th>
    <th>ne</th>
    <th>nl</th>
    <th>no</th>
    <th>ny</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>8.72</td>
    <td>312.58</td>
    <td>169.08</td>
    <td>189.81</td>
    <td>129.33</td>
    <td>62.40</td>
    <td>116.36</td>
    <td>31.15</td>
    <td>127.22</td>
    <td>98.74</td>
    <td>97.91</td>
    <td>6.08</td>
    <td>125.47</td>
    <td>118.66</td>
    <td>213.49</td>
    <td>286.37</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>2.44</td>
    <td>37.91</td>
    <td>15.10</td>
    <td>18.63</td>
    <td>16.50</td>
    <td>9.37</td>
    <td>34.71</td>
    <td>6.75</td>
    <td>36.21</td>
    <td>33.69</td>
    <td>9.95</td>
    <td>2.77</td>
    <td>51.30</td>
    <td>15.22</td>
    <td>22.27</td>
    <td>22.99</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>2.42</td>
    <td>34.92</td>
    <td>12.49</td>
    <td>16.38</td>
    <td>15.14</td>
    <td>8.45</td>
    <td>35.88</td>
    <td>6.31</td>
    <td>36.48</td>
    <td>32.98</td>
    <td>8.94</td>
    <td>2.77</td>
    <td>53.62</td>
    <td>13.01</td>
    <td>19.45</td>
    <td>20.69</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>2.37</td>
    <td>32.79</td>
    <td>10.65</td>
    <td>15.08</td>
    <td>13.82</td>
    <td>8.05</td>
    <td>33.29</td>
    <td>6.19</td>
    <td>34.98</td>
    <td>31.17</td>
    <td>7.42</td>
    <td>2.69</td>
    <td>49.34</td>
    <td>12.22</td>
    <td>17.92</td>
    <td>18.37</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>4.58</td>
    <td>159.69</td>
    <td>83.05</td>
    <td>82.55</td>
    <td>74.40</td>
    <td>36.15</td>
    <td>50.08</td>
    <td>21.05</td>
    <td>61.00</td>
    <td>51.52</td>
    <td>59.86</td>
    <td>4.41</td>
    <td>63.23</td>
    <td>58.05</td>
    <td>97.54</td>
    <td>126.71</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>2.17</td>
    <td>28.49</td>
    <td>10.26</td>
    <td>14.08</td>
    <td>13.02</td>
    <td>6.96</td>
    <td>23.58</td>
    <td>5.37</td>
    <td>24.57</td>
    <td>23.49</td>
    <td>7.26</td>
    <td>2.52</td>
    <td>35.24</td>
    <td>10.26</td>
    <td>15.45</td>
    <td>17.08</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>2.20</td>
    <td>28.23</td>
    <td>9.05</td>
    <td>12.81</td>
    <td>12.46</td>
    <td>6.74</td>
    <td>23.43</td>
    <td>5.35</td>
    <td>24.15</td>
    <td>22.88</td>
    <td>6.67</td>
    <td>2.57</td>
    <td>35.40</td>
    <td>9.67</td>
    <td>14.52</td>
    <td>15.98</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>2.13</td>
    <td>26.04</td>
    <td>8.45</td>
    <td>12.54</td>
    <td>11.84</td>
    <td>6.51</td>
    <td>23.58</td>
    <td>5.13</td>
    <td>24.06</td>
    <td>22.18</td>
    <td>6.27</td>
    <td>2.48</td>
    <td>34.91</td>
    <td>9.37</td>
    <td>13.86</td>
    <td>15.11</td>
  </tr>
</table>

Table 14: The perplexity of language “lo” to “ny” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

<table>
  <tr>
    <th>Model</th>
    <th>oc</th>
    <th>om</th>
    <th>os</th>
    <th>pa</th>
    <th>pap</th>
    <th>pl</th>
    <th>ps</th>
    <th>pt</th>
    <th>rm</th>
    <th>ro</th>
    <th>ru</th>
    <th>sa</th>
    <th>sah</th>
    <th>sd</th>
    <th>se</th>
    <th>sl</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>80.82</td>
    <td>260.13</td>
    <td>30.78</td>
    <td>131.60</td>
    <td>367.89</td>
    <td>91.73</td>
    <td>67.38</td>
    <td>37.27</td>
    <td>354.95</td>
    <td>195.23</td>
    <td>33.77</td>
    <td>181.38</td>
    <td>32.12</td>
    <td>83.51</td>
    <td>255.36</td>
    <td>218.52</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>20.72</td>
    <td>27.23</td>
    <td>8.08</td>
    <td>32.55</td>
    <td>28.52</td>
    <td>12.84</td>
    <td>10.99</td>
    <td>23.22</td>
    <td>27.98</td>
    <td>16.72</td>
    <td>8.13</td>
    <td>43.64</td>
    <td>6.88</td>
    <td>13.97</td>
    <td>27.01</td>
    <td>22.97</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>19.13</td>
    <td>23.77</td>
    <td>7.56</td>
    <td>34.14</td>
    <td>25.53</td>
    <td>11.91</td>
    <td>10.65</td>
    <td>22.23</td>
    <td>23.60</td>
    <td>15.09</td>
    <td>7.40</td>
    <td>47.25</td>
    <td>6.51</td>
    <td>14.48</td>
    <td>23.05</td>
    <td>19.99</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>17.82</td>
    <td>21.06</td>
    <td>7.06</td>
    <td>30.48</td>
    <td>22.65</td>
    <td>11.12</td>
    <td>9.59</td>
    <td>22.51</td>
    <td>18.69</td>
    <td>14.06</td>
    <td>7.24</td>
    <td>41.47</td>
    <td>6.20</td>
    <td>12.62</td>
    <td>21.32</td>
    <td>17.32</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>45.64</td>
    <td>150.00</td>
    <td>21.03</td>
    <td>61.74</td>
    <td>187.88</td>
    <td>46.24</td>
    <td>44.31</td>
    <td>24.85</td>
    <td>183.77</td>
    <td>87.28</td>
    <td>21.70</td>
    <td>104.87</td>
    <td>22.39</td>
    <td>156.62</td>
    <td>107.83</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>14.69</td>
    <td>20.86</td>
    <td>6.20</td>
    <td>23.89</td>
    <td>19.32</td>
    <td>9.20</td>
    <td>8.68</td>
    <td>17.07</td>
    <td>18.42</td>
    <td>11.82</td>
    <td>6.16</td>
    <td>34.04</td>
    <td>5.45</td>
    <td>11.45</td>
    <td>18.82</td>
    <td>15.75</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>14.14</td>
    <td>19.52</td>
    <td>6.36</td>
    <td>24.24</td>
    <td>17.96</td>
    <td>9.25</td>
    <td>8.46</td>
    <td>16.35</td>
    <td>15.92</td>
    <td>11.42</td>
    <td>5.93</td>
    <td>36.76</td>
    <td>5.52</td>
    <td>11.82</td>
    <td>17.97</td>
    <td>14.49</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>13.36</td>
    <td>17.80</td>
    <td>5.68</td>
    <td>23.17</td>
    <td>16.37</td>
    <td>8.66</td>
    <td>7.95</td>
    <td>17.17</td>
    <td>14.33</td>
    <td>10.93</td>
    <td>5.82</td>
    <td>32.97</td>
    <td>5.18</td>
    <td>10.71</td>
    <td>16.31</td>
    <td>13.48</td>
  </tr>
</table>

Table 15: The perplexity of language “oc” to “sl” on the valid split of MADLAD-400 (Kudugunta et al., 2023).
<table>
  <tr>
    <th>Model</th>
    <th>sm</th>
    <th>sn</th>
    <th>so</th>
    <th>sr</th>
    <th>st</th>
    <th>su</th>
    <th>sw</th>
    <th>ta</th>
    <th>te</th>
    <th>tet</th>
    <th>tg</th>
    <th>th</th>
    <th>ti</th>
    <th>tk</th>
    <th>to</th>
    <th>tr</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>114.75</td>
    <td>435.89</td>
    <td>239.57</td>
    <td>51.57</td>
    <td>275.07</td>
    <td>224.84</td>
    <td>224.92</td>
    <td>80.03</td>
    <td>91.45</td>
    <td>206.03</td>
    <td>40.17</td>
    <td>13.71</td>
    <td>16.22</td>
    <td>217.70</td>
    <td>82.62</td>
    <td>153.97</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>15.29</td>
    <td>28.67</td>
    <td>18.92</td>
    <td>9.57</td>
    <td>18.37</td>
    <td>27.30</td>
    <td>45.46</td>
    <td>37.65</td>
    <td>30.32</td>
    <td>16.00</td>
    <td>7.22</td>
    <td>3.64</td>
    <td>5.13</td>
    <td>14.84</td>
    <td>14.60</td>
    <td>14.62</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>13.78</td>
    <td>25.29</td>
    <td>16.27</td>
    <td>8.38</td>
    <td>16.60</td>
    <td>25.26</td>
    <td>43.52</td>
    <td>38.29</td>
    <td>31.56</td>
    <td>14.34</td>
    <td>6.62</td>
    <td>3.49</td>
    <td>4.83</td>
    <td>13.70</td>
    <td>12.54</td>
    <td>13.49</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>12.63</td>
    <td>22.63</td>
    <td>14.53</td>
    <td>8.07</td>
    <td>14.96</td>
    <td>22.70</td>
    <td>39.65</td>
    <td>36.32</td>
    <td>29.26</td>
    <td>12.48</td>
    <td>6.36</td>
    <td>3.43</td>
    <td>4.70</td>
    <td>12.08</td>
    <td>11.60</td>
    <td>12.80</td>
  </tr>
</table>

<table>
  <tr>
    <th>Model</th>
    <th>ts</th>
    <th>tt</th>
    <th>tyv</th>
    <th>udm</th>
    <th>ug</th>
    <th>uk</th>
    <th>ur</th>
    <th>uz</th>
    <th>vec</th>
    <th>vi</th>
    <th>xh</th>
    <th>yi</th>
    <th>yo</th>
    <th>yue</th>
    <th>zh</th>
    <th>zu</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>175.78</td>
    <td>39.70</td>
    <td>40.14</td>
    <td>54.51</td>
    <td>44.53</td>
    <td>44.49</td>
    <td>72.47</td>
    <td>388.73</td>
    <td>319.37</td>
    <td>21.85</td>
    <td>606.11</td>
    <td>17.27</td>
    <td>257.05</td>
    <td>71.22</td>
    <td>36.26</td>
    <td>1278.88</td>
  </tr>
  <tr>
    <td>+ Pre-train</td>
    <td>13.05</td>
    <td>7.46</td>
    <td>8.37</td>
    <td>8.87</td>
    <td>9.21</td>
    <td>7.30</td>
    <td>37.61</td>
    <td>18.66</td>
    <td>73.68</td>
    <td>15.12</td>
    <td>34.44</td>
    <td>5.10</td>
    <td>28.08</td>
    <td>25.72</td>
    <td>15.07</td>
    <td>36.17</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>11.26</td>
    <td>6.97</td>
    <td>7.54</td>
    <td>7.95</td>
    <td>8.99</td>
    <td>6.61</td>
    <td>37.13</td>
    <td>16.74</td>
    <td>64.42</td>
    <td>14.62</td>
    <td>32.00</td>
    <td>4.86</td>
    <td>27.41</td>
    <td>25.82</td>
    <td>14.83</td>
    <td>31.24</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>9.90</td>
    <td>6.71</td>
    <td>7.33</td>
    <td>7.65</td>
    <td>8.10</td>
    <td>6.40</td>
    <td>35.19</td>
    <td>15.80</td>
    <td>55.43</td>
    <td>14.66</td>
    <td>27.73</td>
    <td>4.71</td>
    <td>24.28</td>
    <td>23.99</td>
    <td>14.31</td>
    <td>28.31</td>
  </tr>
</table>

<table>
  <tr>
    <th>Model</th>
    <th colspan="10">High</th>
    <th colspan="4">Medium</th>
    <th colspan="4">Low</th>
    <th>Avg</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>#shot</th>
    <th>en</th>
    <th>de†</th>
    <th>es</th>
    <th>eu</th>
    <th>fr</th>
    <th>ru†</th>
    <th>zh</th>
    <th>ar</th>
    <th>bg†</th>
    <th>el†</th>
    <th>th†</th>
    <th>tr†</th>
    <th>vi</th>
    <th>hi</th>
    <th>sw</th>
    <th>ur</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>0</td>
    <td>43.9</td>
    <td>34.4</td>
    <td>40.5</td>
    <td>37.9</td>
    <td>39.1</td>
    <td>34.6</td>
    <td>35.5</td>
    <td>33.5</td>
    <td>34.0</td>
    <td>35.2</td>
    <td>32.1</td>
    <td>31.6</td>
    <td>39.0</td>
    <td>39.8</td>
    <td>33.9</td>
    <td>34.5</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>40.3</td>
    <td>34.0</td>
    <td>38.6</td>
    <td>35.0</td>
    <td>37.0</td>
    <td>34.4</td>
    <td>32.4</td>
    <td>33.3</td>
    <td>33.4</td>
    <td>31.8</td>
    <td>33.5</td>
    <td>32.1</td>
    <td>36.0</td>
    <td>33.7</td>
    <td>31.8</td>
    <td>34.1</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub> + Pre-train</td>
    <td>0</td>
    <td>43.5</td>
    <td>36.7</td>
    <td>41.6</td>
    <td>40.6</td>
    <td>40.9</td>
    <td>36.1</td>
    <td>33.4</td>
    <td>33.5</td>
    <td>34.1</td>
    <td>33.1</td>
    <td>33.8</td>
    <td>33.4</td>
    <td>42.0</td>
    <td>38.5</td>
    <td>33.9</td>
    <td>37.1</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>40.3</td>
    <td>33.9</td>
    <td>38.5</td>
    <td>36.7</td>
    <td>38.4</td>
    <td>32.7</td>
    <td>35.5</td>
    <td>33.4</td>
    <td>31.6</td>
    <td>33.4</td>
    <td>31.8</td>
    <td>32.8</td>
    <td>36.3</td>
    <td>35.3</td>
    <td>32.2</td>
    <td>34.1</td>
    <td>34.7</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>47.4</td>
    <td>36.0</td>
    <td>41.9</td>
    <td>41.2</td>
    <td>41.1</td>
    <td>36.7</td>
    <td>33.8</td>
    <td>33.5</td>
    <td>34.1</td>
    <td>34.9</td>
    <td>32.6</td>
    <td>34.1</td>
    <td>41.7</td>
    <td>35.5</td>
    <td>33.5</td>
    <td>36.7</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>41.6</td>
    <td>35.8</td>
    <td>39.8</td>
    <td>35.0</td>
    <td>38.6</td>
    <td>36.1</td>
    <td>34.9</td>
    <td>32.7</td>
    <td>34.2</td>
    <td>33.1</td>
    <td>34.3</td>
    <td>33.1</td>
    <td>35.8</td>
    <td>33.9</td>
    <td>32.0</td>
    <td>33.9</td>
    <td>35.3</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>48.3</td>
    <td>36.3</td>
    <td>43.8</td>
    <td>38.6</td>
    <td>42.9</td>
    <td>37.3</td>
    <td>33.4</td>
    <td>33.3</td>
    <td>34.5</td>
    <td>34.1</td>
    <td>34.0</td>
    <td>33.9</td>
    <td>43.0</td>
    <td>37.7</td>
    <td>33.2</td>
    <td>36.2</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>41.2</td>
    <td>35.2</td>
    <td>40.5</td>
    <td>35.2</td>
    <td>39.0</td>
    <td>35.8</td>
    <td>35.2</td>
    <td>33.9</td>
    <td>35.4</td>
    <td>33.9</td>
    <td>33.3</td>
    <td>33.6</td>
    <td>37.7</td>
    <td>35.1</td>
    <td>32.1</td>
    <td>34.0</td>
    <td>35.7</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>0</td>
    <td>49.2</td>
    <td>36.6</td>
    <td>47.7</td>
    <td>47.0</td>
    <td>45.2</td>
    <td>37.8</td>
    <td>34.9</td>
    <td>33.3</td>
    <td>35.6</td>
    <td>33.6</td>
    <td>33.7</td>
    <td>35.3</td>
    <td>42.9</td>
    <td>42.2</td>
    <td>34.1</td>
    <td>38.4</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>46.3</td>
    <td>34.7</td>
    <td>43.1</td>
    <td>40.9</td>
    <td>45.0</td>
    <td>35.3</td>
    <td>38.0</td>
    <td>32.9</td>
    <td>35.0</td>
    <td>33.1</td>
    <td>32.9</td>
    <td>31.2</td>
    <td>37.9</td>
    <td>38.4</td>
    <td>32.6</td>
    <td>35.5</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub> + Pre-train</td>
    <td>0</td>
    <td>49.4</td>
    <td>40.8</td>
    <td>46.7</td>
    <td>42.6</td>
    <td>43.5</td>
    <td>42.4</td>
    <td>33.7</td>
    <td>33.5</td>
    <td>35.1</td>
    <td>34.3</td>
    <td>37.4</td>
    <td>32.7</td>
    <td>42.6</td>
    <td>38.6</td>
    <td>34.9</td>
    <td>38.6</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>45.0</td>
    <td>37.3</td>
    <td>40.4</td>
    <td>37.8</td>
    <td>42.4</td>
    <td>38.8</td>
    <td>35.4</td>
    <td>34.0</td>
    <td>35.2</td>
    <td>33.7</td>
    <td>34.3</td>
    <td>33.4</td>
    <td>37.5</td>
    <td>36.8</td>
    <td>34.4</td>
    <td>36.6</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>49.7</td>
    <td>43.4</td>
    <td>46.0</td>
    <td>44.1</td>
    <td>44.1</td>
    <td>41.4</td>
    <td>33.8</td>
    <td>33.7</td>
    <td>34.5</td>
    <td>35.8</td>
    <td>34.1</td>
    <td>38.4</td>
    <td>45.5</td>
    <td>38.0</td>
    <td>35.5</td>
    <td>37.1</td>
    <td>39.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>44.7</td>
    <td>38.4</td>
    <td>41.2</td>
    <td>37.3</td>
    <td>41.5</td>
    <td>38.2</td>
    <td>35.3</td>
    <td>34.4</td>
    <td>34.9</td>
    <td>33.6</td>
    <td>33.8</td>
    <td>37.8</td>
    <td>35.9</td>
    <td>34.3</td>
    <td>34.1</td>
    <td>36.8</td>
    <td>36.8</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>50.6</td>
    <td>42.6</td>
    <td>45.4</td>
    <td>44.3</td>
    <td>43.1</td>
    <td>41.3</td>
    <td>34.6</td>
    <td>33.5</td>
    <td>36.2</td>
    <td>37.1</td>
    <td>35.1</td>
    <td>34.3</td>
    <td>44.7</td>
    <td>39.2</td>
    <td>36.2</td>
    <td>39.7</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>44.9</td>
    <td>36.9</td>
    <td>40.6</td>
    <td>38.2</td>
    <td>41.8</td>
    <td>39.2</td>
    <td>37.4</td>
    <td>33.9</td>
    <td>36.0</td>
    <td>36.2</td>
    <td>35.9</td>
    <td>32.8</td>
    <td>37.7</td>
    <td>36.8</td>
    <td>34.5</td>
    <td>36.7</td>
    <td>37.5</td>
  </tr>
</table>

Table 16: The perplexity of language “sm” to “tr” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

Table 17: The perplexity of language “ts” to “zu” on the valid split of MADLAD-400 (Kudugunta et al., 2023).

Table 18: In-context learning results on XNLI across all languages. “High”, “Medium” and “Low” denotes the available amount of linguistic resources. † denotes the unseen language in the pre-training corpus of BLOOM.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#shot</th>
    <th colspan="6">High</th>
    <th colspan="3">Medium</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>de<sup>†</sup></th>
    <th>en</th>
    <th>es</th>
    <th>fr</th>
    <th>ja<sup>†</sup></th>
    <th>zh</th>
    <th>ko<sup>†</sup></th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>0</td>
    <td>49.4</td>
    <td>49.9</td>
    <td>50.4</td>
    <td>52.8</td>
    <td>52.8</td>
    <td>54.1</td>
    <td>51.0</td>
    <td>51.5</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>52.5</td>
    <td>50.0</td>
    <td>49.7</td>
    <td>51.9</td>
    <td>51.3</td>
    <td>52.5</td>
    <td>49.7</td>
    <td>51.1</td>
    <td></td>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub> + Pre-train</td>
    <td>0</td>
    <td>49.7</td>
    <td>50.2</td>
    <td>50.7</td>
    <td>54.8</td>
    <td>55.7</td>
    <td>55.1</td>
    <td>54.1</td>
    <td>52.9</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>50.9</td>
    <td>51.5</td>
    <td>51.1</td>
    <td>51.9</td>
    <td>50.3</td>
    <td>54.1</td>
    <td>51.3</td>
    <td>51.6</td>
    <td></td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>51.3</td>
    <td>49.1</td>
    <td>50.9</td>
    <td>54.6</td>
    <td>56.2</td>
    <td>55.3</td>
    <td>54.7</td>
    <td>53.1</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>48.3</td>
    <td>50.5</td>
    <td>53.0</td>
    <td>52.3</td>
    <td>50.1</td>
    <td>55.1</td>
    <td>50.4</td>
    <td>51.4</td>
    <td></td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>51.4</td>
    <td>51.1</td>
    <td>51.2</td>
    <td>54.6</td>
    <td>54.9</td>
    <td>55.2</td>
    <td>54.0</td>
    <td>53.2</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>51.3</td>
    <td>50.1</td>
    <td>52.4</td>
    <td>52.8</td>
    <td>51.8</td>
    <td>54.8</td>
    <td>52.2</td>
    <td>52.2</td>
    <td></td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>0</td>
    <td>52.6</td>
    <td>53.8</td>
    <td>50.7</td>
    <td>54.9</td>
    <td>55.7</td>
    <td>54.8</td>
    <td>54.8</td>
    <td>53.9</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>48.9</td>
    <td>50.4</td>
    <td>49.8</td>
    <td>51.4</td>
    <td>50.1</td>
    <td>50.4</td>
    <td>52.8</td>
    <td>50.5</td>
    <td></td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub> + Pre-train</td>
    <td>0</td>
    <td>53.3</td>
    <td>51.1</td>
    <td>52.7</td>
    <td>54.9</td>
    <td>55.9</td>
    <td>54.9</td>
    <td>53.5</td>
    <td>53.7</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>51.4</td>
    <td>48.6</td>
    <td>51.6</td>
    <td>53.2</td>
    <td>54.0</td>
    <td>53.2</td>
    <td>50.8</td>
    <td>51.8</td>
    <td></td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>51.3</td>
    <td>53.1</td>
    <td>52.8</td>
    <td>54.8</td>
    <td>55.8</td>
    <td>55.3</td>
    <td>51.7</td>
    <td>53.5</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>50.0</td>
    <td>49.4</td>
    <td>52.2</td>
    <td>53.2</td>
    <td>50.3</td>
    <td>52.9</td>
    <td>51.2</td>
    <td>51.3</td>
    <td></td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>53.0</td>
    <td>53.7</td>
    <td>52.8</td>
    <td>54.8</td>
    <td>55.9</td>
    <td>55.4</td>
    <td>53.0</td>
    <td>54.1</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>48.8</td>
    <td>49.2</td>
    <td>50.9</td>
    <td>54.6</td>
    <td>54.7</td>
    <td>54.6</td>
    <td>52.5</td>
    <td>52.2</td>
    <td></td>
  </tr>
</table>

Table 19: In-context learning results on PAWS-X across all languages. “High” and “Medium” denotes the available amount of linguistic resources. † denotes the unseen language in the pre-training corpus of BLOOM.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#shot</th>
    <th colspan="6">High</th>
    <th colspan="3">Medium</th>
    <th colspan="3">Low</th>
    <th colspan="3">Ex-Low</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>zh</th>
    <th>id</th>
    <th>it<sup>†</sup></th>
    <th>th<sup>†</sup></th>
    <th>tr<sup>†</sup></th>
    <th>vi</th>
    <th>et<sup>†</sup></th>
    <th>sw</th>
    <th>ta</th>
    <th>ht<sup>†</sup></th>
    <th>qu<sup>†</sup></th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>0</td>
    <td>57.6</td>
    <td>60.0</td>
    <td>52.4</td>
    <td>53.0</td>
    <td>52.8</td>
    <td>61.0</td>
    <td>48.0</td>
    <td>52.4</td>
    <td>56.4</td>
    <td>50.8</td>
    <td>49.0</td>
    <td>53.9</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>57.4</td>
    <td>60.6</td>
    <td>50.2</td>
    <td>53.0</td>
    <td>50.6</td>
    <td>59.2</td>
    <td>49.4</td>
    <td>50.6</td>
    <td>56.4</td>
    <td>51.6</td>
    <td>48.6</td>
    <td>53.4</td>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub> + Pre-train</td>
    <td>0</td>
    <td>53.6</td>
    <td>57.2</td>
    <td>54.4</td>
    <td>53.8</td>
    <td>51.6</td>
    <td>58.8</td>
    <td>52.0</td>
    <td>51.4</td>
    <td>57.0</td>
    <td>49.0</td>
    <td>50.4</td>
    <td>53.6</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>54.4</td>
    <td>56.8</td>
    <td>54.0</td>
    <td>53.8</td>
    <td>52.8</td>
    <td>58.0</td>
    <td>52.2</td>
    <td>50.8</td>
    <td>56.6</td>
    <td>52.2</td>
    <td>49.8</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>55.8</td>
    <td>57.4</td>
    <td>54.4</td>
    <td>56.2</td>
    <td>53.6</td>
    <td>58.8</td>
    <td>50.2</td>
    <td>53.0</td>
    <td>54.6</td>
    <td>49.8</td>
    <td>51.6</td>
    <td>54.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>55.0</td>
    <td>57.2</td>
    <td>53.2</td>
    <td>55.4</td>
    <td>52.8</td>
    <td>61.2</td>
    <td>50.8</td>
    <td>52.0</td>
    <td>55.6</td>
    <td>49.6</td>
    <td>50.4</td>
    <td>53.9</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>55.8</td>
    <td>59.0</td>
    <td>54.2</td>
    <td>55.6</td>
    <td>53.2</td>
    <td>58.6</td>
    <td>53.2</td>
    <td>51.2</td>
    <td>55.6</td>
    <td>52.4</td>
    <td>49.2</td>
    <td>54.4</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>56.6</td>
    <td>57.4</td>
    <td>53.6</td>
    <td>57.0</td>
    <td>52.8</td>
    <td>59.6</td>
    <td>52.6</td>
    <td>51.0</td>
    <td>56.0</td>
    <td>52.4</td>
    <td>52.4</td>
    <td>54.7</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub></td>
    <td>0</td>
    <td>61.4</td>
    <td>63.2</td>
    <td>52.4</td>
    <td>53.2</td>
    <td>53.0</td>
    <td>66.2</td>
    <td>47.4</td>
    <td>51.8</td>
    <td>56.4</td>
    <td>50.4</td>
    <td>50.8</td>
    <td>55.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>63.8</td>
    <td>62.0</td>
    <td>51.2</td>
    <td>53.0</td>
    <td>52.0</td>
    <td>66.2</td>
    <td>49.2</td>
    <td>52.0</td>
    <td>57.0</td>
    <td>51.0</td>
    <td>50.2</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>BLOOM<sub>1.7B</sub> + Pre-train</td>
    <td>0</td>
    <td>58.6</td>
    <td>61.4</td>
    <td>52.6</td>
    <td>55.0</td>
    <td>52.4</td>
    <td>61.8</td>
    <td>49.6</td>
    <td>54.2</td>
    <td>56.0</td>
    <td>53.4</td>
    <td>50.0</td>
    <td>55.0</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>60.4</td>
    <td>61.6</td>
    <td>53.0</td>
    <td>55.2</td>
    <td>51.4</td>
    <td>63.4</td>
    <td>50.0</td>
    <td>54.8</td>
    <td>56.2</td>
    <td>51.8</td>
    <td>50.0</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td>
    <td>58.6</td>
    <td>61.2</td>
    <td>55.2</td>
    <td>55.2</td>
    <td>54.2</td>
    <td>62.6</td>
    <td>51.0</td>
    <td>52.6</td>
    <td>55.2</td>
    <td>54.0</td>
    <td>51.4</td>
    <td>55.6</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>59.6</td>
    <td>61.4</td>
    <td>56.0</td>
    <td>53.8</td>
    <td>52.2</td>
    <td>63.8</td>
    <td>50.8</td>
    <td>54.0</td>
    <td>56.6</td>
    <td>50.4</td>
    <td>51.8</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td>
    <td>59.6</td>
    <td>62.8</td>
    <td>54.2</td>
    <td>56.2</td>
    <td>54.8</td>
    <td>63.6</td>
    <td>51.2</td>
    <td>54.2</td>
    <td>56.0</td>
    <td>52.6</td>
    <td>51.2</td>
    <td>56.0</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td>
    <td>60.8</td>
    <td>60.2</td>
    <td>53.6</td>
    <td>56.2</td>
    <td>53.8</td>
    <td>63.0</td>
    <td>51.2</td>
    <td>53.4</td>
    <td>55.2</td>
    <td>54.8</td>
    <td>50.6</td>
    <td>55.7</td>
  </tr>
</table>

Table 20: In-context learning results on XCOPA across all languages. “High”, “Medium”, “Low” and “Ex-Low” denotes the available amount of linguistic resources. † denotes the unseen language in the pre-training corpus of BLOOM.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#shot</th>
    <th colspan="4">High</th>
    <th colspan="3">Medium</th>
    <th colspan="3">Low</th>
    <th colspan="3">Ex-Low</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>en</th><th>es</th><th>ru<sup>†</sup></th><th>zh</th>
    <th>ar</th><th>id</th><th>hi</th><th>sw</th><th>te</th>
    <th>cu</th><th>my<sup>†</sup></th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>0</td><td>59.9</td><td>55.9</td><td>48.4</td><td>55.1</td><td>52.5</td><td>55.3</td><td>55.1</td><td>49.9</td><td>55.1</td><td>53.5</td><td>47.3</td><td>53.5</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>59.0</td><td>54.3</td><td>48.6</td><td>54.3</td><td>49.9</td><td>54.9</td><td>53.3</td><td>49.6</td><td>56.5</td><td>51.8</td><td>46.9</td><td>52.6</td>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub> + Pre-train</td>
    <td>0</td><td>59.1</td><td>55.1</td><td>51.6</td><td>54.0</td><td>50.4</td><td>55.6</td><td>54.7</td><td>52.9</td><td>55.3</td><td>54.2</td><td>49.4</td><td>53.8</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>57.8</td><td>54.3</td><td>49.3</td><td>53.9</td><td>48.5</td><td>53.7</td><td>53.5</td><td>52.5</td><td>54.8</td><td>52.8</td><td>48.4</td><td>52.7</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td><td>59.2</td><td>56.1</td><td>51.8</td><td>53.6</td><td>50.6</td><td>55.2</td><td>53.8</td><td>52.5</td><td>55.7</td><td>54.9</td><td>48.2</td><td>53.8</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>58.5</td><td>54.6</td><td>51.4</td><td>53.5</td><td>49.4</td><td>54.7</td><td>53.6</td><td>51.6</td><td>55.7</td><td>52.9</td><td>47.9</td><td>53.1</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td><td>59.0</td><td>56.3</td><td>51.4</td><td>54.7</td><td>51.0</td><td>55.7</td><td>54.3</td><td>52.7</td><td>55.8</td><td>54.7</td><td>49.0</td><td>54.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>58.6</td><td>54.9</td><td>50.5</td><td>54.3</td><td>50.6</td><td>54.5</td><td>53.8</td><td>52.2</td><td>55.7</td><td>53.9</td><td>48.4</td><td>53.4</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td>
    <td>0</td><td>64.4</td><td>61.0</td><td>50.3</td><td>58.1</td><td>54.8</td><td>59.9</td><td>56.9</td><td>52.1</td><td>56.6</td><td>54.9</td><td>47.0</td><td>56.0</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>65.1</td><td>61.7</td><td>50.0</td><td>58.2</td><td>53.7</td><td>59.0</td><td>56.5</td><td>51.8</td><td>55.4</td><td>53.3</td><td>45.9</td><td>55.5</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub> + Pre-train</td>
    <td>0</td><td>63.7</td><td>60.6</td><td>52.4</td><td>57.0</td><td>54.5</td><td>58.0</td><td>55.7</td><td>54.5</td><td>57.1</td><td>56.7</td><td>49.9</td><td>56.4</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>63.1</td><td>60.0</td><td>51.7</td><td>56.7</td><td>54.7</td><td>58.3</td><td>55.3</td><td>55.0</td><td>57.0</td><td>54.9</td><td>49.4</td><td>56.0</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td><td>62.6</td><td>60.2</td><td>54.0</td><td>57.2</td><td>55.1</td><td>58.2</td><td>56.3</td><td>55.0</td><td>57.2</td><td>55.3</td><td>49.2</td><td>56.4</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>64.5</td><td>60.1</td><td>52.8</td><td>56.5</td><td>53.9</td><td>58.6</td><td>55.9</td><td>54.9</td><td>57.6</td><td>55.1</td><td>48.3</td><td>56.2</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td><td>63.4</td><td>60.4</td><td>54.3</td><td>57.0</td><td>53.6</td><td>58.8</td><td>56.8</td><td>55.1</td><td>57.8</td><td>55.5</td><td>49.4</td><td>56.6</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>62.3</td><td>59.6</td><td>53.3</td><td>57.1</td><td>55.3</td><td>57.6</td><td>56.1</td><td>54.9</td><td>57.4</td><td>55.1</td><td>48.4</td><td>56.1</td>
  </tr>
</table>

Table 21: In-context learning results on XStoryCloze across all languages. “High”, “Medium”, “Low” and “Ex-Low” denotes the available amount of linguistic resources. † denotes the unseen language in the pre-training corpus of BLOOM.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#shot</th>
    <th colspan="4">High</th>
    <th colspan="2">Medium</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>en</th><th>fr</th><th>ru<sup>†</sup></th><th>zh</th><th>ja<sup>†</sup></th><th>pt</th>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub></td>
    <td>0</td><td>54.0</td><td>51.8</td><td>50.8</td><td>51.7</td><td>62.3</td><td>51.3</td><td>53.7</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>53.6</td><td>48.2</td><td>50.1</td><td>52.4</td><td>61.7</td><td>53.6</td><td>53.3</td>
  </tr>
  <tr>
    <td>BLOOM<sub>560M</sub> + Pre-train</td>
    <td>0</td><td>52.9</td><td>55.4</td><td>50.2</td><td>52.1</td><td>63.1</td><td>55.9</td><td>54.9</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>53.7</td><td>48.2</td><td>51.1</td><td>53.7</td><td>64.5</td><td>51.7</td><td>53.8</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td><td>52.9</td><td>54.2</td><td>49.8</td><td>54.9</td><td>61.5</td><td>52.9</td><td>54.4</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>54.5</td><td>51.8</td><td>49.4</td><td>55.9</td><td>62.5</td><td>51.0</td><td>54.2</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td><td>53.2</td><td>51.8</td><td>50.7</td><td>57.8</td><td>63.3</td><td>53.6</td><td>55.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>53.7</td><td>55.4</td><td>50.2</td><td>55.6</td><td>64.3</td><td>51.3</td><td>55.1</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub></td>
    <td>0</td><td>55.7</td><td>50.6</td><td>50.8</td><td>54.3</td><td>65.9</td><td>53.2</td><td>55.1</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>56.1</td><td>51.8</td><td>51.8</td><td>54.3</td><td>66.7</td><td>52.1</td><td>55.5</td>
  </tr>
  <tr>
    <td>BLOOM<sub>17B</sub> + Pre-train</td>
    <td>0</td><td>55.1</td><td>51.8</td><td>51.8</td><td>54.6</td><td>65.3</td><td>54.4</td><td>55.5</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>55.4</td><td>53.0</td><td>51.2</td><td>54.6</td><td>65.5</td><td>55.1</td><td>55.8</td>
  </tr>
  <tr>
    <td>Branch-Train-Mix</td>
    <td>0</td><td>55.1</td><td>51.8</td><td>50.6</td><td>55.2</td><td>65.3</td><td>55.5</td><td>55.6</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>55.7</td><td>53.0</td><td>50.8</td><td>55.6</td><td>64.9</td><td>54.8</td><td>55.8</td>
  </tr>
  <tr>
    <td>DMoE</td>
    <td>0</td><td>54.6</td><td>50.6</td><td>52.6</td><td>57.1</td><td>66.7</td><td>57.0</td><td>56.4</td>
  </tr>
  <tr>
    <td></td>
    <td>4</td><td>56.0</td><td>53.0</td><td>51.2</td><td>56.5</td><td>67.1</td><td>54.8</td><td>56.4</td>
  </tr>
</table>

Table 22: In-context learning results on XWinograd across all languages. “High” and “Medium” denotes the available amount of linguistic resources. † denotes the unseen language in the pre-training corpus of BLOOM.
D  Licenses of Scientific Artifacts

We follow and report the licenses of scientific artifacts involved in Table 23.

<table>
  <tr>
    <th>Name</th>
    <th>License</th>
  </tr>
  <tr>
    <td>Transformers</td>
    <td>Apache 2.0 license</td>
  </tr>
  <tr>
    <td>X-ELM</td>
    <td>Apache 2.0 license</td>
  </tr>
  <tr>
    <td>lm-evaluation-harness</td>
    <td>MIT license</td>
  </tr>
  <tr>
    <td>matplotlib</td>
    <td>PSF license</td>
  </tr>
  <tr>
    <td>Bloom</td>
    <td>BigScience RAIL 1.0 license</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>Gemma license</td>
  </tr>
  <tr>
    <td>CulturaX</td>
    <td>ODC-BY and CC0 license</td>
  </tr>
  <tr>
    <td>MADLAD-400</td>
    <td>CC-BY-4.0 license</td>
  </tr>
</table>

Table 23: Licenses of scientific artifacts involved in this work.

E  Additional Information about Language Codes

Table 24 reports more information about the language codes involved in this work.
<table>
  <tr>
    <th>ISO 639-1/2</th>
    <th>Language</th>
    <th>Family</th>
    <th>ISO 639-1/2</th>
    <th>Language</th>
    <th>Family</th>
  </tr>
  <tr><td>am</td><td>Amharic</td><td>Afro-Asiatic, Semitic</td><td>lo</td><td>Lao</td><td>Kra-Dai, Tai</td></tr>
  <tr><td>ar*</td><td>Arabic</td><td>Afro-Asiatic, Semitic</td><td>lus</td><td>Mizo</td><td>Sino-Tibetan, Tibeto-Burman</td></tr>
  <tr><td>av</td><td>Avaric</td><td>Northeast Caucasian, Avar-Andic</td><td>lv</td><td>Latvian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>az</td><td>Azerbaijani</td><td>Turkic, Common Turkic</td><td>mg</td><td>Malagasy</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>be</td><td>Belarusian</td><td>Indo-European, Balto-Slavic</td><td>mi</td><td>Maori</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>bn*</td><td>Bangla</td><td>Indo-European, Indo-Iranian</td><td>mk</td><td>Macedonian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>bo</td><td>Tibetan</td><td>Sino-Tibetan, Tibeto-Burman</td><td>ml</td><td>Malayalam</td><td>Dravidian, Southern</td></tr>
  <tr><td>br</td><td>Breton</td><td>Indo-European, Celtic</td><td>mn</td><td>Mongolian</td><td>Mongolic, Central Mongolic</td></tr>
  <tr><td>ca</td><td>Catalan</td><td>Indo-European, Italic</td><td>mr</td><td>Marathi</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>ce</td><td>Chechen</td><td>Northeast Caucasian, Nakh</td><td>ms</td><td>Malay</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>ceb</td><td>Cebuano</td><td>Austronesian, Malayo-Polynesian</td><td>mt</td><td>Maltese</td><td>Afro-Asiatic, Semitic</td></tr>
  <tr><td>ckb</td><td>Central Kurdish</td><td>Indo-European, Indo-Iranian</td><td>my</td><td>Burmese</td><td>Sino-Tibetan, Tibeto-Burman</td></tr>
  <tr><td>cnh</td><td>Chin Haka</td><td>Sino-Tibetan, Tibeto-Burman</td><td>ne</td><td>Nepali</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>co</td><td>Corsican</td><td>Indo-European, Italian</td><td>nl*</td><td>Dutch</td><td>Indo-European, Germanic</td></tr>
  <tr><td>cs</td><td>Czech</td><td>Indo-European, Balto-Slavic</td><td>no</td><td>Norwegian</td><td>Indo-European, Germanic</td></tr>
  <tr><td>da</td><td>Danish</td><td>Indo-European, Germanic</td><td>ny</td><td>Chewa</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>de*</td><td>German</td><td>Indo-European, Germanic</td><td>oc</td><td>Occitan</td><td>Indo-European, Italic</td></tr>
  <tr><td>dv</td><td>Divehi</td><td>Indo-European, Indo-Iranian</td><td>om</td><td>Oromo</td><td>Afro-Asiatic, Cushitic</td></tr>
  <tr><td>ee</td><td>Ewe</td><td>Niger-Congo, Atlantic-Congo</td><td>os</td><td>Ossetian</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>el</td><td>Greek</td><td>Indo-European, Graeco-Phrygian</td><td>pa</td><td>Punjabi</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>en</td><td>English</td><td>Indo-European, Germanic</td><td>pap</td><td>Papiamento</td><td>Portuguese Creole, Afro-Portuguese</td></tr>
  <tr><td>eo</td><td>Esperanto</td><td>Indo-European, Italic</td><td>pl</td><td>Polish</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>es</td><td>Spanish</td><td>Indo-European, Italic</td><td>ps</td><td>Pashto</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>et</td><td>Estonian</td><td>Uralic, Finno-Ugric</td><td>pt</td><td>Portuguese</td><td>Indo-European, Italic</td></tr>
  <tr><td>eu</td><td>Basque</td><td>Language isolate</td><td>rm</td><td>Romansh</td><td>Indo-European, Italic</td></tr>
  <tr><td>fa</td><td>Persian</td><td>Indo-European, Indo-Iranian</td><td>ro</td><td>Romanian</td><td>Indo-European, Italic</td></tr>
  <tr><td>fi</td><td>Finnish</td><td>Uralic, Finno-Ugric</td><td>ru*</td><td>Russian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>fil</td><td>Filipino</td><td>Austronesian, Malayo-Polynesian</td><td>sa</td><td>Sanskrit</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>fo</td><td>Faroese</td><td>Indo-European, Germanic</td><td>sah</td><td>Yakut</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>fr*</td><td>French</td><td>Indo-European, Italic</td><td>sd</td><td>Sindhi</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>fy</td><td>Western Frisian</td><td>Indo-European, Germanic</td><td>se</td><td>Northern Sami</td><td>Uralic, Sami</td></tr>
  <tr><td>gd</td><td>Scottish Gaelic</td><td>Indo-European, Celtic</td><td>sl</td><td>Slovenian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>gl</td><td>Galician</td><td>Indo-European, Italic</td><td>sm</td><td>Samoan</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>grc</td><td>Ancient Greek</td><td>Indo-European, Hellenic</td><td>sn</td><td>Shona</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>gsw</td><td>Swiss German</td><td>Indo-European, Germanic</td><td>so</td><td>Somali</td><td>Afro-Asiatic, Cushitic</td></tr>
  <tr><td>gu</td><td>Gujarati</td><td>Indo-European, Indo-Iranian</td><td>sr</td><td>Serbian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>ha</td><td>Hausa</td><td>Afro-Asiatic, Chadic</td><td>st</td><td>Sotho</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>haw</td><td>Hawaiian</td><td>Austronesian, Malayo-Polynesian</td><td>su</td><td>Sundanese</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>he</td><td>Hebrew</td><td>Afro-Asiatic, Semitic</td><td>sw</td><td>Swahili</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>hi*</td><td>Hindi</td><td>Indo-European, Indo-Iranian</td><td>ta*</td><td>Tamil</td><td>Dravidian, Southern</td></tr>
  <tr><td>hil</td><td>Hiligaynon</td><td>Austronesian, Malayo-Polynesian</td><td>te*</td><td>Telugu</td><td>Dravidian, Southern</td></tr>
  <tr><td>hhn</td><td>Hmong</td><td>Hmong-Mien, Hmongic</td><td>tet</td><td>Tetum</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>ht</td><td>Haitian Creole</td><td>French Creole, Circum-Caribbean French</td><td>tg</td><td>Tajik</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>hu</td><td>Hungarian</td><td>Uralic, Finno-Ugric</td><td>th*</td><td>Thai</td><td>Kra-Dai, Tai</td></tr>
  <tr><td>id*</td><td>Indonesian</td><td>Austronesian, Malayo-Polynesian</td><td>ti</td><td>Tigrinya</td><td>Afro-Asiatic, Semitic</td></tr>
  <tr><td>ig</td><td>Igbo</td><td>Niger-Congo, Atlantic-Congo</td><td>tk</td><td>Turkmen</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>ilo</td><td>Ilocano</td><td>Austronesian, Malayo-Polynesian</td><td>to</td><td>Tongan</td><td>Austronesian, Malayo-Polynesian</td></tr>
  <tr><td>is</td><td>Icelandic</td><td>Indo-European, Germanic</td><td>tr</td><td>Turkish</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>it*</td><td>Italian</td><td>Indo-European, Italic</td><td>ts</td><td>Tsonga</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>ja*</td><td>Japanese</td><td>Japonic</td><td>tt</td><td>Tatar</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>jv</td><td>Javanese</td><td>Austronesian, Malayo-Polynesian</td><td>tyv</td><td>Tuvan</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>ka</td><td>Georgian</td><td>Kartvelian, Karto-Zan</td><td>udm</td><td>Udmurt</td><td>Uralic, Permic</td></tr>
  <tr><td>kaa</td><td>Karakalpak</td><td>Turkic, Common Turkic</td><td>ug</td><td>Uyghur</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>kbd</td><td>Kabardian</td><td>Northwest Caucasian, Circassian</td><td>uk*</td><td>Ukrainian</td><td>Indo-European, Balto-Slavic</td></tr>
  <tr><td>kha</td><td>Khasi</td><td>Austronesian, Khasi-Palaungic</td><td>ur*</td><td>Urdu</td><td>Indo-European, Indo-Iranian</td></tr>
  <tr><td>kk</td><td>Kazakh</td><td>Turkic, Common Turkic</td><td>uz</td><td>Uzbek</td><td>Turkic, Common Turkic</td></tr>
  <tr><td>kl</td><td>Greenlandic</td><td>Eskaleut, Eskimo</td><td>vec</td><td>Venetian</td><td>Indo-European, Italic</td></tr>
  <tr><td>km</td><td>Khmer</td><td>Austronesian, Khmer</td><td>vi*</td><td>Vietnamese</td><td>Austronesian, Vietic</td></tr>
  <tr><td>kn</td><td>Kannada</td><td>Dravidian, Proto-Dravidian</td><td>xh</td><td>Xhosa</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>ko*</td><td>Korean</td><td>Koreanic, Korean</td><td>yi</td><td>Yiddish</td><td>Indo-European, Germanic</td></tr>
  <tr><td>ky</td><td>Kyrgyz</td><td>Turkic, Common Turkic</td><td>yo</td><td>Yoruba</td><td>Niger-Congo, Atlantic-Congo</td></tr>
  <tr><td>la</td><td>Latin</td><td>Indo-European, Italic</td><td>yue</td><td>Yue Chinese</td><td>Sino-Tibetan, Sinitic</td></tr>
  <tr><td>lb</td><td>Luxembourgish</td><td>Indo-European, Germanic</td><td>zh*</td><td>Chinese</td><td>Sino-Tibetan, Sinitic</td></tr>
  <tr><td>lg</td><td>Ganda</td><td>Niger-Congo, Atlantic-Congo</td><td>zu</td><td>Zulu</td><td>Niger-Congo, Atlantic-Congo</td></tr>
</table>

Table 24: Details of language codes in this work. * denotes the language used in the 18 languages experiment.
(a) ar        (b) bn        (c) de
(d) fr        (e) hi        (f) id
(g) it        (h) ja        (i) ko
(j) nl        (k) ru        (l) ta
(m) te        (n) th        (o) uk
(p) ur        (q) vi        (r) zh

Figure 10: The distribution of parameter deviation \( \| \Delta \theta^x \| \) across layers of BLOOM\(_{560M}\) for 18 languages.
Figure 11: The cosine similarity between 18 languages using all parameter deviation of BLOOM1.7B.

Figure 12: The cosine similarity between 18 languages using the parameter deviation of BLOOM1.7B at the last 3 layers.

Figure 13: The cosine similarity between 18 languages using the parameter deviation of Gemma2B at the last 3 layers.

Figure 14: The cosine similarity between 18 languages using the parameter deviation of BLOOM1.7B on the MADLAD-400 multilingual corpus.
Figure 15: The router distribution of top-1 expert for texts in different languages on models trained with randomly initialized router.
Figure 16: The router distribution of top-1 expert for texts in different languages on models trained with router language classification loss.

Reviewer paper summaries: The authors introduce a method to help LMs add new language abilities while minimising cross-lingual interference. They propose a method that involves fine-tuning a base model on monolingual corpora for a set number of tokens to register the lang-level model diffs. These diffs are used to bottom-up cluster the languages to help decide which groups of languages do not interfere with each other. 

The layers that deviate the most are then chosen for MoE-fication. For new languages, the most similar group's expert is cloned, and the new expert and the router module are trained with everything else being fixed.
Summary of strengths: 1. Doing away with the need for lang-ID is a major plus
2. The choice of datasets for evaluation has good coverage
3. The results on language expansion, do speak very well for the method
Summary of weaknesses: 1. I'm a bit perplexed by the choice of base model. The two choices are either very outdated or not the most competitive models. The analysis could've been more interesting if done with a base model that is at least capably multilingual but also amenable to extension such as Qwen2.5.
2. The authors benchmark against model merging methods but do not benchmark strong recent model releases utilising these methods [1]
3. Some important new multilingual extension papers based on strong base models are not cited or benchmarked against [2, 3]

References:

1. John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan N. Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, Sara Hooker:
Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier. CoRR abs/2412.04261 (2024)
2. Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow:
EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models. CoRR abs/2409.17892 (2024)
3. Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, Xiaohu Zhao, Hao Wang, Heng Liu, Hao Zhou, Huifeng Yin, Zifu Shang, Haijun Li, Longyue Wang, Weihua Luo, Kaifu Zhang:
Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement. CoRR abs/2412.04003 (2024)
Comments, suggestions and typos: N/A
Overall assessment scores: 3.5
Metareview score: 3.5
Metareview: This work proposes a method to dynamically group and scale up the parameters of multilingual LLM while boosting positive transfer among similar languages. Multilingual large language models trained with the proposed method achieve better language modeling and in-context learning performance than the continued pre-trained dense model and other scaling methods.

**Strengths**

- The use of parameter deviations (Δθ) for language grouping, rather than relying on existing linguistic typologies, is innovative.
- The experiments are quite comprehensive and the results are solid.
- The experiments cover both 18 and 128 languages, demonstrating the scalability and effectiveness of the method across different language scales.

**Weaknesses**

- Some reviewers have pointed out a lack of mention of relevant literature for either performance comparison or related-work discussion.
- The potential high computational cost and missing pretraining details (the authors have responded to these points, which could possibly be elaborated on in the work).

The reviewers have given the paper overall scores of 4/3.5/3 with confidence levels of 2/4/4 and this work is worthy of publication (subject to the acceptance rate of the *ACL in the field of multilingual LLMs).
Summary of reasons to publish: - This work is solid, especially in the experimental section. Researchers working on multilingual large models may consider it worth reading.
- Overall, the reviewers did not raise major concerns, and the authors provided detailed responses to the reviewers' comments, which can support further improvements to this work.