Asymmetric Conflict and Synergy in Post-training for LLM-based Multilingual Machine Translation
Authors: Tong Zheng, Yan Wen, Huiwen Bao, Junfeng Guo, Heng Huang
Accepted at: acl25

Asymmetric Conflict and Synergy in Post-training for LLM-based Multilingual Machine Translation

Tong Zheng1, Yan Wen1, Huiwen Bao3, Junfeng Guo1,2, Heng Huang1,2

1Department of Computer Science, University of Maryland College Park
2Institute of Health Computing, University of Maryland College Park
3City University of Hong Kong
{tzheng24, gjf2023, heng}@umd.edu, goodbaohuiwen@gmail.com,

Abstract

The emergence of Large Language Models (LLMs) has advanced the multilingual machine translation (MMT), yet the Curse of Multilinguality (CoM) remains a major challenge. Existing work in LLM-based MMT typically mitigates this issue via scaling up training and computation budget, which raises a critical question: Is scaling up the training and computation budget truly necessary for high-quality MMT, or can a deeper understanding of CoM provide a more efficient solution? To explore this problem, we analyze the linguistic conflicts and synergy, the underlying mechanism of CoM during post-training phase. We identify an asymmetric phenomenon in linguistic conflicts and synergy: the dominance of conflicts and synergy varies in different translation directions, leading to sub-optimal adaptation in existing post-training methods. We further find that a significant bottleneck in MMT appears to lie in post-training rather than multilingual pre-training, suggesting the need for more effective adaptation strategies. Building on these new insights, we propose a direction-aware training approach, combined with group-wise model merging, to address asymmetry in linguistic conflicts and synergy explicitly. Leveraging this strategy, our method fine-tunes X-ALMA-13B-Pretrain—trained only with multilingual pre-training—achieving comparable performance to XALMA-13B (only SFT) while using only 20B pretraining tokens and 17B parameters—5.5x fewer pretraining-tokens and 1.7x fewer model size—with just 0.85 COMET drop on Flores-200 testsets of 50 languages.

1 Introduction

Large language models (LLMs) have shown remarkable general capabilities (Brown et al., 2020; Wei et al., 2022; Dubey et al., 2024) and have advanced multilingual machine translation (Xu et al., 2024a; Yang et al., 2023; Alves et al., 2024). For example, Aya-101 (Aryabumi et al., 2024)

![A scatter plot showing the relationship between pre-training cost, model capacity, and translation performance for various models. The x-axis is labeled 'Pre-Training Tokens (B)' and the y-axis is labeled 'COMET-22'. The plot includes points for Ideal, X-ALMA-DAT (Ours), X-ALMA-DATM (Ours), X-ALMA-13B (Only SFT), and LLaMAX. The size of the circles denotes model capacity.](page_1016_370_393_312.png)

Figure 1: The relationship between pre-training cost, model capacity and translation performance. We evaluate performance on the Flores-200 test sets across 50 languages. The size of circle denotes model capacity.

expands support to 101 languages and achieves strong performance in multilingual machine translation, while LLaMAX (Lu et al., 2024b) further pushes performance beyond 100 languages. The common practice behind these successes is the large-scale pretraining, which typically involves monolingual pretraining 1, parallel pretraining, or both—followed by a small-scale, high-quality post-training phase. However, as LLMs scale to more languages, they suffer from the issue of Curse of Multilinguality (CoM) (Conneau, 2019), which degrades the translation performance.

Understanding and mitigating CoM is not new in the MMT literature. In traditional MMT, existing research has identified critical factors such as resource imbalances, limited model capacity, linguistic similarity, and complex interactions between language pairs, particularly for low-resource languages (Arivazhagan et al., 2019; Aharoni et al., 2019; Shaham et al., 2023; Meng and Monz, 2024), and proposed solutions including language-specific modules (Fan et al., 2021; Zhao et al., 2024; Xu

1We also refer to this as multilingual pretraining, where data from all languages are mixed during the pretraining process.
et al., 2023), vocabulary optimization (Han et al., 2024), data sampling techniques (Wang et al., 2020; Wang and Neubig, 2019; Lin et al., 2019), and continual learning approach (Liu et al., 2023). Based on these studies, recent LLM-based MMT research focuses on designing increasingly complex training pipelines and modular architectures. For instance, Xu et al. (2024b) proposed a five-stage training pipeline incorporating language-specific modules. However, existing analyses primarily focus on the encoder-decoder paradigm, while current LLM-based approaches heavily rely on scaling up model capacity and computational resources, making them prohibitively expensive. This raises a critical question: *Is scaling up the training and computation budget truly necessary for high-quality MMT, or can a deeper understanding of CoM in LLM-based MMT provides a more efficient solution?*

In this work, we systematically investigate linguistic conflicts and synergy during post-training phase. We conduct extensive experiments with different settings: across 5 to 50 languages, three pretrained LLMs - ALMA-7B-Pretrain, ALMA-13B-Pretrain and X-ALMA-13B-Pretrain, three distinct post-training strategies - multilingual training, group multilingual training, and separate training. We observe a consistent pattern: asymmetry in linguistic conflicts and synergy (Figure 2, Appendix B.1and B.2 ). For example, in multilingual training, XX→En translation directions experience significant linguistic conflicts, leading to performance degradation, whereas En→XX translations benefit from linguistic synergy, where XX denotes 49 different languages other than English. We further show this asymmetric phenomenon cannot be easily mitigated through existing training approaches, such as group multilingual training (Table 1). **This finding illustrates the need to develop a direction-aware training strategy for optimal post-training.**

Another key finding of our work is that a simple multilingual pre-training stage can be sufficient to equip foundation models with ideal multilingual capabilities, whereas the bottleneck lies in the post-training stage (dotted lines in Figure 2 (g-i)). Motivated by these findings, we propose a novel *Direction-Aware Training* (DAT) approach and build an efficient MMT starting from a relatively efficient base model, the X-ALMA-13B-Pretrain—utilizing only simple multilingual pre-training on 20 billion tokens. Our approach fully leverages the interactive characteristics of different language directions to reduce conflicts while maximizing synergy. We also present a scalable version of the approach, named DATM, which utilizes model merging to further enhance efficiency with only negligible performance degradation.

Through comprehensive evaluations on Flores-200 and WMT23 Benchmark, we demonstrate the effectiveness of our approach. Notably, as shown in Figure 1, compared to X-ALMA (Only SFT) (Xu et al., 2024b), our model X-ALMA-13B-DAT maintains comparable performance while having two advantages: 1) utilizing a simple and efficient training recipe - starting from base models with fewer pre-training tokens and employing a post-training stage. 2) parameter-efficient - we consume 1.7x fewer parameters compared to X-ALMA (Only SFT). These results demonstrate that simple pre-training combined with dedicated post-training can also achieve good multilingual performance.

2 Experimental Settings

In this section, we introduce the basic experimental settings used in Section 3 and Section 4.

2.1 Datasets

We use the high-quality parallel dataset curated by Xu et al. (2024b), covering fifty languages across low-, medium-, and high-resource categories. Following (Xu et al., 2024b), these languages are grouped into eight linguistic groups based on linguistic similarity and a balanced number of languages. Details are provided in Section A in Appendix. The dataset primarily consists of samples from the Flores-200 development set and NTREX (Barrault et al., 2019). For languages in both Flores-200 and WMT’15-22, corresponding test sets are incorporated, yielding an average of 4K examples per language. For evaluation, we use Flores-200 and WMT23 benchmarks to assess performance.

2.2 Models

We select three representative fully open multilingual LLMs for our study: ALMA-Pretrain (Xu et al., 2024a) (7B–13B parameters) and X-ALMA-Pretrain (Xu et al., 2024b) (13B parameters). The ALMA-Pretrain models were pre-trained on 12B or 20B tokens across six languages, while X-ALMA-Pretrain underwent continued pre-training on 20B tokens from 50 languages, both based on LLaMA-2. We exclude other state-of-the-art multilingual
models for two key reasons: (1) their pre-trained checkpoints are unavailable, as in the case of Aya-series (Aryabumi et al., 2024) and BigTrans (Yang et al., 2023); or (2) they exhibit suboptimal multilingual performance in certain languages as shown in Xu et al. (2024b); Cui et al. (2025), such as LLaMA-3 (Dubey et al., 2024).

2.3 Training

Fine-tuning Strategies We employ three distinct training strategies for fine-tuning the models: Multilingual Training, Separate Training, and Group Multilingual Training.

• Multilingual Training (Tang et al., 2020): This is typically achieved by mixing data from all languages and using it to fine-tune the model. The resulting model is a single model that possesses shared representations across all languages.
• Group Multilingual Training (Xu et al., 2024b): We group the languages and then apply multilingual training within each group, resulting in multiple models, each for its respective languages.
• Separate Training: Separate tuning involves training a distinct model for each language without considering linguistic synergies or conflicts.

Training Configurations In this work, all models are trained with a learning rate of 2e-3 using an inverse square root scheduler, a weight decay of 0.01, and a warmup ratio of 0.01. The total batch size is set to 128. Fine-tuning is conducted for 1 epoch, with both max_new_tokens and max_source_length set to 512. Additionally, FP16 precision training is enabled to optimize performance and efficiency. All models are trained on 4 NVIDIA H100 with LoRA (Hu et al., 2022) as Xu et al. (2024a) has shown a negligible performance gap between LoRA tuning (Hu et al., 2022; Zhang et al., 2025) and full fine-tuning.

2.4 Evaluation

We set the number of beams to 5 and both max_new_tokens and max_source_tokens to 512. We evaluate performance mainly using COMET-22 (Rei et al., 2022) and SacreBLEU (Post, 2018).

3 The Phenomenon: Asymmetry in Linguistic Conflicts and Synergy

In this section, we investigate the phenomenon of Asymmetry in Linguistic Conflicts and Synergy in LLM-based MMT. We begin by illustrating the phenomenon (Section 3.1) and analyzing its distribution across two essential factors: language resources and groups (Section 3.2). Finally, we show how this phenomenon poses challenges to existing post-training strategies (Section 3.3).

3.1 Asymmetry in Linguistic Conflicts and Synergy

We investigate linguistic conflicts and synergy during the post-training phase. To explore this, we utilize three foundation models, as mentioned in Section 2.2, to perform multilingual training with training datasets that include a range of languages, from 5 to 50, and evaluate the average performance on corresponding languages.

To quantify linguistic conflicts and synergy, we compare multilingual training with separate training, where each language pair is trained independently, eliminating cross-lingual interactions.

• Linguistic Conflicts: If multilingual training underperforms compared to separate training (i.e., COMET drop), conflicts dominate over synergy.
• Linguistic Synergy: If multilingual training outperforms separate training, synergy dominates.
• Intensity: the magnitude of the performance gap measures the strength of conflicts/synergy.

Figure 2 displays the results. We can have the following observations:

• Key Findings 1: Asymmetry in Linguistic Conflicts and Synergy. As shown in Figures 2 (a), (d), and (g), the average performance decreases with an increase in the number of languages, a phenomenon known as the CoM (Conneau, 2019; Xu et al., 2024b). However, by decomposing the average performance across all language directions into XX→En and En→XX, we uncover an intriguing asymmetry in the distribution of linguistic conflicts and synergies, as illustrated in Figures 2 (b), (c), (e), (f), (h), and (i). Specifically, in the XX→En direction, linguistic conflicts are more dominant, as shown by multilingual training consistently underperforming separate training. Conversely, in the En→XX direction, linguistic synergy is significant, with multilingual tuning consistently outperforming separate training. Furthermore, comparing different models reveals that increasing model capacity (e.g., from 7B to 13B) or incorporating more languages in the pre-training corpus can mitigate conflicts. However, a significant gap remains between separate and multilingual tuning, indicating that simply increasing model capacity and the
Figure 2: Performance of different models trained on varying numbers of languages. The dotted line represents the performance of separately trained models, serving as a reference point where no language conflicts or synergies occur. Two key findings emerge: (1) Asymmetry in Linguistic Conflicts and Synergy (Figure a–i), highlighting the uneven impact of multilingual training across language pairs; and (2) The Bottleneck of Multilinguality in Post-Training (Figure g–i): While multilingual pre-training provides a solid foundation for handling multiple languages, the multilingual training phase can lead to the CoM.

Figure 3: Δ COMET-22 between separate training and multilingual training in XX → En translation, grouped by resource level and linguistic features. The magnitude of Δ COMET-22 denotes the intensity of linguistic conflicts.

number of languages in the pre-training corpus cannot fully resolve the issue. We observe similar findings in terms of SacreBLEU (Appendix B.1) and across different settings (Appendix B.2).
A potential concern regarding this phenomenon is that it may stem from the limited LoRA rank, leading to linguistic conflicts and synergy issues. However, our results (see Appendix B.3) demonstrate that LoRA rank is not the root cause of this phenomenon. Instead, this issue may arise from an inherent limitation in the model’s ability to encode source language representations effectively, potentially due to the absence of an encoder component. We leave this for future work.
• Key Findings 2: Multilingual Pretraining

Stage can sufficiently facilitate X-ALMA-13B-Pretain with ideal multilingual capabilities, whereas the bottleneck may lie in the post-training stage. Observing the dotted line in Figures 2 (g), (h), and (i), we find that separate training on the X-ALMA-13B-Pretain model achieves ideal multilingual performance, maintaining average performance as the number of languages increases. However, multilingual training in the post-training stage cannot fully activate this multilingual ability, resulting in the CoM. For instance, Figure 2 (h) shows a significant performance gap between multilingual training and ideal performance, which widens as the number of languages increases. Interestingly, previous
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Training Type</th>
    <th colspan="2">Group 1</th>
    <th colspan="2">Group 2</th>
    <th colspan="2">Group 3</th>
    <th colspan="2">Group 4</th>
  </tr>
  <tr>
    <th>XX-En</th><th>En-XX</th>
    <th>XX-En</th><th>En-XX</th>
    <th>XX-En</th><th>En-XX</th>
    <th>XX-En</th><th>En-XX</th>
  </tr>
  <tr>
    <td rowspan="3">ALMA-13B Pretrain</td>
    <td>Group Training</td>
    <td>86.0/30.8</td><td><b>87.7/32.0</b></td>
    <td>86.3/31.7</td><td><b>87.7/33.2</b></td>
    <td>85.2/31.5</td><td><b>88.0/26.8</b></td>
    <td>80.3/24.1</td><td><b>81.3/26.0</b></td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td><b>88.5/43.4</b></td><td>86.6/29.9</td>
    <td><b>88.4/41.4</b></td><td>87.1/31.7</td>
    <td><b>86.9/39.3</b></td><td>86.9/24.7</td>
    <td><b>81.1/31.0</b></td><td>74.7/23.7</td>
  </tr>
  <tr>
    <td>Multilingual</td>
    <td>72.3/14.5</td><td>87.2/31.6</td>
    <td>71.2/13.5</td><td>87.4/32.6</td>
    <td>71.7/13.4</td><td>87.5/<b>26.8</b></td>
    <td>66.2/10.0</td><td>80.6/25.7</td>
  </tr>
  <tr>
    <td rowspan="3"></td>
    <td>Group 5</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
  </tr>
  <tr>
    <td>Group Training</td>
    <td>82.7/25.8</td><td><b>80.0/16.4</b></td>
    <td>82.7/21.9</td><td><b>81.1/18.5</b></td>
    <td>77.4/<b>17.7</b></td><td>68.7/9.7</td>
    <td>73.6/12.6</td><td>69.9/6.9</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td><b>83.3/29.4</b></td><td>75.6/14.2</td>
    <td><b>83.5/24.8</b></td><td>75.8/16.8</td>
    <td><b>77.6/17.6</b></td><td>57.4/5.1</td>
    <td><b>76.5/18.5</b></td><td>55.5/4.0</td>
  </tr>
  <tr>
    <td>Multilingual</td>
    <td>70.7/13.1</td><td>79.4/<b>16.7</b></td>
    <td>72.3/11.1</td><td>80.7/<b>18.7</b></td>
    <td>62.3/4.0</td><td><b>70.6/11.0</b></td>
    <td>62.8/5.9</td><td><b>70.9/8.1</b></td>
  </tr>
  <tr>
    <td rowspan="3">X-ALMA-13B Pretrain</td>
    <td>Group 1</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
  </tr>
  <tr>
    <td>Group Training</td>
    <td>86.3/31.1</td><td><b>88.8/34.7</b></td>
    <td>86.6/31.7</td><td><b>88.6/33.2</b></td>
    <td>85.9/33.2</td><td>89.8/31.3</td>
    <td>83.5/26.6</td><td><b>86.3/29.3</b></td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td><b>88.9/44.4</b></td><td>88.5/34.1</td>
    <td><b>88.6/41.6</b></td><td>88.3/34.9</td>
    <td><b>87.5/41.0</b></td><td>89.7/30.5</td>
    <td><b>85.9/35.8</b></td><td>85.3/27.8</td>
  </tr>
  <tr>
    <td>Multilingual</td>
    <td>79.0/17.4</td><td>88.6/34.3</td>
    <td>78.0/16.3</td><td>88.5/<b>35.4</b></td>
    <td>77.3/15.6</td><td><b>89.9/31.4</b></td>
    <td>75.8/13.8</td><td>86.2/28.8</td>
  </tr>
  <tr>
    <td></td>
    <td>Group 5</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
    <td>XX-En</td><td>En-XX</td>
  </tr>
  <tr>
    <td>Group Training</td>
    <td>85.6/28.2</td><td><b>89.9/24.0</b></td>
    <td>86.2/24.6</td><td><b>89.6/25.8</b></td>
    <td>86.4/27.0</td><td><b>81.0/18.8</b></td>
    <td>83.0/20.6</td><td><b>87.5/17.6</b></td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td><b>87.4/35.3</b></td><td>89.4/23.5</td>
    <td><b>87.7/30.1</b></td><td>88.9/22.6</td>
    <td><b>87.8/33.1</b></td><td>80.2/17.2</td>
    <td><b>86.5/31.1</b></td><td>86.6/15.6</td>
  </tr>
  <tr>
    <td>Multilingual</td>
    <td>79.7/17.8</td><td>89.7/23.6</td>
    <td>80.0/14.7</td><td>89.0/22.8</td>
    <td>77.8/10.6</td><td>80.9/18.3</td>
    <td>76.5/11.4</td><td>87.2/17.1</td>
  </tr>
</table>

Table 1: Performance of ALMA-13B-Pretrain and X-ALMA-13B-Pretrain on 50 languages from the Flores-200 test sets under three training approaches: Group multilingual training, Separate training, and Multilingual training. Results are categorized by language groups. Detailed scores for each group are provided in the Appendix.

work (Xu et al., 2024b) designed complex training regimens with up to five stages, including three pre-training and two post-training stages with language-specific group training, to address this issue. In contrast, our findings suggest there may be a more efficient way to tackle the CoM. For example, we could start with a base model that only undergoes multilingual pretraining and then apply a dedicated post-training approach to achieve high-quality translation.

3.2 Asymmetry in Conflicts and Synergies Across Languages Groups and Resources

We further address a key question: Does Asymmetry in Conflicts and Synergies occur across all language pairs, or is it concentrated in specific pairs? To answer this, we analyze its distribution across different language groups and resource levels.

Figure 3 displays the results. We can have the following observations:
• Asymmetry in linguistic conflicts is consistently observed across languages with varying resource levels and language groups, but its intensity is not uniformly distributed.
• While increasing model capacity or pre-training data can help narrow the performance gap, consistent with findings in previous work (Arivazhagan et al., 2019; Aharoni et al., 2019; Shaham et al., 2023; Meng and Monz, 2024), a substantial gap of nearly 10 COMET-22 points still persists.

3.3 Challenges by Asymmetry in Linguistic Conflicts and Synergy

The asymmetry in linguistic conflicts and synergies may pose challenges for LLM-based MMT, leading to suboptimal performance for existing post-training approaches. Intuitively, translation directions where linguistic conflicts dominate may benefit from post-training strategies that minimize such conflicts. Conversely, translation directions where linguistic synergies prevail may require strategies that effectively enhance high-quality synergy. To see this, we fine-tune foundation models using three key approaches: multilingual training, group multilingual training, and separate training on 50 languages and compare their performance.

Table 1 displays the experimental results on the Flores-200 test set. We observe the following:
• Key Findings 3: The effectiveness of the existing training strategy exhibits an asymmetrical pattern.: In XX→En translations, separate training consistently achieves the best performance, followed by group multilingual training, while full multilingual training performs the worst. This result is expected, as linguistic conflict is prominent in these translation directions.
(a) Seperate/Multilingual/Group Training        (b) Direction-aware Training with Group-wise Model Merging

Figure 4: (a) Separate Training (\( N = N_L \)): Each translation task is trained independently using different datasets for different language pairs, with distinct LoRA model weights fine-tuned separately; Multilingual Training (\( N = 1 \)): All language pairs are combined to fine-tune a single model with shared LoRA weights; Group Multilingual Training (\( N = N_G \)): Language pairs are grouped as specified in Table 3-4, with an adapter trained for each group. (b) Group-wise model merging: For XX→En translation, separate training is applied to each language pair. For En→XX translation, group training is applied, where different tasks share LoRA weights within language groups.

By contrast, in En→XX translations, multilingual training or group multilingual training consistently outperforms separate training. This indicates that while linguistic conflicts dominate in the XX→En direction, the En→XX direction benefits from cross-linguistic knowledge transfer, leading to an enhanced translation quality. When model capacity is sufficiently large, the general pattern observed is: group multilingual training > multilingual training > separate training. This highlights two things: 1) linguistic similarity benefits positive cross-linguistic transfer. 2) the widely adopted group multilingual training approach remains insufficient to address the challenges posed by the asymmetry.

These findings underscore the critical impact of asymmetry in linguistic conflicts and synergy phenomenon on the effectiveness of existing training strategies, highlighting the need for novel training approaches to consider such an asymmetry to achieve optimal performance in both directions.

4 Direction-Aware Training and Merging for Efficient LLM-based MMT

In this section, we show how to construct an efficient MMT system by leveraging the insights from Section 3, starting from a base model with simple multilingual pre-training.

4.1 Motivations and Main Ideas

As demonstrated in Section 3, linguistic conflicts and synergy exhibit asymmetry during the post-training stage, posing significant challenges to multilingual translation. A widely adopted technique to mitigate conflicts and enhance synergy is language-specific group multilingual training (Fan et al., 2021; Zhao et al., 2024; Xu et al., 2023, 2024b). However, it still achieves sub-optimal performance.

The state-of-the-art XALMA system (Xu et al., 2024b) achieves high-quality translations by employing eight large language-specific adapters within a MoE framework combined with group multilingual training. However, this approach incurs high computational and storage costs, as each adapter contains up to 15% of the base model’s parameters, making large-scale deployment challenging. Additionally, XALMA requires a massive amount of tokens during pre-training, further increasing resource consumption. This raises an important question: Can we achieve comparable high translation quality in a more efficient manner?

Intuitively, we could develop a more efficient training approach for high-quality MMT by considering the asymmetry in linguistic conflicts and synergy. To this end, we propose a direction-aware training framework combined with model merging, which fully leverages the inherent asymmetry to enhance both performance and efficiency. Our approach primarily consists of two key components: 1) Direction-aware training strategies for efficiently and effectively mitigating linguistic conflicts and encouraging linguistic synergy and 2) Group-wise model merging for running efficiency.

4.2 Direction-Aware Training Strategies

As shown in Figure 4 (b), we propose a simple yet effective direction-aware training strategy that addresses linguistic conflicts and linguistic synergy separately for different translation directions:
• For XX→En translation directions: We employ separate training to build expert models for each
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2"># Tokens (Pre-training)</th>
    <th rowspan="2"># Params</th>
    <th colspan="2">FLORES200</th>
    <th colspan="2">WMT23</th>
  </tr>
  <tr>
    <th>Base/Adapter</th>
    <th>Total</th>
    <th>XX→En</th>
    <th>En→XX</th>
    <th>XX→En</th>
    <th>En→XX</th>
  </tr>
  <tr>
    <th colspan="8">Existing State-of-the-Art MMT System</th>
  </tr>
  <tr>
    <td>NLLB-3.3B</td>
    <td>-</td>
    <td>3B/-</td>
    <td>3B</td>
    <td>80.7</td>
    <td>87.4</td>
    <td>71.2</td>
    <td>81.5</td>
  </tr>
  <tr>
    <td>Aya-23-8B</td>
    <td>-</td>
    <td>8B/-</td>
    <td>8B</td>
    <td>80.9</td>
    <td>74.4</td>
    <td>81.5</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>Aya-23-35B</td>
    <td>-</td>
    <td>35B/-</td>
    <td>35B</td>
    <td>84.9</td>
    <td>76.0</td>
    <td>82.3</td>
    <td>84.1</td>
  </tr>
  <tr>
    <td>Aya-101</td>
    <td>-</td>
    <td>13B/-</td>
    <td>13B</td>
    <td>86.3</td>
    <td>84.1</td>
    <td>79.7</td>
    <td>80.8</td>
  </tr>
  <tr>
    <td>LLaMAX3-Alpaca-8B</td>
    <td>66B</td>
    <td>8B/-</td>
    <td>8B</td>
    <td>85.9</td>
    <td>84.1</td>
    <td>81.0</td>
    <td>79.8</td>
  </tr>
  <tr>
    <td>X-ALMA-13B (Only SFT, MoE)</td>
    <td>110B</td>
    <td>13B/16B</td>
    <td>29B</td>
    <td><b>88.2</b></td>
    <td><b>88.9</b></td>
    <td><b>83.2</b></td>
    <td><b>85.6</b></td>
  </tr>
  <tr>
    <th colspan="8">Our System</th>
  </tr>
  <tr>
    <td>X-ALMA-13B-DAT (MoE)</td>
    <td>20B</td>
    <td>13B/4B</td>
    <td>17B</td>
    <td>87.6</td>
    <td>87.8</td>
    <td>82.8</td>
    <td>84.8</td>
  </tr>
  <tr>
    <td>X-ALMA-13B-DATM (MoE)</td>
    <td>20B</td>
    <td>13B/1B</td>
    <td>14B</td>
    <td>87.4</td>
    <td>87.8</td>
    <td>82.1</td>
    <td>84.8</td>
  </tr>
</table>

Table 2: Performance on Flores-200 and WMT23 benchmarks. The results of baselines are directly sourced from Xu et al. (2024b) as we utilized same generation configuration. Full results are provided in Appendix.

language direction.
• For En→XX translation directions: We adopt group multilingual training, training one model per language group following Xu et al. (2024b). All training employs LoRA (Hu et al., 2022) with a rank of 16 for parameter efficiency. Using the proposed strategies, we construct a LoRA weight pool of size \( N_G + N_L \), where \( N_G \) is the number of groups and \( N_L \) is the number of languages.

4.3 Group-wise Model Merging

Although the direction-aware training approach achieves promising performance, the number of LoRA weights increases linearly with the number of supported languages, posing challenges for deployment and inference, especially at large language scales. Model merging (Yadav et al., 2024; Zhang et al., 2023) provides a feasible solution to reduce the number of LoRA weights and improve efficiency. However, directly using model merge for efficient MMT is non-trial. In our preliminary experiments, we have two key observations:
• Merging LoRA weights into one for each direction leads to performance degradation. Notably, Dang et al. (2024) find that model merging can improve performance, contrasting our findings. However, this discrepancy may arise because their comparison is against a weaker baseline, such as multilingual training, whereas we compare against the most vigorous baseline—separate training.
• The degradation effect of model merging exhibits an asymmetric nature. The performance degradation per parameter in the En→XX direction is <b>6.86x greater</b> than in the XX→En direction. A potential explanation is that linguistic synergy plays a crucial role in En→XX directions, while model merging introduces low-quality linguistic synergy, leading to a performance drop. Therefore, a more dedicated design is needed to preserve performance as much as possible.

Motivated by these observations, we only apply model merging to XX→En directions in a group-wise manner. Specifically, we apply model merging to languages within each group, resulting in \( N_G \) LoRA weights. We adopt the TIES (Yadav et al., 2024) for model merging. We also compare this approach with other methods such as DARE-TIES (Yu et al., 2024) and find no significant performance difference. With this approach, we can reduce the number of LoRA weights from \( O(N_L) \) to \( O(N_G) \), improving scalability while lead minimal performance degradation.

4.4 Main Results

We evaluated our models using the Flores-200 test set for 50 languages and the WMT23 test sets for five languages (de, ru, uk, ja, zh). We provide more details in Appendix A. We select existing state-of-the-art open multilingual MT system as baselines:
• Aya-101 (Üstün et al., 2024): A 13B multilingual LLM supporting 101 languages.
• LLaMAX (Lu et al., 2024b): An 8B LLM-based MMT system supporting 102 languages.
• Aya-23-8B/35B (Aryabumi et al., 2024): An 8B/35B multilingual LLMs that support 23 languages.
• XALMA (Xu et al., 2024b): A 29B multilingual MoE-based MMT system supporting 50 languages, using language-specific adapters and group multilingual training. Notably, since we focus only on the supervised fine-tuning stage,
we select the version without preference learning, namely XALMA-13B (Only SFT) to ensure fair comparison.

Table 2 shows the results. We can have the following observations:
• Both X-ALMA-13B-DAT and X-ALMA-13B-DATM can achieve high translation performance. Compared to previous multilingual LLMs, such as Aya-101, Aya-23-8B, and LLa-MAX, our approach consistently outperforms them across both benchmarks and translation directions. Moreover, compared to X-ALMA, our X-ALMA-13B-DAT achieves comparable performance in XX→En directions; however, in En→XX, a significant performance gap remains, up to 0.95 COMET-22 on average.
• Our approach provides an efficient way to build effective MMT. Our model is built upon X-ALMA-13B-Pretrain with only 20 billion tokens of simple multilingual pre-training. Moreover, it utilizes multiple small LoRA weight compositions and achieves relatively high translation performance across all directions, which is consistent with previous work (Zheng et al., 2024a)

5 Related Work

5.1 Curse of Multilinguality
Existing research has explored both understanding and addressing this issue in MMT, identifying critical factors such as resource imbalances, limited model capacity, and complex interactions between language pairs, particularly for low-resource languages (Arivazhagan et al., 2019; Aharoni et al., 2019; Shaham et al., 2023). Interestingly, studies have shown that while linguistic similarity enhances positive transfer, dissimilar languages can also act as regularizers, improving training stability (Meng and Monz, 2024). To address these challenges, proposed solutions in recent research include language-specific modules (e.g., adapters, sparse experts) to dynamically allocate capacity and reduce interference (Fan et al., 2021; Zhao et al., 2024; Xu et al., 2023), vocabulary optimization to better support new languages through improved token representations (Han et al., 2024), data sampling techniques to enhance representation for underrepresented languages (Wang et al., 2020; Wang and Neubig, 2019; Lin et al., 2019) and continual learning techniques (Liu et al., 2023). Notably, techniques, such as language-specific modules, have been integrated into LLM-based MMT systems, resulting in substantial improvements in multilingual performance (Xu et al., 2024b). In this work, we systematically investigate how post-training in LLM-based MMT contributes to the CoM, providing a fine-grained analysis of its impact on linguistic conflicts and synergies.

5.2 LLMs for Multilingual MT
Prior to the emergence of LLMs, neural machine translation (NMT) research predominantly focused on optimizing encoder–decoder architectures and attention mechanisms (Li et al., 2022; Zheng et al., 2024a,b; Li et al., 2024) or decoding strategy (Rehe-man et al., 2023; Mu et al., 2023). Recently, many efforts have been made to adapt LLMs for effective machine translation. A key approach is prompting, which enhances translation performance without additional training (He et al., 2024; Lu et al., 2024a). Beyond this, growing research focuses on fine-tuning open and smaller LLMs to achieve high translation quality while ensuring efficiency (Xu et al., 2024a; Yang et al., 2023; Alves et al., 2024; Aryabumi et al., 2024; Luo et al., 2025).

Yang et al. (2023) propose a training pipeline that integrates monolingual pre-training to improve language modeling and parallel instruction fine-tuning for enhanced translation performance. Similarly, Xu et al. (2024a) emphasize the quality over quantity of parallel data, introducing a training recipe: (1) large-scale monolingual pre-training, followed by (2) small-scale, high-quality parallel fine-tuning. Further revisiting the role of parallel data, Guo et al. (2024) highlights its importance in the pre-training stage. Additionally, Xu et al. (2024c) underscore the necessity of alignment in post-training, proposing the CPO algorithm. More recently, with the need to scale models across more languages, Xu et al. (2024b) introduces language-specific modules combined with group training to mitigate language conflicts. In this work, we focus on the post-training stage, which has been underexplored in previous studies, and propose a direction-aware training approach with model merging to achieve efficient and effective MMT.

6 Conclusions
In this work, we systematically investigate linguistic conflicts and synergy during post-training in LLM-based MMT and identify a phenomenon we term asymmetry in linguistic conflicts and synergy. We provide an in-depth analysis of its distribution
and challenges for LLM-based MMT. Based on these insights, we propose a direction-aware training approach combined with model merging to build an effective MMT system from X-ALMA-13B-Pretrain with only multilingual pre-training. Our approach highlights the importance of post-training in LLM-based MMT and offers insights into building MMT resource-efficiently.

Acknowledgments

We thank Zhengmian Hu, Lichang Chen, Peiran Yu, Ruibo Chen, Tianyi Xiong’s feedback on this work.

Limitations

One limitation of this work is that our approach does not surpass state-of-the-art methods like X-ALMA in performance, particularly in En→XX directions, despite requiring less training cost and fewer model parameters. Second, while this work identifies a novel phenomenon and designs an efficient approach leveraging it, it does not provide a deeper or more rigorous analysis of why asymmetry in linguistic conflicts and synergy exists. We leave the analysis of the underlying mechanism of asymmetry in linguistic conflicts and synergy for future work.

Additionally, although this work conducts extensive experiments on fifty languages and three pre-trained models, further scaling is necessary to validate our findings on a broader scale, such as extending to over 100 languages. This would help push the boundaries of multilingual machine translation research, which we also leave for future work.

References

Roe Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884, Minneapolis, Minnesota. Association for Computational Linguistics.

Duarte M Alves, José Pombal, Nuno M Guerreiro, Pedro H Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402.17733.

Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019.

Viraat Aryabumi, John Dang, Dwarak Talpuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, et al. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032.

Loïc Barrault, Ondřej Bojar, Marta R Costa-Jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. 2019. Findings of the 2019 conference on machine translation (wmt19). ACL.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.

Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, et al. 2025. Multilingual machine translation with open large language models at practical scale: An empirical study. arXiv preprint arXiv:2502.02481.

John Dang, Shivalka Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48.

Jiaxin Guo, Hao Yang, Zongyao Li, Daimeng Wei, Hengchao Shang, and Xiaoyu Chen. 2024. A novel paradigm boosting translation capabilities of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 639–649, Mexico City, Mexico. Association for Computational Linguistics.

HyoJung Han, Akiko Eriguchi, Haoran Xu, Hieu Hoang, Marine Carpuat, and Huda Khayrallah.
2024. Adapters for altering llm vocabularies: What languages benefit the most? arXiv preprint arXiv:2410.09644.

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2024. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229–246.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Bei Li, Tong Zheng, Yi Jing, Chengbo Jiao, Tong Xiao, and Jingbo Zhu. 2022. Learning multiscale transformer models for sequence generation. In International Conference on Machine Learning, pages 13225–13241. PMLR.

Bei Li, Tong Zheng, Rui Wang, Jiahao Liu, Junliang Guo, Xu Tan, Tong Xiao, JingBo Zhu, Jingang Wang, and Xunliang Cai. 2024. Predictor-corrector enhanced transformers with exponential moving average coefficient learning. Advances in Neural Information Processing Systems, 37:20358–20382.

Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125–3135. Florence, Italy. Association for Computational Linguistics.

Junpeng Liu, Kaiyu Huang, Hao Yu, Jiuyi Li, Jinsong Su, and Degen Huang. 2023. Continual learning for multilingual neural machine translation via dual importance-based model division. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12011–12027, Singapore. Association for Computational Linguistics.

Hongyuan Lu, Haoran Yang, Haoyang Huang, Dongdong Zhang, Wai Lam, and Furu Wei. 2024a. Chain-of-dictionary prompting elicits translation in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 958–976. Miami, Florida, USA. Association for Computational Linguistics.

Yinqun Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024b. Llamax: Scaling linguistic horizons of llm by enhancing translation capabilities beyond 100 languages. arXiv preprint arXiv:2407.05975.

Yingfeng Luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, et al. 2025. Beyond decoder-only: Large language models can be good encoders for machine translation. arXiv preprint arXiv:2503.06594.

Yan Meng and Christof Monz. 2024. Disentangling the roles of target-side transfer and regularization in multilingual machine translation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1828–1840, St. Julian’s, Malta. Association for Computational Linguistics.

Yongyu Mu, Abudurexiti Reheman, Zhiquan Cao, Yuchun Fan, Bei Li, Yinqiao Li, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2023. Augmenting large language model translators via translation memories. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10287–10299, Toronto, Canada. Association for Computational Linguistics.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

Abudurexiti Reheman, Tao Zhou, Yingfeng Luo, Di Yang, Tong Xiao, and Jingbo Zhu. 2023. Prompting neural machine translation with translation memories. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15139–15157.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhsale. 2023. Causes and cures for interference in multilingual translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15849–15863, Toronto, Canada. Association for Computational Linguistics.

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401.

Ahmet Üstün, Virat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’ouzoua, Gbemileke Onilude, Neel Bhandari, Shivalka Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics.
Xinyi Wang and Graham Neubig. 2019. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5823–5828, Florence, Italy. Association for Computational Linguistics.

Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8526–8537, Online. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations.

Haoran Xu, Kenton Murray, Philipp Koehn, Hieu Hoang, Akiko Eriguchi, and Huda Khayrallah. 2024b. X-alma: Plug & play modules and adaptive rejection for quality translation at scale. arXiv preprint arXiv:2410.03115.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024c. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.

Haoran Xu, Weiting Tan, Shuyue Li, Yunmo Chen, Benjamin Van Durme, Philipp Koehn, and Kenton Murray. 2023. Condensing multilingual knowledge with lightweight language-specific modules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1575–1587, Singapore. Association for Computational Linguistics.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098.

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning.

Jinghan Zhang, Junteng Liu, Junxian He, et al. 2023. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589–12610.

Xueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong, Shuhao Guan, Linbo Cao, and Yining Wang. 2025. Uora: Uniform orthogonal reinitialization adaptation in parameter-efficient fine-tuning of large models.

Xinyu Zhao, Xuxi Chen, Yu Cheng, and Tianlong Chen. 2024. Sparse moe with language guided routing for multilingual machine translation. In The Twelfth International Conference on Learning Representations.

Tong Zheng, Bei Li, Huiwen Bao, Jiale Wang, Weiqiao Shan, Tong Xiao, and JingBo Zhu. 2024a. Partial-Former: Modeling part instead of whole for machine translation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7280–7294, Bangkok, Thailand. Association for Computational Linguistics.

Tong Zheng, Bei Li, Huiwen Bao, Tong Xiao, and JingBo Zhu. 2024b. EIT: Enhanced interactive transformer. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7734–7751, Bangkok, Thailand. Association for Computational Linguistics.
A  Detailed Experimental Setups

In this section, we will discuss the detailed setup of our experiment, including the datasets.

A.1  Details of Dataset in Section 2.1

Following (Xu et al., 2024b), we present a classification of languages based on linguistic families, scripts, and resource availability in Tables 3-4. Fifty languages are grouped into eight distinct categories, primarily guided by linguistic similarity while considering a balanced distribution of languages across groups. Each group encompasses a mix of low-, medium-, and high-resource languages to ensure comprehensive multilingual coverage. Additionally, English is included in each group to facilitate English-centric translation and mitigate catastrophic forgetting. This structured grouping provides a well-rounded dataset for multilingual research, enabling robust language modeling and cross-lingual transfer learning.

We train the translation model on X-ALMA-Parallel-Data, a parallel dataset in (Xu et al., 2024b). The distribution of the parallel datasets for each language is illustrated in Figure 5.

The evaluation dataset primarily consists of samples from the Flores-200 development set and NTREX (Barrault et al., 2019). In our experiment, we follow the setting in (Xu et al., 2024b), where the translation sentences are sampled to contain 1012 sentences in each language pair. We also use WMT23 benchmarks to assess performance for evaluation. The distribution of WMT23 for each language is illustrated in Figure 6.

For languages in both Flores-200 and WMT’15-22, corresponding test sets are incorporated, yielding an average of 4K examples per language.

B  Additional Experiments

B.1  Asymmetry in Linguistic Conflicts and Synergy in terms of SacreBLEU

As shown in Figure 7, we observe a clear asymmetry in linguistic conflicts and synergy based on the SacreBLEU metric. This aligns with our main findings in the paper, where we used the COMET metric, further reinforcing the consistency of the observed phenomenon across different evaluation measures.

B.2  More Experiments on Asymmetry in Linguistic Conflicts and Synergy

We further design another setting to validate the asymmetry in linguistic conflicts and synergy.

Experimental Setup  We select anchor sets of varying sizes and perform post-training using training sets that include different numbers of languages but cover those anchor sets. We then observe the performance changes of these anchor sets. If the performance declines as more languages are included in the training set, this would indicate the presence of linguistic conflicts.

Results  Table 5 displays the results. We can clearly observe that in the XX-En directions, the average performance of each anchor set consistently decreases as the number of languages increases. However, this phenomenon is not observed in the En-XX directions, where performance remains relatively stable. The findings are consistent with Section 3.

B.3  Impact of Lora Rank

We observed an asymmetry in linguistic conflicts and synergies. A natural question arises: could this be due to using a low LoRA rank, which might limit learning capacity and, consequently, degrade performance? To address this concern, we selected the ALMA-13B-Pretrain model and trained it on 16 languages using different LoRA ranks, specifically 16 and 32. We then compared the performance of models with these LoRA ranks on the FLores-200 test sets. As shown in Table 6, increasing the LoRA rank did not yield performance improvements. Therefore, we conclude that the observed asymmetry is not attributed to using a low LoRA rank.

C  Full Results
<table>
  <tr>
    <th>Language</th>
    <th>ISO-639-1</th>
    <th>Script</th>
    <th>Family</th>
    <th>Subgroup</th>
    <th>Resource</th>
  </tr>
  <tr>
    <td>English</td>
    <td>en</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>High</td>
  </tr>
  <tr>
    <th colspan="6">Group 1: Germanic Languages</th>
  </tr>
  <tr>
    <td>Afrikaans</td>
    <td>af</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Danish</td>
    <td>da</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Dutch</td>
    <td>nl</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>German</td>
    <td>de</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Icelandic</td>
    <td>is</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Norwegian</td>
    <td>no</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Swedish</td>
    <td>sv</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Germanic</td>
    <td>High</td>
  </tr>
  <tr>
    <th colspan="6">Group 2: Romance Languages</th>
  </tr>
  <tr>
    <td>Catalan</td>
    <td>ca</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Galician</td>
    <td>gl</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Italian</td>
    <td>it</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Portuguese</td>
    <td>pt</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Romanian</td>
    <td>ro</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>es</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>High</td>
  </tr>
  <tr>
    <th colspan="6">Group 3: Eastern and Southern Slavic Languages</th>
  </tr>
  <tr>
    <td>Bulgarian</td>
    <td>bg</td>
    <td>Cyrillic</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Macedonian</td>
    <td>mk</td>
    <td>Cyrillic</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>ru</td>
    <td>Cyrillic</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Serbian</td>
    <td>sr</td>
    <td>Cyrillic</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Ukrainian</td>
    <td>uk</td>
    <td>Cyrillic</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <th colspan="6">Group 4: Southeast Asian Languages</th>
  </tr>
  <tr>
    <td>French</td>
    <td>fr</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Italic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Indonesian</td>
    <td>id</td>
    <td>Latin</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Malagasy</td>
    <td>mg</td>
    <td>Latin</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Malay</td>
    <td>ms</td>
    <td>Latin</td>
    <td>Austronesian</td>
    <td>Malayo-Polynesian</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Thai</td>
    <td>th</td>
    <td>Thai</td>
    <td>Tai-Kadai</td>
    <td>Kam-Tai</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Vietnamese</td>
    <td>vi</td>
    <td>Latin</td>
    <td>Austronesian</td>
    <td>Vietic</td>
    <td>High</td>
  </tr>
</table>

Table 3: Detailed information of all languages

![Bar chart showing number of sentences per language pair in X-ALMA-Parallel-Data](page_1012_1047_682_246.png)

Figure 5: Number of sentences per language pair in X-ALMA-Parallel-Data (Xu et al., 2024b)

![Bar chart showing number of sentences per language pair in WMT'23](page_1012_1347_682_246.png)

Figure 6: Number of Sentences per language pair in WMT'23
<table>
  <tr>
    <th>Language</th>
    <th>ISO-639-1</th>
    <th>Script</th>
    <th>Family</th>
    <th>Subgroup</th>
    <th>Resource</th>
  </tr>
  <tr>
    <th colspan="6">Group 5: Central and Eastern European Languages</th>
  </tr>
  <tr>
    <td>Czech</td>
    <td>cs</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Greek</td>
    <td>el</td>
    <td>Greek</td>
    <td>Indo-European</td>
    <td>Graeco-Phrygian</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>hu</td>
    <td>Latin</td>
    <td>Uralic</td>
    <td>Finnic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Latvian</td>
    <td>lv</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Lithuanian</td>
    <td>lt</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>pl</td>
    <td>Latin</td>
    <td>Indo-European</td>
    <td>Balto-Slavic</td>
    <td>High</td>
  </tr>
  <tr>
    <th colspan="6">Group 6: Eurasian Language Mix</th>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>zh</td>
    <td>Han</td>
    <td>Sino-Tibetan</td>
    <td>Sinitic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Estonian</td>
    <td>et</td>
    <td>Latin</td>
    <td>Uralic</td>
    <td>Finnic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Finnish</td>
    <td>fi</td>
    <td>Latin</td>
    <td>Uralic</td>
    <td>Finnic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Georgian</td>
    <td>ka</td>
    <td>Georgian</td>
    <td>Kartvelian</td>
    <td>Georgian-Zan</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>ja</td>
    <td>Japanese</td>
    <td>Japonic</td>
    <td>Japanesic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>ko</td>
    <td>Hangul</td>
    <td>Koreanic</td>
    <td>Korean</td>
    <td>High</td>
  </tr>
  <tr>
    <th colspan="6">Group 7: Indo-Aryan Languages</th>
  </tr>
  <tr>
    <td>Gujarati</td>
    <td>gu</td>
    <td>Gujarati</td>
    <td>Indo-European</td>
    <td>Indo-Aryan</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>hi</td>
    <td>Devanagari</td>
    <td>Indo-European</td>
    <td>Indo-Aryan</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Marathi</td>
    <td>mr</td>
    <td>Devanagari</td>
    <td>Indo-European</td>
    <td>Indo-Aryan</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Nepali</td>
    <td>ne</td>
    <td>Devanagari</td>
    <td>Indo-European</td>
    <td>Indo-Aryan</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Urdu</td>
    <td>ur</td>
    <td>Arabic</td>
    <td>Indo-European</td>
    <td>Indo-Aryan</td>
    <td>Mid</td>
  </tr>
  <tr>
    <th colspan="6">Group 8: Turkic and Semitic Languages</th>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>ar</td>
    <td>Arabic</td>
    <td>Afro-Asiatic</td>
    <td>Semitic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Azerbaijani</td>
    <td>az</td>
    <td>Arabic/Latin</td>
    <td>Turkic</td>
    <td>Common Turkic</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Hebrew</td>
    <td>he</td>
    <td>Hebrew</td>
    <td>Afro-Asiatic</td>
    <td>Semitic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Kazakh</td>
    <td>kk</td>
    <td>Cyrillic</td>
    <td>Turkic</td>
    <td>Common Turkic</td>
    <td>Mid</td>
  </tr>
  <tr>
    <td>Kyrgyz</td>
    <td>ky</td>
    <td>Cyrillic</td>
    <td>Turkic</td>
    <td>Common Turkic</td>
    <td>Low</td>
  </tr>
  <tr>
    <td>Persian</td>
    <td>fa</td>
    <td>Arabic</td>
    <td>Indo-European</td>
    <td>Iranian</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Turkish</td>
    <td>tr</td>
    <td>Latin</td>
    <td>Turkic</td>
    <td>Common Turkic</td>
    <td>High</td>
  </tr>
  <tr>
    <td>Uzbek</td>
    <td>uz</td>
    <td>Latin</td>
    <td>Turkic</td>
    <td>Common Turkic</td>
    <td>Low</td>
  </tr>
</table>

Table 4: Detailed information of all languages (cont.)
Figure 7: Performance of different models trained on varying numbers of languages (evaluated by SacreBLEU). The dotted line represents the performance of separately trained models, serving as a reference point where no language conflicts or synergies occur. Two key findings emerge: (1) Asymmetry in Linguistic Conflicts and Synergy (Figure a–i), highlighting the uneven impact of multilingual training across language pairs; and (2) The Bottleneck of Multilinguality in Post-Training (Figure g–i), showing that while monolingual pre-training provides an ideal foundation for handling multiple languages, the post-training stage imposes constraints that lead to the curse of multilinguality.
<table>
  <tr>
    <th>Languages (Trained)</th>
    <th>Languages (Test)</th>
    <th>XX→En</th>
    <th>En→XX</th>
    <th>All</th>
  </tr>
  <tr>
    <th colspan="5">ALMA-7B-Pretrain</th>
  </tr>
  <tr>
    <td>5</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>86.8/32.5</td>
    <td>88.5/32.0</td>
    <td>87.7/32.3</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>79.8/19.0</td>
    <td>88.2/30.9</td>
    <td>84.0/25.0</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>76.4/17.7</td>
    <td>88.4/31.3</td>
    <td>82.4/24.5</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>79.6/18.2</td>
    <td>87.4/26.5</td>
    <td>83.5/22.4</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>76.5/17.3</td>
    <td>87.5/27.2</td>
    <td>82.0/22.3</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>74.3/14.7</td>
    <td>78.8/19.8</td>
    <td>76.6/17.3</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>70.4/13.3</td>
    <td>78.7/20.4</td>
    <td>74.6/16.7</td>
  </tr>
  <tr>
    <th colspan="5">ALMA-13B-Pretrain</th>
  </tr>
  <tr>
    <td>5</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>86.7/31.9</td>
    <td>89.2/33.9</td>
    <td>88.0/32.9</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>83.2/23.3</td>
    <td>89.1/34.5</td>
    <td>86.2/28.9</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>78.0/18.5</td>
    <td>89.0/34.3</td>
    <td>83.5/26.4</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>83.5/23.1</td>
    <td>89.2/30.6</td>
    <td>86.4/26.9</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>78.8/18.9</td>
    <td>89.1/30.4</td>
    <td>84.0/24.7</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>79.8/19.6</td>
    <td>83.3/23.8</td>
    <td>81.6/21.7</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>73.4/14.9</td>
    <td>83.2/23.6</td>
    <td>78.3/19.3</td>
  </tr>
  <tr>
    <th colspan="5">X-ALMA-13B-Pretrain</th>
  </tr>
  <tr>
    <td>5</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>86.5/31.7</td>
    <td>88.4/33.7</td>
    <td>87.5/32.7</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>83.6/23.1</td>
    <td>89.0/33.9</td>
    <td>86.3/28.5</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS]</td>
    <td>80.9/19.5</td>
    <td>88.9/33.4</td>
    <td>84.9/26.5</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>84.2/23.2</td>
    <td>89.9/31.8</td>
    <td>87.1/27.5</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro]</td>
    <td>81.6/19.8</td>
    <td>89.8/31.2</td>
    <td>85.7/25.5</td>
  </tr>
  <tr>
    <td>17</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>83.7/22.2</td>
    <td>88.5/28.4</td>
    <td>86.1/25.3</td>
  </tr>
  <tr>
    <td>50</td>
    <td>[De, Zh, Ru, CS, Ja, Fi, Uk, Ro, Is, Kk, Fr, Lv, Gu, He, Hi, Hu]</td>
    <td>80.6/18.2</td>
    <td>88.4/28.1</td>
    <td>84.5/23.2</td>
  </tr>
</table>

Table 5: The average performance of language in anchor set significantly decreases as the number of trained languages increase in XX-En directions while the average performance in En-XX directions maintain stable. This indicates the linguistic conflicts is predominant in XX-En directions, which is consistent with the findings in Section 3. Performance was evaluated on Flores200 test sets.

<table>
  <tr>
    <th rowspan="2">LoRA Rank</th>
    <th colspan="2">En-XX</th>
    <th colspan="2">XX-En</th>
    <th colspan="2">Avg.</th>
  </tr>
  <tr>
    <th>COMET-22</th>
    <th>SacreBLEU</th>
    <th>COMET-22</th>
    <th>SacreBLEU</th>
    <th>COMET-22</th>
    <th>SacreBLEU</th>
  </tr>
  <tr>
    <td>16</td>
    <td>83.3</td>
    <td>23.8</td>
    <td>79.8</td>
    <td>19.6</td>
    <td>81.5</td>
    <td>21.6</td>
  </tr>
  <tr>
    <td>32</td>
    <td>83.6</td>
    <td>24.1</td>
    <td>79.2</td>
    <td>19.6</td>
    <td>81.4</td>
    <td>21.8</td>
  </tr>
</table>

Table 6: Performance of ALMA-13B-Pretrain on FLores-200 Test Sets for Different LoRA Ranks.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="7">XX→En</th>
    <th colspan="7">En→XX</th>
  </tr>
  <tr>
    <th>Af</th>
    <th>Da</th>
    <th>Nl</th>
    <th>De</th>
    <th>Is</th>
    <th>No</th>
    <th>Sv</th>
    <th>Avg.</th>
    <th>Af</th>
    <th>Da</th>
    <th>Nl</th>
    <th>De</th>
    <th>Is</th>
    <th>No</th>
    <th>Sv</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="16">ALMA-13B-Pretrain</th>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>84.85</td>
    <td>86.55</td>
    <td>84.82</td>
    <td>87.71</td>
    <td>85.19</td>
    <td>85.63</td>
    <td>86.92</td>
    <td>85.95</td>
    <td>83.77</td>
    <td>89.21</td>
    <td>87.44</td>
    <td>88.15</td>
    <td>86.93</td>
    <td>88.51</td>
    <td>89.72</td>
    <td>87.68</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>66.14</td>
    <td>70.25</td>
    <td>69.87</td>
    <td>79.43</td>
    <td>80.38</td>
    <td>69.23</td>
    <td>71.04</td>
    <td>72.33</td>
    <td>82.65</td>
    <td>88.68</td>
    <td>87.01</td>
    <td>87.98</td>
    <td>86.80</td>
    <td>88.04</td>
    <td>89.22</td>
    <td>87.20</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>87.56</td>
    <td>89.60</td>
    <td>87.38</td>
    <td>89.35</td>
    <td>87.04</td>
    <td>88.44</td>
    <td>89.79</td>
    <td>88.45</td>
    <td>80.94</td>
    <td>88.05</td>
    <td>86.81</td>
    <td>87.72</td>
    <td>86.89</td>
    <td>87.35</td>
    <td>88.58</td>
    <td>86.62</td>
  </tr>
  <tr>
    <th colspan="16">X-ALMA-13B-Pretrain</th>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>86.35</td>
    <td>87.07</td>
    <td>84.97</td>
    <td>87.45</td>
    <td>84.77</td>
    <td>86.15</td>
    <td>87.32</td>
    <td>86.30</td>
    <td>86.61</td>
    <td>91.11</td>
    <td>88.18</td>
    <td>87.99</td>
    <td>86.75</td>
    <td>89.94</td>
    <td>91.13</td>
    <td>88.82</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>76.65</td>
    <td>78.71</td>
    <td>76.05</td>
    <td>82.53</td>
    <td>82.65</td>
    <td>77.29</td>
    <td>78.91</td>
    <td>78.97</td>
    <td>86.51</td>
    <td>90.93</td>
    <td>88.27</td>
    <td>87.77</td>
    <td>86.30</td>
    <td>89.81</td>
    <td>90.82</td>
    <td>88.63</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>89.32</td>
    <td>90.23</td>
    <td>87.46</td>
    <td>89.28</td>
    <td>86.70</td>
    <td>88.93</td>
    <td>90.13</td>
    <td>88.86</td>
    <td>86.36</td>
    <td>90.53</td>
    <td>87.96</td>
    <td>87.66</td>
    <td>86.52</td>
    <td>89.65</td>
    <td>90.70</td>
    <td>88.48</td>
  </tr>
</table>

Table 7: Result of various training strategies on Group 1 languages. The performance is evaluated by COMET-22.
<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 2 (Ca, Gl, It, Pt, Ro, Es)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Ca</th><th>Gl</th><th>It</th><th>Pt</th><th>Ro</th><th>Es</th><th>Avg.</th>
    <th>Ca</th><th>Gl</th><th>It</th><th>Pt</th><th>Ro</th><th>Es</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.29</td><td>85.71</td><td>85.97</td><td>86.74</td><td>87.51</td><td>85.43</td><td>86.28</td><td>87.44</td><td>86.43</td><td>88.10</td><td>88.82</td><td>88.95</td><td>86.39</td><td>87.69</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>69.41</td><td>67.79</td><td>70.34</td><td>69.30</td><td>82.55</td><td>67.91</td><td>71.22</td><td>87.14</td><td>85.52</td><td>87.94</td><td>88.69</td><td>88.67</td><td>86.21</td><td>87.36</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>88.59</td><td>87.88</td><td>88.13</td><td>89.33</td><td>89.06</td><td>87.42</td><td>88.40</td><td>87.10</td><td>84.59</td><td>87.83</td><td>88.53</td><td>88.61</td><td>86.09</td><td>87.12</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.59</td><td>86.23</td><td>86.20</td><td>87.06</td><td>87.88</td><td>85.41</td><td>86.56</td><td>88.08</td><td>87.82</td><td>88.55</td><td>89.44</td><td>90.71</td><td>86.78</td><td>88.56</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>76.67</td><td>76.44</td><td>76.87</td><td>77.03</td><td>84.77</td><td>75.93</td><td>77.95</td><td>88.03</td><td>87.72</td><td>88.32</td><td>89.36</td><td>90.70</td><td>86.68</td><td>88.47</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>88.81</td><td>88.46</td><td>88.18</td><td>89.32</td><td>89.36</td><td>87.50</td><td>88.61</td><td>87.77</td><td>87.59</td><td>88.29</td><td>89.21</td><td>90.42</td><td>86.51</td><td>88.30</td>
  </tr>
</table>

Table 8: Result of various training strategies on Group 2 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="5">Group 3 (Bg, Mk, Ru, Sr, Uk)</th>
    <th colspan="5">En→XX</th>
  </tr>
  <tr>
    <th>Bg</th><th>Mk</th><th>Ru</th><th>Sr</th><th>Uk</th><th>Avg.</th>
    <th>Bg</th><th>Mk</th><th>Ru</th><th>Sr</th><th>Uk</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>85.48</td><td>84.30</td><td>85.34</td><td>85.44</td><td>85.49</td><td>85.21</td><td>89.16</td><td>85.24</td><td>89.69</td><td>86.80</td><td>89.16</td><td>88.01</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>69.35</td><td>66.40</td><td>76.99</td><td>68.08</td><td>77.54</td><td>71.67</td><td>88.47</td><td>83.57</td><td>89.44</td><td>86.71</td><td>89.16</td><td>87.47</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>87.45</td><td>85.78</td><td>86.86</td><td>87.12</td><td>87.26</td><td>86.89</td><td>88.21</td><td>81.19</td><td>89.57</td><td>86.14</td><td>89.21</td><td>86.86</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.50</td><td>85.77</td><td>85.60</td><td>86.00</td><td>85.84</td><td>85.94</td><td>90.95</td><td>89.55</td><td>89.64</td><td>88.74</td><td>90.18</td><td>89.81</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>76.11</td><td>75.92</td><td>79.33</td><td>75.76</td><td>79.44</td><td>77.31</td><td>90.90</td><td>89.45</td><td>89.64</td><td>89.67</td><td>89.87</td><td>89.91</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>87.98</td><td>87.76</td><td>86.78</td><td>87.81</td><td>87.36</td><td>87.54</td><td>90.51</td><td>89.44</td><td>89.34</td><td>89.13</td><td>90.08</td><td>89.70</td>
  </tr>
</table>

Table 9: Result of various training strategies on Group 3 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="5">Group 4 (Fr, Id, Mg, Ms, Th, Vi)</th>
    <th colspan="5">En→XX</th>
  </tr>
  <tr>
    <th>Fr</th><th>Id</th><th>Mg</th><th>Ms</th><th>Th</th><th>Vi</th><th>Avg.</th>
    <th>Fr</th><th>Id</th><th>Mg</th><th>Ms</th><th>Th</th><th>Vi</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.98</td><td>85.93</td><td>63.47</td><td>84.55</td><td>77.34</td><td>83.41</td><td>80.28</td><td>87.60</td><td>89.92</td><td>63.79</td><td>86.85</td><td>72.74</td><td>87.01</td><td>81.32</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>79.99</td><td>68.69</td><td>53.49</td><td>66.90</td><td>61.02</td><td>67.33</td><td>66.24</td><td>87.53</td><td>89.60</td><td>63.29</td><td>86.48</td><td>70.57</td><td>86.07</td><td>80.59</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>89.22</td><td>88.76</td><td>56.68</td><td>87.45</td><td>78.61</td><td>86.59</td><td>81.22</td><td>87.46</td><td>89.36</td><td>39.87</td><td>85.64</td><td>59.28</td><td>86.49</td><td>74.68</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.94</td><td>86.35</td><td>73.33</td><td>84.82</td><td>85.07</td><td>84.24</td><td>83.46</td><td>87.96</td><td>90.75</td><td>76.37</td><td>87.89</td><td>86.51</td><td>88.49</td><td>86.33</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>83.49</td><td>77.59</td><td>65.08</td><td>75.95</td><td>76.97</td><td>75.77</td><td>75.81</td><td>88.07</td><td>90.73</td><td>75.70</td><td>87.80</td><td>86.25</td><td>88.40</td><td>86.16</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>89.25</td><td>89.11</td><td>74.75</td><td>87.71</td><td>87.57</td><td>87.21</td><td>85.93</td><td>87.72</td><td>90.75</td><td>71.99</td><td>87.46</td><td>85.84</td><td>88.08</td><td>85.31</td>
  </tr>
</table>

Table 10: Result of various training strategies on Group 4 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 5 (Cs, El, Hu, Lv, Lt, Pl)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Cs</th><th>El</th><th>Hu</th><th>Lv</th><th>Lt</th><th>Pl</th><th>Avg.</th>
    <th>Cs</th><th>El</th><th>Hu</th><th>Lv</th><th>Lt</th><th>Pl</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>87.92</td><td>81.93</td><td>85.57</td><td>79.42</td><td>76.79</td><td>84.70</td><td>82.72</td><td>91.28</td><td>76.51</td><td>86.69</td><td>69.81</td><td>67.41</td><td>88.21</td><td>79.98</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>79.94</td><td>62.88</td><td>68.91</td><td>68.85</td><td>63.48</td><td>80.26</td><td>70.72</td><td>91.11</td><td>75.92</td><td>86.24</td><td>68.90</td><td>66.40</td><td>87.63</td><td>79.37</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>88.77</td><td>82.72</td><td>87.66</td><td>78.75</td><td>75.81</td><td>85.94</td><td>83.28</td><td>90.95</td><td>68.11</td><td>86.23</td><td>61.62</td><td>58.89</td><td>87.96</td><td>75.63</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>87.30</td><td>85.77</td><td>85.81</td><td>85.63</td><td>84.58</td><td>84.64</td><td>85.62</td><td>91.31</td><td>89.03</td><td>89.57</td><td>89.80</td><td>90.07</td><td>89.69</td><td>89.91</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>82.44</td><td>76.80</td><td>76.78</td><td>81.80</td><td>78.77</td><td>81.73</td><td>79.72</td><td>91.31</td><td>88.88</td><td>89.36</td><td>89.49</td><td>89.67</td><td>89.70</td><td>89.73</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>88.58</td><td>87.51</td><td>87.99</td><td>87.58</td><td>86.49</td><td>86.14</td><td>87.38</td><td>90.83</td><td>88.87</td><td>89.03</td><td>89.64</td><td>89.79</td><td>89.61</td><td>89.63</td>
  </tr>
</table>

Table 11: Result of various training strategies on Group 5 languages. The performance is evaluated by COMET-22.
<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 6 (Et, Fi, Ja, Ka, Ko, Zh)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Et</th><th>Fi</th><th>Ja</th><th>Ka</th><th>Ko</th><th>Zh</th><th>Avg.</th>
    <th>Et</th><th>Fi</th><th>Ja</th><th>Ka</th><th>Ko</th><th>Zh</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>80.84</td><td>87.60</td><td>86.18</td><td>73.71</td><td>85.36</td><td>85.22</td><td>83.15</td><td>73.01</td><td>89.85</td><td>89.92</td><td>61.63</td><td>84.44</td><td>87.75</td><td>81.10</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>71.87</td><td>80.54</td><td>77.25</td><td>58.64</td><td>69.78</td><td>75.86</td><td>72.32</td><td>70.26</td><td>89.19</td><td>89.64</td><td>62.08</td><td>85.68</td><td>87.55</td><td>80.73</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>80.97</td><td>89.15</td><td>87.38</td><td>70.05</td><td>86.85</td><td>86.71</td><td>83.52</td><td>61.16</td><td>89.58</td><td>89.64</td><td>41.78</td><td>85.28</td><td>87.29</td><td>75.79</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>87.23</td><td>88.25</td><td>86.16</td><td>84.34</td><td>85.97</td><td>85.01</td><td>86.16</td><td>90.79</td><td>92.29</td><td>90.44</td><td>87.02</td><td>88.14</td><td>87.46</td><td>89.36</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>83.70</td><td>84.38</td><td>80.55</td><td>74.73</td><td>77.59</td><td>79.11</td><td>80.01</td><td>90.55</td><td>92.00</td><td>90.35</td><td>86.23</td><td>87.97</td><td>86.92</td><td>89.00</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>88.75</td><td>89.59</td><td>87.64</td><td>86.37</td><td>87.49</td><td>86.62</td><td>87.74</td><td>90.45</td><td>92.13</td><td>90.40</td><td>86.27</td><td>87.36</td><td>86.99</td><td>88.93</td>
  </tr>
</table>

Table 12: Result of various training strategies on Group 6 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="5">Group 7 (Gu, Hi, Mr, Ne, Ur)</th>
    <th colspan="5">En→XX</th>
  </tr>
  <tr>
    <th>Gu</th><th>Hi</th><th>Mr</th><th>Ne</th><th>Ur</th><th>Avg.</th>
    <th>Gu</th><th>Hi</th><th>Mr</th><th>Ne</th><th>Ur</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>70.05</td><td>83.00</td><td>76.42</td><td>81.54</td><td>76.14</td><td>77.43</td><td>73.06</td><td>69.72</td><td>59.17</td><td>72.70</td><td>68.72</td><td>68.67</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>59.78</td><td>65.44</td><td>60.27</td><td>65.85</td><td>59.95</td><td>62.26</td><td>76.01</td><td>70.68</td><td>61.03</td><td>74.13</td><td>71.06</td><td>70.58</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>66.96</td><td>84.27</td><td>76.60</td><td>82.62</td><td>77.47</td><td>77.59</td><td>59.24</td><td>62.51</td><td>45.03</td><td>59.68</td><td>60.75</td><td>57.44</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>86.43</td><td>87.19</td><td>85.57</td><td>87.82</td><td>84.84</td><td>86.37</td><td>86.68</td><td>79.15</td><td>74.19</td><td>82.71</td><td>82.09</td><td>80.96</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>79.75</td><td>77.66</td><td>76.45</td><td>79.23</td><td>75.84</td><td>77.79</td><td>86.60</td><td>79.37</td><td>73.76</td><td>82.29</td><td>82.21</td><td>80.85</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>87.07</td><td>88.83</td><td>87.44</td><td>89.29</td><td>86.49</td><td>87.82</td><td>85.91</td><td>78.57</td><td>73.17</td><td>81.71</td><td>81.84</td><td>80.24</td>
  </tr>
</table>

Table 13: Result of various training strategies on Group 7 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 8 (Ar, Az, He, Kk, Ky, Fa, Tr, Uz)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Ar</th><th>Az</th><th>He</th><th>Kk</th><th>Ky</th><th>Fa</th><th>Tr</th><th>Uz</th><th>Avg.</th>
    <th>Ar</th><th>Az</th><th>He</th><th>Kk</th><th>Ky</th><th>Fa</th><th>Tr</th><th>Uz</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>74.17</td><td>72.83</td><td>70.54</td><td>73.58</td><td>70.06</td><td>75.22</td><td>82.01</td><td>70.34</td><td>73.59</td><td>77.14</td><td>63.41</td><td>71.63</td><td>68.57</td><td>62.74</td><td>71.52</td><td>77.61</td><td>66.83</td><td>69.93</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>60.33</td><td>61.77</td><td>57.48</td><td>63.74</td><td>60.45</td><td>62.40</td><td>76.40</td><td>59.61</td><td>62.77</td><td>75.78</td><td>65.72</td><td>71.12</td><td>71.08</td><td>66.12</td><td>72.18</td><td>78.45</td><td>69.64</td><td>70.89</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>80.97</td><td>75.94</td><td>77.71</td><td>72.45</td><td>77.85</td><td>81.80</td><td>85.36</td><td>70.53</td><td>76.50</td><td>73.00</td><td>43.13</td><td>62.30</td><td>46.70</td><td>35.71</td><td>64.67</td><td>74.37</td><td>44.37</td><td>55.53</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>81.70</td><td>82.67</td><td>83.28</td><td>83.53</td><td>81.19</td><td>83.14</td><td>87.15</td><td>81.15</td><td>82.98</td><td>86.40</td><td>86.86</td><td>87.74</td><td>89.41</td><td>87.42</td><td>86.69</td><td>88.50</td><td>86.80</td><td>87.48</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>74.48</td><td>75.55</td><td>75.45</td><td>78.74</td><td>74.45</td><td>75.96</td><td>83.65</td><td>73.35</td><td>76.45</td><td>86.11</td><td>86.67</td><td>87.44</td><td>89.33</td><td>87.22</td><td>86.60</td><td>88.35</td><td>86.20</td><td>87.24</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>86.57</td><td>85.65</td><td>87.77</td><td>87.00</td><td>84.49</td><td>87.23</td><td>88.83</td><td>84.21</td><td>86.47</td><td>85.72</td><td>85.81</td><td>87.88</td><td>88.84</td><td>88.45</td><td>86.44</td><td>87.94</td><td>83.71</td><td>86.60</td>
  </tr>
</table>

Table 14: Result of various training strategies on Group 8 languages. The performance is evaluated by COMET-22.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 1 (Af, Da, Ni, De, Is, No, Sv)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Af</th><th>Da</th><th>Ni</th><th>De</th><th>Is</th><th>No</th><th>Sv</th><th>Avg.</th>
    <th>Af</th><th>Da</th><th>Ni</th><th>De</th><th>Is</th><th>No</th><th>Sv</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>35.39</td><td>31.82</td><td>23.66</td><td>34.88</td><td>28.67</td><td>29.25</td><td>31.76</td><td>30.78</td><td>33.13</td><td>38.35</td><td>24.66</td><td>36.48</td><td>25.37</td><td>27.93</td><td>37.83</td><td>31.96</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>10.68</td><td>12.10</td><td>10.23</td><td>21.00</td><td>24.37</td><td>11.10</td><td>12.09</td><td>14.51</td><td>33.00</td><td>36.93</td><td>24.34</td><td>36.82</td><td>25.07</td><td>27.65</td><td>37.18</td><td>31.57</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>52.88</td><td>46.74</td><td>52.21</td><td>44.21</td><td>37.39</td><td>42.92</td><td>47.21</td><td>43.37</td><td>27.63</td><td>35.25</td><td>23.39</td><td>36.22</td><td>25.41</td><td>25.40</td><td>37.72</td><td>28.96</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>37.40</td><td>32.39</td><td>23.63</td><td>33.66</td><td>28.30</td><td>29.68</td><td>32.74</td><td>31.11</td><td>40.68</td><td>42.78</td><td>26.38</td><td>36.09</td><td>24.23</td><td>30.85</td><td>41.80</td><td>34.69</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>16.45</td><td>16.06</td><td>11.58</td><td>22.41</td><td>24.56</td><td>14.76</td><td>16.14</td><td>17.42</td><td>40.21</td><td>41.95</td><td>26.52</td><td>35.32</td><td>23.82</td><td>30.57</td><td>41.79</td><td>34.31</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>57.35</td><td>48.92</td><td>32.66</td><td>43.83</td><td>35.92</td><td>43.87</td><td>48.54</td><td>44.44</td><td>39.08</td><td>42.41</td><td>26.08</td><td>35.04</td><td>24.05</td><td>30.50</td><td>41.25</td><td>34.06</td>
  </tr>
</table>

Table 15: Result of various training strategies on Group 1 languages. The performance is evaluated by SacreBLEU.
<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 2 (Ca, Gl, It, Pt, Ro, Es)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Ca</th><th>Gl</th><th>It</th><th>Pt</th><th>Ro</th><th>Es</th><th>Avg.</th>
    <th>Ca</th><th>Gl</th><th>It</th><th>Pt</th><th>Ro</th><th>Es</th><th>Avg.</th>
  </tr>
  <tr>
    <td><b>ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>33.77</td><td>31.56</td><td>27.82</td><td>34.82</td><td>35.49</td><td>26.81</td><td>31.71</td>
    <td>39.10</td><td>31.01</td><td>27.53</td><td>41.43</td><td>32.21</td><td>27.81</td><td>33.18</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>11.95</td><td>10.61</td><td>11.07</td><td>11.14</td><td>27.33</td><td>9.09</td><td>13.53</td>
    <td>38.50</td><td>29.64</td><td>27.24</td><td>40.89</td><td>32.24</td><td>27.02</td><td>32.59</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>46.06</td><td>41.02</td><td>35.75</td><td>47.77</td><td>43.82</td><td>33.89</td><td>41.38</td>
    <td>37.44</td><td>27.43</td><td>26.37</td><td>40.84</td><td>31.85</td><td>26.38</td><td>31.72</td>
  </tr>
  <tr>
    <td><b>X-ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>33.79</td><td>31.43</td><td>27.45</td><td>34.99</td><td>35.41</td><td>26.29</td><td>31.56</td>
    <td>41.11</td><td>35.11</td><td>29.31</td><td>43.43</td><td>37.63</td><td>28.15</td><td>35.79</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>14.74</td><td>13.75</td><td>13.19</td><td>14.84</td><td>28.91</td><td>12.40</td><td>16.30</td>
    <td>40.59</td><td>34.85</td><td>29.10</td><td>43.23</td><td>36.83</td><td>28.02</td><td>35.44</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>46.01</td><td>42.02</td><td>35.54</td><td>47.73</td><td>44.62</td><td>33.79</td><td>41.62</td>
    <td>39.45</td><td>33.71</td><td>28.55</td><td>43.09</td><td>36.86</td><td>27.55</td><td>34.87</td>
  </tr>
</table>

Table 16: Result of various training strategies on Group 2 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="5">Group 3 (Bg, Mk, Ru, Sr, Uk)</th>
    <th colspan="5">En→XX</th>
  </tr>
  <tr>
    <th>Bg</th><th>Mk</th><th>Ru</th><th>Sr</th><th>Uk</th><th>Avg.</th>
    <th>Bg</th><th>Mk</th><th>Ru</th><th>Sr</th><th>Uk</th><th>Avg.</th>
  </tr>
  <tr>
    <td><b>ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>30.72</td><td>31.47</td><td>30.09</td><td>33.53</td><td>31.89</td><td>31.54</td>
    <td>30.05</td><td>22.85</td><td>29.03</td><td>26.63</td><td>25.21</td><td>26.75</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>9.96</td><td>9.16</td><td>18.08</td><td>10.21</td><td>19.70</td><td>13.42</td>
    <td>30.21</td><td>21.87</td><td>29.62</td><td>26.79</td><td>25.28</td><td>26.75</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>39.56</td><td>38.58</td><td>36.13</td><td>42.05</td><td>40.32</td><td>39.33</td>
    <td>28.31</td><td>17.45</td><td>29.18</td><td>23.40</td><td>25.11</td><td>24.69</td>
  </tr>
  <tr>
    <td><b>X-ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>33.56</td><td>34.13</td><td>30.92</td><td>34.43</td><td>32.81</td><td>33.17</td>
    <td>36.66</td><td>32.41</td><td>28.98</td><td>31.32</td><td>27.09</td><td>31.29</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>12.96</td><td>13.49</td><td>18.33</td><td>14.14</td><td>18.85</td><td>15.55</td>
    <td>36.36</td><td>32.30</td><td>29.38</td><td>32.00</td><td>26.90</td><td>31.39</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>40.40</td><td>43.17</td><td>36.69</td><td>44.00</td><td>40.71</td><td>40.99</td>
    <td>35.61</td><td>31.28</td><td>28.61</td><td>30.38</td><td>26.45</td><td>30.47</td>
  </tr>
</table>

Table 17: Result of various training strategies on Group 3 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 4 (Fr, Id, Mg, Ms, Th, Vi)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Fr</th><th>Id</th><th>Mg</th><th>Ms</th><th>Th</th><th>Vi</th><th>Avg.</th>
    <th>Fr</th><th>Id</th><th>Mg</th><th>Ms</th><th>Th</th><th>Vi</th><th>Avg.</th>
  </tr>
  <tr>
    <td><b>ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>35.54</td><td>30.22</td><td>10.13</td><td>29.61</td><td>13.75</td><td>25.54</td><td>24.13</td>
    <td>43.95</td><td>37.60</td><td>4.01</td><td>28.67</td><td>6.73</td><td>34.86</td><td>25.97</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>24.53</td><td>9.34</td><td>3.85</td><td>9.18</td><td>4.22</td><td>8.58</td><td>9.95</td>
    <td>44.11</td><td>36.40</td><td>4.38</td><td>28.37</td><td>7.15</td><td>33.54</td><td>25.66</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>44.78</td><td>41.77</td><td>7.63</td><td>40.57</td><td>16.36</td><td>34.66</td><td>30.96</td>
    <td>43.97</td><td>35.24</td><td>54.00</td><td>25.13</td><td>4.11</td><td>32.97</td><td>23.66</td>
  </tr>
  <tr>
    <td><b>X-ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>35.04</td><td>31.29</td><td>16.10</td><td>29.44</td><td>21.36</td><td>26.20</td><td>26.57</td>
    <td>46.13</td><td>39.97</td><td>9.99</td><td>30.83</td><td>9.93</td><td>38.68</td><td>29.25</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>27.13</td><td>13.59</td><td>6.83</td><td>13.09</td><td>10.01</td><td>12.26</td><td>13.82</td>
    <td>45.36</td><td>40.01</td><td>9.23</td><td>29.28</td><td>11.36</td><td>37.54</td><td>28.80</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>44.69</td><td>42.99</td><td>19.64</td><td>40.79</td><td>29.92</td><td>36.98</td><td>35.84</td>
    <td>45.30</td><td>39.49</td><td>6.48</td><td>28.01</td><td>9.78</td><td>37.43</td><td>27.75</td>
  </tr>
</table>

Table 18: Result of various training strategies on Group 4 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 5 (Cs, El, Hu, Lv, Lt, Pl)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Cs</th><th>El</th><th>Hu</th><th>Lv</th><th>Lt</th><th>Pl</th><th>Avg.</th>
    <th>Cs</th><th>El</th><th>Hu</th><th>Lv</th><th>Lt</th><th>Pl</th><th>Avg.</th>
  </tr>
  <tr>
    <td><b>ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>36.91</td><td>23.58</td><td>26.31</td><td>21.62</td><td>19.37</td><td>27.05</td><td>25.81</td>
    <td>31.04</td><td>13.04</td><td>17.21</td><td>10.46</td><td>8.63</td><td>18.20</td><td>16.43</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>22.08</td><td>6.40</td><td>8.19</td><td>11.58</td><td>8.37</td><td>21.72</td><td>13.06</td>
    <td>31.02</td><td>13.59</td><td>17.70</td><td>10.33</td><td>9.68</td><td>18.14</td><td>16.74</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>41.62</td><td>27.38</td><td>33.58</td><td>22.89</td><td>19.97</td><td>30.96</td><td>29.40</td>
    <td>30.62</td><td>9.04</td><td>15.56</td><td>6.55</td><td>5.59</td><td>17.93</td><td>14.22</td>
  </tr>
  <tr>
    <td><b>X-ALMA-13B-Pretrain</b></td>
    <td colspan="7"></td>
    <td colspan="7"></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td>
    <td>34.26</td><td>28.15</td><td>25.25</td><td>28.89</td><td>26.61</td><td>26.19</td><td>28.22</td>
    <td>30.91</td><td>23.62</td><td>21.91</td><td>24.56</td><td>22.52</td><td>20.28</td><td>23.97</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td>
    <td>23.16</td><td>12.64</td><td>11.68</td><td>21.52</td><td>16.20</td><td>21.59</td><td>17.80</td>
    <td>30.23</td><td>23.40</td><td>21.72</td><td>23.96</td><td>22.27</td><td>20.02</td><td>23.60</td>
  </tr>
  <tr>
    <td>Separate Training</td>
    <td>40.95</td><td>35.56</td><td>34.62</td><td>35.99</td><td>33.79</td><td>31.12</td><td>35.34</td>
    <td>29.92</td><td>22.18</td><td>21.17</td><td>25.27</td><td>22.27</td><td>20.05</td><td>23.48</td>
  </tr>
</table>

Table 19: Result of various training strategies on Group 5 languages. The performance is evaluated by SacreBLEU.
<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 6 (Et, Fi, Ja, Ka, Ko, Zh)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Et</th><th>Fi</th><th>Ja</th><th>Ka</th><th>Ko</th><th>Zh</th><th>Avg.</th>
    <th>Et</th><th>Fi</th><th>Ja</th><th>Ka</th><th>Ko</th><th>Zh</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>22.49</td><td>27.83</td><td>22.64</td><td>12.35</td><td>21.50</td><td>23.92</td><td>21.79</td><td>10.60</td><td>18.59</td><td>29.63</td><td>4.40</td><td>6.79</td><td>40.49</td><td>18.42</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>13.25</td><td>18.25</td><td>11.69</td><td>3.56</td><td>7.05</td><td>12.82</td><td>11.10</td><td>10.10</td><td>18.29</td><td>28.71</td><td>5.34</td><td>9.87</td><td>39.74</td><td>18.67</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>24.10</td><td>33.33</td><td>26.70</td><td>9.22</td><td>26.77</td><td>28.95</td><td>24.84</td><td>5.44</td><td>17.45</td><td>28.49</td><td>1.52</td><td>9.06</td><td>38.79</td><td>16.79</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>29.46</td><td>27.76</td><td>22.32</td><td>20.23</td><td>22.24</td><td>22.97</td><td>24.16</td><td>21.92</td><td>22.78</td><td>31.87</td><td>12.59</td><td>10.08</td><td>39.99</td><td>23.21</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>22.32</td><td>20.01</td><td>12.44</td><td>9.06</td><td>10.38</td><td>14.09</td><td>14.72</td><td>21.48</td><td>21.38</td><td>30.89</td><td>12.32</td><td>11.82</td><td>38.65</td><td>22.76</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>35.84</td><td>34.19</td><td>26.94</td><td>27.00</td><td>28.37</td><td>28.31</td><td>30.11</td><td>21.52</td><td>21.62</td><td>31.76</td><td>11.54</td><td>10.65</td><td>38.22</td><td>22.55</td>
  </tr>
</table>

Table 20: Result of various training strategies on Group 6 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="5">Group 7 (Gu, Hi, Mr, Ne, Ur)</th>
    <th colspan="5">En→XX</th>
  </tr>
  <tr>
    <th>Gu</th><th>Hi</th><th>Mr</th><th>Ne</th><th>Ur</th><th>Avg.</th>
    <th>Gu</th><th>Hi</th><th>Mr</th><th>Ne</th><th>Ur</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>10.95</td><td>24.45</td><td>16.88</td><td>19.50</td><td>16.89</td><td>17.73</td><td>8.78</td><td>15.68</td><td>5.63</td><td>8.67</td><td>9.90</td><td>9.73</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>3.10</td><td>5.34</td><td>3.71</td><td>4.46</td><td>3.59</td><td>4.04</td><td>10.50</td><td>17.29</td><td>6.25</td><td>9.53</td><td>11.33</td><td>10.98</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>8.09</td><td>26.64</td><td>15.40</td><td>20.37</td><td>17.67</td><td>17.63</td><td>4.89</td><td>9.66</td><td>2.09</td><td>3.89</td><td>5.11</td><td>5.13</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>26.47</td><td>29.68</td><td>25.56</td><td>27.99</td><td>25.45</td><td>27.03</td><td>18.18</td><td>26.33</td><td>13.57</td><td>16.75</td><td>18.94</td><td>18.75</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>12.55</td><td>10.32</td><td>10.44</td><td>10.05</td><td>9.80</td><td>10.63</td><td>17.78</td><td>25.96</td><td>13.17</td><td>15.90</td><td>18.88</td><td>18.34</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>30.82</td><td>36.37</td><td>32.69</td><td>33.91</td><td>31.51</td><td>33.06</td><td>16.55</td><td>23.83</td><td>12.58</td><td>15.01</td><td>17.78</td><td>17.15</td>
  </tr>
</table>

Table 21: Result of various training strategies on Group 7 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="6">Group 8 (Ar, Az, He, Kk, Ky, Fa, Tr, Uz)</th>
    <th colspan="6">En→XX</th>
  </tr>
  <tr>
    <th>Ar</th><th>Az</th><th>He</th><th>Kk</th><th>Ky</th><th>Fa</th><th>Tr</th><th>Uz</th><th>Avg.</th>
    <th>Ar</th><th>Az</th><th>He</th><th>Kk</th><th>Ky</th><th>Fa</th><th>Tr</th><th>Uz</th><th>Avg.</th>
  </tr>
  <tr>
    <td colspan="13"><b>ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>14.67</td><td>9.20</td><td>14.01</td><td>10.83</td><td>7.40</td><td>14.42</td><td>21.09</td><td>9.47</td><td>12.64</td><td>10.04</td><td>3.34</td><td>10.70</td><td>4.42</td><td>2.41</td><td>11.27</td><td>9.99</td><td>3.16</td><td>6.92</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>5.09</td><td>3.87</td><td>4.01</td><td>5.63</td><td>3.46</td><td>5.18</td><td>15.65</td><td>4.19</td><td>5.88</td><td>11.04</td><td>4.76</td><td>10.80</td><td>5.87</td><td>2.87</td><td>12.39</td><td>11.96</td><td>5.05</td><td>8.09</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>28.08</td><td>12.64</td><td>25.00</td><td>11.36</td><td>6.59</td><td>24.96</td><td>28.39</td><td>10.72</td><td>18.47</td><td>8.03</td><td>8.40</td><td>6.60</td><td>1.13</td><td>40.00</td><td>6.09</td><td>7.95</td><td>53.00</td><td>3.95</td>
  </tr>
  <tr>
    <td colspan="13"><b>X-ALMA-13B-Pretrain</b></td>
  </tr>
  <tr>
    <td>Group Multilingual Training</td><td>21.40</td><td>16.27</td><td>24.74</td><td>19.94</td><td>14.91</td><td>20.98</td><td>28.69</td><td>17.76</td><td>20.59</td><td>20.13</td><td>11.89</td><td>25.11</td><td>17.26</td><td>11.53</td><td>21.95</td><td>21.56</td><td>10.98</td><td>17.55</td>
  </tr>
  <tr>
    <td>Multilingual Training (50)</td><td>11.13</td><td>7.89</td><td>11.92</td><td>12.76</td><td>8.08</td><td>10.83</td><td>20.84</td><td>7.77</td><td>11.40</td><td>19.43</td><td>12.11</td><td>24.68</td><td>17.22</td><td>10.65</td><td>21.66</td><td>20.39</td><td>10.52</td><td>17.08</td>
  </tr>
  <tr>
    <td>Separate Training</td><td>38.34</td><td>22.16</td><td>41.85</td><td>30.35</td><td>20.97</td><td>33.66</td><td>35.90</td><td>25.28</td><td>31.06</td><td>18.84</td><td>10.10</td><td>24.62</td><td>15.62</td><td>8.39</td><td>20.92</td><td>19.41</td><td>6.56</td><td>15.62</td>
  </tr>
</table>

Table 22: Result of various training strategies on Group 8 languages. The performance is evaluated by SacreBLEU.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Zh</th>
    <th colspan="2">De</th>
    <th colspan="2">Ru</th>
    <th colspan="2">Ja</th>
    <th colspan="2">Uk</th>
    <th colspan="2">Avg.</th>
  </tr>
  <tr>
    <th>BLEU</th><th>COMET</th>
    <th>BLEU</th><th>COMET</th>
    <th>BLEU</th><th>COMET</th>
    <th>BLEU</th><th>COMET</th>
    <th>BLEU</th><th>COMET</th>
    <th>BLEU</th><th>COMET</th>
  </tr>
  <tr>
    <td>NLLB-3.3B</td><td>11.4</td><td>67.8</td><td>20.1</td><td>66.6</td><td>24.4</td><td>76.7</td><td>6.8</td><td>65.8</td><td>33.1</td><td>79.0</td><td>19.2</td><td>71.2</td>
  </tr>
  <tr>
    <td>Aya-23-8B</td><td>22.6</td><td>78.8</td><td>32.3</td><td>82.1</td><td>30.9</td><td>81.6</td><td>19.8</td><td>80.2</td><td>39.2</td><td>85.0</td><td>29.0</td><td>81.5</td>
  </tr>
  <tr>
    <td>Aya-23-35B</td><td>23.5</td><td>79.7</td><td>32.7</td><td>82.3</td><td>31.7</td><td>82.2</td><td>21.3</td><td>81.6</td><td>39.1</td><td>85.7</td><td>29.7</td><td>82.3</td>
  </tr>
  <tr>
    <td>Aya-101</td><td>13.8</td><td>73.7</td><td>34.9</td><td>81.6</td><td>28.4</td><td>81.4</td><td>13.9</td><td>77.3</td><td>34.9</td><td>84.5</td><td>25.2</td><td>79.7</td>
  </tr>
  <tr>
    <td>LLaMA3X-Alpaca-8B</td><td>22.3</td><td>79.3</td><td>25.6</td><td>79.4</td><td>29.4</td><td>81.3</td><td>17.6</td><td>80.1</td><td>37.8</td><td>84.9</td><td>26.5</td><td>81.0</td>
  </tr>
  <tr>
    <td>X-ALMA-13B (Only SFT, MoE)</td><td>23.8</td><td>80.3</td><td>42.5</td><td>85.3</td><td>32.8</td><td>82.4</td><td>20.4</td><td>81.6</td><td>42.5</td><td>86.4</td><td>32.4</td><td>83.2</td>
  </tr>
  <tr>
    <td>X-ALMA-13B-DAT (MoE)</td><td>21.1</td><td>79.7</td><td>38.4</td><td>84.6</td><td>32.3</td><td>82.4</td><td>19.8</td><td>81.2</td><td>39.5</td><td>85.9</td><td>30.2</td><td>82.8</td>
  </tr>
  <tr>
    <td>X-ALMA-13B-DATM (MoE)</td><td>22.2</td><td>79.2</td><td>37.5</td><td>84.2</td><td>29.0</td><td>82.0</td><td>19.0</td><td>80.4</td><td>34.0</td><td>84.7</td><td>28.3</td><td>82.1</td>
  </tr>
</table>

Table 23: WMT23 XX→En translation results (BLEU and COMET). The results of baselines are directly sourced from Xu et al. (2024b). We keep the same generation configuration as Xu et al. (2024b).
<table>
  <tr>
    <th>Model</th>
    <th>Zh</th>
    <th>De</th>
    <th>Ru</th>
    <th>Ja</th>
    <th>Uk</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th></th>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
    <th>BLEU</th>
    <th>COMET</th>
  </tr>
  <tr>
    <td>NLLB-3.6B</td>
    <td>34.8</td>
    <td>79.6</td>
    <td>33.5</td>
    <td>79.7</td>
    <td>29.1</td>
    <td>83.8</td>
    <td>13.8</td>
    <td>81.6</td>
    <td>25.5</td>
    <td>82.8</td>
    <td>27.3</td>
    <td>81.5</td>
  </tr>
  <tr>
    <td>Aya-23-8B</td>
    <td>44.5</td>
    <td>85.3</td>
    <td>29.3</td>
    <td>80.4</td>
    <td>24.3</td>
    <td>84.3</td>
    <td>19.3</td>
    <td>86.5</td>
    <td>24.3</td>
    <td>84.3</td>
    <td>28.3</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>Aya-23-35B</td>
    <td>42.8</td>
    <td>84.6</td>
    <td>30.7</td>
    <td>80.7</td>
    <td>27.5</td>
    <td>84.7</td>
    <td>20.6</td>
    <td>86.4</td>
    <td>24.9</td>
    <td>84.0</td>
    <td>29.3</td>
    <td>84.1</td>
  </tr>
  <tr>
    <td>Aya-101</td>
    <td>25.4</td>
    <td>78.6</td>
    <td>25.1</td>
    <td>75.1</td>
    <td>22.1</td>
    <td>83.1</td>
    <td>14.1</td>
    <td>84.6</td>
    <td>19.7</td>
    <td>82.7</td>
    <td>21.3</td>
    <td>80.8</td>
  </tr>
  <tr>
    <td>LLaMAX3-Alpaca-8B</td>
    <td>34.0</td>
    <td>81.5</td>
    <td>20.9</td>
    <td>73.3</td>
    <td>23.5</td>
    <td>81.6</td>
    <td>11.9</td>
    <td>81.8</td>
    <td>19.8</td>
    <td>80.6</td>
    <td>22.0</td>
    <td>79.8</td>
  </tr>
  <tr>
    <td>X-ALMA-13B (Only SFT, MoE)</td>
    <td>47.5</td>
    <td>86.1</td>
    <td>40.9</td>
    <td>84.1</td>
    <td>31.5</td>
    <td>85.9</td>
    <td>22.3</td>
    <td>86.8</td>
    <td>27.4</td>
    <td>85.3</td>
    <td>33.9</td>
    <td>85.6</td>
  </tr>
  <tr>
    <td>X-ALMA-13B-DAT (MoE)</td>
    <td>40.3</td>
    <td>85.0</td>
    <td>35.3</td>
    <td>83.3</td>
    <td>27.6</td>
    <td>85.0</td>
    <td>19.0</td>
    <td>86.0</td>
    <td>23.5</td>
    <td>84.6</td>
    <td>29.1</td>
    <td>84.8</td>
  </tr>
  <tr>
    <td>X-ALMA-13B-DATM (MoE)</td>
    <td>40.3</td>
    <td>85.0</td>
    <td>35.3</td>
    <td>83.3</td>
    <td>27.6</td>
    <td>85.0</td>
    <td>19.0</td>
    <td>86.0</td>
    <td>23.5</td>
    <td>84.6</td>
    <td>29.1</td>
    <td>84.8</td>
  </tr>
</table>

Table 24: WMT23 En→XX translation results (BLEU and COMET). The results of baselines are directly sourced from Xu et al. (2024b). We keep the same generation configuration as Xu et al. (2024b).

<table>
  <tr>
    <th rowspan="2">Direction</th>
    <th rowspan="2">NLLB-3.3B</th>
    <th rowspan="2">LLaMAX3-Alpaca-8B</th>
    <th rowspan="2">Aya-101</th>
    <th rowspan="2">Aya-23-8B</th>
    <th rowspan="2">Aya-23-35B</th>
    <th colspan="4">Group 1 (Af, Da, Ni, De, Is, No, Sv)</th>
    <th colspan="4">Group 2 (Ca, Es, Gl, Lt, Ru)</th>
    <th colspan="4">Group 3 (Bg, Mk, Ru, Sr, Uk)</th>
    <th colspan="4">Group 4 (Fr, Id, Mg, Ms, Th, Vi)</th>
  </tr>
  <tr>
    <th colspan="4"></th>
    <th colspan="4"></th>
    <th colspan="4"></th>
    <th colspan="4"></th>
  </tr>
  <tr>
    <td>en→af</td><td>87.4/38.9</td><td>86.0/38.5</td><td>78.8/22.5</td><td>79.6/17.6</td><td>81.2/26.7</td><td>87.5/44.2</td><td></td><td></td><td></td><td>86.6/40.7</td><td></td><td></td><td></td><td>89.1/55.1</td><td></td><td></td><td></td><td>88.1/41.1</td><td></td><td></td><td></td><td>88.0/40.4</td>
  </tr>
  <tr>
    <td>en→da</td><td>90.0/44.5</td><td>88.6/38.2</td><td>87.6/34.2</td><td>76.4/19.3</td><td>82.9/29.0</td><td>91.8/48.6</td><td></td><td></td><td></td><td>91.1/42.8</td><td></td><td></td><td></td><td>89.0/36.1</td><td></td><td></td><td></td><td>88.0/38.2</td><td></td><td></td><td></td><td>87.8/35.1</td>
  </tr>
  <tr>
    <td>en→de</td><td>88.7/40.0</td><td>84.5/31.4</td><td>84.3/29.3</td><td>88.1/16.8</td><td>88.1/17.0</td><td>88.7/41.2</td><td></td><td></td><td></td><td>89.7/31.6</td><td></td><td></td><td></td><td>89.2/43.8</td><td></td><td></td><td></td><td>87.8/35.1</td><td></td><td></td><td></td><td>88.5/42.1</td>
  </tr>
  <tr>
    <td>en→fi</td><td>87.5/42.5</td><td>81.5/24.5</td><td>82.5/16.4</td><td>78.9/11.6</td><td>81.8/19.9</td><td>87.2/35.0</td><td></td><td></td><td></td><td>88.2/26.4</td><td></td><td></td><td></td><td>88.9/30.9</td><td></td><td></td><td></td><td>87.5/38.7</td><td></td><td></td><td></td><td>88.5/42.1</td>
  </tr>
  <tr>
    <td>en→no</td><td>88.9/33.0</td><td>87.8/28.0</td><td>87.5/26.9</td><td>77.3/15.7</td><td>82.4/21.1</td><td>90.6/35.0</td><td></td><td></td><td></td><td>89.9/30.9</td><td></td><td></td><td></td><td>90.7/37.6</td><td></td><td></td><td></td><td>90.0/37.6</td><td></td><td></td><td></td><td>90.0/37.6</td>
  </tr>
  <tr>
    <td>en→sv</td><td>90.7/44.3</td><td>89.1/38.7</td><td>86.9/31.3</td><td>78.3/20.8</td><td>83.7/28.8</td><td>91.7/47.0</td><td></td><td></td><td></td><td>91.1/41.8</td><td></td><td></td><td></td><td>90.1/48.5</td><td></td><td></td><td></td><td>90.1/48.5</td><td></td><td></td><td></td><td>90.1/48.5</td>
  </tr>
  <tr>
    <td>da→en</td><td>80.3/40.6</td><td>89.0/53.1</td><td>86.1/43.2</td><td>85.3/46.9</td><td>88.5/45.3</td><td>89.9/58.8</td><td></td><td></td><td></td><td>89.3/57.4</td><td></td><td></td><td></td><td>89.1/55.1</td><td></td><td></td><td></td><td>88.1/41.1</td><td></td><td></td><td></td><td>88.0/40.4</td>
  </tr>
  <tr>
    <td>da→es</td><td>83.0/34.4</td><td>89.6/45.3</td><td>89.2/42.4</td><td>87.7/42.6</td><td>89.7/47.3</td><td>90.2/49.6</td><td></td><td></td><td></td><td>90.2/48.9</td><td></td><td></td><td></td><td>90.3/48.7</td><td></td><td></td><td></td><td>88.8/28.2</td><td></td><td></td><td></td><td>87.8/35.1</td>
  </tr>
  <tr>
    <td>da→ru</td><td>81.3/28.6</td><td>88.8/40.5</td><td>88.5/39.7</td><td>89.3/43.9</td><td>89.5/41.5</td><td>89.6/45.7</td><td></td><td></td><td></td><td>89.7/43.8</td><td></td><td></td><td></td><td>89.2/43.8</td><td></td><td></td><td></td><td>87.8/35.1</td><td></td><td></td><td></td><td>88.8/28.2</td>
  </tr>
  <tr>
    <td>da→uk</td><td>61.2/16.2</td><td>85.5/32.5</td><td>82.7/27.2</td><td>68.0/13.0</td><td>78.7/14.5</td><td>87.1/37.7</td><td></td><td></td><td></td><td>86.7/35.9</td><td></td><td></td><td></td><td>86.7/35.7</td><td></td><td></td><td></td><td>87.7/36.6</td><td></td><td></td><td></td><td>87.6/35.7</td>
  </tr>
  <tr>
    <td>da→zh</td><td>81.9/38.3</td><td>87.1/80.1</td><td>86.9/37.1</td><td>85.3/37.9</td><td>87.2/38.9</td><td>87.6/40.2</td><td></td><td></td><td></td><td>87.3/38.7</td><td></td><td></td><td></td><td>87.6/38.7</td><td></td><td></td><td></td><td>86.7/35.7</td><td></td><td></td><td></td><td>86.7/35.7</td>
  </tr>
  <tr>
    <td>en→pt</td><td>80.7/32.1</td><td>88.5/41.8</td><td>88.1/39.5</td><td>86.5/38.5</td><td>88.5/42.3</td><td>89.1/45.7</td><td></td><td></td><td></td><td>88.9/43.9</td><td></td><td></td><td></td><td>88.9/44.5</td><td></td><td></td><td></td><td>89.0/37.6</td><td></td><td></td><td></td><td>89.0/37.6</td>
  </tr>
  <tr>
    <td>es→en</td><td>82.3/35.0</td><td>89.5/45.6</td><td>89.4/44.3</td><td>87.9/42.6</td><td>89.5/46.9</td><td>90.2/50.0</td><td></td><td></td><td></td><td>90.1/48.5</td><td></td><td></td><td></td><td>90.1/48.5</td><td></td><td></td><td></td><td>90.1/48.5</td><td></td><td></td><td></td><td>90.1/48.5</td>
  </tr>
  <tr>
    <td>it→en</td><td>87.3/31.7</td><td>83.8/24.9</td><td>86.7/32.7</td><td>87.2/17.2</td><td>84.2/25.3</td><td>88.4/39.0</td><td></td><td></td><td></td><td>87.8/35.1</td><td></td><td></td><td></td><td>88.6/29.5</td><td></td><td></td><td></td><td>88.9/44.5</td><td></td><td></td><td></td><td>88.8/46.4</td>
  </tr>
  <tr>
    <td>pt→en</td><td>86.7/42.3</td><td>89.1/46.3</td><td>88.7/43.8</td><td>89.7/49.7</td><td>89.5/49.1</td><td>90.7/51.0</td><td></td><td></td><td></td><td>89.3/47.7</td><td></td><td></td><td></td><td>89.4/48.4</td><td></td><td></td><td></td><td>89.4/48.4</td><td></td><td></td><td></td><td>89.4/48.4</td>
  </tr>
</table>

Table 25: Full results for Group 1–4 languages on Flores-200 benchmark. The performance of baselines is directly sourced from Xu et al. (2024b) and we keep the generation configuration of our approach the same as those.
<table>
  <tr>
    <th>Direction</th>
    <th>NLLB-3.3B</th>
    <th>LLaMAX3-Alpaca-8B</th>
    <th>Aya-101</th>
    <th>Aya-23-8B</th>
    <th>Aya-23-35B</th>
    <th>X-ALMA-13B (only SFT)</th>
    <th>X-ALMA-13B-DAT (Ours)</th>
    <th>X-ALMA-13B-DATM (Ours)</th>
  </tr>
  <tr>
    <th colspan="9">Group 5 (Cs, El, Hu, Lt, Lv, Pi)</th>
  </tr>
  <tr>
    <td>en→cs</td><td>91.0 / 32.2</td><td>88.1 / 24.6</td><td>90.0 / 26.7</td><td>91.1 / 30.5</td><td>91.4 / 32.2</td><td>91.5 / 33.8</td><td>91.3 / 30.9</td><td></td>
  </tr>
  <tr>
    <td>en→el</td><td>89.0 / 27.4</td><td>86.2 / 20.4</td><td>86.6 / 21.4</td><td>89.5 / 26.1</td><td>89.6 / 27.0</td><td>89.8 / 27.9</td><td>89.0 / 23.6</td><td></td>
  </tr>
  <tr>
    <td>en→hu</td><td>87.9 / 24.6</td><td>86.0 / 18.2</td><td>88.2 / 21.4</td><td>51.7 / 3.6</td><td>87.0 / 20.8</td><td>90.7 / 27.0</td><td>89.9 / 21.9</td><td></td>
  </tr>
  <tr>
    <td>en→lt</td><td>89.3 / 25.2</td><td>86.1 / 20.0</td><td>89.2 / 20.5</td><td>82.4 / 14.0</td><td>81.3 / 21.4</td><td>89.0 / 22.5</td><td>88.0 / 22.5</td><td></td>
  </tr>
  <tr>
    <td>en→lv</td><td>87.4 / 25.0</td><td>85.8 / 21.1</td><td>88.6 / 25.0</td><td>36.5 / 1.5</td><td>62.7 / 7.9</td><td>90.7 / 29.3</td><td>89.8 / 24.6</td><td></td>
  </tr>
  <tr>
    <td>en→pl</td><td>88.9 / 21.6</td><td>86.7 / 17.2</td><td>87.6 / 18.3</td><td>89.2 / 20.7</td><td>89.8 / 22.4</td><td>90.1 / 23.3</td><td>89.7 / 20.3</td><td></td>
  </tr>
  <tr>
    <th colspan="9">cs→en</th>
  </tr>
  <tr>
    <td>80.1 / 29.4</td><td>88.1 / 37.5</td><td>87.6 / 35.6</td><td>88.5 / 40.7</td><td>88.5 / 42.3</td><td>89.0 / 43.3</td><td>88.6 / 41.0</td><td>88.6 / 40.7</td><td></td>
  </tr>
  <tr>
    <td>el→en</td><td>86.1 / 33.0</td><td>87.5 / 34.2</td><td>86.5 / 32.1</td><td>87.8 / 36.1</td><td>88.3 / 39.0</td><td>87.9 / 38.0</td><td>87.5 / 35.6</td><td>87.5 / 35.6</td><td></td>
  </tr>
  <tr>
    <td>hu→en</td><td>70.1 / 14.0</td><td>87.8 / 32.5</td><td>86.4 / 29.9</td><td>81.1 / 23.0</td><td>86.5 / 32.2</td><td>88.7 / 37.3</td><td>88.0 / 34.6</td><td>88.2 / 35.0</td><td></td>
  </tr>
  <tr>
    <td>lt→en</td><td>69.1 / 12.6</td><td>86.0 / 31.0</td><td>85.8 / 30.2</td><td>80.6 / 24.6</td><td>83.4 / 32.9</td><td>87.1 / 36.9</td><td>86.5 / 33.8</td><td>86.3 / 32.8</td><td></td>
  </tr>
  <tr>
    <td>lv→en</td><td>88.1 / 34.4</td><td>87.0 / 37.7</td><td>87.0 / 36.2</td><td>83.1 / 19.1</td><td>87.9 / 39.2</td><td>87.6 / 38.0</td><td>87.6 / 37.3</td><td>87.6 / 37.3</td><td></td>
  </tr>
  <tr>
    <td>pl→en</td><td>77.8 / 20.3</td><td>85.6 / 28.3</td><td>85.6 / 28.0</td><td>86.1 / 30.5</td><td>86.7 / 33.3</td><td>86.5 / 32.8</td><td>86.1 / 31.1</td><td>86.2 / 31.3</td><td></td>
  </tr>
  <tr>
    <th colspan="9">Group 6 (El, Fl, Ja, Ko, Zh)</th>
  </tr>
  <tr>
    <td>en→et</td><td>90.5 / 25.0</td><td>87.7 / 18.1</td><td>90.7 / 21.9</td><td>40.5 / 1.5</td><td>57.8 / 6.1</td><td>91.6 / 26.4</td><td>90.8 / 21.9</td><td></td>
  </tr>
  <tr>
    <td>en→fi</td><td>91.7 / 24.1</td><td>89.3 / 17.5</td><td>90.3 / 18.9</td><td>51.9 / 2.4</td><td>70.0 / 8.1</td><td>92.7 / 25.3</td><td>92.3 / 22.8</td><td></td>
  </tr>
  <tr>
    <td>en→ja</td><td>87.9 / 22.6</td><td>89.0 / 27.5</td><td>89.0 / 27.3</td><td>90.8 / 30.7</td><td>91.0 / 30.9</td><td>91.2 / 34.6</td><td>90.4 / 31.9</td><td></td>
  </tr>
  <tr>
    <td>en→ko</td><td>84.6 / 14.8</td><td>78.6 / 9.6</td><td>85.3 / 11.3</td><td>43.3 / 0.4</td><td>47.6 / 2.0</td><td>87.6 / 14.0</td><td>87.0 / 12.6</td><td></td>
  </tr>
  <tr>
    <td>en→zh</td><td>88.4 / 12.5</td><td>85.6 / 8.8</td><td>87.4 / 10.2</td><td>89.0 / 13.1</td><td>89.4 / 12.8</td><td>89.3 / 15.0</td><td>88.1 / 10.1</td><td></td>
  </tr>
  <tr>
    <td>en→zh</td><td>82.0 / 32.4</td><td>85.6 / 36.3</td><td>82.4 / 27.3</td><td>87.3 / 40.2</td><td>87.5 / 37.3</td><td>88.2 / 43.6</td><td>87.5 / 40.0</td><td></td>
  </tr>
  <tr>
    <th colspan="9">Group 7 (Gu, Hi, Mr, Ne, Ur)</th>
  </tr>
  <tr>
    <td>en→gu</td><td>87.2 / 24.3</td><td>82.7 / 13.7</td><td>83.9 / 15.6</td><td>65.7 / 0.4</td><td>62.2 / 1.5</td><td>88.2 / 25.0</td><td>86.7 / 18.2</td><td></td>
  </tr>
  <tr>
    <td>en→hi</td><td>80.9 / 34.4</td><td>76.6 / 23.5</td><td>75.5 / 21.4</td><td>79.3 / 25.0</td><td>79.1 / 26.0</td><td>81.4 / 34.3</td><td>79.2 / 26.3</td><td></td>
  </tr>
  <tr>
    <td>en→mr</td><td>74.3 / 17.1</td><td>69.5 / 10.1</td><td>69.5 / 10.3</td><td>66.7 / 0.9</td><td>61.1 / 1.3</td><td>75.9 / 18.0</td><td>74.2 / 13.6</td><td></td>
  </tr>
  <tr>
    <td>en→ur</td><td>76.5 / 16.4</td><td>78.4 / 10.7</td><td>77.5 / 10.3</td><td>69.2 / 1.5</td><td>68.3 / 1.4</td><td>84.0 / 11.5</td><td>82.7 / 16.8</td><td></td>
  </tr>
  <tr>
    <td>en→ur</td><td>79.5 / 17.2</td><td>87.5 / 24.6</td><td>86.5 / 23.5</td><td>87.9 / 24.8</td><td>88.4 / 20.4</td><td>88.1 / 28.9</td><td>87.6 / 26.9</td><td>87.5 / 26.9</td><td></td>
  </tr>
  <tr>
    <td>en→ur</td><td>84.8 / 25.6</td><td>50.7 / 1.2</td><td>84.5 / 25.6</td><td>60.1 / 3.6</td><td>79.4 / 19.4</td><td>86.8 / 28.4</td><td>86.4 / 27.0</td><td>86.3 / 26.9</td><td></td>
  </tr>
  <tr>
    <td>en→ur</td><td>84.9 / 26.2</td><td>87.5 / 26.3</td><td>87.0 / 26.5</td><td>89.0 / 29.4</td><td>88.7 / 32.2</td><td>88.1 / 30.6</td><td>87.5 / 28.4</td><td>87.5 / 28.1</td><td></td>
  </tr>
  <tr>
    <td>en→ur</td><td>77.1 / 16.8</td><td>86.6 / 25.9</td><td>84.5 / 28.1</td><td>87.1 / 29.4</td><td>87.6 / 32.2</td><td>87.1 / 30.4</td><td>87.7 / 28.3</td><td>86.6 / 28.2</td><td></td>
  </tr>
  <tr>
    <th colspan="9">Group 8 (Ar, Az, Fa, He, Kk, Ky, Tr, Uz)</th>
  </tr>
  <tr>
    <td>en→ar</td><td>86.3 / 27.5</td><td>82.2 / 14.1</td><td>84.1 / 17.2</td><td>87.3 / 26.5</td><td>87.1 / 27.4</td><td>87.8 / 29.1</td><td>86.4 / 20.1</td><td></td>
  </tr>
  <tr>
    <td>en→fa</td><td>86.9 / 14.0</td><td>80.0 / 7.3</td><td>85.6 / 11.5</td><td>75.5 / 2.0</td><td>67.2 / 3.0</td><td>88.2 / 14.0</td><td>86.9 / 11.9</td><td></td>
  </tr>
  <tr>
    <td>en→he</td><td>86.5 / 22.6</td><td>84.5 / 17.7</td><td>86.4 / 19.1</td><td>87.7 / 23.2</td><td>87.6 / 23.8</td><td>88.5 / 28.4</td><td>86.7 / 22.0</td><td></td>
  </tr>
  <tr>
    <td>en→he</td><td>87.8 / 30.4</td><td>86.2 / 23.5</td><td>85.4 / 20.3</td><td>88.3 / 27.0</td><td>88.2 / 28.9</td><td>89.6 / 32.7</td><td>89.7 / 25.1</td><td></td>
  </tr>
  <tr>
    <td>en→he</td><td>89.6 / 30.6</td><td>86.0 / 27.2</td><td>89.1 / 30.2</td><td>85.1 / 14.7</td><td>85.0 / 14.7</td><td>89.7 / 32.2</td><td>89.7 / 17.5</td><td></td>
  </tr>
  <tr>
    <td>en→ky</td><td>88.1 / 13.2</td><td>82.9 / 7.9</td><td>86.6 / 10.4</td><td>62.9 / 1.2</td><td>49.6 / 0.9</td><td>88.5 / 13.2</td><td>87.4 / 11.5</td><td></td>
  </tr>
  <tr>
    <td>en→tr</td><td>89.7 / 29.0</td><td>84.2 / 13.8</td><td>88.3 / 21.1</td><td>89.9 / 23.7</td><td>88.7 / 23.6</td><td>90.3 / 27.7</td><td>88.5 / 21.6</td><td></td>
  </tr>
  <tr>
    <td>en→uz</td><td>89.8 / 18.6</td><td>74.5 / 6.8</td><td>88.6 / 12.0</td><td>46.5 / 0.5</td><td>37.1 / 0.3</td><td>90.0 / 16.8</td><td>86.8 / 11.0</td><td></td>
  </tr>
  <tr>
    <th colspan="9">ar→en</th>
  </tr>
  <tr>
    <td>86.1 / 38.2</td><td>86.8 / 35.1</td><td>85.8 / 35.0</td><td>87.9 / 41.5</td><td>87.6 / 43.4</td><td>87.5 / 41.2</td><td>86.6 / 38.3</td><td>86.5 / 36.9</td><td></td>
  </tr>
  <tr>
    <td>fa→en</td><td>77.5 / 15.1</td><td>70.0 / 7.9</td><td>85.2 / 21.5</td><td>75.6 / 10.6</td><td>82.6 / 17.9</td><td>86.7 / 25.8</td><td>85.7 / 22.2</td><td>86.1 / 23.1</td><td></td>
  </tr>
  <tr>
    <td>fa→en</td><td>83.5 / 29.8</td><td>87.6 / 33.1</td><td>87.3 / 32.8</td><td>87.9 / 36.8</td><td>88.5 / 39.6</td><td>88.7 / 37.6</td><td>87.7 / 32.1</td><td>87.4 / 32.2</td><td></td>
  </tr>
  <tr>
    <td>fa→en</td><td>88.0 / 31.4</td><td>87.3 / 24.6</td><td>86.4 / 21.4</td><td>88.1 / 33.6</td><td>88.3 / 46.6</td><td>88.1 / 34.5</td><td>87.8 / 19.9</td><td>87.6 / 19.0</td><td></td>
  </tr>
  <tr>
    <td>fa→en</td><td>85.0 / 30.2</td><td>86.2 / 29.0</td><td>86.1 / 29.2</td><td>84.3 / 15.5</td><td>74.0 / 14.2</td><td>87.8 / 33.5</td><td>87.0 / 30.4</td><td>87.0 / 29.4</td><td></td>
  </tr>
  <tr>
    <td>fa→en</td><td>81.6 / 20.1</td><td>84.5 / 20.4</td><td>83.0 / 20.4</td><td>64.3 / 4.2</td><td>74.3 / 11.3</td><td>85.4 / 23.5</td><td>84.5 / 21.0</td><td>84.6 / 21.5</td><td></td>
  </tr>
  <tr>
    <td>tr→en</td><td>75.3 / 16.8</td><td>88.6 / 33.4</td><td>88.1 / 33.2</td><td>88.2 / 35.8</td><td>89.6 / 39.3</td><td>89.6 / 39.9</td><td>88.8 / 35.9</td><td>88.5 / 33.0</td><td></td>
  </tr>
  <tr>
    <td>uz→en</td><td>60.7 / 5.3</td><td>86.1 / 27.9</td><td>84.9 / 28.1</td><td>61.3 / 3.9</td><td>75.9 / 15.3</td><td>86.9 / 32.2</td><td>84.2 / 25.3</td><td>83.5 / 24.1</td><td></td>
  </tr>
</table>

Table 26: Full results for Group 5-8 languages on Flores-200 benchmark. The performance of baselines is directly sourced from Xu et al. (2024b) and we keep the generation configuration of our approach the same as those.

Reviewer paper summaries: This paper focuses on multilingual machine translation (MMT) based on large language models (LLMs). The emergence of LLMs has advanced MMT, but the Curse of Multilinguality (CoM) remains a challenge. Existing approaches often rely on scaling up training and computation, raising the question of whether there is a more efficient solution.

The authors analyze linguistic conflicts and synergy during the post-training phase. They discover an asymmetric phenomenon in linguistic conflicts and synergy, where their dominance varies in different translation directions. This asymmetry affects the performance of existing post-training methods. Also, they find that the bottleneck in MMT lies more in post - training rather than multilingual pre-training.

Based on these findings, the paper proposes a direction-aware training approach combined with group-wise model merging. This method is designed to address the asymmetry explicitly. Experiments on Flores200 and WMT23 benchmarks show that the proposed approach can achieve high translation performance with fewer pre-training tokens and model parameters, providing an efficient way to build MMT systems. However, the approach has limitations, such as not surpassing state-of-the-art methods in performance in some directions, and lacking a deeper analysis of the asymmetry mechanism.

This paper explores the asymmetric conflict and synergy in post-training for multilingual machine translation with LLMs, identifying it as a key factor in the Curse of Multilinguality. The authors propose a Direction-Aware Training (DAT) approach combined with group-wise model merging to address this issue. DAT uses separate training for XX→En directions and group multilingual training for En→XX directions to mitigate conflicts and enhance synergy. Experiments show that this method achieves comparable performance to existing models with fewer pre-training tokens and parameters, offering a more efficient solution for high-quality MMT.

This paper systematically investigates linguistic conflicts and synergy during post-training when training LLMs for multilingual machine translation tasks. The authors find an asymmetric effect in linguistic conflicts and synergy and provide an in-depth analysis of such phenomena. They propose a direction-aware training approach, experimental results show the effectiveness of the proposed method.
Summary of strengths: 1. Identifies an asymmetry in linguistic conflicts and synergy during post-training in LLM-based MMT. Conflicts dominate in XX→En translations, while synergy dominates in En→XX, challenging traditional training assumptions.
2. Finds that the bottleneck in MMT is in post-training rather than pre - training, contrary to common belief.
3. Proposes a direction - aware training approach with group-wise model merging. It addresses the asymmetry, using fewer pre - training tokens and parameters to achieve comparable performance.
4. Validates the approach on Flores200 and WMT23 benchmarks, showing it outperforms previous models and offers an efficient MMT solution.

1. The paper proposes a novel DAT method to address asymmetric conflicts and synergy in MMT, improving efficiency over existing approaches.

2. The method achieves comparable performance to state-of-the-art models with fewer pre-training  tokens and parameters, enhancing cost-effectiveness.

3. The authors provide a comprehensive analysis of the asymmetric phenomenon in linguistic conflicts and synergy, highlighting its impact on different translation directions and language groups.

4. The introduction of group-wise model merging further boosts efficiency by reducing LoRA weights while maintaining translation quality.

5. Experiments on Flores-200 and WMT23 benchmarks demonstrate the effectiveness of the DAT approach in advancing multilingual machine translation.

1. This paper is very well-written and easy to follow.
2. Extensive analyses are conducted to show the asymmetric effect in linguistic conflicts and synergy when learning LLMs for multilingual machine translation.
3. Extensive experimental results are provided to show the effectiveness of the proposed method.
Summary of weaknesses: 1. The direction-aware strategy proposed in the paper requires separate model training and then model merging, leading to a longer process, which is not conducive to practical implementation.
2. The core contributions of the paper, separate training and group-wise merging, are not original techniques. They can be regarded as the application of mature technologies in the translation field, so the innovation is somewhat insufficient.
3. Although the paper analyzes multilingual conflicts and synergies, most of the analysis is based on results. It does not conduct in - depth analysis from the aspects of model principles and deeper levels, thus contributing limitedly to the industry.

1. The effectiveness of method for non-English translation directions is unclear and requires further clarification.

2. The test set size is relatively small and may lack diversity, potentially limiting the generalizability of the results.

1.  The asymmetric effect in linguistic conflicts and synergy is concluded based on the experimental result, simply using the existing high-quality sentence pairs to finetune the base models. The authors do not conduct an in-depth analysis on why such a phenomenon occurs. Given the fact that most of the high-quality sentence pairs are from FLORES-200 dev and NTREX-128, is it possible that the same English sentences are copied multiple times in the sft dataset, and the bad xx -> en performance is simply due to the overfitting issue on the English sentences?
2. The novelty of the direction-aware training approach is limited. The authors adopt the existing methods in their settings.
Comments, suggestions and typos: NA

1. The paper focuses on XX-En and En-XX translation directions. Explore the effectiveness of the DAT approach for non-English-centric translation pairs, such as translating between two low-resource languages.

2. While the paper identifies the asymmetric conflicts and synergy in multilingual translation, it lacks in-depth theoretical analysis. Future work could delve deeper into the underlying mechanisms causing this asymmetry and explore more refined strategies to address it.

N/A
Overall assessment scores: 3

3

3
Metareview score: 3
Metareview: This paper systematically investigates linguistic conflicts and synergy when post-training LLMs for multilingual machine translation. The authors demonstrate an asymmetric effect in linguistic conflicts and synergy and provide extensive empirical results of such phenomena. They propose a direction-aware training approach, and experiments on the FLORES-200 and WMT-23 benchmarks show the effectiveness of the proposed method.
Summary of reasons to publish: 1. Extensive analyses are conducted to show the asymmetric effect in linguistic conflicts and synergy when learning LLMs for multilingual machine translation. 
2. Extensive experimental results are provided to show the effectiveness of the proposed method.
3. This paper is very well-written and structured.