Venue: NAACL 2025

# Multilingual Machine Translation with Open Large Language Models at

Practical Scale: An Empirical Study

 Menglong Cui\({}^{*}\), Pengzhi Gao\({}^{*}\), Wei Liu, Jian Luan, Bin Wang

Xiaomi Inc., Beijing, China

cuimenglongcs@gmail.com {gaopengzhi,liuwei40,luanjian,wangbin11}@xiaomi.com

###### Abstract

Large language models (LLMs) have shown continuously improving multilingual capabilities, and even small-scale open-source models have demonstrated rapid performance enhancement. In this paper, we systematically explore the abilities of open LLMs with less than ten billion parameters to handle multilingual machine translation (MT) tasks. We conduct comprehensive evaluations on six popular LLMs and find that models like Gemma2-9B exhibit impressive multilingual translation capabilities. We then introduce the **F**arallel-**F**irst **M**onolingual-**S**econd (PFMS) data mixing strategy in the continual pretraining stage to further enhance the MT performance and present GemmaX2-28, a 9B model achieving top-tier multilingual translation performance across 28 languages1. Specifically, GemmaX2-28 consistently outperforms the state-of-the-art (SOTA) models such as TowerInstruct Alves et al. (2024) and X-ALMA Xu et al. (2024b) and achieves competitive performance with Google Translate and GPT-4-turbo. 2

Footnote 1: Applied Mathematics Subject Classification: _Keywords:_ (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associations for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443

+
Footnote †: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, (_Volume 1: Long Papers_), pages 5420–5443


We find that open LLMs like Gemma2-9B exhibit remarkable multilingual translation capabilities but still fall behind strong closed-source models. We then systematically investigate the most effective methods to leverage monolingual and parallel data for further boosting Gemma2-9B's capabilities and propose the Parallel-First Monolingual-Second (PFMS) data mixing strategy. Leveraging PFMS strategy when continually pretraining on Gemma2-9B followed by instruction finentuning on a small set of high-quality translation pairs, we learn GemmaX2-28-9B, a many-to-many multilingual MT model that performs competitively with Google Translate and GPT-4-turbo across \(28\) widely spoken languages. The contributions of this paper can be summarized as follows:

* We benchmark the latest open-source LLMs on multilingual MT tasks in \(28\) widely spoken languages, covering English-centric and Chinese-centric translation directions.
* We systematically explore the optimal training recipe for boosting multilingual MT with LLMs and introduce the Parallel-First Monolingual-Second (PFMS) data mixing strategy for the continual pretraining stage.
* We publicly release GemmaX2-28-9B, the many-to-many multilingual translation model supporting \(28\) languages, which consistently outperforms open-source alternates and is competitive with closed-source models such as Google Translate and GPT-4-turbo.

## 2 Related Work

### Multilinguality in LLMs

Large language models have gained substantial attention from the research community due to their outstanding performance on various tasks Winata et al. (2021); Wei et al. (2022); Touvron et al. (2023). However, most advancements in LLMs have primarily focused on English or Chinese, resulting in inadequate performance in low-resource languages Ebrahimi et al. (2022); Asai et al. (2024). The reason is that the language distribution of the training data for LLMs is highly imbalanced and the quality varies across languages Ding et al. (2024). Numerous efforts have been made to enhance the multilingual capabilities of LLMs. Some studies seek to enhance the performance of LLMs on low-resource languages through continual pretraining Cui et al. (2024) or supervised finetuning Ustun et al. (2024) using data from these languages. Additionally, some researchers Li et al. (2024) leverage contrastive learning to align the internal representations of different languages, allowing the model to improve its cross-lingual capabilities with minimal training data. In our study, we assessed the latest open-source LLMs on translation tasks across \(28\) languages. We find that these latest models still show significant performance disparities across different languages, indicating that low-resource languages continue to pose a significant challenge for these models. Therefore, we expand the Gemma2 model to cover \(28\) commonly used languages, enabling it to demonstrate strong generative and translation capabilities across these languages.

### Multilingual MT with LLMs

The use of LLMs has shown significant progress in MT tasks Lu et al. (2024); Xu et al. (2024); Li et al. (2024); Gao et al. (2024), leading to different approaches on LLM-based translation. One line of work focuses on the in-context translation capabilities, and LLMs are provided with parallel sentences to guide the model in generating the target sentence. Some studies Agrawal et al. (2023); Zhu et al. (2024); Cui et al. (2024) show that MT performance can be improved by using semantically related parallel sentences as examples, exhibiting promising results in scenarios with limited computational resources and insufficient parallel data.

Another line involves finetuning with translation instructions Xu et al. (2024) initially pretrained the base model on monolingual data, followed by finetuning on small human-written parallel datasets, which resulted in strong translation performance. Guo et al. (2024) challenge the perspective that the impact of parallel data is reduced in the LLM era and demonstrate the effectiveness of parallel data during continual pretraining on enhancing multilingual translation. However, the distribution of the parallel datasets could be highly imbalanced across different languages for massively multilingual translation scenarios. Alves et al. (2024) incorporates a fixed multilingual mixture of monolingual (two-thirds) and parallel (one-third) data, further enhancing LLMs' translation capabilities. Our work differs in that we primarily focus on exploring the optimal mixing strategy of monolingual and parallel data during continual pretraining to achieve the best translation performance 



## 3 Datasets and Baseline Settings

In this section, we describe the evaluation datasets used in Sections 4 and 5 as well as the model configurations.

### Datasets

We conduct experiments on \(28\) languages across a broad linguistic spectrum. The detailed information of all languages is summarized in Table 3. We evaluate the multilingual translation performance on the FLORES-200 Team et al. (2022) benchmark. To avoid data leakage issues, we also consider the WMT-24 test sets for evaluating multilingual translation capabilities. Only sources are provided in the WMT-24 benchmark, and we adopt English sentences for reference-free evaluation.

### Models

We evaluate the translation performance of six open-source LLMs: Mistral-7B-v0.3 Jiang et al. (2023), Qwen2/2.5-7B Yang et al. (2024), LLaMA3/3.1-8B Dubey et al. (2024) and Gemma2-9B Team et al. (2024). We also summarize the results of several SOTA models as follows:

* Google Translate: The translation performance by leveraging the Google Translate API3. We include it to represent the commercial MT system. Footnote 3: [https://translate.google.com/](https://translate.google.com/)
* GPT-3.5/4 Turbo: The translation performance by leveraging the OpenAI API4 with greedy decoding for inference. Notably, we adopt the same in-context learning strategy as that employed by open-source LLMs. Footnote 4: [https://api.openai.com/vl/chat/completions](https://api.openai.com/vl/chat/completions)
* NLLB-54.5B: The largest and best multilingual NMT model with encoder-decoder architecture released by the No Language Left Behind (NLLB) project Team et al. (2022).

### Evaluation

For the FLORES-200 benchmark, we evaluate MT performance by spBLEU5Goyal et al. (2022) and COMET6Rei et al. (2020). For the WMT-24 benchmark, we adopt two reference-free models, XCOMET7Guerreiro et al. (2023) and COMETKiwi8Rei et al. (2023), each of which has \(10\)B parameters and demonstrates high correlation with human judgments Freitag et al. (2023).

Footnote 5: We calculate the spBLEU scores via sacrBELU Post (2018) with the Flores2E0t tokenizer.

Footnote 6: [https://huggingface.co/Unbabel/wmt22-comet-da](https://huggingface.co/Unbabel/wmt22-comet-da)

Footnote 7: [https://huggingface.co/Unbabel/](https://huggingface.co/Unbabel/)

## 4 Benchmarking Open LLMs for Multilingual Machine Translation

In this section, we report empirical results on tokenizer efficiency and multilingual MT performance across various open-source LLMs.

### Tokenizer Efficiency

The open-source LLMs are typically pretrained on one or a few dominant languages, and tokenization on low-resource languages usually results in undesirable long sequences of subwords which leads to excessive GPU memory consumption during training and slow decoding speed during inference.

Following the line of Liao et al. (2024), we compare the length differences between tokenized English sentences and their corresponding non-English counterparts to evaluate the tokenizer efficiency across different LLMs. Specifically, we define the length ratio as follows:

\[{\text{length~{}ratio}}=\frac{{\text{length}}({\text{tokenizer}}(y))}{{ \text{length}}({\text{tokenizer}}(x))}, \tag{1}\]

where \(x\) represents the English sentence, and \(y\) denotes the corresponding non-English sentence. A smaller length ratio is preferred, meaning that the tokenizer encodes the non-English sentences as efficiently as the English sentence.

We conduct our experiments on the FLORES-200 devtest dataset, which includes \(1012\) sentences for each language. Note that LLaMA3 and LLaMA3.1 share the same tokenizer, as do Qwen2 and Qwen2.5. We also include the tokenizer of the NLLB-54.5B model as a strong baseline for comparison. The averaged length ratio for each language is illustrated in Figure 1. By checking the length ratio for various languages, we have the following observations: 1) Due to its extensive support and optimization for more than \(200\) languages, NLLB-54.5B has the most balanced tokenizer and achieves low length ratios across different languages. 2) Open-source LLMs demonstrate notably high length ratios for low-resource languages such as Khmer, Lao, and Burmeese. Gemma2-9B consistently achieves better length ratios for almost all languages compared with other LLMs.





### In-context Multilingual Translation Performance with Open LLMs

We here investigate the multilingual translation performance of different LLMs with in-context learning strategies on the FLORES-200 and WMT-24 benchmarks. For each open LLM, we report its translation performance across \(28\) languages with five randomly selected translation pairs9 from the FLORES-200 development dataset as the context exemplars. Note that we also adopt the FLORES-200 development dataset for the WMT-24 benchmark. Following the line of Zhu et al. (2024b), we adopt the format "<X>=<Y>" as the in-context template, where <X> and <Y> denote the source and target sentences of the select parallel sentence pairs. All experiments are conducted based on OpenICL10Wu et al. (2023).

Footnote 9: We apply the same five randomly selected translation pairs as exemplars for each direction during evaluation.

Footnote 10: [https://github.com/Shark-NLP/OpenICL](https://github.com/Shark-NLP/OpenICL)

We report the averaged multilingual translation performance on the WMT-24 and FLORES-200 benchmarks in Tables 1. By checking the translation performance for various languages and models, we have the following observations:

* Open LLMs demonstrate impressive multilingual translation capability. Specifically, Gemma2-9B performs the best among the open LLMs evaluated and outperforms GPT-3.5-turbo on average.
* Open LLMs still lag behind the strong supervised NMT models such as NLLB-54.5B, especially in low-resource languages.
* Google Translate exhibits remarkable multilingual translation performance and even outperforms GPT-4-turbo on both the FLORES-200 and WMT-24 benchmarks.
* We do not observe serious data leakage issues on public datasets (FLORES-200) for the open LLMs we evaluated, where the translation performance on both WMT-24 and FLORES-200 benchmarks share a similar trend.

We then illustrate the translation performance on the FLORES-200 benchmark according to varying numbers of in-context exemplars in Figure 2. We can see that the COMET scores increase dramatically with the number of exemplars increasing from \(1\) to \(5\), and the translation performance plateaus afterward except for Mistral-7B-v0.3 which has relatively weak multilingual capability.

Figure 1: The tokenizer efficiency of open-source LLMs for each non-English language. The smaller the length ratio is, the more efficient the tokenizer is.





## 5 GemmaX: Boosting Multilingual Translation with Gemma Models

Given its impressive multilingual capabilities discussed in Section 4, we select Gemma2-9B as our backbone model for learning many-to-many multilingual machine translation across languages. We continue the pretraining of Gemma2-9B with multilingual corpora and finetune the continual pretrained model using a small but high-quality parallel dataset with the translation prompt as follows: Translate this from [_source language_] to [_target language_]:n[_source language_]:<_source sentence>_u[_target language_]:<_target sentence>_.

### Pretraining Data

Monolingual DataWe collect monolingual data from CulturaX Nguyen et al. (2024) and MADLAD-400 Kudugunta et al. (2023). CulturaX contains \(6.3\) trillion tokens in \(167\) languages, which undergo meticulous cleaning and deduplication through a comprehensive pipeline. MADLAD-400 is a multilingual dataset with \(3\) trillion tokens in \(419\) languages, on which the authors perform a self-audit to guarantee the data quality.

Parallel DataWe collect all Chinese-centric and English-centric parallel datasets from the OPUS collection11Tiedemann (2012) up to August 2024, which is comprised of multiple corpora, ranging from movie subtitles Tiedemann (2016) to Bible Christodouloupoulos and Steedman (2015) to web crawled datasets El-Kishky et al. (2020); Schwenk et al. (2021). We download all available corpora and concatenate them without curating the datasets or trying to balance the representation of different domains. After collecting all parallel datasets, we adopt the data-cleaning process as follows: 1) We remove duplicate sentence pairs and discard sentence pairs by utilizing some heuristic approaches. 2) Language identification filtering is applied by utilizing the fastText toolkit Joulin et al. (2016, 2017). 3) Semantic similarity filtering is performed based on LaBSE Feng et al. (2022) and MuSR Gao et al. (2023). Specifically, we only keep the sentence pairs that have similarity scores between \(0.75\) and \(0.99\). After the cleaning process, we have about 3.4 billion cleaned Chinese-centric and English-centric sentence pairs covering 28 languages. The distribution of our parallel datasets for each language is illustrated in Figure 3.

Footnote 11: [http://www.opus.nlpl.eu](http://www.opus.nlpl.eu)

### Supervised Finetuning Data

Inspired by Xu et al. (2024a) that a small amount of high-quality data could dramatically boost the translation performance, we construct our finentuning dataset mostly from human-annotated records: 1) We extract the sentence pairs covering ten languages from the general translation task in the TowerBlock dataset12. 2) For the languages not covered in TowerBlock, we randomly sample \(1000\) English-centric sentence pairs for each language either from the NTREX-128 Federmann et al. (2022) and FLORES-200 dev datasets or the OPUS dataset bidirectionally filtered by _wmt23-comettiwi-da-xrl_ model with quality scores above \(0.85\). 3) To mitigate the off-target issue and enhance the zero-shot translation performance, inspired by Wu et al. (2024), we also randomly select \(25\) non-English-centric translation directions and sample \(100\) sentence pairs for each direction from the NTREX-128 and FLORES-200 dev datasets. In summary, the total number of sentence pairs is around \(196\)K, where there are about \(189\)K in the English-centric directions and \(7\)K in other directions.

Figure 2: MT performance on the FLORES-200 benchmark with different numbers of in-context exemplars.





### Training Configuration

We train all models with the LLaMA-Factory (Zheng et al., 2024) framework for one epoch on \(32\) and \(8\) NVIDIA H800 GPUs for the pretraining and finetuning stages respectively.

In the pretraining stage, we utilize an effective batch size of 1.57 million tokens, corresponding to a batch size of \(4\) per GPU with a gradient accumulation of \(6\) and a maximum sequence length of \(2048\). We use the AdamW optimizer with a learning rate of 2e-5. We adopt full-weight pretraining with bf16 precision by employing a cosine learning rate scheduler. In the finetuning stage, we adopt a batch size of \(4\) per GPU with a gradient accumulation of \(8\) and a maximum sequence length of \(2048\). Additionally, we employ an inverse square root learning rate scheduler. Detailed training configurations can be found in Tables 7 and 8. We use greedy decoding to generate translations.

### Exploring the Best Data Recipe for Multilingual Translation with LLMs

Xu et al. (2024a) conduct continual pretraining solely on monolingual data followed by instruction finetuning on a small number of high-quality translation pairs, achieving impressive translation results. They conclude that the reliance on parallel data is diminished in the era of LLMs. Subsequently, Guo et al. (2024) reemphasize the importance of parallel data and propose a three-stage training strategy: 1) continual pretraining with monolingual data; 2) continual pretraining with parallel data; 3) finetuning with source language consistent translation instructions. However, as illustrated in Figure 3, the parallel data adopted in the second stage could be highly imbalanced for massively multilingual settings, which would weaken the model's understanding and generative capabilities on low-resource languages learned in the first stage. Alves et al. (2024) utilize a multilingual mixture of monolingual (two-thirds) and parallel (one-third) data during continual pretraining, which significantly improves the translation performance of their model. In our experiments, we follow the idea of a multilingual mixture in the continual pretraining stage and raise two questions: 1) _whether a monolingual to parallel ratio of 2:1 is the best data recipe for multilingual machine translation? 2) If not, what is the best data recipe?_

Therefore, we consider the following configurations when preparing the pretraining dataset: 1) monolingual data only, 2) \(2:1\) ratio of monolingual to parallel data, 3) \(1:1\) ratio, 4) \(1:2\) ratio, and 5) parallel data only. For each language, we collect two billion tokens, allocating them between monolingual and parallel sentences based on the specified ratios mentioned above. For each parallel sentence, we construct a new sentence in either <_source sentence>_n<_target sentence>_ or <_target sentence>_n<_source sentence>_ manner, with the order of the source and target sentences determined randomly. We randomly intersperse the monolingual and parallel data. Due to the inherent characteristics of low-resource languages, the corresponding monolingual or parallel data may be insufficient, resulting in the total number of tokens in the pretraining dataset falling short of \(56\) billion. As a result, we construct five pretraining datasets corresponding to these configurations.

We continually pretrain the Gemma2-9B model using five datasets containing varying proportions of monolingual and parallel data, followed by translation instruction finetuning on our high-quality datasets. Based on Table 3, we classify the \(28\) languages into high-resource (\(18\)), mid-resource (\(7\)), and low-resource (\(3\)) languages according to their resource availability. We then report the translation performance during continual pretraining on these three language groups in Figures 4 and 5.

By checking model performance under different

Figure 3: Number of sentences in different languages for Chinese-centric and English-centric parallel dataset.



combinations of data recipes and language groups, we have the following observations:

* The current prevalent training paradigm, which involves continual pretraining on large-scale monolingual data, proves to be suboptimal. While this approach improves the overall performance across various languages, it does so at the cost of translation quality for high-resource languages, particularly in the xx\(\rightarrow\)en directions.
* Incorporating parallel data into the pretraining stage, regardless of the volume, significantly enhances the model's overall translation capabilities. Notably, the translation performance for high-resource language directions improves as the quantity of parallel data increases. However, such improvement was not evident for low-resource and mid-resource languages.
* We believe that the base model has sufficient understanding and generation capabilities for high-resource languages. It mainly needs parallel data during pretraining to align the internal representations across different languages. In contrast, the model's generative and comprehension abilities for low-resource and mid-resource languages are not up to par. To improve its performance with these languages, it's crucial to include more data in the pretraining process to enhance its generative capacities. Unfortunately, due to the nature of low-resource and mid-resource languages, we are unable to obtain enough parallel data. Therefore, we think it's essential to incorporate monolingual data into the pretraining dataset for low-resource and mid-resource languages.

### Main Result

Based on the experimental results in Section 5.4, we propose a Parallel-First Monolingual-Second (PFMS) data mixing strategy, where we give higher priority to parallel data than monolingual data when preparing the continual pretraining dataset. Specifically, for 2 billion tokens per language, we utilize parallel data as much as possible and supplement it

Figure 4: The translation performance (COMET) of models trained with different data recipes during continual pretraining on low-resource (left), mid-resource (middle), and high-resource (right) languages. The upper subfigures illustrate the en\(\rightarrow\)xx translation performance, while the lower subfigures depict the xx\(\rightarrow\)en translation performance. Note that “Gemma2-9B” refers to the direct finetuning of the model without continual pretraining, and its performance is reflected in the right-hand y-axis. The translation performance in BLEU scores is illustrated in Figure 5.

 

with monolingual data if needed. The detailed information on the number of training tokens across different languages adopted in the PFMS strategy is summarized in Table 9. The translation performance by leveraging the PFMS strategy is also illustrated in Figures 4 and 5. We can see that the PFMS strategy consistently outperforms other data mixing strategies except for high-resource languages to English directions. Such a phenomenon might be due to the fact that the dominant part of the parallel-only dataset is the English-centric corpora with high-resource languages.

We continually pretrain Gemma2-9B with PFMS strategy and learn GemmaX2-28-9B by finetuning the pretrained model with high-quality translation pairs covering \(28\) languages. Furthermore, to investigate the impact of our PFMS strategy on models of different scales, we also conduct experiments on the Gemma2-2B model. The experimental results are summarized in Table 2. Besides the models discussed in Section 3.2, we include several of the strongest open-source multilingual models as baselines, including:

* TowerInstruct-7/13B Alves et al. (2024): 7B and 13B LLaMA2-based models for translation-related tasks supporting \(10\) languages.
* Aya-23-8B / Aya-Expanse-8B Dang et al. (2024): 8B models with highly advanced multilingual capabilities, supporting 23 commonly used languages. Note that we only evaluate their multilingual translation performance on languages overlapping with GemmaX2.
* X-ALMA Xu et al. (2024b): a LLaMA2-based multilingual MT model covering \(50\) languages, which consists of a 13B dense model and multiple language-specific modules, with a total of approximately \(29\)B parameters. Note that X-ALMA only supports English-centric directions, and we report the translation performance for the overlapping directions.
* Aya-101 Ustun et al. (2024): a 13B massively multilingual generative language model that follows instructions in \(101\) languages.
* LLaMAX3-Alpaca-8B Lu et al. (2024): an 8B LLaMA3-based model supporting \(102\) languages with powerful multilingual capabilities without loss instruction-following capabilities.

Despite having limited parameters and supporting a large number of languages, GemmaX2-28-9B consistently outperforms the current SOTA open-source models, demonstrating the effectiveness of our proposed PFMS data mixing strategy and the powerful translation capabilities of our model. In addition, our model achieves comparable translation performance to GPT-4-turbo and Google

\begin{table}
\begin{tabular}{l|c|c c c c} \hline Model & WMT-24 & \multicolumn{4}{c}{FLORES-200} \\  & en \(\rightarrow\)xx & en \(\rightarrow\)xx & xx \(\rightarrow\)en & zh \(\rightarrow\)xx & xx \(\rightarrow\)zh \\ \hline \hline \(10\)**languages** & & & & & \\ TowerInstruct-7B-v0.2 & 84.48 / 75.02 & 38.91 / 88.46 & 42.20 / 88.29 & 23.36 / 85.21 & 26.87 / 85.69 \\ TowerInstruct-13B-v0.1 & 86.10 / 76.74 & 40.60 / 88.89 & 43.14 / 88.50 & 25.28 / 86.13 & 29.32 / 86.59 \\ GemmaX2-28-2B & 84.50 / 74.42 & 40.15 / 88.56 & 42.03 / 88.27 & 25.71 / 85.85 & 31.91 / 87.04 \\ GemmaX2-28-9B & **86.59 / 77.25** & **42.58 / 89.15** & **43.95 / 88.62** & **28.55 / 86.84** & **34.10 / 87.70** \\ \hline \hline \(20\)**language** & & & & & \\ Aya-23-8B & 77.72 / 69.55 & 34.95 / 86.81 & 39.56 / 86.57 & 23.68 / 84.28 & 25.33 / 85.15 \\ Aya-Expanse-8B & 80.15 / 74.11 & 37.57 / 87.89 & 40.48 / 87.04 & 25.39 / 85.48 & 28.21 / 86.30 \\ GemmaX2-28-2B & 80.66 / 73.07 & 39.53 / 88.51 & 42.77 / 88.44 & 25.57 / 85.54 & 31.69 / 86.81 \\ GemmaX2-28-9B & **82.95 / 75.82** & **42.27 / 89.21** & **45.23 / 88.92** & **28.68 / 86.67** & **34.43 / 87.59** \\ \hline \hline \(23\)**languagues** & & & & & \\ X-ALMA & 81.67 / 74.73 & 39.31 / 88.69 & 42.09 / 88.54 & - & - \\ GemmaX2-28-2B & 79.83 / 72.77 & 39.32 / 88.27 & 42.72 / 88.46 & 25.54 / 85.24 & 31.42 / 86.73 \\ GemmaX2-28-9B & **82.05 / 75.31** & **41.98 / 88.97** & **45.27 / 88.97** & 28.64 / 86.36 & 34.26 / 87.56 \\ \hline \hline \(28\)**linguags** & & & & & \\ Aya-101 & 71.67 / 63.88 & 27.70 / 84.80 & 33.84 / 86.21 & 17.36 / 81.56 & 18.70 / 81.72 \\ LLAMAX3-Apaca-8B & 67.29 / 60.21 & 28.11 / 83.12 & 35.81 / 87.05 & 18.17 / 80.78 & 21.08 / 83.38 \\ GPT-3.5-turbo & 70.63 / 62.64 & 32.24 / 82.49 & 36.18 / 85.63 & 20.35 / 79.34 & 24.12 / 82.61 \\ NLLB-54.5B & & 37.34 / 87.05 & 43.63 / 88.32 & 25.41 / 84.42 & 20.73 / 80.72 \\ GPT-4-turbo & 77.55 / 70.68 & 37.01 / 87.14 & 41.53 / 88.40 & 24.66 / 84.31 & 28.80 / 85.90 \\ Google Translate & 77.64 / 73.00 & **41.52 / 88.51** & **45.35 / 88.83 & **29.64 / 85.69** & **34.86 / 87.20 \\ GemmaX2-28-2B & 77.20 / 71.99 & 37.00 / 87.54 & 42.16 / 88.32 & 24.27 / 84.44 & 30.59 / 86.41 \\ GemmaX2-28-9B & **79.37 / 74.41** & 39.72 / 88.35 & 45.07 / **88.95** & 27.48 / **85.69** & 33.74 / **87.38** \\ \hline \end{tabular}
\end{table}
Table 2: Translation performance on WMT24 (XCOMET / COMETKiwi) and FLORES-200 (spBLEU / COMET) benchmarks. The detailed results are summarized in Tables 10, 11 and 12.

 

Translate, indicating that our model has achieved translation capabilities on par with industry standards. We also observe that GemmaX2-28-2B demonstrates strong multilingual translation capabilities with minimal model parameters, indicating the effectiveness of our PFMS data mixing strategy for models with varying parameter scales.

## 6 Conclusion

In this paper, we systematically evaluate the multilingual in-context translation of the latest open LLMs at a practical scale. While these models demonstrate strong translation capabilities, they still fall short compared to closed-source models. Furthermore, we explore the best data recipe for multilingual translation with LLMs and propose a Parallel-First Monolingual-Second (PFMS) data mixing strategy. Leveraging the PFMS strategy during continual pretraining on Gemma2-9B, we achieve significant improvements, bringing its translation performance to a level comparable with GPT-4-turbo and Google Translate. To the best of our knowledge, GemmaX2-28-9B is the open model with the highest translation quality. We aim to develop models that support a broader range of languages and possess enhanced translation capabilities as part of our future work.

### Limitations

Due to limited computational resources, we only conduct multilingual in-context translation evaluation and explore the optimal data mixing strategy on open LLMs with parameter sizes below ten billion. The translation performance and optimal data recipe for larger models remain unclear.

## References

* S. Agrawal, C. Zhou, M. Lewis, L. Zettlemoyer, and M. Ghazvininejad (2023)Incre context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 8857-8873. External Links: Link Cited by: SS1.
* E. Almazrouei, H. Alobeidli, A. Albashami, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, P. Pannier, and G. Penedo (2023)The falcon series of open language models. Preprint arXiv:2311.16867. External Links: Link Cited by: SS1.
* D. Miguel Alves, J. Fombal, N. M. Guerreiro, P. Hartinga, J. Alves, A. Farajian, B. Peters, R. Rei, P. Fernandes, S. Agrawal, P. Colombo, J. G. C. de Souza, and A. Martins (2024)Tower: an open multilingual large language model for translation-related tasks. In First Conference on Language Modeling, Cited by: SS1.
* R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Leplikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, Laurent El Shafey, Y. Huang, K. Meier-Hellstern, G. Mishra, E. Moreira, M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. Hernandez Abrego, J. Ahn, J. Austin, P. Barham, J. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepry, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Diaz, N. Du, E. Dyer, V. Feinberg, F. Feng, V. Frenber, M. Freitag, X. Garcia, S. Germmann, L. Gonzalez, G. Gurri, S. Hand, H. Hashemi, L. Hou, J. Howland, A. Hu, J. Hui, J. Hurwitz, M. Isard, A. Ittycheriah, M. Jagielski, W. Jia, K. Kenenaly, M. Krikun, S. Kudugunta, C. Lan, K. Lee, B. Lee, E. Li, M. Li, W. Li, Y. Li, I. Li, H. Lim, H. Lin, Z. Liu, T. Lu, M. Linggioni, A. Mahendru, J. Maynez, V. Misra, M. Moussalam, Z. Nado, J. Nham, E. Ni, A. Nystrom, A. Parrish, M. Pellat, M. Polacek, A. Polozov, R. Pope, S. Qiao, E. Reif, B. Richter, P. Riley, A. Castro Ros, A. Roy, B. Saeta, R. Samuel, R. Shelby, A. Slone, D. Smilkov, D. R. So, D. Sohn, S. Tokumine, D. Valter, V. Vasudevan, K. Vodrahalli, X. Wang, P. Wang, Z. Wang, T. Wang, J. Wieting, Yuhuai Wu, K. Xu, Y. Xu, L. Xue, P. Yin, J. Yu, Q. Zhang, S. Zheng, C. Zheng, W. Zhou, D. Petrov, and Y. Wu (2023)Palm 2 technical report.. Preprint, arXiv:2305.10403. External Links: Link Cited by: SS1.
* A. Asai, S. Kudugunta, X. Yu, T. Blevins, H. Gonen, M. Reid, Y. Tsvetkov, S. Ruder, and H. Hajishirzi (2024)BUFF: benchmarking large language models for few-shot cross-lingual transfer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, Association for Computational Linguistics, New York, NY, USA, pp. 1771-1800. External Links: Link Cited by: SS1.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Won Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhkaran, E. Reif, N. Du, B 

Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanunulayan Sankaranarayana Pilali, Marie Pellat, Aitor Lewkowycz, Eric Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. _Preprint_, arXiv:2204.02311.
* Christodoulopoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. _Language resources and evaluation_, 49:375-395.
* Cui et al. (2024a) Menglong Cui, Jiangcun Du, Shaolin Zhu, and Deyi Xiong. 2024a. Efficiently exploring large language models for document-level machine translation with in-context learning. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 10885-10897, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Cui et al. (2024b) Yiming Cui, Ziqing Yang, and Xin Yao. 2024b. Efficient and effective text encoding for chinese llama and alpaca. _Preprint_, arXiv:2304.08177.
* Dang et al. (2024) John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viratav Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grimsztanjanis, Yannis Flet-Berlica, Acyr Locatelli, Hangyu Lin, Dwark Talapurru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammile Bae, Aleksandra Piktus, Roman Castagne, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, NickFrosst, Marzieh Fadaee, Beyza Ermis, Ahmet Ustin, and Sara Hooker. 2024. Aya expense: Combining research breakthroughs for a new multilingual frontier. _Preprint_, arXiv:2412.04261.
* Ding et al. (2024) Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junje Hu, Anh Tuan Luu, and Shafiq Joty. 2024. Data augmentation using LLMs: Data perspectives, learning paradigms and challenges. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 1679-1705, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Dubey et al. (2020) Abhimanyu Dubey, Abhinav Jauhi, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angelela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archiu Mitra, Archie Sravankumar, Artem Korency, Arthur Husvark, Arunk Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobrie Chern, Charlotte Caucheteux, Chaya Nayak, Choe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikoladis, Damien Allonsius, Daniel Song, Daniele Pintz, Danny Livshits, David Esiobot, Dhruy Choudhary, Drhury Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Milalon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannak Korevaert, Hu Xu, Hugo Tourvon, Hiyan Zarov, Immal Arritea Ibara, Isabel Kuumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshua Saxe, Juneting Jia, Kalyan Vasudeh Alwala, Kartikeyas Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshifting Malik, Kusnely Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madanan, Lubo Malo, Lukas Blecher, Lukas Landzadat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladi Chatterji, Olivier Duchene, Onur Celebi, Patrick Alrassey, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Korura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderera, Ricardo Silreira Cabral, Robert Stojonic, Roberta Raileanu, Rohit Girndhar, Rohit Patel, Romanian Sauvestre, Ronnie Poiddoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabsappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosalbe, Shum Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootila, Stephane Collot, Suchin Gutrangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Sciolon, Tobias Speckbacher, Todor Mihalyo, Tong Xiao, Ujjwal Karn, Vedanuj Gosowami, Vibhor Gupta, Vigineh Ramanathan, Viktor Kerkez, Vincent Gonquet, Virginic Do, Vispo Veget, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenjun Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue 

Li, Yuning Mao, Zacharcle Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafori, Abha Jain, Adam Kelsey, Adam Shainfeld, Adithya Gangidi, Addito Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annite Franco, Aparajita Sarri, Arkabanth Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assif Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamidio, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changshan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Dinana Leskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presnail, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Shothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzman, Frank Kanayet, Frank Seide, Gabriedia Medina Floreze, Gabirella Schwarz, Gada Bedeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Harroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veiliche, Itai Got, Jakse Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Cavill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junje Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartliky Khandeblwal, Katayoun Zandt, Kathy Matosichi, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kwald Lahotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madain Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Markus, Matan Hasson, Matthew Lennie, Matthias Resco, Maxim Groshev, Maxim Naumov, Maya Luthi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Mitquel Jubert Hermoso, Mo Metanat, Mohammad Rastegarai, Munish Bansali, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Baswa, Nagyan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pan Rajual Bajalt, Pedro Rittner, Philip Pontgrager, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandania, Pritish Vivraj, Qian Liang, Rachachad Alao, Rachel Rodriguez, Rafii Ayub, Raghodham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkan Hogan, Robin Battey, Rocky Wang, Rohan Mahheswaris, Russ Howes, Ruyt Binott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shengahn Lin, Shengxin Cindy Zhao, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigibe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudaram Govindparagnrad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Syoudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albierac, Vlad Ionescu, Vlad Poernau, Vlad Theberiu Milhailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xie, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnick, Zhadouhe Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Ebrahimi et al. (2022) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhry, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Gimenez-Lugo, Elisabeth Mager, Graham Neubig, Alexias Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6279-6299, Dublin, Ireland. Association for Computational Linguistics.
* El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhry, Francisco Guzman, and Philipp Koehn. 2020. CCAigned: A massive collection of cross-lingual web-document pairs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5960-5969, Online. Association for Computational Linguistics.
* news test references for MT evaluation of 128 languages. In _Proceedings of the First Workshop on Scaling Up Multilingual Evaluation_, pages 21-24, Online. Association for Computational Linguistics.
* Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari 

vazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. pp. 878-891. External Links: Link Cited by: SS4.
* M. Freitag, N. Mathur, C. Lo, E. Avramidis, R. Rei, B. Thompson, T. Kocmi, F. Blain, D. Deutsch, C. Stewart, C. Zerva, S. Castilho, A. Lavie, and G. Foster (2023)Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, Singapore, pp. 578-628. External Links: Link Cited by: SS4.
* P. Gao, Z. He, H. Wu, and H. Wang (2024)Towards boosting many-to-many multilingual machine translation with large language models. Preprint arXiv:2401.05861. Cited by: SS4.
* P. Gao, X. Zhang, Z. He, H. Wu, and H. Wang (2023)Learning multilingual sentence representations with cross-lingual consistency regularization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, Singapore, pp. 243-262. External Links: Link Cited by: SS4.
* N. Goyal, C. Gao, V. Chaudhary, P. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzman, and A. Fan (2022)The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics10, pp. 522-538. External Links: Link Cited by: SS4.
* N. M. Guerreiro, R. Reid, D. van Stigt, L. Coheur, P. Colombo, and A. F. T. Martins (2023)xcomet: transparent machine translation evaluation through fine-grained error detection. preprintarXiv:2310.10482. Cited by: SS4.
* J. Guo, H. Yang, Z. Li, D. Wei, H. Shang, and X. Chen (2024)A novel paradigm boosting translation capabilities of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, pp. 639-649. External Links: Link Cited by: SS4.
* A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Singh Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. Renard Lavaud, M. Lachaux, P. Stock, T. Le Scao, Thibaut Lavril, T. Wang, T. Lacroix, and W. El Sayed (2023)Mistral 7b. PreprintarXiv:2310.06825. Cited by: SS4.
* W. Jiao, W. Wang, J. t. Huang, X. Wang, S. Shi, and Z. Tu (2023)Is chatgpt a good translator? yes with gpt-4 as the engine. PreprintarXiv:2301.08745. Cited by: SS4.
* P. Joshi, S. Santy, A. Budhiraja, K. Balaji, and M. Choudhury (2020)The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6282-6293. External Links: Link Cited by: SS4.
* A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jegou, and T. Mikolov (2016)Fasttext.zip: compressing text classification models. PreprintarXiv:1612.03651. Cited by: SS4.
* A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (2017)Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain, pp. 427-431. External Links: Link Cited by: SS4.
* S. Kudugunta, I. Caswell, B. Zhang, X. Garcia, C. A. Choquette-Choo, K. Lee, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat (2023)Madlad-400: a multilingual and document-level large audited dataset. PreprintarXiv:2309.04662. Cited by: SS4.
* C. Li, S. Wang, J. Zhang, and C. Zong (2024)Improving in-context learning of multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico, New York, USA, pp. 8058-8076. External Links: Link Cited by: SS4.
* J. Li, H. Zhou, S. Huang, S. Cheng, and J. Chen (2024)BLICIIE the translation ability of large language models via multilingual finentuning with translation instructions. Transactions of the Association for Computational Linguistics12, pp. 576-592. External Links: Link Cited by: SS4.
* B. Liao, C. Herold, S. Khadivi, and C. Monz (2024)Ikun for wmt24 general mt task: lms are here for multilingual machine translation. PreprintarXiv:2408.11512. Cited by: SS4.
* Xi. Lin, T. Mihaloy, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasumuru, S. Sheltier, P. Songuara, V. Chaudhary, B. O'Horo, J. Wang, L. Zettelmoyer, Z. Kozareva, M. Diab, Y. Stoyanov, and X. Li (2022)Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dahbi, United Arab Emirates, pp. 9019-9052. External Links: Link Cited by: SS4.
* Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan (2024)LLAMEX: scaling linguistic horizons of 

lm by enhancing translation capabilities beyond 100 languages. Preprint, arXiv:2407.05975.
* Nguyen et al. (2024) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the _2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 4226-4237, Torino, Italia. ELRA and ICCL.
* OpenAI (2023a) OpenAI. 2023a. Chatgpt [large language model]. [https://chat.openai.com/](https://chat.openai.com/).
* OpenAI (2023b) OpenAI. 2023b. Gpt-4 technical report. Preprint, arXiv:2303.08774.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Rei et al. (2023) Ricardo Rei, Nuno M. Guerreiro, JosA@ Ponbal, Daan van Stigt, Marcos Trevico, Luisa Coheur, Jose G. C. de Souza, and Andre Martins. 2023. Scaling up CometKwi: Unabbel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, Singapore. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the _2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Schwenk et al. (2021) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
* Team et al. (2024) Gemma Team, Morgenae Riviere, Shregya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piot Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabury, Olivier Bachem,anna Walton, Aliakesei Severyn, Alicia Parsh, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimpilayukumar, Dominika Rogozinska, Dustin Heribson, Elisa Bandy, Emman Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska, Harlen Batra, Harsh Dhand, Ivan Nafrini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jeth Chan, Jin Peng Zhou, Joana Carrasqueira, Joan Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersoort, Josh Gordon, Josh Lipschultz, Josh Newman, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sedhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuremann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manwinder Singh, Mark Iverson, Martin Gorner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Mynman, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahlinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengcho Jin, Petko Georgiev, Phil Culliton, Pradep Kuppala, Ramona Connaescu, Ramona Merhely, Rencha Jana, Reza Ardeshit Rozhi, Riashho Agarwal, Ryan Mullins, Samaneh Saadat, Sara McCarthy, Sarah Perrin, Sebastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulse Doshi, Vilhan Jain, Vikas Yadav, Vidobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woodnyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Viuyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gamma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.
* Team et al. (2020) NLLB Team, Marta R. Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Baip Akula, Loic Barrault, Gabriel Meja Gonzalez, Pratinghu Hansanti, John Hoffmann, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzman, Philipp 

Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _Preprint_, arXiv:2207.04672.
* Tiedemann (2016) Jorg Tiedemann. 2016. Finding alternative translations in a large corpus of movie subtitle. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 3518-3522, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Tiedemann (2012) Jorg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)_, Istanbul, Turkey. European Language Resources Association (ELRA).
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwil Bhargava, Shruit Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioub, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Korura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Dias Lakovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavun Xu, Zheng Yan, Iiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. _Preprint_, arXiv:2307.09288.
* Ustin et al. (2024) Ahmet Ustin, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Mueninghoff, Marazieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. AyA model: An instruction finentuned open-access multilingual language model. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15894-15939, Bangskok, Thailand. Association for Computational Linguistics.
* Vilar et al. (2023) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15406-15427, Toronto, Canada. Association for Computational Linguistics.
* Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsumori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. _Preprint_, arXiv:2206.07682.
* Winata et al. (2021) Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In _Proceedings of the 1st Workshop on Multilingual Representation Learning_, pages 1-15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Workshop et al. (2021) BigScience Workshop, ;. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamachi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatuji Ruwas, Rachel Bayden, Stais Bekman, Angeline McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Albham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emzeux, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifelowda Advalian, Dragomir Radev, Eduardo Gonzalez Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gerard Dupont, German Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmunin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chin, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Joseph Tobing, Joydeep Bhattacharge, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allall, Ludovic Tanguy, Manan Dey, Manuel Romero Munoz, Mariamin Masoud, Maria Grandury, Mario Sasko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jaahar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Brexia Harlimann, Rishi Bommasani, Roberto Luis Lopez, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastiaan Nagel, Shamink Bose, Shamsuddeen Hassan Muhammad, Shanaya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, 

Vassilina Nikoulina, Veronika Liappala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davurt Emre Tagar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajnoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulkis Ben-David, Stephen H. Bach, Taewoon Kim, TaiI Bers, Thibault Fevry, Trishahi Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zhengxin Yong, Zhiqing Sun, Shaked Brody, Yallowy tri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phing, Ofir Press, Conglong Li, Deepek Narayanan, Itatim Bourtoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre Francois Lavallee, Remi Lacroix, Samyam Rajbhandari, Sandich Gandhi, Shaden Smith, Stephane Requena, Suraji Pattil, Tim Dettmers, Ahmed Baruva, Amamperei Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurelie Nevoel, Charles Loverning, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoeldopf, Jan Christoph Kalo, Jekaternin Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clineiu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antvery, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Sharvinna, Thomas Scialom, Tian Yun, Tomasz Limisewicz, Verena Riesei, Vitaly Protosov, Vladislav Mikhailov, Yadus Prouksachattun, Yonatan Belinkov, Zachary Bamberger, Zdenek Kaser, Alice Rueda, Amanda Pastana, Amir Feizpour, Amm Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh Hajji Hosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Munoz Ferrandis, Daniel McDuff, Danish Contractor, David Lensky, Davis Dayid, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezwinanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaimal, Irina Sedenko, Isar Nedjadgholij, Jesse Passmore, Josh Seltzler, Julio Bonis Sanz, Livita Dutta, Marion Sangazio, Mariamin Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akininoglu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elcott, Nour Fahmy, Olanrewaja Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alfazideh, Sarmad Shubber, Silas Wang, Souray Roy, Sylvian Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Suyuep, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chunxin Xu, Clementine Fourrier, Daniel Leon Perrian, Daniel Molano, Dian Yu, Enrique Manjucas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Sayrak, Guilly Burns, Helcan U. Vrabec, Imme Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasasi Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeeline Hahn de Bykhovetz, Maiko Takeuchi, Marc Pamies, Maria A Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Ullah, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Mueller, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrig Canalli, Rosaline Su, Rusii Sus, Samuel Calwayujaya, Samuele Garda, Shlosk S Deshmukh, Shubbhanshu Mishra, Sid Kilbhavi, Simon Ott, Since Sang-aroonsiri, Frishi Kumar, Stefan Schweter, Swhill Bharati, Tanmay Laud, Theo Gigant, Tomoya Kaniuma, Wojciech Kusa, Yannis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yue Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilide Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint arXiv:2211.05100.
* Di Wu et al. (2024)Di Wu, Shaomin Tan, Yan Meng, David Stap, and Christof Monz. 2024. How far can 100 samples go? unlocking zero-shot translation with tiny multiparalled data. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15092-15108, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Wu et al. (2023)Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. 2023. OpenICL: An open-source framework for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 489-498, Toronto, Canada. Association for Computational Linguistics.
* Xu et al. (2024a)Haoran Xu, Young Jin Kim, Amr Sharaf, and Hans Assan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
* Xu et al. (2024b)Haoran Xu, Kenton Murray, Philipp Koehn, Hieu Hoang, Akiko Eriguchi, and Huda Khayraliah. 2024b. X-alma: Plug & play modules and adaptive rejection for quality translation at scale. Preprint, arXiv:2410.03115.
* Xu et al. (2024c)Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024c. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, ICM 

2024, Vienna, Austria, July 21-27, 2024_. OpenReviewen.net.
* Yang et al. (2022) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Cheneyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jianxin Yang, Jianwie Zhang, Jinxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe He, Junyang Lin, Kai Dang, Kening Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruizo Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yicheng Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. _Preprint_, arXiv:2407.10671.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhouhii Chen, Christopher Dewan, Mona Diab, Xien Li, Xi Victoria Lin, Todor Miaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Korus, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. _Preprint_, arXiv:2205.01068.
* Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_, pages 400-410, Bangkok, Thailand. Association for Computational Linguistics.
* Zhu et al. (2024) Shaolin Zhu, Menglong Cui, and Deyi Xiong. 2024a. Towards robust in-context learning for machine translation with large language models. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 16619-16629, Torino, Italia. ELRA and ICCL.
* Zhu et al. (2024b) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024b. Multilingual machine translation with large language models: Empirical results and analysis. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 2765-2781, Mexico City, Mexico. Association for Computational Linguistics.
 



\begin{table}
\begin{tabular}{c|l l|r r|r r|r r|r r|r r}  & Direction & Mutant-7B & Opener2-7B & Urban5-7B & Llama3-8B & Llama1-8B & Gmmaa9B & GPT3-5 antho & GPTL-4 antho & Google \\ \hline en+ar & 2.411/11.321 & 59.12/52.88 & 68.43/40.542 & 65.78/58.86 & 51.65/60.33 & 73.16/46.69 & 75.59/1.693 & 73.96/73.98 & 72.73/72.53 \\ \hline en+no & 24.37/13.93 & 39.30/36.79 & 38.31/52.52 & 59.02/75.20 & 56.75/54.98 & 63.67/42.62 & 54.74/43.66 & 70.77/58.66 & 67.36/66.82 \\ \hline en+no & 53.94/47.11 & 53.64/40.26 & 57.05/45.94 & 67.03/55.98 & 67.24/56.36 & 75.26/51.63 & 75.69/1.69 & 81.74/74.05 & 67.69/69.08 \\ \hline en+no & 38.37/62.66 & 84.15/60.77 & 83.62/58.63 & 87.39/68.15 & 87.94/69.69 & 90.17/73.50 & 91.97/55.95 & 72.65/77.84 & 93.63/79.26 \\ \hline en+no & 79.67/66.42 & 81.22/68.50 & 81.29/69.08 & 83.11/70.74 & 83.07/70.57 & 85.25/74.24 & 56.66/75.59 & 86.79/76.74 & 86.32/75.30 \\ \hline en+no & 28.22/18.50 & 4.87/42.86 & 44.99/90.93 & 65.23/61.14 & 65.64/61.87 & 75.12/72.24 & 86.68/63.50 & 76.96/76.22 & 72.30/69.65 \\ \hline en+no & 43.95/63.03 & 76.15/68.68 & 77.95/65.75 & 71.44/62.62 & 74.67/65.00 & 81.31/72.32 & 83.47/43.83 & 83.94/74.68 & 81.77/73.22 \\ \hline en+no & 29.06/14.47 & 4.83/81.86 & 48.27/35.24 & 54.80/51.01 & 66.23/53.03 & 63.18/52.17 & 52.57/56.63 & 74.62/76.59 & 74.62/76.82 \\ \hline en+no & 32.31/25.76 & 34.76/34.46 & 37.76/31.51 & 37.52/35.32 & 53.32/35.32 & 66.92/82.72 & 62.87/40.37 & 84.81/64.25 & 73.07/71.63 \\ \hline en+no & 64.89/49.71 & 74.99/61.66 & 75.48/69.49 & 76.68/62.68 & 77.17/65.35 & 81.31/51.39 & 84.71/75.28 & 84.43/76.32 & 83.71/76.00 \\ \hline en+no & 55.75/67.19 & 77.26/64.09 & 77.36/69.76 & 78.09/62.67 & 67.69/64.84 & 84.41/70.67 & 85.22/75.76 & 78.18/85.87 & 85.27/74.54 \\ \hline en+no & 59.27/61.91 & 71.76/74.69 & 73.05/76.76 & 72.84/74.77 & 72.16/74.59 & 80.98/81.40 & 81.23/78.82 & 83.98/83.47 & 86.64/86.18 \\ \hline en+no & 10.94/24.85 & 15.38/20.69 & 17.36/18.69 & 25.18/25.93 & 27.12/72.32 & 71.22/76.36 & 90.19/93.53 & 26.80/23.54 & 86.63/55.98 & 65.88/72.25 \\ \hline en+no & 39.34/37.53 & 77.61/76.55 & 71.72/70.04 & 73.74/69.03 & 72.80/70.53 & 76.24/71.37 & 79.47/45.45 & 83.84/88.05 & 89.87/87.07 \\ \hline en+no & 10.54/76.25 & 14.30/57.52 & 17.23/77.76 & 16.38/9.72 & 18.85/92.33 & 33.33/23.33 & 23.91/52.52 & 43.33/73.63 & 61.67/63.59 \\ \hline en+no & 60.69/47.94 & 10.71/73.46 & 73.06/10.70 & 23.70/26.10 & 72.60/12.56 & 72.46/84.69 & 82.37/25.48 & 82.41/74.96 & 82.67/71.51 \\ \hline en+no & 117.27/10.83 & 14.34/10.11 & 14.83/53.12 & 53.23/53.23 & 54.29/82.34 & 30.38/32.32 & 23.64/45.99 & 47.87/58.59 & 89.83/73.76 \\ \hline en+no & 18.58/62.12 & 78.66/63.04 & 79.07/69.46 & 89.81/71.15 & 88.49/71.31 & 82.77/74.96 & 89.96/85.93 & 89.16/85.85 & 89.81/85.86 \\ \hline en+np & 66.30/52.96 & 63.52/48.74 & 66.51/53.03 & 72.70/60.61 & 73.06/60.44 & 81.96/71.76 & 81.32/71.33 & 75.85/76.99 & 82.59/74.22 \\ \hline en+op & 78.42/65.35 & 80.69/68.02 & 82.57/60.11 & 81.07/68.00 & 80.81/68.49 & 82.48/52.42 & 83.71/76.98 & 87.89/78.83 & 86.42/75.78 \\ \hline en+op & 77.12/76.82 & 75.28/64.72 & 67.28/62.85 & 67.53/65.79 & 73.67/67.87 & 68.57/62.01 & 80.52/70.20 & 83.73/64.76 & 81.85/74.40 \\ \hline en+on & 34.48/25.88 & 66.46/62.35 & 71.25/63.74 & 67.93/62.62 & 64.96/79.28 & 76.44/71.26 & 82.33/52.32 & 76.65/89.99 & 75.48/71.61 \\ \hline en+op & 41.64/36.37/25.39 & 72.35/35.96 & 73.98/50.65 & 67.31/54.64 & 67.46/64.14 & 73.70/69.53 & 75.30/74.09 & 74.66/49.49 \\ \hline en+up & 40.88/31.49 & 47.60/42.15 & 51.90/49.52 & 60.78/57.13 & 63.12/60.30 & 72.00/60.61 & 72.54/71.03 & 72.41/75.42 & 77.22/75.74 \\ \hline en+ur & 24.00/17.18 & 25.92/21.18 & 25.30/10.12 & 44.41/43.99 & 60.71/54.04 & 54.71/60.24 & 54.55/51.51 & 68.77/62.92 & 73.11/67.63 \\ \hline en+vj & 54.44/42.18 & 27.17/64.68 & 63



\begin{table}
\begin{tabular}{l|c c c c c c c c c|c c c c} Direction & Mistral-7B & Open-27b & Qvern2.5-7b & Llamus-8B & Llame3-1-8B & Gemma2-9b & NLLB-5dB & GPTS-5turbo & GPT+turbe & Google \\ \hline \hline ar\(\rightarrow\)en & 29.99/82.55 & 38.92/86.46 & 40.60/86.94 & 41.42/87.01 & 41.94/87.91 & 46.04/87.94 & 48.26/88.13 & 41.83/87.77 & 46.39/88.44 & 50.39/88.73 \\  & sr=en & 66.63 /97.12 & 76.78/7.17 & 27.11/83.13 & 27.65/82.86 & 28.63/83.34 & 58.87/86.02 & 42.22/89.31 & 29.71/83.68 & 40.97/87.81 & 47.08/87.82 \\  & hu=en & 17.28/78.07 & 28.01/85.65 & 27.36/85.75 & 32.74/87.52 & 32.42/87.15 & 38.35/82.64 & 42.22/89.31 & 29.17/83.67 & 39.19/83.22 & 43.52/89.62 \\  & sse=en & 30.46 /48.48 & 9.1 /66.48 & 8.14/63.41 & 21.85/87.01 & 21.58/80.24 & 23.24/85.26 & 36.04/87.88 & 19.36/87.84 & 29.39/86.18 & 37.49/86.60 \\ cs=en+en & 41.61 /87.91 & 40.78/75.52 & 42.18/87.47 & 43.39/82.53 & 43.87/87.39 & 46.53/89.56 & 43.58/89.56 & 43.59/89.02 & 44.64/89.28 & 44.68/89.34 \\ cs=en & 28.17 /85.32 & 24.17/81.43 & 24.87/87.34 & 32.44/87.57 & 33.35/87.97 & 36.75/90.45 & 42.38/91.94 & 39.65/91.25 & 42.33/91.21 & 45.95/91.51 \\ \hline de-en & 46.23 /89.39 & 45.66/89.89 & 45.69/89.89 & 47.29/88.99 & 47.29/88.15 & 50.72/89.42 & 49.80/89.36 & 45.81/89.00 & 50.68/89.99 & 52.80/90.19 \\ dc=on & 43.55 /84.36 & 33.12/87.86 & 34.51/88.84 & 38.54/86.06 & 39.39/89.52 & 40.51/87.88 & 46.28/88.10 & 47.18/88.52 & 43.67/89.09 & 48.27/89.29 \\ \hline es\(\rightarrow\)en & 34.94 /86.96 & 35.24 /87.70 & 36.05/87.15 & 36.95/87.17 & 37.34/87.24 & 39.02/87.47 & 38.00/87.49 & 36.27/87.70 & 38.23/87.99 & 37.35/87.53 \\ es=en & 28.8 /84.81 & 29.26 /85.42 & 30.14/85.55 & 30.55/85.92 & 39.02/85.96 & 33.45/86.57 & 33.05/86.92 & 34.14/87.36 & 35.41/87.36 & 35.23/87.24 \\ f=an & 24.54 /81.42 & 33.47/86.21 & 33.30/86.26 & 37.92/85.35 & 31.89/87.58 & 43.07/88.65 & 44.19/88.70 & 33.75/87.83 & 41.58/88.82 & 44.68/88.80 \\ f=fa-en & 4.49 /48.09 & 1.49 /72.72 & 13.41/70.00 & 25.68/83.71 & 26.18/84.11 & 22.19/87.31 & 36.13/87.75 & 29.30/85.60 & 34.35/88.10 & 39.72/83.84 \\ fr=an & 43.78 /35.91 & 44.71/89.02 & 47.79/89.15 & 49.17/89.01 & 49.57/89.19 & 51.93/89.63 & 51.34/89.49 & 49.39/89.70 & 54.11/89.88 & 53.44/89.89 \\ fr=en & 45.65 /85.95 & 43.65/89.52 & 42.43/86.80 & 34.93/87.13 & 50.12/87.52 & 51.82/86.36 & 56.18/88.15 & 55.63/89.09 & 55.18/89.23 & 51.89/89.23 \\ \hline he=en & 42.07 /76.46 & 40.28/86.75 & 41.28/86.70 & 44.49/88.71 & 45.02/87.84 & 49.33/88.80 & 49.02/88.62 & 42.60/87.57 & 43.73/89.89 & 51.80/89.19 \\ he=en & 42.07 /76.46 & 40.28/86.57 & 41.28/86.70 & 44.49/88.71 & 45.02/87.84 & 49.33/88.80 & 49.02/88.62 & 42.60/87.57 & 43.73/89.89 & 51.80/89.19 \\ he=en & 43.99 /46.57 & 11.72 /44.72 & 15.95/71.86 & 30.32/83.23 & 30.67/83.55 & 40.69/85.84 & 46.62/87.31 & 32.39/83.78 & 41.69/88.58 & 48.50/88.88 \\ hi=en & 42.34 /82.32 & 33.55/87.73 & 3.38/87.12 & 39.15/88.97 & 39.25/88.97 & 47.56/89.79 & 47.26/90.27 & 37.57/86.69 & 44.75/80.70 & 51.52/99.99 \\ hi=en & 42.52 /49.22 & 11.79/61.00 & 12.18/61.34 & 25.77/75.75 & 27.56/79.52 & 53.53/87.00 & 40.55/81.17 & 27.71/76.49 & 36.37/81.01 & 35.84/83.13 \\ id=en & 41.27 /87.89 & 44.26/86.84 & 45.78/89.16 & 44.26/89.16 & 46.27/89.29 & 50.93/90.00 & 49.88/93.74 & 47.66/89.83 & 47.29/82.02 & 53.52/92.99 \\ id=en-en & 42.9 /82.61 & 34.75/88.55 & 3rd.40/89.19 & 41.02/90.06 & 41.44/90.15 & 48.53/91.95 & 49.18/91.18 & 47.13/91.80 & 49.41/92.92 & 52.76/92.61 \\ it=in & 38.18 /87.576 & 38.32/87.82 & 38.89/89.59 & 39.96/87.93 & 41.01/88.81 & 41.19/85.52 & 41.56/88.13 & 39.53/85.45 & 40.58/86.67 & 41.31/88.54 \\ iit=en & 30.19 /86.13 & 28.35/87.51 & 29.08/86.33 & 2



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} Direction & Mistral-7B & Open-27b & Qvern2.5-79 & Llamus-8B & Llame3-1-8B & Gemma2-9b & NLLB-5dB & GPT3.5-turbo & GPT4+turbo & Google \\ \hline \hline \(\pi\)\(\rightarrow\)zh & 6.78/67.75 & 26.64 /84.24 & 27.49 /84.64 & 22.01 /82.54 & 22.84 /82.66 & 25.84 /84.87 & 21.25 /80.25 & 25.05 /83.20 & 29.73 /85.48 & 36.21 /86.50 \\  \(\pi\)=z & 2.91 /56.15 & 14.77 /72.14 & 15.63 /80.10 & 14.09 /78.08 & 15.96 /79.07 & 22.66 /81.26 & 27.53 /84.08 & 22.66 /83.65 & 26.24 /84.62 & 32.32 /85.00 \\   b=z & 5.16 /54.41 & 21.86 /87.27 & 21.24 /82.69 & 20.77 /82.31 & 21.35 /82.68 & 25.39 /84.94 & 20.01 /83.13 & 19.40 /81.53 & 27.38 /86.37 & 33.55 /87.07 \\   b=z & 0.74 /37.07 & 5.45 /14.14 & 4.85 /89.94 & 12.81 /74.33 & 12.71 /73.98 & 13.27 /76.97 & 23.67 /81.96 & 12.81 /72.07 & 20.64 /82.36 & 27.64 /82.38 \\ c=z & 17.4 /80.14 & 28.93 /85.82 & 28.40 /85.64 & 25.24 /84.11 & 55.26 /84.74 & 30.46 /82.65 & 19.71 /86.06 & 13.31 /86.62 & 36.19 /87.51 \\ c=z & 11.15 /75.53 & 1.45 /79.42 & 14.07 /80.81 & 18.13 /85.07 & 10.24 /86.15 & 22.17 /87.39 & 25.63 /88.81 & 24.22 /89.02 & 26.69 /90.42 & 30.65 /89.81 \\ \hline e=z [t] & 32.82 /83.13 & 30.96 /86.86 & 30.68 /86.81 & 27.33 /85.55 & 27.42 /85.51 & 31.38 /86.93 & 21.48 /88.87 & 30.38 /86.82 & 31.65 /87.07 & 30.73 /87.97 \\ o=e [t] & 16.17 /78.83 & 20.87 /80.72 & 21.06 /81.75 & 21.22 /82.14 & 22.08 /82.45 & 26.83 /83.03 & 21.58 /84.27 & 20.76 /84.84 & 29.73 /85.39 & 23.97 /86.27 \\ \hline e=z [t] & 23.28 /83.37 & 31.71 /88.43 & 37.96 /83.86 & 32.27 /86.99 & 23.57 /87.04 & 37.28 /88.42 & 26.61 /82.25 & 35.72 /84.27 & 37.97 /85.51 & 43.36 /89.94 \\ en=z [t] & 29.33 /85.88 & 34.27 /82.72 & 34.80 /87.23 & 32.66 /87.67 & 32.66 /86.86 & 35.96 /87.52 & 36.18 /87.56 & 32.60 /87.48 & 33.52 /87.78 & 39.82 /88.46 \\ es=z [t] & 16.91 /81.54 & 27.93 /86.63 & 28.40 /86.76 & 24.26 /83.53 & 33.47 /87.68 & 25.80 /86.81 & 19.03 /79.85 & 27.86 /86.63 & 24.86 /76.74 & 32.34 /87.24 \\ es=z [t] & 15.06 /80.17 & 20.56 /83.72 & 20.93 /83.94 & 20.09 /83.44 & 20.65 /83.64 & 23.26 /84.61 & 23.69 /84.46 & 23.40 /85.35 & 24.04 /85.35 & 25.59 /85.47 \\ f=z [t] & 9.14 /71.62 & 24.98 /84.44 & 6.84 /84.16 & 23.02 /83.13 & 15.45 /84.16 & 23.78 /85.66 & 21.49 /81.66 & 24.32 /84.47 & 24.84 /86.17 & 34.43 /87.16 \\ f=z [t] & 17.47 /92.43 & 9.41 /82.11 & 28.32 /84.44 & 19.92 /66.11 & 24.11 /86.39 & 23.39 /84.43 & 18.48 /82.34 & 23.44 /87.53 & 23.78 /85.42 \\ \hline r=z [t] & 19.01 /81.3 & 18.97 /87.69 & 30.97 /86.66 & 26.67 /85.47 & 27.07 /85.35 & 31.56 /86.78 & 21.19 /80.44 & 30.31 /86.79 & 32.32 /87.44 & 37.31 /87.90 \\ fr=r [t] & 22.32 /79.96 & 22.84 /83.33 & 28.62 /83.24 & 26.7 /88.26 & 27.88 /83.34 & 28.41 /89.16 & 34.16 /84.28 & 32.94 /85.22 & 34.58 /85.73 & 32.84 /85.89 \\ h=z [t] & 7.65 /66.32 & 20.88 /83.58 & 25.99 /83.70 & 23.56 /82.83 & 24.38 /83.13 & 23.91 /81.57 & 19.77 /85.20 & 24.92 /83.29 & 34.07 /85.66 & 36.98 /86.30 \\ he=z [t] & 19.41 /21.72 & 89.72 /18.17 & 7.65 /69.60 & 16.25 /87.76 & 16.47 /93.55 & 18.61 /81.91 & 27.04 /84.87 & 17.26 /79.40 & 24.25 /85.04 & 31.91 /87.57 \\ h=z [t] & 7.186 /93.61 & 23.44 /83.71 & 23.66 /83.41 & 23.53 /87.55 & 25.49 /84.99 & 27.58 /85.47 & 21.21 /86.30 & 22.37 /87.85 & 22.90 /80.68 & 25.84 /85.84 & 36.84 /87.51 \\ i=z [t] & 19.39 /40.23 & 6.12 /52.79 & 7.02 /54.20 & 14.35 /75.67 & 14.97 /67.84 & 17.84 /70.81 & 24.20 /73.81 & 16.06 /69.22 & 22.60 /74.71 & 28.30 /75.99 \\ id= z [t] & 16.48 /79.74 & 30.22 /86.13 & 30.57 /86.24 & 26.25 /84.81 & 26.79 /85.12 & 31.15 /86.49 & 21.38 /79.92 & 28.38 /66.10 & 13.36 /86.56 & 37.24 /87.66 \\ id= z [t] & 7.72 /75.08 & 20.73 /85.95 & 22.22 /86.14 & 21.66 /86.24 & 26.2

* [25] M. A. Bader, S. J. Kittel, and D. R. C. Smith. The design of the model for a large number of applications in the field-based systems. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1038-1049, 2016.
* [26] P. L. Bartlett, T. W. H. Freeman, and G. E. Hinton. Visualizing data from incomplete data. _IEEE Transactions on Image Processing_, 7:105-113, 1996.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Training batch size on each device & 4 \\ Number of GPUs & 32 \\ Gradient Accumulation Steps & 6 \\ Maximum Sequence Length & 2048 \\ Number of Epochs & 1 \\ Learning Rate & 2e-5 \\ LR Scheduler & cosine \\ Finetuning Type & full \\ bf16 & true \\ Template & empty \\ Maximum Gradient Norm & 1.0 \\ Warmup & 0.01 \\ Weight Decay & AdamW \\ DeepSpeed & ZeRO2 \\ \hline \end{tabular}
\end{table}
Table 7: Hyperparameter settings for the pretraining experiments.

\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline Training batch size on each device & 4 \\ Number of GPUs & 8 \\ Gradient Accumulation Steps & 8 \\ Maximum Sequence Length & 2048 \\ Number of Epochs & 1 \\ Learning Rate & 2e-5 \\ LR Schedulter & inverse sqrt \\ Finetuning Type & full \\ bf16 & true \\ Template & empty \\ Maximum Gradient Norm & 1.0 \\ Warmup & 0.01 \\ Weight Decay & 0.01 \\ Optimizer & AdamW \\ DeepSpeed & ZeRO2 \\ \hline \end{tabular}
\end{table}
Table 8: Hyperparameter settings for the finetuing experiments.





\begin{table}
\begin{tabular}{c|c c c c c c c c} Direction & Towerhstruct-7B & Towerhstruct-13B & X-ALMA & Aya-101 & L1aMAX3-Apsca-8B & GemmaX2-28-2B & GermanXX2-28-9B \\ \hline \hline en\(\rightarrow\)ar & - & - & 78.72 / 73.56 & 67.77 / 62.85 & 65.41 / 58.95 & 75.26 / 70.21 & 78.16 / 73.06 \\ \hline en\(\rightarrow\)bn & - & - & - & 64.62 / 60.78 & 58.37 / 56.21 & 70.93 / 70.36 & 70.53 / 70.86 \\ \hline en\(\rightarrow\)cs & - & - & 80.17 / 72.76 & 69.62 / 61.35 & 66.97 / 57.09 & 77.75 / 69.11 & 81.48 / 73.48 \\ \hline en\(\rightarrow\)no & 91.05 / 74.12 & 91.99 / 76.28 & 92.64 / 76.76 & 86.09 / 65.64 & 86.44 / 65.96 & 90.84 / 73.67 & 92.42 / 76.63 \\ \hline en\(\rightarrow\)es & 86.46 / 75.03 & 87.44 / 75.97 & 86.93 / 76.16 & 80.22 / 67.42 & 81.36 / 67.9 & 86.45 / 74.22 & 88.05 / 76.96 \\ \hline en\(\rightarrow\)fa & - & - & 78.59 / 74.49 & 68.46 / 65.44 & 65.79 / 63.21 & 76.22 / 72.41 & 79.84 / 75.87 \\ \hline en\(\rightarrow\)fr & 83.01 / 73.73 & 84.02 / 75.33 & 83.05 / 75.71 & 74.04 / 62.09 & 74.28 / 63.05 & 82.07 / 72.28 & 84.46 / 75.66 \\ \hline en\(\rightarrow\)he & - & - & 79.68 / 72.82 & 66.34 / 58.66 & 63.9 / 56.43 & 77.87 / 71.03 & 81.12 / 74.65 \\ \hline en\(\rightarrow\)hi & - & - & 70.12 / 64.37 & 61.31 / 54.32 & 63.35 / 55.65 & 65.08 / 64.38 & 60.18 / 66.03 \\ \hline en\(\rightarrow\)id & - & - & 84.53 / 76.62 & 77.9 / 67.83 & 76.17 / 63.03 & 82.27 / 73.63 & 83.55 / 74.95 \\ \hline en\(\rightarrow\)i+t & 85.87 / 75.57 & 86.91 / 76.71 & 85.53 / 76.12 & 77.5 / 63.45 & 77.78 / 66.23 & 85.69 / 74.16 & 87.1 / 76.89 \\ \hline en\(\rightarrow\)ja & - & - & 82.54 / 81.23 & 73.32 / 73.39 & 71.67 / 74.13 & 79.59 / 79.01 & 81.81 / 80.82 \\ \hline en\(\rightarrow\)kn & - & - & - & 67.42 / 72.55 & 45.71 / 53.23 & 65.64 / 71.71 & 68.21 / 74.38 \\ \hline en\(\rightarrow\)ko & 80.87 / 77.51 & 83.83 / 80.01 & 82.03 / 77.65 & 68.96 / 65.74 & 74.22 / 69.81 & 80.52 / 77.34 & 84.73 / 80.8 \\ \hline en\(\rightarrow\)lo & - & - & - & 69.68 / 67.24 & 43.31 / 32.92 & 62.37 / 64.14 & 64.35 / 66.5 \\ \hline en\(\rightarrow\)ns & - & - & 81.13 / 73.72 & 76.69 / 61.22 & 72.37 / 61.18 & 78.96 / 69.98 & 78.29 / 69.64 \\ \hline en\(\rightarrow\)my & - & - & - & 58.72 / 67.73 & 37.44 / 41.92 & 61.72 / 71.46 & 67.37 / 75.74 \\ \hline en\(\rightarrow\)nl & 89.81 / 78.88 & 90.75 / 79.79 & 90.27 / 79.38 & 83.16 / 68.38 & 81.99 / 69.68 & 88.89 / 77.01 / 80.18 \\ \hline en\(\rightarrow\)pl & - & - & 83.93 / 74.21 & 74.44 / 73.27 & 78.07 / 85.92 & 81.82 / 69.89 / 89.46 & 86.44 / 74.18 \\ \hline en\(\rightarrow\)pt & 86.16 / 76.24 & 87.98 / 77.31 & 86.72 / 77.18 & 81.39 / 64.57 & 80.78 / 67.59 & 86.73 / 75.58 & 88.13 / 77.46 \\ \hline en\(\rightarrow\)ru & 80.39 / 72.45 & 82.6 / 75.19 & 83.09 / 75.77 & 75.44 / 65.96 & 73.71 / 65.99 & 74.93 / 71.83 & 82.54 / 75.36 \\ \hline en\(\rightarrow\)th & - & - & 76.73 / 71.07 & 72.55 / 67.44 & 67.83 / 62.15 & 75.73 / 71.41 & 75.75 / 71.41 \\ \hline en\(\rightarrow\)t1 & - & - & - & 70.4 / 49.57 & 62.96 / 53.84 & 67.74 / 64.78 & 67.41 / 64.79 \\ \hline en\(\rightarrow\)tr & - & - & 76.26 / 73.11 & 66.91 / 63.41 & 63.17 / 62.25 & 75.41 / 72.18 & 77.87 / 75.51 \\ \hline en\(\rightarrow\)ur & - & - & 71.94 / 70.89 & 54.57 / 57.74 & 52.63 / 54.42 & 68.93 / 71.17 & 73.29 / 73.38 \\ \hline en\(\rightarrow\)vi & - & - & 82.48 / 78.01 & 71.57 / 62.53 & 72.89 / 66.38 & 81.39 / 76.66 & 82.03 / 76.85 \\ \hline en\(\rightarrow\)zh & 76.74 / 71.62 & 79.42 / 774.13 & 79.82 / 73.73 & 71.67 / 65.01 & 73.35 / 64.68 & 79.84 / 73.73 & 80.85 / 75.29 \\ \end{tabular}
\end{table}
Table 10: Translation performance (XCOMET / COMETKwi) of different models on the WMT-24 benchmark.

\begin{table}
\begin{tabular}{c|c|c|c} Languages & \# Monolingual tokens & \# Parallel tokens (English-centric) & \# Parallel tokens (Chinese-centric) \\ \hline \hline ar & 0 & 10000000000 & 10000000000 \\ bn & 504825391 & 1452758417 & 42416192 \\ \hline cs & 0 & 1712873789 & 287126211 \\ \hline de & 0 & 1325095947 & 67494053 \\ \hline en & 0 & - & 20000000000 \\ \hline es & 0 & 100000000000 & 10000000000 \\ \hline fa & 1114273750 & 823160488 & 62565762 \\ \hline fr & 0 & 10000000000 & 10000000000 \\ \hline he & 332751438 & 152648706 & 140761486 \\ \hline hi & 0 & 1932427027 & 67572973 \\ \hline id & 0 & 1800539057 & 199460943 \\ \hline it & 0 &



\begin{table}
\begin{tabular}{c|c c c c c c c c} Direction & TowerInstruct-7B & TowerInstruct-13B & X-ALMA & Aya-101 & LLLAMAX3-Alpaca-8B & GemmaX2-28-2B & GemmaX2-28-9B \\ \hline ar–en & - & - & 43.71 / 87.8 & 35.23 / 85.7 & 37.55 / 86.37 & 45.03 / 87.6 & 49.35 / 88.61 \\ ar–en & - & - & 40.73 / 87.74 & 25.07 / 82.98 & 27.14 / 82.49 & 38.52 / 86.37 & 42.9 / 87.76 \\ bn–en & - & - & - & 30.06 / 86.45 & 32.52 / 87.65 & 39.29 / 88.74 & 42.76 / 89.71 \\ bn–en & - & - & - & 20.03 / 82.27 & 23.08 / 81.3 & 32.9 / 86.21 & 34.83 / 86.59 \\ cs–en & - & - & 46.05 / 89.06 & 38.22 / 87.37 & 41 / 87.91 & 45.25 / 88.76 & 47.18 / 89.2 \\ cs+en & - & - & 41.93 / 91.71 & 31.28 / 88.51 & 32.18 / 87.39 & 38.72 / 90.62 & 42.69 / 91.64 \\ \hline de–en & 49.18 / 89.49 & 50.20 / 89.63 & 49.77 / 89.73 & 42.01 / 88.13 & 44.21 / 88.6 & 49.40 / 89.43 / 49.89 / 89.61 \\ de–en & 43.68 / 87.82 & 45.4 / 88.16 & 45.95 / 88.48 & 34.74 / 85.17 & 36.33 / 85.07 & 48.45 / 87.91 & 47.13 / 88.85 \\ \hline es–en & 37.67 / 87.46 & 37.56 / 87.68 & 38.17 / 87.89 & 32.47 / 86.34 & 33.91 / 86.79 & 36.07 / 87.33 & 37.88 / 87.71 \\ es–en & 33.09 / 86.79 & 33.98 / 87.05 & 33.67 / 87.2 & 28.49 & 28.58 / 85.13 & 34.34 / 86.81 & 35.45 / 87.23 \\ \hline fá–en & - & - & 41.12 / 88.5 & 33.13 / 86.67 & 35.27 / 87.32 & 41.67 / 88.42 & 44.95 / 89.18 \\ fät–en & - & - & 35.46 / 88.35 & 26.39 / 84.98 & 26.63 / 84.18 & 35.49 / 88 & 38.66 / 88.88 \\ \hline fr–en & 50.98 / 89.55 & 51.32 / 89.68 & 49.89 / 89.65 & 43.78 / 88.27 & 45.28 / 88.47 & 50.62 / 89.48 & 51.18 / 89.64 \\ fr–en & 53.85 / 88.42 & 55.65 / 88.94 & 55.39 / 88.73 & 43.18 / 85.26 & 45 / 85.79 & 54.2 / 88.39 & 57.69 / 89.11 \\ \hline he–en & - & - & 46.34 / 86.86 & 35.65 / 86.67 & 41.22 / 87.23 & 48.59 / 88.86 & 51.58 / 89.37 \\ he–en & - & - & 44.45 / 89.26 & 26.74 / 83.65 & 31.36 / 83.35 & 41.96 / 88.48 & 46.26 / 89.26 \\ \hline hi–en & - & - & 44.63 / 89.81 & 35.65 / 87.84 & 37.05 / 88.51 & 45.71 / 89.88 & 49.14 / 90.53 \\ hi–en & - & - & 38.46 / 81.45 & 24.29 / 76.16 & 27.68 / 75.99 & 37.86 / 80.17 & 41.27 / 81.1 \\ \hline id–en & - & - & 48.71 / 88.50 & 49.22 / 88.37 & 43.07 / 88.83 & 51.21 / 89.95 & 53.27 / 90.37 \\ id–en & - & - & 49.58 / 92.07 & 38.06 / 89.85 & 38.62 / 89.18 & 51.01 / 91.88 & 52.61 / 92.24 \\ \hline it–en & 40.54 / 88.4 & 41.17 / 88.46 & 40.61 / 88.46 & 35.28 / 87.12 & 35.77 / 87.39 & 39.84 / 88.24 & 42.06 / 88.59 \\ it–en & 36.5 / 88.79 & 38.17 / 89.18 & 37.72 / 89.16 & 29.4 / 86.25 & 31.09 / 86.46 & 36.98 / 88.72 & 38.83 / 89.25 \\ \hline ja–en & - & - & 32.57 / 88.22 & 23.99 / 85.65 & 27.73 / 87.18 & 33.43 / 88.19 & 36.5 / 88.78 \\ jat–en & - & - & 29.58 / 91.19 & 21.87 / 88.79 & 22.01 / 88.94 & 30.67 / 90.94 & 33.26 / 91.29 \\ \hline km–en & - & - & - & 28.08 / 83.98 & 28.74 / 85.69 & 37 / 87.56 & 41.83 / 88.56 \\ km–en & - & - & - & 19.48 / 81.34 & 14.41 / 72.92 & 20.73 / 82.32 & 24.13 / 84.22 \\ \hline ka–en & 34.51 / 88.19 & 36.01 / 88.51 & 34.31 / 88.31 & 27.38 / 86.53 & 29.07 / 87.13 & 35.35 / 88.44 & 38.76 / 89.04 \\ ko–en & 26.39 / 89.4 & 28.39 / 89.87 & 25.74 / 89.27 & 16.34 / 85.94 & 19.62 / 86.79 & 26.7 / 89.24 & 30.37 / 90.13 \\ \hline l–en & - & - & - & 33.44 / 85.25 & 25.56 / 83.19 & 39.44 / 87.51 & 44.22 / 88.82 \\ lo–en & - & - & - & 28.85 / 83.25 & 10.17 / 62.95 & 27.55 / 82.75 & 31.53 / 84.66 \\ \hline ms–en & - & - & 48.33 / 89.31 & 41.62 / 87.89 & 43.67 / 88.38 & 51.1 / 89.55 & 53.12 / 90.03 \\ ms+en & - & - & 43.23 / 89.9 & 34.21 / 87.11 & 35.7 / 87.39 & 46.09 / 89.82 & 47.03 / 89.89 \\ \hline my–en & - & - & - & 20.43 / 80.38 & 22.75 / 84.73 & 29.79 / 86.58 & 36.14 / 88.21 \\ my+er–en & - & - & - & 15.24 / 84.99 & 9.9 / 73.82 & 15.7 / 85.69 & 20.14 / 88.15 \\ n1–en & 37.61 / 87.66 & 38.72 / 87.92 & 38.78 / 88.01 & 33.25 / 86.56 & 34.49 / 86.98 & 37.3



\begin{table}
\begin{tabular}{c|c c c c c c} Direction & TowerInstruct-7B & TowerInstruct-13B & Aya-101 & LLaMAX3-Alpaca-8B & GemmaX2-28-2B & GammaX2-28-9B \\ \hline \hline ar\(\sim\)zh & - & - & 19.62 / 81.35 & 21.34 / 82.33 & 30.33 / 85.23 & 34.11 / 86.37 \\ ar\(\leftarrow\)zh & - & - & 16.31 / 79.75 & 18.35 / 80.54 & 23.2 / 82.56 & 27.62 / 84.39 \\ bn\(\rightarrow\)zh & - & - & 15.91 / 79.83 & 19.3 / 83.08 & 27.13 / 85.48 & 31.43 / 87.14 \\ bn\(\leftarrow\)zh & - & - & 11.43 / 76.82 & 14.78 / 77.21 & 20.71 / 81.4 & 24.57 / 82.68 \\ cs\(\rightarrow\)zh & - & - & 19.73 / 82.54 & 22.82 / 83.86 & 31.8 / 86.71 & 35.03 / 87.46 \\ cs\(\leftarrow\)zh & - & - & 17.93 / 85.82 & 18.86 / 86.4 & 23.55 / 87.96 & 28.17 / 89.81 \\ \hline de\(\to\)zh & 28.43 / 85.96 & 30.43 / 86.71 & 21.55 / 83.32 & 23.39 / 84.54 & 33.77 / 87.24 & 35.49 / 87.78 \\ de\(\leftarrow\)zh & 23.93 / 83.48 & 26.53 / 84.66 & 18.52 / 81.28 & 21.08 / 82.65 & 26.56 / 84.19 & 29.29 / 85.39 \\ \hline es\(\rightarrow\)zh & 25.82 / 85.83 & 27.93 / 86.68 & 24.24 / 84.81 & 26.96 / 85.4 & 30.93 / 87.33 & 32.84 / 87.9 \\ es\(\leftarrow\)zh & 21.34 / 84.21 & 22.84 / 84.76 & 24.85 / 84.38 & 29.9 / 86.4 & 23.51 / 84.77 & 25.45 / 85.54 \\ \hline en\(\rightarrow\)zh & 33.97 / 87.77 & 36.35 / 88.45 & 19.96 / 83.76 & 21.3 / 84.33 & 39.39 / 88.87 & 41.64 / 89.19 \\ en\(\leftarrow\)zh & 33.27 / 86.99 & 34.56 / 87.32 & 17.63 / 82.42 & 19.42 / 83.81 & 34.23 / 87.3 & 36.37 / 87.63 \\ \hline f\(\to\)zh & - & - & 18.66 / 82.13 & 20.83 / 83.74 & 29.39 / 86.16 & 33.12 / 87.17 \\ fa\(\leftarrow\)zh & - & - & 15.92 / 81.43 & 17.85 / 82.72 & 23.56 / 85.04 & 26.12 / 86.01 \\ \hline fr\(\to\)zh & 28.84 / 86.2 & 31.18 / 86.94 & 21.77 / 83.83 & 23.81 / 84.59 & 33.59 / 87.34 & 35.68 / 87.94 \\ fr\(\leftarrow\)zh & 30.07 / 83.92 & 31.55 / 84.96 & 24.29 / 80.98 & 26.03 / 83.06 & 31.93 / 84.77 & 35.25 / 85.63 \\ \hline he\(\to\)zh & - & - & 19.48 / 81.98 & 21.91 / 82.77 & 31.13 / 85.78 & 35.14 / 86.95 \\ hee\(\rightarrow\)zh & - & - & 13.92 / 79.22 & 18.36 / 81.24 & 23.12 / 83.65 & 28.53 / 86.03 \\ \hline hi\(\rightarrow\)zh & - & - & 19.77 / 82.8 & 20.59 / 83.04 & 29.74 / 86.05 & 33.27 / 87.18 \\ hi\(\leftarrow\)zh & - & - & 13.86 / 68.98 & 16.89 / 70.61 & 21.54 / 72.62 & 25.41 / 74.49 \\ \hline id\(\to\)zh & - & - & 19.99 / 82.24 & 23.17 / 83.98 & 33.4 / 86.96 & 35.79 / 87.7 \\ iid\(\gets\)zh & - & - & 19.72 / 86.1 & 20.77 / 86.26 & 30.08 / 88.61 & 32.34 / 89.21 \\ \hline it\(\to\)zh & 26.61 / 86.14 & 29.15 / 86.9 & 19.82 / 83.5 & 22.21 / 84.43 & 31.61 / 87.3 & 33.93 / 87.89 \\ it\(\leftarrow\)zh & 22.33 / 86.17 & 23.68 / 86.71 & 17.38 / 83.25 & 19.88 / 84.92 & 24.38 / 86.4 & 27.14 / 87.36 \\ \hline ja\(\to\)zh & - & - & 18.68 / 85.02 & 19.6 / 85.48 & 28.88 / 87.81 & 32.02 / 88.66 \\ jac\(\to\)zh & - & - & 16.86 / 87.91 & 15.73 / 88.04 & 22.28 / 89.75 & 26.1 / 90.55 \\ km\(\to\)zh & - & - & 7.71 / 69.75 & 18.11 / 81.95 & 26.48 / 85.46 & 30.82 / 86.78 \\ km\(\leftarrow\)zh & - & - & 15.84 / 78.6 & 9.15 / 67.7 & 18.61 / 80.24 & 21.99 / 81.93 \\ \hline ko\(\to\)zh & 24.11 / 85.1 & 27.83 / 86.64 & 19.58 / 83.73 & 21.32 / 84.95 & 29.63 / 86.98 & 32.62 / 88 \\ ko\(\leftrightarrow\)zh & 17.69 / 86.5 & 20.11 / 87.41 & 11.92 / 83.33 & 14.37 / 85.51 & 19.75 / 87.06 & 22.77 / 88.13 \\ \hline
1o\(\rightarrow\)zh & - & - & 17.09 / 80.82 & 13.85 / 79.06 & 26.95 / 85.08 & 31.63 / 86.91 \\
1o\(\leftarrow\)zh & - & - & 18.84 / 80.25 & 6.28 / 85.96 & 20.59 / 80.34 & 25.93 / 82.88 \\ \hline ms\(\rightarrow\)zh & - & - & 18.84 / 81.08 & 22.45 / 82.88 & 32.88 / 86.53 & 35.42 / 87.41 \\ ns\(\rightarrow\)zh & - & - & 17.94 / 82.92 & 18.16 / 83.91 & 26.87 / 86.44 & 28.57 / 86.72 \\ \hline my\(\rightarrow\)zh & - & - & 9.01 / 75.51 & 12.78 / 80.9 & 21.69 / 83.92 & 27.63 / 86.06 \\ my\(\leftarrow\)zh & - & - & 11.44 / 81.88 & 7.3 / 67.8 & 11.32 / 81.5 & 15.21 / 84.47 \\ \hline nl\(\rightarrow\)zh & 25.34 / 84.97 & 27.25 / 85.99 & 18.37 / 81.77 & 20.97 / 83.19 & 29.85 / 86.28 & 31.65 / 86.93 \\ nl\(\leftarrow\)zh & 22.46 / 85.32 & 23.31 / 85.84 &