How Multilingual are Large Language Models Fine-tuned for Translation?
Authors: Aquia Richburg
Marine Carpuat
Venue: COLM (2024)
Keywords: multilingual machine translation, large language models, translation fine-tuning, multilingual evaluation
TL;DR: Translation fine-tuned large language models improve overall translation quality, even for zero-shot languages, but the impact is uneven depending on the language pairs involved.

Abstract: A new paradigm for machine translation has recently emerged: fine-tuning large language models on parallel text has been shown to outperform dedicated translation systems trained in a supervised fashion on much larger amounts of parallel data (Xu et al. 2024, Alves et al. 2024). However, it remains unclear whether this paradigm can enable massively multilingual machine translation or whether it requires fine-tuning dedicated models for a small number of language pairs. How does translation fine-tuning impact the MT capabilities of LLMs for zero-shot languages, zero-shot language pairs, and translation tasks that do not involve English? To address these questions, we conduct an extensive empirical evaluation of the translation quality of the TOWER family of language models (Alves et al. 2024) on 132 translation tasks from the multi-parallel FLORES data. We find that translation fine-tuning improves translation quality even for zero-shot languages on average, but that the impact is uneven depending on the language pairs involved. These results call for further research to effectively enable massively multilingual translation with LLMs.

Published as a conference paper at COLM 2024
How Multilinguul Are Large Language Models Fine-Tuned for Translation?
Aquia Richburg AMSC
Marine Carpuat
Computer Science University of Maryland
University of Maryland
College Park, MD 20742, USA
College Park, MD 20742, USA marine@umd, edu
sichhu1@@mod edu
Abstract
A new paradígm for machine translation has recently emerged: fine-tuning large language models (LLMs) on parallel text has been shown outper- form dedicated translation systems trained in a supervised fashion on much larger amounts of parallel data (Xu et al. 2024a Alves et al. 2024). However, it remains unclear whether this paradigm can enable massively multilingual machine translation or whether it requires fine-tuning dedi- cated models for small number of language pairs. How does translation fine-tuning impact the MT capabilities of LLMS for zero-shot languages, zero-shot language pairs, and translation tasks that do not involve English?
To address these questions, we conduct an extensive empirical evaluation of the translation quality of the TOWER family of language models (Alves et al. (024) on 132 translation tasks from the multi-parallel FLORES-2UU dataset. We find that translation fine-tuning improves translation quality even for zero-shot languages on average, but that the impact is uneven depending on the language pairs involved. These results call for further research to effectively enable massively multilingual translation with LLMS.
1 Introduction & -Background
Machine translation (MT) systems have long been built for a specific pair of input and output languages, and trained on parallel corpus representing this language pair and translation direction (Brown et al. 1990). End-to-end neural models made it easier to train a single model on multiple language pairs, enabling multilingual machine translation models [Johnson et al., 2017 Arivazhagan et al. 2019b Dabre et al., 2020 Team et al. 2022), , where low-resource languages benefit from transfer learning from high-resource languages, including in zero-shot translation directions. While transformative, this paradigm still assumes access to large amounts of parallel data which unrealistic for many of the world's languages less than 5% of languages have labeled data useful enough for applications (Joshi et al. 2020).
Over the past year, large language models (LLMs) have started to exhibit competitive translation quality compared to dedicated supervised MT systems (Kocmi et al.. 2023) While LLMS have been used for translation in zero-shot and few-shot settings ( (Vilar al.)
2023
Hendy et al. 2023) Zhang et al. 2023a Bawden & Yvon! 2023 et al., 2022 et al.
2023. among others), the performance LLMS otten lagged behind dedicated MT systems
trained in supervised fashion on parallel text, particularly for low-resource languages. Instruction fine-tuning of multilingual LLM for MT has emerged as a strong contender to build state-of-the-an systems by providing a straightforward mechanism to combine the strengths of multilingual pre-training on primarily monolingual data with the controlled injection of parallel data- rather than only relying on incidental bilingualism (Briakou et al.) 2023). Xu et al. (002aa) showed that fine-tuning LLAMA2 models in two stages first monolingual data tollowed by small amounts of high-quality parallel data could outperform dedicated supervised MT systems (Team et al. 2022) and large closed models, like GPT3.5, with 7B or 13B parameter open I.LAMA2 models. Translation quality has been

---

Published as a conference paper at COLM 2024
further improved with contrastive preference optimization (Xu et al. 2024b). Meanwhile, Alves et al. (2024) recently introduced a family of LLM for translation-related tasks which is also based on LLAMA2. involves continued pre-training monolingual and parallel text, as well as fine-tuning on parallel text (and related tasks such as post-editing). The resulting models exhibit impressive machine translation performance, outperforming dedicated multilingual models [Team et al.) 2022) and Xu et al. (2024byss 's best models, when translating between English and the non-English languages covered in the training set. In the same vein, recent models such as Big Tigassaatee (Yang et al. 2023) or BayLing (Zhang et al. 2023b) employ continued pre-training on parallel data or translation instruction fine-tuning to improve the translation capabilities of LLMs.
However, it remains unclear how well LLMS fine-tuned for MT translate languages that they have not been trained on. Multilingual LLMS are imbalanced in their language coverage and are affected by the curse of multiinguuliity during pre-training (Chang et al.] 2023 Liang et al., 2023); this is compounded by the fact that exposure to fine-tuning parallel data is necessarily limited to a small number of language pairs raising the same questions about maximizing the benefits of transfer learning for low-resource languages without degrading performance for high-resource languages that have been extensively studied in multilinuua
MT (Arivazhagan et al., 2019a Liu & Niehues 2022
Chang et al., 2023 Yuan et al. 2023
among many others).
To fill this gap, this work seeks to characterize how LLMS fine-tuned for translation perform across diverse set of translation tasks beyond the ones they were fine-tuned on. Addressing this question is crucial to determine whether using LLMS for MT will require fine-tuning models for individual (or small sets of) language pairs, or whether they can truly fulfill the promise of multilingual LLMS and enable massively multilingual MT. We present an extensive empirical evaluation of the TOWER models on 132 machine translation tasks with varying degrees of supervision. We contribute an experimental design and result analysis inspired by Choudhury & Deshpande (2021) 's critique of averaging performance over a set of languages as the primary selection rationale for massively multilingual LLMS: we aim to balance the need to cover an extensive set of languages to understand the generalization ability of the models, with the desire to understand performance beyond a single score which averages potentially disparate behaviors across languages. Our selection of languages is driven by both linguistic typology and data size criteria. Furthermore, we include translation between all language pairs, and do not limit ourselves to translation into and out of English. While MT research naturally does not focus on English as much as other Natural Language Processing research (Bender 2019 Joshi et al., 2020), the vast majority of MT evaluations involve English as either the source or target language. Here we also seek to assess to what degree systems trained on English-centric parallel data generalize to translation tasks that do not involve English!
To be explicit, this work does not introduce any new modeling techniques. Instead, we contribute:
An extensive empirical evaluation of the generalization ability of LLMS fine-tuned for translation, using TOWER models on MT tasks involving 132 translation direc- tions between 12 languages, including German (De), English (En), Korean (Ko), Dutch (NI), Russian (Ru) and Chinese (Zh) - the supervised languages seen during fine-tuning and Czech (Cs), Icelandic (Is), Japanese (Ja), Polish (PI), Swedish (Sv) and Ukrainian (Uk) the zero-shot languages not seen during fine-tuning (Section 2
A fine-grained analysis of results that goes beyond averaging scores across lan- guages and considers best and worst behavior per model and per source and target language to provide a more complete picture of model performance across languages (Choudhury & Deshpande, 2021 (Section 3)
This goal is shared with Gao et al. 22024), however, their evaluation is limited to language pairs that involve languages seen at fine-tuning time, while we also consider generalization to zero-shot languages in addition to zero-shot language pairs.
2

---

Published as a conference paper at COLM 2024
Language
Script
LLaMA-2suppon
Similarity Groups
Czech
Latin
0.03%
West Slavic
Polish
Latin
0.09%
West Slavic
Russian
Cyrillic
0.13%
East Slavic
Ukrainian
Cyrillic
0.07%
East Slavic
German
Latin
0.17%
West Germanic
English
Latin
89.70%
West Germanic
Icelandic
Latin
possibly unseen
North Germanic
Dutch
Latin
0.12%
West Germanic
Swedish
Latin
0.15%
North Germanic
Japanese
Kana
0.10%
Kanji from Hanzi, SOV order
Korean
Hangul
0.06%
SOV order
Chinese
Hanzi
0.13%
Hanzi to Kanji, loanwords to Ja and Ko
Table 1: Evaluated languages with rationales for similarity grouping. The languages marked in bold belong to the supervised set for both TOWER models.
Findings that indicate that fine-tuning on translation-related tasks improve LLM'S ability to handle the task of translation itself beyond the specific language pairs seen during fine-tuning (Section 3
However, the fine-grained analysis reveals an uneven picture with higher variance in translation quality for language pairs that are not fully supervised, and low quality in outlier languages whether they are seen during fine-tuning (Korean) or not (Icelandic) (Section
2 Evaluation Design
Evaluation Languages Since we are interested in measuring the supervised and zero-shot translation capabilities of LLMS before and after fine-tuning, we use a subset of the fine- tuning languages from TOWER as our supervised set and collection of related languages seen as some proportion of LLAMA2's pre-training data as our zero-shot set, as summarized in Table M The supervised language set consists of German (De), English (En), Korean (Ko), Dutch (NI), Russian (Ru) and Chinese (Zh). The zero-shot language set consists of Czech (Cs), Icelandic (Is), Japanese (Ja), Polish (PI), Swedish (Sv) and Ukrainian (Uk). The languages in the zero-shot set are chosen to represent a range in resource support and to be related to languages in the supervised set by language family, typological properties or orthography. This yields (22 = 66 pairs from the set of languages, and thus 132 transla- tion tasks when considering both translation directions. We divide these 132 tasks into subcategories based on the amount of supervision available in TOWER:
Fully supervised translation directions: These consist of De, Ko, NI, Ru and Zh to and from En. This totals 10 pairs.
These are the pairs where at least one language from the supervised set, but the exact language pair was not seen during fine-tuning. For example, De-Zh belongs to this set because no direct De-Zh parallel data was used during fine-tuning. This totals 92 pairs.
Partially supervised translation directions:
Unsupervised translation directions: These are the pairs where both languages are in the zero-shot set. This totals 30 pairs.
Test Data & Metrics For all experiments, we evaluate on the multilingual, multi- FLORES-200 (Tem et al.) 2022) devtest set, which consists of English language articles manually translated into multiple languages. We report reference-based COMET-22 (Rei et al. 2022a) main metric for translation quality. We also computed the commonly
3

---

Published as a conference paper at COLM 2024
used BLEU (Papineni et al. 2002) metric, however in addition to its well-documented weaknesses, it exhibits a higher variance over diverse languages due to the differences in tokenization.
LLMS Translation Models Our zero-shot baseline model is Meta LLAMA2 (Tourron et al. 2023), an LLM which has been shown to be strong baseline on number of English language tasks. For fine-tuning, we use the TOWER (Alves al. 2024) family of models, which fine-tunes LLAMA2 for range of translation-related tasks. TOWERBASE uses two-third/one- one-third mixture of monolingual and parallel data to continue pre-training LLAMA2. The monolingual data is sourced from mC4 (Xue et al. 2021), multilingual webcrawl corpus, and the parallel data comes from several public sources ranging in domain from news to medical to Wikimedia. TOWERINSTRUCT further optimizes TOWERB for translation through instruction fine-tuning, formatting the translation task as natural language instructions. A diverse collection of data resources are formulated into question- answer templates including some zero-shot directions. TOWERINSTRUCT uses data with more task diversity including paraphrasing, dialog data and coding instructions. We consider both the 7B and 13B parameter model variants for each of the LLMS. At inference time, for the TOWER models, we use the same prompt format as presented during fine- tuning. Since LLAMA2 was not trained for translation, we default to basic natural language prompt to translate the source sent from Language to Language B. We use beam search with wedh- of 5 and maximum of 128 new tokens throughout.
Dedicated MT system We also include the state-of-the-art No Language Left Behind
(NLLB) model Team et al. (202)) NLLB is a supervised multilingual NMT system trained to translate between a diverse variety of 200 languages, including using parallel data that does not include English. Among the languages we evaluated, all language pairs that involve English and Korean <+ Japanese are seen at training time. At inference time, we use language specific tokens marking the source and target consistent with NLLB training and use the same beam search setting as for the LLMS.
Limitations While our empirical study extensive, it comes with inherent limitations. Our use of the FLORES-200 test data means that we use text that was originally written in English and translated into other languages, no matter what MT direction is evaluated, which might introduce translationese effects (Team et al. 2022). We accept this limitation as the multi-way parallel nature of FLORES-200 200 has the ad advantage of letting us make more controlled comparisons across languages. Our evaluation is based automatic metrics that are known to correlate highly with human judgments Rei et al. (2022b), it remains to be seen how the translation quality differences observe are perceived by people.
3 Results
Impact of Supervision Type In Figure we plot the distribution of COMET scores for each model according to translation supervision type. The dedicated NLLB MT model represent a strong baseline across supervision types, with mean COMET scores of 0.8 or higher. As can be expected, the LLM systems reach higher mean COMET scores for the fully supervised language pairs than others. When comparing the LLMS with the dedicated NLLB system, we find that the zero-shot LLAMA2 models lag behind by a wide margin. Interestingly, the smaller I.LAMA2-BB model yields higher mean COMET scores than the larger
model. For supervised language pairs, all TOWER models are on par or better than NLLB. However, their performance is more uneven in the other supervision conditions: continued pre-training in the TOWERBASE models improves COMET scores compared to the zero-shot LLAMA2 results but the mean COMET scores lag behind those of the dedicated NLLB systems. Translation-relater fine-tuning in TOWERINSTRUCT models improves translation quality further, but only the larger TOWERINSTRUCT-133 yields higher
We use the distilled 600M to maximize inference speed. On more limited set of language pairs, Alves al. (2024) report consistent results with the larger 54B variant.
4

---

Published as a conference paper at COLM 2024
Distribution of translation quality by supervision type
0.8
CCD CDGD
0.6
COMET Score
0.4
0.2
NLLB
TOWERBASE- 7B
TOWERINSTRUCT-78
LLAMA2-7B
TOWERBASE-138
TOWERINSTRUCT-11B
LLAMA2-138
Full
Partial
Unsupervised
Supervision
Figure 1: Distribution of COMET scores for each translation approach over language pair supervision type: TOWERINSTRUCI-1SE models are competitive with NLLB on average across settings, suggesting that transfer learning benefits many unsupervised languages, but not all based on the increased variance in the least supervised conditions.
mean COMET scores than NLLB. These results suggest the translation supervision at pre- training and fine-tuning time does improve LLM's translation abilities beyond the languages seen in parallel text, and thus that transfer learning benefits other languages.
However, the mean scores only provide partial picture of the translation quality across the 132 language pairs considered. Figure clearly shows, for all models, that the variance of COMET scores across languages also increases in the partial supervision and unsupervised settings compared to the fully supervised settings, which motivates us to look at results finer-grained fashion. The variance increases more for LLM than NLLB models indicating that the dedicated NLLB models have a more reliable behavior across language pairs. TOWERINSTRUCT has large tail of low quality outliers, some of which lower than the mean zero-shot LLAMA2 scores. This shows that translation--elltte fine-tuning does not benefit all language pairs uniformly.
Best/Worst Translation Paths per Model To dig deeper, we examine differences in transla- tion quality across languages for fixed models. Figure 2 reppresetts the spread of COMET scores for each target language for the dedicated NLLB model (top) and the overall top-performing TOWERINSTRUCT-133 TOWERINSTRUCT-133 is the top performing model, its worst performance is consistently worse than that of NLLB, for both supervised and unsupervised target languages. Trans- lation from Icelandic almost always yields the worst COMET score no matter the target language (red points). The best performing dissimilar translation paths (D+ points) inter- estingly tend to overlap with other top performing languages, indicating that dissimilarity between source and target language is not necessarily challenge for TOWERINSTRUCT-132
(bottom) Results show that while on average
'The complete set of graphs for all models and source target language groupings can be found in Appendix C

---

Published as a conference paper at COLM 2024
Translation quality for NLLB into fixed target
Source Language
0.9
S
eng_ Lat_Latn
deu Latn
0.8
kor Hang
nid Latn
rus, Cyrl
COMET Score
zho Hans
ces, Latn
0.6
Latn
jon__pp Jp_aaaa
0.5
pol Latn
swe Latn
ukr Cyrt
0.4
0.3
0.2
eng, Latn
deu Lain
run Cyrl
zho, Hans
ces Lain
fal. Latn
ipn Jpaa
pol Latn
swe Lain
Hang
Latn
Cyrl
Target Language
Supervised (left), Unsupervised (right)
Translation quality for TOWERINSTRUCT-188 into fixed target
Source Language
0.9
S
D+
D+
Latn
Latn
0.8
kor Hang
Latn
rus, Cytl
zho Hans
COMET
ces ces_Latm Latn
0.6
Latn
jan _n__paan
0.5
pol. Latn
swe Latn
ukr, Cyrl
0.4
0.3
eng. Latn
deu, Lain
kas Hang
nid Lain
rus, Cyrl
zho Hans
ces, Lain
tol. Latn
Ipn Jpan
pol, Lain
swe, Lain
ukr, Cyrl
Target Language
Supervised (left), Unsupervised (right)
Figure 2: Spread of translation quality as measured by COMET score for each target language for the dedicated NLLB model (top) and the overall top-performing TOWERINSTRUCI- (bottom). For each data point, the color identifies the source language, and we mark the best/worst dissimilar paths (D+/D-), the similar path (S), and translation from English (E).
Best/Worst Models per Language Pair Next, we show the models with the absolute best and worst translation quality per language pair in Figure 3 TOWERINSTRUCT-131 emerges the top performing model across language pairs (top graph), while the zero-shot LLAMA2 models often yield the worst performance (bottom graph); however there are interesting exceptions in both cases.

---

Published as a conference paper at COLM 2024
Max models per language pair
Model,
Translation Path Simlarity
Lagh
X
TOWERINSTRUCT-BR NOT SIMILAR
datt
TOWERRNSTRUTTJBR SIMLARR
NLLB, NOT, SIMIARR
for Hang
NLLB, SIMLAR
Lain
STRURTT, SOT_L
Target Language
nuc Oyr
TOWERBASE SIMIAR
DOWERNSTRUCT-P2 SIMIAR
zho, Hams
1361
2
Jon Jpan
swe, Lath
eng, Latt
dies, Latt
nit kas Hang
the Here
bes, Lain
Idl Lain
Jon ton_
Del, Siee, Latt
tdr, Cyrl
Source Language
models per language pair
Model, Transiation Path Similarity
Lng_Late
X LLAMA2-7B, NOT, SIMILAR
deu Latn
LLAMAZ-7B, SIMIAR
LLAMA2-138, NOT, SIMLAR
Hang
LLAMA2-13B, SIMILAR
nid Lath
TOWERBASE-78 NOT, SIMILAR
Target Langunge
Ius, Cyt
TOWERBASE 78, SIMLARR
zho Hans
cea Lat
Bl Latn
jpn Jpan
swe, Lath
Cyt
eng, Latn
desu Lats
hat Hong
rid.
nat. Cypt
Sha Hars
Cart, Lais
idl Lat
Smn Jpan
Pal Latn
Siee Lean
Cyrl
Source Language
Figure 3: Models that achieve the absolute best and worst scores per language pair. The group outlined in dotted-blue corresponds to the fully supervised set, the group outlined in dot-dash-red corresponds to the unsupervised set and the remaining pairs are partially supervised. The circle and cross symbols denote whether the language pair consists of similar or dissimilar language, respectively (as defined in Table Symbols marked with asterisks denote statistical significance according to a paired f-test at significance level of 0.05.
Among the best performing systems, see that for translation into English and for pervised languages, the largest TOWERINSTRUCT-13 is not necessarily best: the smaller TOWERINSTRUCT-7T yields higher COMET scores for translation from 5 language pairs into English (top graph, top row), and surprisingly the TOWERBASE-13H model outperforms the fine-tuned TOWERINSTRUCT models for translation from German into English. So bigger models with more supervision are not uniformly better. Furthermore, LLMS all struggle com- pared to NLLB with translation involving Icelandic and Korean. When Korean is the source TOWERINSTRUCT wins 9 out of 11 times but loses completely when Korean is switched to the target. the case Icelandic, TOWERINSTRUCT loses in both directions. Icelandic is not seen at fine-tuning time, accounts for 1% at most of LLAMA2 pre-training data, and a fusional language, so it is perhaps unsurprising that challenging to translate for the LLMS. The Korean results are surprising since is included in fine-tuning. However,
7

---

Published as a conference paper at COLM 2024
its typography (unique Hangul) and grammar (Subject-Object-Vrr)) differ vastly from the other included supervised languages (mainly Subject-Verb-Objee Verb-Object) which might make it harder to model on the target side and explain why NLLB remains stronger. This suggests that when selecting a set of fine-tuning languages, more typological diversity might support better generalization to new languages.
Interestingly, the worst performing systems are often the larger LLAMA2-13B zero-shot models, except for translation into Korean and Icelandic where the smaller LLAMA2-7B zero-shot models are worse. This suggests that the increased capacity in the underlying model benefits the representation of these outlier languages, but not for the more similar higher-resource languages. Additionally, for small number of unsupervised language pairs (English-Swediie) Kerrman-Urriniia, Dutch-Ukrainian and Dutch-Swedish), continued pre- training hurts translation quality and TOWERBASE-77 models underperform the zero-shot LLAMA2 models. Similar versus dissimilar translation directions do not show obviously different patterns.
To understand these results better, we turn to examining the properties of the inputs and outputs of NLLB versus LLM translations.
Off-Target Outputs A known failure mode of LLM is to generate translations in the wrong target language: for instance, Xu et al. (2024a) note that this behavior for LLAMA2 when translating out of English. One possible explanation is that when faced with translation on languages with lower support the model defaults to higher-resource language, usually English. We report the average percentage of translations that are found not to match the specified target language for our LLMS using the lingua-py language identification tool in 4 The analysis of off-target translation across the mulfiple language paths considered here are consistent with prior work, zero-shot LLAMA2 models are often off target. Interest- ingly the larger LLAMA2-13B is more often off-target than the smaller LLAMA2-7B model, which contributes to its worse overall performance (Figure D. Continued pre-training and Ane-tunnng substantially improve the on-target behavior of supervised languages (Figure left), however off- target outputs remain a problem even after fine-tuning when translating into unsupervised languages (Figure rright). These results suggest that LLM translations into Korean and Icelandic are bad for different reasons: translation into Icelandic is only on target about 40% of the time, while translation into Korean is generally on target (more than 90% of the time).
Input Tokenization Based on the typological property of each language and its pre- training representation, parallel texts can be encoded into subword sequences of vastly
different lengths across languages, leading to discrepancies in the quality and cost of
inference across languages (Limisiewicz et al. 2023 Ahia et al., 2023 Petrov et al. 2023). The tokenizers for NLLB and our LLM models are all built using SentencePiece (Kudo & Richardson 2018). NLLB trains a 256k vocabulary model by temperature sampling 100M bitext pairs to downsample higher resource languages and upsample lower resource languages [Team et al. 2022). The LLAMA2 tokenizer (also used by the TOWER models) has a vocabulary size of 32k trained on a mostly English data mixture [Touvron et al. 2023). IigureSppoot the average number of source subwords per segment (top) in parallel with the average COMET scores for all models for translation out of each source language (bottom)
The NLLB and LLAMA2 tokenizers both favor English, with inputs in all languages being longer than English indicating over-segmenntttoo. The LLM tokenizer over-tokenizes at higher rate than the NLLB tokenizer, and has wider discrepancy across languages: Korean text more than times longer than when translating into English, which might be a factor in the lower translation quality of I NSTRUCT-133 into Korean compared to other language pairs, despite its supervised status. Icelandic inputs are more than twice longer than English, but are somewhat shorter than Japanese inputs which are better translated by the LLM models. Input length thus does not entirely explain translation quality.
4https:/ httpss///gtttuu.coo com pemistahl piniistahlllinguaapp
We provide the symmetric figure for output length in Appendix Figure
8

---

Published as a conference paper at COLM 2024
Average on-target ratio from Supervised source
1.0
0.8
0.6
0.2
LLAMA2-78
TOWERBASE 139
LLAMA2-138
TOWERINSTRUCCE RRUCT-78
TOWERBASE-75
TOWERINSTRUCT-L1B
0.0
de
zh
pl
Target language
Supervised (left), Unsupervised (right)
Figure 4: Ratio of on-target translation averaged over the source sentences from the super- vised set. The TOWERINSTRUCT models are generally on target for supervised languages, but off-target translations are frequent for unsupervised languages. By contrast, NLLB translations are on target for 97-100% of outputs.
Conclusion
This work sought to characterize how LLMS fine-tuned for translation perform across diverse set of translation tasks beyond the ones they were fine-tuned on. By conducting an extensive empirical evaluation of the TOWER family of LLMS on 132 translation tasks representing diverse degrees of supervision, we find that fine-tuning on translation-related tasks improves LLM's ability to handle the task of translation itself beyond the specific language pairs seen during fine-tuning. This encouragingly suggests that the fine-tuning paradigm has the potential to enable massively multilingual MT.
However, the fine-grained results show that transfer learning has an uneven impact: outlier languages remain hard to handle whether they are during fine-tuning (Korean) not (Icelandic). Analysis suggest that Korean might be harmed by oversegmentation compared to other fine-tuning languages, while translation involving Icelandic often results in off- target outputs. Furthermore, the worst-case behavior of the best LLM (TOWERI 13B) is consistently worse than that of the dedicated NLLB model. This highlights the need for future work on improving instruction-tuning techniques to benefit the harder language pairs. Initial efforts in that direction are underway. For instance, Gao et al. (2024) have ported a cross-lingual consistency regularization method developed for dedicated multilingual MT (Gao et al. 2023) to instruction fine-tuning - while promising when tested on languages seen during fine-tuning, its impact on translation involving languages unseen during fine-tuning remains be seen. Our results further suggest that ensembling methods that back-off to dedicated MT models might be useful to mitigate off-target translations, and also call for addressing tokenization fairness as a potential cause for discrepancies in translation quality across languages.

---

Published as a conference paper at COLM 2024
100
Avg no. of source tokens
80
20
0.8
0.7
Comet Score
0.6
0.5
0.4
0.3
0.2
MLLB 600M
TOWERBASE 78
TOWERINSTRUCT-FE
0.1
TOWERBASE-130
TOWERINSTRUCT-150
LLAMA2-138
0.0
eng Latn
deu Latn
for, Hang
nid Latn
nus Cyrl
zho Hans
cas Latn
ts Latn
ien Jpan
pol Latn
swe Latn
ukr Cyrl
Source language
Supervised (left), Unsupervised (right)
Figure 5: Average input length per language after subword segmentation for each model (top) and average translation quality out of each source language per model (bottom).
While we have studied overall translation quality as measured by COMET, other aspects of the outputs would be worth studying in future work including how translationese effects (Dutta Chowdhury et al., 2020; Vanmassenhove et al.) 20011) and hallucination patterns (Guerreiro et al. 2023) are impacted by different degrees of instruction-tuning supervision.
References
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, May 2023,
Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. Tower: An open multilingual large language model for translation-related tasks, 2024.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. The Missing Ingredient in Zero-Shot Neural Machine Translation, March 2019a.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges, July 2019b.
Rachel Bawden and François Yvon. Investigating the Translation Performance of Large Multilingual Language Model: The Case of BLOOM. In Mary Nurminen, Judith Bren- ner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu
10

---

Published as a conference paper at COLM 2024
Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 157-170, Tampere, Finland, June 2023. European Association for Machine Translation.
Emily M. Bender. The #BenderRule: On Naming the Languages We Study and Why It Matters, https:/ http://hhhrrraien..tttte the-benderull-on-naninng www.rramniiinneelaaaaassss----ee and-wwhy-itmatteess/, September 2019.
Eleftheria Briakou, Colin Cherry, and George Foster. Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM'ss Translation Capability, May 2023.
Peter
Cocke, Stephen Della Pietra, Vincent Della Pietra, Frederik Jelinek, John Lafferty, Robert Mercer, and Paul Rossin. A statistical approach to machine translation. Computational Linguistics, 16:79-85, 1990.
Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. When is multilin- guality a curse? language modeling for 250 high- and low-resource languages, 2023.
Monojit Choudhury and Amit Deshpande. How Linguistically Fair Are Multilingual Pre- Trained Language Models? Proceedings of the AAAI Conference on Artificial Intelligence, 35
(14):12700-12718, May 2021. ISSN 2374-3468. doi: 10.1609/ aaaii 335114.1755.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. A Survey of Multilingual Neural Machine Translation. ACM Computing Surveys, (1:5:99:1-99:33 September 2020. ISSN 0360-0300. doi: 10.1145/3406095
Koel Dutta Chowdhury, Cristina España-Bonet, and Josef van Genabith. Understand- ing Translationese in Multi-view Embedding Spaces. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, Pp. 6056-6062, Barcelona, Spain (On- line), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1 wttp:///2200.lligmaaaaaa...
Pengzhi Gao, Liwen Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. Improving Zero- shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 12103-12119, Toronto, Canada, July 2023. Association for Computational Lingattiioinnngnnississ doi: 10.18653/v1/200 http://v1/222..innng--aaaa 766.
Improviinii
Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang. Towards Boosting Many-to-Many Multilingual Machine Translation with Large Language Models.
https:/ / /atpix.rxxvvio/aaa/// /abbs//41..566v/// January 2024.
Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. Hallucinations in Large Multilingual Translation Models, March 2023.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How Good Are GPT Models Machine Translation? A Comprehensive Evaluation, February 2023.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's Multilingual Neural Machine Translation System: Enabling Zero- Shot Translation. Transactions of the Association for Computational Linguistics, 5:339-351, October 2017. ISSN 2307-387X. doi: 10.1162 10.1162/tacla.00065
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ 188866551 v1 /2220.00ccc mcl-man.5500
11

---

Published as a conference paper at COLM 2024
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Ken- ton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 1-42, Singapore, December 2023. Association for Computational Linguistics. doi: 201/6653/v11222..mmlll
Taku Kudo and John Richardson. SentencePiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.1863/v1/D18-202. URL http::///dddaattooooooooo//////
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13152-13152, Singapore, December 2023. Association for Computational /006655/1///0
Linguistics. doi: 10.18653 / 2ttp22202...mmmmmimmmmm 2023. wmw.emmann...nn.mm 813
org/
Tomasz Limisiewicz, Jifi Balhar, and David Mareček. Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. In Anna Rogers, Jordan Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa- fion for Computational Linguistics: ACI. 2023, pp. 5661666600 1661-5681, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1. 2.023..indings---i
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot Learning with Multilingual Language Models, October 2022.
Danni Liu and Jan Niehues. Learning an artificial language for knowledge-sharing in multilingual translation, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language Model Tokenizers Introduce Unfairness Between Languages. In Thirty-Seventh Seventh Conference Neural Information Processing Systems, November 2023.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya
Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a, Association for Computational Linguistics.
org/ 2022 1.52
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Philipp Koehn, Loic Barrault, Ondřejj
12

---

Published as a conference paper at COLM 2024
Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-555, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation. In Paola Merlo, Jorg liedemann, and Reut Isarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2203- 2213, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v /v/l 1121.eal--ain.115.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. Prompting PaLM for Translation: Assessing Strategies and Performance, June 2023.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. paradigm shift in machine translation: Boosting translation performance of large language models, 2024a.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation, 2024b.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653 v 202222 m1...44.... URL https://aclanthology) wwwpollooyyoor/2/000 2021 wwacllaclaaa. -main.441
13

---

Published as a conference paper at COLM 2024
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages, November 2023.
Fei Yuan, Yinquan Lu, WenHao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu. Lego-mt: Learning detachable models for massively multilingual machine translation, 2023.
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting Large Language Model for Machine Translation: A Case Study, January 2023a.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models, June 2023b.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis, May 2023.
A Appendix: Other Metrics for Translation Quality
Distribution translation quality by supervision type
NLLB
TOWERBASE- -B TOWERBASE-13B
TOWERINSTRUCT-78
LLAMA2-78
TOWERINSTRUCT-138
ChrF
20
Full
Partial Supervision
Unsupervised
Figure 6: Distribution of ChrF scores for each translation approach over language pair supervision type.
We plot the distribution of ChrF scores in Figure 6 We generally see similar trend where fine-tuning shows improvement across all supervision types. The range in scores is wider compared COMET. This indicates that translation outputs vary to degree from the reference.
B Appendix: Impact of Target Tokenization
14

---

Published as a conference paper at COLM 2024
Avg no. of generated takens
60
10
Comet Score
0.6
0.2
ILAPAZ
ing Latn
det Letp
lor Hang
nd Lain
ne, Cyrl
She Hans
Lain
af Lain
iin Joan
pet Latn
Latn
dkt Cyri
Target language
Supervisec (left). Unsupervised tright)
Figure 7: Average translation quality into target and average number of generated tokens in target.
C Appendix: Translation Quality per Model and Source/Target Language
We illustrate the spread of translation quality scores for all models and source or target languages in Figures to b2loeo to complement Figure in in the main paper which show this spread per target languages, for the NLLB and TOWERINSTRUCT-13B models only.
15

---

Published as a conference paper at COLM 2024
Translation quality for LLAMA2-13B into fixed target
Source Language
0.9
eng.
kor, Hang
nid Latn
nus_ Cyet
COMET
D+
D+
Latn
0.8
IsI Lath
jpn_Jpan
D+
pol_Latn
Latn
ukr, Cyrl
0.3
eng Lain
dea Lain
Hang
nid. Lain
rus, Cytl
zho Hans
cas. Latn
Latn
Jon Jppn
gol. Latn
swe. Lawn
Ukr Cyrl
Target Language
supervised (left), Unsupervised (right)
Translation quality for TOWERBASE- 13B into fixed target
Source Language
0.9
eng Latn
D+
kor Hang
nid Latn
rus, Cyrl
COMET
Latn
0.6
IsI. Latn
Jan Jpan
pol Lain
Latn
ukr, Cyrl
0.3
eng Latn
desu, Lain
kor, Hong
nid, Lain
nus,
zho Hams
cas, Latn
Latn
Jon Jpan
pol.
sHo. Latn
Ukr, Cyrl
Target Language
supervised (left), Unsupervised (right)
Figure 8: Spread of translation quality as measured by COMET score for each target lan- guage for the zero-shot I.AMMA--15 model (top) and the monolingual /bttext fine-tuned TOWERBASE-13B (bottom). For each data point, the color identifies the source language, and we mark the best/ worst dissimilar paths (D+/D-), the similar path (S), and translation from English (E).
16

---

Published as a conference paper at COLM 2024
Translation quality for NLLB from a fixed source
Target Language
0.9
Lain
deu Lath
kor Hang
nid
COMET Score
zho Hans
ces, Latn
Ist_ Latn
Jpn_Jpan
pol, Latn
swe, Latn
ukr, Cyri
04
eng Lain
for Heng
nid, Lain
zho, Hars
cass Latn
fsl. Latt
Latn
Cyrl
Jpan
Latn
Latn
Cyrl
Target Language
Supervised (left), Unsupervised (right)
Translation quality for TOWERINSTRUCT-138 from a fixed source
Target Language
D+
D+
eng_Lav Latn
deu, Latn
kor Hang
nid
Cyrl
zho Hans
COMET
oes, Latn
0.6
[s] Latn
jp__paan
pol. Latn
swe, Latn
ukr Cyrl
eng, Lain
kor Heng
nid Lai
the, Hars
cess Latn
lsd. Latn
Latn
Cyrl
Jpan
Latn
Lath
Cyrl
Source Language
Supervised (left), Unsupervised (right)
Figure 9: Spread of translation quality as measured by COMET score from each source lan- guage for the dedicated NL..B model (top) and the overall best performing ISSRRTT 13B (bottom). For each data point, the color identifies the target language, and we mark the best/ worst dissimilar paths (D+ (D+),-), the similar path (S), and translation from English (E).
17

---

Published as a conference paper at COLM 2024
Transiation quality for LLAMA2-13B from fixed source
Target Language
0.9
eng. Latn
deu Lain
kor Hang
nid Lain
nus, Cyrl
zho Hans
COMET
0.8
Ist Latn
ces, Latn
jpn_Jpan
Latn
9we, Latn
ukr, Cyfl
0.3
eng Lain
deu, Latn
kor Heng
nid Lain
ruB, Cyrl
zho, Hams
cess Latn
Lata
ipn_ JJpan
pol, Latn
tnee, Latn
ukr. Cyrl
Source Language
Supervised (left), Unsupervised (right)
Translation quality for TOWERBASE- 13B from fixed source
Target Language
0.9
eng_ Latn
deu, Latn
D+
nid Latn
nus, Cyrl
zho_Har Hans
Latn
COMET
Ist Latn
0.6
Jpn_Jpan
pot
ukr
0.3
eng Lain
deu, Latn
Vor Heng
nid, Lain
ruB,
zho, Hans
cees Latn
Latn
Iion Jpan
Latn
BuG Latn
Cyrl
Source Language
supervised (left). Unsupervised (right)
Figure 10: Spread translation quality as measured by COMET score from each source lan- guage for the zero-shot I.LMMAZ A2-13B model (top) and the monolingual /bttext fine-tuned TOWERBASE-13B (bottom). For each data point, the color identifies the target language, and we mark the wortt dissimilar paths (D+/D-), the similar path (S), and translation from English (E).
18

---

Published as a conference paper at COLM 2024
Translation quality LLAMA2-7B from foxed
Target Language
ang.
deu Latn
Hang
nid, Latn
D+
Cytl
2ho Hans
COMET
0.6
Is| Latn
Jpan
0.5
pol, Latn
UAZ Cytl
0.4
0.3
0.2
eng
des Hang
Latn
nan, Cyel
2he, Hares
cers Latn
Latn
Jaan
Lain
sue, Lain
Oyt
Source Language
Supervised unsupervised (right)
Translation quality for TOWERBASE-77 from fixed
Target Language
eng
deu. Latn
Hang
nMt Latn
0.7
Cysl
200 Hans
COMET
0.6
Latn
Jppn
0.5
pol. Latn
www.
ukr Cyrl
0.4
0.3
0.2
eng Letn
dea
Hang
Lain
tu0, Cypi
the, Hard
cers Latn
Latn
Jaan
Lain
Oyt
Source Language
Supervised (left, Unsupervised (right)
Translation quality TOWERINSTRUCT-7B from fixed source
Target Language
eng
D++
deu, Latn
nue, Cyrl
0.7
oss, Latn
COMET
0.6
jpn Jpan
Latn
0.5
swe.
ukr. Cyrl
0.4
eng, Lett
des Latn
Hants
Lain
Cyrl
tho Harn
dens Latn
Labn
Joan
Lato
Lem
Cyr
Source Language
Supervised (lef0, Unsupervised (right)
Figure 11: FROM SOURCE; S: similar, E: English, D+: best dissimilar, D-: worst dissimilar
19

---

Published as a conference paper at COLM 2024
Translation quality LLAMA2-7B into fixed target
Source Language
eng, Latn
dau Laon
Hang
nkd, Laon
Cyff
2ho, Hans
COMET
0.6
1s Lain
0.5
pol, Lao
Latn
Ukr Cyt
0.4
0.3
0.2
eng
den Laa
Hang
Latn
run, Oye
2ha
Latt
Latn
Jipan
pol, Latn
Swet,
Cytl
Targei Language
supervised (left). Unsupervised (right)
Transiation quality for TOWERBASE-77 fixed target
Source Language
ong
deu, Loon
Hang
D+
#kg Lam
0.7
ne_Oyt
zho, Hans
COMET
0.6
Lain
0.5
Lan
Lain
Oyt
0.4
0.3
0.2
erg Letn
dea, Lain
Hang
Latn
tun, Oye
2ha,
Lato
Lutt
Jupa
pel, Lain
Swe,,
Cytl
Targel Language
supervised (left). Unsupervised (right)
Translation quality for TOWERINSTRUCT-7 into fixed target
Source Language
ong
deu Laon
Hang
#80 Lam
0.7
Cytl
zho Hans
COMET
0.6
Latn
po__ppmn
0.5
pol_aam
Sn9. Latn
ukz Cytl
0.4
eng, Lett
dea Latn
het. Hang
Labn
not, Cyel
Latn
Ladn
Juan
Dal, Lato
Sie.
Cypl
Target Language
Supervised (left) Unsupervised (righ))
Figure 12: TO TARGET; S: similar, E: English, D+: best dissimilar, D-: worst dissimilar
20


Reviews:
[{"review_id": "74AuxAnPeU", "reviewer": "Reviewer_rg12", "summary": "The authors conduct a large scale empirical evaluation of the translation performance of the Tower language models, which are multilingual foundation models created from the LLama2 line of base models, and targeted at translation and related tasks such as quality estimation and automatic post editing. The evaluation measures translation performance in supervised and zero-shot settings, before and after fine-tuning. As a baseline and semi-oracle, the NLLB massively multilingual model is also evaluated in all settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on the paper.\nAs you noted, this paper exclusively contributes a collection of evaluations to analyze model performance in multilingual settings beyond coarse averaging of metrics.  We believe that providing such a deep dive is critical to complement prior work which only reports average results and might thus obscure large gaps in performance between different language pairs.  We hope that this analysis can promote more comprehensive evaluations in future work, which can in turn guide the design of techniques that improve generalization across diverse language pairs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- clearly written paper with very useful evaluation on partial- and fully-unsupervised MT tasks. \n- in particular, the inclusion of best and worst performance in each setting will be useful for many researchers\n- concrete takeaways, such as that pretraining on translation tasks benefits translation performance on unseen language pairs, but not for all language pairs uniformly/", "reasons_to_reject": "- limited scientific novelty, in that the work is only a collection of evaluations", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hzO3NMuIwP", "reviewer": "Reviewer_zRXH", "summary": "The paper conducts an extensive empirical evaluation of the translation quality of the TOWER family of language models on 132 translation tasks from the multi-parallel FLORES data, aiming to find out how does translation fine-tuning impact the MT capabilities of LLMs for zero-shot languages, zero-shot language pairs, and translation tasks that do not involve English. The conclusion is interesting that the translation fine-tuning improves translation quality even for zero-shot languages on average.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on the paper.\nWe have fixed the formatting errors you mentioned.\nIt would indeed be interesting to assess how trends hold with different LLMs.  We focused our work on the TOWER model family since at the time of writing it was one of the stronger contenders in comparison to the NLLB supervised multilingual MT system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an interesting problem about the multilingual translation ability of LLM models.\n2. Empirical experiments show that the translation fine-tuning improves translation quality even for zero-shot languages on average, which is impressive.\n3. The paper further finds out that the improves are not defined by the language similarity, which leaves rooms for the future research on this direction.", "reasons_to_reject": "1. The paper seems to be written in hurry. Table 2 (ref in Section 2) seems to be missing. Figure 5 is cropped. \n2. The innovation of the paper seems to be limited. However, it is ok for an empirical paper.", "questions_to_authors": "It will be better if the finding can be further confirmed on other LLMs.", "extra_scores": {}}, {"review_id": "25IL8RhgXX", "reviewer": "Reviewer_sAju", "summary": "This paper investigates the capabilities of a large language model (LLM) fine-tuned for machine translation (MT) on certain languages to perform across various language directions for which it has not been tuned. This paper does not present any technical novelty but tries to understand better the capabilities of an LLM challenging it in unseen language directions. The experiments include 132 language directions split into fully supervised, partially supervised, and unsupervised directions, depending on the data used during training. When dealing with a large number of directions, how to report the results is an important problem, because the widely-used average does not allow a reader to highlight the positive and negative performance of a model. This paper addresses this problem by considering the best and worst behaviors of a model and linking them with the source and target languages. A large set of experiments with in-depth analysis are reported to verify the initial research questions. \n\nThe main findings of the paper are:\n\n-) The fine-tuning of an LLM for translation-related tasks helps to improve the performance also on unseen languages. \n\n-) The paper identifies high variance in translation quality for language pairs that are not supervised and outliers languages such as Icelandic and Korean.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments on the paper.\nWe have made the editing fixes suggested.  Thank you!\n\nWhile we do not directly study the impact of writing script, we report how the different amounts of tokenization across languages impact translation quality (in Figure 5), which is impacted by the nature of the script.  The LLAMA tokenizer used by the TOWER models is mainly trained on English text and other text using variants of the Latin alphabet.  And we see translation with the European languages are generally higher than with the Asian languages (in Figure 2, for best performing TOWER variant).  In contrast, NLLB has a more even representation of writing scripts among the language collection and results have less disparity.  It remains to be seen whether a more equitable tokenizer could improve translation quality for the LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although it does not propose any technical enhancement, the paper addresses an interesting problem that should help to understand better the behaviors of LLMs for translation. In this historical period, where there is an increasing emphasis on improving LLM performance on multiple tasks, these papers are important to have a critical view of the existing approaches and provide some directions to work on.\n\nThe paper is supported by a large set of experiments that include results for high- and low-resourced languages. The proposed analysis has clear questions that are addressed with proper experiments. \n\nThe paper is clear and easy to follow.", "reasons_to_reject": "The findings are clear but not conclusive leaving more activities to better understand the capabilities of the LLMs for translation and to motivate the outlier behaviors of some languages.\n\nThe proposed method for overcoming the average limitation when reporting results for a large number of pairs is interesting, but it may not be immediately comprehensible and may require some effort to fully appreciate. However, this is the trade-off for introducing a new metric or visualization strategy. \n\nThe experimental parts of the paper are not always easy to follow and appreciate. The fact that the figures and tables are not on the page where they are discussed creates more difficulties. \n\nOverall the paper is well-written, but it lacks the final proofreading. Some sentences to adjust:\n\n-) Page 1: “Briakou et al. (2023). vast amounts of noisy multilingual pre-training data. Xu” not clear how the sentence starting with “vast” is connected to the previous and next sentences. \n\n-) Page 3: the table with the language information is table 1 and not table 2.\n\n-) Page 5: “for each target language for the for the dedicated” “For the” is repeated.\n\n-) Page 8: “upsample lower resource languages .“ extra space to be removed.", "questions_to_authors": "Do the authors have any insights on the impact of the alphabet used by each language on the translation performance? And what is the role of the vocabulary size (smaller vocabulary can penalize those languages with a high-variable alphabet or an alphabet with a large set of specific symbols)?", "extra_scores": {}}]

Author rebuttal:

Meta-review: There appears to be a consensus among the reviewers. The paper presents a thorough evaluation of the TOWER family of language models, LLaMas specifically fine-tuned for MT. It draws several interesting and somewhat surprsing conclusions; notably, fine-tuning an LM for translation not only enhances performance in supervised language tasks but also improves results on languages / tasks not directly trained on. Additionally, there is considerable variability in performance across zero-shot languages. Although the reviewers acknowledge that the paper does not make a substantial technical contribution, they believe that the findings of this large scale study (132 translation directions!) are valuable to the community.
Decision: Accept