Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
Authors: Duarte Miguel Alves
José Pombal
Nuno M Guerreiro
Pedro Henrique Martins
João Alves
Amin Farajian
Ben Peters
Ricardo Rei
Patrick Fernandes
Sweta Agrawal
Pierre Colombo
José G. C. de Souza
Andre Martins
Venue: COLM (2024)
Keywords: Machine Translation, Multilinguality, Adaptation, Continual Pretraining
TL;DR: We propose a recipe to tailor LLMs for translation workflows.

Abstract: While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and parallel data, creating TowerBase, followed by finetuning on instructions relevant for translation processes, creating TowerInstruct. Our model surpasses open alternatives on several relevant tasks and is competitive with general-purpose closed LLMs. We will release the Tower models, our specialization dataset, an evaluation framework for LLMs focusing on the translation ecosystem, and a collection of model generations on our benchmark.

Published as a conference paper at COLM 2024
TOWER: An Open Multilingual Large Language Model for Translation-Related Tasks
Duarte M. Alves" 2,3, José Pombal 1,2,, Nuno M. Guerreiro 1,2,3,4 Pedro H. Martins', João Alves', Amin Farajian', Ben Peters?, Ricardo Reil, Patrick Fernandes Sweta Agrawal?, Pierre Colombo 455 José G.C. de Souza & André F.T. Martins Shartemalves@teee www..ol..aba............oo. pt, [jose. pombba,,,,,, nuno. enverreiro@@unbabbee com
Abstract
While general-purpose large language models (LLMs) demonstrate profi- ciency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on multi- lingual mixture of monolingual and parallel data, creating TOWERB ASE, followed by finetuning on instructions relevant for translation processes, creating TOWERINSTRÜCT. Our model surpasses open alternatives on sev- eral relevant tasks and is competitive with general-purposs closed LLMs. We will release the TOWER models, our specialization dataset, an evalu- ation framework for LLMs focusing on the translation ecosystem, and collection of model generations on our benchmark.
Introduction
Many important tasks within multilingual NLP, such as quality estimation, automatic post- edition, or grammatical error correction, involve analyzing, generating or operating with text in multiple languages, and are relevant various translation workflows we call these translation-related tasks. Recently, purrree large language models (LLMs) chal- lenged the paradigm per-task dedicated systems, achieving state-of-the-art performance several recent WMT shared tasks (Kocmi et al., 2023; Freitag et al., 2023; Neves et al., 2023). Unfortunately, strong capabilities for multiple translation-related tasks have so far been exhibited by closed LLMs only (Hendy et al., 2023; Koemi & Federmann, 2023; Fernan- des et al., 2023; Raunak et al., 2023). Perhaps because most open LLMs are English-centrre approaches leveraging these models still lag behind, having thus far achieved competitive results only when specializing on single task (Xu et al., 2024a; 2023; Iyer et al., 2023).
Filtered monolingual
High-quality diverse
parallel data
instructions
pretraining
Supervised linetuning
Llama 2
Tower Base
Tower Instruct
Figure Illustration of our method for building TOWERBASE and TOWERINSTRUCT.
In this paper, we bridge this gap with a detailed recipe to develop an LLM for multiple tranllation-related tasks. Our approach, illustrated in Figure 1 and inspired by Xu et al.
'Equal Contribution: ordered alphabetically by first name. Unbabel, Instituto de Telecomunicações, IInstituto Superior Técnico Universidade Lisboa (Lisbon ELLIS Unit), MMISS CentraleSupélec, Université Paris-Saclay, SFquall, 'Carnegie Mellon University.
1

---

Published as a conference paper at COLM 2024
FLORES-200
WMT23
86
89
TOWERINSTRUCT
GFT-4
TONERINSTRUCTTB
GPT-3.5-turho
84
TOWERINSTRUCT.I
GPT-3.S-turbo
COMET-22
88
ALMA- COWERINSTRUCTE
Mixtral-S<7l-Innstrru
LLa
ALMA-R K 7B
Mixtral-&7B-Instrrutt
87
Gomma
LIMMA-2133
* NLLB 5IB
82
Gemma
LLMMA-2700
E LaMA-22 13B Mintral-BBinsruc--02222
86
80
LIaMA-27B
LLMMA-27B
NLLB SIB
85
78
7
13
46
54
70
7
13
46
54
70
Model size (# billion parameters)
Model size (# billion parameters)
Figure 2: Translation quality for TOWERINSTRUCT and collection of open and closed models. GPT models' scale is unknown so we represent them with a horizontal line. TOWERINSTRUCT wwwp://rRRCCCTooooooooooooo larger open alternatives and is competitive with GPT models.
(2024a), relies three steps. First, extend LLaMA-2's (Touvron al., 2023b) multilingual capabilities through continued pretraining, creating TOWERBASE (62.1). Importantly, while Xu et al. (2024a) exclusively employ monolingual data, include parallel data as an additional cross-lingual signal. Second, we curate a dataset to specialize LLMs for translation-related tasks, TOWERBLOCKS ($2.2). Third, we perform supervised finetuning to model for the field of translation, TOWERINSTRUCT ($2.3).
obtain an instruction-followe
We extensively evaluate all our models, comparing with open and closed alternatives on wide range of tasks (53). TOWERINSTRUGE consistently achieves higher translation quality than open alternatives and is competitive with the closed GPT-4 and GPT-3.5-turbo models see Figure 2. Additionally, TOWERI outperforms open models in automatic post- edition, grammatical error correction, and named entity recognition. Careful ablations also outline the influence of each element in our recipe (54). We highlight the importance of adding parallel data during continued pretraining for improved translation quality, and the effectiveness of including conversational and coding data TOWERBLOOCKS.
Accompanying this work, we release 1) the TOWER family, comprising our TOWERBASE and Tow CRWERRISTTUCT models in the sizes of 7B and 13B; 2) our specialization dataset TOWER BLOCKS; 3) TOWEREVAL, the evaluation framework for LLMs for translation-rellted tasks that used to perform all evaluations in this paper; and collection of model generations for our benchmark to ensure reproducibility and encourage future exploration.
2 TOWER: An Open Multilingual LLM for Translation-Related Tasks
Our backbone language model is LLaMA-2 (Touvron et al., 2023b), which is very competitive on a wide range of tasks. Nevertheless, it was trained on relatively little non-English data, limiting its potential for multilingual tasks. We alleviate this effect by continuing LLaMA-2's pretraining on a highly multilingual corpus (62.1). Afterwards, we finetune our continued pretrained model on specialization dataset (52.2), obtaining an instruction-follee following model centered around translation (62.3).
2.1 TOWERBASE: Extending the multilingual capabilities of LLaMA-2
We extend LLaMA-2's training highly-multilinguaa dataset comprising 20 billion tokens measured with the model's tokenizer - for 10 languages: English (en), German (de), French (fr), Dutch (nl), Italian (it), Spanish (es), Portuguese (pt), Korean (ko), Russian (ru),
Links for the TOWER models; TOWERBI.OCKS TOWEREvAI; Zeno (Cabrera et al., 2023) project with model generations.
2

---

Published as a conference paper at COLM 2024
Pre-translation 2%
Translation 27%
Post-translation 28%
General-purpose
Share of each task corresponding branch TowerBlocks, %
Named-entity recognition
Sentence-level translation
Error-span detection (MQMD
Conversational data
100
86
62
Context-aware translation
Automatic post-cdition
Code instructions
Mulli-eference translation
Translation ranking
Paraphrase geueration
Doc-level Terminology
Figure 3: Tasks included in supervised finetuning dataset TOWERBLOCKS.
and Chinese (zh). While Xu et al. (2024b) exclusively leverage monolingual data, we draw inspiration from work including parallel data during pretraining (Anil et al., 2023; Briakou et al., 2023), and mix parallel sentences (one third) along with monolingual data (two thirds). Our results show that this approach greatly benefits translation quality (64).
Monolingual data. We sample uniformly across our languages from mC4 (Xue al., 2021) and apply standard cleaning (Wenzek et al., 2019; Touvron et al., 2023a): deduplication, language identification, and perplexity filtering with KenLM (Heafield, 2011).
Parallel Data. We sample uniformly to-English (xx-+en) and from-English (en-+xx) lan- guage pairs from public sources, removing translations below quality thresholds for Bi- cleaner (Sánchez- Cartagena et al., 2018; Ramirez-Sánchezee al., 2020) as well as COMETKIWI- 22 (Rei et al., 2022b), which is shown to improve translation quality (Peter et al., 2023). We include parallel data with the following template, calculating the loss on all tokens:
[source language] [source]\n[targe] language]: [target]
Model Training. We train our models with a codebase based on Megatron-LLM (Cano et al., 2023) on A100-80GB GPUs, using effective batch size of 1.57 million tokens per gradient step and cosine scheduler with initial and final learning rates of 3 x 10 5 and 3 10-6, We provide additional details on continued pretraining in Appendix D.1.
2.2 TOWERBLOCKS: A dataset to tailor LLMs for translation-related tasks
We build TOWERBLOCKS prioritizing data diversity and quality. Figure illlsstrates all tasks in the dataset. They include tasks important to translation workflows, applied before or after translation, and datasets to improve multilingual understanding and instruction-follooinng
Diversity. We collect records from multiple datasets for each translation-related task datasets are detailed in Appendix E. Akin to Wei et al. (2022), we reformulate all records as natural language instructions using multiple manually curated zero- and few-shot templates. Following insights from Longpre et al. (2023), we create zero-shot instructions with 75% of the records. For the remaining ones, we uniformly sample 1, 3, or in-context examples from the respective dataset. We also include paraphrasing, dialogues from UltraChat (Ding et al., 2023), and code instructions from Glaive-Code- (oode--ssssstnnt (Glaive AT, 2023).
Quality. Similarly Xu al. (2024a), we construct records from datasets, prioritzzin validation and old test sets we exclude datasets from 2023 onwards.
For named entity recognition, we did not find permissively licensed human-annotated dataset, so MultiCoNER (Malmasi al., 2022; Fetahu et al., 2023). For general translation, include
3

---

Published as a conference paper at COLM 2024
For tasks with reference translations, such as translation and automatic post-edition, score references with XCOMET-QE-EESEMBLL (Guerreiro et al., 2023) and discard records with quality scores below 0.85. Additionally, we include only translation pairs in their original direction. Finally, we adopt the UltraChat (Ding et al., 2023) dialogues filtered by Tunstall et al. (2023) and further exclude records that contain translation requests, formatting issues (e.g. instructions starting with punctuation), or assistant refusal cases.
2.3 TOWERINSTRUCT: Specializing TOWERBASE for translation-related tasks
As a final step, we obtain TOWFRINSTRUCT by finetuning TOWERBASE on TOWERBI.OCKS.
Dialog template. We format each dialog single tokenizable string using the chatml template (Open 2023). This template clearly separates between instructions and answers, and allows for multi-turn dialog. The template has three special identifiers (control tokens) to delimit messages: < start >user and <| im start />assis itant precede the beginning of turn, and <|im end|> marks its end. We provide further details in Appendix F.1.
Model training. We finetune the model with the standard cross-entropy loss, enabling bfloat16 mixed precision and packing (Raffel et al., 2020). We only calculate the loss target (answer) tokens. We train for 4 epochs using a low learning rate and a large batch size we detail all hyperparameters Appendix F2.
3 Experiments
3.1 Experimental Setup
Datasets and Tasks. We analyze machine translation (MT) on FLORES-200 (NLLB Team et al., 2022), WMT23 (Kocmi et al., 2023), and TICO-19 (Anastasopoulos et al., 2020). Addi- tionally, we examine three translation-related tasks. First, following Raunak et al. (2023), we evaluate automatic post-edition (APE) by measuring translation quality after post-editing NLLB-3.3B (NLLB Team et al., 2022) translations for WMT23. Second, we evaluate named entity recognition (NER), useful for entity anonymization, using the test split from Multi- CONER 2023 (Fetahu et al., 2023).² 3 Third, we evaluate grammatical error correction (GEC), which is held out from our training data and useful for correcting the source sentence before translation. We test GEC on CoNLL-2014 (Ng et al., 2014) (English), COWSL2H (Yamada et al., 2020) (Spanish), and mlconvgec2018 (Chollampatt & Ng, 2018) (German).
Baselines. We compare TOWER with the open LLaMA-2 70B (Touvron et al., 2023b) and Mixtral-sx77-Insttruct (Jiang et al., 2024), and the closed GPT-3.5-turbo and GPT- 4.4 For MT, we also consider dedicated translation models NLLB-54B (NLLB Team et al., 2022) and ALMA-R (Xu et al., 2024b). In Appendix G, we compare with the open Gemma 7B (Gemma Team, 2024), Mistral-7B-Instruut--0222 (Jiang et al., 2023) and Qwen1.5 72B (Bai et al., 2023).5 Model generations use greedy decoding - we explore alternative decoding methods in Appendix A. We prompt TOWER and closed models in 0-shot fashion and others with examples randomly selected from the development set.
Evaluation. We evaluate translation quality with COMET-22 (Rei et al., 2022a) for both MT and APE. For MT, we also report XCOMET (Guerreiro et al., 2023), COMFTKIW1-22 (Rei et al., 2022b), BLEURT (Sellam et al., 2020), and CHRF (Popović, 2015) in Appendix G.6 6 For GEC, we
small amount of parallel data from OPUS to cover all language pairs. Nevertheless, we apply Bicleaner using a threshold of 0.85 followed by the quality filtering procedure described in this section.
We uniformly sample 1000 of the more than 200k records due to the computational costs of evaluating all models on the whole test
*We use gpt-3. 5- 5-turbo-061 0613 and gpt-4-0613 available from the official OpenAl API. STOWHIINTTUCCT outperforms all these open alternatives.
Perforraacc trends largely hold across metrics. Yet, there is significant quality gap between AI ALMA-R and TOWER models in terms of CHRF e.g, over points in en en-*xx directions WMT23.
4

---

Published as a conference paper at COLM 2024
FLORES-200
WMT 23
TICO 19
Models
en -Xx
xx-te
en-XX
xx-te
en en-
Closed
GPT-3.5-turbo
88.95 2
88.14 3
85.56 2
83.48 2
87.36 2
GPT-4
89.13
88.42
86.01
83.69
87.52
Open
NLLB 54B
86.79 4
87 .5 3
78.60 7
79.06 6
87.05 2
LBM
87.82 4
88.19 2
82.95 6
82.56 4
86.46 4
Mixtral--x77--nstruut
87.76 3
88.17 2
83.60 5
82.84 3
86.60 4
ALMA-R 7B
83.40 5
82.39 4
ALMA-R 13B
84.46 3
83.03 3
TOWERINSTRUCT 7B
88.51 3
88. 27 2
84.28 3
82.77 4
87 01 3
TOWERINSTRUCT 13B
88.88 2
88.47
85.14 2
83.18 2
87.32 2
Table 1: COMET-22 scores for translation test sets aggregated by language pairs. Models with statistically significant performance improvements are grouped in quality clusters. We highlight the best ranked models in bold and underline the best ranked open models.
WMT23 (en XX)
WMT23 (xx cn)
20
10
Versus
A-MA-R-138
Win Margin (9)
Ver sus ALM--R-13
TOMER KUNS
Win Margin
0-
TORET LOSCS
-10-
-10
Ver sus GPT-4
-20
Versusi GPT-4
-0
0-2015 Lergth
8-20th Jength
-30
-28 cokens
-50 takens
-30
porcentile
-40 Tokens
150 Tokens
(1
45
4
Source Length (in tukens) Quantile (4)
Source Length (in takens) Quantile (4)
Figure 4: Win rates margin of TOWERINSTRUCT--3B by tokenized source length WMT23 en-xx (left) and xx-en (right) directions. We compare against GPT-4 (O) and ALMA-R (4). We define win if the sentence-level delta between two systems is above 1 COMET-22 point.
measure edit rate (ER) (Snover et al., 2006) and report ERRANT (Bryant et al., 2017; Felice al., 2016) in Appendix H. For NER, we measure sequence Fl score. We report performance clusters based on statistically significant performance gaps 95% confidence threshold.? We create per-language groups for systems with similar performance, following Freitag et al. (2023), and obtain system-level rankings using a normalized Borda count (Colombo et al., 2022), which is defined average of the obtained clusters. Note that first cluster will not exist if no model significantly outperforms all others on majority of languages.
3.2 Translation
Table reports aggregated results on translation test sets. Table studies translation quality all training languages, considering en-*xx and xx- translation directions.
INWERINSTRUCT 13B is the open system with highest translation quality. TOWERIN- STRUCT 13B consistently outperforms both larger open LLMs and dedicated systems. On FLORES-200, TOWERINSTRUCT 13B is often ranked first, and is close to GPT-4 performance
We posit that ALMA-R's alignment process translations preferred by CIMETKIWI-XXI (Rei et al., 2023) and XCOMET may inadvertently degrade performance on lexical metrics.
FFrr segment-level metrics, such as COMET-22, we perform significance testing at the segment level. For corpus-level metrics, such ER and Sequence F1, consider 100 bootstrapped samples of size 500, similar to Koehn (2004).

---

Published as a conference paper at COLM 2024
FLORES-200
(en
>xx)
Models
de
fr
it
ko
nl
pt
ru
zh
Closed
GPT-3.5-turbo
88.78
87.08
89.02
89.06
89.36
2
88.63
90.46
89.56 3
88.58 2
GPT-4
88.98
87.10
88.93
89.05
90.06
88.56
90.43
90.19
88.87
Open
NLLB 54B
87.18 5
85.9 4
87.71 3
88.10 3
89.00 3
87.33
88. 5
88.89 4
78.26 7
LLaMA-2 70B
87.31 5
86.41
3
87.82 3
88.22 3
88.07
87.47
89.11
88.65
87.32 5
Mixtral-bx77-Instruet
87.99 3
86.80 2
88.53
88.77 2
85.63
87.57
89.45
89.09 4
85.99 6
TOWERINSTRUCT 7B
87.82
4
86.76
88.44
88.73
89.41
88.38
89.60
89.53
87.90 4
TOWERINSTRUCT 13B
88.16 3
87.06
88.92
89.21
89.92
88.63
89.78
89.95 2
88.29 3
FLORES-200 (xx
>en)
Models
de
ko
nl
pt
zh
Closed
GPT-3.5-turbo
89.60 2
87 87.263
3
89.46 3
88.03 3
87.8
87.71
89.78
86.69 4
86.92 2
GPT-4
89.76
87.57
89.61
88.21 2
88.58
87.88
89.94
86.94
87.29
Open
NLLB 54B
89.17 4
87 25
89.29 4
87.91 3
87.86
87.49
89.38
38
86.66
86.55 3
LLaMA-2 70B
89.44 3
87.49
89.55
88.18
87.91
87.52
89.84
86.87
86.91 2
Mixtral-8x77-Insttrue
89.57
2
87.65
89.56
88.44
87.37 4
87.54
89.73 3
86.81 3
86.88 2
TOWERINSTRUCT
89.48
87.48
89.50
88.39
88.16
87.66
89.92
86.90
86.96 2
TOWERINSTRUCT 13B
89.61
87.62
89.67
88.42
88.48
87.92
90.07
87.20
87.27
Table 2: Translation quality (via COMET-22) on FLORES-200 by language pair. Models with statistically significant performance grouped in quality clusters. Best ranked models are highlighted in bold and best ranked open models are underlined.
on WMT23 and TICO-19. Upon inspecting both systems' outputs, we verified that the gap between them increases with longer sentences, as shown in Figure 4. Notably, this trend vanishes when comparing TOWERINSTRUCT 13B to ALMA-R. We posit this difference stems from a prevalence of shorter translations in the training data of both TOWERINSTRUCT 13B and ALMA-R. In future work, we would like to explore how better leverage longer contexts, which can benefit instruction-following (Zhao et al., 2024).
TOWERINSTRUCE 13B achieves high translation quality across all language directions. In Table 2, TOWERINSTRUCT 13B is ranked first for the majority of en-*xx directions, and is among the top performing models for all but one xx xx->e language pair. Notably, TOWERINSTRUCT stands out as the best overall model outperforming GPTT-4 for both pt->en and ru ru-ten directions. The improved performance on xx xx->en directions likely stems from LLaMA-2's English-centric pretraining. A longer, more expensive continued pretraining might further improve performance on en en- directions. In fact, we show in Section that the translation quality gains from LLaMA--2 larger for en-xxx language directions.
TOWERINSTRUCT 7B achieves trade-off between performance and scale. The smaller TOWERINSTRUCT 7B, although behind TOWeRINSTRUCT 13B, is competitive with other open systems and achieves GPTT-3.-turbo translation quality for some language pairs. Importantly, it outperforms the only system of the same size, ALMA-R 7B.
8A similar domain-levdl analysis did not find any domain dissimilar from the others.

---

Published as a conference paper at COLM 2024
APE (COMET-22)*
GEC (ER) Multilingual
NER (FI)1
Models
en en- XX-ten
Multilingual
Baseline (no edits)
76.80
79.99
16.66
Closed
GPT--.5-turbo GPT-4
81.47
78.68
15.06
50.22
85.20
84.30
15.08
59.88
Open
LLaMA-2 70B
78.34
81.03
21.74
44.62
Mixtrra--Sx7B-Instruut
82.64
82.81
17.10
41.77
TOWERINSTRUCT 7B
82.69
81.56
15.13
71.68
TOWERINSTRUCT 13B
83.31
82.26
15.68
74.70
Table Results for translation-related tasks aggregated by language or language pair. Models with statistically significant performance improvements are grouped in quality clusters. We highlight the best ranked models bold and underline the best ranked open models. Since GEC is held out task, we evaluate all models with 5 in-context examples.
WMT 23 zh-ten
WMT 23 zh-
COMET-22 MT (Toww))
COMMET-22 (Tower- CPF-4)
0.5
75
30-
Tooer impruves tranelation quality
No cdit Intervention
-0.5
25
Toneer degrades
quality
-1.0
Tower only
GPT-1 only Edited by
Both
Model
COMET-22 original (NLLB)
Figure Left: Comparison between NLLB original translation quality and TOWERIN- STRUCT 13B post-edition quality on WMT23 zh eh. Marker size and hue represent the translation quality difference. Right: Post-edition quality difference on WMT23 zh-ten when only TOWERINSTRUCT 13B edits (o), only GPT-4 edits (o), or both models edit (O). The bar the right represents the percentage of instances corresponding to each case.
3.3 Translation-Relater Tasks
Table reports aggregated results for all translation-rellted tasks."
TOWERINSTRUCT is an effective translation post-editor. TOWERINSTRUCE outperforms open models and GPT-3.5-turbo on APE, consistently and significantly improving the quality of NILB 3.3B translations. However, GPT-4 is still the top performer on this task. Yet it also has higher edit rate, shown in Figure 5: while TOWERINSTRUCT edits 30% of the instances, GPT-4 edits almost 90%. This tendency suggests that GPT-4 is over-editing, which we further analyze in Appendix B. We posit that TOWERINSTRUCT edits less due to the prevalence of unedited segments in TOWERBLOCKS - roughly 38%.
There is room for improvement on grammatical error correction. On this task, no model significantly outperforms the others majority of languages. We hypothesize that Tow- ERINSTRUCT's average performance is caused by the absence of GEC data in TOWERBLOCKS.
AApendix H details evaluated languages and provides further results for APE and GFC. Ap- pendix C contains preliminary results for MT evaluation.
7

---

Published as a conference paper at COLM 2024
MT (en-
MT (xx-+en
APE (en++xx)
APE (xx xX-)en)
90
90
84
84
COME1-22 22
COMET-22 22
COMET- 22
COMET-22 22
88
88
81
81
86
78
84
75
7B
13B
7B
13B
7B
13B
7B
13B
Model size
Model size
Model size
Model size
(# parameters)
(# parameters)
(# parameters)
(#f parameters)
TOWERINSTRUCT
TOWERBASE
ILaMA-2 Base
Figure 6: Recipe ablation across TOWER scales on MT (FLORES-200) and APE for en-*x and xx- directions. We evaluate pretrained models (ToWER and LLaMA-2) in a 5-shot setting and TOWERINSTRUCT in 0-shot fashion.
FLORES-200 xxx
FLORES-200 (xx xe))
88
88
COMET-22 22
87
87
86
86
85
85
84
84
1
2
3
4
5
6
7
8
1
2
3
5
7
Training tokens (billions)
Training tokens (billions)
TOWERBASE
Parallel only
Monolingual only
TOWERBASE @ 20b
Monolingual only @ 20b
LLaMA-2
Figure 7: Translation quality on FLORES-200 for continue pretraining data recipes. The TOWERBASE recipe, outlined in Section 2.1, mixtures monolingual with parallel data. The "Parallel only" recipe only processed 8 billion tokens due to compute constraints.
TOWERINSTRUCT can identify named entities. TOWERINSTRUCT 13B shows promising performance on NER, surpassing GPT-4 by about 15 F1 points. Similar to APE, TOWERIN- STRUCT 7B reflects most of these improvements, highlighting its capabilities at a smaller scale. Other open models do not perform well on this task, even with in-context examples. We hypothesize these results stem from NER being token-level classification task, as opposed to a generative one. While the models learn the output format from the examples or task description, they struggle to grasp the classification function itself. Conversely, TOWERINSTRUCT learns the task from the NER data TOWERBLOCKS.
4 Dissecting the training recipe
We performed multiple ablations to provide insights the impact of the several design choices made in the development of the TOWER models.
Continued pretraining and supervised finetuning yield independent performance gains. The two leftmost plots of Figure illustrate translation quality after continued pretraining and supervised finetuning. Both steps bring performance improvements at both model scales. Remarkably, TOWERBASE 7B and TOWERINSTRUCT 7B outperform LLaMA-2 13B, and TOWERINSTRUCT 7B outperforms ASSE 13B. The two rightmost plots analyze APE. For this task, while supervised finetuning yields better performance, continued pre- training and in particular parallel data does not improve performance observed for translation. In future work, we would like to explore additional training signals during continued pretraining to increase performance on translation-related tasks.
8

---

Published as a conference paper at COLM 2024
MTT
APET
GEC. Multilingual
NER Multilingual
Models
en -X
Xx-ten
en eX-Xxx
XX -
LLaMA-2 7B
84.23 87.46
87.10 88.02
76.56 76.79
79,91 79.83
15.95
20.09
TOWERBASE 7B
15.41
20.51
Supervised Finetuning
+ MT
88.45
88.28
79.19
79.36
54.76
0.00
+ Pre-MT + Post-MT
87.92
87.96
81.95
81.73
17.44
74.92
+ Gereral-Purpose 88.51
88.27
82.69
81.56
15.13
71.68
Table 4: Ablation results for the components of TOWERBLOCKS. Results for pretrained models are obtained with 5 in-context examples while results for supervised models are obtained in a 0-shot setup. We consider FLORES-200 to evaluate translation quality.
Parallel data during continued pretraining improves translation quality. Figure 7 re- ports 5-shot translation quality on FLORES-200 for multiple continued pretraining data recipes. Mixing monolingual and parallel data achieves the highest quality, outperforming both monolingual-only and parallel-only data mixes. In general, improvements are more noticeable on en-*xx directions, likely due to the English-centric nature of LLaMA-2's training. Nevertheless, while monolingual-onfy data improves over the base LLaMA-2 by 0.1 COMET-22 points on xx-tn directions, our recipe gains nearly full point. 10
Parallel data during continued pretraining sample efficient, but quality continues to improve with more tokens. At the 2 billion token mark, combining parallel sentences with monolingual data (i) yields more than 50% of the improvement over the base model, and (ii) surpasses the recipe leveraging solely monolingual data. Additionally, although training on more tokens has diminishing returns - 85% of the total performance gains appear by the 5 billion token mark it continues to improve translation quality.
Transfer/inteffernnce relations between tasks are complex. Table 4 ablates the compo- nents of TOWER OOCKS, comparing finetuning on translation data, translation-related tasks including pre- and post-translation, and the full dataset with general general-purpose tasks. While adding translation-relate Remarkably, introducing general-purpose instructions recovers translation quality. In future work, we would like to explore transfer maanserrintteffe interferrecce between tasks using scaling laws.
tasks improves their performance, it decreases translation quality.
The TOWER recipe generalizes to other model families. When newer LLMs become available, opportunities for improving TOWER naturally arise. In Table we compare Mistral 7B against LLaMA-2 as a backbone for TOWER. Notably, applying the TOWER recipe to Mistral outperforms starting from LLaMA-2 7B across the board and is competitive with using the larger LLaMA-2 13B, showcasing the generalizability of the recipe."
Related Work
Adapting open models to single tasks within the field of machine translation is competitive with closed models dedicated systems (Xu et al., 2024a; 2023; Iyer et al., 2023). Notably, Xu et al. (2024a) adapt LLaMA-2 for translation with continued pretraining on monolingual data and finetuning on high quality parallel data. Our work adopts similar approach, but introduces parallel data during continued pretraining and leverages LLMs' instruction- following capabilities to build system supporting multiple translation-related tasks.
While 0.1 COMET-22 points translates to 54.9% human agreement, one COMET-22 point translates to 90.9% (Kocmi et al., 2024).
We also release our TOWER models based on Mistral.

---

Published as a conference paper at COLM 2024
MT
APE1
GECL Multilingual
NER
Backbone Models
en XXX
XX->en
en-*xx
XX -
Multilingual
LLaMA-22 7B
88.51
88.27
82.69
81.54
15.13
71.68
LLaMA-2 13B
86.88
88,47
83.31
82.26
15.68
74.70
Mistral
88.98
88.44
83.05
81.95
14.70
72.54
Table Results for translation-related tasks aggregated by language or language pair among different backbone models trained on the TOWER recipe. We consider FLORES-200 to evaluate translation quality.
Multilinguality in LLMs. Previous work building more multilingual LLMs either trains new model "from scratch" (Üstün et al., 2024; Faysse et al., 2024; Wei et al., 2023), extends the pretraining of an existing model, possibly with vocabulary extension (Cui et al., 2023; Xu et al., 2024a; Pires et al., 2023). Building upon the effectiveness of pretraining (Anil et al., 2023; Wei et al., 2023), we include parallel sentences during continued pretraining. We do not extend the vocabulary, as preliminary experiments yielded negative results.
parallel data
Specialization of LLMs. Recent research specializing LLMs closely-related tasks leverages domain-specific data to train new models "from scratch" (Taylor et al., 2022; Wu et al., 2023), extend the pretraining of existing models (Lewkowycz et al., 2022; Chen et al., 2023), perform supervised finetuning (Yue et al., 2024), or combination of the last two (Rozière et al., 2023; Liu et al., 2023). Our specialization approach is broadly inspired by instruction tuning (Wei et al., 2022; Sanh et al., 2022; Wang et al., 2023; Zhou et al., 2023), which finetunes language models tasks formatted natural language instructions.
Conclusion
We propose a new recipe for specializing LLMs on multiple translation-related tasks. First, we expand LLaMA-2's multilingual capabilities with continued pretraining on a highly multilingual corpus. Then, we finetune on a dataset of high-quality and diverse instructions for translation-related tasks. Our final model consistently outperforms open alternatives on multiple translation-relate tasks, and competitive with closed models such as GPT-4.
Acknowledgments
We thank António Farinhas and Manuel Faysse for the fruitful discussion throughout the project. Part of this work was supported by the EU's Horizon Europe Research and Inno- vation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC-2022-C88 101088763), by the Portuguese Recovery and Resilience Plan through project C645008882- 00000055 (Center for Responsible AI), and by Fundação para a Ciência e Tecnologia through contract UIDB/5000S, 2020. We also thank GENCI-IDRIS for the technical support and HPC resources - Jeanzay grants 101838, 103256, 103298 and Adastra grants C1615122, CAD14770, CAD15031 used to partially support this work.
Ethics Statement
Our work builds upon L.LMs to develop an instruction-ffllooiing model centered around translation. The usage of LLMs encompasses several risks, which are thoroughly discussed in Brown et al. (2020) and Chowdhery et al. (2022). While we performed extensive evalu- ation our models, we focused on performance metrics and did not include toxicity or safety analysis. Additionally, we leveraged automatic metrics which are trained on human annotations and can inherit issues from mislabeled data. Annotators fail to consider better alternatives when presented with a given text or wrongfully classify text as high quality (Bansal et al., 2021).
10

---

Published as a conference paper at COLM 2024
References
Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Franscisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. TICO-19: the translation initiative for COvid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, December 2020. Association for Computational Linguistics. URL https://aclanthhlo// org/2020. m1ppovid1ccv.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez
Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vlad Feinberg, Faagxiioyuu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. arXiv preprint XXio:2305.10403 2023. URL https://arxiw.orr/bbbe wttp:or///./...../// 10403.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Sheng- guang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. Xrv preprint Xix::309..1660) 2023.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2021. Association for Computing Machinery. URL https://doi http:////ooooor/////. 10te//1..145555 1145/341111111 764. 3445717.
Eleftheria Briakou, Colin Cherry, and George Foster. Searching for needles in haystack: On the role of incidental bilingualism in palm's translation capability. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. URL https://aclanthologs. org/ www.o/gg........oooo/ acl acl-long. 524.525.. pdff
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner,
11

---

Published as a conference paper at COLM 2024
Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language mod- els are few-shot learners. In Advances in neural information processing systems. Cur- ran Associates, Inc., 2020. URL Paper.pdf. (tt.offfcbb4771188888881111
Christopher Bryant, Mariano Felice, and Ted Briscoe. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, July 2017. Association for Computational Linguistics. URL hatp:://aalanthologyy P17-1074.
Angel Alexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Talwalkar, Jason I. Hong, and Adam Perer. Zeno: An interactive framework for behavioral evaluation of machine learning. In CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2023. Association for Computing Machinery. Utthttss:///of 3544548. 3581
1145/
Alejandro Hernández Cano, Matteo Pagliardini, Andreas Köpf, Kyle Matoba, Amirkeivan Mohtashami, Xingyao Wang, Olivia Simin Fan, Axel Marmet, Deniz Bayazit, Igor Krawezuk, Zeming Chen, Francesco Salvi, Antoine Bosselut, and Martin Jaggi. epflm megatron-llm, 2023. URL https: //githbb. comm/ppfLLM/MMgarronroo
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron-70b: Scaling medical pretraining for large language models. Xiv preprint ar ari::33311 16079, 2023. URL https: http//axiv..g/aaaaaaeee org/abs/////oooooooooo...oo.. http/bbb//311.... 16079.
Shamil Chollampatt and Hwee Tou Ng. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAT Symposium on Educational Advances in Artificial Intelligence. AAAI Press, 2018 URL https: /dl. www..org//oo//// 5555/ 3504035443330 3504741.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanu- malayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellssern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arv preprint ar arXiv:2402.007766 2022. URL https: http:::///axxxv.orrr.. /tgp///////.......// 00782
Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stéphan Clémençon. What are the best systems? new perspectives on nlp benchmarking, In Advances in Neural Information Processing Systemis, 2022. URL https:
orgooggg//goo abs/ 03799.
Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. Xiv preprint ar arXio:2304.08177, 2023. URL https://arxiv. orgg / 2304.08177.
Anna Currey, Maria Nadejde, Raghavendra Pappagari, Mia Mayer, Stanislas Lauly, Xing Niu, Benjamin Hsu, and Georgiana Dinu. MT-GenEval: A counterfactual and contextual dataset for evaluating gender accuracy in machine translation. In Proceedings of the 2022
12

---

Published as a conference paper at COLM 2024
Conference Empirical Methods in Natural Langwage Processing Association for Computa- tional Linguistics, December 2022. URL https://aaxae wttp:/iww...rpppf///////// .11 355. pd.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics. URL https://aclanttholog..oor org/2023. mmlp-mainn. 1aaiii..11... 183.
Bryan Eikema and Wilker Aziz. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Compu- fational Linguistics, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. URL https::/aalaanthology wrg.oogg////// 20200.2020.22 coling-man ielinngmmin... main. 398.
Andreas Eisele and Yu Chen. MultiUN: A multilingual corpus from united nation docu- ments. In Proceedings of the Seventh International Conference on Language Resources and Eval- uation (LREC'1O), Valletta, Malta, May 2010. European Language Resources Association (ELRA). URL http: http://wwwwwweccccc http://reecccco..ooo//rrreeeedee orggproceedingg//rec220 //dp///0000p//////////////////// / .pdf.
Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. CCAligned: A massive collection of gross-lingual web-document pairs. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. URL https: https://aclanthologgy org/2020. emnlp manlp-main... 480.
Miquel Esplå, Mikel Forcada, Gema Gamirez-Sáncheer and Hieu Hoang. ParaCrawl: Web- scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, Dublin, Ireland, August 2019. European Association for Machine Translation. URL
.rg org.:/9--722122222
Europat. Europat. europat.net/.
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Croissantllm: A truly bilingual french-english language model. Xiv preprint Xiv:2402.00786, 2024. URL
http:////axxv..o//bbbbbs 240... 00786.
Christian Federmann, Tom Koemi, and Ying Xin. NTREX-128 news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, Online, nov 2022. Association for Computational Linguistics. URL https: //aclanthology. Sitiaanthology.or/22222 suwevall1ueaa
Mariano Felice, Christopher Bryant, and Ted Briscoe. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https: pnttps://achee /ccaaahhooatoo...... org/C16-1079.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Gra- ham Neubig, and Andre Martins. Quality-aware decoding for neural machine transla- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, July 2022. Association for Computational Linguistics. URL https: /ttp://allaathoogyyyyy org org//022.. naacl-main. 100.
Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference Machine Translation, Singapore, December 2023. Association for Computational Linguistics. URL https:/ htat::://accaathooggg.o org/2023. wmt-1 100.
13

---

Published as a conference paper at COLM 2024
Besnik Fetahu, Zhiyu Chen, Sudipta Kar, Oleg Rokhlenko, and Shervin Malmasi. Mul- tiCoNER v2: large multilingual dataset for fine-grained and noisy named entity recognition. In Findings of the Association for Computational Linguisfics: EMNLP 2023, Singapore, December 2023. Association for Computational Linguistics. URL https: //aclanthology or org/2023. findings- eenps..-ellp.... 133.
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10, 2022. URL https://aclantholl. org/ 2022. 2022.tac1-1.04 tac1-1.47.
Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Confer- ence on Machine Translation, Singapore, December 2023, Association for Computational org/2023. 1m1-1.51.
Linguistics. URL https::/aclantttot
Google DeepMind Gemma Team. Gemma: Open Models Based on Gemini Re- search and Technology, howpublished = h potgg////ecooooog/////////////// gemma-open- models/, note Accessed: 2024-02-27, 2024.
Glaive AI,
2023. saaive--ode-assistaan
URL
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measure- ment scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperabilitty with Discourse, Sofia, Bulgaria, August 2013. Asso- ciation for Computational Linguistics. URL https: https:://ccaanthoogg// org//1 2305.
Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. xCOMET: Transparent machine translation evaluation through fine-grained error detection. ar preprint ar arXiv:2310.10482 2023. URL httpp::/arrxvv.org/abb//// 10482.
Kenneth Heafield. KenLM:
Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https: http://wwwwcccccc/ccc aclweb. orgoorg////oo /atthology/ W11-2123.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? comprehensive evaluation. arXiv preprint ar xrXie:2302.09210, 2023. URL https://arxivv www.ooogg/////e.e absp//as./222//0// 09210.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL
https://opennee wttpeeewwwwweeefomu/inmiiiiae newt/forummdddryyoorrrr.
Vivek Iyer, Pinzhen Chen, and Alexandra Birch. Towards effective disambiguation for machine translation with large language models. In Proceedings of the Eighth Conference on Machine Translation, Singapore, December 2023. Association for Computational Linguistics. URL htt.httss//aaaaanhhllooo org 2223. 1 44.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. aro preprint arX1o=2310.06825, 2023. URL htttp:://axx: http:/:///axxvvor///al// abspa///.//eeeee
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian
14

---

Published as a conference paper at COLM 2024
Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Tim- othée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint ar (xix:2401.04000 2024. URL https: /awxxv/rivvv//a////////ea//aaav/ wwwprwwwwww.ooooooeeeeeeeeoeeeee abwr///bs//44..... 04088.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Infer- national Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015. URL https:/ / /wvvvv..ivvaaa//i///i/////a org/abs/ 11121 6980.
Tom Kocmi and Christian Federmann. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference Machine Translation, Singapore, December 2023. Association for Computational Linguistics. URL org/2023. wmi wwmtt11.. 1.64
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondffj Bojar, Anton Dvorkovich, Chris- tian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Ken- ton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, Singapore, December 2023. Association for Computational Linguistics. URL nttps://aclanthole
org/2023. wmt-1.1.
Tom Kocmi, Vilém Zouhar, Christian Federmann, and Matt Post. Navigating the metrics maze: Reconciling score magnitudes and accuracies. arXiv prepring arXio:2401.06760, 2024.
URL httpss://arrivv.org//////// .67660
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL attps://aclaanthooggy org/ w04-3250
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, Phuket, Thailand, 2005. URL https://aclanthhoogg.ooo///
org/2005 t-aupmmi--ppprs.11 11.
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural In- formation Processing Systems, 2022. URL https://openreview/ wtwpenrrwwieeeeeeeeeeeeeeeeeee met/Forum?idd-IXTZZEN..
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaroo Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, and Haoxing Ren. Chipnemo: Domain-adapted 1lms for chip design. arXiv preprin! ar arXiv:2311.0011 2023. URL /tttp:://arivv org/ hbtpa/as///1.e/ 00116.
Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumâtica: tecnologies de la traducció, 0, 2014.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th international conference on machine learning. PMLR, 2023. URL https: /ttpc//e//cocccooo./c/oo..oo/ mlr. pr22s/v202/ longpre23a. html.
15

---

Published as a conference paper at COLM 2024
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. Multi- CoNER: A large-scale multilingual dataset for complex named entity recognition. In Proceedings of the 29th International Conference Computational Linguistics, Gyeongju, Re- public of Korea, October 2022. International Committee on Computational Linguistics. org. www.ingl.nnn..
URL pattp:://aclanthollie
Thomas Mayer and Michael Cysouw. Creating massively parallel Bible corpus. In Proceed- ings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland, 2014. European Language Resources Association (ELRA). URL http://www.rree. conf org htte//occedings/ree22000 htteee222///dff/22oaaapapp 2ttp://22_aapprrrr .dff.
Mariana Neves, Antonio Jimeno Yepes, Aurélie Névéol, Rachel Bawden, Giorgio Maria Di Nunzio, Roland Roller, Philippe Thomas, Federica Vezzani, Maika Vicente Navarro, Lana Yeganova, Dina Wiemann, and Cristian Grozea. Findings of the WMT 2023 biomed- ical translation shared task: Evaluation of ChatGPT 3.5 as comparison system. In Proceedings of the Eighth Conference on Machine Translation, Singapore, 2023. Association for org/ 2023. wmt-1.2.
Computational Linguistics. URL https:
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, Baltimore, Maryland, 2014. Association for Computational Linguistics. URL https://aclanthho..og htt./1/-rgg/11-1111 ottps:///cccaattooooo..rr/111111
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Вapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scal- ing human-centered machine translation. ar Xiv preprint arXio:2207.0467. 2022. URL http:://arrdd..or//bb//2001 www.orggg////////// 04672..
Open AI, 2023. URL chatml.mc md.
Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraska, and Markus Freitag. There's no data like better data: Using QE metrics for MT data filtering. In Proceedings of the Eighth Conference on Machine Translation, Singapore, December 2023. Association for Computational Linguistics. URL https::/aclantthllll org.2023.. wmt-1. 50.
Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. Sabiá: Por- tuguese large language models. In Intelligent Systems, Cham, 2023. Springer Nature Switzerland. URL https:///inkk: wwiiiik..iinerr...... wwweeerrccm/eeo coom/chapter/11 10071078-3--0335 3-031-45392-2 15#tchpter-inno.
Maja Popović. chrF: the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal, 2015. Association for Computational Linguistics. URL https://aclantthol/
character n-gram F-score for automatic MT evaluation. In Proceedings of
5rg//15- 3049.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL https: //jmlr. waalr.orgg/peee / 0000000422000000 0.4.0d... pdf.
Gema Ramírez-Sánchez, Jaume Zaragoza- Berggaabe,,errabeuu..e Marta Bañón, and Sergio Ortiz Rojas. Bifixer and bicleaner: two open-source tools to clean your parallel data. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal, 2020. European Association for Machine Translation. URL https: //te:///ccllee http://aaaaannny.......o org/
2020.eamt-1. ea 31.
16

---

Published as a conference paper at COLM 2024
Vikas Raunak, Amr Sharaf, Yiren Wang, Hany Awadalla, and Arul Menezes. Leveraging GPT-4 for automatic translation post-editing In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 2023. Association for Computational Linguistics. URL htt.tttss//aaaannhol.gg..//// http://cccaanthoooooooo/////222 org/2023. findiiggs--n- emnlp. 804. 804.
Raj Reddy. Speech understanding systems: A summary of results of the five-year research effort at carnegie mellon university, 1977.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-1ST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Ma- chine Translation (WMT), Abu Dhabi, United Arab Emirates (Hybrid), 2022a. Association for Computational Linguistics. URL https: http:///aaaaatlloog.o/// org/2022. 1mtt-.552
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), Abu Dhabi, United Arab Emirates (Hybrid), 2022b. Association for Computational Linguistics. URL https: httpss//aalaatthooloo
wwg.//22........oo wmt-1.60 60.
Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, Singapore, 2023. Association for Computational Linguistics. URL https://aclanthonn
org/ /223. 13.-
Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. FRMT: A benchmark for few-shot region-aware machine translation. ar Xiv preprint arXiv:2210.00193, 2022. URL http:://arxxv..or///bb//// 00193.
Roberts Rozis and Raivis Skadinš. Tilde MODEL multilingual open data for EU languages. In Proceedings of the 21st Nordic Conference on Computational Linguistics, Gothenburg, Sweden, 2017. Association for Computational Linguistics. URL https: //aclanthology. orgg//7-20/1110235.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. ar Xiv preprint Xiv::2230 12950, 2023. URL /tttps://aaaiiii org/ htrggabb/2308.. 12950.
Victor M. Crinchez-Cartagnnn Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez. Promp- sit's submission to WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, Belgium, Brussels, 2018. Association for Computational Linguistics. URL https: http:://alaaathhhooooo org/W18 8- 8-6488.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chh- ablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Baw- den, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero- shot task generalization. In International Conference on Learning Representations, 2022. URL https://openrevviee met//oormmidd-Wrrsoooot.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arviv preprint ar Arxi::190770511 2019. URL https://arxxv, org. abss/190 009999177777...
17

---

Published as a conference paper at COLM 2024
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. Comatrix: Mining billions of high-quality parallel sentences on the web. arvio preprint 1XXi:1911.004444, 2020. URL https://arxiti
1b9/1991.04944
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020. Association for Computational Linguistics. URL https: // aclanthology. org/2020 acl-main. 704.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, Massachusetts, USA, 2006. Association for Machine Translation in the Americas. URL https://aclanthoology wrg/ aata-papers......... 25.
Felipe Soares, Viviane Moreira, and Karin Becker. A large parallel corpus of full-text scientific articles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language Resources Association (ELRA). URL https: htttps://acllanthologg//// 1-8.111156.
Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadin, Matteo Negri, and Marco Turchi. Translation quality and productivity: A study on rich morphology languages. In Proceedings of Machine Translation Summit XVI: Research Track, Nagoya Japan, 2017. URL
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arviv preprint arXiv:2211.09085 2022. URL https://arxixn wrw..rggg//./.... /
Jörg Tiedemann. The Tatoeba Translation Challenge Realisiic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, Online, 2020. Association for Computational Linguistics. URL https: acwweb. org org/anthology/ 2020. wmt-1. 139.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Proceedings of the eighth international conference on language resources and evaluation (LREC'12), Istanbul, Turkey, 2012. European Language Resources Association (ELRA). URL http://www 1rec-conf. Paper. .ddf.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arviv preprint arXic:2302.13971 2023a. URL 2202. https://arxiv. orgo 2rr02 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar Xiv:2307 09288, 2023b. URL https: /ttp//ixivvvoooo/oooooooo www.vorg//aao. wwt.//bb///b//////// 092288788..8.... 09288.
18

---

Published as a conference paper at COLM 2024
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023. URL https: //arxiv.org/abss: hbrrg/bbs/23101111gg
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbe- mileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilinguul language model. aiv preprint http://ww..rrgbb////////// 220. http://bbb//22222/22 07827.
arXio:2402.07827, 2024. URL
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- gelf-generated instructions. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Toronto, Canada, 2023. Association for Computational Linguistics. URL https: putttps:/aacllee /aclaanhooooogyyy... wwg.//rg........ooo acl aclooon..lloo..... 754.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreiins net/fforum2xd=gEEZrGGoood
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. PolyLM: An Open Source Polyglot Large Language Model. Xiviv preprint arXiv:2307.06018, 2023. URL http::////rriiv.orr abs/2307.060
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. ar Xiv preprint ar 1tt::1991100359 2019. URL htttps://ariiss org/abs/1911 00359.
Philip Williams and Barry Haddow. The elitr eca corpus. arXiv preprint arXiv:2109.07351 2021. URL https: / //wxxvixiii////////v/////////wi/ www.aiwwwwwwwaaa.aee/..eeee. htrg/9 07351.
Krzysztof Wolk and Krzysztof Marasek. Building subject-aligned comparable corpora and mining it for truly parallel sentence pairs. Procedia Technology, 2014. URL http: //dx. doi org/10. 1016/j.
101t/..procc...........oooooo prw.cy. 2014.
Shijie Wu, Ozan Trsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arviv preprint ar arXiv:2303.17564, 2023. URL https: org/abs/2303 17564.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https: //openreview.netee nwtp:/nee/eeeeeeee meet//orrmmidd=frTTTtN
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of ilm performance in machine translation. Xiv preprint XrXi:2200..08411 2024b. URL https://arxins www.:/oggg/b////// 2201 08417.
Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. INSTRUCTSCORE. Towards explainable text generation evaluation with automatic feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. Association for Computational Linguistics. URL https://aclanthologgy org/2023. emnlp-mate camp-mmain... 365.
19

---

Published as a conference paper at COLM 2024
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HLTT) Online, 2021. Association for Computational Linguistics. URL https://aclanthoology org/2021
Aaron Yamada, Sam Davidson, Paloma Fernández-Mira, Agustina Carando, Kenji Sagae, and Claudia Sânchez-Gutiérre.. Cows-I2h: A corpus of spanish learner writing. Research in Corpus Linguistics, 2020. URL https://ricl ael wel.ooo......ioo.o.o...ooooooo.. es/ htt//iddxxxpppp http:/pppp//ccrcccp../ view/109.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual ad- versarial dataset for paraphrase identification. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019, Association for Computational Linguistics. URL ht http://acllann haw.hooooy.nooooooooooooooooooo. hww.hhhooogooorg//11111 org/D19-1382
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. TA Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview met/FForuummddyyL.GG6771....
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilin- gual neural machine translation and zero-shot translation. arXiv preprint ar arXiv:2004.11867, 2020. URL https://arxiii http://wwwwrg/b//////e/// 11867.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint ar 240.:24022048333, 2024. URL https://arxiv. http:///////xxiooo////b/ss 2402. 04833.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-sevntth Conference Neural Information Processing Systems, 2023. URL https: htttp:://ppeneeee htwpss/////ereeew.eeeeeeee KBMOKmX2he.
Michat Ziemski, Marcin Junczys-Dowmune and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Re- sources and Evaluation (LREC'16), Portorož, Slovenia, 2016. European Language Resources Association (ELRA). URL https: /ac.antoooogyyy...... wwwphhhooogo.ooooo...ooooo 15661111.
20

---

Published as a conference paper at COLM 2024
A
Analysis of alternative decoding strategies
FLORES-200 - xx-te en en->xx xx-tn
WMT 23
TICO 19
Models
en -
GPT-3.5-turbo
77.08
78.12 78.51
72.06
72.50
75.91
GPT-4
77.26
72.54
72.91
76.16
TOWERINSTRUCT
13B
Greedy
76.89
78.67
70.87
71.75
75.40
Beam
77.40
78.87
71.31
71.88
75.66
MBR
77.79
78.96
72.29
72.36
76.13
Table 6: Impact of beam search and minimum Bayes risk (MBR) decoding in translation quality for TOWeRINSTRUCT 13B. In bold, we highlight systems in the first quality cluster. For TICO-19 there is no first cluster since no model significantly outperforms the others a majority of the language pairs.
In this section, we analyse the performance of TOWERINSTRUCT 13B with beam- search (Reddy, 1977) using beam size of 5 and minimum Bayes risk (MBR) decoding (Eikema & Aziz, 2020; Fernandes et al., 2022; Freitag et al., 2022) with 20 hypotheses and COMET-22 as an utility function. We generate hypotheses using temperature and nucleus sampling (Holtz- man et al., 2020), with = 0.9 and 0.6. We avoid "optimizing" the evaluation metric (Fernandes al., 2022) by measuring translation quality with BLEURT.
Table reports translation quality across all test sets. Both decoding strategies consistently improve translation quality over greedy decoding, with MBR decoding achieving higher quality, Additionally, for both WMT23 and 1ICO-19, decoding strategies close the gap to GPT-4. Notably, on FLORES-200, TOWERINSTRUCT 13B appears isolated in the first cluster.
B Further analysis on TOWERINSTRUCT and GPT-4 editing tendencies
Figure shows that differences between GPT-4 and TOWERINSTRUCT edit rates are not strongly correlated to differences in COMET-22 (0.34 Spearman p). This means that GPT- edits often do not correspond to gains in performance. This finding, allied with the discussion in Section 3.3 about GPT-4 editing considerably more than TOWERINSTRUCT, suggests that GPT-4 may be editing too much.
WMT
10
J.CEETT-2 (Tooe: -PPS4)
-05
-00
-300
-200
-100
100
300
300
400
A.Edd C17T-1
Figure 8: Difference between TOWERINSTRUCT 13B and GPT-4 edit rate (compared to the original NLLB translation) (x-axis), and difference between TORERINSTRUCT 13B and GPT-4 post-edition COMET-22 (y-axis). The correlation between the two variables is 0.34 Spearman p. Similar patterns are observed for other language pairs.
21

---

Published as a conference paper at COLM 2024
WM123
Models
en-*de
zh-tn
COMET-22
0.3819
0.2731
TOWER
0.3994
0.3292
Table 7: Kendall 7 correlation of reference-based MT evaluation metrics with human judge- ments on WMT23 data.
C TOWERINSTRUCT as an evaluator of translation quality.
We included evaluation data in TOWERBLOCKS to induce positive transfer to machine translation. Nevertheless, recent work has shown that LLMs can also function estimators translation quality (Kocmi & Federmann, 2023; Fernandes et al., 2023). Table 7 shows preliminary experiments where the evaluation data in TOWERBLOCKS improved. Notably, this version of TOWER is competitive with COMET-22 on en dde and zh-ten, suggesting that the TOWER recipe can be used to create models evaluate translation quality. In the future, intend to further explore how to improve evaluation capabilities of TOWERINSTRUCT.
D Details of continued pretraining
This section contains some details the continued pretraining process used to obtain TOWERBASE. Training time was 10 and 20 days for the 7B and 13B models, respectively.
D.1 Filtering thresholds
In Table 8, we report the perplexity floors and ceilings used to filter the monolingual data in the continued pretraining corpus, well as the Bicleaner and CometKiwi-22 thresholds used to filter the parallel data.
D.2 Parallel data sources
Parallel data was obtained from the sources listed in Table 9.
de
fr
nl
pt
ru
zh
ko
Min. perplexity *
50
50
50
50
50
50
50
50
50
Max. perplexity
516
611
322
649
275
257
334
2041
198
Bicleaner t
0.5
0.5
0.5
0.5
0.5
0.5
0.0
0.5
COMETKIW1-22
0.75
0.75
0.75
0.75
0.75
0.75
0.75
0.75
Table 8: Quality filtering thresholds applied on monolingual data (x) and parallel data (+)
by language. On the latter, the to-English language pair's threshold is the same as the corresponding from-English one.
22

---

Published as a conference paper at COLM 2024
Dataset
Version
Europarl (Koehn, 2005)
v8
ParaCrawl (Esplà et al., 2019)
v9
MultiParaCraw/ (Esplå al,, 2019)
v7.1
CCMatrix (Schwenk et al., 2020)
v1
CCAligned (El-Kishky et al., 2020)
v1
MultiCCAligned (El-Kishky et al., 2020)
v1
WikiTitles (Tiedemann, 2012)
v2014
WikiMatrix (Schwenk et al., 2019)
v1
(Tiedemann, 2012)
v16
OPUS100 (Zhang et al., 2020)
v1
TildeModel (Rozis & Skadinš, 2017)
v2018
Bible (Mayer Cysouw, 2014)
vl v14.10
Ubuntu (Tiedemann, 2012)
Tatoeba (Tiedemann, 2012)
v2
GNOME (Tiedemann, 2012)
v1
GlobalVoices (Tiedemann, 2012)
v2018q4
KDE4 (Tiedemann, 2012)
v2
KDE-Doc (Tiedemann, 2012)
v1
PHP (Tiedemann, 2012)
v1
Wikipedia (Wolk & Marasek, 2014)
v1.0
Wikimedia (Tiedemann, 2012)
v20210402
JRC (Tiedemann, 2012)
v3.0
DGT (Tiedemann, 2012)
v2019
EuroPat (Europat)
v3
EUbookshop (Tiedemann, 2012)
v2
EMEA (Tiedemann, 2012)
EUConst (liedemann, 2012)
v1
tico-19 (Anastasopoulos et al., 2020)
v20201028
FCB (Tiedemann, 2012)
v1
Elitr-ECA (Williams Haddow, 2021)
v1
MultiUN (Eisele Chen, 2010)
v1
OpenOffice (Tiedemann, 2012)
V3
Ada83 (Tiedemann, 2012)
vI
infopankki (Tiedemann, 2012)
vI
Scielo (Soares al., 2018)
giga-fren (Tiedemann, 2012)
V2
UNPC (Ziemski et al., 2016)
v1.0
Table 9: The various data sources used to create the parallel data with the number of available language pairs.
23

---

Published as a conference paper at COLM 2024
E
Details of TOWERBLOCKS
This appendix details all datasets utilized in TOWERBLOCKS:
WMT14 to WMT21 12 Evaluation sets for the general machine translation shared task;
WMT22 with quality-shots (Hendy et al., 2023) Evaluation set from WMT23 with high quality in-context examples;
NTREX (Federmann al., 2022) Professional translations of the WMT19 test set;
FlOREs-200 (NLLB Team et 2022) Development set of the FLORES-200 dataset for all languages included in training;
FRMT (Riley al., 2022) - Human translations of English Wikipedia sentences into
regional variants;
OPUS (Tiedemann, 2012) - Parallel corpora from which we sampled very high-quality samples for all language pairs;
QT21 (Specia et al., 2017) and ApeQuest Translation data with post-edits utilized for general translation and automatic post-editing;
MT-GenEval (Currey al., 2022) Gender translation benchmark which we leveraged for general translation and context-aware translation;
WMT20 to WMT22 Metrics MQM MT evaluation data annotated with multidi- mensional quality metrics (Lommel et al., 2014) that we used to perform error span detection;
WMT17 to WMT22 Metrics DAs - MT evaluation data annotated with direct assesse- ments (DAs) (Graham et al., 2013) which we utilized for translation ranking.
WMT21 Terminology 16 Development set for the WMT21 terminology task;
Tatoeba (Tiedemann, 2020) Development set of the Tatoeba dataset which we used to generate translations in different languages for the same source we identified this task as multi-reference translation;
MultiCoNER 2022 and 2023 (Malmasi et al., 2022; Fetahu et al., 2023) - Development sets of the named entity recognition MultiCoNER datasets. For MultiCoNER 2023, adopted the coarse barse-grained entity categorization;
PAWS-X (Yang et al., 2019) - Development set the PAWS-X dataset which we used as paraphrase generation;
UltraChat (Ding et al., 2023) Filtered version of the UltraChat dataset used in Tunstall al. (2023);
Glaive Code Assistant (Glaive AI, 2023) Coding questions and answers across wide range of programming languages.
stammt. stt.ttt..rg//mmm2/rrrsssatttrooo html
//nttps://apequett.
http://www.aatt...oommm///////// html
thttopss/www..taanttorro//t//2/: html
phttppss/ww..taatttoo///mm///// task. html
24

---

Published as a conference paper at COLM 2024
F Details of TOWERINSTRUCT
This appendix further details the supervised finetuning procedure to train TOWERINSTRUCT.
E1
Chat Template
We finetuned TOWERINSTRUCT with the chatml template (Open AI, AT,2023) Table 10 provides example of an interaction using the aforementioned template.
User

<| immmimmts. start puserr
Translate the following text from Portuguese into English.
Portuguese: Ontem, minha amiga foi ao supermercado mas estava fechado. Queria comprar legumes fruta.
English: im_end|>
<I im_starr |>assistant
Model
Yesteday, my friend went to the supermarket but it was closed. She wanted to buy vegetables and [ruit.</im_ende)
User
<| imrstart >suser
Can you now translate into Spanish? <| <minendd>>
<I immsstrrt [>assistane
Model
Ayer mi amiga fue al supermercado, pero estaba cerrado. Quería comprar verduras y fruta.< im_end]>>
Table 10: Example of dialogue with TOWERINSTRUCT'S user and model control tokens.
We avoid the separation <mim start 
 and </im_end|> into multiple tokens by extending the tokenizer for TOWERINSTRUCT with two dedicated tokens. We do not explicitly add new tokens for user and assistant, as both strings already have dedicated tokens. Additionally, overwrite the end-of-sequence token with the <| im_end| token.
Hyperparameters
Table 11 details the hyperparameter configuration for TOWERINSTRUCT'S training. We also utilized bfloatl6 mixed precision and packing.
Global train batch size
256
Number of Epochs
Learning rate
7e-6
LR Scheduler
cosine
Warmup Steps
500
Weight Decay
0.01
Optimizer
Adam (Kingma & Ba, 2015)
Adam B1
0.9
Adam $2
0.999
Adam
1e-8
Maximum Sequence Length
2048
Table 11: Hyperparameter configuration to finetune TOWERINSTRUCT on TOWERBLOCKS.
25

---

Published as a conference paper at COLM 2024
G Translation full results
Tables 12, 13, 14 and 15 report translation quality on all test sets using various metrics: XCOMET, COMETKIW1-22, BLEURT, and CHRF. Tables 16, 17, 18 and 19 analyse translation quality for our languages, considering en-xx and xx-en translation directions, for the same metrics. On Tables 20, 21, and 22, we present translation results for a wider variety of models, broken down by language pair. Importantly, quality trends hold across metrics. TOWERINSTRUCT 13B is the open system with highest translation quality and is competitive with the closed model GPT-4.
FLORES-200
WMT 23 xX-en
TICO
Models
en en-xx
XX-en
en
en-*xX
Closed
GPT-3.--turbo
94.41 2
95.54
88.99
89.75 2
91.19 2
GPT-4
94.75
96.01
89.46
90.28
91.38 2
Open
NLLB 54B
90.04
93.78 4
78.99
81.38
90.11 3
LLaMA-2 70B
92.80 4
94.15 4
84.85 6
87.21 5
89.02 5
Mixtral-Sx7B-Insiiee
91.90 3
94.40 3
85.67 6
87.81 4
89.30 4
ALMA-R 7B
86.50 4
87.67 4
ALMA-R 13B
88.88 2
88.97 3
TOWERINSTRUCT 7B
93.85 2
94.67 3
87.20 4
87 $7.88 4
90.56 3
TOWERINSTRUCT 13B
94.80
95.22 2
88.71 2
88.65 3
91.30 2
Table 12: Translation quality on WMT23 and TICO-19 by language pair measured by XCOMET. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
FLORES-200
WMT 23 xx-+en
TICO 19 en-+XX
Models
en=+xx
xX-tn
en-+xx
Closed
GPT-3.5-turbo
86.25 2
85.64 2
80.82 2 81.20
80.35 2
85.65 2
GPT-4
86.42
85.77
80.54
85.79 2
Open
NLLB 54B
82.93 5
84.89 4
70.96 6
76.69 5
85.16 3
LLaMA-2 70B
85.30 4
84.97 4
78.43 5
79.36 4
84.66 5
Mixtral-6x7B-Ineee Instruct
85.24 3
85.32 3
79.01 5
79.82 3
84.81 4
ALMA-R
79.25
79.79
ALMA-R 13B
80.12 3
80.21 2
TOWERINSTRUCT 7B
85.96 3
85.41 3
79.80 4
79.95 3
85.32 3
TOWERINSTRUCT 13B
86.19 2
85.51 2
80.57 2
80.25
85.59 2
Table 13: Translation quality WMT23 and TICO-19 by language pair measured by COMETKIWI-22 Models with statistically significant performance are grouped in qual- ity clusters. Best performing models are in bold and best performing open models are underlined.
26

---

Published as a conference paper at COLM 2024
FLORES-200
WMT 23
TICO
Models
en -Xx
XX xX-ten
en en-*xx
xX->en
en-XxX
Closed
GPT-3.5-turbo
77.08
78.12 78.51 2
72.06 2
72.50
75.91 2
GPT-4
77.26
72.54
72.91
76.16 2
Open
NLLB 54B
74.29 3
77.99 3
62.73 6
66.46 5
75.49 2
LLaMA-270E
75.04 4
78.28 2
68.03 5
71.01 3
74.00 4
MixtrallSx77-Instruutt
74.78 3
78.10 2
68.81 5
71.32 3
74.22 4
ALMA-R 7B
68.64 5
70.66 4
ALMA-R
70.09 4
71.473 3
TOWERINSTRUCT77 7B
76.10 76.89 2
78.26 2
69.77 4
71.11 3
74.83 4
TOWERINSTRUCT 13B
78.67
70.87 2
71.75 2
75.40
Table 14: Translation quality on WMT23 and TICO-19 by language pair measured by BLEURT. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
FLORES-
WMT 23
TICO
Models
en >xx
xx >en
en >XX
XX >en
en XXX
Closed
GPT-3.5-turbo
58.20
63.75
56.38
60.92
64.18 2
GPT-4
58.61
64.35 2
56.94
61.33
64.34 2
Open
NLLB 54B
54.70 4
63.87 2
42.98 6
52.08 6
63.84 2
LLaMA-2 70B
55.19 4
64.15 2
52.31 4
59.66 2
61.65 4
Mixtral-5x7B-Instruuet
54.50 4
63.38 3
51.22 4
58.63 4
61.34 4
ALMA-R 7B
45.20 7
57.33 4
ALMA-R 13B
46.52 6
58.37 3
TOWERINSTRUCT 7B
56.16
64.08
52.25 4
58.88 4
62.07 4
TOWERINSTRUCT 13B
57.19 2
64.79
54.10 3
59.78 2
62.81 3
Table 15: Translation quality on WMT23 and TICO-19 by language pair measured by CHRF. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
27

---

Published as a conference paper at COLM 2024
FLORES--00
WMT 23
TICO 19
Models
en en-
xx xx->en
en en-+xx
XX -x-
en-*xx
Closed
GPT-3.5-turbo
94.41 2
95.54
88.99 2
89.75 2
91.19 2
GPT-4
94.75
96.01
89.46
90.28
91.38
Open
NLLB 54B
90.04
93.78 4
78.99
81.38
90.11
LLaMA-2 70B
92.80 4
94.15 4
84.85 6
87.21 5
89.02 5
Mixtral-6x/7-Insttrue
91.90
94.40 3
85.67
87.81 4
89.30 4
ALMA-R 7B
86.50 4
87.67 4
ALMA-R 13B
88.88
88.97 3
TOWERINSTRUCT 7B
93.85 2
94.67 3
87 20 4
87 $7.88 4
90.56 3
TOWERINSTRUCT 13B
94.80
95.22
88.71 2
88.65 3
91.30 2
Table 16: Translation quality on FLORES-200 by language pair measured by XCOMET. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
FLORES-200 (en
>xx) nl
Models
de
fr
it
ko
pt
zh
Closed
GPT-335-turbo
85.15
87.04
87.18
87.47
86.92
86.88
85.69
85.58
2
84.37 2
GPT-4
85.27
87.07
87.25
87.51
87.47
86.90
85.68
85.99
84.68
Open
NLLB 54B
82.59 6
85.18 4
85.23 4
85.66 4
86.11 4
84.71
83.45 5
83.56 4
69.88 7
LLaMA-2 70B
84.19
86.403
86.68
86.77
85.46
85.87
84.57
84.59
83.13 5
Mixtral-8xx7B-Isttrae
84.72 3
86.74 2
87.04 2
87.18 2
83.49
6
85.95
84.99
84.78 3
82.30 6
TOWERINSTRUCT 7B
84.41
4
86.77
2
87 00
87.
07.319
2
86.70
86.48
85.57
85.50
2
83.78 4
TOWERINSTRUCT 13B
84.73
3
86.94
87.18
87.45
87 $7.22
86.60
85.85
85.68
84.09 3
FLORES-
(xx (xx-
Models
de
es
it
ko
ml
pt
nu
zh
Closed
GPT-3.5-turbo
84.64
86.27
86.48
86.84
2
85.69
86.18
85.31
84.59
84.76 2
GPT-4
84.71
86.39
86.50
86.95
86.15
86.25
85.31
84.75
84.92
Open
NLLB 54B
84.09 5
85.51
5
86.04 3
86.06 4
85.13 4
85.59
84.45
83.95 4
83.18 6
LLaMA-2 70B
84.29
4
85.78
4
86.05
86.38 3
84.45
85.56
84.87
83.77
4
83.57 5
Mixtral-8x7B-Inaa Instruct
84.45 3
86.07 3
86.34 2
86.78 2
84.74
5
85.78
85.13
84.45 3
84.14 4
TOWERINSTRUCT
84.41
3
86.12
3
86.35
86.79
85.21
85.98
85.17
84.47 2
84.16 4
TOWERINSTRUCT 13B
84.44
3
86.09
3
86.39
86.83
85,47
86.04
85.17
84.69
84.47 3
Table 17: Translation quality on FL.ORFS-22 2000s-200 by language pair measured by COMETKIW1-22. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
28

---

Published as a conference paper at COLM 2024
FLORES-200
(en
>xx)
Models
de
ko
nl
pt
zh
Closed
GPT-3.5-turbo
79.09
76.75
79.54
79.83
2
69.39
2
77.79
80.31
77.31
2
73.69 2
GPT-4
79.13
76.64
79.29
80.00 2
70.31
77.58
80.22
78.16
73.98
Open
NLLB 54B
77.71
3
75.37
4
77.96
79.26
68.95
76.47
77.80 4
76.81
3
58.32 6
LLaMA-2 70B
76.75
75.28
5
76.96 4
78.70 4
67.01
75.98
77.50
75.79
71.41 4
Mixtrrl-s&x7--Instruct
77.73
3
76.08
3
78.39
79.57
61.77
76.35
78.14 3
76.06 4
68.94 5
TOWERINSTRUCT 7B
77.61
3
75.71
4
78.03
3
79.58
3
69.25
77.73
78.43 3
77.02
2
71.53 4
TOWERINSTRUCT 13B
78.15
76.42
78.96
80.39
70.53
77.93
78.78
77.97
72.85 3
FLORES-200
(xx-ten)
Models
de
fr
it
ko
nl
pt
zh
Closed
GPT-3.5-turbo
80.38 2
77.27
3
80.55 3
77.91
3
75.22
3
77.02
80.86
77.73
3
76.12 2
GPT-4
80.74
77.61
80.72
78.14 2
76.51
77.23
81.11
78.02
76.54
Open
NLLB 54B
80.12 3
77.09
3
80.64 2
77.79
3
75.32
76.99
80.81
3
77.95
2
75.19 4
LLaMA-2 70B
80.38
77.65
80.79
78.05
75.58
76.77
81.16
78.18 18
75.96 2
Mixtral-s&x7B-Instruct
80.40 2
77.79
80.75
78.53
74.15
76.87
80.85 3
78.02 2
75.57 3
TOWERINSTRUCT 7B
80.17
3
77.47
80.67
78.40
75.62
76.96
81.30
78. 10
75.68 3
TOWERINSTRUCT 13B
80.55
77.65
81.03
78.54
76.53
77.22
81.51
78.51
76.46
Table 18: Translation quality on FLORES-200 by language pair measured by BLEURT. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
29

---

Published as a conference paper at COLM 2024
FLORES-200
(en
>xx)
Models
de
fr
ko
nl
pt
zh
Closed
GPT-3.5-turbo
67.22 2
57.39
72.79
60.67
35.49
59.57
72.96
58.48 2
39.21
GPT-4
67.89
57.13 2
72.89
60.60
37.18
59.97
72.98
59.50
39.32
Open
NLLB 54B
63.18 5
55.30
5
70.25
58.83 3
36.54
56.99
68.19
57.28 3
25.73 5
LLaMA-2 70B
63.43 5
55.39
5
69.54 4
58.20 3
32.07
56.53
69.61
56.58
35.38 3
Mixtrrl-s&x7B-Instruct
64.14 4
56.14
4
70.91
59.01 2
27.54
56.22
69.43
56.07
31.01 4
TOWERINSTRUCT 7B
63.87
4
56.04 4
70.23 3
59.45
35.44
58.16
68.74 4
57.
3
35.78 3
TOWERINSTRUCT 13B
65.16
3
56.58 3
71.26
60.32
37.10
59.04
69.06
58.77
37.40 2
FLORES-200
(xx-
Models
de
fr
it
ko
nl
pt
zh
Closed
GPT-3.5-turbo
69.31
60.46
3
69.54
62.76
3
57.50
3
60.75
72.56
62.80
3
58.07 2
GPT-4
69.74
61.09
69.94
62.75
59.55
60.88
72.91
63.40
58.87
Open
NLLB 54B
68.54 3
60.72
69.70
62.95 3
58.55
60.67
72.26 3
62.66 3
58.83
LLaMA-2 70B
69.22 2
61.34
70.08
63.51
57.82
60.90
72.96
63.61
57.94 2
Mixtral-s&x7B-Inttruct
69.00 2
61.29
69.32
63.38
55.56
59.98
72.18 4
62.77
3
56.97 3
TOWERINSTRUCT 7B
68.94
2
61.39
69.56
63.59
58.48
60.65
73.00
63.37
57.79 2
TOWERINSTRUCT 13B
69.39
61.50
70.07
64.06
59.81
61.40
73.54
64.41
58.90
Table 19: Translation quality on FLORES-200 by language pair measured by CHRF. Models with statistically significant performance are grouped in quality clusters. Best performing models are in bold and best performing open models are underlined.
30

---

Published as a conference paper at COLM 2024
FLORES-
200 (en (en++x)
Models
de
es
fr
it
ko
ml
pt
ru
zh
Closed
CPTT-35-tuubo
88.78
87.
89.02
89.06
89.36
88.63
90.46
89.56
88.58
GPT-4
88.98
87.10
88.93
89.05
90.06
88.56
90.43
90.19
88.87
Open
NLLB 54B
87.18
85,92
87.71
88.10
89.00
87.33
88.72
88.89
78,26
LLaMA-2 7B
84.03
84.37
85.18
85.18
80.20
84.48
87.01
85.09
82.50
LLaMA-2 13B
85.60
85.45
86.74
87.02
84.22
86.11
88.33
87.02
84.83
1aMMA-2 70B
87.31
86.41
87.82
88.22
88.07
87.47
89.11
88.65
87.32
Distral-7B-Instructt-022
84.27
84.87
86.16
85.86
79.20
84.43
87.53
85.78
82.41
Mixtral-8x77
87.95
86.64
88.39
88.44
85.72
87.26
89.34
88.89
86.23
Hintral-8x7B-Insttee
87.99
86.80
88.53
88.77
85.63
87.57
89.45
89.09
85.99
Qwenl.5 72B
87.20
86.46
87.78
88.19
87.64
87.40
89.13
88.41
88.85
Gemma 7B
86.13
85.84
87.09
87.03
84.89
86.03
88.60
87.24
85.75
COMMAND R
87.60
86.95
87.90
88.93
89.19
87.80
89.74
89.55
88.30
COMMAND R+
88.87
87.45
89.13
89.46
90.25
88.67
90.44
90.57
89.18
ALMA-PRETRAIN 7B
86.47
83.18
84.23
83.59
68.06
81.05
84.80
87.96
85,80
ALMA-PRETRAIN 13B
87.07
84.90
86.05
86.09
77.10
84.36
87.47
88.91
86.58
TOWER
TOWERBASE 7B
86.91
85.95
87.76
87.93
86.55
87.37
89.47
88.72
86.48
TOWERBASE 13B
87.21
86.01
88.34
88.25
88.78
87.52
89.36
88.30
87.14
TOWERINSTRUCT 7B
87.82
86.76
88.44
88.73
89.41
88.38
89.60
89.53
87 .00
TOWERINSTRUCT 13B
88.16
87.06
88.92
89.21
89.92
88.63
89.78
89.95
88.29
FLORES
(ox->en) ko
Models
de
es
fr
it
nl
pt
ru
zh
Closed
GPT--35-turbo
89.60
87.26
89.46
88.03
87.83
87.71
89.78
86.69
86.92
CPT-4
89.76
87.57
89.61
88.21
88.58
87.88
89.94
86.94
87.29
Open
NLLB 54B
89.17
87.25
89.29
87.91
87.86
87.49
89.38
86.66
86.55
LLaMA-2 7B
88.47
86.63
88.78
87. 4848
85.52
86.67
88.98
85.87
85.53
LLaMA-2 13B
89.01
86.98
89.14
87.87
86.95
87.23
89,26
86.37
86.35
LLaMA-2 70B
89.44
87.49
89.55
88.18
87.91
87.52
89.84
86.87
86,91
Distral-7B-Instruce-v1.
88,83
87.07
88.81
87 87.69
85.16
86.93
89.05
86.21
85.65
Mixtral-8x70
89.55
87.57
89.58
88.35
87.03
87.54
89.80
86.79
86.63
89.57
87.65
89.56
88.44
87.37
87.54
89.73
86.81
86.88
Qwen1.5 72B
89.67
87.66
89.58
88.41
88.42
87.72
89.88
87.13
87,94
Gemma 7B
89.17
87.09
89.12
87.81
87.28
87.23
89.48
86.59
86.59
COMMAND R
89.15
87.51
88.91
88.05
88.29
87.49
89.47
86.65
87.10
COMMAND R+
89.07
87.87
89.67
88.53
88.82
87.10
90.15
87.32
87.91
ALMA-PRETRAIH 7B
89.23
86.84
89.01
87.68
83.35
86.92
89.05
86.81
86.59
ALMA-PRETRAIN 13B
89.81
87.42
89.42
88.18
86.26
87.59
89.70
87.23
87.16
TOWER
TOWERBASE 7B
89.26
87.15
89.47
88.14
87.80
87.45
89.77
86.41
86.72
TOWERBASE 13B
89.54
87.42
89.55
88.11
88.24
87.61
89.71
86.18
87.02
TOWERINSTRUCT 7B
89.48
87.48
89 89.50
88.39
88.
87.66
89.92
86.90
86,96
TOWERINSTRUCT 13B
89.61
87.62
89.67
88.42
88.48
87.92
90.07
87.20
87 87.27
Table 20: COMET-22 on FLORES-200 for wide variety of models.
31

---

Published as a conference paper at COLM 2024
WMT23
Models
-de
en en-rr
en- en+2hh
de->en
TU ru->en
zh->en
Closed
GPI-3.5-turbo
84.61
85.38
86.70
85.91
83.02
81.52
GPT-4
84.89
86.07
87.08
86.17
83.63
81.27
Open
NLLB 54B
77.40
83.91
74.48
80.06
80.52
76.60
LLaMA-2 7B
75.02
77.87
79.16
83.36
80.58
77.40
LLaMA-2 13B
78.29
80.44
81.30
83.92
81.54
78.73
LLaMA-2 70B
81.62
83.04
84.19
85.12
82.84
79.73
Distral-7B-Insttuut--022
76.78
80.27
81.26
84.18
81.52
79.11
Mixtral-6x7B
81.92
83.39
83.81
85.04
82.70
79.50
Mixtral-8x7B- -x7B-Insttuct
83.07
83.79
83.94
85.45
83.02
80.04
Qwen1 72B
81.44
83.31
86.48
85.54
83.01
80.60
Gemma 7B
79.56
82.20
83.56
84.60
82.14
79.24
COMMAND
83.29
84.96
84.92
84.66
82.29
79.61
COMMAND R+
84.85
86.22
86.31
84.82
83.38
80.59
ALMA-PRETRANE
ALMA- PRETRAIN 7B
80.20
83.01
82.68
83.51
81.82
78.66
13B
81.18
83.72
83.83
84.32
82.71
79.22
ALMA-R 7B
82.41
84.28
83.51
84.55
82.50
80.13
ALMA-R 13B
83.59
85.37
84.43
85.39
83.23
80.48
TOWER
TOWERBASE 7B
81.03
83.25
84.00
84.09
80.08
78.92
TOWERBASE 13B
81.18
83.46
84.03
83.89
80.03
78.94
TOWERINSTRUCT 7B
83.22
84.73
84.89
85.24
82.94
80.13
TOWERINSTRUCT 13B
83.98
85.51
85.92
85.62
83.21
80.72
Table 21: COMET-22 on WMT23 for a wide variety of models.
WMT23
Models
en >s
en
>fr
en
>pt
en tru
>2h
Closed
GPT-3.5-turbo
88.67
81.86
90.30
87.88
88.09
GPT-4
88.76
81.85
90.30
88.36
88.32
Open
NLLB 54B
88.74
82.01
89.84
88.67
85.97
[LaMA-2
85.77
78.08
86.97
82.99
81.86
[LaMA-2 13B
86.94
79.83
88.48
85.44
84.89
[LaMA-2 70B
87.84
80.67
89.24
87.12
87.44
Distral-7B-Instruct-ve
86.25
79.18
87.87
84.35
84.13
Mixtral-8x7B
88.12
81.15
89.27
87.14
86.58
Mixtral-8x7B -17BB-XnsttBCC
88.23
81.39
89.48
87.04
86.84
Qwenl 5 72B
86.08
80.32
88.20
80.53
86.68
Gemma 7B
87.30
78,20
88.66
86.16
86.78
COMMAND R
88.35
81.41
89.62
88.65
86.43
COMMAND R+
88.99
82.07
90.30
89.06
88.28
ALMA-PRETRAIN
84.42
76.74
84.92
86.53
85.27
ALMA-PRETRAIN 13B
86.17
79.09
87.56
87.27
86.54
ALMA-R 7B
84.63
76.02
82.92
87.80
85.41
ALMA-R 13B
85.93
79.90
87.41
88.58
86.22
TOWER
TOWERBASE 7B
87.90
81.20
89.45
86.94
86.97
TOWERBASE 13B
87.90
81.48
89.54
87.26
87.57
TOWERINSTRUCT
88.34
81.60
89.38
88.11
87.63
TOWERINSTRUCT 13B
88.63
81.82
89.48
88.49
88.20
Table 22: COMET-22 on TICO-19 for a wide variety of models.
32

---

Published as a conference paper at COLM 2024
H Translation-related tasks full results
H.1 Languages considered
For APE, on Table 3, we consider 4 language pairs: en->de, en- de ten, and ru ru- We leave out en-tr and zh-+en, because we had no post editions to serve as fewshot examples for LLaMA-2 and Mixttal-6xx7-Instruuct Nevertheless, we provide results for TOWERINSTRUCT, GPT-3.5-turbo, and GPT-4 the language pairs in Table 23.
For NER, we consider English, German, French, Spanish, Italian, Portuguese, Russian, and Chinese. Finally, we evaluate GEC on English, German, and Spanish. For this task, besides the results in Table 3, we also measure ERRANT in Table 24.
Results broken down by language are in Tables 25, 26, and 27.
APE
Models
en en-+x
xX-ten
Baseline (no edits)
78.84 4
78.80
GPT-3.5-turbo
82.32 3
77.91
GPT-4
85.52
83.12 1
TOWERINSTRUCT 7B
83.10 3
80.19 3
TOWERINSTRUCT 13B
83.65 2
80.89
Table 23: APE results for the WMT23 LPs considered. NLLB corresponds to the translations that were subject to editing, so their quality serves as the baseline for the task. Table 3 did not include zh-en and en-ru to guarantee a fair comparison with open models there were no fewshot examples available for these LPs.
GEC Multilingual
Models
Closed
GPI-3.5-turbo GPT-4
0.49 0.48 3
Open
LLaMA-2 70B
0.43 4
0.43 4
TOWERINSTTUCCT77 7B
0.42 4
TOWERINSTRUCT 13B
0.43 4
Table 24: GEC ERRANT results.
33

---

Published as a conference paper at COLM 2024
WMT23
Models
en-+de
en->ru
en->Zh
de-te
ru ru-ten
zh->en
Baseline (no edits)
77.87
82.93
75.72
79.92
80.05
76.44
Closed
GPT-3.5-turbo
80.67
84.03
82.27
78.48
78.88
76.37
GPT-4
84.65
86.15
85.75
85.39
83.21
80.75
Open
GPT-3.5-turbo
80.67
84.03
82.27
78.48
78.88
76.37
GPT-4
84.65
86.15
85.75
85.39
83.21
80.75
LLaMA-2 70B
78.49
78.20
81.30
80.76
Mixtral-8x7B-Instruut
82.12
83.15
83.40
82.22
TOWER
TOWERINSTRUCT 7B
81.86
83.92 84.34
83.52 84.59
82.29
80.82
77.45
TOWERINSTRUCT 13B
82.03
83.22
81.30
78.15
Table 25: APE COMET-22 results by language pair.
Models
en
de
Baseline (no edits)
13.75
18.23
18.00
Closed
GPT-3.5-turbo
14.71
13.19
17.29
GPT-4
16.48
12.89
15.86
Open
LLaMA-2 70B Mixtral-&&77-Insttruct
17.46
20.67
27.09
16.44
15.38
19.47
TOWER
TOWERINSTRUCT 7B
13.39 13.13
14.77
17.23
TOWERINSTRUCT 13B
14.42
19.48
Table 26: GEC edit rate results by language.
Models
en
de
es
pt
zh
Closed
GPT-3.5-turbo
55.43
60.12
56.82
53.34
55.46
52.57
17.82
GPT-4
63.61
66.58
65.24
58,72
63.39
61.74
39.88
www.............................
Open
LLaMA-2 70B
46.34
48.79
50.69
47.50
53.96
45.60
19.44
45.74
46,94
46.03
46.11
50.86
40.21
16.51
TOWER
TOWERINSTRUCTT7 7B
75.09
78.01
74.89 76.69
70.35
76.39
73.88
53.13
TOWERINSTRUCT 13B
77.52
79.73
74.55
80.36
77. 47
56.57
Table 27: NER F1 results by language.
34


Reviews:
[{"review_id": "oL1d2dd28C", "reviewer": "Reviewer_i13c", "summary": "This paper discusses a method for improving the performance of LLMs in translation workflows by tailoring them to handle multiple tasks. The authors introduce TowerBase, created through continued pretraining on a multilingual mixture of monolingual and parallel data, and TowerInstruct, obtained by finetuning on instructions pertinent to translation processes. Their approach results in models that outperform other open LLM alternatives on various tasks and are competitive with general-purpose closed LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. We are glad that you found the methodology clear and easy to follow, and our dataset and evaluation framework useful for the community.\n\n> The approach of continued pretraining combined with instruction fine-tuning (as depicted in Figure 1) is quite standard in current LLM development. The paper lacks clarity on the specific technical contributions it offers beyond established practices.\n\nRegarding continued pretraining, while we recognize it is an established practice for adapting models to specific domains, its successful usage for expanding capabilities for several languages is less widespread.\n\nThe most similar work to ours is ALMA, which proposes to CPT on monolingual data only and discards parallel data during CPT, advocating that large amounts of parallel data are no longer necessary for training translation systems and deeming this a “paradigm shift in machine translation”. We detach from them in the following ways:\n\n1. Instead of disregarding parallel data, we include it as an extra training signal alongside monolingual data during CPT. We will make sure to emphasize this difference in the final version.\n\n2. Through ablations on the recipe, we find that, contrary to the position put forward in ALMA, parallel data does help when mixed with monolingual data (see Figure 7).\n\nFurthermore, our technical developments go beyond continued pretraining by:\n\n1. Curating a high-quality and diverse instruction-following dataset for translation-related tasks (going beyond translation only), which improves the performance of the trained model across the board;\n\n2. Providing not only extensive experimentation and quantitative results, but also an analysis of relevant trends (e.g., impact of length of the source sentences on translation quality, proneness to edition in automatic post-editing);\n\n3. Releasing all trained models, our instruction following dataset, our evaluation framework and test sets, and a collection of model generations to facilitate reproducibility and future analysis.\n\nWe will make our contributions and technical developments clearer in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores continued pretraining and fine-tuning specifically for translation tasks. The methodology is clear and easy to follow, demonstrating a well-structured approach to enhancing translation performance in large language models.\nThe specialization dataset and evaluation framework will be useful for the community.", "reasons_to_reject": "The approach of continued pretraining combined with instruction fine-tuning (as depicted in Figure 1) is quite standard in current LLM development. The paper lacks clarity on the specific technical contributions it offers beyond established practices.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qQCv8t8ef", "reviewer": "Reviewer_7egk", "summary": "This paper discusses a multilingual recipe for large language models (LLMs) to have the LLMs acquire multilingual capability in the translation task and multilingual relevant tasks. In the recipe, the authors takes LLaMa-2 models as their open LLM models, perform continued pre-training in the LLM on a multilingual mixture of monolingual and parallel data, creating TOWERBASE, and do finetuning on instructions for translation processes, creating TOWERINSTRUCT. The experiments in the translation-related tasks show that the proposed  model outperforms open alternative on the tasks and achieves competitive performance wit closed models such as GPT-4.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> Baselines seems weak […]\n\nNote that we compare with much larger open models trained on multilingual data, and SOTA closed LLMs which have been shown to be strong in multilingual tasks. For MT, we also compare with strong dedicated models.\n\nStill, we'll update the paper with results from newly released multilingual LLMs (e.g., Command R). Our experiments for MT suggest that TowerInstruct-13B outperforms Command R (35B), and is competitive with Command R+ (104B). \n\n> […] comparison against the previous work (Xu et al., 2024b) […] needs more clarification.\n\nTower differs from ALMA in both continued pretraining (CPT) and supervised finetuning (SFT). During CPT, Tower mixes monolingual and parallel data — a key contribution of our paper — while ALMA uses only monolingual data. Our ablations show that the former outperforms the latter (Fig. 7).\n\nFor SFT, Tower includes data from multiple tasks, while ALMA focuses only on MT. Combining all tasks yields a model with similar translation quality that can perform other tasks (Table 4).\n\n> […] how much performance we would get from other open LLMs like Mistral-7B-Instruct.\n\nWe have now trained Mistral-7B with Tower’s recipe. This model outperforms TowerInstruct-7B across the board, and is on par with TowerInstruct-13B on MT while slightly underperforming it on translation-related tasks. We'll add this experiment in the final version.\n\n> […] Did you calculate the loss over the whole template […]\n\nWe calculate the loss for parallel data during CPT on the whole template, including <SRC-LANG>, <TGT_LANG>, source, and target.\n\n> The column of GEC in Table 3 seems wrong.\n\nThis is a valid point, and we alert readers to this possibility in L130: \"a first cluster will not exist if no model significantly outperforms all others on a majority of languages\". Better average performance may not always imply the highest rank.\n\n> Why […] COMET-kiwi for data cleaning?\n\nFiltering MT data with QE metrics has been shown to improve translation quality [1]. We'll clarify this and include the citation in the final version.\n\n> What kind of prompts did you use […]\n\nWe followed prompts from literature when available. We'll specify them in the appendix and release a dataset with all prompts upon acceptance.\n\n> Have you tried […] LoRA […]\n\nPreliminary experiments indicated a drop in performance with LoRA, which led us to adopt full finetuning for all our experiments. \n\n[1] https://aclanthology.org/2023.wmt-1.50/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well organized and easy to follow each section. Most parts are clearly described. Each result table is carefully designed, as those model ranking report helps understand.\n\nThe authors conducted a variety of experiments in the translation-related tasks. Their experimental results are technically sound, showing that the multilingual capability is well acquired by the proposed approach. The proposed approaches work well at different LLM model scale (7B->13B).", "reasons_to_reject": "- Baselines seems weak as they evaluate (non-multilingual) models in multilingual tasks. Also, the system comparison against the previous work (Xu et al., 2024b) seems misleading and needs more clarification. The ALMA models are same LLaMa-based models but trained on different data sets, regardless of its different finetuning recipe. It is not clear to me whether those improvements come from either data set and/or the proposed TOWER recipe.\n\n- The proposed approach is only applied to LLaMa-2 7B and 13B. We would love to know how much performance we would get from other open LLMs like Mistral-7B-Instruct.", "questions_to_authors": "- The authors described the parallel-data training as \"During training, loss is calculated on both source and target.\" in the 67th line. Can you clarify this by referring the prompt template (reported in C.2)? Did you calculate the loss over the whole template including <SRC-LANG> and/or <TGT_LANG>?\n\n- The column of GEC in Table 3 seems wrong. Since it reports the error rate, I guess that lower score is better, hence the GPT-3.5-turbo should be best-ranked with the error rate of 15.06. \n\n- Why did you use COMET-kiwi for data cleaning? \n\n- What kind of prompts did you use in each of LLM baselines? \n\n- Did you do full fine-tuning in all the experiments? Have you tried parameter efficient model finetuning like LoRA finetuning [1]?\n\n[1] \"LoRA: Low-Rank Adaptation of Large Language Models\", Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. arXiv:2106.09685 (cs).", "extra_scores": {}}, {"review_id": "EnDDQVJfpc", "reviewer": "Reviewer_Ls3C", "summary": "This paper sets out to create a large language model specifically designed for translation and translation-related tasks. This is done by continual pre-training of LLaMA-2 on monolingual and parallel data in multiple languages, then instruction-tuning on a curated and filtered labeled dataset of translation-related tasks (translation, post-editing, named entity recognition).\n\nThe resulting model, TowerInstruct, achieves impressive results on translation, named entity recognition, automatic post-editing, and grammatical error correction (with GEC being a held-out task unseen during instruction-tuning), performing on par with much larger models. Furthermore, the authors publicly release the model, dataset, and evaluations.\n\nThis is an important contribution that will be of interest to the industry. My main quibble is that I wish that the set of translation tasks explored were more extensive (e.g. including different domains or modalities).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive reviews and suggestions!\n\n> I don't think named entity recognition or grammatical error correction are specifically translation-related […], while more related tasks are omitted […]\n\nNER and GEC are an important part of a real-world translation pipeline which prompted their inclusion.\n\nYet, tasks such as MT evaluation are also important. We'll run these experiments for the final version. Preliminary results suggest that models trained with Tower’s recipe can be competitive with widespread metrics like COMET-22.\n\n> […] more comprehensive coverage of machine translation\n\nFor this paper, we focused on widely used benchmarks covering different domains and languages: Flores (Wikipedia), WMT23 (news, e-commerce, speech, social, manuals), and TICO (medical). We performed a domain analysis but found no observable pattern in translation quality. We'll include this in the final version. \n\nStill, we acknowledge that exploring low-resource and multimodal MT are exciting avenues for future work.\n\n> […] unclear how experimental settings are selected.\n\nTraining: we followed configurations from other instruction tuned models [1,2,3] and monitored perplexity. Most led to step-wise losses that have been observed with other LLMs, with sudden downward loss shifts at new epochs. Our configuration alleviated these issues. We'll clarify this.\n\nEvaluation: we evaluated pretrained models in a 5-shot fashion and instruction-following models in a 0-shot setting. The exception was Mixtral-Instruct, where a 5-shot prompt corrected output format issues found in responses in a 0-shot setting. For GEC, we evaluated all models in a 5-shot setting as it was a held-out task.\n\n> Why did you select those 10 languages?\n\nWe aimed to cover a range of high-resource languages across multiple scripts, for which there is a lot of monolingual, parallel, and task-specific data. As such, our project differs from “massively multilingual MT” projects which prioritize language coverage. Extending Tower with more languages is an interesting direction for future research.\n\n> […] Why do you need language identification?\n\nEven on a large corpus of web texts such as mC4, there can be texts of other languages in language-specific partitions. By performing language identification, we further ensured we were training on the correct language.\n\n[1] https://arxiv.org/abs/2310.16944\n\n[2] https://huggingface.co/Intel/neural-chat-7b-v3-1\n\n[3] https://huggingface.co/teknium/OpenHermes-7B", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach to training a translation-focused LLM is useful. Furthermore, it is effective, as shown by several extensive evaluations.\n\n2. In addition to the paper, the authors will release their LLM, dataset, and evaluation framework. This will foster future research in the area.\n\n3. The analysis in section 4 is interesting and clearly shows the impact of each component of the training \"recipe\".", "reasons_to_reject": "1. I am a bit perplexed by the choice of \"translation-related\" tasks in the paper: translation, automatic post-editing, named entity recognition, and grammatical error correction. I don't think named entity recognition or grammatical error correction are specifically translation-related (although they may be useful for translation, like several other tasks), while more related tasks are omitted (e.g., metric development, quality estimation, parallel data curation, bilingual lexicon induction). In addition, I think it would be useful to see more comprehensive coverage of machine translation -- including subtasks like domain-specific MT, low-resource or dialectal translation, or multimodal MT.\n\n2. In some cases, it is unclear how experimental settings are selected. Examples: L100 \"We train for 4 epochs using a low learning rate and a large batch size\", L121 \"We prompt TOWER and closed models in a 0-shot fashion and others with 5 examples randomly selected from the development set.\" It would be good to know how these design decisions were made (optimization on a development set?).", "questions_to_authors": "1. Not a question, but a suggestion: \"automatic post-edition\" should be \"automatic post-editing\" (in a couple of places).\n\n2. Why did you select those 10 languages? (EN, DE, FR, NL, IT, ES, PT, KO, RU, ZH)\n\n3. L63: Why do you need language identification?", "extra_scores": {}}, {"review_id": "kPVnhnIRoN", "reviewer": "Reviewer_E4m3", "summary": "The authors describe the process of adapting large language models to translation-related tasks (e.g., translation, pre-editing, post-editing) in great detail. They evaluate the resulting models (TOWER-BASE and TOWER-INSTRUCT) against open and closed models on a variety of tasks, showing clear performance improvements (outperforming open models and approaching or on par with large closed models). The authors pledge to release their models, code, and data sets to the community.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and insightful comments! We are happy to hear that you found our contributions can significantly benefit the field.\n\n> Since the authors' models are based on Llama 2, they inherit Llama's usage restrictions (cannot be used to improve non-Llama models and cannot be used by some companies: https://llama.meta.com/llama-downloads/). This could make it challenging for some members of the research community to build on this work.\n\nThis is an important point. Our Tower recipe can be used with other open models, which are currently launched at a fast pace. In fact, we are planning to release Tower models based on Mistral-7B, which will allow us to reach a wider part of the community.\n\n> Appendix C.2 states that the parallel data used in continued pre-training is prompted with “<SRC LANG>: <SRC>\\n<TGT LANG>: <TGT>”. This is an important enough design decision to move to the main paper (e.g., that parallel data is labeled/prompted instead of simply concatenated).\n\nThank you for pointing this out. We agree that this is an important design decision given our focus on parallel data during continued pretraining. We will add this information to the main text in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work contains two types of contributions that can significantly benefit the field:\n- Methods, results, and observations related to effectively customizing existing LLMs to a class of tasks. These can be directly used or extended in ongoing work.\n- Actual models, code, and data. These can be deployed to benefit real-world users as well as used as starting points for further exploration.\n\nOn a more detailed level:\n- The authors thoroughly describe the processes used to construct data sets, train models, and run evaluations.\n- The authors include detailed analyses (including ablation experiments) to explore where and how different techniques are effective.\n- The authors include a variety of sound baselines that make the results credible.", "reasons_to_reject": "Since the authors' models are based on Llama 2, they inherit Llama's usage restrictions (cannot be used to improve non-Llama models and cannot be used by some companies: https://llama.meta.com/llama-downloads/). This could make it challenging for some members of the research community to build on this work.", "questions_to_authors": "Appendix C.2 states that the parallel data used in continued pre-training is prompted with \"<SRC LANG>: <SRC>\\n<TGT LANG>: <TGT>\". This is an important enough design decision to move to the main paper (e.g., that parallel data is labeled/prompted instead of simply concatenated).", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper explores continued training of LLMs specifically for translation tasks. All reviewers agree to accept the submission. The method is straightforward, and the findings are helpful to the community. Although the technical contribution is limited, it is still a good study for MT.
Decision: Accept