Tower: An Open Multilingual Large Language Model for Translation-Related Tasks

Authors: duarte m alves
josé  pombal
nuno m guerreiro
pedro h martins
joão  alves
amin  farajian
ben  peters
ricardo  rei
patrick  fernandes
sweta  agrawal
pierre  colombo
josé g c de souza
andré f t martins
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 15:42:24

Abstract: While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In this paper, we propose a recipe for tailoring LLMs to multiple tasks present in translation workflows. We perform continued pretraining on a multilingual mixture of monolingual and parallel data, creating TowerBase, followed by finetuning on instructions relevant for translation processes, creating TowerInstruct. Our model surpasses open alternatives on several relevant tasks and is competitive with general-purpose closed LLMs. We will release the Tower models, our specialization dataset, an evaluation framework for LLMs focusing on the translation ecosystem, and a collection of model generations on our benchmark.

Main paper: 
 introduction: 
 Many important tasks within multilingual NLP, such as quality estimation, automatic postedition, or grammatical error correction, involve analyzing, generating or operating with text in multiple languages, and are relevant to various translation workflows -we call these translation-related tasks. Recently, general-purpose large language models (LLMs) challenged the paradigm of per-task dedicated systems, achieving state-of-the-art performance on several recent WMT shared tasks (Kocmi et al., 2023; Freitag et al., 2023; Neves et al., 2023) . Unfortunately, strong capabilities for multiple translation-related tasks have so far been exhibited by closed LLMs only (Hendy et al., 2023; Kocmi & Federmann, 2023; Fernandes et al., 2023; Raunak et al., 2023) . Perhaps because most open LLMs are English-centric, approaches leveraging these models still lag behind, having thus far achieved competitive results only when specializing on a single task (Xu et al., 2024a; 2023; Iyer et al., 2023) . In this paper, we bridge this gap with a detailed recipe to develop an LLM for multiple translation-related tasks. Our approach, illustrated in Figure 1 TOWERINSTRUCT outperforms larger open alternatives and is competitive with GPT models.(2024a), relies on three steps. First, we extend LLaMA-2's (Touvron et al., 2023b) multilingual capabilities through continued pretraining, creating TOWERBASE ( §2.1). Importantly, while Xu et al. (2024a) exclusively employ monolingual data, we include parallel data as an additional cross-lingual signal. Second, we curate a dataset to specialize LLMs for translation-related tasks, TOWERBLOCKS ( §2.2). Third, we perform supervised finetuning to obtain an instruction-following model for the field of translation, TOWERINSTRUCT ( §2.3).We extensively evaluate all our models, comparing with open and closed alternatives on a wide range of tasks ( §3). TOWERINSTRUCT consistently achieves higher translation quality than open alternatives and is competitive with the closed GPT-4 and GPT-3.5-turbo modelssee Figure 2 . Additionally, TOWERINSTRUCT outperforms open models in automatic postedition, grammatical error correction, and named entity recognition. Careful ablations also outline the influence of each element in our recipe ( §4). We highlight the importance of adding parallel data during continued pretraining for improved translation quality, and the effectiveness of including conversational and coding data on TOWERBLOCKS.Accompanying this work, we release 1) the TOWER family, comprising our TOWERBASE and TOWERINSTRUCT models in the sizes of 7B and 13B; 2) our specialization dataset TOWERBLOCKS; 3) TOWEREVAL, the evaluation framework for LLMs for translation-related tasks that we used to perform all evaluations in this paper; and 4) a collection of model generations for our benchmark to ensure reproducibility and encourage future exploration.1  
tower: an open multilingual llm for translation-related tasks: 
 Our backbone language model is LLaMA-2 (Touvron et al., 2023b) , which is very competitive on a wide range of tasks. Nevertheless, it was trained on relatively little non-English data, limiting its potential for multilingual tasks. We alleviate this effect by continuing LLaMA-2's pretraining on a highly multilingual corpus ( §2.1). Afterwards, we finetune our continued pretrained model on a specialization dataset ( §2.2), obtaining an instruction-following model centered around translation ( §2.3).We extend LLaMA-2's training on a highly-multilingual dataset comprising 20 billion tokens -measured with the model's tokenizer -for 10 languages: English (en), German (de), French (fr), Dutch (nl), Italian (it), Spanish (es), Portuguese (pt), Korean (ko), Russian (ru), and Chinese (zh). While Xu et al. (2024b) exclusively leverage monolingual data, we draw inspiration from work including parallel data during pretraining (Anil et al., 2023; Briakou et al., 2023) , and mix parallel sentences (one third) along with monolingual data (two thirds). Our results show that this approach greatly benefits translation quality ( §4).Monolingual data. We sample uniformly across our languages from mC4 (Xue et al., 2021) and apply standard cleaning (Wenzek et al., 2019; Touvron et al., 2023a) : deduplication, language identification, and perplexity filtering with KenLM (Heafield, 2011) .Parallel Data. We sample uniformly to-English (xx→en) and from-English (en→xx) language pairs from public sources, removing translations below quality thresholds for Bicleaner (Sánchez-Cartagena et al., 2018; Ramírez-Sánchez et al., 2020) as well as COMETKIWI-22 (Rei et al., 2022b) , which is shown to improve translation quality (Peter et al., 2023) . We include parallel data with the following template, calculating the loss on all tokens:[source language]: [source]\n[target language]: [target] Model Training. We train our models with a codebase based on Megatron-LLM (Cano et al., 2023) on 8 A100-80GB GPUs, using an effective batch size of 1.57 million tokens per gradient step and a cosine scheduler with initial and final learning rates of 3 × 10 -5 and 3 × 10 -6 . We provide additional details on continued pretraining in Appendix D.1.We build TOWERBLOCKS prioritizing data diversity and quality. Figure 3 illustrates all tasks in the dataset. They include tasks important to translation workflows, applied before or after translation, and datasets to improve multilingual understanding and instruction-following.Diversity. We collect records from multiple datasets for each translation-related taskdatasets are detailed in Appendix E. Akin to Wei et al. (2022) , we reformulate all records as natural language instructions using multiple manually curated zero-and few-shot templates. Following insights from Longpre et al. (2023) , we create zero-shot instructions with 75% of the records. For the remaining ones, we uniformly sample 1, 3, or 5 in-context examples from the respective dataset. We also include paraphrasing, dialogues from UltraChat (Ding et al., 2023) , and code instructions from Glaive-Code-Assistant (Glaive AI, 2023) . Xu et al. (2024a) , we construct records from human-annotated datasets,2 prioritizing validation and old test sets -we exclude datasets from 2023 onwards.For tasks with reference translations, such as translation and automatic post-edition, we score references with XCOMET-QE-ENSEMBLE (Guerreiro et al., 2023) and discard records with quality scores below 0.85. Additionally, we include only translation pairs in their original direction. Finally, we adopt the UltraChat (Ding et al., 2023) dialogues filtered by Tunstall et al. (2023) and further exclude records that contain translation requests, formatting issues (e.g. instructions starting with punctuation), or assistant refusal cases.As a final step, we obtain TOWERINSTRUCT by finetuning TOWERBASE on TOWERBLOCKS.Dialog template. We format each dialog as a single tokenizable string using the chatml template (Open AI, 2023) . This template clearly separates between instructions and answers, and allows for multi-turn dialog. The template has three special identifiers (control tokens) to delimit messages: <|im start|>user and <|im start|>assistant precede the beginning of a turn, and <|im end|> marks its end. We provide further details in Appendix F.1.We finetune the model with the standard cross-entropy loss, enabling bfloat16 mixed precision and packing (Raffel et al., 2020) . We only calculate the loss on target (answer) tokens. We train for 4 epochs using a low learning rate and a large batch size -we detail all hyperparameters in Appendix F.2. 
experimental setup: 
 Datasets and Tasks. We analyze machine translation (MT) on FLORES-200 (NLLB Team et al., 2022) , WMT23 (Kocmi et al., 2023) , and TICO-19 (Anastasopoulos et al., 2020) . Additionally, we examine three translation-related tasks. First, following Raunak et al. (2023) , we evaluate automatic post-edition (APE) by measuring translation quality after post-editing NLLB-3.3B (NLLB Team et al., 2022) translations for WMT23. Second, we evaluate named entity recognition (NER), useful for entity anonymization, using the test split from Multi-CoNER 2023 (Fetahu et al., 2023) . 3 Third, we evaluate grammatical error correction (GEC), which is held out from our training data and useful for correcting the source sentence before translation. We test GEC on CoNLL-2014 (Ng et al., 2014) (English), COWSL2H (Yamada et al., 2020) (Spanish), and mlconvgec2018 (Chollampatt & Ng, 2018 ) (German).Baselines. We compare TOWER with the open LLaMA-2 70B (Touvron et al., 2023b) and Mixtral-8x7B-Instruct (Jiang et al., 2024) , and the closed GPT-3.5-turbo and GPT-4. 4 For MT, we also consider dedicated translation models NLLB-54B (NLLB Team et al., 2022) and ALMA-R (Xu et al., 2024b) . In Appendix G, we compare with the open Gemma 7B (Gemma Team, 2024), Mistral-7B-Instruct-v0.2 (Jiang et al., 2023) and Qwen1.5 72B (Bai et al., 2023) . 5 Model generations use greedy decoding -we explore alternative decoding methods in Appendix A. We prompt TOWER and closed models in a 0-shot fashion and others with 5 examples randomly selected from the development set.Evaluation. We evaluate translation quality with COMET-22 (Rei et al., 2022a) for both MT and APE. For MT, we also report XCOMET (Guerreiro et al., 2023) , COMETKIWI-22 (Rei et al., 2022b) , BLEURT (Sellam et al., 2020) , and CHRF (Popović, 2015) in Appendix G. 6 For GEC, we small amount of parallel data from OPUS to cover all language pairs. Nevertheless, we apply Bicleaner using a threshold of 0.85 followed by the quality filtering procedure described in this section. We define a win if the sentence-level delta between two systems is above 1 COMET-22 point.measure edit rate (ER) (Snover et al., 2006) and report ERRANT (Bryant et al., 2017; Felice et al., 2016) in Appendix H. For NER, we measure sequence F1 score. We report performance clusters based on statistically significant performance gaps at a 95% confidence threshold. 7 We create per-language groups for systems with similar performance, following Freitag et al. (2023) , and obtain system-level rankings using a normalized Borda count (Colombo et al., 2022) , which is defined as an average of the obtained clusters. Note that a first cluster will not exist if no model significantly outperforms all others on a majority of languages. We posit that ALMA-R's alignment process on translations preferred by COMETKIWI-XXL (Rei et al., 2023) and XCOMET may inadvertently degrade performance on lexical metrics.7 For segment-level metrics, such as COMET-22, we perform significance testing at the segment level. For corpus-level metrics, such as ER and Sequence F1, we consider 100 bootstrapped samples of size 500, similar to Koehn (2004 41 3 87.82 3 88.22 3 88.07 4 87.47 3 89.11 4 88.65 5 87.32 5  Mixtral-8x7B-Instruct 87.99 3 86.80 2 88.53 2 88.77 2 85.63 5 87.57 3 89.45 3 89.09 4 85 on WMT23 and TICO-19. Upon inspecting both systems' outputs, we verified that the gap between them increases with longer sentences, as is shown in Figure 4 . Notably, this trend vanishes when comparing TOWERINSTRUCT 13B to ALMA-R. 8 We posit this difference stems from a prevalence of shorter translations in the training data of both TOWERINSTRUCT 13B and ALMA-R. In future work, we would like to explore how to better leverage longer contexts, which can benefit instruction-following (Zhao et al., 2024) .In Table 2 , TOWERINSTRUCT 13B is ranked first for the majority of en→xx directions, and is among the top performing models for all but one xx→en language pair. Notably, TOWERINSTRUCT stands out as the best overall model -outperforming GPT-4 -for both pt→en and ru→en directions. The improved performance on xx→en directions likely stems from LLaMA-2's English-centric pretraining. A longer, more expensive continued pretraining might further improve performance on en→xx directions. In fact, we show in Section 4 that the translation quality gains from LLaMA-2 are larger for en→xx language directions. TOWERINSTRUCT 7B achieves a trade-off between performance and scale. The smaller TOWERINSTRUCT 7B, although behind TOWERINSTRUCT 13B, is competitive with other open systems and achieves GPT-3.5-turbo translation quality for some language pairs. Importantly, it outperforms the only system of the same size, ALMA-R 7B. Table 3 reports aggregated results for all translation-related tasks.9 TOWERINSTRUCT is an effective translation post-editor. TOWERINSTRUCT outperforms open models and GPT-3.5-turbo on APE, consistently and significantly improving the quality of NLLB 3.3B translations. However, GPT-4 is still the top performer on this task. Yet it also has a higher edit rate, shown in Figure 5 : while TOWERINSTRUCT edits 30% of the instances, GPT-4 edits almost 90%. This tendency suggests that GPT-4 is over-editing, which we further analyze in Appendix B. We posit that TOWERINSTRUCT edits less due to the prevalence of unedited segments in TOWERBLOCKS -roughly 38%.There is room for improvement on grammatical error correction. On this task, no model significantly outperforms the others on a majority of languages. We hypothesize that TOW-ERINSTRUCT's average performance is caused by the absence of GEC data in TOWERBLOCKS. TOWERINSTRUCT can identify named entities. TOWERINSTRUCT 13B shows promising performance on NER, surpassing GPT-4 by about 15 F1 points. Similar to APE, TOWERIN-STRUCT 7B reflects most of these improvements, highlighting its capabilities at a smaller scale. Other open models do not perform well on this task, even with 5 in-context examples. We hypothesize these results stem from NER being a token-level classification task, as opposed to a generative one. While the models learn the output format from the examples or task description, they struggle to grasp the classification function itself. Conversely, TOWERINSTRUCT learns the task from the NER data in TOWERBLOCKS. 
dissecting the training recipe: 
 We performed multiple ablations to provide insights on the impact of the several design choices made in the development of the TOWER models.The two leftmost plots of Figure 6 illustrate translation quality after continued pretraining and supervised finetuning. Both steps bring performance improvements at both model scales. Remarkably, TOWERBASE 7B and TOWERINSTRUCT 7B outperform LLaMA-2 13B, and TOWERINSTRUCT 7B outperforms TOWERBASE 13B. The two rightmost plots analyze APE. For this task, while supervised finetuning yields better performance, continued pretraining -and in particular parallel data -does not improve performance as observed for translation. In future work, we would like to explore additional training signals during continued pretraining to increase performance on translation-related tasks. Parallel data during continued pretraining is sample efficient, but quality continues to improve with more tokens. At the 2 billion token mark, combining parallel sentences with monolingual data (i) yields more than 50% of the improvement over the base model, and (ii) surpasses the recipe leveraging solely monolingual data. Additionally, although training on more tokens has diminishing returns -85% of the total performance gains appear by the 5 billion token mark -it continues to improve translation quality.Transfer/interference relations between tasks are complex. Table 4 ablates the components of TOWERBLOCKS, comparing finetuning on translation data, translation-related tasks including pre-and post-translation, and the full dataset with general-purpose tasks. While adding translation-related tasks improves their performance, it decreases translation quality.Remarkably, introducing general-purpose instructions recovers translation quality. In future work, we would like to explore transfer/interference between tasks using scaling laws.The TOWER recipe generalizes to other model families. When newer LLMs become available, opportunities for improving TOWER naturally arise. In Table 5 we compare Mistral 7B against LLaMA-2 as a backbone for TOWER. Notably, applying the TOWER recipe to Mistral outperforms starting from LLaMA-2 7B across the board and is competitive with using the larger LLaMA-2 13B, showcasing the generalizability of the recipe.11  
related work: 
 Adapting open models to single tasks within the field of machine translation is competitive with closed models or dedicated systems (Xu et al., 2024a; 2023; Iyer et al., 2023) . Notably, Xu et al. (2024a) adapt LLaMA-2 for translation with continued pretraining on monolingual data and finetuning on high quality parallel data. Our work adopts a similar approach, but introduces parallel data during continued pretraining and leverages LLMs' instructionfollowing capabilities to build a system supporting multiple translation-related tasks. (Cui et al., 2023; Xu et al., 2024a; Pires et al., 2023) . Building upon the effectiveness of pretraining on parallel data (Anil et al., 2023; Wei et al., 2023) , we include parallel sentences during continued pretraining. We do not extend the vocabulary, as preliminary experiments yielded negative results.Recent research on specializing LLMs on closely-related tasks leverages domain-specific data to train new models "from scratch" (Taylor et al., 2022; Wu et al., 2023) , extend the pretraining of existing models (Lewkowycz et al., 2022; Chen et al., 2023) , perform supervised finetuning (Yue et al., 2024) , or a combination of the last two (Rozière et al., 2023; Liu et al., 2023) . Our specialization approach is broadly inspired by instruction tuning (Wei et al., 2022; Sanh et al., 2022; Wang et al., 2023; Zhou et al., 2023) , which finetunes language models on tasks formatted as natural language instructions. 
conclusion: 
 We propose a new recipe for specializing LLMs on multiple translation-related tasks. First, we expand LLaMA-2's multilingual capabilities with continued pretraining on a highly multilingual corpus. Then, we finetune on a dataset of high-quality and diverse instructions for translation-related tasks. Our final model consistently outperforms open alternatives on multiple translation-related tasks, and is competitive with closed models such as GPT-4. 
ethics statement: 
 Our work builds upon LLMs to develop an instruction-following model centered around translation. The usage of LLMs encompasses several risks, which are thoroughly discussed in Brown et al. (2020) and Chowdhery et al. (2022) . While we performed extensive evaluation on our models, we focused on performance metrics and did not include toxicity or safety analysis. Additionally, we leveraged automatic metrics which are trained on human annotations and can inherit issues from mislabeled data. Annotators can fail to consider better alternatives when presented with a given text or wrongfully classify text as high quality (Bansal et al., 2021) . In this section, we analyse the performance of TOWERINSTRUCT 13B with beamsearch (Reddy, 1977) using beam size of 5 and minimum Bayes risk (MBR) decoding (Eikema & Aziz, 2020; Fernandes et al., 2022; Freitag et al., 2022) with 20 hypotheses and COMET-22 as an utility function. We generate hypotheses using temperature and nucleus sampling (Holtzman et al., 2020), with t = 0.9 and p = 0.6. We avoid "optimizing" the evaluation metric (Fernandes et al., 2022) by measuring translation quality with BLEURT.  
 
 Appendix: Figure 8 shows that differences between GPT-4 and TOWERINSTRUCT edit rates are not strongly correlated to differences in COMET-22 (0.34 Spearman ρ). This means that GPT-4 edits often do not correspond to gains in performance. This finding, allied with the discussion in Section 3.3 about GPT-4 editing considerably more than TOWERINSTRUCT, suggests that GPT-4 may be editing too much. We included evaluation data in TOWERBLOCKS to induce positive transfer to machine translation. Nevertheless, recent work has shown that LLMs can also function as estimators translation quality (Kocmi & Federmann, 2023; Fernandes et al., 2023) . Table 7 shows preliminary experiments where the evaluation data in TOWERBLOCKS is improved. Notably, this version of TOWER is competitive with COMET-22 on en→de and zh→en, suggesting that the TOWER recipe can be used to create models to evaluate translation quality. In the future, we intend to further explore how to improve evaluation capabilities of TOWERINSTRUCT.This section contains some details on the continued pretraining process used to obtain TOWERBASE. Training time was 10 and 20 days for the 7B and 13B models, respectively.In Table 8 , we report the perplexity floors and ceilings used to filter the monolingual data in the continued pretraining corpus, as well as the Bicleaner and CometKiwi-22 thresholds used to filter the parallel data.Parallel data was obtained from the sources listed in Table 9 . Table 8 : Quality filtering thresholds applied on monolingual data ( * ) and parallel data ( †) by language. On the latter, the to-English language pair's threshold is the same as the corresponding from-English one.This appendix details all datasets utilized in TOWERBLOCKS:• WMT14 to WMT2112 -Evaluation sets for the general machine translation shared task;• WMT22 with quality-shots (Hendy et al., 2023) -Evaluation set from WMT23 with high quality in-context examples;• NTREX (Federmann et al., 2022) -Professional translations of the WMT19 test set;• FLORES-200 (NLLB Team et al., 2022) -Development set of the FLORES-200 dataset for all languages included in training;• FRMT (Riley et al., 2022) -Human translations of English Wikipedia sentences into regional variants;• OPUS (Tiedemann, 2012) -Parallel corpora from which we sampled very high-quality samples for all language pairs;• QT21 (Specia et al., 2017) and ApeQuest13 -Translation data with post-edits utilized for general translation and automatic post-editing;• MT-GenEval (Currey et al., 2022) -Gender translation benchmark which we leveraged for general translation and context-aware translation;• WMT20 to WMT22 Metrics MQM14 -MT evaluation data annotated with multidimensional quality metrics (Lommel et al., 2014) that we used to perform error span detection;• WMT17 to WMT22 Metrics DAs15 -MT evaluation data annotated with direct assessements (DAs) (Graham et al., 2013) which we utilized for translation ranking.• WMT21 Terminology16 -Development set for the WMT21 terminology task;• Tatoeba (Tiedemann, 2020) -Development set of the Tatoeba dataset which we used to generate translations in different languages for the same source -we identified this task as multi-reference translation;• MultiCoNER 2022 and 2023 (Malmasi et al., 2022; Fetahu et al., 2023) -Development sets of the named entity recognition MultiCoNER datasets. For MultiCoNER 2023, we adopted the coarse-grained entity categorization;• PAWS-X (Yang et al., 2019) -Development set of the PAWS-X dataset which we used as paraphrase generation;• UltraChat (Ding et al., 2023) Tables 12, 13, 14 and 15 report translation quality on all test sets using various metrics: XCOMET, COMETKIWI-22, BLEURT, and CHRF. Tables 16, 17 , 18 and 19 analyse translation quality for our languages, considering en→xx and xx→en translation directions, for the same metrics. On Tables 20, 21 , and 22, we present translation results for a wider variety of models, broken down by language pair. Importantly, quality trends hold across metrics. TOWERINSTRUCT 13B is the open system with highest translation quality and is competitive with the closed model GPT-4. For APE, on Table 3 , we consider 4 language pairs: en→de, en→zh, de→en, and ru→en. We leave out en→ru and zh→en, because we had no post editions to serve as fewshot examples for LLaMA-2 and Mixtral-8x7B-Instruct. Nevertheless, we provide results for TOWERINSTRUCT, GPT-3.5-turbo, and GPT-4 on the 6 language pairs in Table 23 .For NER, we consider English, German, French, Spanish, Italian, Portuguese, Russian, and Chinese. Finally, we evaluate GEC on English, German, and Spanish. For this task, besides the results in Table 3 , we also measure ERRANT in Table 24 .Results broken down by language are in Tables 25, 26 , and 27. Table 23 : APE results for the 6 WMT23 LPs considered. NLLB corresponds to the translations that were subject to editing, so their quality serves as the baseline for the task. Links for the TOWER models; TOWERBLOCKS; TOWEREVAL; Zeno(Cabrera et al.,2023) project with model generations.For named entity recognition, we did not find a permissively licensed human-annotated dataset, so we use MultiCoNER(Malmasi et al., 2022;Fetahu et al., 2023). For general translation, we include aWe uniformly sample 1000 of the more than 200k records due to the computational costs of evaluating all models on the whole test set.We use gpt-3.5-turbo-0613 and gpt-4-0613 available from the official OpenAI API.TOWERINSTRUCT outperforms all these open alternatives.Performance trends largely hold across metrics. Yet, there is a significant quality gap between ALMA-R and TOWER models in terms of CHRF -e.g., overpoints in en→xx directions on WMT23.A similar domain-level analysis did not find any domain dissimilar from the others.Appendix H details evaluated languages and provides further results for APE and GEC. Appendix C contains preliminary results for MT evaluation.While 0.1 COMET-22 points translates to 54.9% human agreement, one COMET-22 point translates to 90.9%(Kocmi et al., 2024).We also release our TOWER models based on Mistral.https://www2.statmt.org/wmt23/translation-task.htmlhttps://apequest.wordpress.com/https://www.statmt.org/wmt22/results.htmlhttps://www.statmt.org/wmt22/results.htmlhttps://www.statmt.org/wmt21/terminology-task.html 


Decision: The paper explores continued training of LLMs specifically for translation tasks. All reviewers agree to accept the submission. The method is straightforward, and the findings are helpful to the community. Although the technical contribution is limited, it is still a good study for MT.

Reviews: review: paper_summary: paper_summary: This paper discusses a method for improving the performance of LLMs in translation workflows by tailoring them to handle multiple tasks. The authors introduce TowerBase, created through continued pretraining on a multilingual mixture of monolingual and parallel data, and TowerInstruct, obtained by finetuning on instructions pertinent to translation processes. Their approach results in models that outperform other open LLM alternatives on various tasks and are competitive with general-purpose closed LLMs.
strength_weakness: strength_weakness: The approach of continued pretraining combined with instruction fine-tuning (as depicted in Figure 1) is quite standard in current LLM development. The paper lacks clarity on the specific technical contributions it offers beyond established practices.
paper_summary: paper_summary: This paper discusses a multilingual recipe for large language models (LLMs) to have the LLMs acquire multilingual capability in the translation task and multilingual relevant tasks. In the recipe, the authors takes LLaMa-2 models as their open LLM models, perform continued pre-training in the LLM on a multilingual mixture of monolingual and parallel data, creating TOWERBASE, and do finetuning on instructions for translation processes, creating TOWERINSTRUCT. The experiments in the translation-related tasks show that the proposed  model outperforms open alternative on the tasks and achieves competitive performance wit closed models such as GPT-4.
strength_weakness: strength_weakness: - Baselines seems weak as they evaluate (non-multilingual) models in multilingual tasks. Also, the system comparison against the previous work (Xu et al., 2024b) seems misleading and needs more clarification. The ALMA models are same LLaMa-based models but trained on different data sets, regardless of its different finetuning recipe. It is not clear to me whether those improvements come from either data set and/or the proposed TOWER recipe.

- The proposed approach is only applied to LLaMa-2 7B and 13B. We would love to know how much performance we would get from other open LLMs like Mistral-7B-Instruct.
questions: questions: - The authors described the parallel-data training as "During training, loss is calculated on both source and target." in the 67th line. Can you clarify this by referring the prompt template (reported in C.2)? Did you calculate the loss over the whole template including <SRC-LANG> and/or <TGT_LANG>?

- The column of GEC in Table 3 seems wrong. Since it reports the error rate, I guess that lower score is better, hence the GPT-3.5-turbo should be best-ranked with the error rate of 15.06. 

- Why did you use COMET-kiwi for data cleaning? 

- What kind of prompts did you use in each of LLM baselines? 

- Did you do full fine-tuning in all the experiments? Have you tried parameter efficient model finetuning like LoRA finetuning [1]?

[1] "LoRA: Low-Rank Adaptation of Large Language Models", Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. arXiv:2106.09685 (cs).
paper_summary: paper_summary: This paper sets out to create a large language model specifically designed for translation and translation-related tasks. This is done by continual pre-training of LLaMA-2 on monolingual and parallel data in multiple languages, then instruction-tuning on a curated and filtered labeled dataset of translation-related tasks (translation, post-editing, named entity recognition).

The resulting model, TowerInstruct, achieves impressive results on translation, named entity recognition, automatic post-editing, and grammatical error correction (with GEC being a held-out task unseen during instruction-tuning), performing on par with much larger models. Furthermore, the authors publicly release the model, dataset, and evaluations.

This is an important contribution that will be of interest to the industry. My main quibble is that I wish that the set of translation tasks explored were more extensive (e.g. including different domains or modalities).
strength_weakness: strength_weakness: 1. I am a bit perplexed by the choice of "translation-related" tasks in the paper: translation, automatic post-editing, named entity recognition, and grammatical error correction. I don't think named entity recognition or grammatical error correction are specifically translation-related (although they may be useful for translation, like several other tasks), while more related tasks are omitted (e.g., metric development, quality estimation, parallel data curation, bilingual lexicon induction). In addition, I think it would be useful to see more comprehensive coverage of machine translation -- including subtasks like domain-specific MT, low-resource or dialectal translation, or multimodal MT.

2. In some cases, it is unclear how experimental settings are selected. Examples: L100 "We train for 4 epochs using a low learning rate and a large batch size", L121 "We prompt TOWER and closed models in a 0-shot fashion and others with 5 examples randomly selected from the development set." It would be good to know how these design decisions were made (optimization on a development set?).
questions: questions: 1. Not a question, but a suggestion: "automatic post-edition" should be "automatic post-editing" (in a couple of places).

2. Why did you select those 10 languages? (EN, DE, FR, NL, IT, ES, PT, KO, RU, ZH)

3. L63: Why do you need language identification?
paper_summary: paper_summary: The authors describe the process of adapting large language models to translation-related tasks (e.g., translation, pre-editing, post-editing) in great detail. They evaluate the resulting models (TOWER-BASE and TOWER-INSTRUCT) against open and closed models on a variety of tasks, showing clear performance improvements (outperforming open models and approaching or on par with large closed models). The authors pledge to release their models, code, and data sets to the community.
strength_weakness: strength_weakness: Since the authors' models are based on Llama 2, they inherit Llama's usage restrictions (cannot be used to improve non-Llama models and cannot be used by some companies: https://llama.meta.com/llama-downloads/). This could make it challenging for some members of the research community to build on this work.
questions: questions: Appendix C.2 states that the parallel data used in continued pre-training is prompted with "<SRC LANG>: <SRC>\n<TGT LANG>: <TGT>". This is an important enough design decision to move to the main paper (e.g., that parallel data is labeled/prompted instead of simply concatenated).

Comments: title: Thank you for the response.
Re: Rebuttal by Authors
comment: The response clarified my questions. The final ver will be more technically sound with those additional results. Hence, I increased the rating to "7: Good paper, accept".
Thank you for adding this information. The first point is strong enough to mention in the abstract and introduction: beyond training and releasing specific models, you share a recipe that can create Tower-style models from newer and stronger open models as they are available.