A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
Authors: Haoran Xu
Young Jin Kim
Amr Sharaf
Hany Hassan Awadalla
Venue: ICLR 2024 poster (2024)
Keywords: Machine Translation
Language Language Models
Multilingual
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a novel training recipe for decoder-only LLMs in translation which beats NLLB-54B and GPT-3.5 with 7B/13B model size.

Abstract: Generative Large Language Models (LLMs) have achieved remarkable advancements in various NLP tasks. However, these advances have not been reflected in the translation task, especially those with moderate model sizes (i.e., 7B or 13B parameters), which still lag behind conventional supervised encoder-decoder translation models. Previous studies have attempted to improve the translation capabilities of these LLMs, but their gains have been limited. In this study, we propose a novel fine-tuning approach for LLMs that is specifically designed for the translation task, eliminating the need for the abundant parallel data that traditional translation models usually depend on.
Our approach consists of two fine-tuning stages: initial fine-tuning on monolingual data followed by subsequent fine-tuning on a small set of high-quality parallel data.  We introduce the LLM  developed through this strategy as **A**dvanced **L**anguage **M**odel-based tr**A**nslator (**ALMA**). Based on LLaMA-2 as our underlying model, our results show that the model can achieve an average improvement of more than 12 BLEU and 12 COMET over its zero-shot performance across 10 translation directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test datasets. The performance is significantly better than all prior work and even superior to the NLLB-54B model \citep{nllb} and GPT-3.5-text-davinci-003, with only 7B or 13B parameters. This method establishes the foundation for a novel training paradigm in machine translation.

Published as a conference paper at ICLR 2024
A PARADIGM SHIFT IN MACHINE TRANSLATION: BOOSTING TRANSLATION PERFORMANCE OF LARGE LANGUAGE MODELS
Haoran X Young Jin Kim Amr Sharaf Hany Hassan Awadalla
Johns Hopkins University, Microsoft
hxu640 juu.jhuuee jhu.edu edu {youki, amr amrsharaf, banybhemdcrosooft.com
ABSTRACT
Generative Large Language Models (LLMs) have achieved remarkable advance- ments in various NLP tasks. However, these advances have not been reflected in the translation task, especially those with moderate model sizes (i.e., 7B or 13B parameters), which still lag behind conventional supervised encoder translation models. Previous studies have attempted to improve the translation capabilities of these I.MMs, but their gains have been limited. In this study, we propose a novel fine-tuning approach for LLMs that is specifically designed for the translation task, climinating the need for the abundant parallel data that tradi- tional translation models usually depend on. Our approach consists of two fine- tuning stages: initial fine-tuning on monolingual data followed by subsequent fine-tuning on small set of high-quality parallel data. We introduce the LLM developed through this strategy as Advanced Language Model-based trAnslator (ALMA). Based on LLaMA-2 (Touvron et al.) 2023b) our underlying model, our results show that the model can achieve an average improvement of more than 12 BLEU and 12 COMET over its zero-shot performance across 10 trans- lation directions from the WMT'21 (2 directions) and WMT'22 (8 directions) test datasets. The performance is significantly better than all prior work and even superior to the NLLB-54B model (NLLB TEAM et al. 2022) and GPT- 3.5-text-davirci-00
with only 7B 13B parameters. This method estab- lishes the foundation for a novel training paradigm in machine translation.
INTRODUCTION
Generative (decoder-only) large language models (LLMs) such GPT models (Brown al. 2020 OpenAl 2023), PaI.M (Chowdhery et al. 2022), OPT (Zhang et al. 2022), BLOOM (Scao et al. 2022). LLMMA (Touvron et al.) 2023a b), and others have exhibited remarkable capabilities across various NLP tasks. However, for the translation task, only very large models such as GPT-3.5 and GPT-4 can rival the supervised encoder-decoder state-of-the-art (SoTA) models like NLLB (NLB TEAM et al., 2022), while they still fall short in translation for low-resource languages Hendy et al.) 2023, Jiao al. 2023). The discrepancy becomes more evident when comparing other LLMs with traditional translation models (Zhu et al. 2023a). For instance, the OPT-175B model trails behind the NILB- 1.3B model by an average of more than 15 BLEU (Papineni et al.)
2002) points for languages within the Indo- Romance family. The gap is even larger in smaller LLMs; for example, XGLM (Lin et al. (022I. with a parameter size of 7B, lags behind the NLLB-1.3B by substantial 30 BLEU points (Zhu et al. (203aa). Therefore, there is an urgent need to narrow this performance gap between LLMs and conventional SoTA models.
As exemplified by NLLB-1.3BB traditional machine translation models demonstrate proficiency in producing high-quality translations with a small number of parameters. By extension, smaller LLMs
We release our code and models at: httpts:///////t thub.:///tttub.omoo fel XXX/AAIA

---

Published as a conference paper at ICLR 2024
should similarly possess the capability adeptly manage the translation task. Recent research has
sought to enhance translation performance by commencing with smaller LLMs (Yang et al. 2023]
Zeng et al. 2023 Chen et al. 2023 Zhu et al. 2023b Li et al. 2023; Zhang al. 2023b), especially
7B or 13B parameters. Nevertheless, the achieved improvements remain modest and limited. As depicted in Figure I contemporary studies such Balyling (Zhang et al.) (023b) and BigTranslate
Yang et al. 2203)), 3. which use LLaMA as their backbone, exhibit a maximum increment of 3 to BLEU or COMET in relation to the zero-shot performance of LLaMA on the WMT'22 test set (8 directions) While these gains represent promising research direction for smaller LLMs in the translation task, significant performance chasm persists when benchmarked against very large LLMs such GPT-3.5-text-doe
inci-003 and SoTA translation models such as NLLB-54B, We posit that the modest translation gains observed prior studies can be ascribed to an unsuitable training recipe.
ALMA-13B (Ours)
ALMA-13B (Ours)
GPT-3.5
34
GPT-3.5
ALMA-78 (Ours)
-00
-xxt-dvinci--033
ALMA-70 7B (Ours)
32
NLLB-548
84
30
MLLD-540
BLEU
COMET
Bayling-78
Bayling- Bayfing-338
Payling 138
80
BigThansiste
LLMMA 138
2LM 1
78
ALPPO 76
4Lanch 2-m0
54B
175h
76
138
Size (B)
Stoe
(a) BLEU
(b) COMET
Figure 1: Translation performance of contemporary decoder-only LLM translation systems based on LLaMA (Yang et al. 2023] Zhang et al.] 2023b), and zero-shot performance of LLaMA, for the WMT'22 22 test data across 8 directions (translating to or from English for German, Czech, Chinese, and Russian). Benchmark comparisons also include two leading translation models, NLLB-55B and SPT-3.5-teet--davine -003. Our systems, developed on LLaMA-2 with 7B and 13B pa- rameters, surpass previous models by an impressive margin of nearly 10 BLEU and COMET. Furthermore, they even slightly outperform GPT-3.5 and NLLB-54B on average.
We hypothesize that an efficacious training recipe ought to follow two stages: learning general multilingual linguistic knowledge and inducing (instructing) models toward translation genera- tion. Consequently, we propose a two-stage fine-tuning approach and introduce the LLM developed through this strategy Advanced Language Model-based trAnslator (ALMA). Specifically, given most L.LMs are trained on English-dominant data, the first stage is fine-tuning non-English mono- lingual data to enhance the model's 's proficiency in other languages involved in the translation task. Secondly, drawing inspiration from the recognized significance of data quality in other applications (Zhou al., 2023 Maillard et 2023] Gunasekar al. 2023), we fine-tune the model with small amount of high-quality parallel data.
Our main contributions are summarized as follows:
Diminished Necessity of Parallel Data Traditional translation frameworks rely on large amounts of parallel data, which may lead false impression that such data is essential for the translation task with LLMs. Prior studies have fine-tuned LLMs with datasets containing over 300M parallel instances YYang et al. 2023). However, our empirical evaluations suggest that this strategy may not be optimal, and even harm the translation capabilities of LLMss.
LLM Via A New Training Recipe: ALMA We introduce a novel two-stage fine-tuning method for translation with decoder-only LLMs. Leveraging ILaMA-22 the base model, we attain an average improvement of more than 12 BLEU and COMET scores over zero-shot performance
All COMET scores in the paper is COMET-22 (Unbabel /ww:22 comet Rei et al. 2022..
2

---

Published as a conference paper at ICLR 2024
across 10 translation directions from WMT and 2MT test datasets. Notably, the perfor- mance surpasses all previous work and is even better than the NLLB-54B model and GPT-3.5- text-davinci-003.
Efficient Computational Cost Our ablation study reveals both stages are crucial factors for achiev- ing large improvements. The most computationally intensive part is monolingual data fine-tuning, however, we show that only fine-tuning 1B monolingual tokens is sufficient to have comparable performance NLLB-54B in 10 translation directions, which only requires around 18 hours to complete with 16 MI200 GPUs.
2
PRELIMINARY
2.1 TASK DEFINITION
We consider a decoder-only transformer model parameterized by 0 for machine translation. Let x represent the source sentence and y its corresponding target sentence. We utilize fixed prompt template, denoted as I, to guide the model in generating translation. The log- log-likelihood loss of the parallel sentence (x, y) with regard to the model parameters can be formulated follows:
(x, log P(y|x,I:0)
(1)
Σ log Pog
where T' is length of the target sentence, and ye is the 1-th target token. The loss is standard causal language modeling (CLM) loss, which predicts the next token based on prior tokens. We use the same sentence-level translation prompt template suggested by [Hedyy et al. (2023). and illustrate the prompt and the model input/target in Figure Note that we do not compute the loss for the prompt template and source sentence during training (Zhang al., 2023a). In Appendix [ we show that CLM is more suitable for the translation task compared with other modeling methods, such as prefix language modeling (Wang al. 2022) and cecturr-off-denoisse (Tay al. (202a)).
Trnnliate this from (source language) to /larger lenguege): (source language]: esource sentence (target language):
Input
Prompt
Prompt
+
<target sentence>
Target
No loss computted
CLM losa
Figure 2: The prompt used for training and evaluation. /source language] and [target language] represent the full name of the language, e.g., Translate this from German English. Note that do not compute loss for the prompt.
2.2 BACKBONE LLM FOR TRANSLATION
We seek robust LLM to LLMs, we prioritize evaluating the zero-shot translation performance of these models before delv- ing into optimal training recipes. As most of these models provide
foundational model. With the recent emergence of numerous
7B version. comparative
analysis centers on this magnitude: OPT-7B (Zhang et al. 2022). Falcon-7B (Almazrouei et al.
2023 BLOOM-7B (Scao et al. 2022). MPT-7B (MosaicML 2023) ALLMMA---7BB (Touvron al.
2023a), and LLaMA-2-7B (Touvron et al. 2023b). We additionally present results from GPT-3.5- text davinci 303 (hereinafter referred as GPT-3.5-D) and GPT-3.5-turbo- 0301 (here-
inafter referred to as GPT-3.5-T) to show the performance gap
Zero-Shot Evaluation We conduct zero-shot evaluations on English-centric language pairs, considering both from English and to English directions: German (de), Czech (cs), Icelandic (is), Chinese (zh) and Russian (ru), where Icelandic test data is from WMT'21 and the oth- ers are from 2M We choose these test dataset because they are the recent and less likely
https://bette openai wtw..///o//ccmmeeeemc ndex- roreeerchrreeeerceerr

---

Published as a conference paper at ICLR 2024
to overlap the training data used by LLMs, and importantly. they have high-quality data to avoid problems of "translationese" (Zhang & Toral
2019). The beam size is We report sacre- BLEU (zh tokenizer for Chinese and 13a for the others) (Post 2018). We also report COMET (Unbabel w/w.t22 2omt:222-0 -cooett-dd (Rei et al.
2022) because BLEU only reflects the degree of lex- ical match. In this paper, We rely more COMET than BLEU due to its better alignment with human evaluations (Freitag et al. 2022
35-
XX-+on
70-
2351
15-
11/4
12.00
11.4
50
OPP
BLoOM
Fakan 18
UaMe-1 1.8
HPT-78
GPT-1--0
om
BLOOM 78
Fakun
UaM 78
WTT-78
UaM8---78
am--500
Gm-15.I
Modek
Modek
(a) BLEU
(b) COMET
Figure 3: Averaged zero-shot translation performance on 10 directions: csten, detten, istten zh+tnn, rutten, where isten is from WMT'21 test data and the others from WMT test data.
LLM Translation Performance The overall results for the LLMs are presented in Figure 3 with scores averaged across five languages for translations to and from English. Among the 7B LLMs, LLaMA-2-7B exhibits superior performance translating into English, while MPT-7B leads in trans- lations out of English, as measured by BLEU. However, when evaluated with COMET, LLaMA-2- 7B wins in both directions. We show the numeric results in Appendix [B] Consequently, we select LLaMA-2-7B and MPT-7B for further investigation into the necessity of parallel data for LLMs.
Do LLMS HAVE AN APPETITE FOR PARALLEL DATA?
Conventional machine translation training predominantly relies utilizing large volumes of parallel datasets within encoder-decoder frameworks. This trend is not confined to training models from scratch but also pertains strategies that fine-tune pre-trained LLMs, often involving millions of parallel sentences (Rothe et al., 2020 Liu et al., 2020 Xu et al. 2021 2023 Yang et al., 2023). In this section, we examine whether the recently proposed decoder-only LLMs retain a dependence on substantial parallel data and adhere to the traditional training paradigm.
3.1 EXPERIMENTAL DESIGN
Following Section 2.2 our focus narrows to fine-tuning LLaMA-2-7B and MPT-7B. To allow for deep analysis, we concentrate on one language pair, Finglish-Russian (en-tru). We opted for a language pair that is translating
of English and non-Lattin language, since those categories show larger gaps with SoTA models in our initial investigation in Section We use the clean data filtered from 75M parallel sentences from Hendy al. (2023) and split the data size 5 levels: 10K, 100K, 1M, 5M, and 20M. We use the same prompt template and training scheme as described in Section 2.1 and train the model by updating all parameters. Detailed training settings can be found in Appendix
3.2 OBSERVATIONS
The fine-tuning results for LLaMA-2-7B and MPT-7B each data size step are presented in Figure Additionally, we benchmark these against the performance of the NLLB-55B model to show the disparity with one of the SoTA multilingual translation models.
According to Freitag et al. (2022). COMET holds the 2-nd position in alignment with human ratings, whereas BI B.EU is situated the 19-th spot among 20 metrics.
4

---

Published as a conference paper at ICLR 2024
HPT 78
NPT-76
NLLB. 548
L-3MA-2-7B
85.0
ULSMA 2-78
NLLB 54B
Ransoom
Ransoom Init
25.0
22.5
20.0
77.5
(zero-shat)
15K
100k
1M
SM
20M
15K
100K
1M
SM
20M
(a) BLEU
(b) COMET
Figure 4: BLEU and COMET scores obtained during the fine-tuning of MPT-7B and LLaMA-2- 7B across each data step for en-rr.. Additionally, present the results for NLLB-54B and 7B model trained from scratch. A notable decline in LLaMA-2-7B's COMET score suggests that substantial parallel data might dilute its pre-existing knowledge.
Small Training Data Is Enough According to COMET, there is notable difference in the curve of LLaMA-2-78 and MPT-7B: LLaMA-2-77 peaks with 10K and 100K training data before experienc- ing decline, while MPT-7B exhibits continuous improvement with more training data. LLaMA- 2-7B requires only limited training examples (10K and 100K) to achieve competent translation. However, surplus of examples (5M 20M) seems dilute its existing knowledge in Russian. Conversely, MPT-7B, potentially due to its inherently weaker translation capability, exhibits im- proved performance with an increase in training data. This may suggest that LLaMA-2 or other well-trained LLMS may not necessitate substantial parallel data.
Large Parallel Data Wash Out the Knowledge Both LLMs eventually achieve similar BLEU and COMET with 20M training data, regardless of their performance smaller data. We hypothesize that this phenomenon is caused by catastrophic forgetting (French 1999 Kirkpatrick et al.) 2017). suggesting that too many parallel data wash out the pre-existing knowledge. To validate this hy- pothesis, we consider an extreme case: training the model from scratch using 20M data, therehy erasing all prior knowledge F] As expected, it tends up with similar performance in both BLEU and COMET evaluations (triangle in Figure 4. strengthening our speculation regarding the dilution of LLM's intrinsic knowledge with extensive data training.
Beyond BLEU COMET reveals a decline in translation performance for LLaMA-2-7B as the amount of parallel data increases, trend not captured by BLEU which shows increase. This discrepancy arises since BI.UU primarily evaluates lexical overlap, and the extensive WMT training data, heing similar in domain to the test set, likely enhances this measure. This highlights the neces- sity of utilizing additional metrics (like COMET) for a comprehensive evaluation of translation.
From observations, LLaMA-2 (potentially other well-trained LLMs) should not adopt the same training approach as earlier models-wheet whether randomly initialized or the-trained-that rely heavily on vast amounts of training data.
A NEW TRAINING RECIPE
We demonstrate that LLMs like LLaMA-2-7B do not voraciously consume parallel data. We in- troduce novel training strategy that markedly enhances translation performance without relying heavily on parallel data. The recipe comprises two stages: continuous monolingual data fine- tuning and high-quality parallel data fine-tuning. After applying training recipe to LLMs, name the resulting model as ALMA (Advanced Language Model-based trAnslator).
Monolingual Data Fine-tuning LLMs like LLaMA pre-trained on English-dominant corpora. This potentially explains their inadequate translation performance which necessitates cross-lingual capabilities. To ameliorate this, our first stage is fine-tuning I.LMs with monolingual data of non- English languages involved in translation tasks, enhancing their proficiency in these languages. Note
We initialize parameters randomly based on the LLMMA-2-BB model, but the vocahulary.
5

---

Published as a conference paper at ICLR 2024
that we also add English monolingual data during fine-tuning to prevent English knowledge forget- ting. Previous studies also offer some clues that monolingual data help in translation. For instance, Tan al. (2023) utilizes monolingual target corpus to bridge the gap in translation mismatches caused by domain discrepancies. BigTraansatte (Yang et al. 2023) and PolyLM (Wei et al., (023) huge amount of Chinese monolingual data and improve translation or from Chinese. Fur- thermore. Li et al. (2023) utilizes monolingual generation instructions to improve translation. In Section 6.1 show that utilizing small monolingual data and modest computational cost (e.g. IB monolingual tokens mixed by 6 languages and fine-tuning under 18 hours), can facilitate significant improvements in 10 translation directions. Note that we employ full-weight fine-tuning at this stage.
High-Quality Data Fine-tuning Drawing on insights from Section 3.2 that LLMs may require only small parallel data, coupled with previous research emphasizing training data quality (Zhou et al. 2023, Maillard et al. 2023] Gunasekar et al. 2023), we fine-tune the model using a small, yet high-quality parallel dataset this stage. To ensure the data quality, we collect human-written datasets from WMT test data and Flores-200 (NLLB TEAM et al. 2022) development and test sets. Here, we explore both full-weight and lightweight Low-Rank Adaptation (LoRA) (Hu et al. 2022 Mangrulkar et al. 2022 fine-tuning, where apply LoRA to the down-projection layer in each ford-forward network.
5 EXPERIMENTS
5.1 DATA
For our parallel training data, we collect human-written test datasets from 17 to WM plus the development and test sets from Flores-200 (NLLB TEAM et al. 2022). resulting in a total of 58K training examples across all languages. For the test data, we still use the same 10 transla- tion directions to be consistent with our study in Section 2 cattn,, detten, isten, zh+tn,, rutten, where i s is from WMT'21 and the others are from WMT' Test data in WMT"21 (except for iss is used for the development dataset (a total of 8K parallel sentences) 6 The monolin- gual dataset sourced from OSCAR (Ortiz Su' arez al. 2019 Kreutzer al. 2022). We mix the monolingual data and fine-tune the model with a sampling ratio of 20%, 14%, 8%, 19%, 22%, and 17% respectively for de, cs, is. zh, ru and en. We explain the reasoning behind the sampling ratios and show the detailed parallel data information in Appendix D
5.2 TRAINING SETUP
We train the model in many-to-many multilingual translation manner, and use LLaMA-2-7BB (or 13B) as our backbone model given its best zero-shot performance. Our two-stage fine-tuning process yields two model types, differentiated based on the utilization of LoRA:
ALMA-7B/ALMMA-13 Full- Hull-Weight fine-tuning on monolingual data followed by Full- Weil-Weight fine- tuning on high-quality parallel data for LLaMA-2-7B -13B models.
Full-Weight fine-tuning on monolingual data followed by LoRA fine-tuning on high-quality parallel data for LLaMA-2-7B or -13B models.
If using LoRA, the LoRA rank 16 and only updates 0. 1 parameters (7.7M for 7B and 12M for 13B model). Both monolingual data fine-tuning and human-written data fine-tuning basically share the same hyperparameter settings. Specifically, we fine-tune LLaMA-2 with a batch size of 256, warm-up ratio of 0.01, and sequence containing maximum of 512 tokens. For monolingual data fine-tuning, we train the LLaMA-2-7B up to 20B tokens and LLaMA-2-13BB up to 12B tokens. However, is very likely that the model would be better in translation with more monolingual data fine-tuning. For human-written data fine-tuning, we train the model for 2 epochs (enough to see clear convergence) and pick the best model with the lowest validation loss. For both stages, we adopt deepspeed (Rasley et al. 2020) to accelerate our training.
'There is development dataset for Icelandic.

---

Published as a conference paper at ICLR 2024
5.3 BASELINES
We evaluate our method against two baseline categories. First, we consider prior studies with the goal aligning with ours: leveraging LLMs for translation. Secondly, we benchmark against the current SoTA translation models. It's worth noting that this comparison isn't t entirely fair due to discrepancies in training data and model architectures (e.g. 175B GPT-3.5 vs. our 7B models). Nevertheless, utilizing the same test set provides insights into our model's current standing.
Prior Similar Work We compare our model with BigTranslate (Yang et al., 2023), which extends LLaMA-1-13B to over 100 translation directions; TIM (Zeng et al. 2023). ) which uses correct and incorrect examples to help LLM to learn translation; SWIE (Chen et al., 2023). which improves LLM in translation via instruction augmentation; and BayLing (Zhang et al., (023b), which uses interactive translation instructions. Given that the same test data and evaluation metries are utilized, directly report BLEU and COMET from their papers (except for BigTranslate, we assess their released model using the prompt they provided).
TA Models We consider the NLLB-54B model, which is the largest and best translation model released in the NLLB family (NLLB TEAM et al. 2022); and the zero-shot performance of GPT- (GPT-3.5-T). Additionally,
(GPT-3.5-D) and we present the zero-shot results for GPT-4
wettext-dviinne
de
is
zh
Avg.
Models
COMET
BLEU
COMET BLEU
COMET
BLEU
COMET
BLEU COMET
COMET
S07A Models
NLLB-54B
34.50
86.45
37.60
90.15
24.15
81.76
27.38
78.91
30.96
87.92
30.92
85.04
GPT-3.5-D, zero-sbot
31.80
85.61
31.30
88.57
15,90
76.28
38.30
85.76
27.50
86.74
28.96
84.59
GPT:3.5 serl
34.40
87.00
32.92
90.17
18.74
81.04
44.90
87.00
29.90
87.60
32.17
86.56
GPT-4, zero-shot
35.38
87.44
34.53
90.77
43.98
87.49
30.45
88.87
Prior Similar Studies
TIM-BLOOMZZ-78
20.63
74.16
37.20
84.89
TIM-LLaMA-I-78
25.59
82.56
19:33
755.48
SWIE-BLOOMZ-1B
21.85
05.
36.88
844.53
SWTE-LLaMA-IITR
27.21
82.36
B1.224
80.63
BigTrannlate-1338
21.48
78.81
20:67
80.65
2.28
35 50
28.56
811.31
17.66
78
2019
Bayling-131
78
37,92
84.62
12.77
71.01
Our Recipe with Backbome Model: LL
2:7B
LLaMA-2-7B. zero-shot
19.00
76.30
16:02
79.13
1.33
83.83
16.997
711880
16.00
73.24
13.86
68.88
ALMA-7B 7B (Ours) ALMA-7B-LnRR. (Ours)
30.31
85.5 59
29. 88
89.10
25.71
85.52
36.48
88.05
27.09
87.17
29.89
86.49
30.16
45
30.1 17
89.05
15.19
85
36.47
84.87
26.93
87.05
29.78
86.37
Recipe
Backbone Model: LLaMA-2-133
LLaMA- 2- 13B, zero-shn shot
13:69
75.55
0.87
68.57
2 36
38 47
30.00
79.70
0.59
633.8
9.50
655.2
ALMA-13B (Ours)
31.37
31.1
89 42
67
85
85.
28. 76
87.50
39
ALMA 13B LoRA (Ours)
31.47
85.62
32.38
89.79
26. 68
86.08
39.84
85.96
28.96
87.53
31.87
87.00
Table 1: The overall results in en->xx. ALMA models significantly outperform all prior simi- lar studies and are comparable to SoTA models. We categorize BLEU and COMET scores into three groups: scores that are more than 10 points below the higher value of GPT-4/GPT-3.5-T are emphasized in dark red boxes,, those that are more than 5 points below are emphasized in shallow red boxes, and all other scores are emphasized in green boxes. Bold numbers represent the highest scores among ALMA models and prior similar studies.
5.4 RESULTS
We show our main results of en- and xx xx-ten respectively in Table and 2 In summary, our best system (ALMA-13B-LORA) outperforms all previous studies, NLLLB-54B, and GPT-3.5-D, while it marginally underperforms compared to GPT-3.5-T and GPT-4
Comparing With LLaMA-2 Zero-Shot For all 10 translation directions and both 7B and 13B mod- els, LLaMA-2 trained by our recipe significantly outperforms its original zero-shot performance. For instance, ALMA-7B achieves +16.12 BLEU and +17.61 COMET for en- on average. It is worth noting that LLaMA-2-15 13B suffers from the ofr-target issue in en zero-shot translation. However, it can be substantially alleviated by few-shot in-context learning (Brown et al. 2020). but still largely lag behind our methods (e.g., over 10 BLEU and COMET when translating from English). We discuss this further in Appendix B
"GPT-4 results are sourced from Zhang et al. 2023b
7

---

Published as a conference paper at ICLR 2024
Models
de
1a
Avg.
BLBU
COMET BLEU COMET BLEU
COMET BLEU
COMET
BLEU
COMET
BLEU
COMET
So7A Models
NLLB-54B
26.89
78.94
39.11
80.13
23.09
71.66
16.56
70.70
39.11
81.88
28.95
76.66
GPT:3.5 zem shot
30.90
84.79
44.50
86.16
31.90
82.13
25.00
81.62
38.50
84.80
34.16
83,90
GPT-3.5-T, zero-shot
33,10
85.50
47.20
87.30
37.50
85.50
26.60
82.90
42.40
86.10
37.36
85.46
GPT-4, zcro-shot
33.87
85.62
48.67
87.43
27.20
82.79
43.51
86.18
Similar Studies
TIM BLOOMZ.78
24.31
77.65
23.42
79.50
TIM-LLaMA-1-77B
27.91
82.80
19.33
75544
SWIE BLOOMZ-7B
25.95
78.80
23.40
79.36
SWTE-L. -M------MA---77
30.48
82.97
21.30
76.48
BigTranslate-153
81.19
14.16
774.26
77.80
Bayling- 13B
27.34
83.02
33.87
81.65
20.
77.72
33.95
82.07
Our Recipe
with Backzoone Model: LLaMA-2-7B
LLaMA-2-7B, zere-shot
30.42
82.74
36. 56
82.42
10.98
62.79
18.19
775.00
36,02
82.84
26.43
77.16
ALMA-7B (Ours)
29.49
83.99 98
42. 91
85.90
35.26
85,97
23.52
79.73
38.93
84.81
34.02
84.08
ALMA-7B-LoRA (Ours)
83.95
43.49
33.64
79.78
39.21
84.84
84.12
Our Recipe
with Backbone Model:
LLaMA-2-138
LLaMA-2-138, zcro-shot
31.06
83.01
40.02
83,
15.77
66.35
21.81
36.50
82.91
29.03
ALMA-13B (Ours)
30.73
84.42
44.68
86.29
36.46
86.
24.65
79.90
40,37
85.00
35.38
841.40
ALMA 13B LoRA (Ours)
31.14
84.56
45.28
86.47
36.95
86.42
25.46
80.21
40.27
85.27
35.82
84.59
The overall results in xx->en. ALMA models significantly outperform all prior similar studies and are comparable to SoTA models. The color and boldface are the same Table [
Table 2:
Compared with Prior Similar Studies ALMA significantly outperforms all prior studies. Big- Translate, which is fine-tuned Chinese corpus and 300M parallel corpus, struggles surpass ILaMA-2's zero-shot performance, except for en-zh. This observation also aligns with our find- ings that an excessive amount parallel data may damage the model, whereas target monolingual data is helpful to translation. Both TIM and SWIE specifically target two high-resource languages, and zh. Their performance, however, predominantly determined by their backbone models: effective translation is observed for zh. but is lackluster for de when using BLOOMZ, and vice versa with LLaMA-1. In contrast, ALMA versatile, showcasing strong results across all directions.
Compared with SoTA models Our best model (ALMA-13B-LORA, substantially outperforms NLLB-54B and GPT-3.5-D on average. In en- direction, it even outperforms GPT-3.5-T on average COMET (87.00 86.56) and has close performance when it comes to xx->en. Notably, SoTA models typically excel with high-resource languages but falter with low-resource languages such s.. With recipe, the performance of remains strong and performs the best.
077-3.-0
18
18
48
18
78
118
118
112
145
101
175
23.5
BC
BLEU
COMEET
(a) BLEU
(b) COMET
Figure 5: The average performance of ALMA-7B the completion of each 1B IB-token fine-tuning. The scores in the figure are averaged across 10 directions
ANALYSES
How MUCH MONOLINGUAL DATA TO USE?
In our main results, present ALMA with our best settings, fine-tuned on either 20B 12B tokens. Yet, we snapshot all ALMA models after every 1B monolingual tokens (and human-wwitten parallel data) they have been fine-tuned with, and evaluate all their translation performance. As illustrated
8

---

Published as a conference paper at ICLR 2024
Avg. xx->en
Avg. en-
Use Mono.
Parallel Data Quality
BLEU
COMET
BLEU
COMET
X
X
26.43
77.16
13.86
68.88
X
Random
28.24
78.69
19.68
73.89
X
Filtered
28.39
78.94
19.56
74.35
X
HW
29.39
80.00
22.17
76.52
X
28.49
80.32
26.35
84.73
Random
32.47
83.02
26.98
83.15
Filtered
32.32
83.03
27.38
83.98
HW
34.02
84.08
29.89
86.49
Table 3:
Ablation study on the effect of monolingual data and parallel data quality. The backbone model is LLaMA-2-7B. A red cross (x) in the table denotes the omission of monolingual data fine- tuning or parallel data (indicative of zero-shot translation). green check (v) signifies that the model undergoes fine-tuning with monolingual data.
in Figure 5 we report the ALMA-7B's average performance across all directions after fine-tuning every IB tokens. The test dataset remains the same, i.e., the 10 aforementioned directions. We provide detailed numeric results and similar analysis for ALMA-13B to Appendix [F) Importantly, merely fine-tuning on IB monolingual tokens, followed by fine-tuning on human-written data, yields performance comparable to NLLB NLLB-54B and GPT-3.5-D. In practice, we employ 16 MI200 GPUs with a batch size 256 and sequence length of 512, which requires only 18 hours to complete the fine-tuning of 1B tokens and additional hour allocated for human written data fine-tuning. It takes around 19 hours of training to have a strong MMT model.
humtn-written
6.2 THE EFFECT OF MONOLINGUAL DATA AND PARALLEL DATA QUALITY
To scrutinize the impact of monolingual data, we juxtapose LLaMA-2-7B models fine-tuned with and without monolingual data (20B tokens), while keeping the same parallel data. Furthermore, to evaluate the impact of parallel data quality, we introduce three distinct parallel datasets for stage fine-tuning. The first dataset the Humann-ritter data (HW) utilized in prior experiments. The sec- ond is the filtered data (Filtered) referenced in Section [.1] Lastly. we employ randomly selected dataset (Random) sourced from the comprehensive WMT data. We anticipate the quality hierarchy as //W, followed by Filtered, and lastly, Random. For both Filtered and Random, each translation direction has 10K parallel data, aligning the total training dataset size with HW. We show the abla- tion results in Table Using the LLaMA-2-78 same parallel data, incorporation of monolingual data largely enhances translation results, e.g., an increase from 74.35 83.98 in en- COMET scores when training on the same Filtered data. Moreover, regardless of the monolingual data's presence, models fine-tuned with higher-quality data exhibit better performance. Both monolingual and human-writer data emerge as critical factors in improving translation. Detailed results for each language pair are deferred to the Appendix G
foundational model, it's evident that with the
6.3 OTHER ANALYSES
We also explore additional in-depth analyses and elaborate on them in the appendix: 1) The impact of the volume and domain human-written data on translation performance is explored in Appendix H. 2) A comparison between stage 2 fine-tuning (parallel data fine-tuning) and in-context few-shot learning can found in Appendix 3) An evaluation of the zero-shot cross-lingual capabilities of LLaMA-2 after stage fine-tumi fine-tuning other tasks is presented in Appendix
CONCLUSION
In this paper, we show that LLMs do not require as extensive a collection of parallel data as tradi- tional translation models do. Subsequently, we introduce a novel training recipe for decoder-only LLMs in translation, resulting in strong translation models, ALMA. When using our LLaMA-2 foundational model, ALMA exceeds the zero-shot translation performance of LLaMA-2 by more than 12 BLEU and COMET scores across 10 directions on average. Moreover, ALMA models surpass all preceding studies and even outperform NL.LB-54B and GPT-3.5-D.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We extend our gratitude Hieu Hoang, Marcin Junczys-Dowmunt, Yunmo Chen, Steven Tan, Huda Khayrallah, Thamme Gowda, Vikas Raunak, Matt Post, Anoop Kunchukuttan, Roman Grund- kiewicz, Tom Koemi, Kenton Murray and Arul Menezes for their insightful and valuable sugges- tions.
REFERENCES
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. arXiv preprini vrXiiv/903.000 00089, 2019.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-art performance. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Improving translation faithfulness of large language models via augmenting instructions. arXiv preprini arXiv:2308. /2674, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231 2022.
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv: 000.000090000,,, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440 8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 11111163/1/2220..oo main. 747. URL https: //aclanthology.ooom wtt.ogg....////// acl maal-maaiin maim. 747
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. Results WMT22 metrics shared task: Stop using BLEU neural metries better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https:
//aclanthologe www.looy..o..////. 2022. vmmt-.2
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL 5371628
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Texthooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? comprehensive evaluation. arXiv preprint arXiv:2302.092 10, 2023.
10

---

Published as a conference paper at ICLR 2024
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: http///peeeeeeeeeeeeeeeeeeeeeeee
/ www.:/eeettorrrnm
id=nZevKeeFYf9
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study, arXiv preprint arXiv:2301.0877 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-3526, 2017.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-rrsikkh, Allahsera Tapo. Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa- padimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou. Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ball, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayo- dele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality Transactions of the Association for Computational Linguistics, 72, 2022. doi: 10.1162/ .0044.. URLlhttps: http://caaantooooy..oooooooooooa www.://ggg/....ooo http:/22222.ac11....
glance: An audit of web-crawled multilingual datasets.
Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Chen, and Jiajun Chen. Eliciting the translation ability of large language models via multilingual finetuning with translation instructions. arXiv preprint 2rXiv:2305.15088 2023.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual language models. arXiv preprint 1rXiv:2112.1066 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot -arameter-efficient fine-tuning is better and cheaper than in-context learning, Advances in Neural Information Processing Systems, 15:1950-1965, 2022.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine trans- lation. Transactions of the Association for Computationnl Linguistics, 8:726-742. 2020. doi:
16.1////l 00343. URL htR.ttts::// //ac antho ogy . .rgg 2020 taw.taacll....
Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzmán. Small data, big impact: Leveraging minimal data for effective machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2750-2756, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https: com/ hucginngacce/peftt 2022.
MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023.
URL www www.mosaicco. cml. com/ blog/mpt- Accessed: 2023-05-05.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few- shot fine-tuning vs. in-context lcarning: A fair comparison and evaluation. arXiv preprint 2.Xiv:23005/66338 2023.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. LS- DSem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Link- ing Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, Valencia, Spain,
11

---

Published as a conference paper at ICLR 2024
April 2017. Association for Computational Linguistics. doi: 10. 10.18653//1/W17-00000 URL
Marta R NLLB TEAM, Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield,
Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et No language left behind: Scaling human-centered machine translation. arXiv preprint ar arXiv:2207.04672. 2022.
OpenAI. Gpt-4 technical report, 2023.
Pedro Javier Ortiz Su' arez, Benoit Sagot, and Laurent Romary. Asynchronous pipelines for process- ing huge corpora medium to low resource infrastructures. Proceedings of the Workshop Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pp. 9 16, Mannheim, 2019. Leibniz-Institut frur Deutsche Sprache. doi: 10.14618A0 1/446///ds-p
URL http: /bbp::///bn--ie
nbn reso wew.ovinggg..eeee.....e...en.ioe de / urn nbn: de bsz mh39 90215
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics,, pp. 311-318, 2002.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Con- ference Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octo- ber 2018. Association for Computational Linguistics. doi: 10.186653w1/W118-63333 URL
wwtp:///hwooooooo//1//o////111 6319
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Vikas Raunak, Arul Menezes, and Hany Awadalla. Dissecting in-context learning of translations in GPT-3. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 866-872 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/11/2023.. findings-emnlpp..
URL https://
aclanthology. wwwp/rgg.........
20222223..2222 feeffffinninggg emnlp.61 61
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https::///// htta//aaaaanhlllcooooo org/2022.
wmt 1.52
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics. 8: 264-280. 2020. doi: 10.1162/tacl 00033. URL
tacl 11
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXi::221..05100 2022.
Weiting Tan, Haoran Xu, Lingfeng Shen, Shuyue Stella Li, Kenton Murray, Philipp Koehn, Ben- jamin Van Durme, and Yunmo Chen. Narrowing the gap between zero- and few-shot machine translation by matching styles, 2023.
Yi Tay, Mostafa Dehghani. Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Dara Bahri, Tal Schuster. Steven Zheng, et al. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2022a.
12

---

Published as a conference paper at ICLR 2024
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11 199, 2022b.
Alexey Tikhonov and Max Ryabinin. It's All in the Heads: Using Attention Heads as a Base- line for Cross-Lingual Transfer Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3534-3546, Online, August 2021. Associ-
ation for Computational Linguistics. doi: 10. 18653/v1/2022 hit.2222...nniingg-a....... URL https: findingsn 310
//aclanthology.orgg/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. (39,2 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.09288 2023b.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pp- 22964-22984. PMLR, 2022.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. Polylm: An open source polyglot large language model. arXiv preprint MXXiv:3377006116 2023.
Haoran Xu, Benjamin Van Durme, and Kenton Murray. BERT, mBERT, or BiBERT? study on contextualized embeddings for neural machine translation. Proceedings of the 202/ Con- ference on Empirical Methods in Natural Language Processing, pp. 6663-6675, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 10.18633/1/2021 111665/5111222212mmmmmmmm3333330 emm maannp-mmin... 534
hnttps://aclaateee
org/2021
Haoran Xu, Jean Maillard, and Vedanuj Goswami. Language-aware multilingual machine transla- tion with self- supervuuee learning. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 526 226-539, Dubrovník, Croatia, May 2023. Association for Computational Lin- org/
guistics. doi: 10. 2023. findinin -aa.ndiins-aall.38 e8
URL
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtrans: Augmenting large lan- guage models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305. 18098, 2023.
Jiali Zeng, Fandong Meng, Yongjing Yin, and Jie Zhou. Tim: Teaching large language models to translate with comparison. arXiv preprint arXiv:2307.044008 2023.
Mike Zhang and Antonio Toral. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume I: Research Papers), pp. 73-81, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/
v1/W19-522 NWW.. URL
http:/:://aaaaahhhoacass org/ htse/w19-5522 5208
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuni tin-tuning of language models with zero-init atten- tion. arXiv preprint arXiv:2303. 16199, 2023a.
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. arXiv preprint arXiv:2306. 109668.8....8 2023b.
Susan Zhang, Stephen Roller. Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 2022.
13

---

Published as a conference paper at ICLR 2024
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. 11206, 2023.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint 0406552304.04675 2023a.
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Ji- ajun Chen, and Lei Li. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.044948 2023b.
14

---

Published as a conference paper at ICLR 2024
A COMPARING LLM TRAINING OBJECTIVES FOR MACHINE TRANSLATION
We evaluate three potential training objectives for decoder-only LLM in machine translation.
Causal Language Modeling (CLM) We first consider standard language modeling loss that predicts the next token based on all prior tokens.
Prefix Language Modeling (Prefix LM) For decoder-only models, a prefix can be defined with non-causal attention mask. Analogous to standard language modeling, the model predicts each token outside the prefix based on previous tokens. In the context of machine translation, the provided prompt serves the prefix, as depicted in Figure 2
Centure-of-Denoisee (MoD) The UL2 model TTy et al. 2022a) introduces a unified approach masking methods, utilizing a mixture-of-deeoiser (MoD) strategy, which has also been imple- mented in the fine-tume tinettuning of PaLM (Tay et al. 2022b). This strategy is grounded in three objectives:
Regular Denoising: In this approach, noise is sampled in spans and replaced with sentinel tokens, aligning with the standard span corruption technique delineated in Raffel et al. (2020). The parameters set for this objective include a mean of 3 and a corruption rate of 15
Extreme Denoising: This method amplifies the noise to a comparatively "extreme' level, characterized by mean length of 32 and corruption rate reaching up to 25
Sequential Denoising: This is known as the Prefix LM objective previously mentioned.
In our training process, we allocate a 25% probability each for both regular and extreme denoising, and 50% probability for sequential denoising.
We employ the MPT-7B as our backbone model. Our investigation considers four distinct training data sizes: (zero-shot), 100K. 1M, and 5M, with translation directed from Russian English. We use the parallel dataset previously described in Section 3.1 For each data size, the MPT-7B fine-tuned using the corresponding training objective, noting that all trainings utilize full-weight fine-tuning.
The results of the comparison between training objectives can viewed in Figure 6 Although three objectives end up with similar performance under 5M training data, both prefix LM and MoD markedly lag behind CLM under limited parallel data (100K or IM). Surprisingly, with 100K, mod- els fine-tuned using prefix LM and MoD even underperform their zero-shot performance. Con- versely, CLM demonstrates a healthy improvement as the amount of parallel data increases. Conse- quently, adopt CLM as primary training objective for machine translation.
CLM
CLM
Profic UH
Pizfi MoO
35-
6331
3235
76.28 7AJ3
75.25
1.5
6 (000-4100)
100K
1M
9N 62 0 1140-4900
10EK
9N
(a) BLEU
(b) COMET
Figure 6: The comparison of translation performance across various training objectives and parallel data sizes is depicted. or datasets of 100K and IM, both prefix LM and MoD lag behind CLM and even undeperform the zero-shot performance. Notably, only CLM demonstrates a healthy improve- ment as the volume of training data increases.
15

---

Published as a conference paper at ICLR 2024
B FULL RESULTS OF ZERO-SHOT EVALUATION
In Section 2.2
we present the average zero-shot translation performance of recently released LLMs.
Detailed results for each translation direction can be found in Table
Models
de
is
zb
Avg.
BLEU COMET
COMET BLEU COMET BLEU COMET
BLEU
COMET
BLEU
COMET
slatingi from English (en-+xx)
OPT-7B
9.79
65.74
51.55
1.42
45.66
48.84
1.31
41.57
3.41
50.67
BLOOM-7B
7.31
62.21
3.09
56.22
1.49
49.97
20.41
74.03
5.89
56.55
7.64
59.80
Faclon-7B
19.23
77.30
5.86
57.04
1.69
37.53
26.90
79.28
4.61
53.55
11.66
60.94
I.L.MA---7B
21.00
79.50
16.31
16.16
2.42
34.92
15.63
68.03
17.61
76.73
14.59
67.47
MPT-7B
20.91
78.56
11.95
69.80
3.21
41.71
25.41
80,20
13.99
72.43
15.09
68.54
LLaMA-2-7B
19.00
76.39
16.02
79.13
1.33
43.83
16.97
71.80
16.00
73.24
13.86
68.88
Trunslating to English (XX >en)
OPT-7B
24,43
78.37
14.82
66.86 86
3.13
52.63
3.35
54.34
4.47
53.30
10.04
61.10
BLOOM--B
22.06
74.10
6.06
55.18
2.14
48.70
13.66
74.62
20.06
69.27
12.80
64.37
Faclon-7B
29.21
82.02
20.06
71.15
4.29
52.53
19.45
76.68
19.50
73.19
18.50
71.11
LLaMA-1-7B
29.14
81.90
32.93
81.18
6.78
58.15
13.29
72.09
32.93
81.71
23.01
75.01
MPT-7B
29.32
81.80
27.45
76.12
12.44
62.76
19.72
77.25
30.55
79.21
23.90
75.43
L.MMA-2-7B
30.42
82.74
36.56
82.42
10.98
62.79
18.19
75.00
36.02
82.84
26.43
77.16
Table 4: The detailed results of LLM zero-shot performance in Figure
C TRAINING DETAILS
We fine-tune the backbone model using warm-up ratio of 0.01, a maximum sequence length of 512 tokens, and a weight decay of 0.01. The test data from WMT' serves as our development set. The training spans 3 epochs (for MPT-7B as detailed in Section 3 and 2 epochs for LLaMA-2 human-written data fine-tuning). The best model selected based on the lowest validation loss, with validation performed every 10% of the total training progress. We utilize 16 MI200 GPUs for training; each GPU manages batches and has a gradient accumulation step of 4, yielding an effective batch size of 256. The peak learning rate is set at 2e-5, with inverse square learning rate decay to O. The training operates under fp16 6 precision, facilitated by deepspeed Rasley et al. 2020), employing ZeRO stage 2.
D DATA INFORMATION
D.1 SAMPLING RATIO FOR MONOLINGUAL DATA
In 5 we observe a substantial imbalance in the volume of monolingual data available for differ- ent languages, denoted by their respective word count: Specifically, the English language dataset contains 523.9B 9B words, vastly outnumbering other languages, such as Icelandic, which contains 0.3B words. Utilizing unmodified concatenation and shuffling approach for this data would dis- proportionately prioritize English, undermining our objective of enhancing the model's proficiency in non-English languages. To address this, straightforwardly
the sampling ratio for English as of the probability
P(l= g. thereby ensuring balanced learning emphasis. The remaining ee allocation employs temperature sampling, suggested by AAharoni al. (2019), technique preva- lently adopted in the processing of unbalanced multilingual machine translation. Consequently, the process of selecting monolingual example from language adheres to the following distribution:
D: P(I) 'EVEL Dr
PP()) 5
(3)
PEL
where Dj is the amount of the data in language 1, T is the temperature, and L is the set of all languages except for English. The temperature we use is 6.
16

---

Published as a conference paper at ICLR 2024
Parallel Data
Monolingual Data
Train
Development
Test (from English)
Test English)
Words
Sampling Ratio
German (de)
14211
1002
2037
1984
73.8B
20%
Czech (cs)
12076
1002
2037
1448
9.7B
14%
Icelandic (is)
2009
1000
1000
0.3B
8%
Chinese (zh)
15406
1002
2037
1875
44.4B
19%
Russian (ru)
15000
1002
20.37
2016
78.0B
22%
English (en)
523.9B
17%
Table The statisties for the data we utilize for the monolingual data fine-tuning and Human-writted data fine-tuning.
D.2 DATA STATISTICS
We show data statistics in Table 5 The training parallel data is sourced from the WMT' 17 to WM The development data was acquired from WMT'21, and the test data was derived from WM with the exception of the Icelandic dataset, which was procured from WMT This means, Icelandic does not have development dataset. Additionally, the monolingual data was tracted from the Oscar dataset.
E OFF-TARGET ISSUE FOR LLAMA-2-13B
In the zero-shot scenario, the performance of LLaMA-2-13 is reasonable for translations into En- glish. However, we identify significant tar-target issue with LLaMA-2-13B when translating from
English other languages. This issue is highlighted in Table 6 using red highlighted box An illustrative example of the off-target issue is provided below:
Translate this from English to Russian: English: Plug the wall charger (not included) to a power outlet, and then connect your eReader the wall charger.
Russian: Comment: I'm voting to close this question as off-topic because it is not about programming.
Models
de
GS
1s
2h
FU
Avg.
BLEU COMET BLEU COMET BLEU COMET COMET BLEU COMET BLEU COMET
Backbome Model. LLaMA-2-133, Transloting from English (en-*xx)
zero-shot
13.69
75.55
0.87
68.57
2.36
38.47
30.00
79.70
0.59
63.84
9.50
65.23
Prompt in Target Language
20.80
2.05
40.80
22.66
20.65
73.97
Filtered 1-shot
25.71
80.85
20.77
81.30
2.78
42.97
31.70
82.12
22.32
83.03
20.66
74.05
Filtered 5-shot
26.32
81.67
20.89
81.45
2.78
42.62
32.01
82.02
23.26
83.28
21.05
74.21
HW 5-shot
26.33
82.63
21.87
82.66
3.04
41.93
82.65
22.77
84.21
20.95
74.82
ALMA-I3B-LoRA (ours)
31.47
85.62
32.38
89.79
26.68
86.08
39,84
85.96
28.96
87.53
31.87
87.00
Backbonne Model:
LLaMA
138, Trunslating to English
(xx-++
zero-shot
31.06
83.01
40.02
83.27
15.77
66.35
78.10
36.50
82.91
29.03
78.73
Prompt in Target Language
31.06
83.01
40.02
83.27
15.77
66.35
21.81
78.10
36,50
82,91
29.03
78.73
Filtered I-shot
30.75
82.91
39.47
82.90
13.71
64.73
21.00
78.35
37.13
82.85
.1
78.35
Filtered 5-shot
30.92
83,41
41.44
83.81
17.85
68.22
19.86
78.15
36.46
82.26
29.31
79.17
HW 5 S-shot
31.52
83.57
42. 10
84.69
17.88
69.93
23.26
79.36
37.42
84.
30.44
80.33
ALMA--IB-LORA (ours)
84,56
45.28
36.95
86.42
40.27
Table 6: We demonstrate the taf-ttrget problem encountered during shro-shot translation from En- glish to other languages using the LLaMA-2-133 model. Instances of this issue are highlighted within red boxes Implementing prompts in the target languages and incorporating few-shot learn- ing can markedly alleviate this issue. It is pertinent to note that the quality of the shots also influences the final outcomes.
Expectedly, the model should produce translations in Russian. Yet. LLaMA-2-13B outputs "I'm m voting to ...", indicating a misinterpretation of the task, potentially linked to its pre-training phase. We address this off-target behavior through two methods.
Prompt in the Target Language One approach is to utilize prompts in the target language (Rau- nak et al. 2023). For instance, when translating from English to Chinese, the preferred prompt
17

---

Published as a conference paper at ICLR 2024
is: "
\n <source sentence>\nnt \ as opposed to "Translate this from English to Chinese: Chinese:)nEnglishe. nEnglish: sentence" \nChinese:". Employing this technique markedly enhances the zero-shot performance of LLaMA-2-13B. Specifically, the BLEU score calates from 0.87 to 20.80 for en and from 0.59 to 22.66 for en-
In-Context Few-Shot Learning Employing in-context few-shot learning by including several ex- amples within the prompt has proven effective. We investigate both 1-shot and 5-shot learning scenarios. As delincated in Section ![ we utilize two sets of examples: Filtered, extracted from the WMT training data, and another set randomly chosen from humann-wrttern data, termed HW. Table @demonstrates that both I-shot and 5-shot configurations effectively counteract the off-target chal- lenges. Few-shot learning exhibits performance comparable to the strategy of using prompts in the target language. Moreover, echoing observations from Section[] examples of numan-written quality outperform those from the Filtered set.
Nevertheless, both strategies trail behind our proposed solution by a margin of approximately BLEU and COMET points during translations into English, and by over 10 BLEU and COMET points in translations originating from English.
Models
de
G5
5
7
ru
Avg.
BLEU COMET BLEU COMET BLEU COMET BLEU COMET BLEU COMET
BLEU COMET
Translating from English (en +xx)
NLLB-54B
34,50
86.45
37.60
90.15
24.15
81.76
27.38
78.91
30.96
87.92
30.92
85.04
GPT-3.5-D
31.80
85.61
31,30
88.5
15.90
76.28
38,30
85.76
27.50
86,74
28.96
84,59
TB
28.00
84.24
"5:.40"
81
21:38
83.08
"35.34"
84. 80
25.488
86.31
21.16
83.113"
2B
29.68
85.04
27.18
88.00
23.49
84.30
36.12
85.10
16.17
86.56
28.53
85.80
3B
29.25
84.82
28.26
88.31
23.60
84.62
37.06
85.27
26.38
86.72
28.91
85.95
4B
29.61
85.24
28.27
88 29
23.90
84.42
37.26
85.40
27.02
86.91
29.21
86.05
5B
29.52
85.04
28.29
88.43
23.85
84.58
37.19
85.42
26.50
86.85
29.07
86.06
6B
29.49
85.01
28.45
88.43
24.31
84.63
37.16
85.45
26.92
86.90
29.27
86.08
7B
29.46
85.11
28.25
$8.45
24.27
84.78
37.26
85.43
26.80
86.95
29.21
86.14
8B
29.31
84.92
27.93
88.33
23.84
84.74
37.19
85.30
26.27
86.76
28.91
86.01
9B
29.36
84.85
27.86
88.11
24.43
84.60
37.15
85,30
26.41
86.52
29.04
85.88
10B
29.47
84.82
29.18
88.41
25.59
85,09
37.41
85.31
27.71
86.98
29.87
86.
11B
29.55
85.14
28.94
88.41
25.38
85.18
37.60
85.43
27.32
86.96
29.76
86.22
12B
29.71
85.02
28.78
88 49
25.10
84,98
37.75
85.47
27. 64
86.99
29.80
86. 19
12B.
beam 5
31.37
85.45
31.12
89.42
26.67
85.85
39.05
85.76
28.76
87.50
31.39
86.80
Tronslating to Anglish (xx--Fa)
NLLB-54B
26.89
78.94
39.11
80.13
23.09
71.66
16.56
70.70
39.11 11
81.88
28.95
76.66 66
GPT-3.5-D
30.90
84.79
44.50
86.16
31.90
82.13
25.00
81.62
38.50
84,80
34.16
83,90
IB
30.66
84.36
43.71
86.06
34.96
85,54
23.22
79.88
38.87
84.88
34.28
84,14
2B
30.26
84.32
42.46
85.86
34.30
85.63
22.66
79.88
37,30
84,70
33.40
84.08
3B
30.14
84.27
42.22
85.98
34.55
85.79
22.56
79.64
38.31
84.77
33.56
84.09
4B
30.14
84.38
42.84
86.03
34.86
85.75
23.18
79.95
38.45
84,90
33.89
84,20
5B
30.20
84.42
42.89
86.14
34.52
85.87
23.32
80.07
38.07
85.02
33.80
84.30
6B
30.22
84.35
42.85
86.22
34.75
85.96
23.40
79.94
38.25
84,90
33.89
84,27
7B
30.37
84.36
42.77
86.11
35.86
86.12
22.76
79.86
37.95
84,90
33.94
84,27
8B
30.16
84.33
43.25
85.98
34.85
85.83
22.90
79.82
37.42
84.84
33.72
84.16
9B
30.11
84.30
42,90
85.97
35.21
85.85
22.50
79.52
37.74
84.92
33.69
84.
10B
29.93
84.32
43.02
86.10
35.98
86.09
22.54
79.77
37.86
84.88
33.87
84.23
11B
30.57
84.33
43.42
86.11
36.19
86.14
22.98
79.84
38.40
84.88
34.31
84.26
12B
30.40
84.30
43.16
86.17
35.73
86.19
23.89
80.17
38.49
84.89
34.33
84.34
12B,
beam 5ize=5
30.73
84.42
44.68
86.29
36.46
86.30
24.65
79.90
40.37
85.09
35.38
84.40
Table The comprehensive numeric results for LLaMA-2-13B fine-tuned by every 1B monolingual tokens followed by human-written data fine-tuning.
F NUMERIC RESULTS FOR MODELS FINE-TUNED WITH EVERY IB TOKENS
In Table 7 and results for LLaMA-2-13B and LLaMA-2-7B are presented. Both models were fine-tuned at every IB-token interval (comprising six languages) before subsequent fine-tuning with human-written parallel data. Full-weight fine-tuning was employed to ensure a consistent compar- ison. During inference, the 7B models utilized beam search of size 5, while the 13B models adopted a greedy search strategy. For 13B models, we only utilize a beam size 5 for the final models reported in the main manuscript (Table I and
18

---

Published as a conference paper at ICLR 2024
The data from these tables highlight that fine fine-tuning only IB tokens, followed by human-writter data fine-tuning, is adequate to compete with or even outperform the state-of-the of--th--tte-tttt (SoTA) models.
Models
de
GS
za
ru
Avg.
BLEU COMFT
BL.F.
COMET BLEU COMET BLEU COMET BLEUU COMET BLEU COMET
Translating from English (en-
NLLB-54B
34.50
86.45
37.60
90.15
24.15
81.76
27.38
78.91
30,96
87.92
30.92
85.04
GPT-3.5-D
31.80
85.61
31.30
88.57
15.90
76.28
38.30
85.76
27.50
86.74
28.96
84.59
1B
28:40
84,45"
26.99
87.91
20.64"
"3.22"
"33.09
84:41
25.10"
"86.3""
27.224
83:26
2B
28.96
84.62
28.05
88.34
22.23
84.44
34.39
84.08
26.02
86.37
27.93
85.57
3B
29.10
84.66
28.68
88.46
23.23
84.74
35.50
84.40
26.35
86.75
28.57
85,80
4B
29.02
84.75
28.14
88.53
23.78
84.94
35.51
84.65
26.22
86.68
28.53
85,91
SB
29.34
84.89
29.00
88.82
24.16
84.76
35.82
84.71
26.21
86.74 74
28.91
85,98
6B
28.78
84.61
28.31
88.56
23.85
84.82
34.96
84.43
26.03
86.67
28.39
85.82
7B
28.72
84.83
27.72
88.49
23.88
84.86
35.18
84.33
26.17
86.54
28.33
85.81
8B
29.03
84.78
28.76
88. 68.64
23.49
84,94
35.38
84.66
26.42
86.45
28.62
85.89
9B
28.97
84.79
28.06
88.39
23.57
85.04
35.11
84.49
26.20
86.70
28.38
85.88
10B
29.25
84.81
27.97
88.52
23.55
85.08
35.60
84.66
26.18
86.58
28.51
85,93
11B
29.62
85.23
28.77
88.68
24.27
85.08
35.75
84.73
26.55
86,91
28.99
86.13
12B
29.85
85.15
28.90
88.67
24.68
85.27
36.31
84.78
26.95
87.00
29.34
86.17
13B
29.88
85,20
29.30
88.80
24.78
85.24
36.35
84.77
26.98
87.05
29.46
86.21
14B
29.95
85.23
29.59
89.09
25.02
85.20
36.37
84.83
27.00
87.10
29.59
86.29
15B
30.10
85.22
29.79
89.09
25.21
85.40
36.27
84.78
27.37
86.94
29.75
86.29
16B
30.12
85.32
29.65
89.14
24.87
85.34
36.58
84.93
26.97
86,98
29.64
86.34
17B
30.07
85.32
29.32
88.71
25.28
85.13
36.24
84.89
27.43
87.05
29.67
86.22
18B
29.63
85.40
29.14
89.02
25.11
85.33
36.64
84.96
26.96
87.02
29.50
86.35
19B
30.01
85.25
29.75
89.06
25.66
85.37
36.87
85.11
27.13
86,98
29.88 89,88
86.35
20B
30.31
85.59
29.88
89.10
25.71
85.52
36.48
85.05
27.09
87.17
29.89
86.49
Trunslating
Engllish
(xxx)>x))
NLLB-54B
26.89
78.94
39.11
80.
23.09
71,66
16.56
70.70
39.11
81.88
28.95
76.66
GPT-3.5-D
30.90
84.79
44.50
86.16
31.90
82.13
25.00
81.62
38.50
84.80
34.
83,90
TB
29740
83,99
41.64
35.34
33335"
84,76
222:45"
79.12"
"38.15
84:34"
333:00
883355
2B
29.53
84.00
43.32
85.66
33.79
85.17
22.19
78.98
38.82
84.59
33.53
83.68
3
30.15
84.00
43.08
85.79
34.43
85.47
22.70
79.29
39.32
84.61
33.94
83.83
4B
29.82
83.98
43.26
85.92
34.55
85.59
23.27
79.84
39.00
84.62
33.98
83,99
5B
30.09
84.15
43.39
85.97
35.26
85.77
23.65
80.05
38.81
84.65
34.24
84.12
6B
30.26
84.00
43.91
85.86
35.46
85.82
23.75
79.85
39.37
84.58
34.55
84.02
7B
29.44
83.87
42.53
85,90
34.33
85.71
23.23
79.76
38.60
84.58
33.63
83,96
8B
29.69
84.00
42.85
85.68
34.38
85.69
22.92
79.31
38.54
84.47
33.68
83.83
9B
29.76
83,94
42.89
85,90
34.47
85.46
23.03
79.57
38.01
84.50
33.63
83.87
10B
29.05
83.87
41.88
85.61
33.69
85.40
22.97
79.29
37.83
84.28
33.08
83.69
IIB
29.39
83.82
43.42
85.95
34.87
85.69
22.68
79.61
38.13
84.44
33.70
83,90
12B
29, 49
84.00
43.22
85.97
35.24
85.74
22.94
79.65
38.72
84.59
33.92
83,99
13B
29.51
83.94
42.90
85.88
35.34
85.78
22.80
79.64
38.65
84.60
33.84
83.97
14B
29.65
83,93
42.89
85.83
35.47
85.83
22.40
79.65
38.67
84.70
33.82
83,99
15B
29.48
83.89
43.21
85.93
35.72
85.88
22.74
79.34
39.04
84.63
34.04
83,93
16B
29.67
83,99
43.27
86.03
35.54
85,90
22.95
79.57
39.28
84.79
34.14
84.06
17B
29.48
83.93
43.16
85.81
35.19
85.81
22.90
79.35
38.50
84.66
33.85
83.91
18B
29.43
83.98
43.25
85.94
36.16
85.83
23.57
79.69
38.32
84.64
33.95
84.02
19B
29.54
84.06
42.85
85.83
35.97
86.03
23.42
79.56
39.34
84.83
34.22
84.06
20B
29.49
83.98
42.91
85,90
35.26
85.97
23.52
79.73
38.93
84.81
34.02
84.08
Table 8: The comprehensive numeric results for LLaMA-2-7B fine-tuned by every 1B monolingual tokens followed by written data fine-tuning.
G DETAILED RESULTS IN ABLATION STUDY
We show the detailed results of the ablation study on the effect of monolingual data and the quality of the data in Table
H IS MORE HUMAN-WRITTEN PARALLEL DATA BETTER?
The composition of our human-writ written data consists of the prior-year WMT test sets (approximately 10K parallel sentences per pair) and Flores data (around 2K per pair). In this analysis, assess the
19

---

Published as a conference paper at ICLR 2024
Use Purailel Data Quality
de
C5
2h
20
Avg.
BLEU
COMET
BLEU
COMET
BLEU COMET
BLEU
COMET
BLEU
COMET
BLBU
COMET
Tronslatin, from Fngglish (an-
x
19.00
76.39
16.02
79.13
1.33
43.83
16.97
71.80
16.00
73.24
13.86
68. 88
Random
22.74
78.06
19.38
79.59
50.45
27.66
79.70
22.40
81.64
19.68
73.89
Filtered
21.92
77.59
19.93
80.24
6.91
51.42
27.15
80.09
21.90
82.42
19.56
74.35
HW
27.30
83.46
22.59
84.59
3.61
45.81
33.74
83.81
23.63
84,94
22.17
76.52
x
27.44
84.17
28.61
88.57
21.55
83.69
28.51
81.56
25.65
85.67
26.35
84.73
Random
27.38
82.12
28.43
86.82
21.65
80.53
31.68
81.73
25.74
84.53
26.98
83.15
Filtered
27.97
83
28.45
87.26
23.03
82.40
31,55
82.26
25.92
84.84
27.38
83.98
HW
30.31
85.59
29.88
89.10
25.71
85.52
36.48
85.05
27.09
87.17
29.89
86,49
Translating
Engilish
(xxx-een)
X
x
30.42
82.74
36.56
82.42
10.98
62.79
188
75.00
36,02
82.84
26.43
77.16
Random
77.24
Filtered
29.29
82.42
38.41
82,80
17.89
69.05
19.22
77.41
37.12
83.04
28.39
78.94
HW
84.31
22.51
78.77
38.56
83.88
29.39
x
28.28
82.48
38.05
84.18
32.79
84.07
9.44
69.71
33.88
81.18
28.49
80.32
Random
28.89
82.74
40.64
85.01
35.11
85.67
19.50
77.67
38.19
84.01
32.47
83,02
Filtered
28.63
82.85
40.93
84.85
35.12
85.49
19.04
77.92
37.90
84.02
32.32
83,03
HW
29.49
83.98
42.91
83.90
35.26
83.97
23.52
79.73
38.93
84.81
34,02
84.08
Table 9: Detailed results of ablation study on the effect of monolingual data and parallel data qual- ity. The backbone model LLaMA-2-7B. A red cross (X) in the table denotes the omission of monolingual data fine-tuning parallel data (indicative of zero-shot translation). A green check (V) signifies that the model undergoes fine-tuning with monolingual data.
Avg. xx-
Avg. en
Parallel Data Used
BLEU
COMET
BLEU
COMET
Backbone:
LLMMA-2-BB After Stage I
Flores
30.50
83.24
29.28
86.52
WMT
34.02
84.08
29.89
86.49
Table 10: The performance of LLaMa-2-78 (post stage fine-tuning) when fine-tuned exclusively on Flores versus when fine-tuned on both WMT and Flores.
impact of additional human-written parallel data. Specifically, we compare models (LLaMa-2-7B after stage 1) fine-tuned exclusively on Flores against those fine-tuned on both Flores and WMT data. Results can be found in Table TO] Notably, upon integrating WMT data into the training set, we discern a modest improvement in COMET scores. However, there's 's an uptick in BLEU scores, particularly for translations into English. We attribue the increase in lexical match (BLEU) to the domain alignment of WMT data. Consequently, our hypothesis is that while an augmented volume of human-write written data might marginally enhance segment- human judgment correlation (COMET), in-domain data can significantly enhance lexical matching.
Avg. xX xx-
Avg. en -xx
Methods
BLEU
COMET
BLEU
COMET
Backbone:
LLaMA-2-133
13B
After Stage I
Zero-Shot
33.07
83.07
26.76
84.03
Filtered 5-shot
33.12
83.13
27.50
83.78
HW 5-shot
33.75
83.91
27.59
85.24
Our Stage 2
34.31
84.12
29.78
86.37
Our Stage 2 + HW 5-shot
34.14
84.27
28.56
85.87
Table 11: The performance between 5-shot ICL and stage fine-tuning using the L.aMA-2-13BB model post stage 1 as the backbone. Our findings indicate that the quality of shots affects ICI. performance. Notably, stage fine-tuning markedly surpasses the 5-shot ICL and ICL does not help more stage 2.
I PARALLEL DATA FINE-TUNING VS. IN-CONTEXT LEARNING
An alternative way to instruct the model to have better translation is in-context learning (ICL) (Brown et al. 2020), as opposed to additional fine- munituning on parallel data. However, ICL is lim- ited to only few shots given the length of translation examples, while fine-tuning leverage
20

---

Published as a conference paper at ICLR 2024
entirely available data. For ICL, consider 5-shot evaluations. 5 examples are randomly selected from Filtered data (the Quality-Ramdon examples used by Hendy et al. (2023)). We also consider another 5 examples randomly from the suman-written data examine the impact of example qual- ity. We here compare the performance of our fine-tuning method and 5-shot ICI. We assess the LLLMA-2-13B after stage 1 (12B token fine-tuning) and present results in Table 11
Interestingly. ICL also holds the same property that higher quality data leads to better performance (Filtered 5-shot vs. HW 5-shot). Moreover, expected, ICL substantially underperforms our stage 2 fine-tuning possibly due to the small examples provided, which aligns with the findings in the previous work (Liu al., 2022] Mosbach et al. 2023). This could also clarify why implementing ICL subsequent to stage 2 yields no additional benefits, as all high-qualit data has already been incorporated during stage 2 fine-tuning (the last row the Table).
J CROSS-LINGUAL PROFICIENCY OF FINE- TUNED MODELS
We explore the cross-linguat competencies of our models derived from LLaMA-2 after fine-tuning them on monolingual data. Our aim is to discern whether augmenting monolingual data enhances performance cross-lingual tasks. Experiments were conducted on zero-shot pross-lingual tasks en- compassing three benchmarks: Cross-lingual language understanding XStoryCloze- - translation of the English StoryCloze dataset into ten languages (Mostalazadeh et al. 2017), and XWinograd multilingual compilation of Winograd Schemas Tikhonov & Ryabinin (202I). Evaluations were restricted to languages overlapping with our ine-tuned lan- guages, namely, German (with only XNLII being inclusive), Chinese, Russian, and English. Unfor- tunately, none of these datasets covers Icelandic. We first consider baselines for some widely used models: XLM-R large (Conneau al. 22200), XGLM-7.5B (Lin al.. (02I). BLOOM-7B (Scao) itt al. (002), and MPT-7B (MosaicML, 2023). In these comparisons, LLaMA-2 demonstrates the top performance for the tested languages. Subsequent fine-tuning with either IB or 20B monolin- gual tokens on both LLaMA-2-77 and 13B models yields substantial enhancements for non- non-English languages across all tasks. A consistent trend observed was that increased monolingual data cor- responds to greater performance boosts. Only English is observed for a negligible difference after fine-tuning monolingual data, which is an anticipated outcome given LLaMA-22's proficient grasp (Goo et al.
Conneau et al. (2018),
of English. The tool we utilize for LLM evaluation is Im-evaluation harnesaanaessse
2021 1 10
Models
XNLI
Xstarycloze
XWinograd
de
ez:
zu
zh
Avg.
en
z::
Avg.
en
ru
zh
Avg.
XLMR-Lage
32.29
27.51
31.20
33.41
31.10
49.83
48.25
46.46
48.18
47.31
48.89
44.05
46.75
XGLM-7_5B
48.31
$4.06
46.55
34.66
45.90
69.82
63.34
58.90
64.02
79.35
63.17
72.82
71.78
BLOOM-7B
39.04
53.37
42.61
35.50
42.63
70.48
52.68
61.88
61.68
82 62.06
56.83
74.21
71.03
MPT-7B
47.87
46.10
36.71
78.09
57.71
59.50
65.10
86.58
73.21
76.23
LLaMA
45.90
56.51
41.35
34.82
44.64
77.04
63.07
59.56
66.56
87.91
68.89
70.63
75.81
LLaMA-2-7B, mono.
47.63
57.87
34.70
75.84
64.59
71.83
74.81
LLaMA-2-7B, 20B mono.
50.48
57.35
46.47
33.82
47.03
75.71
67.57
62.81
68.70
86.06
66.67
75.20
75.98
LLaMA-2-13B
49.08
66.18
69.31
77.9%
78.41
LLaMA-2-13B. 1B mono.
47.27
49.56
44.14
38.35
44.83
78.09
68.63
62.01
69.58
88.47
69.21
78.17
78.62
LLaMA-2-
12B
49.56
54.02
47.55
40.72
47.96
78.29
69.82
63.67
70.59
70.16
78.17
79.05
Table 12: We evaluate the zero-shot cross-lingual efficacy on three multilingual datasets. Our find- ings indicate that fine-tuning LLaMA-2 with more monolingual data results in enhanced perfor- mance for non-English languages.
Duu to ICL's extended prompt length, employing a large heam size is impractical; hence, we opt for a beam size of for all to ensure a fair comparison.
21


Reviews:
[{"review_id": "Ls28hhc4xO", "reviewer": "Reviewer_TXCQ", "summary": "The paper is motivated by that the translation performance of LLMs are not as good as other tasks compared to the task-specific methods. To improve the translation capabilities of the moderate LLMs, it proposes a fine-tuning paradigm which is firstly fine-tuning on monolingual data followed by subsequent fine-tuning on a small set of high-quality parallel data. It turns out a huge gain in translation quality compared to the zero-short performance across 10 translation directions from the WMT21 and WMT22.", "questions": "How's your fine-tuned models compared with the dedicated encoder-decoder based translation models in similar size? Please discuss in both high and low resource settings.\n\nIf fine-tuning with large amount of parallel data is not optimal, how about using the during pre-training phase? If targeting at the highest translation performance, what's your take? Please include the dedicated translation model in the discussion.\n\nWhat would be the results of evaluating on some out-of-domain test sets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is clearly written and provides many insights. Using LLM to boost the translation quality is an interesting and important topic. It proposes a novel fine-tuning paradigm to let the moderate size LLMs better at translation. Many analyses should be very helpful to the NLP and ML community.", "weaknesses": "The paper is mainly focusing on improve the translation quality of LLMs. It'd be better to compare more with the encoder-decoder translation models and shed light on the best practice of translation itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XozJtG57x1", "reviewer": "Reviewer_vUYo", "summary": "This paper focuses on improving the translation capability of large language models (LLMs). The authors find that LLMs do not require a large amount of parallel data as traditional models do to achieve decent translation quality. Accordingly, they propose a new training recipe including two stages: firstly finetune LLM on monolingual data and then finetune the model on a small set of high-quality parallel data. Experiments with LLAMA-2 show promising performance even on par with GPT-3.5.", "questions": "Firstly, the authors claim the finding/proposal is a \"paradigm shift\" as highlighted in the title, which might be inadequate. The recipe generally follows the pretraining-finetuning paradigm which has already been well-established since BERT and mBart/T5. In addition, similar solutions have already been used in prior studies, such as BigTranslate.\n\nSecondly, the few-shot results are largely undervalued. As shown in Table 11, the quality gap between the HW 5-shot and the proposal is only 0.2 COMET on XX->En, although the finetuning used a large amount of monolingual corpus and more parallel data. Few-shot performance should be able to be further enhanced via beam search and optimized prompt construction, and it should be used as the fair baseline rather than 0-shot prompting. It's also misleading to state in Appendix I that \"ICL substantially underperforms our stage-2 finetuning\". \n\nLastly, finetuning performance is highly dependent on pretraining conditions and the downstream task. Intuitively, when the downstream tasks highly correlate with the pretraining, the demand for a large volume of the supervised corpus is reduced as highlighted in this paper. However, as the MPT-7B performance indicates in Figure 4, when the correlation is low, adding more supervised data is almost always helpful. In other words, the findings in this paper might not be generalization. Do you also have the finetuning results for MPT-7B? Does the recipe also apply to low-resource languages like Gu?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Propose a simple training recipe for LLMs for translation tasks: finetuning first on monolingual data and then on small high-quality parallel data.\n2) Demonstrate impressive performance across 5 language pairs with LLAMA-2.", "weaknesses": "1) The statement of \"paradigm shift\" is somehow overestimated.\n2) The few-shot prompting results are highly undervalued.\n3) The proposed recipe might not apply to other LLMs and languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RzDrUF628F", "reviewer": "Reviewer_AH62", "summary": "This paper examines ways of improving the translation performance of large language models. There is no modelling or training innovation in the paper, but they are the first to show that you can take a smaller English focussed language model (LLaMa 7B,13B) and make it into a translation model with equivalent performance to a large LLM (GPT 3.5 &4) using relatively small amounts of monolingual (20B?) and parallel data (1B) in a fine-tuning step. The paper is generally clearly written but some important details are missing, some claims are not entirely supported, and there are some typos. \n\nIn more detail:\nThey first show that large models (GPT3.5) do extremely well translating into and out of  5 diverse languages and English, mostly high resource except for Icelandic. They then select the best performing smaller model (7B parameters) and experiment on fine-tuning to the translation task using instructions. They show that LLama27B quickly maxes out after 10k examples, whereas MPT-7B continues improving until the final datapoint at 20M. \nThey then experiment with using monolingual data in pretraining, and finetuning with instructions, and call the resulting model ALMA. They show that with a relatively small multilingual (but not parallel) pretraining dataset 1B, they get significant gains in translation quality. Also they used high quality parallel data for finetuning and experimented with full fine-tuning and LORA fine-tuning.", "questions": "They claim/state things which are either not entirely correct or are overstated: \n\"Both monolingual data fine-tuning and human-written data\" - I think they mean parallel data here - both monolingual and translated data are human-written. \nThis claim is not correct: \"We demonstrate that LLMs, such as LLaMA-2-7B, do not voraciously consume parallel data. \" What they demonstrate is that for fine-tuning LlaMa-2-7B does not improve much beyond 10k examples. However, their other model MPT-7B does keep improving and does not max out even at the 20M mark.\nThe citation for this claim in the intro: \"they still fall short in translation for low-resource languages\" Zhang et al. is wrong. They do not look at low resource languages, only experiments with English, Chinese and German. \nThey conclude: \"From our observations, LLMs should not adopt the same training approach as earlier models—whether randomly initialized or pre-trained—that rely heavily on vast amounts of training data\" but do not specify that this is just for the fine-tuning LLaMa - it is a too broad claim to make.\n\nSome things are not explained or described:\nThey do not explain why they selected MPT for experiments in 3.2, and more importantly they do not discuss why it performs contrary to their claimed results - that LLMs to not voraciously consume parallel data. \nAlso for Section 5 they do not say how much parallel data is used for fine-tuning and what ratios of parallel data are we using - same as the monolingual data? This really should be detailed in the main paper. \nThis caption is confusing \"Figure 5: The average performance of ALMA-7B at the completion of each 1B-token fine-tuning\". Is this without the instruction fine-tuning? How do these numbers compare to Tables 1 and 2? I can't figure this out due to the differences in the data - but it seems like the instruction fine-tuning makes little/no difference here. (Fig5) 85.28 COMET vs. (84.12 + 86.27) / 2 (Table 1 and 2) for the ALMA7B?\n\nThere are a number of typos and inconsistencies that need to be polished for final submission: \nThe ALMA models are called AMLA,\nTypo:  \"As expected, it tends up with a similar performance \". \nThe graphics are not very consistent in the paper and don't look clean or very legible. Figure 5 is particularly  hard to read.  \nResults are not very structured. For some (2.2) use NLLB and other not (3.1) .\nMany different result formats vertical/horizontal bar, line, table - it is harder to read and looks messy.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "They show large improvements in the translation capabilities of the most useful size of models (7B,13B) with very affordable limited fine-tuning and data. \nThis is a useful paper for people working in machine translation to see what works in fine-tuning large language models for the translation task.", "weaknesses": "There is not a lot of novelty in the approach - either in training or modelling. I am not sure that the \"New paradigm\" title is justified. \nI have not learned much from reading the paper - it is still not clear what the contribution of the monolingual vs parallel training data is. It is also not clear whether the good performance of the trained models is due to the reduced number of non English languages (5) vs other models (NLLB, GPT3.5,4). \nI am also not sure these results (improvement over LLaMa7B with fine-tuning) would hold if you used few-shot - and it would have been a very easy experiment to conduct. \nThe paper writing is not particularly clear (see questions for details).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VezMUEs1o4", "reviewer": "Reviewer_TMW7", "summary": "This paper proposed an innovative two-stage fine-tuning method: initially fine-tuning on non-English monolingual data to enhance comprehension, followed by further fine-tuning on a small amount of high-quality human-translated parallel text. This approach enabled even smaller LLMs to achieve state-of-the-art translation performance.", "questions": "1. Typically, if an LLM was fine-tuned with specific bilingual corpora to enhance translation capabilities between those two languages, it might impair other NLP capabilities of the LLM, such as document summarization and logical question answering. Did this issue not arise in this study?\n\n2. If possible, please consider using the 'Instruct Score' from EMNLP 2023 (Instructscore: Towards Explainable Text Generation Evaluation with Automatic Feedback) as a metric. I believe it's a better benchmark for evaluating LLM-MT.\n\n3. The outputs of large models were uncertain. Even a minor change in a prompt could lead to variations in the output. During the two-stage fine-tuning process, was the specific impact of the prompt considered?\n\n3. Given the powerful In-Context Learning capabilities of large language models, it would be worth exploring whether adding relevant knowledge to the prompt could further enhance translation capabilities.\n\n4. The section \"Small Training Data Is Enough\" contained many uncertain descriptions, which should be rigorous in a paper and supported by convincing data. Moreover, the best-performing commercial LLM, GPT-4, remained proprietary, leaving us in the dark about the amount of data used, the number of parameters trained, and potential data leakage issues during translation metric testing.\n\n5. The sentence in the \"Large Parallel Data Wash Out the Knowledge\" section: \"As expected, it tends up with a similar performance in both BLEU and COMET evaluations (triangle in Figure 4),\" was hard to understand.\n\n6. The proposed method significantly improved translation metric scores, highlighting ALMA's effectiveness. Based on this, how did the authors believe LLM generalized translation capabilities? Did the proposed method fundamentally assist LLMs in learning deep bilingual text alignment?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results of the paper indicated that smaller models could achieve SOTA translation levels through specialized fine-tuning, suggesting that there might not be a continuous need to expand datasets and models for better performance.\n\nThrough compact, specialized fine-tuning, smaller LLMs could achieve SOTA translation quality without billions of parameters. The focus of this research was on tailored fine-tuning methods that unleashed the potential of LLM's multilingual capabilities on a broader scale.\n\nThe paper demonstrated that instead of increasing data scale, intentional fine-tuning targeting key language capabilities might be the key to maximizing LLM performance.\n\nBy revealing the potential of smaller LLMs for efficient and accurate machine translation, this work laid the foundation for developing more user-friendly and scalable machine translation systems. This training approach offered more possibilities for deploying capable multilingual LLMs in real-world applications.", "weaknesses": "There were certain flaws in the method, and prompts affected the results. \n\nThe evaluation methods had its limitations. \n\nThe stability of the proposed method was not verified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)