Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Authors: Duarte Miguel Alves
Nuno M Guerreiro
João Alves
José Pombal
Ricardo Rei
José G. C. de Souza
Pierre Colombo
Andre Martins
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Translation; In-Context Learning; Efficient Finetuning
TL;DR: We explore finetuning and in-context learning for adapting large language models for machine translation.

Abstract: Large language models (LLMs) are a promising avenue for  machine translation (MT). However, current LLM-based MT systems are brittle: their effectiveness highly depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. Alternatives such as finetuning on translation instructions are computationally expensive and may weaken in-context learning capabilities, due to overspecialization. In this paper, we provide a closer look at this problem.  We start by showing that adapter-based finetuning with LoRA matches the performance of traditional finetuning while reducing the number of training parameters by a factor of 50. This method also outperforms few-shot prompting and eliminates the need for post-processing or in-context examples. However, we show that finetuning generally degrades few-shot performance, hindering adaptation capabilities. Finally, to obtain the best of both worlds, we propose a simple approach that incorporates few-shot examples during finetuning. Experiments on 10 language pairs show that our proposed approach recovers the original few-shot capabilities while keeping the added benefits of finetuning.

Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning
Duarte M. Alves! Nuno M. Guerreiro 1,244,5 João Alves?
José Pombal
Ricardo Refa3, José G. C. de Souza² Pierre Colombo André F. T. Martins 1.rtinsl.2.4 1.2,4 Instituto de Telccomunicações, Lisbon, Portugal Unbabel, Lisbon, Portugal, INESC-ID, Lisbon, Portugal "Instituto Superior Técnico, University of Lisbon, Portugal MIICS, CentraleSupélec, Université Paris-Saclayy France Equall, Paris, France duartemalves@tteniic ulisboa.pp
Abstract
are greatly increased by processing all input pairs. When parallel data available, L.LSs can alterna- tively be finetuned on translation instructions (Li et al., 2023). This method generally outperforms few-shot prompting and eliminates the need for in-context examples. However, it remains unclear whether finetuned models can benefit from the de- sirable properties of in-context learning, such on-the-fly domain adaptation (Agrawal et al., 2022). Additionally, traditional finetuning (Devlin et al., 2019; Radford et al., 2018) incurs a high computa- tional overhead due to the cost of updating all the model weights.
Large language models (LLMs) are promis- ing avenue for machine translation (MT). How- ever, current L.s--ased MT systems are brit- tle: their effectiveness highly depends on the choice of few-shot examples and they often re- quire extra proseprocsssing due overgenera- tion. Alternatives such as finetuning transla- tion instructions are computationally expensive and may weaken in-context learning capabil- ities, due to overspecialization. In this paper, we provide closer look at this problem. We start by showing that celpter--bseed finetuning with LoRA matches the performance of tradi- tional finetuning while reducing the number of training parameters by a factor of 50. This method also outperforms few-shot prompting and eliminates the need for post-processing or in-context examples. However, we show that finetuning generally degrades few-shot perfor- mance, hindering adaptation capabilities. Fi- nally, to obtain the best of both worlds, we propose a simple approach that incorporates few-shot examples during finetuning. Experi- ments on 10 language pairs show that our pro- posed approach recovers the original few-shot capabilities while keeping the added benefits finetuning.
In this paper, we provide closer examination of the impact of finetuning and few-shot prompt- ing for adapting LLMs to perform translation. Our experiments encompass 10 language pairs on gen- eral and specific domains, comprising over 100,000 generated translations ($2). Our main findings are:
We show that finetuning with adapters (Houlsby al., 2019; Hu et al., 2022) is very effec- tive method to steer LLMs for translation ($3.1) This method matches the performance of tradi- tional finetuning
fraction of the computa- tional cost, by training 50 times fewer parame- ters. It also achieves better translation quality than in-context learning and eliminates the need for post-processing the generated outputs and selecting in-context examples.
1 Introduction
Large language models (LLMs) have shown re- markable performance wide range of NLP tasks by leveraging in-context learning (Brown al., 2020). particular, when provided with few- shot examples, these models have demonstrated im- pressive capabilities for performing machine trans- lation (MT) without requiring explicit supervision on parallel data (Garcia et al., 2023). However, this approach exhibits several drawbacks: performance is highly dependent on the quality of examples (Vi- lar al., 2022), outputs are plagued by overgenera- tion (Bawden and Yvon, 2023), and inference costs
We show that finetuning large language models degrades their few-shot performance, limiting their adaptation capabilities ($3.2). In particular, we show that finetuned LLMs perform poorly on domain adaptation scenarios when provided in-context examples.
To address this issue, we propose a simple ap- proach that introduces few-shot examples during finetuning ($4). Our results show that we can recover few-shot capabilities while retaining the benefits of finetuning.
'Code avaliahle at https://githhi com see/dddep-pppne translation_llm.

---

2 Experimental Setup
90
88
our experiments, we use LLaMA 7B and 13B (Touvron et al.. 2023) as backbone language models and finetune them with the standard cross entropy loss.
COMET
84
82
We train our models on general domain OPUS (Tiedemann, 2012) data from the Eu- roparl, Globalvoices, Paracrawl, Tilde, Ubuntu, and Wikipedia domains. We consider the lan- guages Dutch (nl), French (fr), German (dc), Por- tuguese (pt) and Russian (ru), both from and into English (en). To ensure the quality of the training records, first apply Bicleaner (Ramirez-Sánchez et 2020) using threshold of 0.85 and then filter the remaining pairs, ensuring both language direc- tions have a COMETKiwi (Rei et al., 2022b) score above 0.8. Finally, we sample 250K records for each language pair. During training, we uniformly sample from the data to ensure each language pair is seen similar number of times. We perform val-
80
do-en
on-de
fr-ez
o2-fr
ni-en
en-ni
pt-ea
en-pt
ni-en
ea-ru
Language Pair
Pretrained
Finetuned
LoRA
Figure 1: COMET scores on the Flores-200 set by LLaMA 7B pretrained (few-shot) and LLaMA 7B trained with full finetuning and LoRA (zero-shot).
90
COMET
idation on the Flores-200 development set for the language pairs in the training data.
75-
For in-domain evaluation, we consider the Flores-200 (NLLB Team et al., 2022) test dataset on all the translation directions included during training, as well as the WMT22 test sets' for the language pairs considered in our training data. Re- consider garding data for specialized domains, the Medical and Law domains from Aharoni and Goldberg (2020), the TICO dataset (Anastasopou- los al., 2020) and WMT Chat (Farinha et al., 2022). We evaluate our models on zero and five shot settings, uniformly sampling for each test sen- tence five independent few-shot samples from the respective development set.
70
0
8
120
160
Training Examples (in thousands)
Pretrained En-XX
Finetuned En-XX
Protrained XX-En
Finelaned XX-En
Figure 2: COMET scores for zero-shot evaluation on the Flores-200 test by LLaMA 7B finetuned with differing amounts of training data.
Finetuning LLMs on MT instructions
In this section, we investigate the performance of LLMs finetuned on machine translation instruc- tions in relation to few-shot prompting with pre- trained language model.
Our main evaluation metric is COMET (Rei al., 2020, 2022a)4. We also report results with BLEU (Papineni et al., 2002), chrl' (Popović, 2015) and COMETKiwi (Rei et al., 2022b) in Ap- pendix G.
Note that, throughout this section, always analyse few-shot prompting for the pretrained model. We deem that this offers fairer compari- son to finetuning on translation instructions, since both methods have access to training examples. Nevertheless, we also provide the results for zero-shot translation with the pretrained model in Appendix G. Similar to the findings in Bawden and Yvon (2023), zero-shot performance is far be- hind few-shot performance, in particular for out- of-English language pairs, likely due to the preva- lence of English data during the pretraining of the LLaMA models.
We refer the reader to Appendix A for full details hyperparameters and instruction formats used in the following experiments.
"e also consider Chinese (zh). However, it is supported by LLaMA, examine in Appendix ttttps: /tw.ttttttttttttt..t////t.t centtttt.rg//mmt org/wmt222 translate attiinin on- task.html
WWe use the latest COMET model 2mt coneccommetae from version 1.

---

General Domain
Specialized Domains
Flores
WMT 22
Medical
Law
Tico
Chat
84
713
78
86
COMET
80
78
w/o
w/o (ew-shot few-shot
w/ few-sbot few-shot
FT w/ few-shot fow-shot
FT w/o few-shot few-shot
w/
w/o PT w/
few-shot fow-shot
few-shot few-shot
Zero-shot
Five-shot
Figure 3: COMET scores for zero-shot and five-shot translation by models finetuning with and without few-shot examples, Scores are averaged across all language pairs. "FT w/o few-shot" refers finetuning with translation instructions, as in Section 3. "FT w/ few-shot" refers to finetuning with few-shot examples, detailed in Section 4.
3.1 Efficient finetuning with LoRA
3.2 Few-shot prompting of finetuned models
We now direct our attention to comparing zero- and five-shot performance. We argue that, even when L.M can achieve high zero-shot trans- lation quality, few-shot capabilities can be very beneficial for efficient adaptation. As shown by Agrawal et al. (2022), LLMs can leverage very small pool of few-shot examples to perform trans- lation on new domains.
We start by studying parameter efficient training with low-rank adaptation (LoRA) (Hu et al., 2022) and compare it with traditional finetuning.
In Figure 1, observe that LoRA performs comparably traditional finetuning while train- ing 50 times fewer parameters." We also see that both LoRA and traditional finetuning outperform the pretrained model with few-shot prompts- the latter consistent with the findings in Li et al. (2023), which show that finetuning leads to better translations than few-shot prompting of pretrained language models. As a general trend, all methods exhibit better translation quality when translating into English, following recent trends in the litera- ture (Arivazhagan et al., 2019; Vilar et al., 2022).
In the leftmost plots of Figure 3, we examine the zero- and few-shot performance of our fine- tuned models on general domains. Few-shot per- formance degrades and is surpassed by zero-shot performance, suggesting that the finetuning proce- dure is hindering the in-context learning abilities."
In order to further study this phenomenon, we evaluate the above models on specialized domains. General domain examples may be of little help for model already trained on that domain. On the contrary, in specialized domains, examples should information about the prop- erties of the translation, such style, register, and thus help the model achieve better performance.
We also find that finetuning LoRA requires very small number of translations obtain the reported performance, as shown in Figure In particular, it outperforms the few-shot pretrained model with as few as 2,000 training examples.
bring
Considering the high computational costs of full finetuning compared to parameter-efficient finetun- ing and the negligible degradation obtained with the LoRA- SoRA-based model, we use LoRA in subse- quent experiments.
In the rightmost plots of Figure 3, we observe that the above issue happens consistently in all domains, with a larger degradation in performance. This finding further supports our hypothesis that finetuning can degrade the performance of few-shot prompting.
'In this section, only considered model due computational constraints. Concurrent our work. Xu (2023) showed LoRA is competitive with finetuning when applied to LLaMA 13B. ALoRA requires only 134M trainable parameters, whereas traditional finetuning requires 6,7B.
'Regarding the 13B model, the trends are more visible when evaluating with COMETKiwi (see Appendix G) which shown to correlate well with human judgements when eval- uating LLM based MT systems (Hendy et al., 2023).

---

200
40
Translation Length
We also analyze whether finetuned models continue generate context after the desired translation. This issue is present in pretrained LLM outputs and requires post- processonns of the generated content, deleting all words generated after the first new line. In Figure 4, show the length of the tokenized outputs for the 7B models." We observe that the distribution of the length for the outputs generated by both finetuned models matches the distribution of the references. This shows that the finetuned COMET Score A
150
20
100
50
-20
Pretrained FT
FT few-shot few-shot
Reference
de-en
e2-de
fr-en
-f-f
nl-en
en-ni
pt-en
en:pt
ru-en
en-ru
Language Pair
Figure 4: Length of the tokenized outputs when trans- lating the Flores-200 test set for the 7B models.
Figure 5: COMET score difference for zero- vs few- shot translations Flores-200 by the 7B FT w/ few- shot model (A > 0 indicates higher score for few-shot translations).
4 Finetuning with few-shot examples
In order to recover few-shot performance, in- troduce instructions with few-shot examples in the training process: namely, we finetune on data which contains both zero-shot and few-shot instruc- tions. Following Min et al. (2022), we uniformly sample between and few-shot examples for each training example from an example pool previ- ously separated from the training data. 8 From here, we build instruction prompt with the training example and the selected examples and proceed with the training.
models no longer overgenerate.
We also found that these models no longer de- limit their output with the newline symbol and in- stead produce the end of sentence token, removing the necessity for post-processing and increasing computational efficiency. In Appendix F, we pro- vide set of examples to illustrate these findings.
4.2 Influence of in-context examples
In order obtain more fine-grained analysis of the gains obtained by adding in in-context examples, analyzed the difference in COMET scores for each source sentence when prompting the 7B fine- tuned models with and without examples.
In Figure 3, we observe that the models trained with in-context examples recover their few-shot capabilities, both for the general and specialized domains. The few-shot performance is on par or above the zero-shot performance, further suggest- ing that the models are extracting helpful informa- tion from the examples. In Appendix D, present a of examples that highlight these gains.
In Figure 5, we observe that the distributions have high concentration of points slightly above 0. However, also observe very large tails, in particular for out-of-English language pairs.
4.1 Analysis on output format
We manually inspected the examples with the highest differences and found that introducing examples can fix the model generating in the wrong language, supporting the findings in Bawden and Yvon (2023). Surprisingly, examples where the model correctly generated translation in zero-shot scenario and inserting in-context examples lead to hallucinated content.
also discovered
To better characterize this phenomenon, we take inspiration from analysis on hallucinations under perturbation (Lee et al., 2018), and measured how many times prompting the model without examples lead translation above 30 BLEU points, and introducing examples reduced the score below
"We also considered a training mixture where 50% the contained examples and remaining data had between 5 uniformly sampled examples. We did not further explore this as preliminary results (see Appendix C) show results are similar to ones obtained with the procedure above.
III Appendix we show model finetuned without examples also same behavior.
"The 13B models follow a similar distribution.
We show several extracted examples in Appendix E.

---

7B
13B
Domain
FT w/o few-shot
FT w/ few-shot
FT w/o few-shot
FT w/ few-shot
Flores 0.12%
0.00%
0.02%
0.00%
Medical
1.09%
0.05%
0.29%
0.00%
Law 2.70%
0.05%
0.48%
0.15%
Tico
0.60%
0.04%
0.04%
0.04%
Chat 1.80%
0.23%
0.51%
0.00%
Table I: Hallucination Rates for finetuned models each evaluation dataset, considering all languages pairs.
3 (these thresholds were selected based on previ- ous work (Lee et al., 2018; Raunak et al., 2021; Guerreiro et al., 2
in-context examples. To address this issue, we propose a simple approach that mixes few-shot prompts during finetuning. Our results show that we recover the original few-shot capabilities and retain the benefits of finetuning.
In Table 1, out examples have higher hallucination rates than their respective counterparts, further showing their degradation few-shot performance. Through a manual inspection of the obtained outputs, we observed that the models generate hallucinations of different categories. In particular, they gener- ate both detached (fully and strongly) and oscilla- tory hallucinations, and can also generate off- target translations. One common case is that the models copy from the instruction (either from the source the examples).
that the models finetuned with-
Limitations
In this paper, we focus on English-centric high- resource language pairs. remains an open ques- tion how these findings generalize for non-English language pairs or in low-resource settings.
We also do not perform human assessment the quality of the translations quality due the time and of performing this study, Instead, we base our evaluation on COMET, state-of-the-ard metric for MT evaluation, and provide results for other metrics in Appendix G.
The models finetuned with few-shot examples exhibit lower hallucination rates, suggesting that the training procedure reduced the prevalence of this issue. In particular, these models no longer copy from the instruction. However, they still pro- duce hallucinations and their impact very serious. As such, we believe that it motivates further study on the influence of in-context examples and the generated output.
Ethics Statement
This paper is based large language models. These models can encompass several risks, which are discussed in detail Brown et (2020) and Chowdhery et al. (2022). Namely, they are trained on large web corpora, which can contain toxic con- tent (Gehman et al., 2020), and have a high energy consumption, in particular during training (Strubell et al., 2019).
5 Conclusion
this paper, we provide study on finetuning and few-shot prompting for adapting LLMs for translation. We show that clapter-based finetuning matches the performance of traditional finetuning while training 50 times fewer parameters. Addition- ally, finetuning with adapters outperforms few-shot prompting of large language models and eliminates the need for output post- processog and in-context examples.
Additionally, evaluation is based auto- matic metrics finetuned based on human prefer- ences. In such cases, annotators may not con- sider better alternatives when evaluating generated text and wrongfully classify the text as high qual- ity (Bansal et al., 2021).
Acknowledgements
In addition, we show that finetuned models exhibit poor performance when prompted with
This work supported by EU's Horizon Eu- rope Research and Innovation Actions (UTTER. contract 101070631), by the project DECOL- LAGE (ERC-2022-CoG 101088763), by Fun- dação para a Ciência e Tecnologia through con-
12 Note that this analysis is similar to that of hallucinations under perturhation, when considering the introduction of the examples us the input perturbation.

---

tract UIDB/500088/0020, and by the Portuguese Recovery and Resilience Plan through project C645008882- 00000055 (Center for Responsi- ble AI). Part of this work was performed using HPC resources from GENCI-IDRIS (Grants 2022- AD01101838, 2023-10355 and 2023-101838).
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mad- eling with pathways.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In- context examples selection for machine translation.
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Asso- ciution for Computational Linguistics, pages 7747- 7763, Online. Association for Computational Lin- guistics.
Antonios Anastasopoulos, Alessandro Cattelan, Zi- Yi Dou, Marcello Federico, Christian Federmann, Dmiirry Genzel, Francisco Guzmán, Junjie Hu, Mac- duff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Ök- tem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. TICO-19: the Translation initiative for COvid-19. LiXiv:2007001788
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. CoRR, abs/1907.05005
Ana C Farinha, M. Amin Farajian, Marianna Buchic- chio, Patrick Fernandes, José G. C. de Souza, He- lena Moniz, and André F. T. Martins. 2022. Find- ings of the WMT 2022 shared task on chat transla- tion. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 724-743, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Ray- mond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2027 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Computing Machinery.
Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng. Melvin Johnson, and Orhan Firat, 2023. The unreasonable effectiveness of few-shot learning for machine trans- lation.
Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: the case of bloom.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Asvociation for Computational Linguistics: EMNLP 2020, pages 2356-3369, Online. Association for Computational Linguisties.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances neural information processing systems, volume 33, page 1877-1901. Curran Associates, Inc.
Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan

---

Awadalla. 2023. How good are gpt models at ma- chine translation? a comprehensive evaluation.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, ceemeettr-efficeett transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-279999 PMLR.
Gema Samfrez-Sánchern Jaume Zaragoza- Bernbaeuu Marta Bañón, and Sergio Ortiz Rojas. 2020. Bifixer and bicleaner: two open-source tools to clean your parallel data. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 291-298, Lisboa, Portugal. Euro- pean Association for Machine Translation.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Vikas Raunak, Arul Menezes, and Marein Junczys- Dowmunt, 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1188 113, Online, Association for Computational Linguistics.
Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fan- njiang, and David Sussillo. 2018. Hallucinations in neural machine translation.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metries shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Chen, and Jiajun Chen. 2023. Eliciting the translation abil- ity of large language models via multilingual finetun- ing with translation instructions.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online, Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedamuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safyyaah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 3645-3650, Florence, Italy. Asso- ciation for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation (LREC"12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Maja Popović. 2015, chrF: character n- n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard

---

Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance.
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Has- Awadalla. 2023. A paradigm shift in machine translation: Boosting translation performance of large language models.

---

Translate the source text from X to Y.
85
Source:
80
Target:
COMET
Table 2: Prompting template for finetuning without few- shot examples.
70
65
A Details on experimental setup
60
zh-en
en-2h
Language Pair
A.1 Instruction format
Pretrained
Finetuned
The training data for finetuning without few-shot examples follows the template shown in Table 2. The same format is used when testing all models in zero-shot setting.
Figure 6: COMET scores the Chinese language pairs of the Flores-200 test set by LLaMA 7B trained with full finetuning and LoRA.
We treat the few-shot instruction template as a hyperparameter and experiment with three dif- ferent methods, as shown in Table 3. Our first template follows recent trends in the literature and repeats the zero-shot instruction for each example (Vilar et al., 2022). However, in our experiments, found that pretrained language models see the repeating pattern and continue generate more examples besides the target translation. In order to circumvent this issue in the finetuned models, we designed the two remaining templates with sep- arate examples sections. Our goal was to better separate the examples from the input and thus re- duce the propensity for overgeneration. We found that all templates lead to overgeneration with the pretrained model and none suffered from this issue when the model is finetuned.
parameter search by finetuning the LLaMA 7B model with each configuration on the training data. We only consider zero-shot translation and use the template format in Table 2. We find the best con- figuration based on the average COMET score on all language pairs in the validation set.
Table specifies the hyperparameters experi- mented when training LLaMA 7B with traditional finetuning.
first chose the learning rate and weight decay, while not using warm-up steps. We then tuned the scheduler and warm-up steps. Our final configuration has learning rate of le-6, no weight decay, and constant learning scheduler with no warm-up steps.
Table 6 details the hyperparameters experi- mented when finetuning with LoRA. We based our experiments on the best configurations for the GPT- 2 models trained in Hu et al. (2022). Initial experi- ments with lower values lead to underfitting model so our configurations focused on increasing model capacity, with higher values, while keeping regularization through label smoothing and weight decay. In Table 7, we present the results for all the runs. We saw very little variation on the obtained scores. We adopted the best configuration, with an r value of 256, weight decay of 0.0, and label smoothing of 0.001.
In order to select the template format for our remaining experiments, we test them by finetuning with examples the LLaMA 7B model and choos- ing the template with the highest average COMET score the languages in the validation set. In or- der collect examples for few-shot prompting in the validation set, sampled from the validation set ensuring the predicted example was not in the in-context examples.
In Table 4, we observe that the templates lead to very similar results, suggesting that the finetuning procedure is not very sensitive the template used. Nevertheless, their ranking is consistent across met- rics, with the second one obtaining the best scores, As such, we use it when prompting models in few-shot scenario.
Regarding the 13B models, used the same hyperparameters as in the 7B models.
B Analysis on Chinese language pairs
In this section, we explore the results for the lan- guage pairs including Chinese with the LLaMA 7B model, order to study if our previous results hold.
A.2 Training hyperparameters
order to choose the best hyperparameters for both finetuning approaches, we perform hyper-

---

Format
Format
Format
Translate the text from X to Y.
Consider the following N translations
Consider the following translations from X to Y.
Source:
from X to Y.
Target:
Example
Source:
Source:
Target: ...
Translate the text from to Y.
Target:
Source:
Source:
Target:
Example
Target: ...
Translate the text from Y.
Source:
Source:...
Target: ...
Translate the source text from
Target:
Translate the source text from
Source: Target:
Source:
Target:
Table 3: Prompting templates for finetuning with in-context examples.
Format
COMET
COMETKiwi
BLEU
chrF
Format
85.25
82.49
32.44
57.25
Format 2
85.34
82.54
32.62
57.37
Format 3
85.27
82.51
32.39
57.22
Table 4: Scores for the few-shot formats on the Flores-200 validation set.
0.78
indicating it is extracting helpful information from the examples in the prompt.
0.77
C Experiments with more zero-shot data
COMET
0.76
We explored alternative method for combining few-shot examples during finetuming. Instead of uniformly sampling between and examples, we build a training mixture where 50% of the training examples were zero-shot and the remaining ones had between and 5 uniformly sampled examples.
0.75
0.74
FT w/o few-shot
FT w/ few-shot
Zero-shot
Five-shat
In Figure 8, we compare the training mixes by finetuning LLaMA 7B. We see that the results are very similar for both configurations. The alterna- tive configuration (Unbalanced) obtains slightly lower results. As such, adopted the method de- scribed in Section for mixing few-shot examples during finetuning.
Figure 7: COMET scores for Chinese language pairs by the 7B finetuned models on zero-shot and five-shot scenarios for the Flores-200 test set.
We start by investigating whether LoRA still competitive with full finetuning. In Figure 6, we observe that LoRA performs comparably to the finetuned model and outperforms the pretrained LLM, following the trend of other language pairs (see Section 3.1).
D Examples of domain adaptation
In this section, we provide examples of translations where the LLaMA 7B model trained with few-shot example was able to absorb domain knowledge from the examples in the prompt.
We also investigate the performance of the mod- cls finctuned with and without examples. In Fig- 7, observe similar trend the results above. The model finetuned without few-shot ex- amples exhibits performance degradation, while the model finetuned with few-shot examples ob- tains higher performance with fhw-shot prompting,
In the first example from Table 8, model correctly translates the terminology GVO GMOs (Genetically Modified Organisms), instead of adopting the acronym in the source sentence. the second example, the model able to correctly order the words the translation.
that the

---

Optimizer
Adam
Optimizer Batch Size
AdamW 256 Se-3, 2e-4, 5e-4, 2e-5, 5e-5, le-6
Batch Size
Learning Rate
2e-4
Learning Rate
Scheduler
Linear
Warm-up Steps Dropout
500
Scheduler
Constant, Cosine, Lincar 0, 1000, 2000 0.1
0.05
Warm-up Steps
128, 256
Weight Decay
O Label Smoothing
2 2-r
0.01, 0.05, 0.1, 0.2 0.0, 0.1
Table 5: Hyperparameters for traditional finetuning experiments.
Weight Decay
Table 6: Hyperparameters for LoRA experiments.
LoRA-R
Weight Decay
Label Smoothing
COMET
COMETKiwi
BLEU
chrF
128
0.0
0.0
84.74
81.96
31.47
56.36
128
0.1
0.0
84.76
81.98
31.45
56.34
128
0.0
0.01
84.79
81.99
31.48
56.37
128
0.0
0.05
84.80
82.00
31.28
56.30
128
0.0
0.1
84.61
81.85
31.10
56.15
128
0.0
0.2
84.36
81.61
30.71
55.89
256
0.0
0.0
84.78
82.01
31.47
56.40
256
0.1
0.0
84.72
81.94
31.41
56.32
256
0.0
0,01
84.87
82.08
31.57
56.45
256
0.0
0.05
84.78
81.97
31.47
56.39
256
0.0
0.1
84.65
81.92
31.28
56.25
256
0.0
0.2
84.48
81.77
30.95
56.01
Table 7: Scores for the LoRA hyperparameters on the Flores 200 validation set.
E
Analysis on the distributions of COMET score differences
F Examples of generated outputs
In this section, we present translations where prompting the pretrained LLaMA 7B model leads to overgeneration, and both 7B finetuned models correctly stopped to translate. Table 11, we see that, although all models generated the same trans- lation, the pretrained model continued to generate, repeating the prompt and translation, while both finetuned models correctly stopped to generate to- kens.
We also provide a more in-depth analysis on the distributions of COMET score differences, with focus on the examples with the highest differences.
observe that the distributions for the LLaMA 7B model finetuned without in-context examples also have large tails, similar to the results of the model finetuned with in-context examples (see in Section 4.2).
In Figure
We also analyzed whether the same long tails appear on the specialized domains. Figure 10, we observe that this is in fact the case. The dis- tributions of the differences are centered around zero and have extreme values both sides for all domains and finetuned models.
G Results with all evaluation metries
We provide the evaluation for the models con- sidered in this paper using three other MT eval- uation metries: BLEU (Papineni et al., 2002), chrF (Popović, 2015) and COMETKiwi (Rei al., 2022b).
Finally, we show several examples where few- shot prompting both helped degraded the model performance. In Table 9, prompting the model with few-shot examples fixed the generation in the wrong language. In Table 10, introducing in- context examples the model prompt leads to hallucinated content.
In Figure 11, we show the comparison between both finetuning approaches with the L.LMMA 7B model. The results consistent across all metries, with the LoRA model performing similarly the finetuned and outperforming the pretrained model.
In Figures 12, 13 and 13, we compare finetun-

---

General Domain
Specialized Domains
Flores
Medical
Law
Tico
88
86
COMET
84
82
80
Balanced
Unbalanced
Balanced
Unbalanced
Balanced
Unbalanced
Balanced
Unbalanced
Zero-shot
Five-shot
Figure 8: COMET scores for zero-shot and five-shot translation by finetuning the LLaMA 7B model with the two methods for combining few-shot examples. Balanced is the method described in Section 4 and Unbalanced the alternative method described in Appendix C.
Source
"bezeichnet "genetisch veränderte Futtermittel" Futtermittel. GVO enthalten, daraus bestehen oder hergestellt werden;"
Reference
""genttically modified feed" means feed containing, consisting of produced from GMOs;" ""Genetically modified feed" means feed containing GVO, derived from GVO or produced from GVO,"
Zero-shot translation
Few-shot translation
14 geneticalll modified feed" means feed containing, consisting produced from GMOs,"
Source
"VERORDNUNG (EG) Nr. 538/2000 DER KOMMISSION"
Reference
"COMMISSION REGULATION (EC) $38/2000" "(EG) No 538/2000 OF THE COMMISSION"
Zero-shot translation
Few-shot translation
"COMMISSION REGULATION (EC) No $38/2000"
Table 8: Examples of translations where the LLaMA 7B finetuned with few-shot examples was able extract domain information from the examples in the prompt.
40
when prompting with few-shot examples.
For the lexical metrics, the degradation in few- shot performance is visible on the 13B models. However, these metrics may not be reliable for eval- uating translations from LLMs (Hendy et al., 2023), as LLMs tend to produce less literal translations which are poorly captured by lexical overlap with the reference.
COMET Score
20
-20
-40
do-40
on-de
fr-en
en-fr
nl-en
en1 nl
pt-en
en-pt
ru-en
In Tables 12, 13, 14, 15, 16, 17 and 18 also provide the exact scores for all metrics in tabular format.
Language Pair
Figure Difference in COMET scores for zero- vs few- shot translations by the LLaMA 7B FT w/o few-shot model on Flores-200) (A > means that the transla- tion with sew-shot examples was scored higher than the translation without examples).
ing with and without examples. We observe that the results with COMETKiwi follow the trends obtained with COMET, with performance degra- dation when fhw-shot prompting the model trained with examples and a recovery of the performance

---

FT w/o few-shot
FT w/ few-shot
COMET Score A
40
-60
Flores
Medical
Law
Tico
Chat
Flores
Medical
Law
Tico
Chat
Figure Difference in COMET scores for translations obtained with zero- and few-shot prompting for all domains for the finetuned LLaMA 7B models.
Source
May take your order numher please?
Reference
Darf ich bitte Ihre Bestellnummer haben?
Zero-sho shot translation
May take your order number please?
Few-shot translation
Bitte geben Sie mir Ihren Bestellnummer
Source
May know
you're referring to please
Reference
A qual aula você está se referindo?
7ero-shot translation
May know the class you're 'r referring to please
Fow-shot translation
Poderia saber classe que você está referindo por favor
Source
Thanks, so you are still waiting for the #PRS #PRS_ORG* Wew.ooo--eeeeeeeleeeeeeeeee Pemdelleuchte, Messing und mehrfarbiges Glas?
Reference
Messing und Thanks, so are still waiting for the HPRS_ORG* Pendant lamp. brass and multi-coloured glass?
Vielen Dank, wartest mehrfarhiges Glas?
noch auf die SeSSSORRG-PPeeddd
Zero-shot translation
Few-shot translation
Danke, so warten Sie noch auf die SPRS_ORG#-BUE Pendelleenll,,,httt. Messing und mchrfarhiges Glas?
Table 9: Examples of translations by the 7B FT w/o few-shot model where adding examples corrected the language in which the model was generating.
Source
"Age: Age was the most important factor for the prognosis of SARS, which is also true for 1OV1D-19."
Reference
"Âge : l'âge constituait le principal facteur pour le pronostic SRAS, et cela semble également
être le cas pour COVID-19."
Zero-shot translation
"À Age L'âge le facteur plus important pour la prognostication du SRAS, qui
également vrai pour le COVID-19.
Fow-shot translation
AÁÄAAAA GAAAAААAAAAAAAA
À AAAAAAAЯЯЯ
Source
"Chinese people well other Asians in the United Kingdom and the United States have reported increasing levels of racist abuse, well assaults."
Reference
"O povo chinés, assim como outros asiáticos Reino Unido nos Estados Unidos, relataram níveis crescentes de insultos racistas, bem como de ataques."
Zero-shot translation
"Pessnas chinesas, hem como outros asiáticos no Reino Unido nos Estados Unidos, relataram um aumento abusos racistas, hem como assaltos."
Few-shot translation
"Em 6 de abril, férias foram estendidas todas escolas de nível médio do Turcomenistão."
Table 10: Examples of translations by the 7B FT w/o few-shot model where adding examples introduced an hallucination.

---

Source
""We have 4-month-old mice that are non-diahetic used be diahetic" he added." ""Agora temos ratos de 4 meses de idade que não são diabéticos que antes eram diabéti- cos, "ommlemmntooo.
Reference
Pretrained
""Agora temos ratos 4 meses que não são diabéticos que eram diabéticos", acrescen- tou, iou. Translate source text from English to Portuguese old mice that are non-diabetes that used to be diabetic," he added.AnTarge AnTarget: "Agora temos ratos de 4 meses que não são diabéticos que eram di"
"We now have 4-month-
w/o Examples
""Agora ratos meses que são diahćticos que diahéticos", acrescentou." "'Agora temos ratos de 4 meses que não são diabéticos que eram diabéticos", acrescentou."
FT w/ Examples
Table Examples of translations where finetuning the LLaMA model eliminated the overgeneration in the outputs.
COMETKiwi
BLEU
chrF
85
40
20
70
60
de-en en-lo
fr-on
en-fr
nl-n
on-nl pt-en en-pt ru-en ru zi-en en-2h
en-
do-en en-de
fr-en
en-fft
nlen
en-nl
pt-en
en-pt
ru-on el rt ah
en-28
de-on
en-de
fren
on-fr
nl-en
e2-221
plen
en-pf riren en-ru zh-en en-2h
Language Pair
Language
Pair
Language Pair
Pretrained
Finetuned
LoRA
Figure 1I: Scores of the pretrained model (few-shot prompting) and both 7B finetuned models (zero-shot prompting) the Flores-200 test set.
General Domain
Specialized Domains
Flores
WMT
Medical
Tico
Chat
COMETKiwi
Model Sizee
86
COMETKiwS
84
82
80
FT w/o FT w/ few-shot few-shot
w/o FT few-shot few-shot
FI w/o few-sbot few-shot
FT
FT w/ few-shot few-shot
FT
FI w/o few-shot fewishot
FT w/
FT w/o few-shot few-shot
FT w/
Zero-shot
Five-shot
Figure 12: COMETKiwi scores for zero-shot and five-shot translation by models finetuning with and without few-shot examples. Scores are averaged across all language pairs. "FT w/o few-shot" refers to finetuning with translation instructions, as Section "FT w/ few-shot" refers to finetuning few-shot examples, detailed Section 4.

---

General Domain
Specialized Domains
Flores
WMT
Medical
Law
Tico
Chat
40
BLEU
Madel Size
20
BLEU
13B
30-
FT w/o PT w/ fow-shot fow-shot
FT w/o FT w/ few-sbot few-shot
FI w/o FT w/ few-sbot fow-shot
FT wo FT w/ few-shot fow-shot
FI w/o few-shot fow-sbot
FT
PT w/o PT w/ fow-shot few-shot
Zero-shot
Five-shot
Figure 13: BLEU for zero-shot and five-shot translation by models finetuning with and without few-shot examples. Scores are averaged across all language pairs. "FT w/o few-shot" refers to finetuning with translation instructions, as in Section 3. "FT few-shot" refers to finetuning with few-shot examples, detailed in Section
General Domain
Specialized Domains
Flores
WMT
Medical
Law
Tico
Chat
cheF
Madel Size
cheF
w/o FT w/ few-shot few-shat
w/o few-shot fewishot
FT w/o w/ few-shot few-shoot
FT w/o FT w/ few-shot few-shatt
FT w/o FT w/ few--shot fewishot
FT w/o FT w/ feweishat few-shot
Figure 14: chrF scores for zero zero-shot and five-shot translation by models finetuning with and without few-shot examples. Scores are averaged across language pairs. "FT w/o few-shot" refers finetuning with translation instructions, in Section 3. "FT w/ few-shot" refers to finetuning with few-shot examples, detailed in Section 4.

---

COMET
COMETKiwi
BLEU
chrF
Language Pair
Model
Context
de-en
Pretrained
Zero-Shot
87.23
82.99
36.77
62.34
Five-Shot
88.06
83.49
39.01
64.42
Finetuned
Zero-Shot
88.66
83.95
41.05
66.17
LoRA
Zero-Shot
88.66
83.93
41.22
66.15
en-de
Pretrained
Zero-Shot
71.36
57.87
13.99
38.73
Five-Shot
82.74
79.82
24.77
54.00
Finetuned
Zero-Shot
83.47
81.19
28.30
57.69
LoRA
Zero-Shot
83.63
81.30
28.15
57.62
fr-en
Pretrained
Zero-Shot
87.04
84.61
37.27
62.79
Five-Shot
88.41
85.65
40.71
65.85
Finetuned
Zero-Shot
88.80
86.02
42.99
67.30
LoRA
Zero-Shot
88.77
86.04
42.93
67.39
en-fr
Pretrained
Zero-Shot
76.54
65.17
23.43
47.02
Five-Shot
84.52
84.53
36.71
61.40
Finetuned
Zero-Shot
85.39
85.28
39.23
64.10
LoRA
Zero-Shot
85.36
85.10
39.69
64.43
nl-en
Pretrained
Zero-Shot
84.60
83.59
25.42
53.94
Five-Shot
86.46
84.99
28.55
57.22
Finetuned
Zero-Shot
86.44
85.22
29.43
58.26
LoRA
Zero-Shot
86.40
85.27
29.78
58.30
en-nl
Pretrained
Zero-Sho
73.99
61.17
12.45
38.88
Five-Shot
83.12
81.86
19.26
50.58
Finetuned
Zero-Shot
83.40
82.63
20.71
53.10
LoRA
Zero-Shot
83.10
82.50
20.46
52.79
pt-en
Pretrained
Zero-Shot
87.26
83.21
41.14
65.78
Five-Shot
88.57
84.22
44.35
68.49
Finetuned
Zero-Shot
88.88
84.60
46.58
70.13
LoRA
Zero-Shot
88.91
84.68
46.32
70.16
en-pt
Pretrained
Zero-Shot
74.60
55.91
17.46
40.46
Five-Shot
86.43
82.03
35.25
61.41
Finetuned
Zero-Shot
87.11
82.86
38.75
64.45
LoRA
-hoo-Shot
86.93
82.67
38.94
64.57
ru-en
Pretrained
Zero-Shot
83.85
82.03
28.37
57.00
Five-Shot
85.21
83.11
31.08
59.25
Finetuned
Zero-Shot
85.54
83.49
32.70
60.12
LoRA
Zero-Shot
85,42
83.48
32.28
59.59
Pretrained
Zero-Shot
57.63
46.49
7.16
17.48
Five-Shot
83.44
79.45
19.28
47.60
Finetuned
-hro-Shot
84.91
81.06
20.76
49.32
LoRA
Ecro-Shol
84.73
80.74
20.88
49.53
zh-en
Pretrained
Zero-Shot
76.74
77.46
12.54
34.99
Five-Shot
82.27
79.55
19.23
48.13
Finetuned
Zero-Shot
83.68
80.78
21.73
49.88
LoRA
Zero-Shot
83.15
80.44
21.61
49.71
en-zh
Pretrained
Zero-Shot
49.12
45.10
4.54
7.20
Five-Shot
66.04
61.20
14.68
15.11
Finetuned
Zero-Shot
72.94
68.59
17.07
17.72
LoRA
Zero-Shot
72.43
67.99
16.85
17.74
Table 12: Scores for the 7B pretrained model and 7B both finetuned models the Flores-200 test set.

---

COMET
COMETKiwi
BLEU
chrF
Language Pair
Model
Context
de-en
FT w/o few-shot
Zero-shot
88.66
83.93
41.22
66.15
Five-shot
88.53
83.88
40.41
65.66
FT w/ few-shot
Zero-shot
88.67
84.09
41.50
66.54
Five-shot
88.69
84.12
41.47
66.63
en-de
w/o few-shot
Zero-shot
83.63
81.30
28.15
57.62
Five-shot
83.44
80.98
28.01
57.43
FT w/ few-shot
Zero-shot
83.99
81.51
29.21
58.98
Five-shot
84.06
81.49
29.55
59.06
fr-en
FT w/o few-shot
Zero-shot
88.77
86.04
42.93
67.39
Five-shot
88.79
85.95
43.20
67.32
FT w/ fho-shot
Zero-shot
88.94
86.17
43.34
67.72
Five-shot
88.98
86.17
43.60
67.86
en-fr
FT w/o few-shot
Zero-shot
85.36
85.10
39.69
64.43
Five-shot
85.12
84.82
39.09
63.95
FT w/ sho-shot
zero-shot
85.76
85.49
40.57
65.14
Five-shot
85.98
85.64
40.85
65.25
nl-en
FT w/o few-shot
Zero-shot
86.40
85.27
29.78
58.30
Five-shot
86.44
85.18
29.70
58.09
FT w/ few-shot
Zero-shot
86.67
85.45
30.08
58.74
Five-shot
86.68
85.47
30.06
58.73
en-nl
FT w/o few-shot
Zero-shot
83.10
82.50
20.46
52.79
Five-shot
82.99
82.29
20.24
52.44
w/ sew-shot
Zero-shot
83.45
82.84
20.98
53.50
Five-shot
83.60
83.02
21.09
53.66
pt-en
FT w/o few-shot
Zero-shot
88.91
84.68
46.32
70.16
Five-shot
88.79
84.50
46.37
70.02
FT w/ few-shot
Zero-shot
88.92
84.70
46.95
70.45
Five-shot
88.96
84.70
47.32
70.62
en-pt
FT w/o few-shot
Zero-shot
86.93
82.67
38.94
64.57
Five-shot
86.86
82.46
38.64
64.32
w/ few-shot
Zero-shot
87.36
83.06
40.10
65.36
Five-shot
87.35
83.07
40.11
65.25
ru-en
FT w/o few-shot
Zero-shot
85.42
83.48
32.28
59.59
Five-shot
85.30
83.20
32.08
59.18
FT w/ few-shot
Zero-shot
85.68
83.56
33.12
60.31
Five-shot
85.73
83.62
33.18
60.35
en-ru
FT w/o few-shot
Zero-shot
84.73
80.74
20.88
49.53
Five-shot
84.50
80.18
20.09
48.74
FT w/ few-shot
Zero-shot
85.41
81.40
22.08
50.36
Five-shot
85.57
81.47
22.19
50.73
zh-en
FT w/o few-shot
Zero-shot
83.15
80.44
21.61
49.71
Five-shot
82.57
79.54
20.62
48.23
FT w/ few-shot
Zero-shot
83.71
81.12
22.26
50.70
Five-shot
83.73
80.96
22.74
50.83
en-zh
FT w/o few-shot
Zero-shot
72.43
67.99
16.85
17.74
Five-shot
71.77
66.56
16.11
16.93
FT w/ few-shot
Zero-shot
74.25
70.08
19.05
19.20
Five-shot
74.74
70.69
19.32
19.50
Table 13: Scores of the 7B finetuned models (zero- and few-shot prompting) the Flores-200 test set.

---

COMET
COMETKiwi
BLEU
chrF
Language Pair
Model
Context
de-en
w/o few-shot
Zero-shot
82.86
79.66
29.09
53.99
Five-shot
82.73
79.24
29.27
53.92
FT w/ few-shot
Zero-shot
83.10
79.75
29.43
54.49
Five-shot
83.06
79.73
29.40
54.58
en-de
FT w/o few-shot
Zero-shot
80.02
78.29
23.69
52.80
Five-shot
80.11
78.01
23.11
52.00
FT w/ few-shot
Zero-shot
80.94
78.94
24.00
53.81
Five-shot
81.19
79.05
24.33
54.04
ru-en
w/o few-shot
Zero-shot
82.58
79.08
36.18
61.15
Five-shot
81.90
78.31
35.01
60.35
FT w/ few-shot
Zero-shot
82.97
79.37
37.06
61.93
Five-shot
83.12
79.46
37.30
62.11
en-ru
FT w/o few-shot
Zero-shot
82.04
77.77
20.01
46.18
Five-shot
81.79
77.41
19.41
45.33
FT w/ few-shot
Zero-shot
82.33
78.31
20.74
46.71
Five-shot
82.24
78.14
20.88
46.91
zh-en
FT w/o few-shot
zero-shot
74.90
72.17
15.14
43.29
Five-shot
74.04
71.14
15.52
43.32
FT w/ few-shot
Zero-shot
75.56
72.96
15.91
44.03
Five-shot
75.52
73.09
15.98
44.53
en-zh
FT w/o few-shot
Zero-shot
74.27
69.08
18.19
19.96
Five-shot
73.86
67.77
17.12
18.88
FT w/ few-shot
Zero-shot 76.05
70.97
20.37
21.54
Five-shot 76.25
71.14
19.89
21.43
Table 14: Scores of the 7B finetuned models (zero- and few-shot prompting) on the WMT 2022 test set.

---

COMET
COMETKiwi
BLEU
chri
Domain
Language Pair
Model
Context
Medical
de-en
FT w/o few-shot
Zero-shot
83.54
80.92
39.98
60.32
Five-shot
83.03
80.32
39.29
59.43
FT w/ few-shot
Zero-shot
83.73
81.08
40.62
60.88
Five-shot
83.77
81.15
40.57
60.89
en-de
FT w/o few-shot
Zero-shot
80.15
79.64
30.25
53.10
Five-shot
79.75
78.87
29.86
52.35
FT w/ few-shot
Zero-shot
80.58
79.93
31.61
54.51
Five
80.58
79.87
31.48
54.32
Law
de-en
FT w/o few-shot
Zero-shot
84.11
80.53
38.18
59.38
Five-shot
82.67
78.90
36.90
57.85
FT w/ few-shot
Zero-shot
84.26
80.81
39.84
60.62
Five-shot
84.43
80.96
40.11
60.82
en-de
FT w/o few-shot
Zero-shot
80.83
78.63
26.04
50.44
Five-shot
79.35
76.87
25.03
48.88
FT w/ few-shot
Zero-shot
81.55
79.07
27.80
52.12
Five-shot
81.64
79.11
27.71
52.00
Tico
en-fr
FT w/o few-shot
Zero-shot
78.44
83.99
31.57
56.80
Five-shot
78.10
83.40
32.72
57.11
FT w/ few-shot
Zero-shot
78.60
84.20
32.47
57.59
Five-shot
78.62
84.21
32.58
57.66
en-pt
FT w/o few-shot
Zero-shot
86.87
81.73
39.76
65.92
Five-shot
86.48
81.03
39.31
65.26
FT w/ few-shot
sero-shot
87.03
82.00
40.96
66.81
Five-shot
87.13
82.07
41.11
66.91
Chat
en-de
FT w/o few-shot
Zero-shot
82.56
77.97
27.32
50.81
Five-shot
82.70
77.75
28.53
51.76
FT w/ few-shot
Zero-shot
83.54
78.70
28.19
52.54
Five-shot
84.01
78.76
28.83
53.05
en-fr
FT w/o few-shot
Zero-shot
86.71
80.81
44.70
62.90
Five-shot
86.38
80.12
43.97
61.69
FT w/ few-shot
Zero-shot
86.52
80.61
44.87
63.26
Five-shot
86.62
80.72
44.89
63.20
en-pt
FT w/o few-shot
Zero-hhot
89.54
80.58
41.79
62.93
Five-shot
88.70
79.80
41.06
61.98
FT w/ few-shot
Zero-shot
89.64
80.79
43.52
63.85
Five-shot
89.98
80.76
43.88
64.17
Table 15: Scores of the 7B finetuned models (zero- and few-shot prompting) on the test sets for specialized domains.

---

COMET
COMETKiwi
BLEU
chrF
Language Pair
Model
Context
de-en
FT w/o few-shot
Zero-shot
88.82
84.11
42.03
67.00
Five-shot
88.86
84.05
41.95
66.93
FT w/ few-shot
Zero-shot
88.94
84.13
42.86
67.40
Five-shot
89.03
84.20
42.77
67.53
en-de
w/o few-shot
Zero-shot
84.87
82.38
30.60
60.04
Five-shot
84.64
82.03
30.73
60.01
FT w/ few-shot
Zero-shot
85.01
82.43
31.52
60.49
Five-shot
85.01
82.52
31.51
60.50
fr-en
FT w/o few-shot
Zero-shot
89.09
86.16
44.14
68.12
Five-shot
89.12
86.10
44.05
68.10
FT w/ fho-shot
Zero-shot
89.13
86.12
44.83
68.46
Five-shot
89.15
86.14
44.95
68.58
en-fr
FT w/o few-shot
Zero-shot
86.08
85.69
41.33
65.59
Five-shot
86.07
85.75
41.19
65.59
FT w/ sho-shot
zero-shot
86.35
85.93
41.96
66.07
Five-shot
86.41
85.92
42.10
66.20
nl-en
FT w/o few-shot
Zero-shot
86.81
85.39
30.37
59.02
Five-shot
86.99
85.45
30.83
59.24
FT w/ few-shot
Zero-shot
86.89
85.45
30.71
59.15
Five-shot
86.90
85.51
30.89
59.25
en-nl
FT w/o few-shot
Zero-shot
84.10
83.26
21.87
54.25
Five-shot
84.63
83.69
22.14
54.20
w/ few-shot
Zero-shot
84.16
83.38
22.10
54.61
Five-shot
84,42
83.57
22.44
54.85
pt-en
FT w/o few-shot
Zero-shot
89.26
84.86
47.81
71.25
Five-shot
89.30
84.82
48.07
71.17
FT w/ few-shot
Zero-shot
89.35
84.86
48.30
71.40
Five-shot
89.32
84.88
48.16
71.39
en-pt
FT w/o few-shot
Zero-shot
87.98
83.75
41.26
66.19
Five-shot
87.88
83.57
41.43
66.24
w/ few-shot
Zero-shot
88.01
83.78
41.63
66.49
Five-shot
88.00
83.81
42.08
66.71
ru-en
FT w/o few-shot
Zero-shot
85.93
83.78
33.57
60.86
Five-shot
86.01
83.82
33.82
61.03
FT w/ few-shot
Zero-shot
86.03
83.84
34.02
61.20
Five-shot
86.13
83.84
34.40
61.44
en-ru
FT w/o few-shot
Zero-shot
86.46
82.68
23.37
52.00
Five-shot
86.08
82.43
23.89
52.21
FT w/ few-shot
Zero-shot
86.92
83.10
24.21
52.80
Five-shot
86.64
82.92
24.07
52.62
zh-en
FT w/o few-shot
Zero-shot
84.16
81.48
23.23
51.41
Five-shot
83.87
81.26
22.59
50.65
FT w/ few-shot
Zero-shot
84.48
81.81
24.09
51.95
Five-shot
84.51
81.78
24.28
52.33
en-zh
FT w/o few-shot
Zero-shot
76.49
73.15
19.96
20.03
Five-shot
76.52
72.52
20.13
20.24
FT w/ few-shot
Zero-shot 78.02
74.89
21.60
21.23
Five-shot
77.89
74.45
21.82
21.32
Table 16: Scores of the 13B finetuned models (zero- and few-shot prompting) the Flores-200 test

---

COMET
COMETKiwi
BLEU
chrF
Language Pair
Model
Context
de-en
w/o few-shot
Zero-shot
83.40
80.09
30.16
55.15
Five-shot
83.50
80.02
30.62
55.45
FT w/ few-shot
Zero-shot
83.44
80.03
30.58
55.39
Five-shot
83.54
80.19
30.64
55.53
en-de
FT w/o few-shot
Zero-shot
81.88
79.82
25.33
54.72
Five-shot
81.87
79.71
25.12
54.62
FT w/ few-shot
Zero-shot
82.43
80.30
25.88
55.42
Five-shot
82.18
80.18
25.91
55.42
ru-en
w/o few-shot
Zero-shot
83.18
79.54
38.04
62.55
Five-shot
83.08
79.56
37.55
62.79
FT w/ few-shot
Zero-shot
83.49
79.83
38.23
62.88
Five-shot
83.51
79.87
38.28
62.94
en-ru
FT w/o few-shot
Zero-shot
83.90
79.60
21.57
48.19
Five-shot
84.03
79.75
22.17
48.51
FT w/ few-shot
Zero-shot
84.46
80.10
22.71
49.21
Five-shot
84.44
80.13
22.73
49.17
zh-en
FT w/o few-shot
zero-shot
76.11
73.54
16.66
45.52
Five-shot
75.65
72.88
17.99
46.38
FT w/ few-shot
Zero-shot
76.48
73.91
17.20
45.85
Five-shot
76.44
73.92
17.51
46.14
en-zh
FT w/o few-shot
Zero-shot
77.48
72.69
21.38
22.13
Five-shot
77.28
72.22
20.75
21.86
FT w/ few-shot
Zero-shot 78.80
74.15
23.31
23.63
Five-shot 78.94
74.40
23.32
23.49
Table 17: Scores of the 13B finetuned models (zero- and few-shot prompting) the WMT 2022 test set.

---

COMET
COMETKiwi
BLEU
chrF
Domain
Language Pair
Model
Context
Medical
de-en
FT w/o few-shot
Zero-shot
83.99
81.19
42.01
62.18
Five-shot
84.06
80.98
42.39
62.42
FT w/ few-shot
Zero-shot
84.18
81.24
43.15
62.60
Five-shot
84.26
81.26
43.11
62.76
en-de
FT w/o few-shot
Zero-shot
81.31
81.09
32.36
56.33
Five-shot
81.12
80,92
31.89
56.05
FT w/ few-shot
Zero-shot
81.77
81.40
33.00
57.08
Five-shot
81.72
81.21
33.28
57.16
Law
de-en
FT w/o few-shot
Zero-shot
84.73
81.07
41.08
61.87
Five-shot
84.57
80.74
41.01
61.77
FT w/ few-shot
Zero-shot
85.03
81.25
42.72
63.04
Five-shot
85.13
81,40
42.74
63.06
en-de
FT w/o few-shot
Zero-shot
83.46
81.73
28.25
54.84
Five-shot
83.16
81.33
28.35
54.54
FT w/ few-shot
Zero-shot
83.91
82.05
29.97
55.89
Five-shot
83.87
82.04
29.89
55.90
Tico
en-fr
FT w/o few-shot
Zero-shot
79.56
85.37
34.22
59.72
Five-shot
79.43
85.24
36.47
60.62
FT w/ few-shot
Zero-shot
79.78
85.62
35.22
60.33
Five-shot
79.79
85.68
35.18
60.40
en-pt
FT w/o few-shot
Zero-shot
87.89
83.36
43.30
69.13
Five-shot
87.82
83.15
43.31
69.01
FT w/ few-shot
Zero-shot
88.16
83.67
44.21
69.68
Five-shot
88.29
83.74
44.48
69.87
Chat
en-de
FT w/o few-shot
Zero-shot
84.62
79.89
29.24
54.11
Five-shot
84.84
79.67
31.11
54.75
FT w/ few-shot
Zero-shot
85.56
80.28
30.33
54.51
Five-shot
85.64
80.36
30.90
55.09
en-fr
FT w/o few-shot
Zero-shot
87.46
81.01
47.39
64.62
Five-shot
88.18
81.32
49.43
66.37
FT w/ few-shot
Zero-shot
86.41
80.88
47.74
64.97
Five-shot
86,86
81.04
47.92
64.94
en-pt
FT w/o few-shot
Zero-shot
91.02
81.67
45.99
66.36
Five-shot
90.82
81.49
44.49
66.19
FT w/ few-shot
Zero-shot
90.56
81.31
44.75
65.93
Five-shot
90.50
81.38
44.89
66.00
Table 18: Scores of the 13B finetuned models (zero- and few-shot prompting) the test sets for specialized domains.


Reviews:
[{"review_id": "mo1dN2Kms7", "reviewer": "Reviewer_ceRx", "summary": "", "questions": "This is just out of curiosity - not sure whether using COMETKiwi for quality filtering and using COMET as the main metric would be a problem. Do BLEU scores and other metrics show the same trends? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n> “This paper can benefit from some editing and re-organization” (...) “The comparison between fine-tuning, adaptor fine-tuning, and LoRA does not seem very relevant to the main point of this work”\n\nYou are right that the study on the impact of finetuning and few-shot prompting for adapting large language models is the main contribution of this paper. However, we believe our findings about the effectiveness of efficient adaptation are also relevant and can be highly valuable to the research community due to the computational savings they enable. We will, nevertheless, consider these editing improvements in the final version.\n\n> “It would be nice to see more ways to combine fine-tuning with few-shot examples, why was uniform sampling used?”\n\nThis is a good question. In fact, we also experimented with considering 50% of the dataset as zero-shot and the remaining 50% containing few-shot instances, uniformly sampling between 1 and 5 examples. This strategy leads to results similar to the ones in the paper, which we will include in the camera ready version. Regarding sampling of examples, our approach is similar to that of Min et al., 2022, which uniformly sampled from the training set. We will make sure to cite this paper in the final version. We recognize that other example selection strategies could bring benefits and deem this an interesting future work direction.\n\n> “[...] Do BLEU scores and other metrics show the same trends?”\n\nThis is an important consideration. In our submitted version, we have provided results with several other reference-based and reference-free metrics (CometKiwi, BLEU, and chrF) in Appendix E. Importantly, the trends hold across all metrics. We will make sure to highlight this in the final version.\n\nFinally, we would like to thank you for pointing out the missing references and suggesting presentation improvements. We will make sure to consider them in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work presents an innovative approach to harvest the benefits of fine-tuning and few-shot learning by fine-tuning the LLM with few-shot examples. The problem that the authors are trying to solve - the trade-off between fine-tuning and few-shot learning - is a very important problem with LLMs. The experiments in this work are comprehensive and support the claims made by the authors. I also appreciate the nicely designed figures.", "reasons_to_reject": "- This paper can benefit from some editing and re-organization.\n\n- The comparison between fine-tuning, adaptor fine-tuning, and LoRA does not seem very relevant to the main point of this work, consider moving this to the appendix when re-organizing the paper.\n\n- It would be nice to see more ways to combine fine-tuning with few-shot examples, why was uniform sampling used? There does not seem to be any motivation or empirical experiments to explain this choice. ", "questions_to_authors": "This is just out of curiosity - not sure whether using COMETKiwi for quality filtering and using COMET as the main metric would be a problem. Do BLEU scores and other metrics show the same trends? ", "paper_topic_and_main_contributions": "The authors point out the problem that fine-tuning degrades few-shot performance when using LLMs for machine translation. They propose a method to include few-shot examples in the adaptor fine-tuning process, which boost MT performance with fine-tuning while keeping the few-shot capabilities of LLMs. Fine-tuning with few-shot examples improves translation performance, recovers few-shot capabilities, does not suffer from overgeneration, and reduces the rate of hallucination.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Line 087: Europarl: https://aclanthology.org/2005.mtsummit-papers.11/ Paracrawl: https://aclanthology.org/2020.acl-main.417/ please check and add necessary references for all the data sources used.", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3: including the model performance with (1) only fine-tuning and (2) only prompt learning as baselines would better contextualize the results.\n\nLine 087: red comma\n\nLine 220: out-of-English"}, "extra_text": {}}, {"review_id": "kY84hX10Vh", "reviewer": "Reviewer_3izU", "summary": "", "questions": "1. How about using larger foundational models (e.g., 13B and others)? \n \n2. Why do the three length distributions in Figure 4 appear the same? \n\n3. Why does fine-tuning with few-shot examples improve the performance of zero-shot learning? Your explanation is that \"the model is extracting helpful information from the examples.\" However, I find this explanation less convincing, considering the LLaMA's strong capability to extract information.\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\n> \"Regarding the first finding, some open-source projects (e.g., Stanford Alpaca and Alpace-Lora) have demonstrated the effectiveness of finetuning based on the adapter method.\"\n\nOur paper differs from the Alpaca models as they focus on how to train large language models to follow instructions as a means to generalize to arbitrary tasks. The purpose of our paper is rather to study how large language models can be specialized to machine translation. Note that none of the aforementioned Alpaca projects test efficient finetuning techniques in this specific scenario. One important challenge is that our task is highly multilingual and that LLaMA is highly English-centric. Thus, we find it noteworthy to point out that it is possible to capture the multilinguality aspect of machine translation with efficient finetuning.\n\n> \"The second finding is interesting, but it lacks in-depth analysis and only presents a result.\"\n\nOur second finding reports that finetuning language models degrades their few-shot capabilities. This is an important limitation of current methods, which we believe can be very useful for researchers and practitioners. Note also that our results are consistent across several domains. We welcome any suggestions you may have for deepening the analysis.\n\n> \"As for the third finding, the in-context learning, which is restored by the proposed method, only shows slight improvement in Figure 3.\"\n\nWhile the gains are relatively small, they are consistent across the several domains. The main take-away of that experiment is that with this method we are able to recover in-context learning abilities, a very important characteristic of language models. This shows that finetuning and in-context learning can coexist. \n\n> \"How about using larger foundational models (e.g., 13B and others)?\"\n\nThank you for the suggestion. For the camera ready version, we will run our experiments on the 13B LLaMA model. In the meantime, we have the following preliminary results. We have found that, similarly to LLaMA-7B, the training of the 13B version also seems to plateau around the same number of training examples. As for the results in Table 3, we found that the overall trends reported in the paper seem to hold for the 13B version, with a consistent degradation (as evaluated with COMET and COMETKiwi; we report these results in the tables below) of few-shot performance for finetuned models and a restoration of this performance when finetuning with examples. We will include more detailed results in the final version.\n\nResults with COMET:\n| Model                 | Context   | Flores | WMT | Law | Medical | Tico | Chat |\n| ------------------- | ----------- | ------: | ------: | ----: | --------: | -----: | -----: |\n| FT w/o few-shot | Zero-shot | 85.83 | 80.99 | 84.09 | 82.65   | 83.73 | 87.70 |\n|                            | Few-shot  |85.84  | 80.90 | 83.87 | 82.59   | 83.63 | 87.95 |\n| FT w/ few-shot   | Zero-shot | 86.11 | 81.51 | 84.47 | 82.97   | 83.97 | 87.51 |\n|                            | Few-shot  | 86.12 | 81.51 | 84.50 | 82.99   | 84.04 | 87.67 |\n\nResults with COMETKiwi:\n| Model                 | Context   | Flores | WMT | Law | Medical | Tico | Chat |\n| ------------------- | ----------- | ------: | ------: | ----: | --------: | -----: | -----: |\n| FT w/o few-shot | Zero-shot | 83.06 | 77.54 | 81.40 | 81.14  | 84.36 | 80.86 |\n|                            | Few-shot  | 82.96 | 77.36 | 81.03 | 80.95  | 84.20 | 80.83 |\n| FT w/ few-shot   | Zero-shot | 83.31 | 78.05 | 81.65 | 81.32  | 84.65 | 80.82 |\n|                            | Few-shot  | 83.30 | 78.11 | 81.72 | 81.23  | 84.71 | 80.93 |\n\n> \"Why do the three length distributions in Figure 4 appear the same?\"\n\nThe length distributions measure the number of tokens for the reference translations and the translations generated with both finetuning approaches. The fact that they are similar indicates that the translations generated by the finetuned models are similar in length to the reference translation, suggesting that the models do not overgenerate and learn when to stop. Note that this property is desirable — as it removes any need to post-process/parse the model output — and is not found prior to finetuning (prompting the pretrained model with few-shot exemples leads to overgeneration, as shown in Figure 4). In the camera ready version, we will make this more clear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is straightforward and easy to understand. \n\n2. The finding that the fine-tuned LLM cannot benefit from in-context learning is interesting. \n", "reasons_to_reject": "1. The contribution is limited, even for a short paper. Regarding the first finding, some open-source projects (e.g., Stanford Alpaca and Alpace-Lora) have demonstrated the effectiveness of finetuning based on the adapter method. The second finding is interesting, but it lacks in-depth analysis and only presents a result. As for the third finding, the in-context learning, which is restored by the proposed method, only shows slight improvement in Figure 3. \n\n2. The analysis is not sufficiently thorough, and more experiments are needed, particularly on larger LLMs.\n", "questions_to_authors": "1. How about using larger foundational models (e.g., 13B and others)? \n \n2. Why do the three length distributions in Figure 4 appear the same? \n\n3. Why does fine-tuning with few-shot examples improve the performance of zero-shot learning? Your explanation is that \"the model is extracting helpful information from the examples.\" However, I find this explanation less convincing, considering the LLaMA's strong capability to extract information.\n", "paper_topic_and_main_contributions": "The paper initially compared the performance of a fully fine-tuned large language model (LLM) and an adapter-based LLM in the context of a machine translation task. The results showed that few-shot learning is not beneficial for the LLM after fine-tuning. Subsequently, the authors introduced few-shot samples into the instructions during the fine-tuning stage. The experiments show that their method can restore the effectiveness of in-context learning.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 2, Line 087, there is a misplaced comma (red comma)."}, "extra_text": {}}, {"review_id": "3MIWcA5FxS", "reviewer": "Reviewer_mF6X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> \"The reason is that we might want to analyze whether this fine-tuning process is just specializing the LLAMA model to a translation model\"\n\nIndeed, in this work we aimed to explore how to finetune large language models to machine translation. Our main goal was comparing both in-context learning and finetuning methods, and their overall impact when adapting LLMs for translation. \n\n> \"For example, when translating with Meidcal/Law, was the model able to absorb the domain knowledge shown in the few-shot examples, such as domain-specific terms, expressions, and styles?\"\n\nThis is a great question. We followed previous work that explored domain adaptation through in-context learning (Agrawal et al., 2022) and only reported performance in terms of translation quality. However, verifying whether the model is indeed “adapting” is relevant and we believe our paper can benefit from this analysis. We have analyzed the translations generated in this setting and did find examples in which the model is adapting and using domain-specific terms and styles (see examples below). What is more, perhaps surprisingly, we also found examples in which the introduction of these few-shot examples proved to be harmful (e.g., few-shot examples can introduce hallucinations — see Table 7 in Appendix C for examples of such pathological translations). We will make sure to add this more in-depth analysis in the final version.\n\nExamples of adaptation:\n\nSource: bezeichnet \"genetisch veränderte Futtermittel\" Futtermittel, die GVO enthalten, daraus bestehen oder hergestellt werden;\n\nReference: \"genetically modified feed\" means feed containing, consisting of or produced from GMOs;\n\nTranslation with Zero-shot: \"Genetically modified feed\" means feed containing GVO, derived from GVO or produced from GVO;\n\nTranslation with Few-shot: \"genetically modified feed\" means feed containing, consisting of or produced from GMOs;\n\nNote how in this case the model correctly translates the terminology GVO to GMOs (Genetically Modified Organisms).\n\n--------------------------------------------------------------------------------\n\nSource: VERORDNUNG (EG) Nr. 538/2000 DER KOMMISSION\n\nReference: COMMISSION REGULATION (EC) No 538/2000\n\nZero-shot: (EG) No 538/2000 OF THE COMMISSION\n\nFew-shot: COMMISSION REGULATION (EC) No 538/2000\n\nNote how the model is able to correctly order the words in the translation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiment results show that FT w/ few-shot is generally beneficial for improving the translation quality when used with few-shot examples", "reasons_to_reject": "1/ The study is lacking analysis on finetuning with in-domain translation examples vs. general-domain translation examples\n\n\n\nThe reason is that we might want to analyze whether this fine-tuning process is just specializing the LLAMA model to a translation model\n\n\n\n\n\n2/Since the main purpose of performing few-shot inference is to adapt the domain of translation. The study is lacking concrete insights on whether the domain-specific translation features are conceretely improved.\n\n\n\nFor example, when translating with Meidcal/Law, was the model able to absorb the domain knowledge shown in the few-shot examples, such as domain-specific terms, expressions, and styles?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides an analysis into the tradeoffs between few-shot prompting and finetuning for adapting large language models to machine translation. \n\n\n\n1/ Adapter-based finetuning matches full finetuning performance with 50 times fewer parameters \n\n\n2/ Finetuning compromises few-shot performance and the model's ability to adapt to new domains \n\n\n3/ The authors introduce few-shot examples during finetuning to bring the performance up for few-shot use cases", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s5od6eXXrU", "reviewer": "Reviewer_G2rv", "summary": "", "questions": "1. Where are the baseline numbers on few-shot when compared with zero-shot on the pre-trained model itself? Knowing these numbers per test sets (both Flores-200 and domain test sets) is crucial for understanding how much drop in few-shot translation quality is observed post-finetuning.\n2. Why limit the finetuning data to 160K in Figure 2? Did you stop observing gains after this or was it the maximum data used for finetuning?\n3. What was the nature of hallucinations observed in Table 1? Is it output in the wrong language or mainly detached ('Curious Case of hallucinations in NMT', NAACL'21) or overgenerated outputs ('Finding-Memo', EMNLP'22)? Could you quantify it into a few categories based on the hallucination literature (wrong-lang, detached, oscillatory, etc.)?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> \"(...) it would have been very interesting to vary the LLMs across two dimensions: scale (at least upto 13B) and diversity (more LLM instances such as MPT, Falcon, Bloom, etc.).\"\n\nWe recognize that only performing experiments on the 7B LLaMA model may limit the generalizability of our results. We provide a comprehensive answer to your concerns regarding this in the response to R2, who also shared the same concerns. We will make sure to include this analysis in the final version.\n\n> \"The evaluations report only one metric, the reference based COMET-22 metric. LLMs such has PALM, GPT-3.5 have shown to significantly impacted by reference bias and as such reference-free QE metrics such as COMET-KIWI should also be included to understand the impact of finetuning.\"\n\nPlease note that we reported results with COMETKiwi in appendix E, as well as BLEU and chrF for completeness. The trends are similar to the ones obtained with COMET-22. We will give more emphasis to the results with the other metrics in the final version.\n\n> \"Similarly, while Flores-200 is a useful test set, I am surprised the authors did not conduct any experiments on WMT-22 test sets.\"\n\nThis is a great suggestion. We tested our models on the WMT-22 test set and found that our trends hold across all metrics, as shown below. These results will be included in the camera ready version.\n| Model                  | Context   | COMET | COMETKiwi | BLEU | chrF   |\n| ------------------- | ----------- | --------: | -------------: | ------: | ------: |\n| FT w/o few-shot | Zero-shot | 79.44    | 76.01            | 23.72 | 46.23 |\n|                            | Few-shot  | 79.07    | 75.31            | 23.24 | 45.63 |\n| FT w/ few-shot   | Zero-shot | 80.16    | 76.72            | 24.58 | 47.09 |\n|                            | Few-shot  | 80.23    | 76.77            | 24.63 | 47.27 |\n\n> \"Where are the baseline numbers on few-shot when compared with zero-shot on the pre-trained model itself?\"\n\nThis is a good question. In our paper, both in Figure 1 and Figure 2, we report results with the pretrained model using few-shot, by randomly sampling examples from the development sets (see L135, L143). We will make sure that this is more explicit in the camera ready version. \n\nWe did not report zero-shot performance of the pretrained model since it is far behind, particularly in en-xx directions. As such, we considered it more interesting to compare the finetuned models with the pretrained model prompted with few-shot. In any case, the zero-shot results are shown below. We will add these results in the final version (possibly in Appendix). \n\n| Language Pair | COMET | COMETKiwi | BLEU | chrF   |\n| ---------------- | ---------: | -------------: | ------: | ------: |\n| de-en              | 87.23      | 82.99           | 36.77 | 62.34 |\n| en-de              | 71.56      | 57.87           | 13.99 | 38.73 |\n| fr-en                | 87.04      | 84.61           | 37.27 | 62.79 |\n| en-fr                | 76.54      | 65.17           | 23.43 | 47.02 |\n| nl-en               | 84.60      | 83.59           | 25.42  | 53.94 |\n| en-nl               | 73.99      | 61.17           | 12.45  | 38.88 |\n| pt-en              | 87.26      | 83.21            | 41.14 | 65.78 |\n| en-pt              | 74.60      | 55.91            | 17.46 | 40.46 |\n| ru-en              | 83.85      | 82.03            | 28.37 | 57.00 |\n| en-ru              | 57.63      | 46.48            | 7.16   | 17.48 |\n| zh-en             | 76.74      | 77.46            | 12.54 | 34.99 |\n| en-zh             | 49.12      | 45.10            | 4.54   | 7.20  |\n\n> \"Why limit the finetuning data to 160K in Figure 2? Did you stop observing gains after this or was it the maximum data used for finetuning?\"\n\nRegarding our training procedure, we stopped training because the validation loss plateaued, suggesting that the model had converged, which was supported by the fact that the COMET scores in Figure 2 stabilized.\n\n> \"What was the nature of hallucinations observed in Table 1?\"\n\nIt is true that we did not elaborate on the nature of hallucinations, but we will do so in the CR version. Briefly, we observed that the model generates hallucinations of different categories. In particular, it generates both detached (fully and strongly) and oscillatory hallucinations, and can also generate off-target translations. One common case is that the model copies from the instruction (either the source or from the examples). We will report this information in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is a timely paper, with not very comprehensive but interesting and logical results, and can help practitioners help leverage competitive open source LLM implementations such as LLAMA for the task of translation.", "reasons_to_reject": "1. Non-Comprehensive Experiments: The biggest problem with the paper as it currently stands is that all of the conclusions are made solely based on one 7B LLM, this makes adjudicating the scientific merit of the conclusions very difficult. While the results are quite intuitive, it would have been very interesting to vary the LLMs across two dimensions: scale (at least upto 13B) and diversity (more LLM instances such as MPT, Falcon, Bloom, etc.).\n2. Limited Evaluation: The evaluations report only one metric, the reference based COMET-22 metric. LLMs such has PALM, GPT-3.5 have shown to significantly impacted by reference bias and as such reference-free QE metrics such as COMET-KIWI should also be included to understand the impact of finetuning. Similarly, while Flores-200 is a useful test set, I am surprised the authors did not conduct any experiments on WMT-22 test sets. Doing these experiments is crucial since a lot of existing literature has benchmarked results on WMT-22 (e.g. Hendy et-al, How Good are GPTs at MT?) and the absence of any comparisons with the state-of-the-art makes the evaluation look less impressive. Similarly, the baseline numbers on few-shot translation performance on Flores-200 test set is missing in the paper, knowing this number is crucial, since a lot of the mistakes in the zero-shot mode is due to the model not knowing the appropriate output space ('Rethinking the role of demonstrations', Min et al, 2022) and hence producing translations in the wrong language and I am curious if this is the key benefit of finetuning itself.", "questions_to_authors": "1. Where are the baseline numbers on few-shot when compared with zero-shot on the pre-trained model itself? Knowing these numbers per test sets (both Flores-200 and domain test sets) is crucial for understanding how much drop in few-shot translation quality is observed post-finetuning.\n2. Why limit the finetuning data to 160K in Figure 2? Did you stop observing gains after this or was it the maximum data used for finetuning?\n3. What was the nature of hallucinations observed in Table 1? Is it output in the wrong language or mainly detached ('Curious Case of hallucinations in NMT', NAACL'21) or overgenerated outputs ('Finding-Memo', EMNLP'22)? Could you quantify it into a few categories based on the hallucination literature (wrong-lang, detached, oscillatory, etc.)?", "paper_topic_and_main_contributions": "This paper studies the impact of finetuning LLMs (LLAMA-7B only) for the translation task, on the translation capabilities of the LLMs, mainly the general translation quality (measured over 10 directions, both E-X and X-E on the Flores-200 test sets) and few-shot domain adapted translation quality (measured on 4 domain-specific test sets: Medical, Law, TICO and WMT-Chat). The key finding is that while fine-tuning (with and without LoRa) improves general translation quality on Flores-200, with LoRa fine-tuning only slightly behind full finetuning performance; the domain adaptation abilities of the model through few-shot learning gets severely impacted. The authors then finetune LLAMA with few-shot examples directly and this recovers the few-shot performance of the finetuned, which is evident in improved domain adaptation performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Kudos to the authors for a well-written paper. "}, "extra_text": {}}, {"review_id": "57sYicCFrj", "reviewer": "Reviewer_Rre2", "summary": "", "questions": "1. Could you explain Lines 517-519 in more detail? I don't get it.\n\n2. Could you offer the results for different templates in Table 3? \n\n3. For Table 4 and 5, could you also offer the results for different settings? I'm interested in the variance.\n\n4. In Figure 2, do you have results for full fine-tuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> \"Only one pre-trained model is applied. I would be interested in the analysis of different models, like different sizes of LLaMA, and FLAN-finetuned models.\"\n\nWe agree that considering scaled-up LLaMa versions can strengthen our work. We provide a comprehensive answer to your concerns regarding this in the response to R2, who also shared the same concerns. We will make sure to include this analysis in the final version.\n\n> \"Could you explain Lines 517-519 in more detail? I don't get it.\"\n\nRegarding lines 517-519, the first template repeats the translation instruction multiple times. When a pretrained model is prompted with it, it sees the repeating pattern and continues to generate more examples besides the target translation. With our remaining templates, we designed a separate examples section that would better separate the examples from the input. Our goal was to reduce the propensity for overgeneration. However, we found that all templates lead to overgeneration with the pretrained model and none suffer from this issue when the model is finetuned. In the camera ready version, we will better explain this information.\n\n> \"Could you offer the results for different templates in Table 3?\"\n\nThe results for the different templates were very similar, as shown in the Table below. We believe that once the model learns which template to expect, it has little influence on the obtained results. Nevertheless, we would like to note that the ranking of the templates according to performance is very consistent across metrics.\n\n| Format | COMET | COMETKiwi | BLEU | chrF   |\n| -------: | ---------: | -------------: | ------: | ------: |\n| 1          | 85.25     | 82.49           | 32.44  | 57.25 |\n| 2          | 85.34     | 82.54           | 32.62  | 57.37 |\n| 3          | 85.27     | 82.51           | 32.39  | 57.22 |\n\n> \"For Table 4 and 5, could you also offer the results for different settings? I'm interested in the variance.\"\n\nIn the Table below, we present the average scores on the validation set for the hyperparameter configurations for the LoRA models. Note that we only tried label smoothing without weight decay, as our best model in initial experiments did not have weight decay. Overall, the results show a small variability across all the metrics.\n\n| LoRA-R | Weight Decay | Label Smoothing | COMET | COMETKiwi | BLEU | chrF   |\n| --------: | ---------------: | -------------------: | ---------: | -------------: | ------: | ------: |\n| 128       | 0.0                  | 0.0                        | 84.74     | 81.96            | 31.47 | 56.36 |\n| 128       | 0.1                  | 0.0                        | 84.76     | 81.98            | 31.45 | 56.34 |\n| 128       | 0.0                  | 0.01                      | 84.79     | 81.99            | 31.48 | 56.37 |\n| 128       | 0.0                  | 0.05                      | 84.80     | 82.00            | 31.28 | 56.30 |\n| 128       | 0.0                  | 0.1                        | 84.61     | 81.85            | 31.10 | 56.15 |\n| 128       | 0.0                  | 0.2                        | 84.36     | 81.61            | 30.71 | 55.89 |\n| 256       | 0.0                  | 0.0                        | 84.78     | 82.01            | 31.47 | 56.40 |\n| 256       | 0.1                  | 0.0                        | 84.72     | 81.94            | 31.41 | 56.32 |\n| 256       | 0.0                  | 0.01                      | 84.87     | 82.08            | 31.57 | 56.45 |\n| 256       | 0.0                  | 0.05                      | 84.78     | 81.97            | 31.47 | 56.39 |\n| 256       | 0.0                  | 0.1                        | 84.65     | 81.92            | 31.28 | 56.25 |\n| 256       | 0.0                  | 0.2                        | 84.48     | 81.77            | 30.95 | 56.01 |\n\n> \"In Figure 2, do you have results for full fine-tuning?\"\n\nDue to storage constraints, we had to delete intermediate checkpoints when using full-finetuning (note that each copy of the model in bf16 takes up to around 13GB). For the final version, we will rerun the best configuration and obtain the necessary results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow;\n2. Extensive experiments are conducted to support the main findings;\n3. The analysis is thorough.", "reasons_to_reject": "1. The novelty of the method is limited, but I like the analysis;\n2. Only one pre-trained model is applied. I would be interested in the analysis of different models, like different sizes of LLaMA, and FLAN-finetuned models.", "questions_to_authors": "1. Could you explain Lines 517-519 in more detail? I don't get it.\n\n2. Could you offer the results for different templates in Table 3? \n\n3. For Table 4 and 5, could you also offer the results for different settings? I'm interested in the variance.\n\n4. In Figure 2, do you have results for full fine-tuning?", "paper_topic_and_main_contributions": "This paper mainly discusses how to tune a pre-trained language model for machine translation. Firstly, as previous works, it finds that parameter-efficient fine-tuning offers comparable performance as full-finetuning while only tuning a negligible amount of parameters. Next, it explores the role of prompting during fine-tuning and finds out that fine-tuning with few-shot prompting offers better results for both in-domain and out-of-domain translation.\n\nThe main contributions of this paper are:\n1. It combines both in-context tuning and fine-tuning to tune a large pre-trained language model, which is novel.\n2. The main conclusions are supported by extensive experiments and analysis. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 3, it's better to present zero/few-shot results of the pre-trained model as baselines."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that finetuning with LoRA-based SFT is a good choice for improving the translation performance of LLaMa 7B. It also shows that such a way of finetuning may harms the few-shot translation ability and can be refined by introducing few-shot samples into finetuning. The experiments were conducted on datasets of 10 language pairs.

Fortunately, we have 5 reviewers for this submission. The reviewers agree that this paper investigates interesting issues, and some of the results could be of interest to the community, especially given that tuning LLMs for downstream tasks plays an important role in LLM applications. However, some of the reviewers have concerns about the novelty and experiments. Also, the base model selection is another concern.

My major concern is that the claims here should be examined carefully. For example, the author states that finetuning is harmful to few-shot translation. This might not be a solid conclusion because the author did not finetune the LLM in a few-shot manner. As usual, to achieve good translation abilities for LLMs, we need to use SFT to involve few-shot and zero-shot promotes, rather than zero-shot promotes. Therefore, the finding here is due to the way the author chooses, but not the conclusion we can draw in a general setup. A related problem is that the author uses a relatively large number of samples to tune the model. As is pointed out in related work, more tuning data is not always helpful, and the resulting model tends to overfit this data in many cases.

Another concern is that the use of LLaMa 7B narrows the scope of this work. I basically agree with Reviewer G2rv in that LLaMa is not the best choice for translation. I suspect that the conclusion may change if we use multi-lingual LLMs and/or increase the model size. In this sense, the results are highly dependent on the LLM used in this work.
Decision: Accept-Findings