Dissecting In-Context Learning of Translations in GPT-3
Authors: Vikas Raunak
Arul Menezes
Hany Hassan Awadalla
Venue: EMNLP 2023 Findings (2023)
Keywords: translation
large language models
in context learning

Abstract: Most of the recent work in leveraging Large Language Models (LLMs) such as GPT-3 for Machine Translation (MT) has focused on selecting the few-shot samples for prompting. In this work, we try to better understand the role of demonstration attributes for the in-context learning of translations through perturbations of high-quality, in-domain demonstrations. We find that asymmetric perturbation of the source-target mappings yield vastly different results. We show that the perturbation of the source side has surprisingly little impact, while target perturbation can drastically reduce translation quality, suggesting that it is the output text distribution that provides the most important learning signal during in-context learning of translations. We propose a method named Zero-Shot-Context to add this signal automatically in Zero-Shot prompting. We demonstrate that it improves upon the zero-shot translation performance of GPT-3, even making it competitive with few-shot prompted translations.

Dissecting In-Context Learning of Translations in GPTs
Vikas Raunak
Hany Hassan Awadalla
Arul Menezes
Microsoft Azure Al Redmond, Washington (viraunak, hanyh, arunmammmccmooff..tt.....
Abstract
the most useful samples for few shot learning, a topic that has received considerable attention for cliciting better translations from LLMs (Vilar et al., 2022; Agrawal et al., 2022). Our contributions are: We explore the role of demonstration at- tributes within in-context learning of transla- tions in the GPT family of LLMs, through per- turbations of the input-output (source-target) mappings. We show that the target text distri- bution is the most important factor in demon- strations, while the source text distribution provides an inconsequential learning signal.
Most of the recent work in leveraging Large Language Models (LLMs) such as GPT-3 for Machine Translation (MT) has focused se- lecting the few-shot samples for prompting. In this work, we try to better understand the role of demonstration attributes for the in-context learning of translations through perturbations high-quality, in in-domain demonstrations. We find that asymmetric perturbation of the source- target mappings yield vastly different results. We show that the perturbation of the source side has surprisingly little impact, while tar- get perturbation can drastically reduce trans- lation quality, suggesting that it is the output text distribution that provides the most impor- lant learning signal during in-context learning of translations. We propose method named Pero-Shot-Contee to add this signal automati- cally in Zero-Shot prompting. We demonstrate that it improves upon the zero-shot translation performance GPT-3, even making compet- itive with few-shot prompted translations.
2. Based on our findings, we propose Zero-Shot- Context prompting, which tries to automati- cally provide the learning signal correspond- ing to the target text distribution without any source-target examples. This greatly improves GPT-3's zero-shot performance, even making it competitive with few-shot prompting.
2 Related Work
1 Introduction
Our work related to two key themes, namely prompting LLMs for translation and analysis in-context learning in LLMs. In this section, we situate our work within these two themes.
Recent work has put into question the importance of the correctness of demonstrations for prompt- ing in Large Language Models (LLMs) (Min et al., 2022). One key conjecture is that the latent zero- shot capabilities of LLMs might be considerably higher than their observed zero-shot capabilities for range of tasks (Min al., 2022; Kojima et al., 2022). One way to elicit higher zero-shot perfor- mance is to qualify the role of demonstration at- tributes towards task performance and then simu- late such in-context learning signals in zero-shot manner. However, realizing this goal hinges on explicitly dissecting the role of various demonstra- tion attributes (format, inputs, outputs, input-output mapping) towards task performance within few- shot in-context learning. In this work, we explore these questions for the task Machine Translation (MT). Our line of inquiry is orthogonal to finding
LLM Prompting for MT: LLMs have achieved close the ttate-of-the-att art translation performance under few-shot prompting (Hendy al., 2023; Lin et al., 2022). Most of the work for prompting in MT' has focused on selecting the training or devel- opment instances to be used as examples during prompting. Vilar et al. (2022) experiment on PaLM (Chowdhery et al., 2022) and find that quality of examples is the most important factor in few-shot prompting performance. Agrawal al. (2022) periment with XGLM (Lin et al., 2021) and report that translation quality and the domain of the exam- ples are consequential. Our work builds on these with a different aim, in that we do not explore se- lecting the examples, rather apply perturbations on

---

Ground Truth
Shuffled Targets
Jumbled Source
Jumbled Target
Reversed Target
English: A B C
English: A B C
English: B A C
English: A B C
English: A B C
German: D E F
German: X Y Z
German: D E F
German: E F
German: F D
English: U V W
English: U V W
English: U W V
English: U V W
English: U W
German: X Y
German: D F
German: X Y
German: Z
German: Z ZYX
Table I: Perturbations Applied:
The four types of perturbations (shown here as applied on abstract source-target example sequences) manipulate the demonstration attributes differently. For example, while Jumbled Source and Jumbled Target both corrupt the souree-target mapping, they modify different learning signals in in-context learning.
high-quality, in-domain examples to better qual- ify the role of certain demonstration attributes for in-context learning of translations.
2022). We also investigate the veracity of our ob- servations with text-davinci-001 and text-curie-001, two prior LLM versions in the GPT family as well as the more recent text-davinci-003.
Analyzing In-Context Learning: Theoretical and empirical investigation of in-context learning is ongoing research endeavor (Xie et al., 2021; von Oswald et al., 2022; Akyürek al., 2022: Dai et al., 2022). Min et al. (2022) demonstrate that label correctness demonstrations of limited importance for open-set classification tasks, while Yoo et al. (2022) show that negated labels do matter. Our experiments differ from these works both on the choice of the task (translation, which has exponential output space) well the types of perturbations applied to the demonstrations.
Datasets: We experiment with the WMT'21 News Translation task datasets (Barrault et al., 2021), for the following four language pairs: English-German (En-De), German-Englin (De- En), English-Russian (En-Ru) and Muussian-Englise (Ru-En). On each of these datasets text-davinci- 002 achieves highly competitive performance with the WMT-21 winning NMT model (Tran al., 2021), with eight demonstrations (k: in k- shot prompting). We list the full test set perfor- mance with text-davinci-Oe and text-davinci-O0 for k = 8 in Table 2, while the perturbation exper- iments are reported on 100 random samples from the test sets in each case.
3 The Role of Demonstration Attributes
To produce outputs for specific task, LLMs are typically prompted with demonstrations (input- output examples pertaining the specific task) appended with the test input. Similar to Min al. (2022), we posit that there exist four aspects of demonstrations the translation task that provide a learning signal: the input- mapping, the input text distribution, the output text distribution and the format. In this section, we conduct an empirical investigation on how LLMs such as GPT- 3 leverage the demonstrations provided to them for the task of translation by perturbing the input- output srource-targed) mappings provided during prompting. Through these experiments, we hope to compare the importance of three key demonstration attributes the input text distribution, the output text distribution and their mapping for translation.
Method
En-De
De-En
Ru-En
En- Rn-Ru
Facebook- WMT-21
39.36
39.88
35.25
46.41
davinci-002 (k=8)
39.57
40.28
35.67
39.06
davinci-003 (k=8)
40.31
41.31
36.03
41.82
Table COMET-QE Scores on WMT-21 Test Sets:
Both the translations from the WMT-21 winning system (Tran et al., 2021) as well as the GPT translations were obtained through greedy decoding.
Vilar et al. (2022) report than the choice of the format inconsequential for few- shot prompting on the translation task. As such, we use the standard prompt used for MT in prior works, namely [Source]] ABC ('n) [Tarret]: DEF, where Source (c.g., English) and Target (e.g., German) represent the language names. Further, we use high- quality, in-domain sentence pairs sampled from the development set for few-shot prompting.
Prompt Details:
Models: In this section, we mainly report results for text-davinci-002 one of the most capable LLM models publically accessible (Liang et al.,
To minimize reference-bing in eval- uation, which has been shown to be detrimental
Evaluation:
LLLMs: btttps://bbta. http://bbbtaa..oi....ee htt.://cc//////mee/........

---

estimating the LLM output quality in related se- quence transduction tasks (Goyal et al., 2022; Gar- cia et al., 2023; Raunak et al., 2023), we make use a state-of-the-ar Quality Estimation (Fomicheva et al., 2020) metric named COMET-QE (Rei et al., 2020) for quality evaluation. Further, one caveat of using the reference-free metric is that it allo- cates high scores to a translation if it is in the same language the source sentence, i.e. it doesn't 't pe- nalize copy errors translation. To mitigate this evaluation shorteoming, we use a language-id clas- sifier (Joulin et al., 2017) and set the translation to empty if the translation is produced the same language as the source.
Translation quality declines by a large value for JT, an effect that becomes larger with increasing k e.g., for JT perturbation at 8, the translation quality is considerably worse. On the other hand, JS produces very little to no effect on the quality of translations. Further, owing to the nature of the perturbation ST becomes more disruptive at higher values of k, while yielding no impact for l 1.
We repeat the same experiment as above (Experiment 1) with four different lan- guage pairs from WMT-21 and text
Experiment 2:
Perturbation Effect Language Pairs
40
None
ST
We apply four perturbations to the demonstrations used for prompting. Table enumerates the four perturbations with abstract source-target sequences: Shuffled Targets (ST) ran- domizes the mappings between the source and tar- gets in the prompt, Jumbled Source (JS) random- izes the position of the words in the source sen- tences, Jumbled Ref (JT) randomizes the positions of the words in the target sentences and Reversed Ref (RT) reverses the order of the words in the target sentence. Among these perturbations, ST impacts both the input and output spaces symmetri- cally, while the other perturbations (JS, JT and RT) perturb only one of the input/output spaces.
Experiment 1:
JS
COMET-0E Sce
JT
25-
20
15
Detn
Em-Ru
Ru-En
De-fr
Language
Figure 2: Perturhation effects different W language pairs for text-davinci-00) under few-shot prompting with k=8. The asymmetric effect of source and target perturbation holds true throughout the pairs.
The results are reported in Figure 2. We find that the trends are similar to the first experi- ment (Figure 1). Across the language pairs, JS and JT have asymmetric impact translation quality, showing that in each case the critical learning sig- nal arrives from the target text distribution, while the source text distribution an inconsequential factor with respect to the output translation quality.
Results:
Perturbation Effect vs Number of Demonstrations
30
COMET-DE Score
20
ST
15
JT
RT
10
We repeat Experiment 2, by keep- ing the language pair fixed to En-De and varying the LLMs. We report results in Figure for three other models from the GPT family, namely text- curie-001, text-davinci-002 and text-davinci-003.
Experiment 3:
Numbar Demonstrations
Figure 1: Perturbing the demonstrations for WMT-21 Baglish-GGerman test set. Source and Target perturba- tions have asymmetric effects despite the input-output mapping getting severely damaged in both cases.
Results: We find that across different models, JS and JT have asymmetric impact on the translation quality, consistent with the prior two experiments.
The results of upplying these perturba- tions on En-De are presented in Figure 1, across different number of demonstrations (k 1, 8). The results show that while ST and JT both signif- icantly disrupt the source-targe mappings in the demonstrations, they have greatly different impact.
Results:
Analysis: Compared to Min et al. (2022), wherein the randomization of the input-output map- pings in the demonstrations leads better perfor- mance than no demonstrations (zcro-shot prompt- ing) for open-set classification tasks, our results

---

Perturbation Effect vs GPT-3 Variants
the choice of the MT quality metric well: with COMET-KIWI (the state-of-the-art QE metric in the WMT--2 Quality Estimation Shared Task (Rei et al., 2022)), no perturbation (None in the plots) with spt-3.5-turbo-instee
None
ST
25
RT
COMET-OE
20
obtains a score of 83.75, the JS perturbation a score of 83.94 and the JT per- turbation obtains score of 73.26. Similarly, with COMET-KIWI
25
10
obtains score of 83.94, the JS perturbation score of 83.85 and the JT perturbation obtains a score of 72.72. These results point to the robustness of our findings.
culle001 davinci-000
davinci-002
devinci-003
Model
Figure 3: Perturbation effect across GPT-3 model vari- for the WMT-21 English-German test set, The asymmetric effect of source and target perturbation holds across different models, suggesting that this is stable trait of the in-context learning mechanism.
Our findings suggest that the data representing the output space might be the most im- portant attribute demonstrations for in-context learning of translations. Besides suggesting an in-built robustness towards perturbations on the source side, this result points to interesting ex- ploratory directions for data selection for prompt- ing, e-g., that target-original data might be more useful as demonstration examples than source- original. We leave such questions to future work.
Implications:
are quite different. We find that depending on the type of perturbation, in-context translation learn- ing results can be vastly different even when all the perturbations break the correct input-outpul map- ping. For some perturbations (e.g.. JT and RT) the translation quality is much worse than zero-shot. To reconcile these results, we hypothesize that the difference arises from the increased complexity of the auto-regressive search in the case of transla- tion, i.e., a clear specification of the output space in the demonstrations becomes much more critical to constrain the search space.
4 Zero-Shot-Contex for Translation
Previously, we demonstrated that the most impor- tant demonstration attribute for in-context learning of translations is the output text distribution. In this section, we present a method providing this learning signal zero-shot manner. Our experi- ment here represents an inverse of experiments section 3, i.e., here we add a useful learning signal to zero-shot prompting, rather removing learning signals from few-shot prompting to gauge their im- portance. We present a method named 'Zero-Shot- Context' and show that it greatly improves upon zero-shot performance for GPT-3, eliciting perfor- mance competitive even with few-shot prompting. Note that this method is one example of adding a particular signal in zero-shot prompting and that there could be multiple ways to add such a signal to bolster zero-shot performance including direct instruction finetuning on the translation task. How- ever, we leave a thorough analysis of improving zero-shot translation performance by adding rele- vant signals from demonstrations future work and focus only exploring on our key hypothesis.
Further, the ST results in Figures 2 & 3 show that souree-target mapping is also critical demon- stration attribute, fact consistent with prior results emphasizing the importance of example quality (Vi- lar al., 2022; Agrawal al., 2022). However, we show that it is not the primary learning signal in in- context learning of translations and even therein the source word order matters for little, suggesting that only an approximation of the input text distribution is sufficient for effective in-context learning.
Generality of Our Findings: We also conduct experiments on ppt-3.5-turbo-instuee and gpt-3.5- turbo-instruct-9914 two of the more recent LLMSs in the GPT family. With spt-3.5-turbo-instruce on En-De, no perturbation (None in the plots) obtains a COMET-QF score of 34.21, the JS perturbation score of 35.20 and the JT perturbation obtains score of 25.45. Similarly, with gpt-3.5-turbo- instruct-0914 on En-De, no perturbation obtains COMET-QE score of 33.64, the JS perturbation a score of 34.35 and the JT perturbation obtains a score of 24.42. This observed behavior is agnostic
Proposed Method: We propose a new zero-shot prompting method named ure 4), which auto nuto-generates the output space spec- ification learning signal from the LLM itself (the
(Fig-

---

Context Generation
the Context-Generation step's output with the ran- dom target-side sentence. Ideally, an in-domain, high-quality target-side sentence should also be able to provide a learning signal regarding the out- put text distribution. We find that this is indeed the case, and simply replacing the context gener- ation step with the random target-side sentence also improves upon zero-shot performance, achiev- ing 36. 10 COMET-QE score for WMT-21 En-De test set and 37.86 COMET-QE score for WMT-21 En-Ru However, these scores are lower than Zero- Cont-Cont,xt, suggesting that the contextual nature of Zero-Shot-CCaee Shot-Context is also important.
ABCD
Context:
Explain the above text in German.
English: ABCD
German:
Translation
Figure 4: Schematic for ZZrroShhot-ontextt The Con- text Generation step provides an automatic learning signal to the LLM about the output text distribution, simulating the most important demonstration attribute.
Context) and uses it condition the translation.
Further Analysis: Our findings indicate that the latent zero-shot GPT-3 performance for translations could indeed be higher than currently reported and that it is possible to leverage direct computation to improve LLM translation performance instead of manually retrieving selecting examples. In particular, we showed that simple addition of signal pertaining to the output space improved the zero-shot performance of text-davinci-002, a useful step towards better zero-shot utilization LLMs for translation. As pointed out in Bawden and Yvon (2023), generating zero-shot translations often suffers from outputs in the wrong language and find that Zero-Sher Shot-Context considerably alleviates this, leading better performance. How- ever, further rigorous analysis of this phenomenon across different L.LMs hindered by the fact that we do not have access to the training or the in- struction finetuning dataset used for the underlying state-of-the LLMs.
Method
CQET
BLEUT
Chrftt
TER.
Zero-Shot
32.29
22.6
54.3
71.4
Zero-Shun Chot-Context
37.65
23.1
55.4
68.5
Few Shot (k=l)
39.92
22.4
54.1
71.8
Few Shot (k=2)
39.04
24.7
56.6
64.8
Few Shot (k=4)
40.36
24.0
55.7
65.4
Table 3: Zero-Shot-Conter
Baselines on WMT-21 En-De: Paro-Shot-Contex greatly improves upon Zero- Shot Translations, gaining +5 QE points in quality.
Method
CQET
BLEU
ChrF't
TER.
Shot
35.39
19.8
49.4
74.3
Zero-Shot-Ge ConteCot
40.67
18.8
48.7
75.6
Few Shot (k=1)
37.92
20.5
50.1
72.3
Few Shot (k=2)
39.35
19.3
50.0
72.7
Few Shot (k=4)
39.25
20.2
50.1
72.3
Table 4: Zero-Shot-Conterr
Basclines on WMT-21 En-Ru: Zero-Shot-Comme greatly improves upon Zero- Shot and is even competitive with few-shot translations.
Experiment and Results: In Table 3 we com- pare Zero-Shot-Contex with Zero-Shot prompt- ing, as well as few-shot prompting (for k=1, 2. 4) with high-quality, in-domain examples sampled from the development set, En-De WMT-21 test set with text-davinci-002. The results show that Pcro-Shot-Contex greatly improves upon Zero- Shot translation quality as measured by COMET- QE (CQE). Note that the gains are not visible in reference-basee evaluation with BLEU and ChrF and limitations of these metrics have been pointed our in the literature (Freitag et al., 2022). Table 4 presents comparison on the WMT-21 En-Ru test set.
Summary and Conclusions
We analyzed the relative importance of demonstra- tion attributes learning signals within few-shot in-context learning of translations in LLMs from the GPT family. We demonstrated that the crit- ical learning signal arrives from the output text distribution, followed by the input-output mapping, while the input text distribution matters for little. We use this finding to propose Zero-Shor method that tries to automatically generate the critical learning signal. Zero-Shot-Comee greatly improves upon zero-shot translation quality in GPT- 3, further validating our findings. We hope that our work could serve as useful contribution towards better understanding of in-context learning of trans- lations LLMs.
Ablation on Zero-Shot Context: We consider the following experiment: we pick a random target- side sentence from the development set and replace

---

6 Limitations
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo. Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Our work experiments with high-quality, in- domain examples for few-shot prompting. It is conceivable that perturbations could have different impacts on examples with varying quality. Also, while our proposed zero-shot method does not con- sume any manual examples, it suffers from the limitation that it involves two passes over LLM. While this is mitigated by the method presented as an ablation, we believe that simpler methods to add the relevant demonstration signal could be derived by pre-compuring the singular target-side context once for the entire test set, a proposal we didn't investigate.
Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few- shot learning for machine translation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the of gpt-3.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? a comprehensive evaluation.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis. Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In- context examples selection for machine translation.
Armand Joulin, Edouard Grave. Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pu- pers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learn- ing algorithm is in-context learning? investigations with linear models.
Loic Barrault, Ondrej Bojar, Fethi Bougares. Rajen Chatterjee, Marta R. Costa-jussa, Christian Feder- mann, Mark Fishel, Alexander Fraser, Markus Fre- itag, Yvette Graham, Roman Grundkiewiez, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes. Philipp Koehn, Tom Kocmi, Andre Martins. Makoto Morishita, and Christof Mouz, edi- tors. 2021. Proceedings of the Sixth Conference on Machine Translation. Association for Computational Linguistics, Online.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models, arXiv preprint wwxiv:2221.091111
Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: the case of BLOOM. In Proceed- ings of the 24th Annual Conference of the European Association for Machine Trunslation, pages 157-170, Tampere, Finland. European Association for Machine Translation.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint 22Xxx:2212.105599
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig. Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint
Marina Fomicheva, Shuo Sun. Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel. Nikolaos Aletras, Vishrav Chaudhary, and Lucia Spe- cia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP.

---

Vikas Raunak, Arul Menezes, Matt Post, and Hany Has- san. 2023. Do GPT's produce less literal translations? In Proceedings of the st Annual Meeting of the A Association for Computational Linguisties (Voluee
2: Short Pupers), pages 1041-1050, Toronto, Canada. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. TT. Martins. 2022. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021. Facebook Al's WMT21 news translation task submission. In Proceedings of the Sixth Conference Machine Translation, pages 205-215, Online, Association for Computational Linguistics.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance.
Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning implicit bayesian inference.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyun- Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Tacuk Kim. 2022. Ground-truth labels matter: deeper look into input-label demonstrations. arXiv.


Reviews:
[{"review_id": "mLJDyCskbD", "reviewer": "Reviewer_gFZk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments. Please find the response below:\n\n1. Practicality: We apologize for not very clearly conveying the scope of the paper. This is a very focused analysis paper; we only posit one hypothesis and validate the hypothesis through multiple experiments. Our key hypothesis is that the output space has the most influence on in-context learning of translations. The input space can be perturbed without damaging the translation quality, this means that LLM is not learning the input-output mapping from the demonstrations. We also conduct a reverse experiment: that if we take zero-shot translations and just add a target side text prior to generating the translations, the translation quality jumps dramatically as measured by COMET.\n\n2. Confusion about Performance: Please note that surface-level metrics are not adequate for measuring LLM translation quality. This has been observed in multiple prior works, hence, to compare LLM translation quality, neural metrics must be used. In the neural metric used, we do see large jumps in performance from the proposed simple method. We cannot trust non-neural metrics for evaluating LLM translation quality and we will clarify this in the paper. Hence, the proposed reason to reject is not very relevant to the findings in the existing literature [e.g., Garcia et al, Unreasonable effectiveness of few-shot learning, etc.] since the primary metric of comparison should be COMET. We only included the other metrics for completeness and much weight should not be put on other metrics, owing to their well-known deficiencies.\n\n3. Zero-Shot Performance Variability: Under zero-shot with temperature=0, we noticed very little variability, hence the numbers can be trusted without std-dev. The only source of randomness comes from non-deterministic hardware, but empirically we find the diffs between the same runs to be <1% of the cases. We will add std-dev over 3 runs in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents an interesting finding that the target side offers important learning signals.\n2. Building upon this discovery, the authors propose a zero-shot prompting method.", "reasons_to_reject": "1. This paper lacks practicality, as it is easy to find high-quality examples of in-context learning in real-world applications.\n2. The performance improvement is not significant either, as seen in Table 3, where some results are even worse than those achieved by the zero-shot approach.\n3. It would be nice if the author could report the avg with std performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper examines the role of in-context examples in machine translation. The authors find that perturbing the source text has little impact while perturbing the target side can significantly reduce translation quality. Based on this observation, the paper proposes a zero-shot prompting method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uWtodqCobH", "reviewer": "Reviewer_uUav", "summary": "", "questions": "A: Addressing the elephant in the room: using GPT models that are soon to be deprecated hinders reproducibility; at this point (and at the time of the submission), for the language pairs that are studied in this paper (high-resource directions like en-ru, en-de, de-en), there are more than good enough open large language models that would dramatically increase the reproducibility and impact of these results. Have you replicated these results with any of those models (e.g., LLaMA models)?\n\nB: Beyond quality or aggregated corpus-level metrics, how do the translations of the LLM differ qualitatively under each different perturbation in your suite of perturbations? This analysis can provide way more insight than just presenting quality alone. \n\nC: What is the main intuition behind using the context generation process in Section 4? In which scenarios do you see this extra computation (i.e., an additional pass) being more useful than providing in-context examples? \n\nD: Could you provide some examples of the generated context in the initial pass required for the method in Section 4? Are there failure cases where this additional pass actually led the translation output to be of significantly lower quality than the vanilla zero-shot computation?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. Please find our response below:\n\n1. Experimental Setup: We have experimented with multiple random samples, sampled from the dev set and the trends remain the same. The effect sizes are too large, and we didn't include std-dev. We will make this point clear and report averages as well. For our experiments, X-E translation is not going to yield any useful analysis since the output space has a large factor of variance removed: the language. We will emphasize this point as to why the experiments are predominantly in E-X. At the time of writing this paper, we didn't have a public implementation of COMET-22, however, we will report COMET-22 numbers as well in the revision. The reported phenomenon and the central object of our analysis are pretty agnostic to the choice of the metric though, since we will only observe changes in magnitudes, not trends (the effect sizes are very large).\n\n2. Impact of Perturbations: Different perturbations on the target side yield different translations: for jumbled target, the model produces words in jumbled order (but relevant to translation). While source side perturbations do not yield significant differences to the translation surface form.\n\n3. Section 4: We mainly report COMET-QE + LID as the metric here. Our only observation here is that, adding an additional signal about the output space leads to a large jump in COMET-QE scores, almost equivalent to few-shot quality.\n\n4. Questions:\n\nA. The models investigated are not deprecated (https://platform.openai.com/docs/models/gpt-3-5) and as such our experiments can be readily replicated. We chose these LLMs owing to their state-of-the-art quality (HELM, Liang et al). We are quite confident as to the generality of our results (observed on both text-davinci-002 and text-davinci-003) and we will include more results with Llama 2 in the appendix.\n\nB. The translations do preserve the characteristics of the task that the model receives under perturbations, e.g., for jumbled target, a jumble of target words is produced. The key signal is that the model does not learn about the input-output mapping at all from these perturbed demonstrations, only what the output space represents.\n\nC. The key intuition is this: In the perturbation experiments, we removed learning signals from the few-shot examples through different perturbations. In context generation, we add a learning signal, i.e., describing what the output space will look like (e.g., German). So, we supply that information to the model and hope that, should our hypothesis be true, this will perform better than the zero-shot translation. \n\nD. Generated context is sometimes a paraphrase or paraphrase + continuation of the source text in the target language. The quality metric does not indicate a quality drop against zero shot, the biggest quality gain comes from the model almost completely stopping to produce wrong language outputs under context generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a valuable analysis study on the robustness of large language models when prompted to translate with in-context examples;\n- The paper studies several models (although closed) and language pairs;\n- The experiments in Section 3 are well motivated, explained, and the results are well synthesised and presented", "reasons_to_reject": "- The experimental setup has some relevant problems: (i) the samples that are used for the in-context examples are drawn from the development set; however, no study of variance to multiple sets of examples has been done -- this is critical as few-shot examples can impact the output quality; (ii) the evidence and analysis in this paper (Fig 1, Fig 3, Tables 2 and 3) are overwhelmingly presented for \"en-xx; out-of-English\" directions; however, LLMs exhibit different characteristics when translating into English [1,2] (e.g., much better quality in xx-en than en-xx directions) and it is not clear whether the results presented in the paper would translate for these directions as well; (iii) the choice of using a quality estimation system rather a neural metric for evaluation is odd and justified on a paper that studies a summarisation task; models like COMET-22 have shown superior correlations with human judgements than the reference-less counterparts [3].\n- The impact of the perturbations was only studied from the lens of quality; different vectors of analysis, such as error analysis, are relevant and could provide way more insight (e.g., hallucination, surface form, off-target translation, etc.)\n- Section 4 is convoluted: it is not clear what and how the \"context generation\" step helps generation of the translation; moreover, although reference-based metrics were used for Tables 2 and 3, the natural choice given the rest of the paper (COMET) was not employed.", "questions_to_authors": "A: Addressing the elephant in the room: using GPT models that are soon to be deprecated hinders reproducibility; at this point (and at the time of the submission), for the language pairs that are studied in this paper (high-resource directions like en-ru, en-de, de-en), there are more than good enough open large language models that would dramatically increase the reproducibility and impact of these results. Have you replicated these results with any of those models (e.g., LLaMA models)?\n\nB: Beyond quality or aggregated corpus-level metrics, how do the translations of the LLM differ qualitatively under each different perturbation in your suite of perturbations? This analysis can provide way more insight than just presenting quality alone. \n\nC: What is the main intuition behind using the context generation process in Section 4? In which scenarios do you see this extra computation (i.e., an additional pass) being more useful than providing in-context examples? \n\nD: Could you provide some examples of the generated context in the initial pass required for the method in Section 4? Are there failure cases where this additional pass actually led the translation output to be of significantly lower quality than the vanilla zero-shot computation?", "paper_topic_and_main_contributions": "This paper studies the robustness of large language models from the GPT family to perturbations on translation demonstrations. They use a suite of perturbations on both the source and target space (e.g., mismatched pairs, reversed word order, and others). Their experiments suggest that large language models, when translating under the paradigm of in-context learning, are more robust to source perturbations than target perturbations. In fact, a defective specification of the output space can have major quality impact. Finally, they propose a zero-shot-context method for translating with LLMs that does not require feeding in-context examples. Their method shows some benefits over vanilla zero-shot performance.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] https://aclanthology.org/2022.emnlp-main.616/\n\n[2] https://arxiv.org/abs/2303.01911\n\n[3] https://aclanthology.org/2022.wmt-1.2/"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Note that, given the experiments presented in the paper, writing \"source word order matters for little\" in L227, \"perturbation of the source side has surprisingly little impact\" is not only different but also highly more accurate than writing \"the input text distribution matters for little\" (L274) and \"the source text distribution provides an inconsequential learning signal\" (L53)."}}, {"review_id": "JStndBr8jk", "reviewer": "Reviewer_999s", "summary": "", "questions": "- This work only focuses on the impact of prompting LLMs for the translation task. However the finding is quite general. Have the authors looked into other tasks how well they can generalize there? \n \n- You find that \"the critical learning signal arrives from the output text distribution\", but the perturbations you applied were more complex on target side. Could that be the reason?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciative comments. Please find the response below:\n\n1. Other Tasks: We have not looked at other generation tasks. We wanted to investigate our hypothesis by both experiments that remove the learning signals (through perturbations) as well as experiments that add a learning signal (through target-space informing context). It is quite conceivable that tasks such as cross-lingual summarization would also show similar dynamics but in our work, our scope has been limited to translation and we leave further exploration to future works.\n\n2. Complexity of Perturbations: No, even on perturbations with equal complexity, e.g., Jumbled Source (JS) and Jumbled Target (JT), the impact on the translation quality if very different. In Figures 1, 2, 3, the translation quality is severely impacted by JT but not by JS. So, the results are not due to differences in perturbation complexity. We hope that this answers your question clearly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It's a well written paper with experiments to support the hypothesis\n- The problem of understanding the impact of prompts on the performance is an open question in the field", "reasons_to_reject": "- There are a few open questions [section bellow]", "questions_to_authors": "- This work only focuses on the impact of prompting LLMs for the translation task. However the finding is quite general. Have the authors looked into other tasks how well they can generalize there? \n \n- You find that \"the critical learning signal arrives from the output text distribution\", but the perturbations you applied were more complex on target side. Could that be the reason?", "paper_topic_and_main_contributions": "The paper presents an exploration into the integration of Large Language Models (LLMs) such as GPT-3 for Machine Translation (MT), with a specific focus on in-context learning and the role of demonstration attributes. This work delves into the impact of perturbing high-quality, in-domain demonstrations to facilitate translation improvement.\nThe authors find that while perturbations on the source side exhibit minimal impact, perturbing the target side significantly reduces translation quality. This observation leads to a hypothesis that the output text distribution plays a pivotal role as the primary learning signal during in-context translation learning.\n\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the impact of individual source and target context in GPT-3 based translation. The authors find that the target context matters more than the source context, and accordingly propose a zero-shot prompting approach. While the finding is interesting, this paper can be further improved from several aspects:

1. Clarify the importance of the findings (e.g. response to Reviewer gFZk);
2. Clarity the neural-based metric is the main evaluation assessment;
3. More results on open-sourced Llama2 are appreciated (as promised in the response to Reviewer uUav);
4. Add the necessary content of the response to Reviewer uUav (e.g. answering in 1., 2., B., C. and D) to make the paper more clear and fluent.
Decision: Accept-Findings