Towards Making the Most of ChatGPT for Machine Translation
Authors: keqin Peng
Liang Ding
Qihuang Zhong
Li Shen
Xuebo Liu
Min Zhang
Yuanxin Ouyang
Dacheng Tao
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Translation
Large Language Model
Prompt Engineering

Abstract: ChatGPT shows remarkable capabilities for machine translation (MT). Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g, low-resource and distant-language-pairs translation. However, they usually adopt simple prompts which can not fully elicit the capability of ChatGPT. In this report, we aim to further mine ChatGPT's translation ability by revisiting several aspects: temperature, task information, and domain information, and correspondingly propose two (simple but effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts (DSP). We show that: 1) The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; 2) Emphasizing the task information further improves ChatGPT's performance, particularly in complex MT tasks; 3) Introducing domain information can elicit ChatGPT's generalization ability and improve its performance in the specific domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community. We also explore the effects of advanced in-context learning strategies and find a (negative but interesting) observation: the powerful chain-of-thought prompt leads to word-by-word translation behavior, thus bringing significant translation degradation.

Towards Making the Most of ChatGPT for Machine Translation
Keqin Peng Ri Liang Ding*; Qihuang Zhong', Shen Xuebo Liu', Min Zhang', Yuanxin Ouyang Dacheng Tao Beihang University RJD Explore Academy $Wuhan University The University of Sydney 'Harbin Institute of Technology, Shenzhen keqin. pengebuaa. edu. cn, liangding www.egmail...mmmmmmmmmmm.. com https: //g thub. com Cccm///ommiinngg/hhtttt.....
Abstract
model developed by OpenAl, based on Instruct- GPT (Ouyang al., 2022a), that has been trained to follow instructions with human feedback. Chat- GPT possesses diverse abilities of NLP, including question answering, dialogue generation, code de- bugging, generation evaluation, and so on (Qin et al., 2023; Zhong et al.. 2023; Wang et al., 2023a: Koemi and Federmann, 2023; Lu et al., 2023b; Wang et al., 2023b). We are particularly interested in how well ChatGPT can perform on the machine translation task.
ChatGPT shows remarkable capabilities for ma- chine translation (MT). Several prior studies have shown that it achieves comparable results commercial systems for high-resource lan- guages, but lags behind in complex tasks, c.g., low-resource and sastantlanguaag--aiin trans- lation. However, they usually adopt simple prompts which can not fully elicit the ca- pability of ChatGPT. In this paper, we aim to further mine ChatGPT's translation ability by revisiting several aspects: I temperature, task information, and domain informa- tion, and correspondingly propose optimal temperature setting and two (simple but effec- tive) prompts: Task-Specific Prompts (TSP) and Sommiin-Specifiee Prompts (DSP). We show that: The performance of ChatGPT depends largely temperature. and lower tempera- ture usually can achieve hetter performance; Emphasizing the task information can fur- ther improve ChatGPT's performance, partic- ularly in complex MT tasks; @ Introducing domain information can elicit ChatGPT's gen- eralization ability and improve its performance in the specific domain; O ChatGPT tends to generate hallucinations for non-English--cntricc MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community. We also explore the effects of advanced in-context learning strategies and find a (negative but in- teresting) observation: the powerful chain-of- thought prompt leads to word-by-wond transla- tion behavior. thus bringing significant transla- tion degradation.
Previous studies (Jiao et al., 2023; Hendy et al., 2023) on translation tasks have found that ChatGPT performs competitively with commercial transla- tion products (c.g., Google Translate and Microsoft Translator) on high-resource languages, but has lim- ited capabilities for low-resource and distant lan- guages. However, they only adopt simple prompts and basic settings regardless of the significant in- fluence of the prompts' quality (Zhou et al., 2022), which may limit ChatGPT's performance. In this paper, we aim to further elicit the capability of ChatGPT by revisiting the following three aspects and correspondingly propose an optimal tempera- ture setting and two simple but effective prompts: Specifpe Prompts (TSP) and Domain-Specite Prompts (DSP).
1 Temperature. Temperature is an important parameter to ensure ChatGPT generates varied re- sponses to human queries. Basically, decoding with higher temperatures displays greater linguistic va- riety, while the low one generates grammatically correct and deterministic text (Ippolito et al., 2019). However, for tasks with high degree of certainty, such as machine translation, generation may impede its translation quality. We evaluate the performance of ChatGPT different temperatures to verify its effect and find the optimal temperature setting for the following experiments.
1 Introduction
Recently, the emergence of ChatGPT' has brought remarkable influence on natural language process- ing (NLP) tasks. ChatGPT is a scage-scaleiie language
argue, diverse
Work was done when Kegin was interning JD Explore Academy.
Conresponding Author.
httts::///chat.openee

---

c Task Information. ChatGPT is fine-tuned on high-quality chat datasets and thus essentially a conversational system that has a certain distance from the translation system, we argue that the task inconsistency will limit its translation ability to certain degree. In response to this problem, we proposed Task-Specifie Prompts (TSP) to further emphasize the task information bridge the task gap, i.c., conversation and translation.
Task
Test Set
Direction
Domain
Size
Flores-200
Any
General
1,012
En=>hh
2,001
WMT19 News
News
De=
3,004
En
224
WMT19 Bio
Biomedical
Zh=>En
241
SMMT22E-Commers
En->Zh
E-Commerce
530
Table 1: Data statistics and descriptions.
Domain Information. Compared with tradi- tional machine translation systems, ChatGPT can incorporate additional information, like human in- teractions, through the input prompts (Dong et al., 2023). We argue that such flexible interaction may alleviate some classical MT challenges, e.g., cross- domain generalization (Kochn and Knowles, 2017). We, therefore, propose Domain-Speciffe Prompts (DSP) to introduce the domain navigation infor- mation elicit ChatGPT's generalization ability across different domains. Through extensive experiments, we find that:
word-by-we ford-by-word translation behavior, thus bringing significant translation degradation. Also, we call for improving ICL and CoT for MT upon ChatGPT by incorporating the philosophy of example-hased and statistical MT (Nagao, 1984; Koehn, 2009).
The remainder of this paper is designed as fol- lows. We present the evaluation settings in Sec- tion 2. In Section 3, we revisit the performance of ChatGPT from three aspects (temperature, task, and domain information) and show the zero-shot translation performance of ChatGPT with our pro- posed advanced prompt recipes. Section 4 summa- rizes the few-shot in-context learning and chain-or of- thought results. Section 6 presents conclusions.
ChatGPT's performance largely depends on the temperatures, especially in difficult lan- guages. Generally, setting lower tempera- ture can result higher performance.
Evaluation Setting
We provide brief introduction of the evaluation setting, which mainly includes the used models, test set, and evaluation metrics.
Emphasizing the task information in prompts can further improve ChatGPT's performance, especially in complex tasks.
Models. We mainly compare ChatGPT² with the commercial translation product Google Translator?, which supports translation in 133 languages. By default, the results in this paper come from the gpt- 3. 3.5-turbo-0301 models, which power the ChatGPT.
Introducing the correct domain informa- tion consistently improves ChatGPT's perfor- mance while wrong domain information leads to significant degradation in performance.
Data. For multilingual translation and in-context learning, evaluate the performance of the mod- els the Flores-200 (Goyal al., 2022)* test sets, which consists of 1012 sentences translated into 204 languages. To evaluate the effect cross-domain translation, we adopt the test set of WMT19 Biomedical (Bawden et al., 2019), News Translation Task (Barrault et al., 2019) and WMT22 E-Commerce task (Kocmi et al., 2022). Table lists the statistics of these test sets. We test all samples through OpenAl API.
When tackling the non- oon-English--entric tasks (both the input and expected output English), ChatGPT may generate hallucina- tions, which should be paid more attention to by the MT/NLP community.
Furthermore, we explore the effects of several ad- vanced in-context learning strategies (Brown et al., 2020b). Specifically, we investigate ChatGPT's few-shot in-context learning (ICL.) and chain-of- thought (CoT) (Wei et al., 2022c; Kojima et al., 2022) abilities on MT tasks. Experimental results show that few-shot ICL can further improve Chat- GPT's performance, which is identical the find- ings of Hendy et al. (2023), and we also find a negative but interesting observation: CoT leads to
Metric. The translation metrics shared task (Fre- itag et al., 2022) recommends using neural network-
com/chat cttaccccccaaac.ccmcct
a hhttps://tanslaate wooeee.ooooee..ooooooo len...ccom com
'https://github. wtt.ub..oooaaceeeeeeeeeeeee comm//Facbbokreeerrcc//oors

---

ZH DE
RO
Method
Translation Prompt
ChatGPT
"role": "user", "content"; "Please provide the [TGT] translation for the following sentence:"
82.6
822
82
904
ChaiGPT+
"role" "system", "content" "You are a machine translation systen.", ",
"role": "user", "content": "Please provide the [TGT] translation for the following sentence:"
COMET
69.5
89.0
89.1
98.0
Table 2: Multilingual translation prompts.
67
66.4
03.9
86
based metrics since they have demonstrated a high correlation with human evaluation and are resilient domain shift. Hence, we adopt the mostly used COMET (Rei et al., 2020) as our primary metric and use the default parameters of "comet- compare" for significance test". Specifically, we use the reference- based metric COMET-20 (wmr20- also report BLEU
82-4
Figure 1: The relationship between temperature and ChatGPT's performance (in terms of COMET scores) when translating from English to other languages.
COMET-da). Additionally, scores (Papineni et al., 2002) and ChrF (Popović, 2015) using SacreBLEU (Post, 2018) for com- pleteness, but notably, we mainly analyze the per- formance terms of model-based metric COMET.
ZH
DE
RO
47
45
44.8
45
43.6
43
3 Zero-Shot Translation
BLEU
40.6
40.6
38.9
41.4
392
In this section, explore the performance of Chat- GPT from three aspects: TEMPERATURE, TASK INFORMATION, and DOMAIN INFORMATION, and correspondingly propose an optimal temperature setting and two simple and effective prompts to improve ChatGPT's performance.
39
404
363
38.3
37
35
0.2
0.4
0.8
T
Figure 2: The relationship between temperature and ChatGPT's performance (in terms of BLEU scores) when translating from English to other languages.
3.1 The Effect of Temperature B
ChatGPT is a chatting machine designed to provide fluent and diverse responses to a wide range of human requests. is intuitive that the diversity of responses may hinder its performance on tasks with high degree of certainty, such machine translation, to some extent.
clear degradation both in COMET and BLEU scores. Furthermore, it is noteworthy that Chat- GPT's sensitivity to the temperature varies depend- ing on the language pair: the impact of temperature relatively small when translating to high-resoune languages, e.g., German, while for complex lan- guages, e.g., Chinese, it has a large degradation in performance (-1.3 COEMT points and 3.7 BLEU points for Chinese) when the temperature changes from 0 to 1. We speculate that the huge re- source variance in training data leads differences in the confidence of languages, which partially ex- plains the different performances. In the following experiments, we adopt
To investigate the influence of diversity, compare the performance of ChatGPT in differ- ent temperature settings, including 0, 0.2, 0.4, 0.6, 0.8, and 1, across three translation direc- tions: Rooganian English~> Chinese, and Germman The relationship between tem- perature and performance of ChatGPT is shown in Figure and 2.
Results. Figure and show that ChatGPT's performance largely depends on the value tem- peratures, and as the temperature rises, there is
0 as our default set- ting to make the most of ChatGPT and ensure the stability generation to avoid a result of noise.

---

System
COMET
BLEU
ChrF
COMET
BLEU
ChrF
DE=EN
EN=DE
Google Translator
77.7
47.4
70.5
70.5
44.4
68.9
ChatGPT
77.2
43.5
69.4
69.3
40.6
67.1
ChatGPT + TSP
77.51
44.1
69.7
69.4
40.4
67.0
EN
EN=ZH
Google Translator
73.5
33.5
61.2
68.5
48.8
43.8
ChaiGPT
71.3
26.4
58.3
66.4
45.1
39.0
ChatGPT + TSP
71.3
26.7
58.4
67.21
45.3
39.3
RO=EN
EN-RO
Google Translator
82.4
48.0
71.2
91.6
43.3
67.0
ChalGPT
80.6
41.8
68.8
92.4
40.6
65.5
ChatGPT + TSP
80.8
41.9
69.0
92.9
40.8
65.7
ZH=RO
RO=ZH
Google Translator
73.9
25.8
53.9
62.3
42.3
37.8
ChatGPT
73.8
20.9
51.5
58.9
37.7
33.3
ChatGPT + TSP
74.1*
21.0
51.3
59.1
38.0
33.7
Table 3: Performance with different prompts on language pairs from Flores-200. "TSP" denotes our proposed task-specific prompting method. The best scores across different systems marked in bold and the best scores of ChatGPT are underlined. Notably, we set the temperature as for ChatGPT in this experiment. We can see that our TSP method consistently boosts the performance of ChatGPT in most settings. Shadowed areas mean difficult English-cene tramslation tasks, Green areas mean non English-centric translation tasks. "f" indicates a statistically significant difference from the ChatGPT baseline (p 0.05).
3.2 The Effect of Task Information
RO ZH
Previous studies (Jiao et al., 2023; Hendy et al., 2023) have shown that ChatGPT can achieve ex- ceptional performance in conversational domain translation, which is attributed to its ability to gen- erate more natural and diverse spoken language. However, given that ChatGPT deliberately de- signed as general task solver (Qin et al., 2023), when asking the ChatGPT to perform as a specific task engine, there will arise a task gap. This task inconsistency may limit ChatGPT's effectiveness in translation tasks other than the spoken domain.
60
20
ChaIGPT(T-1)
CheiGPT(T-0)
ChaiGPT(T-0) TSP
Figure 3: Number of Post-Edited sentences in non- English-centric language pairs, where a higher value means the translation contains hallucinations. RO represents the translation for ZH= while ZH repre- sents the translation for ZO-RO.
To bridge the task gap and generate more translation-like sentences, we propose Task- Specific Prompts (TSP) to emphasize the transla- tion task information. Specifically, prepend the sentence "You are a machine translation system." to the best translation template in Jiao al. (2023), and adopt it to query ChatGPT. The templates of prompts present in Table 2, and [TGT] represents the target languages of translation.
models on four language pairs, covering eight dis- tinct translation directions. These languages com- prise 1) German, which is of the most English languages in the GPT training data, 2) Ro- manian, less frequently encountered non-English language the GPT training data, and 3) Chinese, a large-scale language with a script distinct from
We have compared the performance of various

---

English. We also adopt Chinese-Romanial as non-English-centre use case. Table 3 lists the full results, where we list both English-centric and non- English-centric language directions (marked with green ), and also, among English-centric direc- tions, we highlight the difficult pairs (EN-ZH and EN-RO with shadow in terms of their resources and language distance.
Target Language
Template
Chinese
[Ro] would be translated to: [2h];
[2h] (Note: _
Romanian
[2h] can translated Into Ronanian
as [Ro]; [Ro] (Note: ....)
Table 4: Some templates about irrelevant information in generated sentences for Chinesc Romeniian.. Semi- colon is used to separate different templates. [Ro] repre- sents the sentence in Romanian while [Zh] represents that in Chinese.
3.2.1 English-Centric Language Pairs
We first consider the performance of Chat- GPT in English-centric translation language pairs. Specifically, we conduct experiments in three language pairs: German+>English (high- resource), Romanian< (low-resource), and Chinesc+Engglish (distant language).
formation obeyed some patterns followed the trans- lation, such as "Translation may vary depending on context", which will greatly affect the MT per- formance. We used post-processing method to remove irrelevant information from the generated text. Specifically, we summarize some templates about irrelevant sentences and remove them from the generation texts. Some templates are shown in Table 4 and the number post- pross-prcessedd sentences is presented in Figure 3.
Results. Our results presented in Table show that our TSP method achieves comparable results on COMET score compared to Google Translator and even outperforms it in some language pairs, e.g., Beglish-RRmmaiiann (92.9 v.8. 91.6). We also observe that our TSP method consistently im- proves the performance of vanilla ChatGPT, espe- cially when translating to low-resource or distant languages. Specifically,, our TSP method brings +0.8 and +0.5 COMET score improvements in re-
Results. Figure shows that lower temperature can reduce the number of hallucinations (especially in distant languages, e.g., Chinese) and our TSP method can further reduce its number, which sug- gests that our method can help ChatGPT to better serve as machine translation system. The full results Romanian=CChines lists are in Table 3. As seen, TSP method can only slightly improve ChatGPT's performance, which could be due to the difficulty both understanding and generating the language pairs. Mcanwhile, our used post-editing approach could only roughly remove the halluci- nation patterns, the NLP/MT community should pay more attention to the potential hallucination when using ChatGPT to tackle the non-English text.
Chinese and English=Romane spectively, and +0.2 on average when translating to English. We speculate that the high- righures training data can help the model better understand the specific task from a few task-related naviga- tions, thereby reducing the need for additional task- specific information. Although our proposed TSP consistently improves the performance in terms of semantic metric, i.e., COMTE, notably, we have not consistently bridged the task gap terms of lexical metrics (BLEU and ChrF), which consis- tent with similar findings from Vilar et al. (2022) on PALM-540B model.
The subsequent experiments will use ChatGPT with TSP as the default setting.
3.2.2 To--English-Centrii Language Pairs
3.3 The Effect of Domain Information
We also evaluate the performance of ChatGPT in con-EEglish--enee language pairs (since the pre- training process was dominated by the English tokens and the multilingual MT community ar- gues it may harm the non-Ennlish-centriic per- formance (Costa-jussà et al., 2022; Zan et al., 2022a, 2023).). We have an important finding that, when tackling non-Fnglish-centriic MT lan- guage pairs, ChatGPT tends to generate trans- lation hallucinations, that is, some unrelated in-
Compared with traditional machine translation sys- tems, ChatGPT can incorporate additional infor- mation through the prompts to further improve its performance. While previous studies have shown that ChatGPT has great robust translation capabil- ities (Hendy al., 2023), we believe that further enhance its performance by incorporating domain-specific guidance.
To this end, we propose Domain SSeccifc Prompts (DSP) that identify the domain informa-

---

Method
Translation Prompt
that our method can further improve the generaliza- tion ability of ChatGPT and narrow the gap with one of the most advanced commercial systems Google Translator. Nonetheless, our method's im- pact on BLEU is inconsistent, and still lags sig- nificantly behind Google Translator's performance. To verify that the observed improvement is in- deed due to the introduction of the domain infor- mation, we deliberately provided incorrect domain information for each sentence, namely F-DSP, attack the improvement brought by the DSP strat- egy. Specifically, We exchange domain informa- tion for the biomedical sentences and the news sen- tences, We expect that the wrong domain guidance (F-DSP) will under-perform the DSP, and even per- form worse than the vanilla ChatGPT. The results of these experiments shown in the last row of Ta- ble which clearly shows consistent degradation in COMET, proving that the domain information is the key to the success our method.
ChatGPT
"role": "system", "content":
machine translation system.
"role": "user", "content": 'Please
provide the [TGT] translation for the following sentence:
ChaiGPT-DSP
"role": "systen", "content": "You machine translation systen that translates sentences in the [DOM] domain.", "role": "user", "content"; 'Please pravide the [TGT] translation for the follewing sentence:
ChatGPT+F-DSP 'role": "systen", "content": "You machine translation systen that translates sentences in the [FDOM] domain.", "role": "user", "content": 'Please provide the [TGT] translation for the following sentence:
Table 5: Dpmain-Specific translation prompts. "[DOM]" and "[FDOM]" denote the correct and incorrect domain instructions, respectively.
tion of translated sentences in prompts to facili- tate ChatGPT's generalization. Specifically, we ask ChatGPT with the following prompts "You are a machine translation system that translates sen- tences in the [DOM] domain", as shown in Ta- ble 5. Here, [DOM] represents the correct domain of the translated sentence, while [FDOM] represents the wrong domain of that, which is used to verify whether the improvement comes from domain in- formation. For example, for a biomedical sentence, [DOM] is biomedical, while [FDOM] can be any field except biomedical.
All the above DSP and F-DSP results confirm the importance of domain-specific prompting guidance in using ChatGPT for MT tasks.
4 Few-shot Machine Translation
In this section, we simply explore the effeets of ad- vanced in-context learning (ICL) strategies, specifi- cally, we investigate ChatGPT's few-shot ICL and Chain-or of Tafohhh (CoT) abilities on MT tasks.
4.1 Few-Shot In-Context learning
We evaluate our method on the WMT19 Bio and News datasets followed Jiao et al. (2023), which allows us to examine domain bias's impact. For example, the WMT19 Bio test set comprises Med- line abstracts that require domain-specific knowl- edge, while the WMT19 News dataset features news-style texts that are significantly different from dialogues. To further prove the effectiveness of our method, we conduct our method on WMT22 English-Chin Chinese E-Commerce test set, which is less likely to overlap with the GPT training data.
In-context learning (Brown al., 2020b) has shown its remarkable ability for many NLP tasks (Liu et al., 2023). To further explore the ca- pabilities of the ChatGPT, we conduct experiments with different sample selection strategies. Specif- ically, we evaluate the performance of few-shot machine translation in the following three direc- tions: English=Chinese English~ and English=German in Flores-200. We conducted experiments with randomly and TopK (Liu et al., 2022) sampled demonstrations from development sets the 1-shot and 3-shot settings.
Results. The results are listed in Table 6. Ob- viously, the original ChatGPT does not perform well as Google Translator in both COMET and lexical metrics (c.g., BLEU). However, our DSP method can consistently improve the performance of ChatGPT in terms of COMET score and even outperforms Google Translator in two datasets (WMT19 Bio Chinese > English and WMT19 News English + Chinese). This finding indicates
Results. Our results are listed in Table 7. As seen, in-context learning with random examples consistently improves the performance in both lexi- cal metric (BLEU) and COMET score compared the zero-shot approach, and increasing the number of shots can lead to further improvement, which is consistent with previous finding (Hendy et al., 2023). The advanced salple-selection strategy like

---

WMT19 Bio
WMT19 News
WMT22 E-Commerce
System
EN EN->ZH
ZH HH
EN-
EN->DE
EN-ZH
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
Google Translator
59.4
38.8
59.4
36.1
59.3
43.4
64.1
33.7
71.7
48.0
ChatGPT
58.6
35.5
58.7
31.1
58.8
39.6
63.1
31.3
68.2
43.5
ChaGGP + DSP
58.9
35.8
59.61
31.3
59.6
39.8
63.2
31.5
68.6
43.8
CharGPT F-DSP
58.6
35.6
58.4
31.3
57.9
39.0
62.0
31.2
67.1
43.3
Table 6: Performance of ChatGPT translation robustness, i.e., different domains. "DSP" denotes our proposed domain-specife prompting method, while "F-DSP" denotes the false domain-specific prompting, i.c., we specify wrong/unrelated domain information
prompt. results in green denote that "DSP" improves ChatGPT by clear margin (0.5 (1) score), while the red results denote the significant performance drops caused by "F-DSP". indicates a statistically significant difference from the ChatGPT baseline (p < 0.05).
EN = DE
EN
ZH
EN = RO
System
COMET
BLEU
COMET
BLEU
COMET
BLEU
Google Translator
70.5
44.4
68.5
48.8
91.6
43.3
ChatGPT
69.4
40.4
67.2
45.3
92.9
40.8
Random Sampling few- shot prompting
-w/ 1-shot
69.8
40.6
67.6
45.4
93.1
40.7
-w/ 3-shot
70.0
40.7
68.3
45.9
93.6
40.9
TopK Sampling feetiingfew-ssee few-shot prompting
-w/ 1-shot
69.8
41.0
68.4
45.8
93.1
40.7
-w/ 3-shot
70.5
40.9
68.8
45.8
94.0
41.2
Table 7: Few-shot translation performance of ChatGPT on Flores-200. In the random sampling few-shot prompting setting, we randomly sample 1/3 examples from the development set with 3 runs. The best scores across different systems are marked in bold and the best scores of ChatGPT are underlined.
TopK, which chooses test-sample similar examples as demonstrations, can further improve the perfor- mance, even outperform Google Translator in some language pairs, e.g., English= Romanian (94.0 v.s. 91.6) and English=Chinese (68.8 v.s. 68.5).
but its inffuence on machine translation tasks has hardly been investigated.
To investigate this further, randomly select and adopt the zero-shot
20 samples from the CoT technique (Kojima et al., 2022) and the 1-shot CoT technique. Specifically, as shown in Table 8, for zero-shot CoT, we use the prompt "Please provide the [TGT] translation for the following sentence step by step" to extract step-by-step translation. We also add the sentence 'and then provide the complete sentence:' to the end of the prompting to ensure that ChatGPT can generate the complete translation. While for the 1-shot CoT, we provide the manual interme- diate reasoning steps inspired by zero-shot CoT, as shown in Table 8. Here, [S] and [T] represent the corresponding source and target sentence in the demonstration, respectively, and [S_i] and [T are the i-th matching tokens in the source and target sentence.
We encouragingly find that the advanced sample- selection strategy for in-context learning for MT tasks upon ChatGPT is extremely similar to the de- sign philosophy of example-based machine trans- lation (EBMT, Nagao, 1984), where the EBMT is often characterized by its use of bilingual corpus as its main knowledge base, at run-time. It is wor- thy of designing better ICL strategies inspired by EBMT in future work.
4.2 Chain-of-Thought
Chain-of-Thougee (CoT) prompting (Wei et al., 2022c) has been demonstrated be effective in eliciting the reasoning ability of large language models. Previous studies have shown that CoT can improve the ChatGPT's performance in natural language understanding tasks (Zhong et al., 2023),
Results. We conduct experiments the follow- ing two translation directions: English~>German

---

Method
Translation Prompt
tion (Kaplan et al., 1989).
Zero-Shot CoT "role": "systen", "content": "You are a machine translation system.", "role": "user", "content": 'Please provide the German translation for the following sentence step by step and then provide the complete sentence:
Related Work
Large Language Models. Large language mod- els (LLMs) usually refer to language models with hundreds of billions of parameters, which are trained on massive text data (Zhao et al., 2023). LLMs usually can be classified into three groups based on model architectures: 1) encoder-only LLMs (Devlin et al., 2019; Liu et al., 2019; Zhong et al., 2022), usually used for NLU tasks; 2)
1-Shot CoT
"role": "systen", "content": "You are a machine translation system.", "role"; "user", "content"; "Please provide the German translation for the fol following sentence step by step and then provide the complete sentence: [S] 1. [$_1] [T_1]
[S_2]
[T_2] ... n. [S_n] [T_n] The
decoder-only LLMs (Radford et al., 2019; Brown et al., 2020a), more suitable for NLG tasks; and 3) encoder-decoder LLMs (Raffel et al., 2020; Lewis et al., 2020; Zan et al., 2022b; Peng et al., 2023), which can achieve better performance on condi- tional text generation tasks.
complete sentence in [TGT] is: [T] Please provide the German translation for the following sentence step by step and then provide the complete sentence:'
Table 8: The templates of Zero-Shot CoT and I-shot COT. [S_n] represents the n-th token in source demon- stration [S], [T_n] represents the n-th token in target demonstration [T].
Traditionally, these PLMs can achieve remark- able performance in various natural language pro- cessing (NLP) tasks through fine-tuning on specific tasks. But with the scaling up and the develop- ment of LLMs (Brown et al., 2020a: Ouyang et al.. 2022b), decoder-only LLMs exhibit remarkable zero-shot and few-shot abilities, denoted emergent abilities (Wei et al., 2022b), and achieve compa- rable results with other LLMs in NLU and con- ditional NLG tasks. Especially the emergency of ChatGPT, developed by OpenAl, takes LLMs a big step forward in both academia and industry. ChatGPT possesses diverse abilities of NLP and can generate luman-like responses by instruction- tuning (Wei et al., 2022a) and Reinforcement Learning from Human Feedback (RLHF) tech- nique (Ouyang et al., 2022b).
EX-DE
EN-ZH
Method
COMET
BLEU
COMET
BLEU
ChaiGPT
72.4
36.5
68.3
41.4
-W 2ato-shet Coll"
(U3.1
35.1
()14))
39,5 (U88)
362 (15.5
-w I-shhoot CoT
69.6 (12.8)
(105)
61.1 (17.2)
37.6 (13.8)
Table 9: Performance ChatGPT equipped with CoT prompting methods randomly selected 20 samples from English= German and English=CCinnse.
and English~ The results are listed in Ta- ble 9, which shows that there is a significant degra- dation in COMET score with zero-shot CoT setting, especially in English=CCinese, which drops 8.8 COMET points. 1-shot CoT prompting can con- sistently outperform zero-shot CoT but still lags behind zero-shot prompting COMET.
ChatGPT for Machine Translation. The abil- ity of ChatGPT has been widely studied in vari- ous domains (Qin et al., 2023; Zhong et al., 2023), but its ability machine translation tasks has not been fully investigated. Jiao al. (2023) and Hendy al. (2023) first provided an evaluation on the performance of ChatGPT for machine transla- tion, they found that ChatGPT can perform com- petitively with commercial translation products on high-resource European languages but lags behind significantly on low resource or distant languages. However, they usually adopt simple prompts and basic settings which cannot fully exploit the capa- bilities of ChatGPT, we first proposed that Chat- GPT can achieve comparable results with proper settings and investigate how to make the most of GPT for machine translation.
We looked in detail at the sentences generated by different prompts. presented in Table 10, and we have a negative but interesting observation: the CoT prompt leads to word-by-word translation behavior, which is the main reason for the signifi- cant translation degradation.
For more CoT variants designed with different principles inspired by the philosophy in statistical MT (Zens et al., 2002; Kochn, 2009) will be ex- plored in the future. For example, word-by-wort and then reordering the translation (Du and Way, (Feng
2017: Ding et al., 2020), phrase-to-phre et al., 2018; Ding et al., 2021) and then reordering the translation, and structure-to- struuuue transla-

---

Subsequent work follows our work further explore the performance of ChatGPT, Gao al. (2023) and Lu et al. (2023a) introduce new infor- mation (e.g., POS or multilingual dictionaries), He et al. (2023) proposed a CoT-like framework to generation human-like translation.
Ethics Statement
We take ethical considerations very seriously and strictly adhere to the EMNLP Ethics Policy. This paper focuses exploring the translation ability of ChatGPT open-sourced machine translation datasets, not involving any ethics problem. Both the compared models and evaluation datasets used in this paper are publicly available and have been widely adopted by researchers. Therefore, we be- lieve that this research will not pose ethical issues.
6 Conclusion
In this paper, we investigate how to further mine ChatGPT's translation ability from three perspec- tives, namely temperature, task, and domain in- formation, and correspondingly propose an opti- mal temperature setting and two simple but effec- tive prompts. We empirically demonstrated that there a high correlation between temperature and ChatGPT's performance, and a lower temperature usually can achieve better performance. Experi- mental results across various language pairs and domains proved the effectiveness of our proposed prompts. We further explore the effectiveness of advanced in-context learning strategies for Chat- GPT, we find that the few-shot in-context learning method can consistently improve ChatGPT's per- formance, while conventional Chain-of -hoiugof-Thhughh (CoT) prompting will degrade its performance be- cause of its word-by-word translation behavior.
References
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussà, et al. 2019. Findings of the 2019 conference on machine translation (9MT19). In WMT.
Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, et al. 2019. Findings of the WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies. In WMT.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. NewrIPS.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, al. 2020b. Language mod- are few-shot learners. NeurIPS.
In future work, besides the aforementioned ex- plorations (EBM'T-inspired prompts designing, sta- tistical MT-inspired tfaiinof-tthughh designing), we would like to investigate how to further elicit the ability of ChatGPT by designing more effective prompts (e.g., design human-like CoT to navigate the LLMs, and better demonstration selection algo- rithms in few-shot ICL.) and investigate the ability of ChatGPT for more MT settings (e.g., document translation).
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human- centered machine translation. arXiv preprint.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- Pre-training of deep bidirectional transformers for language under- standing. In NAACL.
Limitations
Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Progres- sive multi-granularity training for non-autoregressive translation. In Findings of ACL.
Our work has several potential limitations. First, we only propose some simple prompts that have not been carefully designed to investigate the ca- pabilities of ChatGPT, which may not sufficiently elicit the power of ChatGPT. Second, we have not fully studied the performance of ChatGPT in few- shot scenarios, especially the effect of Chain-Of- Thought in machine translation. In future work, we would like design different types of prompts to further improve ChatGPT's performance ma- chine translation and conduct more in in-depth analy- ses and discussions.
Liang Ding, Longyue Wang, and Dacheng Tao. 2020. Self-attention with cross-lingual position representa- tion. ACI.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning. arXiv preprint.
Jinhua Du and Andy Way. 2017. Pre-reordering for neu- machine translation: Helpful or harmful? Prugue Bulletin of Mathematical Linguistics.

---

Jiangtao Feng, Lingpeng Kong, Po-Sen Huang. Chong Wang, Da Huang. Jiayuan Mao, Kan Qiao, and Dengyong Zhou. 2018. Neural phrase-to-phrase ma- chine translation. arXiv preprint.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequencc-to-ssqqueee pre-training for natural language generation, translation, and com- prehension. In ACL.
Markus Freitag, Ricardo Rei, Nitika Mathur, et al. 2022. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In WMT.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In DeeLIO.
Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: empirical study. arXiv e-prints.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, et al. 2022. The flores-101 101 evaluation henchmark for low-resource and multilingual ma- chine translation. TACI.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint.
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang. Zhaopeng Tu, Shum- ing Shi, and Xing Wang. 2023. Exploring human- like translation strategy with large language models. arXiv preprint.
Hongyuan La, Haoyang Huang, Dongdong Zhang, Hao- ran Yang. Wai Lam, and Furu Wei. 2023a. Chain- of-dictionary prompting elicits translation in large language models. arXiv preprint.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, et al. 2023, How good are gpt models at machine transla- tion? a comprehensive evaluation. arXiv preprint.
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch 2019. Compar- ison of diverse decoding methods from conditional language models. In ACL.
Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023b. Error analysis prompting ables human like translation evaluation in large lan- guage models: A case study on chatgpt. arXiv preprint.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? preliminary study. arXiv preprint.
Makoto Nagao. 1984. A framework of a mechanical translation between japanese and english by analogy principle. Artificial and human intelligence.
Ronald M Kaplan, Klaus Netter, Jurgen Wedekind, and Annie Zaenen. 1989. Translation by structural corre- spondences. In EACL.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, et al. 2022a. Training language models to follow instructions with human feedback. arXiv preprint.
Tom Koemi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In WMT.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022b. Training language models to follow instruc- tions with human feedback. NeurlPS.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In ACL.
Tom Koemi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint.
Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, and Dacheng Tao. 2023. Token-level self-evolution training for sequence-to-scqqence learning. In ACL.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In WMT.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In WMT.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NewrlPS.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT.

---

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint.
Richard Zens, Franz Josef Och, and Hermann Ney. 2002. Phrase- based statistical machine translation. In KI.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjic Zhang, Zican Dong, al. 2023. A survey of large language models. arXiv preprint.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models unsupervised multitask learners. OpenAl blog.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? comparative study on chatgpt and fine-tuned bert. arXiv preprint.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. JMLR.
Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, al. 2022. Toward efficient lan- guage model pretraining and downstream adaptation via self-evolution: A case study superglue. arXiv preprint.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance. arXiv preprint.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster. Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-leve prompt engineers. arXiv preprint.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint.
Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. 2023b. Re- cursively summarizing enables long-term dialogue memory large language models. arXiv preprint.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. ICLR.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022b. Emergent abilities of large language models. arXiv preprint.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022c. Chain thought prompting elicits reasoning large language models. arXiv preprint.
Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022a. Bridging cross-lingual gaps during leveraging the multilingual sequence- to-sequence pretraining for text generation. arXiv preprint.
Changtong Zan, Liang Ding. Li Shen, Yibin Lei, Yibing Zhan, Weifeng Liu, and Dacheng Tao. 2023. Un- likelihood tuning on negative samples amazingly im- proves zero-shot translation. arXiv preprint.
Changtong Zan, Keqin Peng, Liang Ding, Baopu Qiu. Boan Liu, Shwai He, Qingyu Lu, Zheng Zhang, Chuang Liu, Weifeng Liu, Yibing Zhan, and Dacheng Tao. 2022b. Vega-MT: The JD explore academy ma- chine translation system for WMT22, In WMT.

---

Zero-shot prompting
System: You are a machine translation system.
User: Please provide the German translation for the following sentence: Sadly, as newer methods of writing have emerged, the pencil has been relegated to lesser status and uses.
Response: Leider wurde der Bleistift durch neuere Schreibmethoden eine geringere Position und Verwendung gedrängt.
Zemo-shot CoT prompting
System: You are a machine translation system.
User: Please provide the German translation for the following sentence step by step and then
provide the complete sentence: Sadly, as newer methods of writing have emerged, the pencil has been relegated lesser status and uses.
Response: 1. Sadly Bedauerlicherweise 2. as newer methods of writing have emerged da neuere Schreibmethoden aufgetaucht sind 3. the pencil der Bleistift 4. has been relegated wurde zurüickesstuft 5, to lesser status and uses geringerem Status und Verwendungszweckenn Bedauerlicherweise wurde der Bleistift aufgrund neuerer Schreibmethoden geringerem Status und Verwendungszweeken zurückgestuff.
I-shot CoT prompting
System: You are machine translation system.
Please provide the German translation for the following sentence step by step and then provide the complete sentence: That said, expect to be out of breath, and take care in the steeper portions, especially when wet, as it can become dangerous quickly. 1. That said jedoch damit 2. expect Rechnen sie 3. to be zu sein 4. out of breath außer Atem 5. and und 6. take care vorsichtig 7. in the steeper portions in den steileren Abschnitten 8. especially besonders 9, when wet wenn nass ist 10. become werden 11. dangerous gefährlich 12. quickly schnell The complete sentence in German is: Rechnen Sie jedoch damit, außer Atem sein, und seien Sie in den steileren Abschnitten vorsichtig, besonders wenn nass ist, da dort schnell gefährlich werden kann. Please provide the German translation for the following sentence step by step and then provide the complete sentence: Sadly, as newer methods of writing have emerged, the pencil has been relegated to lesser status and uses.
User:
Response: 1. Sadly Leider 2. as da 3. newer methods neuere Methoden 4. of writing des Schreibens 5. have emerged aufgetaucht sind 6. the pencil der Bleistift 7. has been relegated wurde zurückgestuft to lesser status zu einem geringeren Status 9, and uses und Verwendungen. The complete sentence in German is: Leider wurde der Bleistift aufgrund des Auftauchens neuerer Methoden des Schreibens zu einem geringeren Status und Verwendungen zurückgestuff.
Table 10: Examples of zero-shot and I-shot CoT prompting on English The sentences red are the reasoning step for CoT and those in blue are the one-shot example. Sentences in bold are the instruction of CoT.


Reviews:
[{"review_id": "LTuBMtC1Aq", "reviewer": "Reviewer_gZrL", "summary": "", "questions": "1. Using WMT19 for evaluation raises concerns because ChatGPT might use WMT19 dataset for training, making the conclusion biased. Besides, the biomedical test set is small.\n2. The authors conclude that \"TSP benefits translation\" but without whether task-general system prompt works, like \"you are a professional system assistant and can help users with different tasks\". \n3. There are no significance tests for nearly all results in this paper. Considering that most improvements are modest, the gain might be a result of noise.\n4. In line 384, the authors claim studying \"different sample selection strategies\" but they actually only experimented with random sampling. Did you also work with other selection methods? How many runs did you perform with random selection? \n5. What does the \"advanced sampling selection strategies\" in line 403 mean?\n\nWhile ChatGPT gains popularity, using it for scientific research should be more careful and rigorous. This is more concerned when only reporting its downstream performance. ChatGPT is a product. We all have no idea what data has been used for its training. Stating \"Zero-Shot\" as in Section 3 is not well justified.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, which will serve to improve our paper considerably. We will attend to all comments to the best extent, and try to address your concerns as follows:\n\n**Q1: “WMT19 makes the conclusion biased ... because ChatGPT might use dataset for training and biomedical test set is small”**\n\nThanks for pointing this out, we have to explain that the aim of this study is to maximize the effectiveness of ChatGPT when functioning as a machine translator, therefore, we follow Jiao et al. (2023) (who is the first to show the possibility of translating with ChatGPT) to verify our proposed DSP on the WMT19 dataset.  \n\nAlthough it is unclear what data has been used for ChatGPT training, we provide two pieces of evidence to dispel your concern about the convincing of our conclusion:  \n1) Table 6 shows that DSP can improve ChatGPT’s translation performance while FDSP degrades the translation performance, demonstrating the effectiveness of DSP;  \n2) We conduct experiments on a larger WMT22 English-Chinese e-commerce test set, which is less likely to overlap with the GPT training data, and report the results below. The extra experiments on the updated test data confirm the effectiveness of our proposed DSP.\n\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  ChatGPT  |  43.5  |  68.2 |\n|  +DSP  |  **43.8**  |  **68.6** |\n|  +FDSP  |  43.3  |  67.1 |\n\n[1] Jiao et al., Is ChatGPT a good translator? a preliminary study. In arXiv preprint 2023.  \n\n**Q2: “without .. task-general system prompt”**\n\nGiven that ChatGPT is deliberately designed as a general task solver [2,3], we argue that ChatGPT is already the task-general system even without a task-general system prompt. To address your concern, we followed your suggestion and added the task-general system prompt “You are a professional system assistant.” in Flores-200 English-Chinese. The results below show that although ``+general prompt’’ slightly improves the COMET score, our TSP is still the best among them, showing the effectiveness of our method. \n\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  ChatGPT  |  45.1  |  66.4 |\n|  +general prompt  |  45.1  |  66.8 |\n|  +TSP  |  **45.3**  |  **67.2** |\n\n[2] Qin et al., Is ChatGPT a General-Purpose Natural Language Processing Task Solver? In arXiv preprint 2023.\n\n[3] OpenAI, GPT-4 Technical Report. In OpenAI blog 2023.\n\n**Q3: “no significance tests … gain might be a result of noise”**\n\nWe followed your suggestion and conducted the significance test (p<0.05) with comet-compare, and found that 1) TSP has a significant improvement over baseline on 5/8 tasks, and 2) DSP has a significant improvement over baseline on 2/4 tasks, illustrating the consistent effectiveness of our method. We will add the above significance tests to the revised version. \n\nAs for the ``noise’’ concern, due to the zero temperature setting, all the better translation performance can be easily reproducible.\n\n**Q4: “Results of different sample selection strategies”**\n\nThanks for pointing this out, in our current few-shot in-context learning (ICL) experiment (Table 7), we only report the results of the random sampling strategy in 1 run. \n\nTo make the experiments more convincing, we adopted TopK strategies [4] and a random sampling strategy with 3 runs following the tasks in Table 7. The results are shown below. We can notice that 1) few-shot settings can consistently improve ChatGPT’s performance, and 2) the use of advanced few-shot sampling strategies, e.g., TopK, can further enhance its performance.\n\nFor EN-> ZH:\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  ChatGPT  |  45.3  |  67.2 |\n| *Random Sampling*  |\n|  -w/ 1-shot  |  45.4  |  67.6 |\n|  -w/ 3-shot  |  **45.9**  |  68.3 |\n| *TopK*  |\n|  -w/ 1-shot  |  45.8  |  68.4 |\n|  -w/ 3-shot  |  45.8  |  **68.8** |\n\nFor EN-> DE:\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  ChatGPT  |  40.4  |  69.4 |\n| *Random Sampling*  |\n|  -w/ 1-shot  |  40.6  |  69.8 |\n|  -w/ 3-shot  |  40.7  |  70.0 |\n| *TopK*  |\n|  -w/ 1-shot  |  **41.0**  |  69.8 |\n|  -w/ 3-shot  |  40.9  |  **70.5** |\n\nFor EN-> RO:\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  ChatGPT  |  40.8  |  92.9 |\n| *Random Sampling*  |\n|  -w/ 1-shot  |  40.7  |  93.1 |\n|  -w/ 3-shot  |  40.9  |  93.6 |\n| *TopK*  |\n|  -w/ 1-shot  |  40.7  |  93.1 |\n|  -w/ 3-shot  |  **41.2**  |  **94.0** |\n\n[4] Liu et al., What makes good in-context examples for GPT-3? In DeeLIO 2022.\n\n**Q5: \"Zero-Shot\" as in Section 3**\n\nIn section 3, the \"zero-shot’’ corresponds to the ``few-shot’’ in the in-context learning (ICL) scenario.\n \n**Q6: Missing References**\n\nThanks for your advice, we promise to add these references in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper extensively experiments on prompting for ChatGPT for translation, which offers insights into the translation capability of ChatGPT.\n", "reasons_to_reject": "* Experimental setup has problems.\n* Some conclusions need further evidence.", "questions_to_authors": "1. Using WMT19 for evaluation raises concerns because ChatGPT might use WMT19 dataset for training, making the conclusion biased. Besides, the biomedical test set is small.\n2. The authors conclude that \"TSP benefits translation\" but without whether task-general system prompt works, like \"you are a professional system assistant and can help users with different tasks\". \n3. There are no significance tests for nearly all results in this paper. Considering that most improvements are modest, the gain might be a result of noise.\n4. In line 384, the authors claim studying \"different sample selection strategies\" but they actually only experimented with random sampling. Did you also work with other selection methods? How many runs did you perform with random selection? \n5. What does the \"advanced sampling selection strategies\" in line 403 mean?\n\nWhile ChatGPT gains popularity, using it for scientific research should be more careful and rigorous. This is more concerned when only reporting its downstream performance. ChatGPT is a product. We all have no idea what data has been used for its training. Stating \"Zero-Shot\" as in Section 3 is not well justified.", "paper_topic_and_main_contributions": "This paper examines different prompting strategies for ChatGPT for machine translation tasks. The authors studied the impact of decoding temperature, task/domain-specific information, and few-shot and chain-of-thought prompting on translation performance with Flores and WMT benchmarks. The results offered some insights: 1) lower temperature benefits translation; 2) adding task and domain-specific information is helpful; 3) few-shot prompting improves translation but chain-of-thought doesn't work well.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "That using more few-shot examples doesn't necessarily improve performance has been observed and discussed in the following papers:\n[1] PaLM: Scaling Language Modeling with Pathways\n[2] Prompting Large Language Model for Machine Translation: A Case Study"}}, {"review_id": "S6QGWk9U46", "reviewer": "Reviewer_ZQYG", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and detailed comments. We will attend to all comments to the best extent, and try to address your concerns as follows:\n\n**Q1: Contribution and Novelty of the work**\n\nOur initial motivation is to prove the powerful ability of ChatGPT in machine translation and provide several simple strategies to maximize its performance, which has been questioned in previous work, for example, Jiao et al. (2023) reported in their position paper (arXiv: 2301.08745v1) that ChatGPT lags behind significantly on low-resource/ distant languages and different domains like biomedical.\n\nIn this work, we are (nearly the first) to find that ChatGPT’s translation performance largely depends on the temperatures, and emphasizing the task information and domain information can improve the performance. Through our experiments, we encouragingly show the MT and LLM community that it's not that ChatGPT is not effective in translation, but that we used it in an improper way.\n\n**Q2: “The slight improvement in Table 6 and Table 7”**\n\nFirst, we conducted the significance test (p<0.05) and showed that our TSP and DSP strategies bring significant improvement on 7/12 tasks. \n\nAlso, given that the call for using model-based neural metrics, e.g., Comet, rather than BLEU score [1,2] and the evidence from previous ``translation with LLM’’ studies [3], improvement in terms of Comet is more convincing than BLEU, and in our experiments, we achieved up to 0.9 Comet improvement (on the WMT19 bio zh-en task). \n\nMoreover, our method can be further improved by adjusting more hyperparameters, such as temperature and top-k/p decoding, and our aim is to provide simple but effective guidance to enhance translation performance. Therefore, we believe our improvements are reasonable.\n\n[1] Matt Post, A Call for Clarity in Reporting BLEU Scores. In WMT 2018.\n\n[2] Freitag et al., Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust. In WMT 2022.\n\n[3]  Hendy et al., How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. In arXiv preprint 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. The authors conducted experiments on multiple test sets (Multilingual / Multi-domain ) to confirm their findings. \n", "reasons_to_reject": "1. The contributions and novelty of the work are limited. After reading the paper, I am trying to figure out the research question, which may be a better translation prompt. The paper is more like a product manual telling me how to use ChatGPT for translation. \n\n2. The slight improvement （ ChatGPT vs. ChatGPT+DSP in Table 6 and Table 7）can not support the claim  “Experimental results across various language pairs and domains proved the effectiveness of our proposed prompts.” ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors conducted empirical experiments on the ChatGPT with the aim of enhancing its translation performance. These experiments were focused on the multilingual test sets Flores-200 and WMT19（News + Biomedical). The authors investigated both decoding temperature and translating prompts and obtained some empirical conclusions.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lztd6DPVr8", "reviewer": "Reviewer_4hHT", "summary": "", "questions": "1. Can you explain why lower temperatures can lead to better performance?\n2. Can you explain why non-centric translation can lead to more hallucinations?\n3. Just curiosity, what is the performance of GPT4?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Appreciate your positive comments. We will attend to all comments to the best extent, and try to address your concerns as follows:\n\n**Q1: “Too empirical without any theoretical understanding”**\n\nThanks for your advice, we will consider adding some theoretical formulations from the perspective of explaining why prompting engineering and in-context learning works.\n \n**Q2: “The slight improvement”**\n\nWe conducted the significance test (p<0.05) and showed that our TSP and DSP strategies bring significant improvement on 7/12 tasks, and we will add the significance test to the revised version.\n\nAlso, our method can be further improved by adjusting more hyperparameters, such as temperature and top-k/p decoding, and our aim is to provide simple but effective guidance to enhance translation performance. In the revised version, we will try to optimize the performance according to the hyperparameter grid search.\n\n**Q3: “Why lower temperatures can lead to better performance”**\n\nAs mentioned in section 3.1 and the introduction, previous work [1] has proved that decoding with higher temperatures displays greater linguistic variety, while the low one generates grammatically correct and deterministic text. For tasks with a high degree of certainty, such as translation, a lower temperature can generate more grammatically correct and deterministic text which can avoid stochasticity and achieve a stable and better performance.\n\n[1] Ippolito et al., Comparison of diverse decoding methods from conditional language models. In ACL 2019.\n \n**Q4: “Why non-centric translation can lead to more hallucinations”**\n\nWe argued that it may be because ChatGPT is mainly trained by English corpus, which may limit its ability to understand and generate other languages, which will lead to more hallucinations.\n\n**Q5: “Performance of GPT4”**\n\nThanks for pointing this out, given the significantly higher cost and inference latency (than GPT3.5), we only test our method in the WMT19 Bio En-Zh dataset with GPT-4. The results are shown below. \nThe results show that our method can stably improve the performance of GPT-4.\n\n|        | BLEU |  COMET | \n|:-----------:|:----:|:----:|\n|  GPT-4  |  33.6  |  56.9 |\n|  +TSP  |  **34.0**  |  **58.0** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to follow.\n2. Interesting and timely topic, which can contribute to the machine translation field.\n3. The evaluation is solid, with multiple datasets in different metrics.", "reasons_to_reject": "1. Although it is acceptable for EMNLP, this paper is too empirical without any theoretical understanding.\n2. The improvement of the two proposed prompts is not significant.", "questions_to_authors": "1. Can you explain why lower temperatures can lead to better performance?\n2. Can you explain why non-centric translation can lead to more hallucinations?\n3. Just curiosity, what is the performance of GPT4?", "paper_topic_and_main_contributions": "ChatGPT is widely adopted in various NLP tasks. However, the previous works on using ChatGPT for machine translation only adopt simple prompts and analyses. This paper conducts a more in-detailed study on using ChatGPT for MT. Specifically, the authors take temperature, task information as well as domain information into account and propose two novel prompting strategies. The main findings are: 1) ChatGPT’s performance largely depends on the temperatures and a lower temperature can result in higher performance; 2) Emphasizing the task information and domain information can improve the performance; 3) the translation between non-English pairs suffer from more hallucination.\n\nPresentation: Good\nThis paper is well-written, with different colors and icons, making it easier for the reader to understand.\nSoundness: Good\nThe evaluation is well and sufficient, with multiple datasets in different metrics.\nNovelty: Good\nThis is the first paper that takes a deeper look at using ChatGPT for MT.\nSignificant: Fair\nThe improvement of the proposed two prompts is not significant.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}]

Author rebuttal:

Meta-review: *Summary*: This work investigates different prompting strategies to improve ChatGPT's performance on machine translation tasks. The authors c
onsider the impact of decoding temperature, inserting task/domain-specific information to the prompts, and the effect of few-shot
and chain-of-thought prompting on translation quality on two (FLORES, WMT) benchmarks.

*Evaluation*: R1 and R3 have rated this work as 3 or higher on soundness (3/4), while R2 has rated this as a 2. R2's main concerns are that 1) the investigation in this work is too empirical without any other solid backing and 2) the content is a bit thin with the focus on identifying good prompting strategies to improve ChatGPT's translation capabilities. Regarding 1), well-conducted empirical analyses do have a place in EMNLP. With regards to 2), the authors have offered many natural dimensions (adding task/domain information, few-shot setting) along which prompting might be carried out for translation. The CoT-based prompting could have been made more competitive; it is not surprising that just adding "step by step" to the prompt does not help.
Decision: Accept-Findings