Document-Level Machine Translation with Large Language Models
Authors: Longyue Wang
Chenyang Lyu
Tianbo Ji
Zhirui Zhang
Dian Yu
Shuming Shi
Zhaopeng Tu
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Document-Level Machine Translation
Evaluation and Explaination
TL;DR: An in-depth evaluation of LLMs' ability on discourse modeling and document-level machine translation.

Abstract: Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs (We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM).

Document-Level Machine Translation with Large Language Models
Longyue Wang
Chenyang Lyu²*
Tianbo Ji Zhirui Zhang"
Dian Yu'
Shuming Shi'
Zhaopeng Tu
Tencent AI Lab "MBZUAI Dublin City University
(vinnylywang, jackzrzhang, shumingshi, zptu) @tencent.com
chenyang. lyu@ mbzuai.acc.e tianbo.jiz @ meil.
Abstract
3 Trandate this document from English Chiraaa
Large language models (LLMs) such as Chat- GPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document- level machine translation (MT) as a testbed, this paper provides in-depth evaluation of LLMs' ability on discourse modeling. The study fo- cuses on three aspects: 1) Effects of Context- Aware Prompts, where we investigate the im- pact of different prompts on document-level translation quality and discourse phenomena: 2) Comparison of Translation Models, where compare the translation performance of Chat- GPT with commercial MT systems and ad- vanced document-leww MT methods; 3) Anal- ysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evalu- surpris- ating number of benchmarks,
lue pilinna an mueld peinder
merwaeesn be bee Sonment
and oir
SURB W PESUNEE 1
(August Horch)
1118149 2E
# 112200 (ALE BEE00N
An example of translating a document- level text from English to Chinese using GPT-4 (Date: 2023.03. 17). We highlight the discourse phenomena using figures and lines, which are invisible to GPT-4.
Figure 1:
ingly find that LLMs have demonstrated supe- rior performance and show potential to hecome a new paradigm for document- 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms human evaluation:' 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evalua- tion of LLMs.?
translation:
T5 (Raffel et al.. 2020). These models have demon- strated impressive performance MT (Zhu et al., 2020; Guo al., 2020; Xue et al., 2021). However, most of the existing work has focused on sentence- level translation, which can result in translations that lack coherence and context. Recent years have seen a growing interest in document-level transla- lion, which is crucial task that involves translat- ing entire documents (Wang al., 2017; Bawden et al., 2018; Wang. 2019; Zhang al., 2022) while modelling specific discourse phenomena (Wang et al., 2016: Voita et al., 2018; Wang et al., 201 20, 2019; Voita et al., 2019b; Wang et al., 2023b). The most popular large language model (LLM) Chat- GPT shows the ability of maintaining long-tent term coherence and consistency in conversation by conditioning previous conversational turns. Ad-
1 Introduction
the past several years, machine translation (MT) has seen significant advancements with the intro- duction of pre-trained models such as BERT (De- vlin et al., 2018), GPT-2 (Radford et al., 2019), and
"Equal contribution.
'The protocol employed in work was approved by the Tencent Institutional Review Board (IRB). "We release our data and annotations at https: //gi:::///tthbb com/ conm/onnyyewangdcu wtwpe//wwwwdw/////ueeeeeeee Dww./ent- LLM.
hittps:/ccat..orr openai. aivenai.com. com. All corresponding results were obtained from GPT-3.5 and GPT-4 in March 2023. The reproducibility is discussed in Section Limitation.

---

ditionally, the model is trained on a large dialogue dataset, which allows it learn the patterns and conventions of human communication, further im- proving its ability to document-level understanding and generation (as shown in Figure 1).
English and English-<Rusiia)) and seven domains (i.e. news, social, fiction, Q&A, TED, Europarl, and subtitle). We adopt com- prehensive set of evaluation methods to mea- sure the performance of the models on document- level translation, including general-purpose met- ries, discourse-specific metrics, and human evalua- tion. The main contributions are:
In this paper, are particularly interested in how LLMs such as ChatGPT perform for modeling document-level text, encompassing discourse phe- nomena such as entity consistency, referential ex- pressions, and coherence. Taking document-level MT as a testbed, we conduct an empirical study from three in-depth perspectives:
Our empirical study shows the superior capabil- ities of L.Mss over advanced MT systems and methods on document-level translation, indicat- ing their potential to form a new paradigm.
Effects of Context-Aware Prompts: ChatGPT needs prompt as guidance to trigger its trans- lation ability. Thus, enable prompts to guide ChatGPT to consider document-level contexts as long as possible. Jiao al. (2023) has found that the candidate prompts generally work well and show minor performance differences on sentence- level translation. In this work, we further inves- tigate the effects of prompts on the translation quality and specific discourse phenomena.
We establish a benchmark with a probing method to thoroughly assess the document-level transla- tion quality and the ability of learning discourse knowledge, which will be made available for fu- ture research.
To facilitate future research on document MT, we publicly release the instruction-base benchmark, system outputs well human annotations.
2 Experimental Step
Comparison of Advanced Translation Models: While ChatGPT has demonstrated remarkable abilities in long-text NLP tasks, we are specifi- cally interested in how it performs document- level translation. Consequently, we conduct systematic comparison of commercial MT prod- ucts and advanced decument-level approaches, utilizing both automatic and human evaluations to assess their discourse awareness.
2.1 Dataset
Table shows statistics document-level datasets used in our experiments. About Group #1, we utilized the latest datasets, mZPRT (Wang et al., 2022) and WMT2022 (Koemi et al., 2022), for evaluation to ensure that the testing data had not been used in commercial systems (e.g. Google Translate and ChatGPT) before. As seen, this cov- ers four domains (i.e. news, social media, web fiction, and Q&A forum) in Chinese=Fngllshh Regarding Group #2, we utilized four widely- used benchmarks to compare established document- level methods with GPT-like applications. This cov- ers three domains (i.e. TED talks, news commen- tary, European Parliament) in Chinese- and English=Germman In Group #3, employed an English=<Rusiian contrastive testset (Voita et al., 2019b) that specifically targets discourse phe- nomena, such as deixis, ellipsis, and lexical cohe- sion. We use this dataset to further exploit models' capacity for modeling discourse knowledge.
A nalysis of Discourse Modelling Abilities: more challenging question is the extent to which ChatGPT capture and utilize discourse knowl- edge. To answer this question, we introduce a probing method through contrastive testing and explanation. In addition, the impact of various training techniques on the ability of LLMs to model discourse has not been thoroughly investi- gated. We compare variant models of ChatGPT that incorporate techniques such as code pretrain- ing, supervised fine-tuning (SFT), and reinforce- ment learning from human feedback (RLHF). However, this is not strict comparison because there are other confounding variables employed during the evolution of ChatGPT. In general, hope pose this open question that stimulates reflection and sparks further investigation.
As seen, we also report the average length of document (IWIIDI), which can be considered a measure of the complexity of discourse modeling.
As the length of the document increases, it becomes more challenging model accurate cohesive de- vices and discourse structure. From this perspec- tive, the mZPRT Fiction and IWSLT TED datasets pose a greater challenge compared to others.
We conduct experiments on a variety of document-level translation benchmarks, cover- ing three language pairs (i.e. Chinese-+English,

---

ID
Domain
Source
Language
IDI
ISI
IWI
IWVIDI
News
505
16.1K/18.5K
424
WMT2022
Zh=>En
Social
25
478
16.4K/13.3K
656
Fiction
12
857
17.1K/16.68
1,425
mZPRT
Zh=>En
Q&A
182
1,171
15.0K/22. IK
82
IWSLT2015
Zh
62
6,047
116.9K/101.5k
1,885
TED
IWSLT2017
En
23
2,271
38.1K/33.8K
1,657
2
News
News Commentary vll
155
2,999
56.8K/53.99
366
En=
Europarl
Europarl v7
360
5,134
130.1K/120.9K
361
Subtitle
OpenSub2018
En=>Ru
6,000
24,000
187 187.8K/514.8K
Table I: Statistics of datasets for document-level translation and analysis. We select the four latest benchmarks and four commonly-used ones, covering diverse domains and languages. We count the number of documents IDI, sentences ISI, and words |W] in terms of source/target language. The average length of document /WI/IDI can be considered a measure of the complexity of discourse modeling. Note taht, WMT2022 includes two references, whereas others have only one.
Discussion on Data Contamination We con- ducted ChatGPT in March 28~31 2023 with the official notice "the training data is up Septem- ber 2021".4 Different from previous work that leaned on dated or single-type datasets to assess ChatGPT's capabilities (Jiao et al., 2023; Lu et al., 2023), we carefully chosen both lastst, public and diverse testsets to mitigate the risks associated with data contamination. Taking Probing Discourse Knowledge (Section 5.1) for example, while the contrastive testset used for the prediction task orig- inated 2019, the evaluation on the explanation task remains devoid of public references. Our con- clusions are comprehensively made by considering both prediction and explanation results, balancing out any potential data contamination concerns. De- spite precautions, there remains a risk of data con- tamination, given that publicly available datasets are easily incorporated into LLM training (e.g. pre- training, SFT, RLHF). better way is con- sistently integrate and employ the latest datasets when evaluating LLMs.
models to achieve high correlations with human quality judgments. About document-level metrics, report document-level sacreBLEU (d-BLEU) (Liu et al., 2020), which computed by matching n-grams in the whole document. Note that all eval- uations are case-sensitive. To facilitate sentence- level evaluation of document outputs, we imple- mented automatic sentence spliting/alignmen" on the output and then manually corrected errors.
Discourse Awareness To target specific dis- course phenomena, we utilized two targeted met- rics, namely, cTT and AZPT, which respectively evaluate the consistency of terminology transla- tion and accuracy of zero pronoun (ZP) translation. Regarding CTT, one repeated terminology should keep the same translation throughout the whole document (Xiao et al., 201 1). We adopt lexical translation consistency metric (Lyu et al., 2021):
Σ Σ ((4=))) C2
CTT ΣieT
TT
2.2 Evaluation Method
for each terminology word TT, the C de- notes the size of the combination of translation (t,,..., tt), and function 1(t; = -y) returns L; is same as (j. otherwise The metric illus- trates how frequently translation pairs of w is same within document. The higher the metric value is, the more likely w is translated consistently. Regarding ZP. it is a discourse phenomenon that appears frequently in pronoun-dropping (pro-drop)
Translation Quality We evaluate different ap- proaches and systems using classical sentence- and document-level evaluation metrics. About sentence-level metries, we employ the commonly- used Lac (Post, 2018) and TER (Snover et al., 2006). Additionally, we utilize COMET (Rei et al., 2020), which leverages pretrained language
'https://plaffor: http://aaaforroogeaiccooooo///oo wwt.:/ccs//oomee//...../ gpt-4.

---

BLEU
TER
COMET
d-BLEU
CTT Fiction
AZPT
ID
News
Fiction
News
Fiction
News
Fiction
News
Fiction
Fiction
Base
25.5
12.4
62.7
85.0
0.420
0.095
28.2
15.4
0.19
0.39
25.8
13.9
63.8
82.8
0.483
0.124
28.7
17.0
0.29
0.41
P2
26.2
13.8
61.4
81.8
0.479
0.155
28.8
16.5
0.28
0.41
P3
26.5
14.4
61.1
81.1
0.469
0.154
29.1
17.4
0.33
0.44
Table 2: Ablation study of document-level prompts (detailed in Table 3) Chinese=>English datasets using ChatGPT. We use BLEU and d-BLEU to measure sentence- and document-level translation quality. We also conduct two targeted evaluation metrics measure specific discourse phenomena: accuracy of zero pronoun translation (AZPT) and consisteney of terminology translation (CTT). Base is sentence-level baseline system using InstructGPT API without any context-base based chat box.
Prompt
cordingly, each output will be assigned two distinct scores (0~5). For each domain subset, we assessed 100 instances, with each instance containing out- puts from different systems. This amounted an evaluation of roughly 70K words total. The scores were assigned each window of neighbor- ing sentences, taking into account the context pro- vided by the entire document. Our intent was for evaluators to consider discourse properties beyond single sentences, while also avoiding the difficult task of evaluating entire document. We em- ployed two professional evaluators for our study. The payment and background is detailed in Section Ethical Considerations and Appendix A.2, respec- tively. Besides, our annotators were given practice items, and the annotations reaches 0.86 Cohen's ss kappa scores (McHugh, 2012), demonstrating that the annotators work efficiently and consistently un- der this guideline.
P1
Please provide the TGT translation for the sentence: S
P2 Translate the following SRC sentences into
TGT: [S1]. [S2]...
(Continue) Translate this document from SRC TGT: S S
Table 3: The prompts suggested by ChatGPT for document-level translation. SRC and TGT denote source and target languages, respectively. Each document orderly processed in one "Chat Box" while each prompt is fed into one "Conversational Turn". P1 represents separately translating each sentence in document. P2 or P3 means translating document w/wo a sentential boundary tag "["".
languages such as Chinese and Japanese. Recover- ing ZPs in target language (non-pro-drop) needs understanding of the discourse structure. We used the AZPT score measure the accuracy of ZP translation (Wang al., 2022):
Effects of Context-Aware Prompts
3.1 Motivation
AZPT Σ=Ezp A(t2/2) IPP
Existing document NMT methods can be mainly classified into two categories: multi- sentence (Wang et al., 2017; Voita et al., 2018; Tu et al., 2018) and whole-document (Macé and Ser- van, 2019; Bao al., 2021) approaches. ChatGPT is capable not only handling long text in sin- gle conversational turn but also recalling the entire context in the chat box. Accordingly, we design prompts to trigger the document-level translation ability of ChatGPT.
(2)
where ZP is the list of zero pronouns in the source sentences, to is the generated translation for the zero pronoun 2, and A(tzlz) binary scorer to judge whether &g is the correct translation z.
Human Evaluation To thoroughly validate our conclusions, we also conduct a human evaluation (see Scction Limitation.). We establish two sets of evaluation criteria: 1) general quality, covering aspects such as fluency and adequacy; 2) discourse- aware quality, including factors such as consis- tency, word choice, and anaphora. The detailed scoring criteria are listed in Appendix$ A.2. Ac-
The prompt engineering is necessary to ensure ChatGPT's robust ability interpret instructions and to model long-term dependencies. Our re- search confirms the neutrality and representative- ness of various prompts, allowing other researchers

---

Automatic (d-BLEU)
Human (General/Discourse))
Model
News
Social
Fiction
Q&A
Ave.
News
Social
Fiction
Q&A
Ave.
Google
27.7
35.4
16.0
12.0
22.8
1.9/2.0
1.2/1.3
2.1/2.4
1.5/1.5
1.7/1.8
DeepL
30.3
33,4
16.1
11.9
22.9
2.2/2.2
1.3/1.1
2.4/2.6
1.6/1.5
1.9/1.9
Tencent
29.3
38.8
20.7
15.0
26.0
2.3/2.2
1.5/1.5
2.6/2.8
1.8/1.7
2.1/2.1
GPT-3.5
29.1
35.5
17.4
17.4
24.9
2.8/2.8
2.5/2.7
2.8/2.9
2.9/2.9
2.8/2.8
GPT-4
29.7
34.4
18.8
19.0
25.5
3.3/3.4
2.9/2.9
2.6/2.8
3.1/3.2
3.0/3.1
Table 4: Comparison between commercial MT systems and LLM applications on Chinexe-+English datasets using both automatic human evaluation methods. The human evaluation based on scale from 0~55 encompasses two dimensions: general quality and discourse awareness (detailed in Table 12). The significant test is detailed in Appendix I.
utilize them with confidence, unburdened by concerns of unintended biases.
of P2 and combine all sentences together. Take- away: (1) Despite translating document sentence by sentence, CharGPT's ability to model long-term dependencies already exists within the chat box. (2) Increasing document length as a input can further enhance translation quality and discourse aware- ness. (3) ChatGPT tends to translate a document without adhering to strict sentential boundaries, mirroring natural approach adopted by humans during document translation, which doesn' neces- sitate sentence-to-sentence translation.
3.2 Comparison of Different Prompts
We query ChatGPT itself for advice and obtain a number of candidate prompts, and then refine them into three document-level prompts as shown in Table 3. We utilize Pl to translate document sentence by sentence, with each sentence placed in single conversational turn and the entire docu- ment contained within chat box. This mainly takes advantage of ChatGPT's 's long-term modeling ability in the chat box. P2 and P3 combine mul- tiple continuous sentences and translate them in one conversational turn until the entire document is finished. This aims to maximize the length of document as input. The only difference is whether or not the sentential boundary tag "[]" is inserted into each sentence.
4 Comparison of Translation Models
In this section, we compare various systems and methods for the document-level translation task. In the following experiments, we use the P3 prompt for ChatGPT and the same document-level window size for MT models as the default setting.
4.1 ChatGPT vs. Commercial Systems
We compare the three candidate prompts on the Zh->En En translation task using two testsets, WMT2022 News and mZPRT Fiction. Table 2 shows the translation quality in terms of variety of automatic evaluation metrics. In general, Chat- GPT reliably performs well with three candidate prompts, showing only minor variations perfor- mance. This aligns with prior findings sentence- level translation with ChatGPT (Tiao et al., 2023). Out of the three prompts, the prompt involved multi-turn contexts without sentence boundaries (P3) achieves the best scores in most evaluation metrics, except for COMET. Regarding discourse phenomena, P3 outperforms other candidates with better consistency of terminology translation and higher accuracy of ZPP translation. Upon examin- ing the output samples, we noticed that ChatGPT may sometimes forget the sentential boundary tag
Commercial systems are known for their high curacy and efficiency in translation, making them strong contender for any machine translation evalu- ation. By comparing with commercial systems, we can gauge ChatGPT's performance relative to the best available MT technologics. We compare GPT- 3.5/GPT-4 with three commercial translation prod- ucts, including Google Translate," Deepl. Trans- late,? and Tencent TranSmart (Huang et al., 2021). We employ both automatic (d-BLEU) and human evaluation (general/disccouree-wvaa quality) as de- tailed in Section 2.2.
Table shows the results. When evaluated using d-BLEU, commercial MT systems generally out-
"https:/ttranslate.r www.gooooggcooooo " hhttps://ww.dee dewpl. com.
"https://tranmmart.cg con.

---

ZH>EN
EN-DE
Model
TED
TED
News
Europarl
BLEU
d-BLEU
BLEU
d-BLEU
BLEU
d-BLEU
BLEU
d-BLEU
MCN
19.1
25.7
25.1
29.1
24.9
27.0
30.4
32.6
G-Trans
25.1
27.2
25.5
27.1
32.4
34.1
Sent2Sent
19.2
25.8
25.2
29.2
25.0
27.0
31.7
33.8
MR-Doc2Sent
19.4
25.8
25.2
29.2
25.0
26.7
32.1
34.2
MR-Doc2Doc
25.9
29.3
26.7
34.5
Sent2Sent* 21.9
27.9
27.1
30.7
27.9
29.4
32.1
34.2
MR-Doe2Sent*
22.0
28.1
27.3
31.0
29.5
31.2
32.4
34.5
MR-Doe2Doc*
28.4
31.4
32.6
34.9
ChatGPT
28.3
33.6
39.4
30.4
Table 5: Comparison between document-level NMT methods and LLM applications on Chinese==Englihh and English=Geeraal benchmarks using commonly-used BLEU and d-BLEU metrics. "x" indicates using additional sentence-level corpus for model pre-training.
perform LLM-based systems, except for the Q&A domain, which involves informal spoken language. While the difference in performance is not signif- icant in the news domain (e.g. the gap between DeepL and GPT-4 is only 0.6 points), it is consid- erable in the social media and web fiction domains (i.e. the gaps are 3.3 and 1.9 points). A surpris- ing finding is that GPT-4 and GPT-3.5 perform significantly better than MT systems in terms of human evaluation. The potential reasons may be: (1) d-BLEU only measures the similarity of the n-grams between the MT output and the reference translations. However, human takes into account additional factors such as coherence, fluency, and naturalness of the translation, which may not neces- sarily correlate with d-BLEU scores. (2) ChatGPT and MT systems may have different strengths and weaknesses. For example, ChatGPT may be better at modeling long-term dependencies and capturing discourse-leve information, which could result in higher human evaluation. On the other hand, MT systems may perform better in terms of word-level accuracy, which is reflected in d- BLEU. Note that, our findings is distinguished from Neubig and He (2023). Focusing on long-tex translation, com- pare ChatGPT with MT systems, and underscore ChatGPT's enhanced capacity to model long-term dependencies comparison to MT systems. On the other hand, Ncubig and He (2023) investigate the varying performances of GPT models based on sentence length. They found that GPT models perform better on shorter sentences while worse on longer ones. Karpinska and lyyer (2023) recently
highlighted that GPT-3.5 has the capability uti- lize document-level context effectively for literary translation, yet it not free from critical errors. While the Fiction testset in our work is categorized under literary, we did not find obvious omission errors in the output. A more detailed comparison is earmarked for future exploration. Karpinska and Tyyer (2023) recently pointed that GPT-3.5 can effectively leverage document-level context for literary translation, but critical errors persist. Al- though the Fiction subset belongs to literary, we did not find omission errors in the output and we leave this fine-grained comparison for future work. Takeaway: (1) There is a certain degree of discrep- ancy discrepancy between human and automatic evaluation, which potentially provide complemen- tary reference points when measuring the quality of document-level translations; (2) This discrepancy underlines the complexity inherent accurately evaluating the capabilities of such systems. We further explore evaluation methods in Section 5.I.
4.2 ChatGPT Document NMT Methods
Document NMT methods are specifically designed to handle part entire documents, making them relevant point comparison for evaluating Chat- GPT's ability to model long-tern dependencies and discourse phenomena. We compare with five advanced document-level NMT models:
MCN (Zheng al., 2020): A multi-channel net- work that integrates hierarchical encoder and parallel decoder, which leverages the document structure and semantics for translation.

---

ID
Prompt
Model
deixis
lex.c
ell.infl
ell.
P4
Given an SRC (D). Which one is the correct TGT translation as follows: [Til,
Sent2Sent
51.1
45.6
55.4
27.4
MR-Doc2Doc
64.7
46.3
65.9
53.0
.... ...ITmm Why?
CADec
81.6
58.1
72.2
80.0
DocRepair
91.8
80.6
86.4
75.2
Table 6: The prompt for probing discourse knowledge encoded LLMss. SRC and TGT denote source and target languages, respectively. D represents document contains several sentences. T ... Tmm refer to the trans- lation candidates, where only one of them is positive translation and the others negative due to the modifi- cation discourse-specific words.
GPT-3.5
57.9
44.4
75.0
71.6
GPT4
"85.9
72.4
69.8
*81.4
Table 7: Accuracy [%] of translation prediction for spe- eific contextual phenomena (deixis, lexical consistency, ellipsis (inflection), and VP ellipsis) between different models the English=Russian contrastive testset. indicates a significant difference (p < 0.001) between GPT-4 and GPT-3.5.
G-Trans (Bao et al., 2021): A graph-based based transformer that incorporates document-level dis- course structure directed acyclic graph, en- hancing the representation of the context.
the domain distribution bias of the training data. Moreover, we find that ChatGPT is unstable, and its translation results sometimes exhibit omissions and obvious copying behaviors. Note that, the commonly-usee datasets were created between 2012 and 2017, a time frame that raises the pos- sibility of these datasets being incorporated into the training data of newer language models. Take- away: (1) ChatGPT has exhibited superior perfor- mance and may become a new promising paradigm for document-level NMT; (2) It is still debatable whether these benchmarks can be considered as up- propriate measures for evaluating document-level translation methods. We advocate for greater trans- parency from model developers regarding their training datasets. Additionally, this highlights the importance of designing innovative evaluation tech- niques that can reliably assess model capabilities while sidestepping concerns related to data con- tamination.
Sent2Sent: A superior sentence-level baseline that employs transformer architecture to trans- late each sentence independently and then merges the translations into document- output.
MR-Doc2Doc and MR-Doc2Sent: Sun et al. (2022) explore resolve document transla- tion with the end-to-end, namely document-to- document (Doc2Doc) pattern, and utilize Multi- resolutional Training, which combines docu- ments with shorter segments like sentences or paragraphs to improve translation quality (de- noted as MR-Doc2Doc). Additionally, they re- produce the document-to-senteee Doc2Sent) that introduces extra model modules to capture contextual information.
baseline (MR-
To enable fair comparison with previous work, we use four widely used document-level translation benchmarks: TED (ZH-EN and EN-DE), News (EN-DE), and Europarl (EN-DE). We adopt tok- enized case-insensitive BLEU and d-BLEU as the evaluation metrics. As MR-Doc2Doc and Chat- GPT generate document-level translations that are difficult to separate into individual sentences, we only report d-BLEU scores for these models.
Analysis of Large Language Models
We analyze the ability of LLMs to capture dis- course knowledge from perspectives: (1) prob- ing the discourse knowledge encoded in LLMs, and (2) examining the impact of different training techniques on discourse modeling.
Table 5 lists the results. The MR-Doc2Doc with extra model pre-training achieves the best document-level performance among previous mod- els. Thanks to the document-level LM pre-training, ChatGPT casily outperforms MR-Doc2Doc* TED (EN-DE) and News (EN-DE) datasets, obtain- ing similar performance on TED (ZH-EN) dataset, Surprisingly, ChatGPT performs poorly on the Eu- roparl (EN-DE) dataset, even worse than Sent2Sent. We suspect this phenomenon may be caused by
5.1 Probing Discourse Knowledge in LLM
In order to verify whether LLMs truly learn to uti- lize the context to resolve discourse inconsistencies, adopt the contrastive test sets proposed by Voita et al. (2019b). This dataset includes deixis, lexi- con consistency, ellipsis (inflection), and ellipsis (verb phrase) for evaluating discourse phenomena in English-Russian Russian translations. Each instance has

---

GPT-3.5
GPT-4
Subset
Prodiction
Explanation
Prediction
Explanation
rg
deixis
58.0
18.0
0.293
*89.0
*93.0
0.279
1cx.c
42.0
11.0
0.089
*72.0
*86.0
0.293
ell.infl
75.0
58.0
0.398
71.0
*91.0
0,184
cP
74.0
75.0
0.448
82.0
*94.0
0.539
Table 8: Human evaluation results of GPT-3.5 and GPT-4 on contrastive sets. For each test set, we randomly select 100 examples and ask annotators to assess whether the responses generated by the models include the correct prediction and explanation, respectively.
count the accuracy (%) of prediction and explanation for GPT-3.5 and GPT-4, based which the Phi coefficient (ro)) is calculated to measure the association between two binary variables (i.e., prediction and explanation). "*" indicates a significant difference (p < 0.001) between GPT-4 and GPT-3.5.
positive translation and few negative that differ by only one specific word. The goal to determine if model more likely to generate a correct translation than incorrect variations. this experiment, we compare GPT-3.5/GPT-4 with ad- vanced methods, such as Sent2Sent, MR-Doc2Doc, CADec (Voita et al., 2019b) and DocRepair (Voita et al., 2019a), where CADec and DocRepair intro- duce context-aware post-editing modules to refine the sentence-level translations. For these baselines, we adopt force decoding to generate scores for all translation candidates each instance. If the score of the positive translation is the highest, then this instance is counted as correct. For ChatGPT, we query them with the prompt P4 in Table 6 to obtain responses and correspondent explanations for each instance. Then some heuristic rules and manual verification used calculate final performance.
Evaluation Explanation We conduct human evaluations assess the quality of LLM-generated explanations. This provides an additional way explore the discourse knowledge contained within LLMs. As illustrated Table 8, we randomly select 100 examples for each contrastive test set and request native speakers to evaluate whether the models' responses contain the correct predic- tion and explanation, respectively. Then the Phi coefficient (ra) is further calculated better mea- sure the correlation between two binary variables (i.e., prediction and explanation). We can observe that the accuracy of explanation often not re- flective of the accuracy of prediction, indicating mismatch in utilizing discourse knowledge for prediction and explanation. In addition, GPT-3.5 is not good at explaining the reason for selecting the correct translation, while GPT-4 exhibits high per- formance in this aspect and brings better accuracy of prediction. Takeaway: (1) GPT-4 demonstrates a strong ability to explain discourse knowledge. (2) Despite GPT-4's superior performance in pre- diction and explanation, the correlation between prediction and explanation does not appear to be significantly improved compared to GPT-3.5.
Evaluation on Prediction As shown in Table 7, GPT-3.5 performs worse than DocRepair (discouse- enhanced method) across all discourse phenomena, with particularly significant gaps present in deixis and lexical consistency tests. These results show that it is difficult to handle deixis and lexical consis- tency phenomena with large-scale document-level pre-training. GPT-4 exhibits significant improve- ments in these areas, but still lags behind DocRe- pair in deixis, lexical consistency, and ellipsis (in- flection) phenomena. Takeaway: (1) GPT-3.5 demonstrates lower accuracy in contrastive pre- diction compared to conventional translation mod- els, whereas GPT-4 exhibits significant improve- ment. (2) As there is no detailed technical report available for GPT-4, we argue that its significant improvements are likely due to the use of super- vised data and RLHF. We further explore this in Section 5.2.
5.2 Potential Impacts of Training Techniques
LLMs have become the foundation of natural lan- guage processing research (Brown et al., 2020), with recent advancements such as learning from source code (Chen et al., 2021) and RLHF showing promise in improving LLM performance (Ouyang et al., 2022). To investigate the potential impacts of these approaches on discourse modelling, we con- duct experiments on Chinese-+English Fiction and datasets using different variants of LLMs trained with distinct techniques (detailed in SA.3). Accordingly, we use P3 and P4 prompts.
English=

---

ZH
EN-RU
Model
Training Techniques in LLMs
Automatic
Human
Probing
GPT-3
3.3
n/a
n/a
GPT-3
InstructGPT
+ SFT
7.1
n/a
n/a
Code Premaavigg
Supervised Ficeuniig
FeedME-1
14.1
2.2/2.5
30.5/28.6
InstructGPT
[SFT)
CodexGPT
Highquality Sumplea Supervised Pasotuning
Highquullty Samples
+ FeedME-2
16.1
2.2/2.3
34.4/30.
InstructOPT (FeddME
InstructGPT
+ PPO
17.2
2.6/2.7
58.0/39.4
(FecdME
Reinforommnnt Learning Hummn Fsecback
GPT-3.5
17.4
2.8/2.9
62.3/40.5
Comversatinnl
GPT-4
18.8
2.6/2.8
78.5/91.0
InstrucGGPT
(PPO)
Abilly
CharGPT
Table 9: Translation quality on Chinesc=EEnglish Fiction (automatic d- U-BLEU and human general/chscourse) and probing performance Rugsiah==Rusian (constrative prediction and explanation). The figure the left illustrates LLMs along with their training methods (from ChatGPT reports and API notes). "n/a" skips low performances.
Table 9 shows the results. When SFT with high- quality demonstration examples, the translation per- formance can achieve d-BLEU, which reaches to an acceptable level (InstructGPT +FeedME-1 vs. +SFT). Moreover, code pretraining can im- prove the document translation quality by 2 d- BLEU points and the discourse knowldge probing by 4/1.5 (CodexGPT +IeeddMB-2 vs. InstructGPT +FeedME-1). When further adding PPO method, it outperforms all other combination strategies on translation quality, discourse awareness and dis- course knowledge probing (CodexGPT +FeedME- 2 +PPOO vs. others). This shows that RLHF strongly enhances LLM's capability of translation. Lastly, GPT-3.5 and GPT-4 excel in d- B-BLEU and human evaluation scores as well as probing performance, demonstrating the importance of contextual infor- mation for complex, lengthy document translation. Takeaway: (1) Methods like code pretraining, SFT and RLHF appear to enhance the performance of document translation and discourse modeling; (2) However, it is quite challenging to explore the non-open source systems due to various con- founding factors introduced during their develop- ment. Therefore, advocate for organizations like OpenAl to provide greater transparency to aid researchers in such explorations.
prompts, (2) comparison of advanced translation models, and (3) analysis of discourse modelling abilities. With the release of the GPT-4 model, the discourse-award aware performance has been signifi- cantly improved, making it a promising paradigm for document-level translation. Despite its prowess in generative tasks, it struggles with discerning sub- tle distinctions for ranking.
In our future work, we plan to explore more document-level evaluation method (Castilho, 2021; Jiang et al., 2023; Koemi and Federmann, 2023), latest long-text benchmarks (Wang et al., 2023a; Thai et al., 2022; Wang et al., 2023c) and other MT scenarios (Ghazvininejad al., 2023; Guer- reiro et al., 2023; Lyu al., 2023). Furthermore, we intend delve into more detailed analysis and comparison in future work. For instance, we will employ appropriate significant test methods to account for multiple comparisons (e.g. non- parametric Kruskal-Walli test, Bonferroni correc- tion) and conduct a power analysis (Card et al., 2020; Graham et al., 2020; Vilar et al., 2022; Hendy et al., 2023). About annotation consistency, we fur- ther apply the Krippendorff's alpha coefficient and check the confidence interval (Krippendorff,, 2011). 1)..
Limitations
6 Conclusion and Future Work
We list the main limitations of this work follows: Potential Inaccuracy of Conclusions. Our con- clusions are derived from experiments conducted on a limited set of datasets, which may not guar- antee accuracy or applicability across all contexts. These limitations might inadvertently introduce
We provide a comprehensive evaluation of LLMS (such as GPT-3.5 and GPT-4) for document-level machine translation. Our evaluation covers three main aspects: (1) the effects of discourse-aware

---

with a background in translation and linguistics (detailed in Tablel To understand any polen- tial biases, annotators were inquired about their familiarity with the specific translation models under evaluation and any affiliations with Al or translation companies. We ensured that none of the annotators had conflicts of interest, and their annotations were routinely cross-checked for consistency. In terms of compensation, an- notators received an hourly wage of $37.4. This rate aligns closely with the mean hourly wages and for- bias or overlook certain phenomena, potentially impacting the comprehensiveness of our findings. In response to this concern, we strive to use an extensive array of the most recent datasets, span- ning various language pairs and domains. This broad coverage aims to encompass wide range of linguistic and contextual variations, thereby enhancing the generalizability of our findings.
Model Updates in ChatGPT and Reproducibil- ity. When the underlying model or its parame- ters of ChatGPT are updated or changed, the conclusions derived from prior evaluations may longer be valid or entirely accurate. To mit-
observed for U.S. interpreters/transleee eign language teachers."
igate this issue, this paper has tried utmost to ensure the reproducibility of our findings: (1) We release all system outputs accompanied by exact timestamps and change logs. This ensures that researchers can reliably reproduce and vali- date our results. (2) We evaluated all systems at two distinct points: in March and August 2023. While there were minor variations in the exact performance figures between these two evalua- tions, our overarching conclusions and core find- ings remained unchanged and consistent.
Annotator Consent and IRB Review. Prior to the commencement of the study, all participating annotators gave their informed consent, confirm- ing their understanding of the study's objectives and the intended research use of their annotations. An exhaustive IRB review was undertaken and finalized before the project's onset (IRB Protocol Number: IBB-2023-00067 and Approval Date: 01/12/2023). The protocol employed in this work was approved by the Tencent Institutional Review Board. We remain steadfast our commitment to uphold and adhere to the institution's 's highest ethical and professional benchmarks throughout our research endeavors.
Criteria of Human Evaluation and Refine- ment. The design of the criteria still has room for improvement. For example, in the "Discourse Awareness" category, there is only a slight dif- ference between Scores 5 and 4, but more sig- nificant gap between Scores 3 and 2. Given the scarcity of benchmark standards discourse evaluation from past research, we published the detailed scores for further analysis and highlight this area as an opportunity for refinement.
Reproducibility Challenges and Mitigation Strategies. The evolving nature of closed com- mercial platforms indeed presents challenges to the reproducibility of research. As these plat- forms continue to upgrade and improve, previ- ous versions of models may be retired or modi- fied, which can make replication efforts problem- atic. To tackle this issue, we have made several additions to our paper: (1) Documentation of Specifics: We have included exact versions of models we evaluated, along with the precise date of evaluation and other pertinent details. This allows for clear record of the conditions under which our study was conducted. (2) Release of System Outputs: We release all system outputs, which ensures that researchers can reliably re- produce and validate our results. (3) Advocacy for Archiving Historical Versions: We emphasize the importance for both the AI community and commercial organizations to maintain archives of previous model iterations. By doing this,
Ethical Considerations
Annotation Process and Annotator Profile. A one-week trial annotation phase was conducted for bidding (five companies participated) on our enterprise-level annotation platform. The authors answered questions posted by the annotators of these companies and updated annotation guide- lines accordingly. The Q&& history is recorded and updated in the formal annotation phase. After evaluating the trial annotations based on accuracy and consistency, we selected a professional lan- guage service company (large enterprise") head- quartered in Beijing, China. Their annotators were experts in both source and target languages,
IIt is based on the information provided by https:///wer bls. http://s//curreet//e2/////3///// -00999hhmmmmm .tm and ht https: //www. bls. bls..oo/oos/curreete /wwp:///ee/ccrrre///s25111111111 2tmmt/44.......mmmmmmmm
"It is hased on the information provided by https: //www. kanzhun.com com/firm.

---

searchers can readily access and evaluate past versions, ensuring continuity in analysis even as new model versions emerge.
bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805.
Marjan Ghazvininejad, Hila Gonen, and Luke Zettle- moyer. 2023. Dictionary-ban -basd phrase-level prompt- ing of large language models for machine translation. arXiv preprint arXhv:2302.07856
Acknowledgements
We are grateful to the anonymous reviewers, area chairs and ethics committee for their insightful comments and suggestions which will serve to im- prove the paper considerably. Their insights are in- valuable, not only in helping us present our findings more cautiously, but also in educating us beyond the scope of this single paper.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in ma- chine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langwage Processing.
Nuno M. Guerreiro, Elena Voita, and André Martins. 2023. Looking for a needle in haystack: A com- prehensive study of hallucinations neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics.
References
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Junliang Guo, Zhirui Zhang. Linli Xu, Hao-Ran Wei. Boxing Chen, and Enhong Chen. 2020. Incorpo- rating bert into parallel sequence decoding with adapters. Advances in Neural Information Process- ing Systems.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? comprehensive evaluation. arXiv preprint 2rxiv:2302.092/0
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenom- ena neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- ler of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Pa- pers).
Guoping Huang, Lemao Liu, Xing Wang. Longyue Wang. Huayang Li, Zhaopeng Tu, Chengyan Huang, and Shuming Shi. 2021. Transmart: A practical in- teractive machine translation system. arXiv preprint arXiv:2 13072.
Welch BI. 1947. The generalization of 'student's' prob- lem when several different population varlances involved. Biometrika.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan. Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems.
Yuchen Elcanor Jiang. Tianyu Liu, Shuming Ma, Dong- dong Zhang. Mrinmaya Sachan, and Ryan Cotterell. 2023. Discourse centric evaluation of machine trans- lation with a densely annotated parallel corpus. arXiv preprint arXiv:2305.1 11142.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? a preliminary study. arXiv preprint arXiv:2301.08745.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020. With little power comes great responsibility. arXiv preprint arxiv:2010.06595.
Marzena Karpinska and Mohit lyyer. 2023. Large lan- guage models effectively leverage document-level context for literary translation, but eritical errors per- sist, arXiv preprint 2rx.::2300.03245.
Sheila Castilho. 2021. Towards document-level human mt evaluation: On the issues of annotator agreement, effort and misevaluation. Association for Computa- tional Linguistics.
Tom Koemi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel. Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák. Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Trunslation.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint a7Xiv:2107.03374.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep

---

Tom Koemi and Christian Federmann. 2023, Large language models are state-of-the the-art art evaluators of translation quality, arXiv preprint arXiv:2302. 74520.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of trans- lation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transactions of the Association for Computational Linguistics.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chenggi Zhao, Shujian Huang. Jiajun Chen, and Lei Li. 2022. Re- thinking document-level neural machine translation. In Findings of the Association for Computational Lingaistics: ACL 2022.
Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Hao- ran Yang. Wai Lam, and Furu Wei. 2023. Chain- of-dictionary prompting elicits translation in large language models. arXiv preprint 27X:v:2305.06555.
Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray. Moira Inghilleri, John Wieting, and Mohit Iyyer. 2022. Exploring document-leve literary ma- chine translation with parallel paragraphs from world literature. arXiv preprint arXiv: 14250.
Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023. New trends in machine translation using large lan- guage models: Case examples with chatgpt. arXiv preprint 2xXhv:2305.01188
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with continuous cache. Transactions of the Association for Computational Linguistics.
Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. 2021. Encouraging lexical translation consis- tency for document-level neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance. arXiv preprint arXiv:2211.09111
Valentin Macé and Christophe Servan. 2019. Using whole document context in neural machine transla- tion. In Proceedings of the 16th International Confer- ence Spoken Language Translation, Hong Kong. Association for Computational Linguisties.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural ma- chine translation. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing.
Mary I. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica.
Graham Neubig and Zhiwei He. 2023. Zeno gpt ma- chine translation report.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019b.
When good translation is wrong in context: Context- aware machine translation improves on deixis, ellip- sis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroli Wainwright, Pamela Mishkinn Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. 2022. Training language models to follow instruc- tions with human feedhack. In Advances in Neural Information Processing Systems.
Elena Voita, Pavel Serdyukov, Rico Semnrich, and Ivan Titov. 2018. Context- aware neural machine trans- lation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers).
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAI.
Longyue Wang. 2019. Discourse-avee neural machine translation. Ph.D. thesis, Ph. D. thesis, Dublin City University, Dublin, Ireland.
Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research.
Longyue Wang, Zefeng Du, Donghuai Liu, Cai Deng, Dian Yu, Haiyun Jiang, Yan Wang. Leyang Cui, Shuming Shi, and Zhaopeng Tu. 2023a. Disco-bench: A discourse-awan evaluation benchmark for lan- guage modelling. arXiv preprint ar arXiv:2307.08072
Ricardo Rei Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing. Association for Computational Linguistics.
Longyue Wang, Siyou Liu, Mingzhou Xu, Linfeng Song, Shuming Shi, and Zhaopeng Tu. 2023b. A survey on zero pronoun translation. arXiv preprint arXiv:2305. 10796.

---

Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao- Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Kochn, Andy Way. Yulin Yuan, and Shuming Shi. 2023c. Findings of the WMT 2023 shared task discourse-level literary translation. In Proceedings of the Eighth Conference on Machine Translation (WMT).
Biao Zhang, Ankur Bapna, Melvin Johnson, Ali Dabir- moghaddam, Naveen Arivazhagan, and Orhan Firat. 2022. Multilingual document-level translation ables zero-shot transfer from sentences to documents. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers).
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Toward making the most of context in neural machine translation. ArXiv.
Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu. 2018a. Trans- lating pro-drop languages with reconstruction mod- els. In Proceedings of the 2018 AAAI Conference on Artificial Intelligence, volume 32.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. 2020. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06823.
Longyue Wang, Zhaopeng Tu, Xing Wang. and Shum- ing Shi. 2019. One model learn both: Zero pro- noun prediction and translation.
Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing.
Longyue Wang, Zhaopeng Tu, Andy Way. and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings the 2017 Conference on Empirical Methods in Natural Language Processing.
Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2018b. Learning jointly translate and predict dropped pronoums with a shared reconstruction mech- anism. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016. novel approach to dropped pronoun translation. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies.
Longyue Wang, Mingzhou Xu, Derek F. Wong, Hongye Liu, Linfeng Song. Lidia S. Chao, Shuming Shi, and Zhaopeng Tu. 2022. GuoFeng: A benchmark for zero pronoun recovery and translation. In Proceed- ings the 2022 Conference Empirical Methods Natural Language Processing.
RF Woolson. 2007. Wilcoxon signed-rank test, Wiley encyclopedia of clinical trials.
Tong Xiao, Jingbo Zhu, Shujie Yao, and Hao Zhang. 2011. foeument-levee. consistency verification chine translation. Proceedings of Machine Trans- lation Summit XIII: Papers.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mTT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 202/ Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies.

---

A Appendix
Level
Evaluator A (Zh-En)
A.1 Significance Testing
Position
lecture at an international university
Education
Ph.D degree in Translation Studies, in-
Machine Translation For automatic evaluations, we used the non-parametric one-tailed Wilcoxon signed-rank test (Woolson, 2007). About results in Table 4, the significance test contrasting GPT- 3.5/GPT-4 with others yields p-value of less than 0.05, indicating they do significantly boosts trans- lation quality. For human evaluation, Welch's t-test (BI, 1947). Table 10 shows the over- all significance test by combining all datasets, and the results for each domain are also consistent.
ternational university
Certification
CATTI Translation Level
Experience
Translator for the academic journal; Participant in the Construction and Ap- plication Research of the bilingual ter- minology knowledge base, National Social Science Fund project.
employ
Level
Evaluator B (Zh-En)
Position
manager of quality control at famous
translation company
Education
Master in English, international univer-
Automatic
Human
sity
Certification TEM8
GPT-3.5
GPT-4
GPT-3.5
GPT-4
Experiencc
Translator for the National Internet In- formation Center; Translator and proof- reader for top company.
Google
0.0000
0.0000
0.0000
0.0000
DeepL
0.0000
0.0000
0.0000
0.0000
Tencent
0.0309
0.0001
0.0000
0.0000
Level
Evaluator (Ru-En)
GPT-3.5
n/a
0.0022
n/a
0.0028
Position
interpreter at an import&export trading
GPT-4
n/a n/a
n/a n/a
company
Education
Master in Russian Written Translation.
Table 10: The results of significance test in Table 4.
international university
Certification
Russian Professional Level English CET6
Probing Task We performed a Welch's t-test (B), 1947) with unequal variances to verify the signifi- cance between GPT-3.5 and GPT-4 in Table 7 and 8. We find that the corresponding two-tailed value is smaller than 0.001, which indicates the significance between them.
Experience
Interpreter in several import&export
trading companies.
Level
Evaluator (Ru-En)
Position
translator at translation company
Education
Master in Russian Written Translation,
international university
A.2 Human Evaluation Guidelines
Certification
Russian Professional Level English
CET6
Guidelines Table 12 presents the human evalu- ation criteria for document-level translation, with scores ranging from 5. A score of 5 indicates excellent overall translation quality, with no gram- matical errors, accurate word choice, consistent key terms, and consistent context and tone throughout the passage. score of 0 indicates poor overall translation quality, with more than half of the trans- lation being mistranslated or missing, inconsistent key terms, and poor fluency and clarity. In between scores reflect varying degrees of translation quality, with factors such fluency, accuracy, consistency of key terms, and context and tone consistency af- fecting the score.
Experience
Work in several translation companies.
Table
The basic background of human annotators.
A.3 Training Methods in LLMs
GPT-3 (Brown al., 2020): A LLM with 175B parameters pre-trained on large-scale web cor- pora (approximately 400B tokens) We used Ope- nAI API davinci.
InstructGPT (SFT) (Ouyang et al., 2022): A GPT-3 model trained with supervised fine-tuning on human demonstrations similar to Ouyang et al. (2022)."
InstructGPT (FeedME-1) (Ouyang et al., 2022): An improved version of GPT-3 with supervised
We employed two profes- sional evaluators for our study through Tencent's designated supplier. Table 11 detailed their back- ground related to this task.
Human Evaluators
"https:/ opw.ai.penacc..mm com/ com/docs/ model snel-index-Fo reseeassaschers.

---

fine-tuning on human-written demonstrations and model-generated examples rated by humans with high quality.
InstructGPT (FeedME-2) (Ouyang al., 2022): An improved version of Codex (Chen al., 2021) with supervised fine-tuning on human-written demonstrations and human-rated examples with high quality." 12
InstructGPT (PPO) (Ouyang et al., 2022): An improved version of InstructGPT (FeedME-2) with extra training of RLHF, which trained with reward model learned from human com- parisons. 13
ChatGPT: A further improved version of In- strucGPT that can perform tasks via dialogue with users, which is able to take contextual infor- mation in dialogue into consideration.
Phttpps://ppeailc http:///ppeena...mmmo//b///eeerp
'hhttps///og httpss:///ppen..ic/m httpiaagaaccooo//////oo////
cotruuctioo-folowwinn

---

Score
General Quality
Discourse Awareness
5
Translation passes quality control; the overall translation is excellent. Translation is very fluent with grammatical errors and has been localized to fit larget language. Word choice is accurate with no mistranslations. The translation a 100% true to the source text.
No inconsistency relating key terms such names, organization, etc. Linking words or expressions between sentences keeps the logic and language of the passage clear and fluent. Context and tone are consistent throughout. The style of the text conforms to the culture and habit the target language.
4 Translation passes quality control; the overall translation is very good. Translation is fluent. Any errors that may be present does not affect the meaning or comprehension of the text. Most word choice is accurate, but some may cause ambiguity. Key terms are consistent. Inconsistency is limited to non-key terms.
Logical and language is clear and fluent. Some sentenees lack transition but does not affect contextual comprchension. Topic is consistent. Tone and word choice may be in- consistent, but comprehension is not affected. Translation conforms to the culture and habit.
3 Translation passes quality control; the overall translation is ok. Translation is mostly flu- ent but there many sections that require rereading due to language usage. Some word choice is inaccurate or errors but meaning of the sentence can be inferred from context.
Some key terms may be inconsistent. Most sentences translation smoothly and logically but some sentences that may seem abrupt due to lack of linkage. Topic is consistent. Tone and word choice is inconsistent, noticeably affecting the accuracy of reading comprehen- sion.
2 Translation does not pass quality control; the overall translation is poor. Meaning is unclear disjointed. Even with multiple rereading, passage may still be incomprehen- sible. Translation is not accurate to the source text or is missing in large quantities, causing the tramslation to deviate from the source text.
Many key terms inconsistent, needing multiple rereading understand context of the passage. Some linkages present but overall, the passage lacks fluency and clar- ity, causing trouble with comprehension, The topic or tone is different from the other pas- sages, alfecting reading comprehension.
Translation does not pass quality control; the overall translation is very poor. More than half of the translation is mistranslated or miss- ing.
Key terms are inconsistent, causing great trouble with comprehension. Some linkages present but overall, the passage lacks fluency and clarity, heavily interfering with comprehension. The topic or tone is different from the other passages, heavily interfering with comprehension.
Translation output unrelated to the souree text.
Output is unrelated to previous or following sections.
Table 12: Human evaluation criteria on document-level translation.


Reviews:
[{"review_id": "JaN5ZeVWWg", "reviewer": "Reviewer_GjYL", "summary": "", "questions": "Question A\n- In ZenoGPT report, they insist that \"GPT models are relatively good on shorter sentence and relatively bad on longer sentences.\". In this paper, ChatGPT is good at catching long-term dependency. I wonder which one is correct.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About difficulties of reproducibility because ChatGPT is constantly updated.**\n\nThank you for highlighting this concern, a challenge inherent to assessing all non-open-sourced systems such as ChatGPT. \n\nThis paper has tried utmost to ensure the reproducibility of our findings:\n1. We release all system outputs accompanied by exact timestamps and change logs. This ensures that researchers can reliably reproduce and validate our results.\n2. We evaluated all systems at two distinct points: in March and August 2023. While there were minor variations in the exact performance figures between these two evaluations, our overarching conclusions and core findings remained unchanged and consistent. We will report this in our revised version.\n\n**Q2. Different conlusions with ZenoGPT report.**\n\nThe two observations are not contradictory. \n1. In our study, we compare ChatGPT with MT systems, focusing only on long-text translation. Our results underscore ChatGPT's enhanced capacity to model long-term dependencies in comparison to MT systems. \n2. Conversely, the ZenoGPT report [1] delves into the differential performance of GPT models between shorter and longer sentences. They found that GPT models perform better on shorter sentences while worse on longer ones. \nWe will add this discussion in the revised version.\n\n[1] Graham Neubig and Zhiwei He. 2023. Zeno GPT Machine Translation Report.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting findings about analysis of discourse modelling abilities\n* They do an in-depth comparison between (ChatGPT/GPT-4) and advanced machine translation service\n", "reasons_to_reject": "* Difficulties of reproducibility. ChatGPT is constantly updated.", "questions_to_authors": "Question A\n- In ZenoGPT report, they insist that \"GPT models are relatively good on shorter sentence and relatively bad on longer sentences.\". In this paper, ChatGPT is good at catching long-term dependency. I wonder which one is correct.", "paper_topic_and_main_contributions": "They do the document-level translation by using ChatGPT and GPT-4 and analyze the result thoroughly.  They show the superiorities of LLMs over advanced MT systems and established a benchmark with a probing method to assess the document-level translation quality and the ability of learning discourse knowledge. They also find that code pre-training and RLHF really help to produce coherent and cohesive translations.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQ9KkR3JnT", "reviewer": "Reviewer_AmZG", "summary": "", "questions": "* are there any insights / changes in scores etc. when you'd exchange the prompts? or asked differently: how long did you \"prompt-engineer\" and by how much would the results look differently if you were to spend yet more time finding the ideal prompts or if you were not to prompt-engineer at all?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. How did changing prompts affect scores? How long was spent on prompt engineering? What would be the impact of further refinement or no prompt engineering?**\n\nIn Section 3, we explored various prompt strategies specifically to answer the questions above. \n\nTable 2 reveals that ChatGPT consistently performs well across three candidate prompts, with minor variations in performacne (evaluated using two datasets and six evluation metrics). This aligns with prior findings in sentence-level translation with ChatGPT [1]. It is pivotal to mention that our prompt strategies consider factors inherent to document-level translation. We highlited that the prompt involved multi-turn contexts without sentence boundaries (P3) performs the best.\n\nIt is necessary for us to spend approximately one weeks on this prompt engineering to ensure ChatGPT's robust ability to interpret instructions and to model long-term dependencies. Our findings serve as a testament to the representativeness and neutrality of different prompts, paving the way for future research. Other reseachers can employ these prompts confidently, without concerns over unintended biases, thereby ensuring the objectivity of their findings.\n\n[1] Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* emergent field, very important measurements and an assessment of a technology that surely will change the \"regular MT\" research as well as commercial MT model deployment\n* thorough experimentation, which all the rights questions asked and addressed", "reasons_to_reject": "* none, if my questions are addressed", "questions_to_authors": "* are there any insights / changes in scores etc. when you'd exchange the prompts? or asked differently: how long did you \"prompt-engineer\" and by how much would the results look differently if you were to spend yet more time finding the ideal prompts or if you were not to prompt-engineer at all?", "paper_topic_and_main_contributions": "This paper discusses the suitability of Large Language Models, such as GPT and ChatGPT, when being tasked with document-level machine translation. Among the contributions are effects of discourse-aware prompts, a comparison of LLMs vs other commercial MT systems, and an examination which of the components in the LLM pipeline (finetuning, reward modeling etc.) contribute most to the LLM's ability to translate in a discourse-aware manner. \nThe languages looked at are Chinese⇒English, English⇒German and English⇒Russian and the test data is from seven domains (i.e. news, social, fiction, Q&A, TED, Europarl, and subtitles).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XW4Dx3Slg1", "reviewer": "Reviewer_Dtpc", "summary": "", "questions": "Question A: Did you perform any significance testing, visualize the scores, inspect the trends, and analyze the data and its distribution beyond looking at the  average scores?\n\nQuestion B: How many evaluators were employed? How much were they compensated? How many documents were evaluated (only for one task it is mentioned that 100 were sampled)? Was there only one rating per document? Who were these people?\n\nQuestion C: In line 361 you write that it is debatable whether these benchmarks can be considered appropriate, I agree, but I don't see this discussed anywhere apart from takeaways.\n\nQuestion D: How were COMET/BLEU/TER scores computed for P3 in table 2? Since later for this prompt only d-BLEU is used due to the lack of sent-to-sent alignments.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About effects of training methods (Section 5.2) and analyzing black box.**\n\nIndeed, we are curious about ChatGPT's evolution in terms of discourse modelling, thus we leave this open question at the end the this paper (Section 5.2). We hope to stimulate further research in areas such as the interplay of code and text in pre-training. A detailed dissection of the impact of individual training factors (e.g. complex interplay between training methods and training data), while valuable, is beyond the scope of this study.\n\nWe recognize the intricacies of pinpointing specific causes due to the black-box nature of the models. Thus, this analysis is grounded on observable outcomes, and we aim to provide a comparative perspective rather than absolute claims on the internal workings of the models. Your feedback is invaluable, and we will make these distinctions more evident in the revised version.\n\n**Q2. About data contamination.**\n\nOur experiments on ChatGPT were conducted in March 2023, and the OpenAI's official notice about the date of training data is \"Up to Sep 2021\" (according to https://platform.openai.com/docs/models/gpt-4). Different from previous work that often leaned on dated or single-type datasets to assess ChatGPT's capabilities [1-2], we have carefully chosen both lastst, public and diverse testsets to mitigate the risks associated with data contamination.\n\nGiven the concerns regarding data contamination, we took rigorous steps to ensure the reliability and integrity of our evaluation:\n1. **Latest Datasets**. Instead of using older testsets, we have carefully chosen the latest document-level translation datasets, specifically those published in 2022, ensuring a gap between publication and the model's training cut-off.\n2. **Diverse Domains and Languages**. Instead of  relying on single-type testsets, we have consciously chosen eigth datasets covering six domains and two language pairs, offering a holistic view of ChatGPT's performance.\n3. **Multi-dimensional Evaluation**. Beyond merely assessing translation quality, we have additionally proposed probing task, diving deeper into the discourse knowledge embedded within models.\n\nTaking contrastive testing for example (Section 5.1), while the dataset used for prediction originated in 2019 [3] , the evaluation on explanation remains devoid of public references. This makes it improbable for ChatGPT to have been trained on it. Our conclusions are comprehensively made by considering both Prediction and Explanation Evluation, balancing out any potential data contamination concerns.\n\n[1] Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study.\n\n[2] Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, Furu Wei. 2023. Chain-of-Dictionary Prompting Elicits Translation in Large Language Models.\n\n[3] Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.\n\n**Q3. About significance testing.**\n\nWe have indeed carried out significance testing for all our results: \n\n1. **Machine Translation**. For automatic evaluations, we used the non-parametric one-tailed Wilcoxon signed-rank test. Taking results in Table 4 for example, the signficance test contrasting ChatGPT/GPT-4 with others yields a p-value of less than 0.05, indicating they do significantly boosts translation quality. The following table only shows the overall significance test by combining all datasets, and the detailed results for each domain are also consistent. For human evaluation, we will employ statistical power analysis to gauge the reliability and consistency of human judgments.\n\n|       p      |   Google  |   DeepL    |  Tencent   | GPT-3.5 | GPT-4 |\n|-----------|:------------:|:-------------:|:-------------:|:-----------:|:---------:|\n| **GPT-3.5** | 0.000000 | 0.000000 | 0.030863 |     n/a      |    n/a   |\n| **GPT-4**    | 0.000000 | 0.000000 | 0.000148| 0.002180 |    n/a   |\n\n2. **Probing Task**. We performed a ttest with unequal variances to verify the significance of different methods on deixis and lexical consistency tests, i.e., DocRepair vs. ChatGPT and  ChatGPT vs. GPT-4. We find that the corresponding two-tailed pvalue is smaller than 0.001, which indicates the significance between them.\n\nOur revised version will incorporate heatmaps displaying all significance test outcomes in the Appendix.\n\n[4] Yvette Graham, Barry Haddow, Philipp Koehn. 2022. Statistical Power and Translationese in Machine Translation Evaluation. \n\n**Q4. About lacking of multiple references and human evaluation.**\n\nThe official WMT2022 datasets used in our paper indeed contain multiple references, which are associated with News and Social testsets in Table 2 and 4. We will add this details in Section 2.1. \n\nYour observations about the importance of human evaluation resonate with our stance. We noticed that in sections like \"Main Contributions\" and \"Reasons To Accept,\" our human evaluations were positively acknowledged, whereas, under \"Reasons To Reject,\" there's a mention of an absence of human evaluation (\"That is why the human evaluation is so important, and the one in this paper is lacking.\"). \n\n**Q5. About human evaluation critera and missing details.**\n\nPlease see Q8 for more details regarding human evaluators.\n\nWe appreciate your pointing out the nuances in the general quality scale. We will follow your suggestions to conduct more detailed analysis instead of relying solely on simple averages. Besides, all translation outputs along with human scores are released, enabling other researchers to further evaluate and analyze them. \n\nIt is worth noting that we did not ask evaluators to assign a single score to an entire document. Instead, scores were assigned to each window of neighboring sentences, taking into account the context provided by the entire document. Our intent was for evaluators to consider discourse properties beyond single sentences, while also avoiding the difficult task of evaluating an entire document. We believe our results hold reliability and were not do not contradict with Sheila Castilho's works. Besides, our annotators were given practice items, and the annotations reaches 0.86% kappa scores, demonstrating that the annotators work efficiently and consistently under this guideline.\n\nAs we found limited standards on discourse evaluation in previous studies to reference, we would genuinely appreciate more specific recommendations or references concerning evaluation criteria.\n\n**Q6. About discourse-aware prompts.**\n\nThis section is not designed for exploiting how the availability of discourse information can improve the translation. Instead. this is a abliation study about effects of different prompt strategies on document-level translation. This kind of abliation study was usually exploited at the beginning of LLM evaluation works like Jiao et al. [1]. However, document-level translation with ChatGPT-like conversational systems is quite different from sentence-level task. Apart from just the instruction content, it is crucial to consider inherent document factors like multi-turn contexts and sentence boundaries.\n\nWe reveal that ChatGPT consistently performs well across three candidate prompts (evaluated using two datasets and six evluation metrics). Among them, the prompt (P3) involved multi-turn contexts without sentence boundaries performs the best. It is necessary on this prompt engineering to ensure ChatGPT's robust ability to interpret instructions and to model long-term dependencies. Our findings serve as a testament to the representativeness and neutrality of different prompts, paving the way for future research. Other reseachers can employ these prompts confidently, without concerns over unintended biases, thereby ensuring the objectivity of their findings.\n\n**Q8. About evaluators.**\n\nWe engaged two evaluators for our study. The basic information is as follows:\n\n| Evaluator 1 |\n|-------------------------------------------------|\n| Position: lecture at a top university |\n| Education: Ph.D degree in Translation Studies, top university |\n| Certification: CATTI Translation Level 1 |\n| Professional Experiences: Translator for the academic journal; Participant in the Construction and Application Research of the bilingual terminology knowledge base, a National Social Science Fund project. |\n|**Evaluator 2**|\n| Position: manager of quality control at top translation company |\n| Education: Master in English, top university |\n| Certification: TEM8 |\n| Professional Experiences: Translator for the National Internet Information Center; Translator and proofreader for top company |\n\nFor each domain subset, we assessed 100 instances, with each instance containing outputs from 5 different systems. This amounted to an evaluation of roughly 70K words in total. While we count instance in sentence, the actual number of documents varied due to the lengths of different domain. Besides, our annotators were given practice items, and the annotations reaches 0.86% kappa scores, demonstrating that the annotators work efficiently and consistently under this guideline. The compensation provided to the evaluators for this task amounted to $5000 US dollars. We will add the above details in the revised version.\n\n**Q9. About whether these benchmarks can be considered appropriate in line 361.**\n\nIndeed, Line 361 alludes to the debatable nature of the appropriateness of these benchmarks, especially in the context of evaluating LLMs like ChatGPT. This concern stems from the potential risk of data contamination, which could render the benchmark results unreliable. Many commonly-used datasets used in Table 5 were created between 2012 and 2017, a time frame that raises the possibility of these datasets being incorporated into the training data of newer language models.\n\nWe advocate for greater transparency from model developers regarding their training datasets. Additionally, this highlights the importance of designing innovative evaluation techniques that can reliably assess model capabilities while sidestepping concerns related to data contamination.\n\nWe will add such discussions in the revised verson when given more space.\n\n**Q10. How were COMET/BLEU/TER scores computed for P3 in table 2? Since later for this prompt only d-BLEU is used due to the lack of sent-to-sent alignments.**\n\nWe post-processed the document translation output for sentence-level evaluation. Specifically, we conduct automatic sentence alignment [5] and then mannually revise alignment errors. \n\n[5] https://github.com/rsennrich/Bleualign.\n\n**Q11. About missing references, typos and others.**\n\n1. All paper mentioned will be citated and carefully discussed in our revised version.\n2. I will revise the statements in the revised version: \"asking model\" to \"query model\"; \"for comprehending\" to \"for probing\"; \"ability to document-level awareness\"  to \"ability to document-level translation\".\n3. I will reivse the date in the footnote 1: All corresponding results were obtained from ChatGPT and GPT-4 in 28~31 March 2023. \n4. No, the GPT-3 model is davinci.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors explore an important and very timely task of document-level translation, \n- The authors add human evaluation (though details are missing and data analysis is problematic),\n- The authors experiment with many types of texts (news, fiction, Q&A)", "reasons_to_reject": "UPDATE: The authors addressed most of my concerns however, I believe that the first and second points are still valid and should be discussed as potential limitations (i.e., there are too many confounding variables to claim that one is investigating an impact of different training methods; and the datasets might have been - and probably were - used for RLHF).\n\nUPDATE 2: the concerns were addressed by the authors to the extend it was possible with the current design.\n\n- The authors claim to investigate the effect of different training methods on the processing of discourse-level information (as one of three main experiments), however, it is questionable whether what we see is the effect of different training methods, different training data, or perhaps the data used for RLHF (rather than RLHF alone, that is it is possible that the MT datasets used in this study were used to create RLHF examples). Since the authors research black box models behind an API, I do not think we can make any claims about the effect of the training method (of which we know little) on the model's performance.\n\n- Data contamination might have influenced the evaluation - The authors employ various existing datasets. While two of these datasets do have publication date past the openAI's models' training cutoff point (made public in August 2022), this seems not to be the case for the other datasets employed in this study (including the dataset with contextual errors). It is likely that these were included in the training data of the LLMs being evaluated. Furthermore, with the RLHF models, it is also possible (and quite likely) that MT datasets published post-training were employed to create the RLHF data. For instance the WMT22 dataset was made public in August 2022, which gives companies like OpenAI plenty of time to retrieve it, reformulate it into training examples, and use for RLHF.\n\n- The authors discuss how certain methods are significantly different from others, yet no significance testing is done to support these claims. For example, in line 486 the authors write \"The conversational ability of ChatGPT and GPT-4 significantly boosts translation quality and discourse awareness\" -- the difference between zh->en ChatGPT (17.4 d-BLEU; 2.8/2.9 humeval) and GPT-4 (18.8 d-BLEU; 2.6/2.8 humeval) and the scores for FeedME-2 (16.1 d-BLEU; 2.2/2.3 humeval) and PPO (17.2 d-BLEU; 2.6/2.7 humeval) is minimal and it's hard to say whether it is significant without proper testing (including checking the distribution and accounting for multiple comparisons).\n\n- The main automatic method used throughout this paper is d-BLEU, which works best with multiple references, yet I believe only one is given. I understand that there are limited automatic options for document level evaluation where the sentences cannot be aligned. Some researchers used sliding windows for COMET, but that's not ideal (yet worth trying?). That is why the human evaluation is so important, and the one in this paper is lacking.\n\n- Human evaluation - many important details are missing so it is hard to judge the research design (more questions below); however what bothers me most is that the authors construct an ordinal scale with a clear cutoff point between 2 and 3 (for general quality especially), yet they present only average scores. I do believe that \"5: Translation passes quality control; the overall translation is excellent (...)\" minus \"4: Translation passes quality control; the overall translation is very good\" is not the same \"one\" as \"3: Translation passes quality control; the overall translation is ok. (...)\" minus \"2: Translation does not pass quality control; the overall translation is poor. (...)\". It is clear that the difference between 5 and 4 is minimal, while between 3 and 2 is much bigger. Simple average is not a proper way to analyze this data (a proper analysis would include histograms of scores, possibly a stacked bar with proportion of choices, statistical testing). \n\n- Another issue with the human evaluation is that it appears that the evaluators were asked to evaluate an entire document by assigning it one score. Note that this is a cognitively demanding and difficult task for the evaluators. The results are likely to be unreliable (please see Sheila Castilho's work on this topic). There is also no indication that the annotators were at least given some practice items.\n\n\n- \"Discourse-aware prompts\" - I am not sure what this experiment was about. It seems that the idea was to evaluated how the availability of discourse information can improve the translation, but if that is so, then all three setups did have discourse level information (hence this evaluation is impossible). The only thing this seems to be doing is checking in which way the information should be presented (one sentence at a time in a chat, all sentences at once but clearly marked, or the entire document at once without sentence boundaries).", "questions_to_authors": "Question A: Did you perform any significance testing, visualize the scores, inspect the trends, and analyze the data and its distribution beyond looking at the  average scores?\n\nQuestion B: How many evaluators were employed? How much were they compensated? How many documents were evaluated (only for one task it is mentioned that 100 were sampled)? Was there only one rating per document? Who were these people?\n\nQuestion C: In line 361 you write that it is debatable whether these benchmarks can be considered appropriate, I agree, but I don't see this discussed anywhere apart from takeaways.\n\nQuestion D: How were COMET/BLEU/TER scores computed for P3 in table 2? Since later for this prompt only d-BLEU is used due to the lack of sent-to-sent alignments.", "paper_topic_and_main_contributions": "In this paper, the authors investigate the capabilities of large language models, namely GPT-3.5-turbo (chatGPT) and GPT-4, to leverage contextual information. Using a document-level translation task as a testbed, the authors compare the performance of GPT-3.5 and GPT-4 against (1) other commercial systems (GoogleTranslate, DeepL, and Tencent's MT), and (2) publicly available MT systems capable of translating entire documents. The authors study three translation directions: Chinese to English, English to German, and English to Russian.\n\nThe authors employ document-level BLEU (d-BLEU), TER, and COMET scores to evaluated the outputs, with d-BLEU being their main choice due to the lack for sentence-to-sentence alignments. They supplement these with a simplified human evaluation, based on two properties rated on a 6-point scale (\"discourse awareness\" and \"overall quality\"). In addition, the authors examine the ability of large language models to distinguish correct from incorrect translations, specifically where context is crucial (zero-pronouns, discourse deixis, etc.), using a dataset with contextual errors where there is one correct and multiple incorrect translations. The authors also claim that they investigate the effects of different training methods on the model's ability to exploit discourse-level information.\n\nThe three main experiments are:\n\n(1) \"Discourse-aware prompts\" -- Here the authors use 3 prompts (sentence by sentence in chat conversation, entire text with marked sentences, and just the entire text) in order to choose which one works best. The output is evaluated using automatic metrics (BLEU, d-BLEU, COMET, and TER).\n\n(2a) Comparison of chatGPT/GPT-4 with commercial models -- Here the authors employ the best prompt for GPT-3.5-turbo and GPT-4 and evaluate their performance against three commercial systems (GoogleTranslate, DeepL, Tencent TranSmark). For this experiment two datasets published in 2022 are being used. The outputs are evaluated with d-BLEU and simplified human evaluation (6-point scale);\n\n(2b) Comparison of chatGPT/GPT-4 with document-level NMT models -- Here the authors compare the GPT models with five document-level NMT models. The models are tested on datasets which were published over the years the newest being from 2018. The outputs are evaluated using BLEU and d-BLEU. \n\n(3a) Analyzing the capabilities of LLMs to process discourse-level information -- Here the authors employ an existing dataset with translations containing contextual errors published in 2019. The task is to ultimately favor the correct translation over multiple incorrect candidates. The two GPT models are compared with NMT models. In case of GPT models, the authors prompt the models to indicate the correct translation and produce an explanation as to why it is correct. For the other models the authors use forced decoding. The authors report accuracy for each of these models. For the GPT models the authors also report human evaluation of 100 examples where both the choice and explanation were assigned a good or bad label (binary choice).\n\n(3b) Effect of training methods on the ability to model/process contextual information -- Here the authors test different GPTs models (including InstructGPT and Codex) on the translation task. It appears that the datasets used are from 2022 and 2018 but I believe this is not made clear. The authors then use d-BLEU and simplified human evaluation (6-point scale).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Several papers have explored the capabilities of LLM in terms of MT. At least the following two should be mentioned as they directly relate to the research done by the authors:\n\n(1) Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit Iyyer. 2022. Exploring document-level literary machine translation with parallel paragraphs from world literature. (paragraph-level post-editing with LLMs)\n\n(2) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. (includes document level MT in a setup that resembles P2 setup from this study).\n\nSome other papers would include: Vilar et al., 2022; Guerreiro et al., 2023, Kocmi and Federmann, 2023 (this is about mt-eval with llms)", "Typos_Grammar_Style_and_Presentation_Improvements": "I believe that we should avoid talking about \"asking model\", model's ability \"for comprehending\", and \"ability to document-level awareness\". This sounds as if the model had ability to comprehend and be aware, while I believe that the model processes text, and leverage information (rather than \"is aware\" or \"understands\"). \n\nline 299: discrepancy is repeated\n\nIf possible, the exact date for when GPT-4 and chatGPT (GPT-3.5-turbo) were used should be provided, given that these two models receive constant updates, sometimes more often than once a month.\n\nIs the GPT-3 model davinci-002? This information seems to be missing.", "Justification_for_Ethical_Concerns": "No important details were reported about the human evaluation. We don't know how much the human evaluators were paid, whether the study was IRB-reviewed, who they were or whether the provided a consent to take part in this study."}}]

Author rebuttal:

Meta-review: In this work, the authors perform document-level translation using ChatGPT and GPT-4, and present a thorough comparison of these LLMs with state-of-the-art MT systems on a benchmark evaluating translation quality going from Chinese to English, English to German and Englishto Russian. They also probe which parts of the LLM training pipeline (fine-tuning, reward modeling, etc.) contribute most to its document-level translation capabilities and experiment with different discourse-aware prompts to identify what works best.

After detailed discussions with the authors during the rebuttal phase, the reviewers are all unanimous in their ratings for soundness (4/4/4) and excitement (4/4/4). R3 pointed to many updates that the authors should make in their revised draft, including details of human evaluation criteria, citations to relevant references and to be more precise about the impact of different training techniques.
Decision: Accept-Main