Challenges in Context-Aware Neural Machine Translation
Authors: Linghao Jin
Jacqueline He
Jonathan May
Xuezhe Ma
Venue: EMNLP 2023 Main (2023)
Keywords: neural machine translation
document-level neural machine translation
context-aware neural machine translation

Abstract: Context-aware neural machine translation, a paradigm that involves leveraging information beyond sentence-level context to resolve inter-sentential discourse dependencies and improve document-level translation quality, has given rise to a number of recent techniques. 
However, despite well-reasoned intuitions, most context-aware translation models show only modest improvements over sentence-level systems. In this work, we investigate and present several core challenges that impede progress within the field, relating to discourse phenomena, context usage, model architectures, and document-level evaluation. 
To address these problems, we propose a more realistic setting for document-level translation, called paragraph-to-paragraph (PARA2PARA) translation, and collect a new dataset of Chinese-English novels to promote future research.

Challenges in Context-Aware Neural Machine Translation
Linghao Jin Jacqueline He? Jonathan May'
Xuezhe Ma'
IIffrmmaion Sciences Institute, University of Southern California
UUiiverstty of Washington (linghaoj, jonmay, xuezhema edu
jyyhecs. washington. edu
Abstract
translation continues surpass that of MT sys- tems (Läubli et al., 2018), thus underscoring the need for incorporating long-range context.
Context-aware neural machine translation, paradigm that involves leveraging information beyond sentence-level context to resolve inter- sentential discourse dependencies and improve document-level translation quality, has given rise to a number of recent techniques. How- ever, despite well-reasoned intuitions, most context-aware translation models yield only modest improvements over sentence-level sys- tems. In this work, investigate and present several core challenges, relating to discourse phenomena, context usage, model architectures, and document-lewe evaluation, that impede progress within the field. To address these prob- lems, we propose more realistic setting for document-level translation, called paragraph- to-paragraph (PARA2PARA) translation, and collect dataset of Chinese-Fnglish novels to promote future research.'
Despite some efforts to meaningfully exploit inter-sentential information, many context-aware (or interchangeably. document-level) NMT systems only show meager gains across sentence-level and document-level translation metrics (Tiedemann and Scherrer, 2017; Miculicich al., 2018; Müller et al., 2018; Tu al., 2018; Maruf et al., 2019; Lupo et al., 2022a,b; Wu et al., 2022). Perfor- mance improvements against sentence- base- lines on overall translation accuracy, pronoun reso- lution, or lexical cohesion become less pronounced when context-aware systems are trained real- istic, high-resourced settings (Lopes al., 2020), casting doubt on the efficacy of such approaches. this paper, we conduct thorough empirical analysis and present some key obstacles that hinder progress in this domain:
1 Introduction
Neural machine translation (NMT) has garnered considerable scientific interest and commercial suc- cess in recent years, with current tate-of-the-ane art systems approaching exceeding human quality for few resource-rich languages when translat- ing individual sentences (Wu al., 2016; Has- san et al., 2018; Yang al., 2020). Despite the strong empirical performance of such systems, the independence assumption that underlies sentence- level NMT raises several issues. Certain textual elements, such as coreference (Guillou and Hard- meier. 2016), lexical cohesion (Carpuat, 2009), lexical disambiguation (Rios Gonzales al., 2017) are impossible correctly translate with- out access to linguistic cues that exist beyond the present sentence (Sim Smith, 2017). When evalu- ating documents rather than individual sentences, the adequacy and fluency of professional human
1. Existing document-level corpora contain sparse number of discourse phenomena that require inter-sentential context to be accu- rately translated.
2. Though context is necessary for pronoun reso- lution and named entity consistency, it is less helpful for tense and discourse markers.
The sentence-level Transformer baseline al- ready performs up to par with concatenation- based NMT settings.
4. Advanced model architectures do not mean- ingfully improve document-leve' translation on existing document-level datasets.
5. Current metrics designed for document-level translation evaluation do not adequately mea- sure document-level translation quality.
Equal contribution.
WWe release the paper's code and dataset here: https: //github com/L inghao Sacaaao-Jnn/aaa hiin/aant-challeeee challenges.
The above findings suggest that paragraph-to- paragraph (PARA2PARA) translation, wherein a

---

document translated at the granularity of para- graphs, may serve more suitable and realistic setting for document-level translation, which in practice is unencumbered by sentence-level align- ments. To this end, we develop and release a Chinese-Englise dataset,
'he two-to-two (2-2) setup incorporates additional context from the previous souree sentence (C; { 3/i 9)-1)))... The target context is integrated the same manner as in one-to-two.
In order to investigate the importance of con- text after the current sentence, we also explore a three-to-one (3-1) setting, wherein introduce additional source-side context by concatenating the previous and subsequent sentences to the current one (C; (œ--1, r++1) )) and do not incorporate any target context.
new paragraph-aligned consisting of 10,545 parallel paragraphs harvested from 6 novels within the public domain, in order to spur future research.
2 Background
The high-level objective of sentence-level machine translation is to model the sentence-level condi- tional probability P(y[æ), which the source =
Related Work
and target sentences æ (V'...,"')) are textual sequences of respective lengths M and N. Under the dominant paradigm of neural machine translation (Sutskever et al., 2014), the conditional probability Po(y) (z) is typically de- composed into the following auto-regressive for- mulation (with 0 denoting parameterized weights):
......M,,
3.1 Model Architectures
Recent progress in context-aware NMT generally falls along two lines: multi-encoder approaches and concatenation- ones (Kim et al.. 2019).
Under the first taxonomy, additional sentences are encoded separately, such that the model learns an internal representation of context sentences in- dependently from the current sentence. The inte- (I)
N II (,
gration of context and current sentences can oceur either prior to being fed into the decoder (Maruf and Haffari, 2018; Voita et al., 2018; Miculicich et al., 2018; Zhang et al., 2018; Maruf et al., 2019), 9), or within the decoder itself (Bawden et al., 201 88; Cao and Xiong, 2018; Kuang and Xiong, 2018: Sto- janovski and Fraser, 2018; 1; Tu et al., 2018; Zhang et al., 2018). The effectiveness of these multi- encoder paradigms is subject to debate; in stan- dardized analysis, Li et al. (2020) finds that rather than effectively harnessing inter-sentential informa- tion, the context encoder functions more as noise generator that provides richer self self-training signals, since even the inclusion of random contextual input can yield substantial translation improvement. In addition, Sun et al. (2022) finds that BLEU-score improvements from context-aware approaches of- ten diminish with larger training datasets or thor- ough baseline tuning.
Equation 1 implies that when predicting the target token y², the model could only access the current source sentence, x, as well as all previously trans- lated tokens in the current target sentence, yh Translating sentences in a document in such an isolated fashion, without any extra-sentential infor- mation that lies beyond sentence boundaries, has been found produce syntactically valid, but se- mantically inconsistent text (Läubli et al., 2018).
To remedy this, contexi-aware neural machine translation considers a document 1) that entails a set of logically cohesive source sentences X {x), D2 T.), and a parallel set of target sen- tences Y = [y1, yy,,.., ya}. Under a left-to-right translation schema, the model computes the prob- ability of translating the source sentence æ; con- ditioned on the context C₁ wherein 0 I i d:
Po(yi C+) II Polod l M C.). j=1
(2)
On the other hand, concatenation--aased NMT approaches are conceptually simpler and have been found to perform on par with or better than multi- encoder systems (Lopes al., 2020; Ma et al., 2021). Under this paradigm, context sentences are appended the current sentence, with special tokens mark sentence boundaries, and the con- catenated sequence is passed as input through the encoder-decoder architecture (Ma et al., 2020).
In practice, there are multiple ways to formulate C. Passing in C. = = reduces to the sentence- level case (1). Throughout this paper, explore concatenation-bate based setups first presented by Tiedemann and Scherrer (2017). The one-to-two (1-2) setup prepends the preceding target sentence to the current target sentence (C;- = {) de- noting sentence boundaries with a <SEP> token.

---

3.2 Datasets
ing ContraPro (Müller et al., 2018) in English-to- German, Bawden (Bawden et al., 2018) in English- to-French, and Voita (Voita et al.. 2019) in English- to Russian translation. Though targeted, these test sets tend to be small, and are constricted to par- ticular language pair and discourse phenomenon.
Until recently, the bulk of context-aware NMT re- search has focused on document-level, sentence- aligned parallel datasets. Most commonly used corpora, including IWSLT-17 (Cettolo et al., 2012), NewsCom (Tiedemann, 2012), Europarl (Kochn, 2005), and OpenSubtitles (Lison al., 2018) are sourced from news articles or parliamentary pro- ceedings. Such datasets often contain a high vol- ume of sentences that is sufficient for training sentence-level NMT systems, yet the number of documents remains comparatively limited. 2
4 Challenges
We identify key obstacles that account for the lack of progress in this field, based on careful em- pirical analysis over a range of language pairs, model architectures, concatenation schemas, and document-level phenomena."
In an attempt to address the scarcity of document- level training data, recent works have developed datasets that are specifically tailored for context- aware NMT. Jiang et al. (2023) curated Bilingual Web Books (BWB), a document-level parallel cor- pus consisting of 9.6 million sentences and 196 thousand documents (chapters) sourced from En- glish translations of Chinese web novels. Thai et al. (2022) introduced PAR3, multilingual dataset of non-English novels from the public domain, which is aligned at the paragraph level based on both hu- man and automatic translations. Using automatic sentence alignments, Al Ghussin al. (2023) ex- tracted parallel paragraphs from Paracrawl (Bañón et al., 2020), which consists of crawled webpages.
4.1 Discourse phenomena is sparse in surrounding context.
Contextual sparsity is a bottleneck to document- level neural machine translation that manifests in two forms (Lupo et al., 2022a). First, the majority of words within sentence be accurately trans- lated without additional access inter-sentential information: context poses as a weak training signal and its presence has not been found to substantially boost translation performance. Second, only a few words in neighboring sentences may actually con- tribute to the disambiguation of current tokens at translation time.
3.3 Evaluation
We investigate contextual sparsity via a fine- grained analysis on the BWB (Jiang et al., 2022) test set, which has been manually tagged with spe- cific discourse-level phenomena.* Specifically, we use it to probe NMT models' ability to exploit long-range context by analyzing the frequency of particular discourse phenomena that can only be resolved with context.
addition to metries that evaluate sentence-level translation quality, e.g., BLEU (Papineni et al., 2002) and COMET (Rei et al., 2020), num- ber of automatic metrics designed specifically for document-level MT have been recently proposed. Jiang et al. (2022) introduced BlonDe, document- level automatic metric that calculates the similarity- based Fl measure of discourse-related spans across four categories. Vernikos et al. (2022) show that pre-trained metrics, such as COMET, can be ex- tended to incorporate context for document-level evaluation. To measure the influence of context usage in context-aware NMT models, Fernandes et al. (2021) proposed Context-aware Cross Mutual Information (CXMI), a language-agnostic indicator that draws from cross-mutual information.
For the manual analysis, we randomly sample 200 discourse-annotated annotated instances from the test set and ask bilingual annotators who are fluent in Chinese and English to identify and count in- stances that contain a particular context-dependent discourse phenomenon. Annotators are asked to discern if the following document-level discourse phenomena exist in each sentence pair:
Another approach to document-level MT eval- uation focuses on hand-crafted contrastive evalua- tion sets to gauge the model's 's capacity for captur- ing inter-sentential discourse phenomena, includ-
Pronoun Ellipsis: The pronoun is dropped in Chinese, but must be included in the English translation.
UUnless otherwise specified, experiments are conducted using Transformer (Vaswani et al., 2017) as default archi- tecture. Full training details are Appendix A. "Examples of annotated paragraphs are in Appendix A.4.
As an example, the IWSLT-17 (Cettolo et al., 2012) EN-FF split contains 239854 sentences and 2556 docu- ments.

---

Lexical Cohesion: The same named entity must be translated consistently across the cur- rent sentence and context sentences.
Discourse Phenomena
Freg.
d=1 (%)
d=2 (%)
d=3 (6)
d> (9)
Ellp. Promoun
76.6
12.5
7.8
3.1
Lexical Cobesion
56.6
23.2
7.1
Tense
100.0
0.0
0.0
0.0
Ambiguity
90.1
9.9
0.0
0.0
Tense:
Tense information that can be omitted in Chinese, and must be inferred based on context to be correctly translated in English.
Discourse Marker
1000
No Conteal
Table I: Frequency of context-dependent discourse phe- nomena in 200-count sample of the BWB test set, and the percentage of cases where relevant context can be found distance d 1, 2, 3, > 3 sentences.
Instances in which an ambiguous word or phrase the current sentence requires context to be correctly translated.
Ambiguity:
Discourse Marker: A discourse marker, e.g., while, as long as, else, that is not explicit in Chinese, but must be pragmatically inferred and present in English.
603
350
400
150
200
400
290
230
303
200
130
200
150
200
300
Table indicates that lexical cohesion (83.2%) and pronoun ellipsis (53.8%) constitute the majority of discourse phenomena found in the 119 sentences that require nnter-sentential signals for correct trans- lation. In contrast, other categories- tense (4.2%). ambiguity (9.2%) and discourse marker (16.8%) oceur much less frequently.
50
88
Sont.
Scnt.
Sent. length
Figure 1: Sentence length distributions on test sets.
BWB test set and show that this premise does not always hold; while pronoun resolution or named entity consistency is often better resolved with the incorporation of context, tense and discourse mark- ers are relatively insensitive to context and yield meager improvement.
We next examine how far the useful context tends to be from the cross-lingually ambiguous sentence. Taking d as the sentence distance, the majority of discourse phenomena can be disam- biguated based on the nearest context sentence (d=1). Specifically, the necessary information for tense, umbiguity, and discourse markers can almost always be found by d=1, whereas relevant context for pronoun ellipses and lexical cohesion tends to be more spread out. Hardly any useful information can be found in very distant context (d>3).
4.2.1 Pronoun Resolution
We examine two types of pronoun translation: pro- noun ellipsis and anaphoric resolution.
Pronoun ellipsis. As Chinese is a pro-drop lan- guage, pronouns can be freely omitted and are im- plicitly inferred from surrounding context. In con- trast, grammatical and comprehensible translation into English requires that the pronoun be made plicit. To test concatenation- based NMT systems' ability to resolve Chinese-Engllit pronoun ellip- sis, we conduct inference on subset BWB that contains 519 instances of pronoun ellipsis.
A significant fraction (40.5%) of sentences in the sampled test set can be translated independently. i.e., without access inter-sentential information. Correspondingly, we notice that many sentences across document-level data are not lengthy with discourse-level phenomena, but rather simple con- structions. Figure indicates that the majority of sentences are relatively short in BWB and IWSLT- 17, ranging from 20-50 characters (Chinese) or 10-30 words (French and German).
Table 2 indicates that the disambiguation of pro- noun ellipsis is particularly responsive to context. Incorporating single target-side context sentence (the 1-2 setting) improves the BlonDe F1-score from 55.88 to 63.91; adding another source-side context sentence (the 2-2 setting) marginally im- proves to 65.91. In this scenario, more source-side context may carry useful information, as the 3-1 setting performs the best overall on BlonDe (66.06).
4.2 Context does not help disambiguate certain discourse phenomena.
An implicit assumption in context-aware NMT is that the inclusion of the proper context would influ- ence the model leverage it resolve any poten- tial discourse ambiguities. To this end, investi- gate different types of discourse phenomena on the EExamplss for each DM category are in Appendix A.4,
Anaphoric resolution. When translating to lan- guages that contain grammatical gender, anaphoric pronouns form another instance of cross-lingual

---

Th-tEn
De
En->Fr
Type
contrast
cause
cond.
conj.
(a)syn.
Count
2042
624
361
226
123
705
Setting
BLONDE
CONTRAPRO
BAWDEN
1-1
55.68
58.97
40.99
71.68
47.15
56.59
1-2
55.39
$7.05
37.12
70.80
52.03
57.51
1-1
55.88
57.75
50.00
2-2
54.99
57.05
37.12
70.80
51.21
56.79
1-2
63.91
55.45
64.00
3-1
53.57
59.97
37.12
65.48
43,90
54.46
2-2
65.91
69.74
72.00
3-1
66.06
Table 4: Accuracy across discourse marker categories and concatenation settings on the BWB test set.
Table 2: BLONDE evaluation of pronoun translation on the BWB test subset and accuracy for anaphoric pronoun resolution on CONTRAPRO and BAWDEN. The 3-1 selting requires the surrounding context sentences, and therefore cannot be applied to contrastive sets.
Consistency. We extract 780 examples (705 per- son entities, 75 non-person entities) to construct consistency test subset. Each instance includes Sentence with named entity that is also mentioned in the preceding sentence. We then measure the frequency which different context- aware transla- tion models could consistently translate the entity across the two consecutive sentences.
Consistency (%)
Acc. (%)
Setting
Person
Non-person
Person
1-1
32.34
14.67
54,55
1-2
49.36
21.33
51.96
According to Table 3. this task proves to be challenging no system achieves above-random performance-buut the presence of context facil- itates consistency as each context-aware setting performs better than the 1-1 baseline on person entities (32.34%). Adding target-side context (1-2 and 2-2 settings) appears strictly more helpful. By contrast, source-side context (3-1 setting) results in marginal performance gains relative to the baseline.
2-2
45.53
14.67
52.42
3-1
36.17
17.33
51.15
Table 3:
Named entity analysis consistency and accuracy on relevant samples from the BWB test set.
ambiguity. For example, when translating into Ger- man, the English pronoun can become either es, sie, or er, depending on the grammatical gender of its referent.
Accuracy. To explore the frequency at which named entities are accurately translated, next examine the 1734 person entities from the BWB test set. Surprisingly, the sentence-level model is better than context-aware aware models at correctly trans- lating named entities, with the best accuracy of 54.55% (Table 3). While context is important for ensuring named entity consistency, these findings suggest that adding context may introduce addi- tional noise and do not necessarily lead more accurate translations. We hypothesize that the de- pendency on context might hurt the model's down- stream performance when the NMT model tries to be consistent with the context translation, which re- sults in propagation of errors down the sequence. In addition, when comparing all the results us- ing the entity category in BlonDe across the three language pairs in Table and Table 6, becomes clear that additional context does not meaningfully increase the accuracy of named entity translation.
Thus, we also conducted experiments from En- glish to German (En->De) and French (En->Fr), both grammatically gendered languages, and evalu- ated on the contrastive sets ControPro (Müller et al.. 2018) and Bawden (Bawden et al., 2018), 8), respec- tively. While Table 2 shows steady improvement for anaphoric resolution ContraPro, curiously, the 1-2 concatenation-basee model exhibits slight dip compared its sentence-level counterpart on Bawden. We hypothesize that the small size (200 examples) of the Bawden dataset causes the signifi- cant variance in the results.
4.2.2 Named Entities
Named entities-real-vee objects denoted with proper names-are domain-specific and low- frequency, and thus tend to be absent from bilingual dictionaries (Modrzejewski et al., 2020). Their translations are often either inconsistent (c.g.. different target translations for the same source phrase) or inaccurate (with regards to some target reference). In this section, we examine for named entity consistency and accuracy on the annotated BWB test set.
4.2.3 Discourse Marker and Tense
Discourse makers. The omission of discourse markers (DM) particles that signal the type of co- herence relation between two segments (Grote and

---

BLEU
BlonDe
COMET
all
pron.
entity
tense
d.m.
1-1
20.80g.20
38.380.38
72.931.02
53.171.66
73.29b.14
60.03g.sp
0.24190.01
1-2
19. 19.17
35.770.28
74.481.76
43.34
70.701.15
1..67
0.221 10.01
XFMR
2-2
0..45
37.630.40
76.540.27
48.092.63
72.930.35
59.86p.31
0.24350.01
3-1
19.87
37.620.42
76.590.32
49.762.61
72.610.14
59.070.49
0.22590.00
1-1
20.600.07
o.12
23.08
49.561.20
73.430.27
60.32g.3u
0.24030.00
1-2
20.32..30
36.680.50
73.56u.16
46.041.93
73.170.22
60.35g.49
0.2378p.01
MEGA
2-2
20.340.27
36.740.76
73.83p.55
48.786.80
73.39p.27
60. 10.333.36
0.23540.01
3-1
19.87
37.520.38
76.620.49
49.321.56
72.650.06
59.230.23
0.22999.01
Table 5: Automatic metric results BWB (Zh-EEn) across different architectures (XFMR and MEGA) and concatenation settings (1-1, 1-2, 2-2, and 3-1). We report average and standard deviations across three runs.
Stede, 1998), e.g., so, because, for this reason requires context awareness when translating from morphologically poorer languages to morphologi- cally richer ones. Following Jiang et al. (2022), we separate DMs into five categories: contrast, cause, condition, conjunction, and (a-Jsynchronous, and examine how different context-aware settings fare with each discourse relation.
thetic language yields marginally higher BlonDe scores, at over 75. One reason that the BlonDe score for tense may be relatively inflexible across language pairs is that most sentences from the cor- pora generally adhere to a particular tense, such past tense in literature, thus diminishing the neces- sity of context.
4.2.4 Is source or target context more helpful?
As Table shows, the sentence-level (1-1) base- line performs the best across discourse markers in aggregate, and across the cause and condition cat- egories. The incorporation of context does not significantly improve the accuracy of discourse marker translation; interestingly, the 3-1 setting fares poorly, with the lowest performance across all categories except on contrast DMs.
Fernandes et al. (2021) finds that concatenation- based context-aware NMT models lean target context more than source context, and that incorpo- rating more context sentences on either side often leads to diminishing returns performance.
However, according to Table 2-6, this is not uni- versally the case; the effectiveness of target-side versus source-side context is largely dependent on the language pair. Though target-side context often helps with translation consistency, such as preserv- ing grammatical formality across sentences, does not necessarily guarantee a better translation qual- ity than source-side context (e.g., the 3-1 setting performs best on pronoun translation for French and German according to Table 6, and pronoun ellipsis for Chinese in Table 2).
Tense. Tense consistency another extra- sentential phenomenon that requires context for disambiguation, particularly when translating from an analytic source language (e.g., Chinese) to a synthetic target language (c.g., English), wherein tense must be made explicit.
From experimental results the BWB (Ta- ble 5) and IWSLT (Table 6) data, there is min- imal variance across all translation settings in the BlonDe scores for tense and DM, suggesting that context is not particularly conducive for any lan- guage pair. Tense is generally consistently resolv- able, with all models surpassing 70 on Zh- As expected, translating from French- more syn-
4.3 The context-agnostic baseline performs comparably to context-aware settings.
Experimental results across both the BWB (Ta- ble 5) and IWSLT-17 (Table 6) datasets demon- strate that vanilla 1-1 baseline performs par with, or even better than its context-aware coun- terparts on the sentence-level automatic metrics, BLEU and COMET. This suggests that, due problems with common document-level datasets (e.g., relative lack of contextual signals) ($4.1)
'In analytic languages, concepts are conveyed through root/stem words with few affixes. Synthetic languages use numerous affixes to combine multiple concepts into single words, incurring higher morpheme-to-word ratio (O' O''rray et al., 1997).
WWe train on the IWSLT dataset in reverse order (Fr+FRn and Dc)+En) in order to evaluate with BlonDe.

---

BLEU
BlonDe
COMET
all
pron.
entity
tense
d.m.
1-1
34.93n.15
52.22p.60
71.640.29
64.70
75.91
77.78p.68
0.47940.01
XFMR
3-1
15.37
52.88o.44
75.42
67.182.87
76.09..40
78.59
0.49499.02
Fr
1-1
35.00p.72
51.271.34
68. 16.56
63.675.71
75.09%.05
77.530.65
0.45066.02
MEGA
3-1
36.03p.25
53.376.19
77.733.55
64.884.53
76.38p.17
7..2
0.5095..02
1-1
30.00
47.76p.17
70.801.75
65. 1..05
71.58p.64
75.720.38
0.3250p.00
XFMR
3-1
30.60p.26
48, 48.230.31
76.210.49
59.44
72.450.44
75.51..25
0.3540p.01
De
1-1
30.860.25
48, 48.48
72.523.48
67.52
73.462.03
75.98o.70
0.34000.01
MEGA
3-1
31.211..7
49 49.22..10
76.102.88
68.484.20
72.470.27
75.48
0.3563
Table 6: Automatic metric results on IWSLT-17 (Fr-En and De-En), on different architectures (XFMR and MEGA) and concatenation settings (1-1 and 3-1). We report average and standard deviations across three runs.
and the inability of sentence-level metrics to cap- ture document-level attriibtes, context-aware mod- els do not exhibit a meaningful improvement over context-agnostic models at the sentence level. In terms of document-level improvement, the sentence-lever baseline even outperforms context- aware models in select instances, such when translating named entities (53.17% on Zh, 65. 11% on De). There are no notable differences in han- dling tense and discourse markers across contex- tual settings, which aligns with our observations in $4.2.3. These results demonstrate that on com- monly used datasets, context-aware models also do not significantly improve document-level transla- tion over sentence-level Transformer baseline.
level metrics, BLEU and COMET. At the document level, MEGA has the highest overall BlonDe FI- score when translating from both German (53.37 vs. 52.88) and French (49.22 vs. 48.23). While MEGA tends to outscore XFMR on the pronoun and entity categories, there is no significant improvement, if any for tense and discourse marker. Furthermore, MEGA usually starts from higher sentence-level baseline (except pronoun resolution for Fr-+En); when moving from the sentence-level to the con- textual 3-1 setting, MEGA does not show higher relative gains than XFMR.
One potential explanation as to why MEGA per- forms better on automatic metrics is because it is a stronger model and better translation overall (Ma et al., 2023), rather than it being able to leverage context in more useful manner. The lack of im- provement in particular discourse categories does not necessarily indicate that existing context-aware models are incapable of handling long-range dis- course phenomena. Rather, it suggests that current data may not sufficiently capture the complexities in such situations. As discussed, discourse phenom- ena are sparse; some of them could not be resolved even with necessary context.
4.4 Advanced model architectures do not meaningfully improve performance.
Motivated by the limitations of the self-attention mechanism on long-range dependency model- ing (Tay et al., 2022), recent work has proposed more advanced architectures to better leverage con- textual signals into translation (Lupo et al., 2022b; Sun al., 2022; Wu al., 2022, 2023). The hy- pothesis is that long-range sequence architec- tures can effectively model longer context windows, they are better-equipped to handle the lengthier na- ture document-leve translation.
This finding aligns with similar work (Sun et al., 2022; Post and Junczys- Dowmunt, 2023) which also propose that, on existing datasets and under current experimental settings that use sentence- level alignments. the standard Transformer model remains adequate for document- translation.
To test this theory, we replace the Transformer (XFMR) attention mechanism with a recently intro- duced MEGA architecture (Ma al., 2023), which overcomes several limitations of the Transformer on long-range sequence modeling. As Table shows, MEGA always performs better than XFMR across both the 1-1 and 3-1 settings on the sentence-
4.5 There a need for an appropriate document-level translation metric.
Though BLEU and COMET are both widely used for sentence-level machine translation, they pri- marily focus on assessing sentence-level transla-
We refer to Appendix A.2 for a more detailed discussion on MEGA's design.

---

We never back aguin, for the past is still too close.. The things we have tried to forect would stir again and that sense of fear building up the blind unreasoning panie. now mercifully stilled might once again become.. living companion,
* + A
H RE it 22 E PEETEE
IE FER DE
Source panagragh from Reberca (Diphne du Maunier) in English
Corresponding haman translaned paragraph in Chiness
An example of parrappg-p-- translation. Aligned sentences are underlined in the same color. Highlighted parts are added by translators and do not have a corresponding source segment.
Figure 2:
tion quality, and do not adequately encapsulate discourse-level considcrations. Contrastive sets are a more discourse-oriented means towards evalu- ating document-leve translation quality, but they too contain shortcomings. First, contrastive sets are not generalizable beyond particular discourse phenomena and language pair, and the curation of these sets is both time- and labor-intensive. Further- more, contrastive sets evaluate in a discriminative manner- by asking the model to rank and choose between correct and incorrect translation pairs which is at odds with, and does not gauge, the MT model's generative capacity. Post and Junczys- Dowmunt (2023) (concurrent work) proposes gen- erative version of contrastive evaluation, and finds that this paradigm able make finer-grained distinction between deeu NMT systems,
PARA2PARA Translation
recurrent theme throughout our analyses is that existing datasets are not conducive to meaning- ful context usage in document-level translation. The majority of datasets used in the literature of document-level NMT are aligned at the sentence level, which is artificial in design and not reflective of how documents are translated in practice.
As such, paragraph-level parallel data (Figure 2) may be more suited for document-level NMT and provide richer contextual training signals. Recent work have turned toward literary translation as a challenging, realistic setting for document-level translation (Zhang and Liu, 2020; Thai et al., 2022; Karpinska and lyyer, 2023), given that literary texts typically contain complex discourse structures that mandate document- frame of reference. As Figure illustrates, sentence alignment not al- ways feasible when translating literature. Karpin- ska and lyyer (2023) finds that language models can effectively exploit document-level context and cause fewer discourse-level translation errors based on human evaluation, when the paragraph is taken as the minimal discourse-level unit.
The recently proposed BlonDe (Jiang et al., 2022) score, which calculates the similarity mea- sure of discourse-related spans in different cat- egories, is first step towards better automatic document-level evaluation. However, BlonDe re- quires the source language's data to be annotated with discourse-level phenomena, and its applica- bility is restricted language pairs in which the target language is English.
To promote future rescarch on document-lovel translation in a realistic setting, we collect profes- sional English and Chinese translations of classic novels, and format the data by manually correct- ing paragraph-level alignments. The PARA2PARA dataset consists of 10,545 parallel paragraphs across six novels from the public domain. 9 To our knowledge, the only other paragraph-aligned, parallel dataset sourcing from the literary domain is PAR3 (Thai et al., 2022), which uses Google Trans- late and fine-tuned GPT-3 (Brown et al., 2020) to automatically generate reference translations. In
Finally, incorporating pre-trained models into metrics another promising direction. To this end, Vernikos et al. (2022) present a novel approach for extending pre-trained metrics such as COMET to incorporate context for document-level evaluation, and report better correlation with human prefer- ence than BlonDe. Nevertheless, the incorporation of pre-trained models raises the issue of metric interpretability, yielding opaque numbers with no meaningful linguistic explanations. Thus, we note the need to develop more robust, automatic, and interpretable document-level translation metrics.
Data and preprocessing details are in Appendix B.I.

---

Domain
Pre-training
BLEU
BlonDe
COMET
all
pron.
entity
tense
d.m.
NONE
1.37p.06
8,440.39
47 27.282.18
18.736.19
41.221.48
16.873.60
0.3949..0
XFMR
16.00p.10
35.36p.19
79.16u.
12.330.10
72.470.
60.63o.12
0.73390.00
Closed
LIGHTCONV
16.877.06
36.70p.14
79.28).28
55,38
72.80p.0a
61.66p.61
0.74090.00
MBART25
15.63p.25
35.370.20
78.72..27
54.04g.49
72.01o.12
60.599.62
0.73850.00
NONE
0.73p.32
1.820.21
48. 125554
0.00p.00
39.272.11
13.91 3,90
0.35870.02
XFMR Big
9.17
15.35
72.20
32.542.18
67.170.59
51.83
0.7003
Open
LIGHTCONV
8.60p.10
25.480.10
72.50..43
38.83
67.40g:35
51.790.92
0.7027d.00
MBART25
7.970.06
22.41o.71
72.240.61
20.074.17
67.25%.40
50.520.84
0.70120.00
Table 7:
Baseline translation results on the Zh PARA2PARA dataset. Bold denotes best performance.
contrast, the source and target paragraphs in our dataset are culled from professional translations." 10
issue of low-domain data; all of them exhibit a marked performance improvement over the NONE setting, attesting to the challenging constitution of paragraph-to--paaggra translation.
We then benchmark the dataset under two ex- perimental settings for Zh > En translation: i). a standard closed-domain setup, in which both the training and testing data are sourced from the same novels; ii). more challenging open-domain setup, wherein two novels are held and used as only the test set. We experiment with training a Transformer-basee based model on PARA2PARA data from scratch (NONE), as well as incorporating pre- trained baselines, which the model is first trained on the sentence-level WMT17 Zh-En dataset (Bo- jar et al., 2017), before further fine-tuning on the PARA2PARA data, using the following backbone architectures:
On the closed-domain setting, LIGHTCONV Big yields the highest score across all three metrics. Open-domain results are mixed: scores are lower across the board as this setting challenging. XFMR Big has the best BLEU and discourse marker Fi-score on BlonDe, although all pre-training baselines perform similarly. LIGHT- CONV Big performs the best on pronoun, entity, and tense on BlonDe and has the highest COMET score.
expected,
Conclusion
XFMR (Vaswani et al.. 2017), the Transformer-BIG.
Despite machine-humant parity at the sentence level, NMT still lags behind human translation on long collections of text, motivating the need for context- aware systems that leverage signals beyond the current sentence boundary. In this work, we high- light and discuss key obstacles that hinder momen- tum in context-aware NMT. We find that training signals that improve document-level discourse phe- nomena occur infrequently in surrounding context, and that most sentences can be accurately trans- lated in isolation. Another challenge is that context benefits the resolution of some discourse phenom- ena over others. A context-agnostic Transformer baseline is already competitive against context- aware settings, and replacing the Transformer's self-attention mechanism with a more complex long-range mechanism does not significantly im- prove translation performance. We also note the need for generalizable document-level translation metric. Finally, we make the case for paragraph- aligned translation, and release a new PARA2PARA dataset, alongside baseline results, to encourage further efforts in this direction.
LIGHTCONV Big (Wu et al., 2019), which replaces the self-attention modules in the Bransformer-BIG with fixed convolutions.
MBART25 (Liu et al., 2020), which is pre- trained on 25 languages the document level.
Table 7 shows preliminary baseline results on BLEU, BlonDe, and COMET. the NONE set- ting, the Transformer's relatively low performance and incoherent output underscores the difficulty of training from scratch on the PARA2PARA cor- pus, due to two reasons-the the inherent difficulty of training on paragraph-level, longer-sequence data, and the limited dataset size (especially rel- ative to that of sentence-level MT datasets). To disentangle the two factors, we report additional baselines that leverage pre-training to offset the
10 Another distinction is that Zh-En split in PAR3 sources from ancient novels in Classical Chinese (which different from the modern language) and consists 1320 paragraphs. Example translations are in Appendix B.2.

---

7 Limitations
lowship. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program con- tract #2022-22072200006. The views and conclu- sions contained herein are those of the authors and should not be interpreted as necessarily represent- ing the official policies, either expressed or im- plied, of ODNI, TARPA, or the U.S. Government. The U.S. Govermment authorized to reproduce and distribute reprints for govermmental purposes notwithstanding any copyright annotation therein.
Several limitations restrict the scope of this work. To begin, our choice of languages in this Enggish, Chinese, French, German-is non- exhaustive, and it is possible that our findings would not generalize well to scenarios that involve low-resourced languages or distant language pairs. In particular, significant portion of our investi- gation on discourse relations that necessitate con- text for proper disambiguation targets the Chinese- English BWB test set, which is the only public dataset that has been manually annotated with this type of information. Some of the discourse phe- nomena that consider may not occur fre- quently in other languages. While this work is pre- liminary step that sheds light on the current nature of data that drives context-aware neural machine translation, future directions could entail extending similar analysis to other languages or discourse phe- nomena (e.g., the disambiguation of deixis when translating from Russian to English (Voita et al., 2019)).
References
Yusser Al Ghussin, Jingyi Zhang, and Josef van Gen- abith. 2023. Exploring paracrawl for document-leww neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 1304- 1310, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Hcafield, Hicu Hoang, Miquel Esplà-Gomis Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ranírez-Sánchezz Elsa Sarria,, Marek Strelee, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acqui- sition parallel corpora. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555 4565, Online. Association for Computational Linguistics.
Another restriction is that this work only exam- ines concatenation-hasee architectures, which tend be conceptually simple, effective, and hence subject to widespread adoption in recent years (Fer- nandes et al., 2021). While the purported advan- tages of multi-encoder NMT models are mixed (Li et al.. 2020), for comprehensiveness. it would be insightful to examine whether they behave differ- ently relative to concatenation--uee based systems under experimental setup. Other potential avenues for exploration entail loss-based approaches context- aware neural machine translation, such as context discounting (Lupo et al., 2022b) or contrastive learning- schemas (Hwang et al., 2021).
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenom- ena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pa- pers), pages 1304-1313, New Orleans, Louisiana. Association for Computational Linguisties.
Lastly, although the PARA2PARA dataset may pose as a more natural setting for context-aware translation, it is considerably smaller than other document-level datasets. Given that the small scale of training data is a prevalent issue in context-aware neural machine translation (Sun et al., 2022), fu- ture efforts could focus on expanding this dataset (as is casier to source paragraph- aligned parallel translations in the wild than sentence-level ones) or moving beyond the literary domain.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang. Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logucheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Sec- ond Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169-214. 14, Copenhagen. Denmark. Association for Computational Linguis- tics.
A Acknowledgements
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voo Voss, Gretchen Krueger, Tom Henighan, Rewon Child,
We thank the anonymous reviewers for their con- structive feedback in improving this work. JH is supported by an NSF Graduate Research Fel-

---

Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
machine translation using coreference information. In Proceedings of the Sixth Conference on Machine Translation, pages I 1135-1 144, Online. Association for Computational Linguistics.
Yuchen Eleanor Jiang. Tianyu Liu, Shumingg Ma, Dong- dong Zhang, Mrinmaya Sachan, and Ryan Cotterell. 2023. Discourse-centric evaluation of document- level machine translation with a new densely anno- tated parallel corpus of novels. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 7853-7872, Toronto, Canada. Association for Com- putational Linguistics.
Qian Cao and Deyi Xiong. 2018. Encoding gated translation memory into neural machine translation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3042-3047, Brussels, Belgium. Association for Com- putational Linguistics.
Marine Carpuat. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evalu- ations: Recent Achievements and Future Directions (SEW-2009), pages 19-27, Boulder, Colorado. Asso- ciation for Computational Linguisties.
Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dong- dong Zhang, Jian Yang. Haoyang Huang, Rico Sen- nrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation met- ric for document-level machine translation. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1550-1565, Seattle, United States. Association for Computational Linguistics.
Mauro Cettolo. Christian Girardi, and Marcello Fed- erico. 2012. WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation, pages 261-268, Trento, Italy. European Association for Machine Translation.
Marzena Karpinska and Mohit lyyer. 2023. Large lan- guage models effectively leverage document-leve context for literary translation, but critical errors per- sist. In WMT.
Patrick Fernandes, Kayo Yin, Graham Neubig, and An- dré F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6467-6478, Online. Association for Computational Linguistics.
Yunsu Kim, Duc Thanh Tran, and Hermann Ney. 2019. When and why is document-level context useful in neural machine translation? In Proceedings of the Fourth Workshop on Discourse in Machine Trans- lation (DiscoMT 2019), pages 24-34, Hong Kong, China. Association for Computational Linguistics.
Brigitte Grote and Manfred Stede. 1998. Discourse marker choice in sentence planning, In Natural Language Generation, Niagara-on-the Lakee, Ontario, Canada. Association for Computational Linguisties.
Philipp Koehn, 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86. Phuket, Thailand.
Liane Guillou and Christian Hardmeier. 2016. PROTEST: A test suite for evaluating pronouns in machine translation. Proceedings of the Tenth In- lernational Conference on Language Resources and Evaluation 16). pages 336-643, Portorož, Slovenia. European Language Resources Association (ELRA).
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Ballis Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Downn Lewis, Mu Li, Shujie Liu, Tin Liu, Rengian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dong- dong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic chinese to en- glish news translation. CoRR, abs/1 0bss/8803000 803.05567
William
Shaohui Kuang and Deyi Xiong. 2018. Fusing recency into neural machine translation with an inter-sentence gate model. In Proceedings of the 27th International Conference on Computational Linguistics, pages 607- 617, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
J. Stuart Hunter. 1986. The exponentially weighted moving average. In Journal of Quality Technology.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In
Yongkeun Hwang. Hyeongu Yun, and Kyomin Jung. 2021. Contrastive learning for context-aware neural

---

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Annual Meeting of the Association for Computational Linguistics, pages 5505-3511 Online. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2023. Mega: Moving average equipped gated attention. In The Eleventh Interna- tional Conference on Learning Representations.
Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 4791-4796, Brus- sels, Belgium. Association for Computational Lin- guistics.
Zhiyi Ma, Sergey Edunov, and Michael Auli. 2021. A comparison of approaches to document-level ma- chine translation.
Bei Li, Hui Liu, Ziyang Wang. Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongram Liu, and Changliang Li. 2020. Does multi-encoder help? case study on context- aware neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3512-3518, Online. Association for Computational Linguistics.
Sameen Maruf and Gholamreza Haffari. 2018. Docu- ment context neural machine translation with mem- ory networks. Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 1275-1284, Melbourne, Australia, Association for Computational Linguistics.
Proceedings of the 56th Annual
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018 0penSubtinlee20188: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re- sources Association (ELRA).
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-ave aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages
Minneapolis, Min- nesota. Association for Computational Linguistics.
Chunxi Liu, Qiaochu Zhang, Xiaohui Zhang, Kritika Singh. Yatharth Saraf, and Geoffrey Zweig. 2020. Multilingual graphemic hybrid ASR with massive data augmentation. In Proceedings of the 1tt Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboru- tion and Computing for Under-Resourced Languages (CCURL), pages 46-52, 52, Marseille, France. European Language Resources association.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas. and James Henderson. 2018 Document-level neural machine translation with hierarchical attention net- works. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954, Brussels. Belgium. Association for Computational Linguistics.
António Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and André F. T. Martins. 2020. Document-level neural MT: A systematic compar- ison. In Proceedings of the 22nd Anmual Conference of the European Association for Machine Translation, pages 225-234, Lisboa, Portugal, European Associa- tion for Machine Translation.
Maciej Modrzejewski, Miriam Exel, Bianka Buschbeck, Thanh-Le Ha, and Alexander Waibel. 2020. Incorpo- rating external annotation to improve named entity translation in NMT. In Proceedings of the 22nd Annual Conference of the Furopean Association for Machine Translation, pages 45 51, Lisboa, Portugal. European Association for Machine Translation.
Lorenzo Lupo, Marco Dinarelli, and Laurent Besacier. 2022a. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). pages 4557-4574 4572, Dublin, Ireland. As- sociation for Computational Linguistics.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evalua- tion of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Pa- pers, pages 61-72, Brussels, Belgium. Association for Computational Linguistics.
Lorenzo Lupo, Marco Dinarelli, and Laurent Besacier. 2022b. Focused concatenation for context-aware neural machine translation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 830-842, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
William O'Grady, Michael Dobrovolsky, and Mark Aronoff. 1997. Contemporary Linguistics: An In- troduction, edition. St. Martin's 's Press.
OpenAI. 2022. ChatGPT: A large-scale conversational model.
Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. simple and effective unified encoder for document- level machine translation. In Proceedings of the 58th
Mylc Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.
fairseq: A fast, extensible toolkit for

---

sequence modeling, In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 14, page 3104-3112, 12, Cambridge, MA, USA. MIT Press.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- kational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani. Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey, ACM Comput. Surv., 55(6).
Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray. Moira Inghilleri, John Wieting, and Mohit Tyyer. 2022. Exploring document-level literary ma- chine translation with parallel paragraphs from world literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9882 0932-9902,.2 Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Matt Post. 2018. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Matt Post and Marcin Junczys-Dowman Dowmunt. 2023. Escap- ing the sentence-level paradigm in machine transla- tion.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in OPUS. In Proceedings of the Eighth In- ternational Conference on Language Resources and Evaluation (LREC' pages 22 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Jörg Tiedemann and Yves Scherrer. 2017. Neural ma- chine translation with extended context. In Proceed- ings of the Third Workshop on Discourse in Machine Translation. pages 82-92, Copenhagen. Denmark. Association for Computational Linguistics.
Annette Rios Gonzales, Laura Mascarell, and Rico Sen- nrich. 2017. Improving word sense disambiguation neural machine translation with sense embeddings. Proceedings of the Second Conference on Machine Translation, pages 11-19, Copenhagen, Denmark. Association for Computational Linguistics.
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang- 2018. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, 6:407-420 420.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Karin Sim Smith. 2017. On integrating discourse in ma- chine translation. In Proceedings of the Third Work- shop on Discourse in Machine Translation, pages 110-121, Copenhagen, Denmark. Association for Computational Linguistics.
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level mt metrics: How convert any pretrained metric into a document- metric. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Dario Stojanovski and Alexander Fraser. 2018. Corefer- ence and coherence in neural machine translation: study using oracle experiments. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pages 49-60, Brussels, Belgium. Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When good translation is wrong context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 1198-1212, Florence, Italy. Asso- ciation for Computational Linguistics.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Re- thinking document-level neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3537-3548, Dublin, Ireland. Association for Computational Linguistics.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018 Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings

---

of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264-1274, Melbourne, Australia. Association for Computational Linguistics.
Appendix
A Implementation Details
A.1 Training
Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large lan- guage models. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing.
We train all models on the fairseq framework (Ot et al., 2019). Following Vaswani et al. (2017); Fer- nandes et al. (2021), we use the Adam optimizer with B1 = 0.9 and B2 = 0.98, dropout set to 0.3, an inverse square root learning rate scheduler with an initial value of 10-4, and the warm-up step set to 4000. We run inference the validation set and save the checkpoint with the best BLEU score, We compute all BLEU scores using the sacreBLEU toolkit (Post, 2018)." Wherever possible, we re- port the average and standard deviation across three randomly seeded runs.
Felix Wu, Angela Fan, Alexei Bacvski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In Interna- tional Conference on Learning Representations.
Minghao Wu, George Foster, Lizhen Qu, and Gho- lamreza Haffari. 2023. Document flattening: Be- yond concatenating context for document-level neu- ral machine translation. In Proceedings of the 17th Conference of the European Chapter of the Associa- tion for Computational Linguistics, pages 448-462, Dubrovnik, Croatia. Association for Computational Linguisties.
A.2 Models
Transformer The Transformer (Vaswani et al., 2017) is an encoder-deooder architecture that relies on self-attention mechanism, in which every po- sition of a single sequence relates to one another in order to compute a representation of that sequence. An n-length output sequence of d-dimensional rep- resentations Y E Ruxdd can be computed from an input sequence of d-dimensional representations X E Raxd as follows:
Yonghui Wu, Mike Schuster, Z. Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens. George Kurian, Nishant Patil, Wei Wang, Cliff Young. Ja- son R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's 's neural machine trans- lation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08)
QK" ((X)
Y = Atm(X)
V
(3)
Zhaofeng Wu, Hao Peng. Nikolaos Pappas, and Noah A. Smith. 2022. Modeling context with linear atten- tion for scalable document-level translation. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2022. Association for Computational Linguistics.
Q. K, and V are sequences of queries, keys, and values, respectively, with learnable weights and biases. Here, f(-) is an attention function, most commonly set to softmax, and is a correspondent scaling term. We use the Transformer base version across all experiments, which consists of 6 encoder layers, 6 decoder layers, model dimension of 512, and FFN hidden dimension of 2048.
Shuoheng Yang. Yuxin Wang, and Xiaowen Chu. 2020. A survey of deep learning techniques for neural ma- chine translation.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 533-542, Brussels, Bel- gium. Association for Computational Linguistics.
MEGA The recently introduced MEGA (Moving Average Equipped Gated Attention) (Ma et al.. 2023) architecture solves for two limitations of the traditional Transformer, which have long since resulted in sub-optimal performance sequence tasks: weak inductive bias, and a quadratic computational complexity. This mech- anism applies a multi-dimennsonal, damped expo- nential moving average (Hunter, 1986) (EMA) to
long-
Yuqi Zhang and Gongshen Liu. 2020. Paragraph- parallel based neural machine translation model with hierarchical attention. Journal of Physics: Confer- ence Series, 1453(1))01206...
12ehe sacreBI.Fr EEU signature is Bl.FUUccaee.miiee www.emmeexeeddn.....mm igt+numrefs. I ++moothh. tetttthheeppttttsscttttttteeeeee test-sct +tok.13a.

---

single-head gated attention, in order to preserve inductive biases. MEGA serves as drop-in re- placement for the Transformer attention mecha- nism, and full details can be found (Ma et al., 2023). MEGA is of comparable size to the Trans- former, with encoder and 6 decoder layers, model dimension of 512, and an FFN hidden di- mension of 1024, alongside additional shared representation dimension (128), value sequence di- mension (1024), and EMA dimension (16).
Cause: so, thus, hence, result, therefore. thereby, accordingly, consequently, for this reason
Condition: if, long as, provided that, suming that, given that
Conjunction: also, in addition, moreover, ad- ditionally, besides, else. plus, furthermore
(A)synchronous: when, after, then, before, until, after, once, after, next
In total, the Transformer architecture is around 65M parameters; the MEGA architecture around 67M parameters.
Contrastive set examples. Contrastive evalua- tion examples for anaphoric pronoun resolution are in Table 10. Following standard practice, the model is evaluated in discriminative manner: rather than generating translated sequences, the model is pro- vided with the previous sentence context, and is asked to choose the current sentence with the correct pronoun from the incorrect ones.
A.3 Data
For the Ent+Fr and En++De language pairs, we train on the IWSLT17 (Cettolo et al., 2012) datasets, which contain document-level transcriptions and translations culled from TED talks. The test sets from 2011-2014
used for validation, and the 2015 test is held for inference. For Zh-+En, we use the BWB (Jiang et al., 2023) dataset, which consists of Chinese webnovels.
B PARA2PARA Translation
B.1 Data and Preprocessing
Data for each language pair is encoded and vectorized with bytc-pair encoding (Sennrich et al., 2016) using the SentencePiece (Kudo and Richardson, 2018) framework. We use 32K joint vocabulary size for Zh- EEn, and 20K vocabulary size for the other language pairs.
We gather the Chinese and English versions of six novels within the public domain, which freely available online (Table 11). Prior to the tokeniza- tion step, we normalize punctuation and segment Chinese sentences using the open-sourced Jieba package. English sentences are tokenized using the Moses toolkit (Koehn et al., 2007). We em- ploy byte-pair encoding (Sennrich et al., 2016) for subword tokenization.
Full corpus statisties are Table 8.
Dataset
Lg. Pair
Train
Valid
Test
BWB
Zh-=En
9576566
2632
2618
In the open- setting, Tale of Two Cities and Twenty Thousand Leagues Under the Seas are withheld as the test set.
WMT17
Zh->En
25134743
2002
2001
IWSLT17
En
232825
5819
1210
IWSLT17
En
206112
5431
1080
B.2 Translation Examples
Table 8: Sentence counts across parallel datasets.
Translation examples on the PARA2PARA dataset are in Figure
A.4 Evaluation
B.3 LLM Evaluations
Annotated BWB test set. Some manually an- notated paragraphs from the BWB test set can be found in Table 9, which used in the discourse phenomena analysis.
Large Language Models (LLMs) (e.g.. Chat- GPT (OpenAI, 2022)) have recently accrued a great deal of mainstream and scientific interest, as they are found to maintain considerable fluency, consistency, and coherency across multiple NLP tasks, including document-level NMT (Wang al., (conuurnnr work). To investigate how LLMs would fare on the PARA2PARA dataset, also ob- tain translations using GPT-3.5 (gpt-3.5-turbo), a commercial, black-box LLM. Table 12 shows GPT-3.5's performance alongside that of the three
Discourse marker cafegories. Following (Jiang al., 2022), categorize discourse markers into the following:
Contrast: but, while, however, although, though, yet, whereas, in contrast, by compari- son, conversely

---

<PER, T.I>/Qiao Lian] clenched <0.1>(her] fists and lowered <0,1> [
Actually, (he) was right.
<0.1<(She) was indeed an idiot, as only idiot would believe that they could find true love online.
<P,I>[hhe] curled < lips and took deep breath. <ORG, T, (>eWChhtt) account.
Q,1><<ER,, Tiao Lian]: "What happened?" <Q>
<8 (Song Cheng) was extremely nervous and followed <P.10>(him).
<PER,T,I0><SM Liangchuan| walked forward, one step at time,
until <0,10>(hee) reached the front of <FAC,N, 19t.,N199111 (the room].
<PER, Wang Wenhao] was currently ingratiating <0.12> [122((himsef]] with <PER,N,20> C-list celebrity).
<PER,N,20> (The celebrity) asked, <Q.20> I heard that you beat <PER,N,21>(a paparazzi] |?"<QQ> paparazzi) nowadays are so disgusting.
<0.2>>Yeeah, <PER,N,21> have wanted to teach (P,21>(hhmm lesson myself for some time now!"<lQ>
",20>*Are not you afraid of becoming enemy of <P,21>/(hhem ?"<
Table 9: Annotated paragraphs from the BWB test set.
Context Sentence
Current Sentence
ContraPro (En-De)
src
There were spring nights. Through open windows it came in, dancing.
-
1. Bei offenen Fenstern tanzt es herein.
gtt
Es gab FrChlingsmächte.
2. Bei offenen Fenstern tanzt sie herein.
3. Bei offenen Fenstern
herein.
Only road led the Huseby summer farm, and it passed right by the main farm.
Bawden (En-Fr)
src
The next Saturday night.
1. Une seule route conduisait à la ferme d' été
des Huseby, et elle passait devant la grande ferme.
Igt
Le dimanche soir suivant.
2. Une scule route conduisait la ferme d'
des Huseby, il passait devant la grande ferme.
Table 10: Examples from the ContraPro and Bawden contrastive evaluation Highlighted pronouns in the current sentence require the preceding context sentence for proper disambiguation.
pre-trained baselines, for reference. This exper- iment is similar to that of Karpinska and lyyer (2023), who test GPT-3.5 on paragraphs from recently-published literary translations, and show that while LLMs can provide better paragraph-level translation (as they are better-equipped to handle long context), there are nevertheless critical trans- lation errors that human translator would be able avoid. Given that OpenAI did not disclose the composition of ChatGPT's training data, is likely that there may be data leakage from pre- pre-training (cspecially as our dataset is sourced from public- domain data). Thus, we do not believe these results represent fair comparison with the pre-training baselines; we report them for the sake of compre- hensiveness.
B.4 Pre-trained baseline performance
To investigate how fine-tuning on the PARA2PARA dataset affects the baselines' performance, we eval- uate the pre-trained baselines on the same test set without any training on the PARA2PARA corpus, As Table 13 illustrates, all three baselines exhibit significantly worse performance across the board (X), and improve after fine-tuning (V).

---

Title
Author
Year
# Paras.
APL
Gone with the Wind
Margaret Mitchell
1936
3556
143
Rebecca
Daphne du Maurier
1938
1237
157
Alice's Adventure in Wonderland
Lewis Carroll
1865
218
144
Foundation
Isaac Asimov
1951
3413
76
A Tale of Two Cities
Charles Dickens
1859
696
225
Twenty Thousand Leagues Under the Seas
Jules Verne
1870
1425
117
Table 11: Corpus information for the PARA2PARA dataset. APL = average paragraph length in tokens.
Domain
Pre-training
BLEU
BlonDe
COMET
all
pron.
entity
tense
d.m.
GPT-3.5
11.60
28.29
83.21
35.63
73.98
63.82
0.7644
Closed
XFMR Big
16.00
35.36
79.16
52.33
72.47
60.63
0.7339
LIGHTCONV Big
16.87
36.70
79.28
55.38
72.80
61.66
0.7409
MBART25
15.63
35.37
78.72
54.04
72.01
60.59
0.7385
GPT-3.5
11.90
27.86
86.02
30.64
75.71
66.93
0.7648
Open
XFMR Big
9.17
25.35
72.20
32.54
67.17
51.83
0.7003
LIGHTCONV Big
8.60
25.48
72.50
38.83
67.40
51.79
0.7027
MBART25
7.97
22.41
72.24
20.07
67.25
50.52
0.7012
Table 12: GPT-3.5 evaluations on the PARA2PARA dataset.
Domain
Pre-training
Fine-tuning
BLEU
BlonDe
COMET
all
pron.
entity
tense
d.m.
XFMR Big
7.30
25.42
67.41
37.24
60.82
54.06
0,6662
Closed
LIGITCONV Big
6.30
23.30
58.19
34.32
56.16
47.57
0.6487
MBART25
6.70
21.35
63.56
22.10
60.96
47.52
0.6699
XFMR
4.70
21.22
59.81
335.45
54.31
44.12
0.6342
Open
LIGHTCONV Hig
4.20
20.78
53.35
49.80
39,96
0.6160
MBART25
4.10
18.61
55.89
26.27
54.39
39,35
0.6407
NONE
1.370.00
8.44g.39
47.282.18
18.73a.19
41.22
16.873.00
0.3949%.00
XFMR Dig
16.00 16
35.360.19
79.169.1
52.35o.10
72.47o.08
60.63a.12
0.73390.00
Closed
LIGHTCONVDR Dio
16.87
36.700.14
79.28
55.381.27
72.800.09
61.66u.11
0,7409
MBART25
15.63
35.370.21
78.724.7
54.044.49
72.01o.12
60.59
0.7385
NONE
0.73 32
1.82
48.125.51
0.000
39.272.11
13.91s.00
0.3587
XFMR
9.17
25.35 .05
72.20p.us
32.542.18
67.176.59
51.83
0.7003
Open
LIGHTCONV Die
8.600.10
25.480.10
72.50..4
38.83
67.40%.35
51.79a.92
0.7027
MBART25
7.97
22.41o.m1
72.24m.61
20.07417
67.250..
50.52
0.701 20.00
Table 13: Ablation study on the effect of fine-tuning on Zh->En PARAZPARA dataset. no fine-tuning; denotes fine-tuning, Bold denotes best performance.

---

Source
These thoughts were clearly readable on my face but Captain Nemo remained content with inviting me to follow him. and did like man resigned to the worst. We arrived at dining where found breakfast served. Aromnax captain told beg you share my without formality..
Reference
Open Domain Base
think
knew he did
lunch with
his of courtesy. should his meal.
Open Domain
My face told me so cleerly, but Captain Nemo not sey: "What. just ask me to go with him and follow him like a men who has no choice but to listen to his fate. We are in the dining room, and breakfast has been arranged there." Mr. you
Base+ TraarrrannoooreMMTTT
"said the captein."
your dinner, end be
talk
give you any
Open Domain
My face told ma this clearl. but Captain Nemo did not say, WWhtt to ask me to follow him like following him like a man who listens desperately to fate. We the dining room, and breakfast is there. Mr Aronas, 'said the captain me. I dinner, and talk. Though promised you might go out
Base-LightConvWWMTTY
ask you have dinner, to polite.
Open Domain Base-mBART-WMTT1
told clearly, Captain Nemo say anything. asked desperately to his own wil. We went to the dining-room, and breakfast was already ready. "Professor Aronnax," the captain told me. "I ask eat well. Don't be rude. We dine and talk. Although I promise you to for walk the
like
listans
Closed Domain Base
Nemo gave replid,, sir."
sir, replied." replied,
Closed Domain Base+ Treeeerrrrnffrme-WMMert
My faca betrayad my idea clearly, but Captain Nemo said nothing: Let me follow him, and tollow him like a desperate We were in the dining-romm, and breskfest had been arranged there. "Mr. Aronnax the captain said
beg
Closed Domain
My face orders recklessly. We arrived at the diring room, and breakfast was ready. "Prafessor Aronnax, the captain told me, 'lll treat you kunch,
cleerty. but Captain Nemo
say anything. Just ask to follow him, Just like man who obeys
Base-LightConwWWMIT
offended. Well have dinner together
Though promisad you could go
Closed Domain Base-mBART-WMTI1
My face told me plainly, but Captain Nemo didn't anything. He merely asked me follow him, he were dining-room,
desperately will. the caplain lold me, 'I ask you to eat well. Don't be rude. We dine and talk. Though promise you you walk in
ready "Prolessor Aronnax,"
Figure example of PARA2PARA translation across open-domain and closed- closed-domain settings.


Reviews:
[{"review_id": "2iyTDg1HJl", "reviewer": "Reviewer_bWHY", "summary": "", "questions": "A. What does synthetic and analytic mean to describe languages?\n\nB. For table 7, what would be the model performances before fine-tuning (i.e trained on WMT17 Zh-En only)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R1 for their time and feedback! We are encouraged that R1 finds that the analysis is “fairly detailed” and that the experiments are “generally sound,” and respond point-by-point to their comments below: \n\\\n\\\n**“Section 4.3….is potentially misleading….both types of model [could] generate deficient translations.”**\n\nWe agree that the claim (\"The sentence-level NMT baseline is already competitive\") may be confusing. \n\nTo be precise, we argue in Sect. 4.3 that the performance across the context-agnostic baseline and context-aware models is largely similar. Due to problems with common document-level datasets (e.g., relative lack of contextual signals) (Section 4.1),  context-aware models do not exhibit a meaningful improvement over context-agnostic models.\n\nAcross different language pairs and model architectures, we find that this trend is consistent across not just sentence-level automatic metrics (e.g., BLEU, COMET), but surprisingly, across some document-level metrics as well (e.g., tense and discourse markers in BlonDe). \n\nOur empirical results could be interpreted to mean that both context-agnostic and context-aware models generate either high-quality or low-quality translations. The interpretation here is valid in either direction, and does not detract from our overall argument.\n\nAs it is critical to keep our arguments precise, we will clarify the wording in our revision. \n\\\n\\\n**“High level observations in section 4 are often already known…Data sparsity is [previously] mentioned [and] limited differences in BLEU scores have…motivated the design of metrics to capture specific discourse phenomena.”**\n\nSome of the challenges in Section 4 have indeed been raised in prior work. We believe it is necessary to include them for the sake of completeness.  \n\nHowever, in contrast to past literature, we conduct a more rigorous and finer-grained quantitative analysis of such observations over a range of language pairs, model architectures, concatenation schemas, and document-level phenomena. To date, no other work has attempted a comprehensive analysis of the obstacles that impede progress in context-aware NMT. \n\nFor example, data sparsity is first posited by Lupo et al., 2022 [1], though they do not perform any data investigation and only focus on multi-encoder architectures. Our analysis is the first to quantitatively show the presence of context sparsity. By carefully examining document-level data (BWB, IWSLT), we statistically find that most sentences are short, simple constructions which do not require context to be accurately translated. We show that concatenation-based architectures also struggle with this phenomenon, resulting in meager document-level translation gains.  \n\nAs for the second point, it is well known that BLEU, COMET, etc. are inadequate measurements of document-level improvements, spurring the development of document-level metrics (e.g., BlonDe, contrastive evaluation sets) that explicitly  measure different discourse-level phenomena. Our study is the first to show that these metrics are only selectively helpful based on discourse category (Sect. 4.2). We additionally note several limitations in Sect. 4.5.  \n\n[1]  [Lupo et al. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. Annual Meeting of the Association for Computational Linguistics. 2022](https://arxiv.org/abs/2103.17151)\n\\\n\\\n**“The collected [Para2Para] dataset is quite small…its usefulness is not well established.”**\n\nAt 10,545 paragraphs, our Para2Para dataset is indeed smaller than many benchmark sentence-level translation datasets, due to the challenge of finding paragraph-aligned, high-quality, professionally translated data. Para2Para is comparatively larger than the Zh-En split of Par3 [1], another open-sourced, paragraph-aligned literary translation dataset, which stands at only 1320 paragraphs. \n\nAlthough we are definitely open to expanding the dataset in future work, the main purpose of Para2Para is to establish preliminary baselines and demonstrate that taking the paragraph as a minimal discourse-level unit can serve as a challenging, realistic setting for context-aware NMT. We introduce pre-trained baselines to offset the impact of low domain size. From Table 7, these baselines perform noticeably better compared to a baseline training from scratch (the NONE setting), though the numbers across the board are all fairly low, showing that paragraph-to-paragraph translation is quite difficult in nature.    \n\n[1] [ Thai et al. Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature. Conference on Empirical Methods in Natural Language Processing. 2022.](https://arxiv.org/abs/2210.14250)\n\\\n\\\n**1A.**\\\nIn an analytic language, concepts can be mainly  expressed as root/stem words, and bound morphemes (affixes) are rare; examples include Chinese, Vietnamese, and Cambodian. In contrast, synthetic languages  have many affixes, in which multiple concepts would be fixed into a single word; examples include English, Latin, and French. The morpheme-to-word ratio is lower in analytic languages, and higher in synthetic ones. [1] We will add this elaboration in our revision.\n\n[1] O'Grady, W., Dobrovolsky, M., & Aronoff, M. (1997). Contemporary linguistics: an introduction. 2nd, U.S. ed.\n\\\n\\\n**1B.** \\\nWe include a table showing model performances without fine-tuning (i.e., trained only on WMT17 Zh-En) below: \n| Domain | Pre-training | BLEU  | BLONDE (all) | BLONDE (pron.) | BLONDE (entity) | BLONDE (tense) | BLONDE (d.m.) | COMET    |\n|--------|--------------|-------|--------------|----------------|-----------------|----------------|--------------|----------|\n| Closed | XFMR       | 7.30  | 25.42        | 67.41          | 37.24          | 60.82        | 54.06  | 0.6662   |\n|        | LightConv  | 6.30  | 23.30        | 58.19          | 34.32          | 56.16        | 47.57  | 0.6487   |\n|        | mBART25    | 6.70  | 21.35        | 63.56          | 22.10          | 60.96        | 47.52  | 0.6699   |\n| Open   | XFMR       | 4.70  | 21.22        | 59.81          | 35.45          | 54.31        | 44.12  | 0.6342   |\n|        | LightConv  | 4.20  | 20.78        | 53.35          | 38.94          | 49.80        | 39.96  | 0.6160   |\n|        | mBART25    | 4.10  | 18.61        | 55.89          | 26.27          | 54.39        | 39.35  | 0.6407   |\n\nFor reference, this is the results including fine-tuning:\n| Domain | Pre-training   | BLEU  | BLONDE (all) | BLONDE (pron.) | BLONDE (entity) | BLONDE (tense) | BLONDE (d.m.) | COMET    |\n|--------|----------------|-------|--------------|----------------|-----------------|----------------|--------------|----------|\n| Closed | None       | 1.90  | 7.65         | 47.28          | 15.50           | 40.51          | 8.84   | 0.3725   |\n|        | XFMR      | 15.80 | 34.80        | 78.55          | 50.39           | 71.87          | 60.19  | 0.7352   |\n|        | LightConv  | **16.80** | **36.86**    | **79.01**      | **57.70**       | **72.90**       | **61.18**  | 0.7375   |\n|        | mBART25    | 15.70 | 35.60        | 78.80          | 55.06           | 72.05          | 58.92  | **0.7391** |\n| Open   | None      | 0.90  | 1.74         | 48.15          | 0.00            | 35.69          | 19.94  | 0.3429   |\n|        | XFMR      | **9.70**  | **25.96**    | 72.31          | **35.51**       | 65.71          | 51.43  | 0.7022   |\n|        | LightConv  | 8.90  | 24.89        | **73.41**      | 30.31          | **68.14**       | **52.87**  | 0.7038   |\n|        | mBART25    | 8.10  | 23.06        | 72.09          | 23.37          | 67.39          | 51.23  | **0.7076** |\n\nAs expected, we can see that without additional training on the Para2Para dataset, all three models exhibit worse performance across the board. \n\\\n\\\n**Missing References** \\\nWe thank R1 for providing these two references and will add both to our revision! We should note that the first paper (Zhang and Liu, 2020) [1] is primarily a model paper.  While this work also introduces a parallel paragraph corpus, this dataset is divided based on word alignments, whereas our Para2Para dataset is aligned at the paragraph level. The second paper (Ghussin et al., 2023) [2] is technically concurrent work; the dataset in this paper is cultivated from Paracrawl, which sources from online webpages. Our dataset, in comparison, focuses exclusively on the literary domain. \n\n[1] [Zhang and Liu. Paragraph-Parallel based Neural Machine Translation Model with Hierarchical Attention. Journal of Physics: Conference Series. 2020.](https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012006) \\\n[2] [Ghussin et al. Exploring Paracrawl for Document-level Neural Machine Translation. EACL. 2023.](https://arxiv.org/abs/2304.10216)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The analysis, especially in sections 4.1 (discourse phenomena sparsity) and 4.2 (model performance for contextual phenomena), is fairly detailed. We can observe that some phenomena on which the context-aware models are struggling (e.g. tense, discourse markers) are infrequent (table 1).\n\nThe experiments are generally sound.", "reasons_to_reject": "Section 4.3 (\"The sentence-level NMT baseline is already competitive\") is potentially misleading. Because BLEU scores are similar between context-aware and context-agnostic models, the authors mention that \"This suggests that [...] context-agnostic models are already capable of delivering high-quality translations.\" However, this could also indicate that both types of model generate deficient translations.\n\nHigh-levels observations in section 4 are often already known (although relevant papers are cited and the specific results/experiments are novel). Data sparsity is mentioned for example in [1]. Limited differences in BLEU scores have (in part) motivated the design of metrics to capture specific discourse phenomena (but these metrics are not perfect).\n\nThe collected paragraph-to-paragraph dataset is quite small, so its usefulness is not well established.\n\n[1] Lupo et al. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. ACL. 2022", "questions_to_authors": "A. What does synthetic and analytic mean to describe languages?\n\nB. For table 7, what would be the model performances before fine-tuning (i.e trained on WMT17 Zh-En only)?", "paper_topic_and_main_contributions": "This paper investigates challenges for context-aware neural machine translation. The authors show that discourse phenomena are sparse and that a context-aware transformer model (concatenating the context) doesn't perform well for some of them. They also replace the transformer model with another architecture (MEGA), with limited improvement, and argue for a better document-level translation metric. A paragraph-to-paragraph dataset of Chinese-English novels is also collected.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Thai et al. Exploring document-level literary machine translation with parallel paragraphs from world literature. EMNLP. 2022. is duplicated in the references."}, "extra_text": {"Missing_References": "Zhang and Liu. Paragraph-Parallel based Neural Machine Translation Model\nwith Hierarchical Attention. Journal of Physics: Conference Series. 2020. https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012006/pdf\nNote: The previous paper appears to be almost the same as Zhang et al. Paragraph-Level Hierarchical Neural Machine Translation. ICONIP. 2019.\n\n[Contemporaneous] Ghussin et al. Exploring Paracrawl for Document-level Neural Machine Translation. EACL. 2023."}}, {"review_id": "eqciyZBr2J", "reviewer": "Reviewer_LPga", "summary": "", "questions": "L603, why do you say it shows difficulty to train on scratch on longer-sequence data? To me, the relatively low performance comes from limited training data. On top of this point, we know LLMs can handle long contexts quite well, maybe you could prompt some LLMs to generate hypotheses on your test sets as well? Would be interesting to see how out-of-the-box \"ChatGPT\" compares with pre-trained and finetuned in-domain document-level NMT systems.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R2 for their time and feedback! We are encouraged that R2 finds the that “the paper is well-written” and “provides value for future research in context-aware NMT,”  that “the analyses are extensive,” and that our Para2Para dataset is a “nice and challenging resource to boost further research.” \n\nWe respond point-by-point to R2’s feedback below: \\\n\\\n**“Language pairs/model variations are not that extensive.”**\n\nWe agree that our analysis could definitely benefit from more language pairs / model variations. Given that many popular document-level metrics are constrained to particular language pairs, we selected En-Fr, En-De, and Zh-En as representative language pairs so that we could use BlonDe and contrastive evaluation sets. We chose to focus on the concatenation-based architecture given its simplicity, good performance, and widespread adoption in recent years [1] over alternative architectures, and we additionally explore different combinations of context window settings (1-2, 2-2, 3-1, etc.). \n\n[1] [Fernandes et al. Measuring and Increasing Context Usage in Context-Aware Machine Translation. Annual Meeting of the Association for Computational Linguistics. 2021](https://arxiv.org/abs/2105.03482) \\\n\\\n**2A.**\n\n“[In] L603, why do you say it shows difficulty to train on scratch on longer-sequence data?” \n\nWe think that both reasons—the difficulty of training on longer-sequence data, as well as the relatively small size of the Para2Para corpus—could explain the extremely low performance of the NONE  (i.e., Transformer with no pre-training) setting (Table 7). \n\nTo disentangle the two factors (is the dataset difficult because of its limited size, or because paragraph-to-paragraph translation is inherently a challenging task?), we also test additional baselines that leverage pre-training to alleviate the low-domain issue. Our pre-trained baselines (using the Transformer, LightConv, and mBART-25) all exhibit a marked performance improvement over the NONE setting, although the numbers across the board are relatively low, attesting to the difficult nature of paragraph-to-paragraph translation.  \n\nWe will clarify this point in our revision. \\\n\\\n**“Maybe you could prompt some LLMs [e.g., ChatGPT] to generate hypotheses on your test sets as well?”**\n\nHow LLMs would fare on our Para2Para dataset is indeed an interesting question. From recent studies such as [1], we hypothesize that while LLMs can provide better paragraph-level translation (as they are better at handling long context), there would still be critical translation errors that a human translator would be able to avoid.\n\nAs for using ChatGPT— given that OpenAI did not disclose the composition of ChatGPT’s training data, it is likely that there may be instances of data leakage from pre-training (especially as our dataset is sourced from public-domain data); thus, we do not believe these results represent a fair comparison with our other baselines in Table 7. The out-of-box ChatGPT results, included with the other pre-trained baselines on the Para2Para dataset, are as follows:\n\n| Domain | Pre-training   | BLEU  | BLONDE (all) | BLONDE (pron.) | BLONDE (entity) | BLONDE (tense) | BLONDE (d.m.) | COMET    |\n|--------|----------------|-------|--------------|----------------|-----------------|----------------|--------------|----------|\n| Closed | GPT-3.5     | 11.60 | 28.29        | **83.21**      | 35.63          | **73.98**      | **63.82**  | **0.7644** |\n|        | XFMR      | 15.80 | 34.80        | 78.55          | 50.39          | 71.87          | 60.19  | 0.7352   |\n|        | LightConv  | **16.80** | **36.86**    | 79.01          | **57.70**      | 72.90          | 61.18  | 0.7375   |\n|        | mBART25   | 15.70 | 35.60        | 78.80          | 55.06          | 72.05          | 58.92  | 0.7391   |\n| Open   | GPT-3.5    | 11.90 | **27.86**    | **86.02**      | 30.64          | **75.71**      | **66.93**  | **0.7648** |\n|        | XFMR       | 9.70  | 25.96        | 72.31          | **35.51**      | 65.71          | 51.43  | 0.7022   |\n|        | LightConv  | 8.90  | 24.89        | 73.41          | 30.31          | 68.14          | 52.87  | 0.7038   |\n|        | mBART25    | 8.10  | 23.06        | 72.09          | 23.37          | 67.39          | 51.23  | 0.7076   |\n\n[1] [Karpinska et al. Large language models effectively leverage document-level context for literary translation, but critical errors persist. arxiv preprint. 2023 ](https://arxiv.org/abs/2304.03245)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written. The text is smooth and easy to follow.\n\nThe analyses are extensive. Although context-aware translation itself is a hard topic, the authors did a good job disecting the problem, and provided multiple angles in understanding the challenges and the current landscape.\n\nThe introduced paragraph-aligned dataset is a nice and challenging resource to boost further research. While comparing scores of automatic metrics is easily envisioned, I can also see people getting interested in proposing better evaluation metrics for such paragragh-aligned test sets.", "reasons_to_reject": "Solid work, no apparent reasons to reject.\n(Although one may argue that the language pairs/model variations are not that extensive, I don't think this point alone justifies a rejection.)", "questions_to_authors": "L603, why do you say it shows difficulty to train on scratch on longer-sequence data? To me, the relatively low performance comes from limited training data. On top of this point, we know LLMs can handle long contexts quite well, maybe you could prompt some LLMs to generate hypotheses on your test sets as well? Would be interesting to see how out-of-the-box \"ChatGPT\" compares with pre-trained and finetuned in-domain document-level NMT systems.", "paper_topic_and_main_contributions": "This paper talks about context-aware (or document-level) neural machine translation.\n\nOn a high level, there are two major parts in the paper.\nThe first part analyze the efficacy and chanllenges of context-aware NMT, providing detailed information on aspects like how often document-level context is useful, how well can different types of discourse phenomena be resolved, and how existing document-level systems compare to a strong sentence-level baseline.\nThe second part introduced a new dataset, which is a paragraph-aligned translation dataset that are based on open-domain books. The dataset is quite challenging, because of the relatively small size, the translation style, and the occasionally added translations without corresponding source sentences. The authors also provided baseline results on this dataset.\n\nOverall, the paper is solid, aligned with the interests of EMNLP, and provides value for future research in context-aware NMT.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Table 7 and the corresponding text describing it, you conclude that it is necesary to pretrain the model the get decent performance on the proposed paragraph-level dataset. For this purpose, I think it also makes sense to include scores for the pre-trained but not finetuned models. Including these results can also eliminate the reasoning that the pre-trained models are already good enough without finetuning."}}, {"review_id": "RVmWW1SntT", "reviewer": "Reviewer_dPV2", "summary": "", "questions": "A, where does the target context come from during inference? the model decoding results or the gold reference? How do you close the gap between training and inference if you use the decoded context?\n\nthanks for the response, with Question A, I want to know whether the target-side input contexts are noised or not during training.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R3 for their time and feedback! We are encouraged that R3 finds our analysis to point out “key obstacles for document-level MT studies,” and respond point-by-point to their feedback below: \\\n\\\n**“Some of the findings are already proposed in previous studies.”** \n\nSome findings have indeed surfaced in previous studies, but the purpose of this work is to explore such claims in greater and more rigorous detail across a standardized experimental set-up. Our experiments either corroborate or refute such commonly-accepted beliefs, yielding new insights that we believe to be useful to the scientific community. \n\nGiven that R3 did not provide specific references to back up this claim, we refer to our response to R1, in which we provide some concrete examples wherein our paper adds quantitative analysis to common perceptions in this field. For example,\n\n- *Data sparsity*: this is first posited by Lupo et al., 2022 [1], though they do not perform any data investigation and only focus on multi-encoder architectures. Our analysis is the first to quantitatively show the presence of context sparsity across document-level data (BWB, IWSLT).\n\n- *There is a need for an appropriate document-level translation metric*: Our study is the first to show that existing metrics are only selectively helpful based on discourse category (Sect. 4.2). We additionally note several limitations in Sect. 4.5.\n\nTo date, no other work has attempted a comprehensive analysis of the obstacles that impede progress in context-aware NMT.  \n\n[1] [Lupo et al. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. Annual Meeting of the Association for Computational Linguistics. 2022](https://arxiv.org/abs/2103.17151) \\\n\\\n\\\n**“Paragraph-to-paragraph translation is simple in practice and lacks novelty”**\n\nThough paragraph-to-paragraph translation may be conceptually simple, translating literary works presents a significantly greater challenge compared to standard MT tasks [1]. Moreover, paragraphs from literary texts provide richer contextual signals. We empirically show in Table 7 that this task is in fact quite challenging, as even powerful pre-trained models struggle on this task, and the performance discrepancy becomes even more apparent under open-domain settings.\n\nFurthermore, to the best of our knowledge, our proposed dataset (Para2Para) is the first paragraph-aligned, parallel translation dataset in which source and target paragraphs are harvested from high-quality professional literary translations. The only other paragraph-aligned, literary translation dataset that we are aware of is Par3 [1],  which uses fine-tuned GPT-3 to produce target translations, and whose Zh-En split sources from ancient Chinese novels written in Classical Chinese (which is very different in from the modern language). \n\n[1] [Thai et al. Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature. Conference on Empirical Methods in Natural Language Processing. 2022.](https://arxiv.org/pdf/2210.14250) \\\n\\\n**3A.**\\\nFollowing previous work on context-aware NMT [1, 2, 3], we use the preceding decoded sentence  as target context during inference for a fair and realistic comparison (as one would not have access to gold target data when translating in real-world scenarios) . \nThere is certainly a performance difference between leveraging either the decoded translation versus the gold reference during test-time translation; however, closing the gap is beyond the scope of this work. \\\n\\\n[1] [Fernandes et al. Measuring and Increasing Context Usage in Context-Aware Machine Translation. ArXiv abs/2105.03482. 2021: n. pag.](https://arxiv.org/abs/2105.03482) \\\n[2] [Yin et al. Do Context-Aware Translation Models Pay the Right Attention? Annual Meeting of the Association for Computational Linguistics. 2021](https://arxiv.org/abs/2105.06977) \\\n[3] [Lupo et al. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. Annual Meeting of the Association for Computational Linguistics. 2022](https://arxiv.org/abs/2103.17151)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The analysis on the context-aware translation testset points out key obstacles for document-level MT studies (existing document-level corpora contain a sparse number of discourse phenomena that rely upon inter-sentential context to be accurately translated and context is less helpful for tense and discourse markers). Based on these findings, the paper suggests to perform paragraph-to-paragraph translation and provides a paragraph-aligned Chinese-English dataset.", "reasons_to_reject": "A, some of the findings are already proposed in previous studies.\n\nB, paragraph-to-paragraph translation is simple in practice and lacks novelty.", "questions_to_authors": "A, where does the target context come from during inference? the model decoding results or the gold reference? How do you close the gap between training and inference if you use the decoded context?\n\nthanks for the response, with Question A, I want to know whether the target-side input contexts are noised or not during training.", "paper_topic_and_main_contributions": "This paper takes an in-depth look into the testset of document-level MT, it finds that 1) existing document-level corpora contain a sparse number of discourse phenomena that rely upon inter-sentential context to be accurately translated, and 2) context is less helpful for tense and discourse markers. The paper suggests to perform paragraph-to-paragraph translation and construct a new paragraph-aligned Chinese-English dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive study of NMT systems that exploit context beyond the current sentence. It identifies several challenges to progress in this area, including sparsity of context signals and the need for accurate metrics. It proposes a simple framework based on paragraph-level translations, and presents baseline results on a new public dataset.

Reviewers were generally positive, finding the paper clearly written, and the analyses comprehensive and well organized, with solid experiments. The released dataset and benchmarks were also appreciated. On the negative side, one reviewer felt that the paper didn’t break new ground, and that the proposed paragraph-to-paragraph framework was overly simplistic.

Moving beyond sentence-level translation is an important problem for the field, and this paper goes further than previous studies in analyzing reasons for the relative lack of progress. Given the rather confused state of the art in this area, starting afresh with a clear and simple benchmark and dataset - armed with a detailed set of empirical observations about the signals available from extra-sentence context - seems like an excellent strategy. I think this work has the potential to make a large impact.
Decision: Accept-Main