Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels
Year: 2023

Abstract:
Several recent papers claim to have achieved human parity at sentence-level machine translation (MT)—especially between highresource language pairs. In response, the MT community has, in part, shifted its focus to document-level translation. Translating documents requires a deeper understanding of the structure and meaning of text, which is often captured by various kinds of discourse phenomena such as consistency, coherence, and cohesion. However, this renders conventional sentence-level MT evaluation benchmarks inadequate for evaluating the performance of context-aware MT systems. This paper presents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. (2022a). The new BWB annotation introduces four extra evaluation aspects, i.e., entity, terminology, coreference, and quotation, covering 15,095 entity mentions in both languages. Using these annotations, we systematically investigate the similarities and differences between the discourse structures of source and target languages, and the challenges they pose to MT. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures. This gives us a new perspective on the challenges and opportunities in document-level MT. We make our resource publicly available to spur future research in document-level MT and its generalization to other language translation tasks. https://github.com/EleanorJiang/ BlonDe/tree/main/BWB

heading: 1 Introduction
text: The field of machine translation (MT) has made tremendous strides in recent years thanks to neural machine translation (NMT) models that can utilize massive quantities of parallel training data (Vaswani et al., 2017; Junczys-Dowmunt et al.,
2018, inter alia). Unfortunately, most parallel corpora are aligned only at the sentence level, and document-level translation has remained limited to small-scale studies (Ansari et al., 2020; Lison et al., 2018; Koehn, 2005; Tiedemann, 2012).
There are, however, inherent differences between sentence-level translation and documentlevel translation, as documents consist of complex discourse structures that go beyond the mere concatenation of individual sentences. Three key discourse features are particularly important in document-level translation. First, the translations of named entities have to be consistent. For example, in Fig. 1, the same terminologies are not consistently referred to with the same translations (i.e., Weibo vs. micro-blog, Qiao Lian vs. Joe vs. Joe love), and as a result, the sentence-level MT system fails to capture discourse dependencies across sentences. Second, the coreference relationship in the source language needs to be preserved. In particular, the relations between entities and their pronominal anaphora should be preserved, as well as the transition of discourse centers1. In Fig. 1 and Fig. 8, coreference chains are color-coded, visually demonstrating the preservation of anaphoric referential relations and the transition chains of centers. Third, the conversational structure, such as transitions between speakers, must be maintained.
Inferring latent discourse structures from documents is essential for translation coherence because of these discourse features. As a result, conventional sentence-level MT pipelines that do not consider context are unable to generate natural and coherent translations (Lapshinova-Koltunski et al., 2018; Werlen and Sadiht, 2021). While efforts have been made to develop context-aware NMT models over the past few years (Tiedemann and
1In this context, discourse centers refer to the entities that are in the readers’ focus at a certain point in the discourse. It can be realized by either pronouns or nominal mentions (Grosz et al., 1995).
7853
Scherrer, 2017; Agrawal et al., 2018; Voita et al., 2018; Bawden et al., 2018; Zhang et al., 2018), a reliable evaluation method for document-level MT has not yet been developed.
Traditional MT evaluation metrics, such as BLEU (Papineni et al., 2002), TER (Snover et al., 2006), METEOR (Banerjee and Lavie, 2005), are limited to evaluating translation quality at the sentence level and do not consider contextual information. Even with the use of more complex neural evaluation methods, e.g., COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020) and BERTSCORE (Zhang et al., 2020), it is not possible to accurately assess the coherence of translations within the context of a document. BLONDE (Jiang et al., 2022a) has attempted to evaluate documentlevel translation quality and obtained a higher correlation with human evaluation. Yet, it is a coarsegrained automatic evaluation metric based on surface syntactic matching, thus being unable to evaluate deeper discourse features, such as coreferences and quotations. Vernikos et al. (2022) propose to concatenate context to extend pretrained metrics to the document level. However, its lack of interpretability leaves it unclear as to why it assigns a particular score to a given translation, which is a major reason why some MT researchers are reluctant to employ learned metrics in order to evaluate their MT systems (Karpinska et al., 2022).
To address this issue, we have annotated a new benchmark based on the large-scale bilingual parallel corpus BWB proposed in previous work (Jiang et al., 2022a). This benchmark includes four evaluation aspects annotated on both source and target sides: entity, terminology, coreference, and quota-
tion. It consists of 80 documents across multiple fiction genres, with 15,095 mentions covered in 150,287 words in total.
Using these annotations, we systematically evaluate the latent discourse structures of MT models. We show that, while context-aware MT systems perform better than phrase- and sentence-level ones in terms of entity translation, they still lag behind human translation in terms of proper noun and personal name translation. In addition, we demonstrate that humans are far more adept at preserving coreference chains than MT models.
The main contributions of our paper are:
• We propose a new benchmark that includes four types of discourse annotations that are closely related to document-level translation. • We systematically investigate the similarities and differences between the discourse structures of source and target languages, and the challenges they pose to machine translation. • We demonstrate that machine translations differ fundamentally from human translations in terms of their latent discourse structures. We believe these new insights can lead to potential improvements in future MT systems.

heading: 2 Corpus
text: We draw our source material for annotation from the texts in the test set of BWB (Jiang et al., 2022a), which consists of 65,107 words, 2,633 sentences in 80 different documents drawn from 6 web novels across different genres. The distribution of genres in BWB are shown in Fig. 2. The statistics of BWB are shown in Tab. 1. We refer the readers to App. D
for the dataset creation details.

heading: 3 Annotation
text: In this section, we describe the annotation scheme and the process of annotating the corpus, and provide examples of how the annotations can be used to study discourse phenomena in the two languages. The annotation was conducted by eight professional translators. A total of 8,585 mentions on the English side and 6,070 mentions on the Chinese side have been annotated.

heading: 3.1 Named Entities
text: The mistranslation of named entities (NEs) can significantly harm the quality of translation, although often under-reported by automatic evaluation metrics (e.g., BLEU, COMET). Therefore, we annotate named entities according to the ACE 2005 guidelines (Walker et al., 2005). These annotations identify six categories of entities: people (PER), facilities (FAC), geo-political entities (GPE), locations (LOC), vehicles (VEH), and organizations (ORG). In addition, each unique entity is assigned an entity_id for the purpose of coreference resolution. For example, the following text includes annotations for two entities:
(1) · · · [PER,6 the scion of [ORG,5 the Ri family]]
Nested entities are also annotated. The annotated
dataset contains a total of 5,984 English entities and 4,853 Chinese entities. There are fewer entities annotated in Chinese compared to English because the subject is often dropped in Chinese. This phenomenon also serves as a significant contributor to translation errors in context-agnostic MT systems when translating from Chinese to English, as shown in Fig. 1 sentence (3).

heading: 3.2 Terminology
text: Terminology refers to specialized words or phrases conventionally associated with a particular subject matter or whose use is agreed upon by a community of speakers. In the context of a novel, terminology refers to the specific words and phrases that are used to describe the concepts, characters, settings, and events in the story. These words and phrases may be specific to the genre or style of the novel, or they may be unique to the world or setting created by the author. For example, in a fantasy novel that uses made-up magic terms, incorrect terminology translation could lead to misunderstandings of the powers and abilities of characters, the rules of the magical world, or the significance of certain events. Similarly, incorrect translations of named entities could lead to confusion or misunderstandings about the identities and roles of characters in the story. Inconsistent terminology translation can compromise the integrity and cohesion of a work and make it difficult for readers to fully grasp the intended meaning. Therefore, we have included a layer for terminology identification. This layer is a binary classification of whether a certain span counts as terminology (T) or not (N). Tab. 2 shows some term and non-term annotations in the dataset. There are 2,156 English terms and 2,290 Chinese terms that have been annotated, accounting for approximately 52% of all entities.

heading: 3.3 Coreference
text: Coreference is the phenomenon in which a word or phrase in a text refers to another word or phrase that has been previously mentioned in the text. Establishing coreference is important for determining gender and number marking on pronouns, determiners, and adjectives as well as for making lexical decisions. Errors can compromise coherence and accuracy. For example, consider the following sentence in English:
SRC In fairness, Miller did not attack the statue itself. ... But he did attack its meaning ... REF Um fair zu bleiben, Miller griff nicht die Statue selbst an. ... Aber er griff deren Bedeutung an ... MT Fairerweise hat Miller die Statue nicht selbst angegriffen. ... Aber er griff seine Bedeutung an ...
In this example, both itself and its refer back to the statue. However, in MT, its is incorrectly translated as seine, which is the masculine possessive pronoun in German, as a result of the incorrect coreference resolution.
We follow the OntoNotes Coreference Annotation Guidelines for Chinese and English (Weischedel et al., 2012), and consider proper noun phrases, common noun phrases, and personal pronouns as coreference candidates. In particular, we follow the three important principles of the OntoNotes guidelines.
Maximal Spans. We include all modifiers in an annotated span. I.e., we annotate [the surrounding passersby, who were actually reporters in disguise] rather than simply [passersby].
Rather Lack Than Abuse. When the annotators were in doubt, they were told to not mark a difficult coreference decision.
Ellipsis. Omitted pronouns are marked with O and other pronouns are marked with P. For instance, consider:
SRC [PER,T,1乔恋] 攥紧了拳头,[P,1她]垂下了头。 REF [PER,T,1Qiao Lian] clenched [O,1] her] fists and
lowered [P,1 her] head.
In this example, the first her is omitted in Chinese and is therefore marked as <O,1>, where 1 is the entity id of Qiao Lian.

heading: 3.4 Quotations
text: The final annotation layer is quotation. In this stage of the process, we identify instances of direct speech and attribute the speech to its speaker.
The inclusion of direct speech is common in literature, and its proper translation is essential. For instance, the same person can be addressed by different names by different people. Furthermore, MT systems that lack contextual awareness may have difficulty correctly identifying the speaker in instances of direct speech, leading to inconsistencies in overall contextual translation. For example,
(2) [Q,2 “Oh dear! Oh dear! I shall be late!”]
where 2 is the entity id of the speaker and “Oh dear!” is an exclamation. In this example, discriminating exclamations from vocatives is vital for the cohesiveness of the story. In addition, there are cases where knowing the speaker is important for coreference, e.g., John (Mary, respectively) said to Mary (John, respectively), “Oh, this is your dog. Her (His, respectively) dog barked.” In BWB test set, there are 840 (31.9%) sentences that contain quotations, and there are 25 distinct speakers in total.
Inter-Annotator Agreement. To ensure annotation quality, we randomly select 10 documents and have them independently annotated by another expert, following previous work (Bamman et al., 2019). We then calculate the mention overlap and F1 scores for entity, terminology, coreference, and quotation between the two annotations, using the regular annotator’s result as the hypothesis and the second annotator’s as the reference. We achieve comparable inter-annotator agreements with previous work (Cohen et al., 2017; Bamman et al., 2019). The results are reported in Tab. 3.

heading: 4 Bilingual Analysis
text: We conduct a thorough bilingual analysis of the novel evaluation aspects using the new annotation. The following two research questions are being investigated:
• How different (or similar) are the discourse structures in the source language (Chinese) and the target language (English)? • How do the differences in discourse structures affect the translation quality of MT systems?
Entity types and terminology. Tab. 4 demonstrates the distribution of entity annotations and term annotations in both Chinese and English in the BWB test set. First, the BWB test set is dominated by person and facility entities, with a much lower proportion of geo-political entities. In addition, terminology entities and person entities are repeated more frequently, requiring better translation consistency. An important sanity check from our analysis is that the number and distribution of entities in Chinese and English are similar. This suggests that at the discourse level, the information being conveyed in these two languages is largely the same.
Pronouns. Pronoun translation has been the focus of discourse-level MT evaluation (Hardmeier, 2012; Miculicich Werlen and Popescu-Belis, 2017). We compare the numbers of different types of pronouns in Chinese and English in Tab. 5. As can be seen, Chinese has significantly fewer pronouns due to its pronoun-dropping property. This poses additional challenges for NMT systems, as they must be able to resolve anaphoric references. In addition, Tab. 5 reveals a notable difference in the frequency of neuter pronouns. English exhibits a higher prevalence of neuter pronouns, possibly due to the presence of a larger number of expletive subjects in the language.
Coreference. We further conduct analyses to investigate the differences and similarities between coreference behaviors in Chinese and English.
Fig. 3a examines the distribution in distances to the nearest antecedent for both Chinese and English. The average distance between antecedents and anaphora in English is shorter than in Chinese. This may be connected to the more common use of pronoun ellipsis in Chinese—when referring to closer antecedents, pronouns are often omitted. Fig. 3b illustrates the distance between the first and last mention of an entity within each coreference chain. It can be observed that, although English coreference chains tend to be longer in general compared to Chinese coreference chains, the distribution of these lengths is consistent. This represents another language-independent discourse feature in addition to entity distribution. The spread distributions of the two languages are further depicted in Fig. 8 and Fig. 9. Finally, we present an analysis of the size of coreference chains in Fig. 3c. Our results indicate that the number of mentions in English coreference chains tends to be larger than in Chinese.
Challenges for MT. So far our analyses have revealed that the Chinese source language and English target language exhibit several distinct linguistic characteristics, leading to various discourse phenomena, including the omission of pronouns in Chinese and the use of expletive subjects in English, as well as shorter English coreference chains, among others. These discourse phenomena present challenges for MT. In order to systematically analyze translation errors caused by these phenomena, we conducted a study in which four professional translation experts compared reference translations to those generated by a commercial MT system
Google Translate. The annotation guidelines are as follows:
1. Identify translation errors and identify whether the translation error is at the DOCUMENT level, e.g., the translation is inconsistent with the context or does not comply with the global criterion of coherence. 2. Categorize the DOCUMENT errors in accordance with the discourse phenomena and mark the corresponding spans in the reference (English) that cause the MT output to be incorrect.
The results, summarized in Tab. 6, indicate that named entity translation is the most significant issue, with terminology translation, coreference resolution failures, and pronoun omission also being notable problems. This analysis emphasizes the importance of accurately inferring latent discourse structures in the translation of long texts.

heading: 5 Exploring Discourse Features of MT Systems and Human Translation
text: The existence of a bilingual corpus with discourse annotations allows us to test the performance
of both human translations and MT systems— including the popularly used commercial systems and those trained on the in-domain dataset BWB. The following 6 MT systems are adapted:2
• SMT: phrase-based baseline (Chiang, 2007). • BING, GooGLe, BaiDu: commercial systems. • SENT: the sentence-level transformer-based
(Vaswani et al., 2017) baseline model. • DOC: the document-level NMT model that
adopts two-stage training (Zhang et al., 2018). • PE: the post-edited BING outputs produced by
professional translators. They were instructed to correct only discourse-level errors with minimal modification.
Overall Quality. The quality of the translations produced by each system was first evaluated using a range of automatic translation metrics, including BLEU, METEOR, BERTSCORE, COMET, and BLONDE. The results of this evaluation are presented in Tab. 7. In addition, we conducted a human evaluation of the translations, the results of which are shown in Fig. 4.3 The large gap between the performances of HT and MT indicates that the genre of BWB, i.e., literary translation, is challenging for MT, and NMT systems are far beneath human parity. DOC performs significantly better than SENT, suggesting that BWB contains rich discourse phenomena that can only be translated accurately when the context is taken into account. It is also worth noting that even though PE is the post-edit of the relatively poor-performing system BING, it still achieves surprisingly better performance than DOC at the document level. This observation confirms
2SENT and DOC are trained on BWB by fairseq (Ott et al., 2019), and the training details are in App.C.
3App. E describes how human assessment is carried out. The inter-rater agreement is reported in Tab. 11.
our claim that discourse phenomena have a huge impact on human judgment of translation quality.
Terminology Translation. We next turn to evaluating the performance of each system on terminology translation, depicted in Fig. 5. The recall rate of terminology is reported. As can be seen, although the accuracy of each system in the translation of common phrases is not significantly different, the performance of the PE system is superior to that of the MT systems in the translation of proper names. Notably, despite having been trained on in-domain data, the SENT system is unable to accurately translate terms on the test set due to the varied terminologies used in different novels, resulting in a performance that is even lower than that of commercial MT systems.
Entity Translation. For the evaluation of entity translations, we compared translations of PER and translations of other entities. As illustrated in Fig. 6, the translation of personal names and terms exhibits a similar trend, with the MT systems performing significantly worse than PE. This outcome is expected, as both personal names and terms present similar challenges for MT, including (1) the need for consistent translation, (2) the potential for multiword personal names to be combined in specific
ways that may not translate well when each word is translated individually, and (3) the possibility of personal names not having a direct equivalent in other languages. For example, in Fig. 9, the character name Ye Qing Luo (literal: night, clear, fall) is mistranslated into The night fell. Additionally, wordplay and puns based on character names may be difficult to convey in translation, as character names may be chosen for their sound or meaning in the original language, which may not translate effectively into other languages.
Pronoun Translation. Our evaluation of pronoun translation focuses on the impact of omitted pronouns on translation quality. As depicted
in Fig. 7, an increase in the proportion of omitted pronouns leads to a decrease in the translation quality of the human post-editing system. This finding suggests that pronoun omission is a major issue in Chinese-to-English translation. However, the BERTSCORE scores of the sentence-level and document-level MT systems do not follow a monotonic trend with the increase in the proportion of omitted pronouns. Rather, the highest value is reached when the proportion of omitted pronouns is within the interval of [0.6, 0.7]. This result indicates that NMT models process pronoun omissions in a manner that is fundamentally different from that of humans. In contrast, the BLONDE scores exhibit a pattern that is consistent with the trend observed in the PE system, indicating that while the omission of pronouns may not significantly affect the quality of translation at the sentence level, it is crucial for ensuring the overall quality of documentlevel translation.
Translation Coherence. Finally, the coreference annotations allow us to measure the coherence of translated texts. We first fine-tune a neural coreference model (Lee et al., 2017) trained on OntoNotes (Hovy et al., 2006). This end-toend model jointly performs mention detection and antecedent linking. We modify the model by replacing the span representations with SpanBERT embeddings (Joshi et al., 2020). Additionally, during inference, we inject mentions that match the reference annotations into the antecedent candidate sets after the mention detection stage. The model then greedily selects the most likely antecedent for each mention in a document, or a dummy antecedent that begins a new coreference chain with the mention. Coreference resolution performance and the coherence ranking are reported in Tab. 8. Coherence is compared at the document level according to the centering theory (Grosz et al., 1995).
We use the operationalization of Jiang et al. (2022b) and compare KP scores of each document.
As shown in Tab. 8, HT surpasses MT in terms of entity coherence. Additionally, human post-editing was found to achieve entity coherence comparable to that of HT. These results highlight the challenges that MT currently faces in regard to maintaining contextual coherence when translating longer texts.

heading: 6 Comparing to Related Work
text: Evaluation Test Suites for Context-Aware MT. Several context-aware test suites have been proposed in recent years (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and Hardmeier, 2018). Although they facilitate the development of context-aware machine translation, they are not without limitations: First, the previous test suites are limited in size, with the largest being Parcorfull (LapshinovaKoltunski et al., 2018) which has 82,379 words from three different domains. Moreover, it is often the case that test sets are annotated based on a small portion of a parallel corpus that comprises documents. However, the parallel corpus from which previous test sets are selected is generally quite small, which makes it impossible to train large indomain NMT models. In contrast, our test set is based on the ultra-large-scale parallel corpus BWB, which is twice as large as the OpenSubtitles fr-en corpus (Lison et al., 2018) upon which most previous challenge sets were based (see Tab. 9). This enables us to differentiate between the source of mistranslation being underfitting or the model not having the capability to model discourse structure. In addition, the scope of most test sets has been restricted to a single discourse phenomenon (the majority of which focus on pronoun translation), which makes it impossible to objectively compare a model’s translation capability for different discourse phenomena in the same language and domain. For example, it is hard to decipher whether the model misinterpreted a pronoun’s gender as a result of a coreference resolution error or as a consequence of misjudging the gender of the entity. In sharp contrast to this, we have four annotation layers on the same parallel corpus.
Monolingual Corpora with Discourse Annotations. Linguistically annotated corpora have contributed significantly to the advancement of key
natural language technologies such as named entity recognition (Tjong Kim Sang and De Meulder, 2003), coreference resolution (Lee et al., 2017), and discourse parsing (Surdeanu et al., 2015). The majority of evaluation has, however, only been conducted on monolingual corpora such as the BBN named entity and pronoun coreference corpus (Weischedel and Brunstein, 2005), the Penn Discourse Treebank (Miltsakaki et al., 2004; Webber et al., 2019), and the OntoNotes corpus (Hovy et al., 2006). And yet, languages differ considerably regarding the discourse phenomena they exhibit. In particular, different languages have different linguistic features that influence the application of cohesive devices, and there are language-specific constraints governing the choice of referring expressions. For instance, Fig. 1 demonstrates a prominent feature distinguishing Chinese from English. In Chinese, it is a common practice to omit pronouns, and an anaphoric link can be inferred from context without explicit assertion. English, in contrast, does not generally allow the omission of pronouns. Our dataset contains aligned parallel discourse annotations in two languages, allowing analysis of the transferability of current NER and coreference resolution models across languages.

heading: 7 Conclusion
text: In this paper, we introduced the bilingual parallel corpus BWB-test, which includes annotations of various discourse phenomena. Our analysis of these annotations revealed the significant challenges posed by discourse phenomena to MT. Therefore, we advocate for greater attention on discourse coherence and consistency of the outputs of NMT models. The main discourse challenges faced by MT include entity consistency, entity recognition, anaphoric information loss, and coreference. Additionally, the BWB corpus, with its rich discourse annotations, serves as a valuable resource for a variety of purposes, including studying the transferability of named entity recognition and coreference resolution, as well as the development of multilingual structured prediction models.
Limitations
There are several limitations to the current study. Firstly, as for now, the corpus used in this study only consists of a single language pair. Secondly, the coherence of the MT systems was evaluated using a fine-tuned conference model, as no anno-
tations were available for the MT outputs. However, as shown in Tab. 8, the fine-tuned conference model is not perfect and may affect the quality of our coherence evaluation. Thirdly, this paper focuses on using discourse annotations to reveal and analyze discourse phenomena and the challenges they present to machine translation. Using the annotations to improve MT models is beyond the scope of this study and is left for future work.
Ethical Considerations
The annotators were paid a fair wage, and the annotation process did not solicit any sensitive information from the annotators. In regard to the copyright of our dataset, as stated in the paper, the crawling script that we plan to release will allow others to reproduce our dataset faithfully and will not be in breach of any copyright. In addition, the release of our annotated test set will not violate the doctrine of Fair Use (US/EU), as the purpose and character of the use is transformative. Please refer to https://www.nolo.com/legal-encyclopedia/ fair-use-the-four-factors.html for relevant laws.

heading: A Document-Level Parallel Corpora
text: There are some document-level parallel corpora in the market: TED Talks of IWSLT dataset (Ansari et al., 2020), News Commentary (Tiedemann, 2012), LDC4 and OpenSubtitle (Lison et al., 2018). The sizes and average length of these corpora are summarized in Tab. 9. Below we review these document-level parallel corpora in detail.
LDC. This corpus consists of formal articles from the news and law domains. The articles ave syntactic structures such as conjoined phrases, which make machine translation challenging. However, the news articles in this corpus are relatively outdated.
IWSLT. This corpus contains the TED Talks that covers the variety of topics. However, it is quite small in scale, which makes training large transformer-based models impractical.
News Commentary. This corpus consists of political and economic commentary crawled from the web site Project Syndicate.5 However, the scale of this corpus is also quite small. Moreover, there are no parallel Chinese-to-English data available in this corpus.
Europarl. The corpus is extracted from the proceedings of the European Parliament. Only European language pairs are available in this corpus.
OpenSubtitle. This corpus is a collection of translated movie subtitles (Lison and Tiedemann, 2016). Besides being simple and short, the documents in this corpus are usually verbal and informal as well.
BWB. The BWB corpus is the largest corpus in terms of size. Moreover, the sentences and documents in BWB are substantially longer than previous corpora. It is also worth noting that BWB differs from previous corpora in terms of genre – in-depth human analysis shows that it is very challenging for current NMT systems due to its rich discourse phenomena.

heading: B Case Study
text: We provide two example chapters in BWB with coreference annotation in Fig. 8 and Fig. 9. We observe that the BWB dataset poses challenges for NMT in the following ways.
4https://www.ldc.upenn.edu 54https://wit3.fbk.eu
Entity Consistency. There are many named entities in the dataset that have a high repetition rate, such as fictional characters. Therefore, named entity consistency is a significant challenge in machine translation on this dataset. For example, the translations of Weibo and Qiao Lian in Fig. 8 are not consistent in context.
Entity Recognition and Retrieval. In addition to the fluency of entity translation, the adequacy of entity translation is another challenge in BWB. In the case of fictional characters with strange names, the NMT model may not correctly recognize named entities, resulting in extremely poor translation quality, as in Fig. 9. “Ye Qing Luo” could be literally translated as “night”, “clear”, “fall”; however, it is actually the name of a fictional character. Even though fictional characters are difficult to translate, they are relatively rare throughout the text, so it would be beneficial to abandon the assumption of inter-sentence independence in consideration of global contextual information. One potential way to alleviate this problem is to equip NMT models with an entity recognition module.
Anaphoric Information Loss. Chinese, being one of the pro-drop languages, omits many pronouns, while the English language does not, as shown in Tab. 5. Translating from Chinese to English thus requires context to infer the correct English pronouns to compensate for the anaphoric information loss of sentence-level Chinese-to-English translations.
Morphological Information Loss. Tense information is also frequently absent in Chinese and can only be inferred from context. In general, this problem, which we refer to as morphological information loss, is often encountered when translating from a morphologically poorer language to a morphologically richer one. In the case of Chinese-to-English translation, tense information is often lost, while in other language pairs, such as English-to-French and English-to-German, gender information is often missed since as French and German are morphologically richer than English.
Coreference. In addition, in Fig. 8, we observe that the focus entity of the document is shifting throughout the text (Qiao Lian −→ Shen Liangchuan −→ Wang Wenhao −→ Shen Liangchuan −→ Song Cheng), and this information is language-independent, i.e., consistent in source and target. This information could be used to improve the coherence of translation.

heading: C Experiment Setup
text: We adopt the parameters of Transformer Big (Vaswani et al., 2017) for both SENT and DOC. More precisely, the layers in the big encoders and decoders are N = 12 , the number of heads per layer is h = 16, the dimensionality of input and output is dmodel = 1024, and the inner-layer of a feed-forward networks has dimensionality dff = 4096. The dropout rate is fixed as 0.3. We adopt Adam optimizer with β1 = 0.9, β2 = 0.98, = 10−9, and set learning rate 0.1 of the same learning rate schedule as Transformer. We set the batch size as 6,000 and the update frequency as 16 for updating parameters to imitate 128 GPUs on a machine with 8 V100 GPU. The datasets are encoded by BPE with 60K merge operations.

heading: D The BWB Corpus
text: In this section, we describe three stages of the dataset creation process: collecting bilingual parallel documents, quality control and dataset split.
D.1 Bilingual Document Collection
385 Chinese web novels across multiple genres were selected, including action, fantasy, romance, comedy, science fiction, martial arts, etc. The genre distribution is shown in Fig. 2. We then scrape their corresponding English translations from the Internet.6 The English versions are translated by professional translators who are native speakers of English, and then corrected and aligned by professional editors at the chapter level. The text is converted to UTF-8 and certain data cleansing (e.g., deduplication) is
6https://readnovelfull.com
performed in the process. Chapters that contain poetry or couplets in classical Chinese are excluded as they are difficult to translate directly into English. Further, we exclude chapters with less than 5 sentences and chapters where the sequence ratio is greater than 3.0. The titles of each chapter are also removed, since most of them are neither translated properly nor at the document level. The sentence alignment is automatically performed by Bleualign7 (Sennrich and Volk, 2011). The final corpus has 384 books with 9,581,816 sentence pairs (a total of 461.8 million words).8
D.2 Quality Control
We hired four bilingual graduate students to perform the quality control of the aforementioned process. These annotators were native Chinese speakers and proficient in English. We randomly selected 163 chapters and asked the annotators to distinguish whether a document was well aligned at the sentence level by counting the number of misalignment. It is identified as a misalignment if, for example, line 39 in English corresponds to line 39 and line 40 in Chinese, but the tool made a mistake in combining the two sentences. We observed an alignment accuracy rate of 93.1%.
D.3 Dataset Split
We construct the development set and the test set by randomly selecting 160 chapters from 6 novels, which contain 3,018 chapters in total. To prevent any train-test leakage, these 6 novels are removed from the training set. Tab. 1 provides the detailed statistics of the BWB dataset split. In addition, we asked the same annotators who performed the quality control to manually correct misalignments in the development and test sets, and 7.3% of the lines were corrected in total.

heading: E Human Evaluation
text: We conducted human evaluation on the BWB test set following the protocol proposed by (Läubli et al., 2018, 2020). As stated in §5, we evaluated two units of linguistic context (SENTENCE and DOCUMENT) independently based on their respective FLUENCY and ADEQUACY. We showed raters isolated sentences in random order in the SENTENCE-level evaluation, whereas in the DOCUMENT-level evaluation, we presented entire documents and asked raters to evaluate a sequence of five sequential sentences at a time in order. The ADEQUACY evaluation was based solely on source texts, whereas neither source texts nor references were included in the FLUENCY evaluation.
The ADEQUACY evaluation was conducted by four professional Chinese to English translators, and the FLUENCY evaluation was conducted by four native English revisers. The four translators were different from the professional translators who performed human translation. For human evaluation, we deliberately invited another group of specialists to avoid making judgments biased towards human translation.
We adopted relative ranking because it has been shown to be more effective than direct assessment when conducted by experts rather than crowd workers (Barrault et al., 2019). In particular, raters were presented with the system outputs and were asked to evaluate the system outputs vis-à-vis one another, e.g., to decide whether system A was better than system B (with ties allowed).
By randomizing the order of presentation of the system outputs, we were able to blind the origin of the output sentences and documents. While in the SENTENCE-level evaluation, the system outputs were presented in different orders for each sentence, the DOCUMENT-level evaluation used the same ordering of systems within a document to help raters better assess global coherence.
Additionally, we used spam items for quality control.(Kittur et al., 2008). At the SENTENCE-level, we make one of the five options nonsensical in a small fraction of items by randomly shuffling the order of the translated words, except for 10% at the beginning and end. At the DOCUMENT-level, we randomly shuffle all translated sentences except the first and last sentence at the document level, rendering one of the five options nonsensical. If a rater marks a spam item as better than or equal to an actual translation, this is a strong indication that they did not read both options carefully.
7https://github.com/rsennrich/Bleualign 8We will release a crawling and cleansing script pointing to a past web arxiv that will enable others to reproduce our dataset
faithfully.
Each raters evaluated 180 documents (including 18 spam items) and 180 sentences (including 18 spam items). The 180 sentences were randomly sampled from PART1 or PART2. We spited the test set into two non-overlapping subsets, referred to as PART1 and PART2. Note that PART1 and PART2 were chosen from different books. Each rater evaluated both sentences and documents, but never the same text in both conditions so as to avoid repetition priming (Gonzalez et al., 2011). Each document or sentence was therefore evaluated by two raters, as shown in Tab. 10.
We report pairwise inter-rater agreement in Tab. 11. Cohen’s kappa coefficients were used:
κ = P (A)− P (E) 1− P (E) (1)
where P (A) is the proportion of times that two raters agree, and P (E) is the likelihood of agreement by chance.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 2 + Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2 + Appendix
C 3 Did you run computational experiments? 5
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. we used existing models
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? in lab experts
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Sec. 3.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Sec. 3.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Sec. 3.

heading: Future Work
text: Using the annotations to improve MT models is beyond the scope of this study and is left for future work. Ethical Considerations
The annotators were paid a fair wage, and the annotation process did not solicit any sensitive information from the annotators. In regard to the copyright of our dataset, as stated in the paper, the crawling script that we plan to release will allow others to reproduce our dataset faithfully and will not be in breach of any copyright. In addition, the release of our annotated test set will not violate the doctrine of Fair Use (US/EU), as the purpose and character of the use is transformative. Please refer to https://www. nolo. com/legal-encyclopedia/ fair-use-the-four-factors. html for relevant laws