Venue: NAACL 2022

# BlonDe: An Automatic Evaluation Metric for

Document-level Machine Translation

 Yuchen Eleanor Jiang\({}^{1*}\) Tianyu Liu\({}^{1}\) Shuming Ma\({}^{2}\) Dongdong Zhang\({}^{2}\) Jian Yang\({}^{2}\) Haoyang Huang\({}^{2}\) Rico Sennrich\({}^{3}\) Mrinmaya Sachan\({}^{1}\) Ryan Cotterell\({}^{1}\) Ming Zhou\({}^{4}\)

\({}^{1}\)ETH Zurich \({}^{2}\)Microsoft Research Asia \({}^{3}\)Universitat Zurich \({}^{4}\)Langboat.com

{yucjiang,tianyu.liu,ryan.cotterell,mrinmaya.sachan}@inf.ethz.ch

{shuming.ma,dongdong.zhang,t-jianya,haohua}@microsoft.com

sennrich@cl.uzh.ch ming.zhou@chiangxin.com

Most of the work was done while the first author was an intern at Microsoft Research Asia.BlinOne is more selective than bleu for document-level MT, and also shows a larger quality difference between human and machine translations.

###### Abstract

Standard automatic metrics, e.g., bleu, are not reliable for document-level MT evaluation. They can neither distinguish document-level improvements in translation quality from sentence-level ones, nor identify the discourse phenomena that cause context-agnostic translations. This paper introduces a novel automatic metric BlonDe1 to widen the scope of automatic MT evaluation from the sentence to the document level. BlonDe takes discourse coherence into consideration by categorizing discourse-related spans and calculating the similarity-based F1 measure of categorized spans. We conduct extensive comparisons on a newly constructed document-level translation dataset bwb. The experimental results show that BlonDe possesses better selectivity and interpretability at the document-level, and is more sensitive to document-level nuances. In a large-scale human study, BlonDe also achieves significantly higher Pearson's \(r\) correlation with human judgments compared to previous metrics.

Footnote 1: Bitlingual Evaluation of Document Translation.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1550 - 1565

July 10-15, 2022 ©2022 Association for Computational Linguistics

[https://github.com/EleanorJiang/BlonDe](https://github.com/EleanorJiang/BlonDe)

## 1 Introduction

Over the past years neural machine translation (NMT) models have become the models of choice in Machine Translation (MT; Luong et al., 2015; Vaswani et al., 2017; Zhang et al., 2018, _inter alia_). Although some recent work Hassan et al. (2018); Popel (2018); Bojar et al. (2018) suggests that NMT has achieved human parity at the sentence level, the reliability of these human-parity claims was quickly contested by Laubli et al. (2018, 2020), showing that there is a larger difference between human and machine translation quality when inter-sentential context is taken into account.

Therefore, document-level machine translation has received increased attention in the MT community. However, despite various modeling advances, the MT community still lacks an efficient and effective evaluation metric for document-level translation. Standard evaluation metrics for MT, e.g., bleuPapinieni et al. (2002), terSnover et al. (2006) and meteorBanerjee and Lavie (2005), focus on the quality of translations at the sentence level and do not consider discourse-level features.

Thus, test suites that perform context-aware evaluation by targeting characteristic discourse-level phenomena have been proposed Hardmeier et al. (2015); Guillou and Hardmeier (2016); Burchardt et al. (2017); Isabelle et al. (2017); Rios Gonzales et al. (2017); Muller et al. (2018); Bawden et al. (2018); Voita et al. (2019); Guillou and Hardmeier (2018), _inter alia_) for document-level MT. However, such test suites need to be re-created for new domains or even language pairs, and their construction can be very labor-intensive. We still lack an

Figure 1: BlonDe is more selective than bleu for document-level MT, and also shows a larger quality difference between human and machine translations.



easy-to-use automatic metric that can reliably discriminate the quality of document-level translation.

In this paper, we curate a large-scale document-level parallel corpus (bwb) from heterogeneous data sources, and quantify document-level translation mistakes by performing a large human study. On this dataset, we found that inconsistency,2 ellipsis, and ambiguity were the most noticeable phenomena critical for document-level MT, together amounting to 86.73% of MT mistakes. Based on this analysis, we propose BlonDe, an automatic metric that evaluates translation quality at the document level. At the core of the metric is the similarity-based bijection between subsets of reference and system categories, e.g., pronouns, inflected forms, discourse relations and lexicons, and phrases, e.g., named entities. It computes recall, precision and F1, along with the corresponding measure of \(n\)-grams. Furthermore, BlonDe can incorporate human annotation easily by computing scores of human-annotated categories in the same way.

Footnote 2: By inconsistency we mean the mistakes related to coreference and lexical cohesion Carpuat (2009); Guillou (2013).

We compare BlonDe with 11 other metrics and demonstrate that BlonDe is better at distinguishing between context-aware and context-agnostic MT systems. We also observe that the degree to which BlonDe correlates with sentence-level metrics (e.g., bleu) is lower than the degree to which the sentence-level metrics correlate with each other. This signals that BlonDe indeed captures additional aspects of translation quality beyond the sentence-level. Finally, our human evaluation also reveals significantly higher Pearson's \(r\) correlation coefficients between BlonDe and human assessments compared to other metrics.

## 2 bwb: Bilingual Web Book Dataset

To design a metric that is sensitive to document-level phenomena, we first curate a document-level Chinese-English parallel corpus, called bwb (**B**ilingual **W**eb **B**ooks). bwb consists of Chinese web novels across multiple genres (sci-fi, romance, action, fantasy, comedy, _inter alia_) and their corresponding English translations crawled from the Internet.

Dataset Creation.The novels are translated by professional native English speakers, and are corrected by editors. The sentence alignment of the training set is done by Bluealign3Sennrich and Volk (2011). We hired four bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting bwb parallel corpus and observe an alignment accuracy rate of 93.1%. We further asked the same batch of annotators to correct such misalignments in both the development and the test set. The details of the corpus creation and quality control are described in Appendix A.

Statistics.Table 1 summarizes the statistics of the bwb dataset. It is a much larger dataset, and contains longer documents and richer discourse phenomena compared to all previous documented datasets Lison and Tiedemann (2016); Koehn and Knowles (2017); Barrault et al. (2019); Koehn (2005); Liu and Zhang (2020). To the best of our knowledge, this is the largest Chinese-English document-level translation dataset to date.

Dataset Split.We treat chapters in our books as _documents_. The maximum, median, and minimum number of sentences per document are 46, 30 and 18, respectively. To prevent any train-test leakage, we split the dataset into a training, development and a test set such that chapters from the same book are part of the same split. We use 377 books for training, and randomly select 80 and 79 documents from the 3,018 documents in the remaining 6 books as the development and test sets, respectively.4

Footnote 4: One document in the development set was dropped due to its poor annotation quality.

## 3 Analyzing Discourse Errors

Next, we conduct a human study on the test set of bwb, in which we identify and categorize the discourse errors made by MT systems that are not captured in sentence-level evaluation. This human study is conducted by eight professional translators. The annotators are asked to classify translation errors into document-level and sentence-level errors (some cases can be both). sentence-level

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Statistic** & **Train** & **Test** & **Dev** & **Total** \\ \hline \#Docs & 196,304 & 80 & 79 & 196,463 \\ \#Sents & 9,576,566 & 2,632 & 2,618 & 9,581,816 \\ \#Words & 325.4M & 68.0K & 67.4K & 460.8M \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the bwb dataset.

 

errors refer to those errors that render the translations to be inadequate or not fluent as stand-alone sentences, while document-level errors reflect a coherence violation across multiple sentences in the document. document-level errors are further categorized according to the linguistic phenomena leading to the lower performance in the context-dependent translation.5

Footnote 5: The annotation guidelines are described in Appendix B.

Table 2 shows the result of our error analysis. A substantial proportion of translations have document-level errors (71.9%). This supports the claim that bwb contains rich discourse phenomena that common MT systems cannot address. We observe that three categories, i.e., inconsistency (64.4%), ellipsis (20.3%) and ambiguity (7.3%), account for the vast majority of document-level errors. Below we discuss these three categories of document-level errors and the design intuitions behind BlonDe.

Inconsistency.We consider two kinds of consistency in translation: lexical and grammatical. Lexical consistency is defined as a repetitive term that keeps the same translation throughout the whole document Carpuat and Simard (2012). Inconsistent translation of named entities can significantly impact translation output, although bleu may not be adversely affected Agrawal and Singla (2012); Hermjakob et al. (2008). Therefore, in the design of BlonDe, we also focus on the reiteration of named entities (e.g., Qiao in Figure 3). On the other hand, typical examples of grammatical consistency are tense and gender consistency. Tense consistency refers to the tense being _compatible_ with the context, rather than being exactly the same across the whole document. Tense inconsistency can arise when the source language is an isolating language and does not mark tense explicitly, e.g., in Chinese, and the target language is a synthetic language, e.g., English (_teal_ in Figure 3), where tense is marked explicitly. In the same spirit, the same entity should maintain a consistent grammatical gender. 6

Footnote 6: It is worth noting that the metric proposed in this study can be applied to a wider range of language pairs by extending the definition of grammatical consistency.

Ellipsis.Elipsis denotes the omission from a clause of one or more words that are nevertheless inferred in the context of the remaining elements Yamamoto and Sumita (1998); Voita et al. (2019). Translation errors arise when there are elliptical constructions in the source language while the target language does not allow for the same types of ellipsis. For example, the ellipsis of subjects or objects is very common in Chinese while it is ungrammatical in English--especially for pronouns. In Figure 3, _she (Qiao)_ is omitted in Chinese. However, it is hard to guess the gender of _Qiao_ from this stand-alone sentence: the correct pronoun choice can only be inferred from context (there is a _her_ in the previous sentence). Another common form of ellipsis is the omission of discourse markers, especially when the source language has more zero connective structures PoChing and Rimmington (2004) than the target language. In the example, _However_ and _So_ are ignored in src, which misleads the sentence-level system mta to ignore the discourse relations between sentences.

Ambiguity.Translation ambiguity occurs when a word in one language can be translated in more than one way into another language Tokowicz and Degani (2010). The cross-language ambiguity comes from several sources of within-language ambiguity including lexical ambiguity, polysemy, and near-synonymy. A unified feature of these is that ambiguous terms satisfy the form of one-to-many mappings. For the example in Figure 2, the word mta can be translated to _look_, _see_, _watch_ or _view_. Without access to the context, all the lexical choices are sensible.

\begin{table}
\begin{tabular}{l r r} \hline \hline Error Type & \# & \% \\ \hline no error & 451 & 17.1\% \\ sentence & 1351 & 51.3\% \\ document & 1893 & 71.9\% \\ \hline inconsistency & 1695 & 64.4\% \\ named entity & 1139 & 43.3\% \\ tense & 1018 & 38.7\% \\ ellipsis & 534 & 20.3\% \\ pronoun & 456 & 17.3\% \\ other & 103 & 4.0\% \\ ambiguity & 193 & 7.3\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: The statistics of translation errors in human analysis.

 



## 4 BlonDe

The aforementioned document-level phenomena have little impact on the \(n\)-gram statistics of translations. However, as is shown in Section 3, they can be key considerations for human readers when evaluating translations at the document level. Standard automatic metrics ignore the importance of contextual coherence of translations, which implies that the document-level nuances are not being properly modeled Zhou et al. (2008); Xiong and Zhang (2014). In this section, we describe BlonDe, an automatic metric that explicitly tracks discourse phenomena.

### Document-Level Evaluation

We first give the formulation of measuring discourse phenomena. We define a **document**\({\mathfrak{D}}=[S_{1},\ldots,S_{N}]\) as a sequence of \(N\) sentences. We take a **sentence**\(S\) of length \(T\) to be a string of tokens \(t_{1}\cdots t_{T}\) where each token \(t_{i}\) is taken from the vocabulary \(\mathcal{V}\). Let \(\mathsf{spans}\left(S\right)=\left\{m_{1},m_{2},\ldots\right\}\) be the set of spans in the sentence \(S\). Here, a **span** is a subsequence of the tokens in \(S=t_{1}\cdots t_{T}\).

Let us assume that we are interested in \(K\)**discourse categories**. Each of these categories capture a discourse phenomenon of interest. As shown in Section 3, named entity inconsistency, tense inconsistency and pronoun ellipsis make up the majority of discourse errors (67.8%) on the data analyzed. We therefore introduce three types of categories: entity, tense and pronoun. In addition, we introduce **discourse markers** dm as a category, which are the essential contextual links between the various discourse segments (See Figure 3).

For a certain discourse category of interest, \(k\), we assume that there are \(L_{k}\)**features**. In our case, the features of entity \(\mathcal{E}\) are a list of named entities in \({\mathfrak{D}}\); the features of tense are \(\mathcal{V}=[\texttt{MD},\texttt{VBD},\texttt{VBN},\texttt{VBP},\texttt{VBZ}, \texttt{VBG},\texttt{VB}]\),7 the features of pronoun are \(\mathcal{P}=[\texttt{masculine}\), feminine, neuter, epicene];8 the features of dm are \(\mathcal{M}=[\texttt{contingency},\texttt{temporal}\), expansion, comparison].9 Note that different categories can have different numbers of features and the number of features can be dynamic: \(\mathcal{E}\) depends on \({\mathfrak{D}}\) while \(\mathcal{V}\) and \(\mathcal{P}\) are fixed. The intuition behind this is that we want to encourage the system output to keep consistent tense and pronouns as well as the consistent translation for a specific named entity.

Footnote 7: MD: Modal; VBD: Verb (past tense verb); VBN: Verb (past participle); VBP: Verb (non-3rd person singular present); VBZ: Verb (3rd person singular present); VBG: Verb (gerund or present participle); VB: Verb (base form).

Footnote 8: nasculine: he, him, his himself; feminine: she, her, hers, herself, neuter? it, its, itself; epicene: they, them, their, their, theirs, themselves.

Footnote 9: A detailed explanation is provided in Table 6.

Let us now define \(\mathbf{C}_{k,l}\left(S\right)\subseteq\mathsf{spans}\left(S\right)\) as the set of spans in \(S\) that share the \(l^{\text{th}}\) feature in the \(k^{\text{th}}\) discourse category. To give a concrete example, let us assume that tense is the \(k^{\text{th}}\) category and VBD is the \(l^{\text{th}}\) feature in this category. The corresponding \(\mathbf{C}_{k,l}(S)\) is the set of the spans (in this case, unigrams) tagged with VBD in the sen

tence \(S\). In Figure 3, all the spans are colored. We then let \({\bf C}_{k}(S)\) refer to a vector of size \(L_{k}\) where each element of that vector is the set \({\bf C}_{k,l}(S)\). The sets of spans for entity, tense, pronoun and dm can be produced by an NER model, a POS tagger, a rule-based string match and a discourse marker, respectively. We also define a weight vector \(\mathbf{w}_{k}=[w_{k,l}:l\in\{1,\dots,L_{k}\}]\) for each discourse category \(k\), where each entry \(w_{k,l}\) corresponds to the weight given to a feature.

We then define the **discourse representation** of sentence \(S\) as the concatenation of all categories:

\[{\bf C}(S)=[{\bf C}_{k}\left(S\right):k\in\{1,\dots,K\}] \tag{1}\]

Similarity.Let \(\textbf{sim}:{\bf C}(S^{s})\times{\bf C}(S^{r})\rightarrow\mathbb{R}^{K}\) represent a similarity vector which measures category-wise similarity between the discourse representations of two sentences \(S^{r}\) and \(S^{s}\). Each entry of the vector **sim** takes non-negative values: The entry being zero if \(S^{s}\) and \(S^{r}\) have no shared spans with the same discourse category.

The similarity vector **sim** defined here can be implemented in several ways. A possible implementation of **sim** can be achieved by counting the number of functionally similar spans for each feature and then taking a weighted sum over all features:

\[\textbf{sim}(S^{s}, S^{r})= \tag{2}\] \[[\textbf{sim}_{k}(S^{s},S^{r}):k\in\{1,\dots,K\}]\]

where each entry \(\textbf{sim}_{k}\) is defined as follows:

\[\textbf{sim}_{k}(S^{s},S^{r})= \tag{3}\] \[\mathbf{w}_{k}~{}\odot~{}\min\left(\textbf{count}({\bf C}_{k}(S^{s}) )\,,\textbf{count}({\bf C}_{k}(S^{r}))\right)\]

where

\[\textbf{count}({\bf C}_{k}(\cdot)) \tag{4}\] \[=[|{\bf C}_{k,l}(\cdot)|:l\in\{1,\dots,L_{k}\}]\]

denotes the cardinality of \({\bf C}_{k,l}\) applied entry-wise and \(\min\) denotes the minimum function applied element-wise. Intuitively, \(\textbf{sim}_{k}\), measures the number of functionally similar spans shared by \(S^{s}\) and \(S^{r}\). Assume that tense is the \(k^{\text{th}}\) category.10

Footnote 10: In Figure 3, \(\textbf{sim}_{k}(U_{b}^{\textsc{int}},U_{b}^{\textsc{int}})=0\) since mta mistranslated the verbs as present tense due to the exclusion of context. The total similarity \(\textbf{sim}(\mathfrak{D}^{\textsc{mta}},\mathfrak{D}^{\textsc{infer}})\) is the number of functionally similar spans across all features: (1, 2, 4, 0) for \(\mathcal{E}\), \(\mathcal{V}\), \(\mathcal{M}\)). Here, we assume all category weights are 1.

It is worth noting that there are many other reasonable ways to operationalize **sim**. For entity, partial credit could be assigned to two named entities if they have overlapping tokens; for tense and pronoun, partial credit could be assigned to two similar categories, e.g., VBP and VB; for dm, partial credit could be assigned according to the sense hierarchy and the confidences in the detected discourse markers. We leave the expansion of the **sim** definition to future work.

A Document-level Similarity Measure.Now we turn from measuring the similarity at the sentence level to the document level. We first lift \(\textbf{sim}(\cdot,\cdot)\) to measure the similarity between two documents:

\[\textbf{sim}\left(\mathfrak{D}^{s},\mathfrak{D}^{r}\right)=\underset{S^{s} \in\mathfrak{D}^{s}}{\sum}\textbf{sim}(S^{s},S^{r}) \tag{5}\]

where the sum is applied element-wise.

We then define \(\textbf{sim}(\cdot,\cdot)\) for a system document \(\mathfrak{D}^{s}\) and a set of reference documents \(\mathbb{D}^{r}=\{\mathfrak{D}^{r_{1}},\mathfrak{D}^{r_{2}},\ldots\}\) by aggregating the **sim** of all sentences in \(\mathfrak{D}^{s}\) and \(\mathbb{D}^{r}\):

\[\textbf{sim}\left(\mathfrak{D}^{s},\mathbb{D}^{r}\right)=\underset{S^{s}\in \mathfrak{D}^{s}}{\sum}\bigoplus_{S^{r}\in\mathbb{D}^{r}}\textbf{sim}(S^{s},S^ {r}) \tag{6}\]

Here, \(\oplus\) is a generic aggregator over multiple references, e.g., \(\oplus=\max\), if we take the reference which has the maximum similarity with the system output; or \(\oplus=\sum\) if we sum up the similarity scores of all references. Again, \(\oplus\) is applied element-wise.11 We also reuse the notation \(\textbf{sim}(\cdot,\cdot)\) for two sets of documents \(\mathbb{D}^{s}\) and \(\mathbb{D}^{r}\):

Footnote 11: In Figure 3, since we only have one reference, \(\textbf{sim}(\mathfrak{D}^{\textsc{mrfa}},\mathbb{D}^{r})=\textbf{sim}( \mathfrak{D}^{\textsc{mrfa}},\mathfrak{D}^{\textsc{mrf}})=(1,2,4,0)\).

\[\textbf{sim}\left(\mathbb{D}^{s},\mathbb{D}^{r}\right)=\underset{\mathfrak{D}^ {s}\in\mathbb{D}^{s}}{\bigoplus}\textbf{sim}\left(\mathfrak{D}^{s},\mathbb{ D}^{r}\right) \tag{7}\]

Note that the similarity vector can also be computed for the same (set of) documents. For example, if **sim** is implemented as counting the number of functionally similar spans for each feature, then, \(\textbf{sim}\left(\mathfrak{D}^{s},\mathfrak{D}^{s}\right)\) and \(\textbf{sim}\left(\mathbb{D}^{r},\mathbb{D}^{r}\right)\) denote the total number of spans of each category in the system output and the reference, respectively.12

Footnote 12: In Figure 3, \(\textbf{sim}(\mathfrak{D}^{\textsc{mrfa}},\mathfrak{D}^{\textsc{mrfa}})=(1,7,6,0)\) and \(\textbf{sim}\left(\mathbb{D}^{r},\mathbb{D}^{r}\right)=\textbf{sim}\left( \mathfrak{D}^{\textsc{mrf}},\mathfrak{D}^{\textsc{mrf}}\right)=(2,7,5,2)\).

Scoring.We are now ready to define the "goodness" of a system output with respect to our discourse phenomenon of interest. We compute the 

precision, recall and F1 for all \(K\) discourse categories defined as follows:

\[{\mathbf{p}}(\mathfrak{D}^{s},\mathbb{D}^{r})=\frac{\text{\bf sim}\left( \mathfrak{D}^{s},\mathbb{D}^{r}\right)}{\text{\bf sim}\left(\mathfrak{D}^{s}, \mathfrak{D}^{s}\right)}, \tag{8}\] \[{\mathbf{r}}(\mathfrak{D}^{s},\mathbb{D}^{r})=\frac{\text{\bf sim} \left(\mathfrak{D}^{s},\mathbb{D}^{r}\right)}{\text{\bf sim}\left(\mathbb{D}^ {r},\mathbb{D}^{r}\right)},\] (9) \[{\mathbf{F}}(\mathfrak{D}^{s},\mathbb{D}^{r})=2\cdot\frac{{\mathbf{p}} \odot{\mathbf{r}}}{{\mathbf{p}}+{\mathbf{r}}}. \tag{10}\]

Here, \({\mathbf{p}}(\mathfrak{D}^{s},\mathbb{D}^{r})\), \({\mathbf{r}}(\mathfrak{D}^{s},\mathbb{D}^{r})\) and \({\mathbf{F}}(\mathfrak{D}^{s},\mathbb{D}^{r})\) are all \(K\)-dimensional vectors; where, the \(k^{\text{th}}\) element of these vectors represents the precision, recall and F-score for the \(k^{\text{th}}\) category. Thus, the addition, multiplication and division operations above are also defined element-wise.13

Footnote 13: In Figure 3, recall that \(\text{\bf sim}(\mathfrak{D}^{\textsc{mt}},\mathbb{D}^{r})=(1,2,4,0)\), \(\text{\bf sim}(\mathfrak{D}^{\textsc{mt}},\mathfrak{D}^{\textsc{mt}})=(1,7,6,0)\) and \(\text{\bf sim}\left(\mathbb{D}^{r},\mathbb{D}^{r}\right)=(2,7,5,2)\). Thus, we have \({\mathbf{p}}(\mathfrak{D}^{\textsc{mt}},\mathbb{D}^{r})=\left(\frac{1}{1},\frac{2} {7},\frac{4}{6},0\right)=(1,\frac{2}{7},\frac{4}{3},\text{NA})\) where NA denotes an missing value. Furthermore, we have \({\mathbf{r}}(\mathfrak{D}^{\textsc{mt}},\mathbb{D}^{r})=\left(\frac{1}{2},\frac{2 }{7},\frac{4}{5},\frac{9}{2}\right)=\left(\frac{1}{2},\frac{2}{7},\frac{4}{5}, \frac{6}{2}\right)\) where \(\delta\) denotes a small value (0.0001) for sonothing. Finally, we have \({\mathbf{F}}(\mathfrak{D}^{\textsc{mt}},\mathbb{D}^{r})=\left(\frac{2}{7},\frac{ 2}{7},\frac{8}{11},\text{NA}\right).\)

BlonD-d.P\((\mathfrak{D}^{s},\mathbb{D}^{r})=\)

\[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\left(\prod_{k=1}^{ K}({\mathbf{p}}_{k}(\mathfrak{D}^{s},\mathbb{D}^{r})^{a_{k}})\right)^{1/\sum\limits_{ k=1}^{K}a_{k}}}\] BlonD-d.R \[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\left(\prod_{k=1}^{K}({\mathbf{r}}_{k}( \mathfrak{D}^{s},\mathbb{D}^{r})^{a_{k}})\right)^{1/\sum\limits_{k=1}^{K}a_{k}}}\]

where \(a_{k}\) denotes the importance weight of the \(k^{\text{th}}\) category, and \({\mathbf{p}}_{k}\) and \({\mathbf{r}}_{k}\) denote the \(k^{\text{th}}\) entry of \({\mathbf{p}}\) and \({\mathbf{r}}\), respectively.14 Therefore, BlonD-d.F1 is defined as follows:

Footnote 14: BlonD–d adopts uniform weights.

BlonD-dopts uniform weights.

\[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\left(\prod_{i=1}^{3}\right)}\]

\[2\cdot\frac{\text{\sc BlonD-d.P}\cdot\text{\sc BlonD-d.R}}{\text{\sc BlonD- d.P}+\text{\sc BlonD-d.P}+\text{\sc BlonD-d.P}+\text{\sc BlonD-d.R}}\]

Whenever not otherwise specified, we simply use BlonD-d to refer to BlonD-d.F1.15

Footnote 15: E.g., the BlonD-d scores of mta in Figure 3 are:

[MISSING_PAGE_POST]

\begin{table}
\begin{tabular}{c|c|c c c|c} \hline \multicolumn{4}{l}{} & \multicolumn{2}{c}{bleu} & \multicolumn{2}{c}{BlonDe} & \multicolumn{2}{c}{Blon 

BlonDe covers both discourse coherence features and sentence-level adequacy, thus providing a comprehensive measurement of translation quality. Table 3 compares BlonDe with bleu using the two MT outputs found in Figure 3. It is striking that bleu rates mta higher than mtb given that mtb is clearly better than mta to human readers. In sharp contrast, their BlonDe scores reflect the correct ranking in translation quality.

### BloD+: Combining BlonDe with Human Annotations

BlonDe is easy to generalize--for instance, it would be easy to incorporate human annotations, e.g., one could annotate spans related to discourse errors and treat them as categories. The automatically inferred categories and human annotated categories are then combined by adopting the same weighted averaging approach, which we call BlonD+. We hired the same translators who analyzed discourse errors in Section 3 to annotate ambiguous and omitted word/phrases on the test set of bwb.17

Footnote 17: We also make this annotated test publicly available as a testbed for evaluating the ability of MT systems to disambiguate word senses and to predict coherent pronouns or discourse markers in the case of omission.

## 5 Experiments

In this section, we examine the effectiveness of BlonDe at the document-level MT evaluation through experiments. We answer the following question: Do differences in BlonDe reliably reflect differences in the document-level translation quality of different MT systems? To answer this question, we run several MT baselines and compare their BlonDe scores to eleven other metrics:

Standard Sentence-level Metrics.bleu(Papineni et al., 2002), meteorBanerjee and Lavie, 2005), terSnover et al. (2006), rouge-lLin (2004), CIDErVedantam et al. (2015).

Document-level Metrics.lc and rcWong and Kit (2012)--these are ratios between the number of lexical cohesion devices (repetition and collocation) and repeated content words over the total number of content words in a target document, which are direct measurements of lexical cohesion.

Embedding-based Metrics.We consider four embedding-based metrics in this work: SkipThought cosine similarity Kiprois et al. (2015), embedding average cosine similarity Ayer; Sharma et al. (2017), Vector extrema cosine similarity Vector; Forgues et al. (2014), Greedy Match Greedy; Rus and Lintean (2012).

### MT Systems

We test BlonDe on the following system outputs: an smt system Chiang (2007), three well-known commercial NMT systems Omt-a, omt-b, omt-c), a sentence-level transformer-based system (mt-s) and a document-level system Ortd) trained on bwb. mt-dZhang et al. (2018) trains sentence-level model parameters and then estimates document-level model parameters while keeping the sentence-level Transformer model parameters fixed. We adopt Transformer Big Vaswani et al. (2017) for both mt-s and mt-d. The final "system" is a human post-editing pe on omt-c provided by professional translators, so it is supposed to be the strongest baseline.18

Footnote 18: We trained models by fairseq Ott et al. (2019). Model parameters and the post-editing details are in Appendix F.2 and C, respectively.

### The BlonDe Evaluation

Firstly, we leverage the test set of bwb and evaluate the above-mentioned systems by BlonDe and other metrics. Figure 4 presents the means of all metrics along with the 95% confidence interval estimated from bootstrap resampling. We observe that the BlonDe scores demonstrate an exponentially increasing trend from sentence-level towards document-level and human post-editing, while the trends of standard metrics are mostly linear. Specifically, the difference between the BlonDe scores of mt-s and mt-d (denoted as \(\Delta\)(mt-s, mt-d)) is significantly higher than the difference between the \(\Delta\)(mt-s, mt-d) in their bleu scores. An even larger \(\Delta\) between mt-d and pe in their BlonDe scores is observed, indicating mt-d is still far away from achieving human parity. Note that the trend of BlonD-d scores is even more exponential, which indicates that BlonD-d indeed distills documentation-level translation quality.

The \(t\)-statistics of the paired sample \(t\)-tests of individual documents are given in Table 4. Unlike bleu, meteor and other metrics, which either fails to distinguish human and machine translation or has lower discriminative power compared to distinguishing different machine translations, the BlonDe family maintain similar discriminative 

power across the pair-wise comparisons. Interestingly but not surprisingly, the non-reference-based lc and rc fail to distinguish both (mt-s, mt-d) and (mt-d, pe), since sentence-level MT is by nature more repetitive than human translation and thus it is hard to distinguish accidental repetition from document-level cohesion.

In addition, the \(t\)-statistics of BlonD-d categories provide rich diagnostic information. As can be seen, although transformer-based NMT models have substantially higher bleu scores than smt systems, mt-s is not statistically superior to smt in terms of named entity translation. However, human post-editing scores significantly better on entity translation--meaning that named entity translation accounts for a substantial part of quality differences between machine and human. In terms of tense and and dm translation, mt-d does not significantly out-perform mt-s, which could be taken into consideration in future document-level MT model designs.

We also show the pairwise Pearson correlations between different metrics in Figure 5. It illustrates the homogeneity/heterogeneity of different metrics. We report the absolute value of the correlation for ter. We see that while sentence-level metrics (bleu, meteor and rouge-l) have strong correlations with each other, BlonDe correlates less well with those metrics.

Figure 4: The mean scores of different system outputs given by different metrics on the bwb test set. Shaded region represents 95% confidence interval.

Figure 5: Absolute Pearson correlation pairs of automatic metrics. Computed over the scores of individual documents in bwb test set.

 



### Human Evaluation

We then evaluate BlonDe along with other metrics in terms of their Pearson correlation with human assessment. Our human assessment is provided by four professional Chinese to English translators and four native English revisiers. Two experimental units (sentence vs document) are assessed independently in terms of fluency and adequacy, respectively. In the sentence-level evaluation, we show the raters isolated sentences, while in the document-level evaluation, we show them entire documents and we only ask raters to evaluate the overall quality of sequential blocks of sentences (5 sentences per block) as used in the Relative Ranking (RR) evaluation Bojar et al. (2016). We use the Williams significance test Williams (1959); Graham and Baldwin (2014) following the practice adopted by WMT Mathur et al. (2020) to identify correlation differences are statistically significant. The detailed protocol is presented in Appendix D.

The results are shown in Table 5. BlonDe obtains the highest correlation with human assessment at both the sentence level and the document level. However, BlonDe correlates remarkably better with human assessment when context is taken into account, and it only significantly outperforms all other metrics at document level.

It is worth noting that BlonDe also correlates well with fluency assessment, even though it is, in essence, still a reference-based metric. One possible explanation for this unexpected positive result is that it tracks span categories that directly relate to cohesion and coherence. Another important observation is that the recall-based BlonDe variants generally correlate better with human assessment, yet appears to be less selective compared to the precision-based variants (see mt-d\(\rightarrow\) pe in Table 4). This provides support for adopting the F1 in order to get the best of both worlds.

## 6 Related Work

There have been a few studies on automatic evaluation metrics for specific discourse phenomena.

Pronoun TranslationHardmeier and Federico (2010) measured the precision and recall of pronouns directly and Miculicich Werlen and Popescu-Belis (2017) proposed to estimate the accuracy of pronoun translation (APT) by aligning source and target texts. However, as shown in Guillou and Hardmeier (2018), APT does not take the antecedents of an anaphoric pronoun into account. They cannot handle the mismatches in the numbers of pronouns either. Jwalapuram et al. (2019) also proposed a specialized measure for pronoun evaluation which involves training. In comparison, BlonDe does not rely on any alignment or training.

Lexical CohesionWong and Kit (2012) proposed lc and rcGong et al. (2015) described a cohesion function to measure text cohesion via lexical chain and a gist consistency score based on topic model. However, they fail to distinguish accidental repetition from document-level cohesion.

Discourse RelationsHajlaoui and Popescu-Belis (2013) proposed to assess the accuracy of connective translation (ACT). However, such an assessment requires a bilingual dictionary of all possible DM translations, whereas BlonDe only requires a list of monolingual DMs. Guzman et al. (2014) and Joty et al. (2014) compute a metric based on the similarity between the discourse trees of reference and system output. Those discoursrepresentation-based metrics are indirect, and rely on discourse parsing tools, which are much more inaccurate than syntactic and semantic parsing tools used in BlonDe. Unlike previously proposed metrics, BlonDe does not only focus on one specific discourse phenomenon, and thus has significantly

\begin{table}
\begin{tabular}{c|r r|r r} \hline \hline \multirow{2}{*}{Unit} & \multicolumn{2}{c}{sentence} & \multicolumn{2}{c}{document} \\  & ade & Flu & ade & flu \\ \hline BlonDeR & **363** & .327 & .436\({}^{\dagger}\) & .371\({}^{\dagger}\) \\ BlonDeP & .331 & .296 & .383\({}^{\dagger}\) & .344\({}^{\dagger}\) \\ BlonDeF1 & .35 & .314 & .417\({}^{\dagger}\) & .358 \({}^{\dagger}\) \\ BlonD+R & .364 & **.329** & **.44\({}^{\dagger}\)** & **.373\({}^{\dagger}\)** \\ BlonD+P & .334 & .3 & .39\({}^{\dagger}\) & .349\({}^{\dagger}\) \\ BlonD+Fl & .351 & .318 & .422\({}^{\dagger}\) & .362\({}^{\dagger}\) \\ \hline bleu & .325 & .308 & .343 & .266 \\ meteor & .338 & .31 & .339 & .278 \\ rootg-l & .275 & .262 & .29 & .211 \\ ter & .063 & .027 & .044 & .092 \\ CIDER & .139 & .116 & .114 & .087 \\ skip & .213 & .174 & .163 & .171 \\ aver & .163 & .163 & .16 & .111 \\ vector & .25 & .243 & .248 & .218 \\ greedy & .323 & .3 & .307 & .265 \\ \hline lc & .086 & .061 & .153 & .116 \\rc & .096 & .07 & .169 & .13 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Absolute Pearson correlation with human judgments on bwb. The highest correlations are in bold. Correlation of metrics not significantly outperformed by any other metrics are highlighted with \(\dagger\). The BlonDe family are not tested against each other.



higher Pearson correlation coefficients with human assessments.

## 7 Conclusion

In this paper, we build a large-scale parallel dataset for document-level translation, bwb. We analyze it for common document-level translation errors in practice and propose BlonDe, an interpretable automatic metric for document-level MT evaluation. We further improve BlonDe by diagnosing and distilling discourse-related errors in MT outputs and human-annotations to obtain two improved metrics BlonD-d and BlonD+. These metrics were shown to have better selectivity than various sentence-level metrics and correlate better with human judgments.

## Ethical Considerations

The annotators were paid a fair wage and the annotation process did not solicit any sensitive information from the annotators. Finally, while our approach is not tuned for any specific real-world application, the approach could be used in sensitive contexts such as legal or health-care settings, and any work must use our approach undertake extensive quality-assurance and robustness testing before using it in their setting.

Replicability.As part of our contributions, we will release the annotated bwb test set, and release the crawling script of the training set under Fair Use rules. The BlonDe package is also publicly available at [https://github.com/EleanorJiang/BlonDe](https://github.com/EleanorJiang/BlonDe).

## Acknowledgements

Mrinmaya Sachan acknowledges support from an ETH Zurich Research grant (ETH-19 21-1) and a grant from the Swiss National Science Foundation (project #201009) for partially supporting this work.

## References

* Agrawal and Singla (2012) Neeraj Agrawal and Ankush Singla. 2012. Using named entity recognition to improve machine translation. Technical report, Stanford University.
* Ansari et al. (2015) Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondrej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stuker, Marco Turchi, Alexander Waibel, and Changhan Wang. 2020. Findings of the IWSLT 2020 evaluation campaign. In _Proceedings of the 17th International Conference on Spoken Language Translation_, pages 1-34, Online. Association for Computational Linguistics.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
* Barrault et al. (2019) Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 1-61, Florence, Italy. Association for Computational Linguistics.
* Bawden et al. (2018) Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1304-1313, New Orleans, Louisiana. Association for Computational Linguistics.
* Bojar et al. (2018) Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
* Bojar et al. (2016) Ondrej Bojar, Yvette Graham, Amir Kamran, and Milos Stanojevic. 2016. Results of the WMT16 metrics shared task. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_, pages 199-231, Berlin, Germany. Association for Computational Linguistics.
* Burchardt et al. (2017) Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Peter Jan-Thorsten, and Philip Williams. 2017. A linguistic evaluation of rule-based, phrase-based, and neural MT engines. _The Prague Bulletin of Mathematical Linguistics_, 108(1):159.
* Carpuat (2009) Marine Carpuat. 2009. One translation per discourse. In _Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009)_, pages 19-27, Boulder, Colorado. Association for Computational Linguistics 

Marine Carpuat and Michel Simard. 2012. The trouble with SMT consistency. In _Proceedings of the Seventh Workshop on Statistical Machine Translation_, pages 442-449, Montreal, Canada. Association for Computational Linguistics.
* Chiang (2007) David Chiang. 2007. Hierarchical phrase-based translation. _Computational Linguistics_, 33(2):201-228.
* Forgues et al. (2014) Gabriel Forgues, Joelle Pineau, Jean-Marie Larcheveque, and Real Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In _Workshop on Modern Machine Learning and Natural Language Processing_, volume 2.
* Gong et al. (2015) Zhengxian Gong, Min Zhang, and Guodong Zhou. 2015. Document-level machine translation evaluation with gist consistency and text cohesion. In _Proceedings of the Second Workshop on Discourse in Machine Translation_, pages 33-40, Lisbon, Portugal. Association for Computational Linguistics.
* Gonzalez et al. (2011) Cleotile Gonzalez, Brad Best, Alice F. Healy, James A. Kole, and Lyle E. Bourne Jr. 2011. A cognitive modeling account of simultaneous learning and fatigue effects. _Cognitive Systems Research_, 12(1):19-32.
* Graham and Baldwin (2014) Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 172-176, Doha, Qatar. Association for Computational Linguistics.
* Guillou (2013)Liane Guillou. 2013. Analysing lexical consistency in translation. In _Proceedings of the Workshop on Discourse in Machine Translation_, pages 10-18, Sofia, Bulgaria. Association for Computational Linguistics.
* Guillou and Hardmeier (2016) Liane Guillou and Christian Hardmeier. 2016. PROTEST: A test suite for evaluating pronouns in machine translation. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 636-643, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Guillou and Hardmeier (2018) Liane Guillou and Christian Hardmeier. 2018. Automatic reference-based evaluation of pronoun translation misses the point. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4797-4802, Brussels, Belgium. Association for Computational Linguistics.
* Guzman et al. (2014) Francisco Guzman, Shafiq Joty, Lluis Marquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 687-698, Baltimore, Maryland. Association for Computational Linguistics.
* Hajlaoui and Popescu-Belis (2013) Najeh Hajlaoui and Andrei Popescu-Belis. 2013. Assessing the accuracy of discourse connective translations: Validation of an automatic metric. In _International Conference on Intelligent Text Processing and Computational Linguistics_, pages 236-247. Springer.
* Hardmeier and Federico (2010) Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In _Proceedings of the 7th International Workshop on Spoken Language Translation: Papers_, pages 283-289, Paris, France.
* Hardmeier et al. (2015) Christian Hardmeier, Preslav Nakov, Sara Stymne, Jorg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT shared task on pronoun translation. In _Proceedings of the Second Workshop on Discourse in Machine Translation_, pages 1-16, Lisbon, Portugal. Association for Computational Linguistics.
* Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. _arXiv preprint arXiv:1803.05567_.
* learning when to transliterate. In _Proceedings of ACL-08: HLT_, pages 389-397, Columbus, Ohio. Association for Computational Linguistics.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
* Isabelle et al. (2017) Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2486-2496, Copenhagen, Denmark. Association for Computational Linguistics.
* Joty et al. (2014) Shafiq Joty, Francisco Guzman, Lluis Marquez, and Preslav Nakov. 2014. DiscoTK: Using discourse structure for machine translation evaluation. In _Proceedings of the Ninth Workshop on Statistical Machine Translation_, pages 402-408, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Jwalapuram et al. (2019) Prathyusha Jwalapuram, Shafiq Joty, Irina Temnikova, and Preslav Nakov. 2019. Evaluating pronominal anaphora in machine translation: An evaluation measure and a test suite. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2964-2975, Hong Kong, China. Association for Computational Linguistics 

Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In _Advances in Neural Information Processing Systems_, volume 28.
* Kittur et al. (2008) Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_, pages 453-456.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _Proceedings of Machine Translation Summit X: Papers_, pages 79-86, Phuket, Thailand.
* Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In _Proceedings of the First Workshop on Neural Machine Translation_, pages 28-39, Vancouver. Association for Computational Linguistics.
* Laubli et al. (2020) Samuel Laubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human-machine parity in language translation. _Journal of Artificial Intelligence Research_, 67:653-672.
* Laubli et al. (2018) Samuel Laubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? A case for document-level evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lison and Tiedemann (2016) Pierre Lison and Jorg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 923-929, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Lison et al. (2018) Pierre Lison, Jorg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Liu and Zhang (2020) Siyou Liu and Xiaojun Zhang. 2020. Corpora for document-level neural machine translation. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 3775-3781, Marseille, France. European Language Resources Association.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
* Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020. Results of the WMT20 metrics shared task. In _Proceedings of the Fifth Conference on Machine Translation_, pages 688-725, Online. Association for Computational Linguistics.
* Werlen and Popescu-Belis (2017) Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (APT). In _Proceedings of the Third Workshop on Discourse in Machine Translation_, pages 17-25, Copenhagen, Denmark. Association for Computational Linguistics.
* Muller et al. (2018) Mathias Muller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 61-72, Brussels, Belgium. Association for Computational Linguistics.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weihg Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Po-Ching and Rimmington (2004) Yip Po-Ching and Don Rimmington. 2004. _Chinese: A Comprehensive Grammar_. Routledge.
* Popel (2018) Martin Popel. 2018. CUNI transformer neural MT system for WMT18. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 482-487, Belgium, Brussels. Association for Computational Linguistics.
* Gonzales et al. (2017)Annette Rios Gonzales, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In _Proceedings of the Second Conference on Machine Translation_, pages 11-19, Copenhagen, Denmark. Association for Computational Linguistics.
* Rus and Linean (2012) Vasile Rus and Mihai Linean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In _Proceedings of the Seventh Workshop on Building 

Educational Applications Using NLP_, pages 157-162, Montreal, Canada. Association for Computational Linguistics.
* Sennrich and Volk (2011) Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In _Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011)_, pages 175-182, Riga, Latvia. Northern European Association for Language Technology (NEALT).
* Sharma et al. (2017) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. _CoRR_, abs/1706.09799.
* Sileo et al. (2019) Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3477-3486, Minneapolis, Minnesota. Association for Computational Linguistics.
* Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_, pages 223-231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
* Tokowicz and Degani (2010) Natasha Tokowicz and Tamar Degani. 2010. Translation ambiguity: Consequences for learning and processing. _Research on Second Language Processing and Parsing_, pages 281-293.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30.
* Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In _Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition_, pages 4566-4575.
* Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1198-1212, Florence, Italy. Association for Computational Linguistics.
* Williams (1959) Evan James Williams. 1959. _Regression Analysis_, volume 14. Wiley.
* Wong and Kit (2012) Billy T. M. Wong and Chunyu Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 1060-1068, Jeju Island, Korea. Association for Computational Linguistics.
* Xiong and Zhang (2014) Deyi Xiong and Min Zhang. 2014. Semantics, discourse and statistical machine translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Tutorials_, pages 11-12, Baltimore, Maryland, USA. Association for Computational Linguistics.
* Yamamoto and Sumita (1998) Kazuhide Yamamoto and Eiichiro Sumita. 1998. Feasibility study for ellipsis resolution in dialogues by machine-learning technique. In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2_, pages 1428-1435, Montreal, Quebec, Canada. Association for Computational Linguistics.
* Zhang et al. (2018) Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 533-542, Brussels, Belgium. Association for Computational Linguistics.
* Zhou et al. (2008) Ming Zhou, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. In _Proceedings of the 22nd International Conference on Computational Linguistics_, pages 1121-1128, Manchester, UK.

 

Dataset Creation

The Background of Translators.The original Chinese books are translated by professional native English speakers, and are corrected by editors.

Data Collection.This process is implemented by a python web crawler, and certain data cleaning is also done in the process. We crawl the books chapter by chapter, and convert the text to UTF-8. After deduplication, we remove the chapters with less than 5 sentences. We further remove the titles of each chapter, because most of them are neither translated properly nor in the document-level.

Alignment and Quality Control.After collecting the web books, we align the bilingual books chapter by chapter according to the indices, while removing those chapters without parallel data. Then, we use Blucalign, which is an MT-based sentence alignment tool, to align the chapters into parallel sentences, while retaining the document-level information. We further deduplicate the parallel corpus and filter the pairs with a sequence ratio of 3.0. The scale of the final corpus is 384 books with 9,581,816 sentence pairs (a total of 460 million words). To estimate the accuracy of this process, we hired 4 bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting bwb parallel corpus. These students are native Chinese speakers who are proficient in English. More specifically, they were asked to distinguish whether a document is well aligned at the sentence level by counting the number of misalignment. For example, if Line 39 in English actually corresponds to Line 39 and Line 40 in Chinese, but the tool made a mistake that it combines the two sentences, it is identified as a misalignment. We observe an alignment accuracy rate of 93.1%.

We further asked the same batch of annotators to correct such misalignments in both the development and the test set. The annotation result shows that 7.3% lines are corrected.

## Appendix B Error Analysis and BlonD+ Annotation

Error analysis and BlonD+ annotation are conducted together. This task is conducted by eight professional Chinese-English translators who are native in Chinese and fluent in English.

The guideline is as follows:

* First, identify cases which have translation errors. The annotators are instructed to mark examples as "translations with no error" only if it satisfies the criteria of both adequacy and fluency as well as satisfies the criterion that it is coherent in the context.
* Second, identify whether the translation contains document-level error or sentence-level error (or both). The annotators are instructed to mark examples as "cases with sentence-level errors" when they are not adequate or fluent as stand-alone sentences; while "document-level errors" mean those errors that cause the example violating the global criterion of coherence.
* Third, categorize the examples with document-level errors according to the linguistic phenomena that lead to errors in MT outputs when considering context.

We first conduct a test annotation and observe that the annotators categorize document-level errors into mainly into 3 categories, namely inconsistency, ellipsis, and ambiguity. According to this observation, we instruct annotators to mark document-level errors as _inconsistency_, _ellipsis_, and _ambiguity_, or _other document-level error_ during the annotation process for the entire test set.

In the formal annotation process, we also added the requirement to annotate BlonD+ spans. The detailed requirement is as follows:

* Third, categorize the examples with document-level into 4 categories: inconsistency, ellipsis, and ambiguity, or other document-level error which cannot be categorized.
* Fourth, if the example is categorized as _ambiguity_, mark the specific word/phrase in the reference (English) that cause ambiguity and give the correct word/phrase.
* Fifth, if the example is categorized as _ellipisis_ and it is _not_ related to pronouns or discourse markers, mark the omitted word/phrase in the reference (English).

## Appendix C Human Post-Editing

This task is conducted by the same eight professional Chinese-English translators who carry out the annotation in Appendix B. We asked them to follow guidelines for achieving "good enough 

quality at the sentence-level (comprehensible, accurate but as not being stylistically compelling) but especially pay attention to document-level errors and correct them.

## Appendix D The Human Evaluation Protocol

The human evaluation is conducted on outputs of four systems (omt-b, mt-s, ctx, pe) and human translation. We follow the protocol proposed by Laubli et al. (2018, 2020). We conduct the evaluation experiment with a \(2\times 2\) mixed factorial design, carrying both Document-level and sentence-level evaluation in terms of adequacy and fluency. In the sentence-level evaluation, we show raters isolated sentences in random order; while in the document-level evaluation, entire documents are presented and we only ask raters to evaluate a sequence of 5 sequential sentences at a time in order.

To avoid reference bias, the adequacy evaluation is only based on source texts, while no source texts nor references are presented in the fluency evaluation. We adopt Relative Ranking (RR): raters are presented with outputs from the aforementioned five systems, which they are asked to evaluate relative to each other, e.g., to determine system A is better than system B (with ties allowed).

We use source sentences and documents from the bwb test set, but blind their origins by randomizing both the order in which the system outputs are presented. Note that in the document-level evaluation, the same ordering of systems is used within a document. The order of experimental items is also randomised. Sentences are randomly drawn from these documents, regardless of their position.

We also use spam items for quality control Kittur et al. (2008): In a small fraction of items, we render one of the five options nonsensical by randomly shuffling the order of all translated words, except for 10% at the beginning and end. If a rater marks a spam item as better than or equal to an actual translation, this is a strong indication that they did not read both options carefully. At the document-level, we render one of the five options nonsensical by randomly shuffling the order of all translated sentences, except for the first and the last sentence.

We recruit four professional Chinese to English translators and four native English revisiers for the adequacy and fluency conditions respectively. Note that the eight translators are different from those professional translators who carry out the human translation pe. We deliberately invite another group of specialists for human evaluation to avoid making unreasonable judgments biased towards pe. In each condition, each raters evaluate 162 documents (plus 18 spam items) and 162 sentences (plus 18 spam

\begin{table}
\begin{tabular}{c c l} \hline \hline
**CATEGORIES** & **DESCRIPTION** & **MARKERS** \\ \hline contingency & only consider “cause” & [“but", “while”, ”however”, “although”, “though”, “still”, “yet”, “whereas”, “on the other hand”, “in contrast”, “by contrast”, “very comparison”, “conversely”] \\ \hline comparison & combine “concession” and “contrast” & [“if”, “because”, “so”, “since”, “thus”, “hence”, “as a result”, “therefore”, “thereby”, “accordingly”, “consequently”, “in consequence”, “for this reason” \\ \hline expansion & only consider “ conjunction” & [“also”, “in addition”, “moreover”, “additionally”, “ besides”, “else, “plus”] \\ \hline temporal & “synchronous” & [“meantime”, “meanwhile”, “simultaneously”] \\  & “asynchronous” & [“when”, “after”, “then”, “before”, “until”, “later”, “once”, “afterward”, “next”] \\ \hline \hline \end{tabular}
\end{table}
Table 6: Explanations of the discourse marker types (discourse relations) in dm.

 

items). We use two non-overlapping sets of documents and two non-overlapping sets of sentences, and each is evaluated by two raters. Specifically, we refer the first half of the test set as part1 and the second half as part2. Note that part1 and part2 are chosen from different books. Each rater evaluates both sentences and documents, but never the same text in both conditions so as to avoid repetition priming (Gonzalez et al., 2011): rater1 and rater2 conduct the document-level adequacy evaluation on 180 documents sampled from part1 and the sentence-level adequacy evaluation for part2; rater3 and rater4 conduct the sentence-level fluency evaluation on 180 documents sampled from part1 and the document-level fluency evaluation for part2; rater5 and rater6 conduct the document-level fluency evaluation on 180 documents sampled from part1 and the sentence-level fluency evaluation for part2; rater7 and rater8 conduct the sentence-level fluency evaluation on 180 documents sampled from part1 and the document-level fluency evaluation for part2.

## Appendix E Statistical Analysis of Human Evaluation

We calculate Cohen's kappa coefficient:

\[\kappa=\frac{P(A)-P(E)}{1-P(E)} \tag{16}\]

where \(P(A)\) is the proportion of times that two raters agree, and \(P(E)\) is the likelihood of agreement by chance. We report pairwise inter-rater agreement in Table 8.

## Appendix F Experiment Settings

### Blonde

We use the named entity recognition module and the POS tagger of spaCy Honnibal and Montani (2017) to implement the categorizing function cat for entity and tense, respectively. We use the script provided by Sileo et al. (2019) as the discourse marker minor.

### Model Hyperparameters

We follow the setup of Transformer big model for bwb experiments. More precisely, the parameters in the big encoders and decoders are \(N=12\) , the number of heads per layer is \(h=16\), the dimensionality of input and output is \(d_{model}=1024\), and the inner-layer of a feed-forward networks has dimensionality \(d_{ff}=4096\). The dropout rate is fixed as 0.3. We adopt Adam optimizer with \(\beta_{1}=0.9,\beta_{2}=0.98,\epsilon=10^{-9}\), and set learning rate \(0.1\) of the same learning rate schedule as Transformer. We set the batch size as 6,000 and the update frequency as 16 for updating parameters to imitate 128 GPUs on a machine with 8 V100 GPU. The datasets are encoded by BPE with 60K merge operations.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & sentence & document \\ \hline rater1-rater2 & .171 & .169 \\ rater3-rater4 & .294 & .346 \\ rater5-rater6 & .323 & .402 \\ rater7-rater8 & .378 & .342 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Inter-rater agreements measure by Cohen’s \(\kappa\), where rater1-4 are professional translators whose native language is Chinese, rater5-8 are native English revisers.