SlowGuess/NAACL_2022_OCR huggingface dataset:
Here,  $p(\mathfrak{D}^s,\mathbb{D}^r)$ ,  $r(\mathfrak{D}^s,\mathbb{D}^r)$  and  $\boldsymbol {F}(\mathfrak{D}^s,\mathbb{D}^r)$  are all  $K$  -dimensional vectors; where, the  $k^{\mathrm{th}}$  element of these vectors represents the precision, recall and F-score for the  $k^{\mathrm{th}}$  category. Thus, the addition, multiplication and division operations above are also defined element-wise. $^{13}$
BLOND-D. Further, we combine the scores of all categories into an overall score with a simple weighted average, named BLOND-D. By computing BLOND-D, one can distill the document-level translation quality from translation quality at the sentence level. More formally, we have
$$
\operatorname {B L O N D} - \mathrm {D}. \mathrm {P} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = \tag {11}
$$
$$
\left(\prod_ {k = 1} ^ {K} \left(\boldsymbol {p} _ {k} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) ^ {a _ {k}}\right) ^ {1 / \sum_ {k = 1} ^ {K} a _ {k}} \right.
$$
$$
\operatorname {B L O N D} - \mathrm {D}. \operatorname {R} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) = \tag {12}
$$
$$
\left(\prod_ {k = 1} ^ {K} \left(\boldsymbol {r} _ {k} \left(\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}\right) ^ {a _ {k}}\right) ^ {1 / \sum_ {k = 1} ^ {K} a _ {k}} \right.
$$
where  $a_{k}$  denotes the importance weight of the  $k^{\mathrm{th}}$  category, and  $\pmb{p}_k$  and  $\pmb{r}_k$  denote the  $k^{\mathrm{th}}$  entry of  $\pmb{p}$  and  $\pmb{r}$ , respectively. Therefore, BLOND-D.F1 is defined as follows:
$$
\operatorname {B L O N D} - \mathrm {D}. \mathrm {F} 1 (\mathfrak {D} ^ {s}, \mathbb {D} ^ {r}) = \tag {13}
$$
$$
2 \cdot \frac {\mathrm {B L O N D - D . P} \cdot \mathrm {B L O N D - D . R}}{\mathrm {B L O N D - D . P} + \mathrm {B L O N D - D . R}}
$$
$^{13}$ In Figure 3, recall that  $\mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^r) = (1,2,4,0)$ ,  $\mathbf{sim}(\mathfrak{D}^{\mathrm{MTA}},\mathfrak{D}^{\mathrm{MTA}}) = (1,7,6,0)$  and  $\mathbf{sim}(\mathbb{D}^r,\mathbb{D}^r) = (2,7,5,2)$ . Thus, we have  $\pmb {p}(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^r) = \left(\frac{1}{1},\frac{2}{7},\frac{4}{6},\frac{0}{0}\right) =$ $(1,\frac{2}{7},\frac{2}{3},\mathrm{NA})$  where NA denotes a missing value. Furthermore, we have  $\pmb {r}(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^r) = \left(\frac{1}{2},\frac{2}{7},\frac{4}{5},\frac{0}{2}\right) = \left(\frac{1}{2},\frac{2}{7},\frac{4}{5},\frac{\delta}{2}\right)$  where  $\delta$  denotes a small value (0.0001) for smoothing. Finally, we have  $\pmb {F}(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^r) = \left(\frac{2}{3},\frac{2}{7},\frac{8}{11},\mathrm{NA}\right)$ .
BLEU P	BLONDE	BLOND-D F1
P	R	F1
MTA	41.5	10.5	51.3	17.4	7.6
MTB	35.9	60.6	58.9	59.8	97.7
Table 3: The BLEU and BLONDE scores of the two system outputs in Figure 3. P, R and F1 represent precision, recall and F1, respectively.
Whenever not otherwise specified, we simply use BLOND-D to refer to BLOND-D.F1. $^{15}$
4.2 BLONDE: Combining BLOND-D with  $n$ -grams
However, focusing on discourse phenomena solely is not enough to provide comprehensive MT evaluation that correlates strongly with human judgments. Consider the following example:
(1) REF Qiao lifted her heavy eyelids. MT Qiao scrunched her brows together.
The output of MT is far from "good" in terms of adequacy, whereas BLOND-D(MT) = 1, since MT translates both named entities and tenses correctly. Thus, in order to account for sentence-level adequacy of our final metric BLONDE, we augment the set of categories and features to include each  $n$ -gram (for a value of  $n$ ) as a category and each span of  $n$ -tokens as a feature for the  $n$ -gram category. Formally, we have
$$
\mathbf {C} ^ {\prime} (S) = \tag {14}
$$
$$
[ \mathbf {C} _ {k} (S): k \in \{1, \dots , K + N \} ]
$$
$$
\mathbf {C} _ {K + n} = \{n \text {- g r a m}: n \in \{1, \dots , N \} \} \tag {15}
$$
The calculation of BLONDE.P, BLONDE.R and BLONDE.F1 is then done exactly in the same manner as BLOND-D. Whenever not specified, we simply use BLONDE to refer to BLONDE.F1.
$^{15}$ E.g., the BLOND-D scores of MTA in Figure 3 are:
BLOND-D.P  $(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^{r}) = \left(\frac{1}{1}\right)^{\frac{1}{3}}\left(\frac{2}{7}\right)^{\frac{1}{3}}\left(\frac{2}{3}\right)^{\frac{1}{3}}$
$= .114^{16}$
BLOND-D.R  $(\mathfrak{D}^{\mathrm{MTA}},\mathbb{D}^{r}) = \left(\frac{1}{2}\right)^{\frac{1}{4}}\left(\frac{2}{7}\right)^{\frac{1}{4}}\left(\frac{4}{5}\right)^{\frac{1}{4}}\left(\frac{\delta}{2}\right)^{\frac{1}{4}}$
$= .057$
BLOND-D.F1(  $\mathfrak{Q}^{\mathrm{MTA}},\mathbb{D}^r) = \frac{2\cdot.057\cdot.114}{.057 + .114}$
$= .076$
BLONDE covers both discourse coherence features and sentence-level adequacy, thus providing a comprehensive measurement of translation quality. Table 3 compares BLONDE with BLEU using the two MT outputs found in Figure 3. It is striking that BLEU rates MTA higher than MTB given that MTB is clearly better than MTA to human readers. In sharp contrast, their BLONDE scores reflect the correct ranking in translation quality.
4.3 BLOND+: Combining BLONDE with Human Annotations
BLONDE is easy to generalize—for instance, it would be easy to incorporate human annotations, e.g., one could annotate spans related to discourse errors and treat them as categories. The automatically inferred categories and human annotated categories are then combined by adopting the same weighted averaging approach, which we call BLOND+. We hired the same translators who analyzed discourse errors in Section 3 to annotate ambiguous and omitted word/phrases on the test set of BWB. $^{17}$
In this section, we examine the effectiveness of BLONDE at the document-level MT evaluation through experiments. We answer the following question: Do differences in BLONDE reliably reflect differences in the document-level translation quality of different MT systems? To answer this question, we run several MT baselines and compare their BLONDE scores to eleven other metrics:
Standard Sentence-level Metrics. BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006), ROUGE-L (Lin, 2004), CIDER (Vedantam et al., 2015).
Document-level Metrics. LC and RC (Wong and Kit, 2012)—these are ratios between the number of lexical cohesion devices (repetition and collocation) and repeated content words over the total number of content words in a target document, which are direct measurements of lexical cohesion.
Embedding-based Metrics. We consider four embedding-based metrics in this work: SkipThought cosine similarity (SKIP; Kiros et al.,
2015), embedding average cosine similarity (AVER; Sharma et al., 2017), Vector extrema cosine similarity (VECTOR; Forgues et al., 2014), Greedy Match (GREEDY; Rus and Lintean, 2012).
5.1 MT Systems
We test BLONDE on the following system outputs: an SMT system (Chiang, 2007), three well-known commercial NMT systems (OMT-A, OMT-B, OMT-C), a sentence-level transformer-based system (MT-S) and a document-level system (MT-D) trained on BWB. MT-D (Zhang et al., 2018) trains sentence-level model parameters and then estimates document-level model parameters while keeping the sentence-level Transformer model parameters fixed. We adopt Transformer Big (Vaswani et al., 2017) for both MT-S and MT-D. The final "system" is a human post-editing (PE) on OMT-C provided by professional translators, so it is supposed to be the strongest baseline.[18]
5.2 The BLONDE Evaluation
Firstly, we leverage the test set of BwB and evaluate the above-mentioned systems by BLONDE and other metrics. Figure 4 presents the means of all metrics along with the  $95\%$  confidence interval estimated from bootstrap resampling. We observe that the BLONDE scores demonstrate an exponentially increasing trend from sentence-level towards document-level and human post-editing, while the trends of standard metrics are mostly linear. Specifically, the difference between the BLONDE scores of MT-S and MT-D (denoted as  $\Delta (\mathrm{MT - S},\mathrm{MT - D})$ ) is significantly higher than the difference between the  $\Delta (\mathrm{MT - S},\mathrm{MT - D})$  in their BLEU scores. An even larger  $\Delta$  between MT-D and PE in their BLONDE scores is observed, indicating MT-D is still far away from achieving human parity. Note that the trend of BLOND-D scores is even more exponential, which indicates that BLOND-D indeed distills document-level translation quality.
The  $t$ -statistics of the paired sample  $t$ -tests of individual documents are given in Table 4. Unlike BLEU, METEOR and other metrics, which either fails to distinguish human and machine translation or has lower discriminative power compared to distinguishing different machine translations, the BLONDE family maintain similar discriminative
<sup>17</sup>We also make this annotated test set publicly available as a testbed for evaluating the ability of MT systems to disambiguate word senses and to predict coherent pronouns or discourse markers in the case of omission.
18We trained models by fairseq (Ott et al., 2019). Model parameters and the post-editing details are in Appendix F.2 and C, respectively.
Figure 4: The mean scores of different system outputs given by different metrics on the BwB test set. Shaded region represents  $95\%$  confidence interval.
BLEU	BLONDE	BLOND+	BLOND-D	Categories
R	P	F1	R	P	F1	R	P	F1	E	V	P	M
SMT→ MT-S	25.8	13.5	7.42	10.9	14.5	8.51	12.0	8.02	1.32	5.10	-2.12	23.6	11.4	13.6
MT-S→ MT-D	8.97	6.32	5.45	5.92	6.58	5.61	6.13	4.85	4.57	4.79	4.93	1.88	7.43	1.62
MT-D→ PE	2.6	4.51	7.77	6.06	4.20	7.27	5.66	6.44	11.1	8.58	12.9	2.44	2.76	5.35
Other Standard Metrics	Discourse Cohesion	Embedding-based Metrics
METEOR	ROUGE-L	TER	CIDER	LC	RC	SKIP	AVER	VECTOR	GREEDY
SMT→ MT-S	25.3	19.8	-8.28	.853	11.7	12.9	12.2	9.50	18.0	22.3
MT-S→ MT-D	13.4	11.8	.148	-3.03	-1.23	-1.45	1.62	3.13	5.05	5.83
MT-D→ PE	3.58	9.65	19.9	-6.67	-4.23	-4.44	-6.23	.628	-1.03	-3.15
Table 4: The paired  $t$ -statistics of different MT systems. The cells with  $p$ -value  $> .05$  are marked in gray. While BLEU distinguishes SMT and the sentence-level MT-S significantly, it fails to possess the same discriminative power towards document-level and human translations. BLONDE maintains similar discriminative power across the three  $t$ -tests.
Figure 5: Absolute Pearson correlation pairs of automatic metrics. Computed over the scores of individual documents in BwB test set.
power across the pair-wise comparisons. Interestingly but not surprisingly, the non-reference-based LC and RC fail to distinguish both (MT-S, MT-D) and (MT-D, PE), since sentence-level MT is by nature more repetitive than human translation and thus it is hard to distinguish accidental repetition
from document-level cohesion.
In addition, the  $t$ -statistics of BLOND-D categories provide rich diagnostic information. As can be seen, although transformer-based NMT models have substantially higher BLEU scores than SMT systems, MT-S is not statistically superior to SMT in terms of named entity translation. However, human post-editing scores significantly better on entity translation—meaning that named entity translation accounts for a substantial part of quality differences between machine and human. In terms of TENSE and and DM translation, MT-D does not significantly out-perform MT-S, which could be taken into consideration in future document-level MT model designs.
We also show the pairwise Pearson correlations between different metrics in Figure 5. It illustrates the homogeneity/heterogeneity of different metrics. We report the absolute value of the correlation for TER. We see that while sentence-level metrics (BLEU, METEOR and ROUGE-L) have strong correlations with each other, BLONDE correlates less well with those metrics.
Unit	SENTENCE	DOCUMENT
ADE	FLU	ADE	FLU
BLONDE.R	.363	.327	.436†	.371†
BLONDE.P	.331	.296	.383†	.344†
BLONDE.F1	.35	.314	.417†	.358†
BLOND+.R	.364	.329	.44†	.373†
BLOND+.P	.334	.3	.39†	.349†
BLOND+.F1	.351	.318	.422†	.362†
BLEU	.325	.308	.343	.266
METEOR	.338	.31	.339	.278
ROUGE-L	.275	.262	.29	.211
TER	.063	.027	.044	.092
CIDER	.139	.116	.114	.087
SKIP	.213	.174	.163	.171
VER	.163	.163	.16	.111
VECTOR	.25	.243	.248	.218
GREEDY	.323	.3	.307	.265
LC	.086	.061	.153	.116
RC	.096	.07	.169	.13
Table 5: Absolute Pearson correlation with human judgments on BwB. The highest correlations are in bold. Correlation of metrics not significantly outperformed by any other metrics are highlighted with  $\dagger$ . The BLONDE family are not tested against each other.
We then evaluate BLONDE along with other metrics in terms of their Pearson correlation with human assessment. Our human assessment is provided by four professional Chinese to English translators and four native English revisers. Two experimental units (SENTENCE vs DOCUMENT) are assessed independently in terms of FLUENCY and ADEQUACY, respectively. In the SENTENCE-level evaluation, we show the raters isolated sentences, while in the DOCUMENT-level evaluation, we show them entire documents and we only ask raters to evaluate the overall quality of sequential blocks of sentences (5 sentences per block) as used in the Relative Ranking (RR) evaluation (Bojar et al., 2016). We use the Williams significance test (Williams, 1959; Graham and Baldwin, 2014) following the practice adopted by WMT (Mathur et al., 2020) to identify correlation differences are statistically significant. The detailed protocol is presented in Appendix D.
The results are shown in Table 5. BLONDE obtains the highest correlation with human assessment at both the sentence level and the document level. However, BLONDE correlates remarkably better with human assessment when context is taken into account, and it only significantly outperforms all other metrics at document level.
It is worth noting that BLONDE also correlates
well with FLUENCY assessment, even though it is, in essence, still a reference-based metric. One possible explanation for this unexpected positive result is that it tracks span categories that directly relate to cohesion and coherence. Another important observation is that the recall-based BLONDE variants generally correlate better with human assessment, yet appears to be less selective compared to the precision-based variants (see MT-D  $\rightarrow$  PE in Table 4). This provides support for adopting the F1 in order to get the best of both worlds.
There have been a few studies on automatic evaluation metrics for specific discourse phenomena.
Pronoun Translation. Hardmeier and Federico (2010) measured the precision and recall of pronouns directly and Miculicich Werlen and Popescu-Belis (2017) proposed to estimate the accuracy of pronoun translation (APT) by aligning source and target texts. However, as shown in Guillou and Hardmeier (2018), APT does not take the antecedents of an anaphoric pronoun into account. They cannot handle the mismatches in the numbers of pronouns either. Jwalapuram et al. (2019) also proposed a specialized measure for pronoun evaluation which involves training. In comparison, BLONDE does not rely on any alignment or training.
Lexical Cohesion. Wong and Kit (2012) proposed LC and RC. Gong et al. (2015) described a cohesion function to measure text cohesion via lexical chain and a gist consistency score based on topic model. However, they fail to distinguish accidental repetition from document-level cohesion.
Discourse Relations. Hajlaoui and Popescu-Belis (2013) proposed to assess the accuracy of connective translation (ACT). However, such an assessment requires a bilingual dictionary of all possible DM translations, whereas BLONDE only requires a list of monolingual DMs. Guzmán et al. (2014) and Joty et al. (2014) compute a metric based on the similarity between the discourse trees of reference and system output. Those discourse-representation-based metrics are indirect, and rely on discourse parsing tools, which are much more inaccurate than syntactic and semantic parsing tools used in BLONDE. Unlike previously proposed metrics, BLONDE does not only focus on one specific discourse phenomenon, and thus has significantly
higher Pearson correlation coefficients with human assessments.
In this paper, we build a large-scale parallel dataset for document-level translation, BwB. We analyze it for common document-level translation errors in practice and propose BLONDE, an interpretable automatic metric for document-level MT evaluation. We further improve BLONDE by diagnosing and distilling discourse-related errors in MT outputs and human-annotations to obtain two improved metrics BLOND-D and BLOND+. These metrics were shown to have better selectivity than various sentence-level metrics and correlate better with human judgments.
The annotators were paid a fair wage and the annotation process did not solicit any sensitive information from the annotators. Finally, while our approach is not tuned for any specific real-world application, the approach could be used in sensitive contexts such as legal or health-care settings, and any work must use our approach undertake extensive quality-assurance and robustness testing before using it in their setting.
Replicability. As part of our contributions, we will release the annotated BWB test set, and release the crawling script of the training set under Fair Use rules. The BLONDE package is also publicly available at https://github.com/EleanorJiang/BlonDe.
Mrinmaya Sachan acknowledges support from an ETH Zürich Research grant (ETH-19 21-1) and a grant from the Swiss National Science Foundation (project #201009) for partially supporting this work.
Neeraj Agrawal and Ankush Singla. 2012. Using named entity recognition to improve machine translation. Technical report, Stanford University.
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondrej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco
Turchi, Alexander Waibel, and Changhan Wang. 2020. Findings of the IWSLT 2020 evaluation campaign. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 1-34, Online. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Rachel Bawden, Rico Senrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1304-1313, New Orleans, Louisiana. Association for Computational Linguistics.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
Ondrej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojevic. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 199-231, Berlin, Germany. Association for Computational Linguistics.
Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Peter Jan-Thorsten, and Philip Williams. 2017. A linguistic evaluation of rule-based, phrase-based, and neural MT engines. The Prague Bulletin of Mathematical Linguistics, 108(1):159.
Marine Carpuat. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009), pages 19-27, Boulder, Colorado. Association for Computational Linguistics.
Marine Carpuat and Michel Simard. 2012. The trouble with SMT consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442-449, Montreal, Canada. Association for Computational Linguistics.
David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201-228.
Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and RÉal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Workshop on Modern Machine Learning and Natural Language Processing, volume 2.
Zhengxian Gong, Min Zhang, and Guodong Zhou. 2015. Document-level machine translation evaluation with gist consistency and text cohesion. In Proceedings of the Second Workshop on Discourse in Machine Translation, pages 33-40, Lisbon, Portugal. Association for Computational Linguistics.
Cleotilde Gonzalez, Brad Best, Alice F. Healy, James A. Kole, and Lyle E. Bourne Jr. 2011. A cognitive modeling account of simultaneous learning and fatigue effects. Cognitive Systems Research, 12(1):19-32.
Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172-176, Doha, Qatar. Association for Computational Linguistics.
Liane Guillou. 2013. Analysing lexical consistency in translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 10-18, Sofia, Bulgaria. Association for Computational Linguistics.
Liane Guillou and Christian Hardmeier. 2016. PROTEST: A test suite for evaluating pronouns in machine translation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 636-643, Porto Roz, Slovenia. European Language Resources Association (ELRA).
Liane Guillou and Christian Hardmeier. 2018. Automatic reference-based evaluation of pronoun translation misses the point. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4797-4802, Brussels, Belgium. Association for Computational Linguistics.
Francisco Guzmán, Shafiq Joty, Lluis Márquez, and Preslav Nakov. 2014. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687-698, Baltimore, Maryland. Association for Computational Linguistics.