SlowGuess/NAACL_2022_OCR huggingface dataset:
Najeh Hajlaoui and Andrei Popescu-Belis. 2013. Assessing the accuracy of discourse connective translations: Validation of an automatic metric. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 236-247. Springer.
Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In Proceedings of the 7th International Workshop on Spoken Language Translation: Papers, pages 283-289, Paris, France.
Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT shared task on pronoun translation. In Proceedings of the Second Workshop on Discourse in Machine Translation, pages 1-16, Lisbon, Portugal. Association for Computational Linguistics.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567.
Ulf Hermjakob, Kevin Knight, and Hal Daumé III. 2008. Name translation in statistical machine translation - learning when to transliterate. In Proceedings of ACL-08: HLT, pages 389-397, Columbus, Ohio. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486-2496, Copenhagen, Denmark. Association for Computational Linguistics.
Shafiq Joty, Francisco Guzmán, Lluis Márquez, and Preslav Nakov. 2014. DiscoTK: Using discourse structure for machine translation evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 402-408, Baltimore, Maryland, USA. Association for Computational Linguistics.
Prathyusha Jwalapuram, Shafiq Joty, Irina Temnikova, and Preslav Nakov. 2019. Evaluating pronominal anaphora in machine translation: An evaluation measure and a test suite. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2964-2975, Hong Kong, China. Association for Computational Linguistics.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. **Skip-thought vectors. In Advances in Neural Information Processing Systems, volume 28.**
Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 453-456.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human-machine parity in language translation. Journal of Artificial Intelligence Research, 67:653-672.
Samuel Läubli, Rico Senrich, and Martin Volk. 2018. Has machine translation achieved human parity? A case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Siyou Liu and Xiaojun Zhang. 2020. Corpora for document-level neural machine translation. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3775-3781, Marseille, France. European Language Resources Association.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the
2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688-725, Online. Association for Computational Linguistics.
Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (APT). In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 17-25, Copenhagen, Denmark. Association for Computational Linguistics.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 61-72, Brussels, Belgium. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yip Po-Ching and Don Rimmington. 2004. Chinese: A Comprehensive Grammar. Routledge.
Martin Popel. 2018. CUNI transformer neural MT system for WMT18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 482-487, Belgium, Brussels. Association for Computational Linguistics.
Annette Rios Gonzales, Laura Mascarell, and Rico Senrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11-19, Copenhagen, Denmark. Association for Computational Linguistics.
Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building
Educational Applications Using NLP, pages 157-162, Montreal, Canada. Association for Computational Linguistics.
Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 175-182, Riga, Latvia. Northern European Association for Language Technology (NEALT).
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3477-3486, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
Natasha Tokowicz and Tamar Degani. 2010. Translation ambiguity: Consequences for learning and processing. Research on Second Language Processing and Parsing, pages 281-293.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 4566-4575.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198-1212, Florence, Italy. Association for Computational Linguistics.
Evan James Williams. 1959. Regression Analysis, volume 14. Wiley.
Billy T. M. Wong and Chunyu Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1060-1068, Jeju Island, Korea. Association for Computational Linguistics.
Deyi Xiong and Min Zhang. 2014. Semantics, discourse and statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Tutorials, pages 11-12, Baltimore, Maryland, USA. Association for Computational Linguistics.
Kazuhide Yamamoto and Eiichiro Sumita. 1998. Feasibility study for ellipsis resolution in dialogues by machine-learning technique. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, pages 1428-1435, Montreal, Quebec, Canada. Association for Computational Linguistics.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533-542, Brussels, Belgium. Association for Computational Linguistics.
Ming Zhou, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 1121-1128, Manchester, UK.
A Dataset Creation
The Background of Translators. The original Chinese books are translated by professional native English speakers, and are corrected by editors.
Data Collection. This process is implemented by a python web crawler, and certain data cleaning is also done in the process. We crawl the books chapter by chapter, and convert the text to UTF-8. After dedduplication, we remove the chapters with less than 5 sentences. We further remove the titles of each chapter, because most of them are neither translated properly nor in the document-level.
Alignment and Quality Control. After collecting the web books, we align the bilingual books chapter by chapter according to the indices, while removing those chapters without parallel data. Then, we use Bluealign, which is an MT-based sentence alignment tool, to align the chapters into parallel sentences, while retaining the document-level information. We further deduplicate the parallel corpus and filter the pairs with a sequence ratio of 3.0. The scale of the final corpus is 384 books with 9,581,816 sentence pairs (a total of 460 million words). To estimate the accuracy of this process, we hired 4 bilingual graduate students to manually evaluate 163 randomly selected documents from the resulting BwB parallel corpus. These students are native Chinese speakers who are proficient in English. More specifically, they were asked to distinguish whether a document is well aligned at the sentence level by counting the number of misalignment. For example, if Line 39 in English actually corresponds to Line 39 and Line 40 in Chinese, but the tool made a mistake that it combines the two sentences, it is identified as a misalignment. We observe an alignment accuracy rate of  $93.1\%$ .
We further asked the same batch of annotators to correct such misalignments in both the development and the test set. The annotation result shows that  $7.3\%$  lines are corrected.
B Error Analysis and BLOND+ Annotation
Error analysis and BLOND+ annotation are conducted together. This task is conducted by eight professional Chinese-English translators who are native in Chinese and fluent in English.
The guideline is as follows:
- First, identify cases which have translation
errors. The annotators are instructed to mark examples as "translations with no error" only if it satisfies the criteria of both adequacy and fluency as well as satisfies the criterion that it is coherent in the context.
- Second, identify whether the translation contains document-level error or sentence-level error (or both). The annotators are instructed to mark examples as "cases with sentence-level errors" when they are not adequate or fluent as stand-alone sentences; while "document-level errors" mean those errors that cause the example violating the global criterion of coherence.
- Third, categorize the examples with document-level errors according to the linguistic phenomena that lead to errors in MT outputs when considering context.
We first conduct a test annotation and observe that the annotators categorize document-level errors into mainly into 3 categories, namely inconsistency, ellipsis, and ambiguity. According to this observation, we instruct annotators to mark document-level errors as inconsistency, ellipsis, and ambiguity, or other document-level error during the annotation process for the entire test set.
In the formal annotation process, we also added the requirement to annotate BLOND+ spans. The detailed requirement is as follows:
- Third, categorize the examples with document-level into 4 categories: inconsistency, ellipsis, and ambiguity, or other document-level error which cannot be categorized.
- Fourth, if the example is categorized as ambiguity, mark the specific word/phrase in the reference (English) that cause ambiguity and give the correct word/phrase.
- Fifth, if the example is categorized as ellipsis and it is not related to pronouns or discourse markers, mark the omitted word/phrase in the reference (English).
C Human Post-Editing
This task is conducted by the same eight professional Chinese-English translators who carry out the annotation in Appendix B. We asked them to follow guidelines for achieving "good enough"
CATEGORIES	DESCRIPTION	MARKERS
contingency	only consider “cause”	[“but”, “while”, “however”, “although”, “though”, “still”, “yet”, “whereas”, “on the other hand”, “in contrast”, “by contrast”, “by comparison”, “conversely”]
comparison	combine “concession” and “contrast”	[“if”, “because”, “so”, “since”, “thus”, “hence”, “as a result”, “therefore”, “thereby”, “accordingly”, “consequently”, “in consequence”, “for this reason”]
expansion	only consider “conjunction”	[“also”, “in addition”, “moreover”, “additionally”, “besides”, “else”, “plus”]
temporal	“synchronous”
“asynchronous”	[“meantime”, “meanwhile”, “simultaneously”]
[“when”, “after”, “then”, “before”, “until”, “later”, “once”, “afterward”, “next”]
Table 6: Explanations of the discourse marker types (discourse relations) in DM.
Dataset	Domain	#Docs	#Sents
WMT (Barrault et al., 2019)	News	68.4k	3.63M
OpenSubtitles (Lison et al., 2018)	Subtitles	29.1k	31.2k
TED (Ansari et al., 2020)	TALKs	1K	219M
BWB	Books	196k	9M
Table 7: Comparison of different document-level datasets.
quality at the sentence-level (comprehensible, accurate but as not being stylistically compelling) but especially pay attention to document-level errors and correct them.
D The Human Evaluation Protocol
The human evaluation is conducted on outputs of four systems (OMT-B, MT-S, CTX, PE) and human translation. We follow the protocol proposed by (Läubli et al., 2018, 2020). We conduct the evaluation experiment with a  $2 \times 2$  mixed factorial design, carrying both DOCUMENT-level and SENTENCE-level evaluation in terms of ADEQUACY and FLUENCY. In the SENTENCE-level evaluation, we show raters isolated sentences in random order; while in the DOCUMENT-level evaluation, entire documents are presented and we only ask raters to evaluate a sequence of 5 sequential sentences at a time in order.
To avoid reference bias, the ADEQUACY evaluation is only based on source texts, while no source texts nor references are presented in the FLUENCY evaluation. We adopt Relative Ranking (RR): raters are presented with outputs from the aforementioned five systems, which they are asked to evaluate relative to each other, e.g., to determine system A is better than system B (with ties allowed).
We use source sentences and documents from
the BwB test set, but blind their origins by randomizing both the order in which the system outputs are presented. Note that in the DOCUMENT-level evaluation, the same ordering of systems is used within a document. The order of experimental items is also randomised. Sentences are randomly drawn from these documents, regardless of their position.
We also use spam items for quality control (Kittur et al., 2008): In a small fraction of items, we render one of the five options nonsensical by randomly shuffling the order of all translated words, except for  $10\%$  at the beginning and end. If a rater marks a spam item as better than or equal to an actual translation, this is a strong indication that they did not read both options carefully. At the DOCUMENT-level, we render one of the five options nonsensical by randomly shuffling the order of all translated sentences, except for the first and the last sentence.
We recruit four professional Chinese to English translators and four native English revisers for the adequacy and fluency conditions respectively. Note that the eight translators are different from those professional translators who carry out the human translation PE. We deliberately invite another group of specialists for human evaluation to avoid making unreasonable judgments biased towards PE. In each condition, each raters evaluate 162 documents (plus 18 spam items) and 162 sentences (plus 18 spam
SENTENCE	DOCUMENT
RATER1-RATER2	.171	.169
RATER3-RATER4	.294	.346
RATER5-RATER6	.323	.402
RATER7-RATER8	.378	.342
Table 8: Inter-rater agreements measure by Cohen's  $\kappa$ , where RATER1-4 are professional translators whose native language is Chinese, RATER5-8 are native English revisers.
items). We use two non-overlapping sets of documents and two non-overlapping sets of sentences, and each is evaluated by two raters. Specifically, we refer the first half of the test set as PART1 and the second half as PART2. Note that PART1 and PART2 are chosen from different books. Each rater evaluates both sentences and documents, but never the same text in both conditions so as to avoid repetition priming (Gonzalez et al., 2011): RATER1 and RATER2 conduct the DOCUMENT-level ADEQUACY evaluation on 180 documents sampled from PART1 and the SENTENCE-level ADEQUACY evaluation for PART2; RATER3 and RATER4 conduct the SENTENCE-level FLUENCY evaluation on 180 documents sampled from PART1 and the DOCUMENT-level FLUENCY evaluation for PART2; RATER5 and RATER6 conduct the DOCUMENT-level FLUENCY evaluation on 180 documents sampled from PART1 and the SENTENCE-level FLUENCY evaluation for PART2; RATER7 and RATER8 conduct the SENTENCE-level FLUENCY evaluation on 180 documents sampled from PART1 and the DOCUMENT-level FLUENCY evaluation for PART2.
E Statistical Analysis of Human Evaluation
We calculate Cohen's kappa coefficient:
$$
\kappa = \frac {P (A) - P (E)}{1 - P (E)} \tag {16}
$$
where  $P(A)$  is the proportion of times that two raters agree, and  $P(E)$  is the likelihood of agreement by chance. We report pairwise inter-rater agreement in Table 8.
F Experiment Settings
F.1 BLONDE
We use the named entity recognition module and the POS tagger of spaCy (Honnibal and Montani, 2017) to implement the categorizing function cat
for ENTITY and TENSE, respectively. We use the script provided by Sileo et al. (2019) as the discourse marker minor.
F.2 Model Hyperparameters
We follow the setup of Transformer big model for BwB experiments. More precisely, the parameters in the big encoders and decoders are  $N = 12$ , the number of heads per layer is  $h = 16$ , the dimensionality of input and output is  $d_{model} = 1024$  and the inner-layer of a feed-forward networks has dimensionality  $d_{ff} = 4096$ . The dropout rate is fixed as 0.3. We adopt Adam optimizer with  $\beta_1 = 0.9, \beta_2 = 0.98, \epsilon = 10^{-9}$ , and set learning rate 0.1 of the same learning rate schedule as Transformer. We set the batch size as 6,000 and the update frequency as 16 for updating parameters to imitate 128 GPUs on a machine with 8 V100 GPU. The datasets are encoded by BPE with 60K merge operations.