SlowGuess/ACL_2020_OCR huggingface dataset:
translation benefit from larger context? CoRR, abs/1704.05135.
Prathyusha Jwalapuram, Shafiq Joty, Irina Temnikova, and Preslav Nakov. 2019. Evaluating pronominal anaphora in machine translation: An evaluation measure and a test suite. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2957-2966, Hong Kong, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Conference Proceedings: the 10th Machine Translation Summit, pages 79-86. AAMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 177-180. Association for Computational Linguistics.
Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 252-261, Uppsala, Sweden. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from Movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, pages 923-929, Portož, Slovenia. European Language Resources Association.
Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275-1284, Melbourne, Australia. Association for Computational Linguistics.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2018. Contextual neural model for translating bilingual multi-speaker conversations. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 101-112, Brussels, Belgium. Association for Computational Linguistics.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers), pages 3092-3102, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954.
Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (APT). In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 17-25, Copenhagen, Denmark. Association for Computational Linguistics.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311-318. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the  $54^{th}$  Annual Meeting of the Association for Computational Linguistics, pages 1715-1725.
Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82-92, Copenhagen, Denmark. Association for Computational Linguistics.
5976
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003-3008, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264-1274, Melbourne, Australia. Association for Computational Linguistics.
Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826-2831, Copenhagen, Denmark. Association for Computational Linguistics.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533-542, Brussels, Belgium. Association for Computational Linguistics.
A.1 Model Configuration
We use similar configuration as the Transformer-base model (Vaswani et al., 2017) except that we reduce the number of layers in the encoder and decoder stack to 4 following Maruf et al. (2019). For training, we use the default Adam optimiser (Kingma and Ba, 2015) with an initial learning rate of 0.0001 and employ early stopping.
A.2 English  $\rightarrow$  Russian Experiments
We wanted to compare the two variants of Transformer-HAN with  $k = +1$  and  $k = -1$  in the same experimental setting as done by Voita et al. (2018). The data they made available only contains the previous context sentence. Thus, we extract training, development and test sets following the procedure in this work but of roughly the same size as Voita et al. (2018) for a fair comparison of the two context settings. While they extract their test set as a random sample of sentences, we extract
(a) Europarl
(b) TED Talks
Figure 3: Plots showing proportion of intersentential English pronouns versus size of coreference resolution window for Europarl and TED Talks corpora.
from a random sample of documents, resulting in a test set which has document-level continuity between sentences. The pre-processing pipeline is the same as the one used for our English-German and English-Portuguese experiments except that we perform lowercasing (instead of truecasing) and learn separate BPE codes for source and target languages following Voita et al. (2018). We also evaluate the models trained with our training set on the test set provided by Voita et al. (2018) after removing the sentences overlapping with our train and dev sets (corpora statistics are in Table 7).
Results Table 8 indicates that the model trained with the next sentence as context not only statistically significantly outperforms the Transformer baseline (+0.9 BLEU) but also demonstrates comparable performance to the HAN model trained
Origin	#Sentences	Document length
Voita et al. (2018)	2M/10K/10K	-
Ours	2M/11K/10K	606.3/620.6/631.6
Our, Voita et al. (2018)*	2M/11K/7.3K	606.3/620.6/-
Table 7: Train/dev/test statistics for English-Russian: number of sentences (K: thousands, M: millions), and average document length (in sentences). The first row mentions statistics of data used by Voita et al. (2018), the second row mentions statistics of data we extracted, and the third row mentions the data statistics for our train/dev sets and Voita et al. (2018)'s test after removing overlap (referred as Voita et al.  $(2018)^{\star}$
5977
Data Setting	Baseline	HAN(k = +1)	HAN(k = -1)
Ours	23.35	24.25	24.18
Our, Voita et al. (2018)★	27.15	-	28.23
with the previous sentence. This finding is consistent with our main results. We also evaluate the model trained with our training data on Voita et al. (2018) test set and report almost four points jump in the absolute BLEU score for both the baseline and the context-dependent model. In addition, we note that for their test set, the HAN  $(k = -1)$  has greater percentage improvement over the baseline  $(4\%)$  than what they report for their context-aware model  $(2.3\%)$ .
A.3 Cataphora-Focused Test Suite
We segment the cataphora test set into the following subsets based on the part-of-speech of the postcedent being referred to:
- DET Postcedents prefixed with possessive determiners, e.g., his son or their child.
PROPN Postcedents which are proper nouns, i.e., named entities.
- NOUN Postcedents which are nouns, including proper nouns and common nouns, such as boy or child.
A.3.1 Results for HAN  $(k = -1)$
We evaluate Transformer-HAN  $(k = -1)$  enriched with anaphoric context on the cataphora test set (Table 9) to determine if this context-aware model is making use of coreference information to improve the overall translation quality (in BLEU). We find that HAN  $(k = +1)$  performs better than HAN  $(k = -1)$  when English is in the target-side, which we hypothesise to be because of the extraction of the cataphora test suite from the English-side. However, when English is in the source-side, both models perform comparably showing that the Tranformer-HAN (a context-aware NMT model) is able to make better use of coreference information to improve translation of pronouns.
Table 8: BLEU on tokenised lowercased text for the Transformer baseline and Transformer-HAN with following sentence  $\left( {k =  + 1}\right)$  and previous sentence  $\left( {k = }\right.$  -1) for English  $\rightarrow$  Russian. All reported results for the HAN variants are statistically significantly better than the baseline.
Lang. Pair	Baseline	HAN(k = -1)
English→German	32.33	32.94
German→English	36.91	37.23
English→Portuguese	36.29	37.24
Portuguese→English	40.74	41.25
Table 9: BLEU on the cataphora test set for the Transformer and Transformer-HAN  $(k = -1)$ . All results for HAN  $(k = -1)$  are statistically significantly better than the baseline.
A.4 Pronoun Sets
Language | Pronouns
English	i, me, my, mine, myself, we, us, our, ours, ourselves, you, your, yours, yourself, yourselves, he, his, him, himself, she, her, hers, herself, it, its, itself, they, them, their, themselves, that, this, these, those, what, whatever, which, whichever, who, whoever, whom, whose
German	ich, du, er, sie, es, wir, mich, dich, sich, ihn, uns, euch, mir, dir, ihm, ihr, ihre, ihrer, ihren, meiner, mein, myeine, deiner, dein, seiner, sein, seine, unser, unsere, euer, euere, denen, dessen, deren, meinen, meinem, deinen, deinem, deines, unserer, unseren, unseres, unserem, ihrem, ihres, seinen, seinem, seines
Portuguese	eu, nós, tu, você, vocês, ele, ela, eles, esas, me, te, nos, vos, o, lo, no, a, la, na, lhe, se, os, los, as, las, nas, lhes, mim, ti, si, meu, teu, seu, nosso, vosso, minha, tua, sua,{nossa, vossa, meus teus, seu, nossoos, vossos, minhas, tuas, suas nossas, vossas, dele, dela, deles, delas, quem que, qual, quais, cujo, cujos, cuja, cujas, onde
Table 10: Pronoun sets used in our pronoun-focused automatic evaluation.
15The BLEU score for the baseline on Voita et al. (2018)\* is less than the one reported in their original work because of the reduced size of the test set and the different training set.
5978