SlowGuess/ACL_2020_OCR huggingface dataset:
KayYen Wong† Sameen Maruf‡ Gholamreza Haffari‡
Faculty of Information Technology, Monash University, VIC, Australia
$\dagger$ kywon63@student.monash.edu
$^\ddagger$ firstname_lastname@monash.edu
The advent of context-aware NMT has resulted in promising improvements in the overall translation quality and specifically in the translation of discourse phenomena such as pronouns. Previous works have mainly focused on the use of past sentences as context with a focus on anaphora translation. In this work, we investigate the effect of future sentences as context by comparing the performance of a contextual NMT model trained with the future context to the one trained with the past context. Our experiments and evaluation, using generic and pronoun-focused automatic metrics, show that the use of future context not only achieves significant improvements over the context-agnostic Transformer, but also demonstrates comparable and in some cases improved performance over its counterpart trained on past context. We also perform an evaluation on a targeted cataphora test suite and report significant gains over the context-agnostic Transformer in terms of BLEU.
Standard machine translation (MT) systems typically translate sentences in isolation, ignoring essential contextual information, where a word in a sentence may reference other ideas or expressions within a piece of text. This locality assumption hinders the accurate translation of referential pronouns, which rely on surrounding contextual information to resolve cross-sentence references. The issue is further exacerbated by differences in pronoun rules between source and target languages, often resulting in morphological disagreement in the quantity and gender of the subject being referred to (Vanmassenhove et al., 2018).
Rapid improvements in NMT have led to it replacing SMT as the dominant paradigm. With this, context-dependent NMT has gained traction, overcoming the locality assumption in SMT through the
use of additional contextual information. This has led to improvements in not only the overall translation quality but also pronoun translation (Jean et al., 2017; Bawden et al., 2018; Voita et al., 2018; Miculicich et al., 2018). However, all these works have neglected the context from future sentences, with Voita et al. (2018) reporting it to have a negative effect on the overall translation quality.
In this work, we investigate the effect of future context in improving NMT performance. We particularly focus on pronouns and analyse corpora from different domains to discern if the future context could actually aid in their resolution. We find that for the Subtitles domain roughly  $16\%$  of the pronouns are cataphoric. This finding motivates us to investigate the performance of a context-dependent NMT model (Miculicich et al., 2018) trained on the future context in comparison to its counterpart trained on the past context. We evaluate our models in terms of overall translation quality (BLEU) and also employ three types of automatic pronoun-targeted evaluation metrics. We demonstrate strong improvements for all metrics, with the model using future context showing comparable or in some cases even better performance than the one using only past context. We also extract a targeted cataphora test set and report significant gains on it with the future context model over the baseline.
Pronoun-focused SMT Early work in the translation of pronouns in SMT attempted to exploit coreference links as additional context to improve the translation of anaphoric pronouns (Le Nagard and Koehn 2010; Hardmeier and Federico 2010). These works yielded mixed results which were attributed to the limitations of the coreference resolution systems used in the process (Guillou, 2012).
5971
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5971-5978 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Domain	#Sentences	Document length
Subtitles	English-German
Europarl	9.39M/9K/14.1K	565.8/582.2/591.0
TED Talks	1.67M/3.6K/5.1K	14.1/15.0/14.1
0.21M/9K/2.3K	120.9/96.4/98.7
Subtitles	English-Portuguese
15.2M/16.1K/23.6K	580.4/620.6/605.3
Context-Aware NMT Multiple works have successfully demonstrated the advantages of using larger context in NMT, where the context comprises few previous source sentences (Wang et al., 2017; Zhang et al., 2018), few previous source and target sentences (Miculicich et al., 2018), or both past and future source and target sentences (Maruf and Haffari, 2018; Maruf et al., 2018, 2019).
Further, context-aware NMT has demonstrated improvements in pronoun translation using past context, through concatenating source sentences (Tiedemann and Scherrer, 2017) or through an additional context encoder (Jean et al., 2017; Bawden et al., 2018; Voita et al., 2018). Miculicich et al. (2018) observed reasonable improvements in generic and pronoun-focused translation using three previous sentences as context. Voita et al. (2018) observed improvements using the previous sentence as context, but report decreased BLEU when using the following sentence. We, on the other hand, observe significant gains in BLEU when using the following sentence as context on the same data domain.
3 Contextual Analysis of Corpora
To motivate our use of the future context for improving the translation of cataphoric pronouns in particular and NMT in general, we first analyse the distribution of coreferences for anaphoric and cataphoric pronouns over three different corpora - OpenSubtitles2018 $^{1}$  (Lison and Tiedemann, 2016), Europarl (Koehn, 2005) and TED Talks (Cettolo et al., 2012) - for English-German. For Europarl and TED Talks, we use the publicly available document-aligned version of the corpora (Maruf et al., 2019). For Subtitles, we align the English and German subtitles at the document-level using publicly available alignment links. $^{2}$  To control for the length and coherency of documents, we keep
Table 1: Train/dev/test statistics: number of sentences (K: thousands, M: millions), and average document length (in sentences). The #Documents can be obtained by dividing the #Sentences by the Document Length.
Pronoun	Subtitles	Europarl	TED Talks
Intrasentential	30.1	75.6	64.1
Anaphora (< 0)	54.3	19.6	28.5
Cataphora (> 0)	15.6	4.7	7.4
Table 2: Percentage of different pronoun types.
Figure 1: Plot showing proportion of intersentential English pronouns versus size of coreference resolution window for the Subtitles corpus (plots for Europarl and TED Talks are in the appendix).
subtitles with a run-time less than 50 minutes (for English) and those with number of sentences in the hundreds. The corpus is then randomly split into training, development and test sets in the ratio 100:1:1.5. Table 1 presents the corpora statistics.
Analysis of Coreferences We find the smallest window within which a referential English pronoun is resolved by an antecedent or postcedent using NeuralCoref. Table 2 shows that the majority of pronouns in Europarl and TED Talks corpora are resolved intrasententially, while the Subtitles corpus demonstrates a greater proportion of intersentential coreferences. Further, anaphoric pronouns are much more frequent compared to cataphoric ones across all three corpora. For Subtitles, we also note that a good number of pronouns (15.6%) are cataphoric,  $\sim 37\%$  of which are resolved within the following sentence (Figure 1). This finding motivates us to investigate the performance of a context-aware NMT model (trained on Subtitles) for the translation of cataphoric pronouns.
Datasets We experiment with the Subtitles corpus on English-German and English-Portuguese language-pairs. To obtain English-Portuguese data, we employ the same pre-processing steps as reported in §3 (corpus statistics are in Table 1). We use  $80\%$  of the training data to train our models and the rest is held-out for further evaluation as discussed later in §4.2. The data is truecased using
<sup>3</sup>https://github.com/huggingface/neuralcoref
Due to resource constraints, we use about two-thirds of the final training set ( $\sim 8\mathrm{M}$  sentence-pairs) for En-Pt.
http://www.opensubtitles.org/
http://opus.nlpl.eu/OpenSubtitles2018.php
5972
Lang. Pair	Baseline	HAN(k = +1)	HAN(k = -1)
English→German	31.87	32.53	32.48
German→English	35.92	36.64♠	36.32
English→Portuguese	35.45	36.04	36.21
Portuguese→English	39.34	39.96♠	39.69
the Moses toolkit (Koehn et al., 2007) and split into subword units using a joint BPE model with 30K merge operations (Sennrich et al., 2016).<sup>5</sup>
Description of the NMT systems As our baseline, we use the DyNet (Neubig et al., 2017) implementation of Transformer (Vaswani et al., 2017). For the context-dependent NMT model, we choose the Transformer-HAN encoder (Miculicich et al., 2018), which has demonstrated reasonable performance for anaphoric pronoun translation on Subtitles. We extend its DyNet implementation (Maruf et al., 2019) to a single context sentence. For training, Transformer-HAN is initialised with the baseline Transformer and then the parameters of the whole network are optimised in a second stage as in Miculicich et al. (2018) (details of model configuration are in Appendix A.1). For evaluation, we compute BLEU (Papineni et al., 2002) on tokenised truecased text and measure statistical significance with  $p < 0.005$  (Clark et al., 2011).
We consider two versions of the Transformer-HAN respectively trained with the following and previous source sentence as context. From Table 3, we note both context-dependent models to significantly outperform the Transformer across all language-pairs in terms of BLEU. Further, HAN  $(k = +1)$  demonstrates statistically significant improvements over the HAN  $(k = -1)$  when translating to English. These results are quite surprising as Voita et al. (2018) report decreased translation quality in terms of BLEU when using the following sentence for English→Russian Subtitles. To
Table 3: BLEU for the Transformer baseline and Transformer-HAN with following sentence  $(k = +1)$  and previous sentence  $(k = -1)$ .  $\spadesuit$  Statistically significantly better than HAN  $(k = -1)$ .
Model	English→German
APT	Precision	Recall	F1-score	CRC
Baseline	60.8	47.4	54.3	50.7	-
+HAN(k = +1)	61.4	48.3	54.3	51.1	-
+HAN(k = -1)	62.0	48.0	54.6	51.1	-
Model	German→English
APT	Precision	Recall	F1-score	CRC
Baseline	77.9	56.9	50.4	53.4	50.4
+HAN(k = +1)	78.3	57.9	50.6	54.0	50.9
+HAN(k = -1)	78.3	58.0	50.5	54.0	51.0
Model	English→Portuguese
APT	Precision	Recall	F1-score	CRC
Baseline	46.4	54.8	56.0	55.4	-
+HAN(k = +1)	47.0	55.8	55.2	55.5	-
+HAN(k = -1)	47.3	56.0	55.4	55.7	-
Model	Portuguese→English
APT	Precision	Recall	F1-score	CRC
Baseline	64.3	54.9	51.1	53.0	50.2
+HAN(k = +1)	64.6	55.7	51.5	53.5	50.9
+HAN(k = -1)	64.3	55.6	51.2	53.4	51.6
Table 4: Pronoun-focused evaluation on generic test set for models trained on different types of context.
identify if this discrepancy is due to the language-pair or the model, we conduct experiments with English  $\rightarrow$  Russian in the same data setting as Voita et al. (2018) and find that HAN  $(k = +1)$  still significantly outperforms the Transformer and is comparable to HAN  $(k = -1)$  (more details in Appendix A.2).
Pronoun-Focused Automatic Evaluation For the models in Table 3, we employ three types of pronoun-focused automatic evaluation:
1. Accuracy of Pronoun Translation (APT) (Miculicich Werlen and Popescu-Belis, 2017) $^9$ . This measures the degree of overlapping pronouns between the output and reference translations obtained via word-alignments.
2. Precision, Recall and F1 scores. We use a variation of AutoPRF (Hardmeier and Federico, 2010) to calculate precision, recall and F1-scores. For each source pronoun, we compute the clipped count (Papineni et al., 2002) of overlap between candidate and reference translations. To eliminate word alignment errors, we compare this overlap over the set of dictionary-matched target pronouns, in contrast to the set of target words aligned to a given source pronoun as done by AutoPRF and APT.
3. Common Reference Context (CRC) (Jwalaipuram et al., 2019). In addition to the previous
$^{9}$ https://github.com/idiap/APT
5Tokenisation is provided by the original corpus.
$^{6}$ https://github.com/duyvuleo/
Transformer-DyNet
<sup>7</sup>Where in the original architecture,  $k$  sentence-context vectors were summarised into a document-context vector, we omit this step when using only one sentence in context.
The code and data are available at https://github.com/sameenmaruf/ac12020-contextnmt-cataphora.
5973
Model	BLEU	APT	F1-score
Baseline (Transformer)	31.87	61.6	49.1
+HAN(k = ∅)	32.30	61.6	49.1
+HAN(k = +1,+2)	32.56♣	62.0	49.8
+HAN(k = -2,-1)	32.47	61.9	49.8
+HAN(k = -2,-1,+1,+2)	32.59♣	62.0	49.9
two measures which rely on computing pronoun overlap between the target and reference translation, we employ an ELMo-based (Peters et al., 2018) evaluation framework that distinguishes between a good and a bad translation via pairwise ranking (Jwalapuram et al., 2019). We use the CRC setting of this metric which considers the same reference context (one previous and one next sentence) for both reference and system translations. However, this measure is limited to evaluation only on the English target-side.[10]
The results using the aforementioned pronoun evaluation metrics are reported in Table 4. We observe improvements for all metrics with both HAN models in comparison to the baseline. Further, we observe that the HAN  $(k = +1)$  is either comparable to or outperforms HAN  $(k = -1)$  on APT and F1 for  $\mathrm{De} \rightarrow \mathrm{En}$  and  $\mathrm{Pt} \rightarrow \mathrm{En}$ , suggesting that for these cases, the use of following sentence as context is at least as beneficial as using the previous sentence. For  $\mathrm{En} \rightarrow \mathrm{De}$ , we note comparable performance for the HAN variants in terms of F1, while for  $\mathrm{En} \rightarrow \mathrm{Pt}$ , the past context appears to be more beneficial.[11] In terms of CRC, we note HAN  $(k = -1)$  to be comparable to  $(\mathrm{De} \rightarrow \mathrm{En})$  or better than HAN  $(k = +1)$ $(\mathrm{Pt} \rightarrow \mathrm{En})$ . We attribute this to the way the metric is trained to disambiguate pronoun translations based on only the previous context and thus may have a bias for such scenarios.
Ablation Study We would like to investigate whether a context-aware NMT model trained on a wider context could perform well even if we do not have access to the same amount of context at decoding. We thus perform an ablation study for
Table 5: Evaluation on English  $\rightarrow$  German generic test set for HAN trained with  $k = \{-2, -1, +1, +2\}$  but decoded with varying context.  $\spadesuit$ : Statistically significantly better than HAN with no context ( $k = \emptyset$ ).
Model	English→German
Cataphora	DET	PROPN	NOUN
Baseline	32.33	32.14	33.02	32.93
+HAN(k = +1)	32.93♠	32.68♠	33.98♠	33.76♠
Model	German→English
Cataphora	DET	PROPN	NOUN
Baseline	36.91	36.35	38.81	38.84
+HAN(k = +1)	37.68♠	37.19♠	39.51	39.45
Model	English→Portuguese
Cataphora	DET	PROPN	NOUN
Baseline	36.29	35.91	37.91	37.60
+HAN(k = +1)	37.08♠	36.70♠	38.49	38.19
Model	Portuguese→English
Cataphora	DET	PROPN	NOUN
Baseline	40.74	40.12	42.77	42.63
+HAN(k = +1)	41.63♠	41.06♠	43.60♠	43.42♠
Table 6: BLEU on the cataphora test set and its subsets for the Transformer and Transformer-HAN  $(k = +1)$ . Statistically significantly better than the baseline.
English  $\rightarrow$  German using the HAN model trained with two previous and next sentences as context and decoded with variant degrees of context.
From Table 5, we note that reducing the amount of context at decoding time does not have adverse effect on the model's performance. However, when no context is used, there is a statistically significant drop in BLEU, while APT and F1-scores are equivalent to that of the baseline. This suggests that the model does rely on the context to achieve the improvement in pronoun translation. Further, we find that the future context is just as beneficial as the past context in improving general translation performance.
Cataphora-Focused Test Suite To gauge if the improvements in Table 3 for the HAN  $(k = +1)$  model are coming from the correct translation of cataphoric pronouns, we perform an evaluation on a cataphoric pronoun test suite constructed from the held-out set mentioned earlier in § 3. To this end, we apply NeuralCoref over the English side to extract sentence-pairs which have a cataphoric pronoun in one sentence and the postcedent in the next sentence. This is further segmented into subsets based on the part-of-speech of the postcedent, that is, determiner (DET), proper noun (PROPN) or all nouns (NOUN) (more details in the appendix).<sup>12</sup>
From Table 6, we observe HAN  $(k = +1)$  to outperform the baseline for all language-pairs when evaluated on the cataphora test suite. In general, we observe greater improvements in BLEU when trans
<sup>12</sup>We note that there may be some overlap between the three pronoun subsets as a test sentence may contain more than one type of pronoun.
10We use the same English pronoun list for all pronoun-focused metrics (provided by Jwalapuram et al. (2019) at https://github.com/ntunlp/eval-anaphora). All pronoun sets used in our evaluation are provided in Appendix A.4. 11It should be noted that for Portuguese, adjectives and even verb forms can be marked by the gender of the noun and these are hard to account for in automatic pronoun-focused evaluations.
5974
Figure 2: Example attention map between source (y-axis) and context (x-axis). The source pronoun he correctly attends to the postcedents Richard in the context.
lating to English, which we attribute to the simplification of cross-lingual pronoun rules when translating from German or Portuguese to English. $^{13}$  We also observe fairly similar gains in BLEU across the different pronoun subsets, which we hypothesise to be due to potential overlap in test sentences between different subsets. Nevertheless, we note optimum translation quality over the noun subsets (PROPN and NOUN), while seeing the greatest percentage improvement on the DET subset. For the latter, we surmise that the model is able to more easily link pronouns in a sentence to subjects prefixed with possessive determiners, for example, "his son" or "their child".
We also perform an auxiliary evaluation for Transformer-HAN ( $k = -1$ ) trained with the previous sentence as context on the cataphora test suite and find that the BLEU improvements still hold. Thus, we conclude that Transformer-HAN (a context-aware NMT model) is able to make better use of coreference information to improve translation of pronouns (detailed results in Appendix A.3).
Qualitative Analysis We analyse the distribution of attention to the context sentence for a few test cases.14 Figure 2 shows an example in which a source pronoun he attends to its corresponding postcedent in context. This is consistent with our hypothesis that the HAN  $(k = +1)$  is capable of exploiting contextual information for the resolution of cataphoric pronouns.
In this paper, we have investigated the use of future context for NMT and particularly for pronoun translation. While previous works have focused on the
use of past context, we demonstrate through rigorous experiments that using future context does not deteriorate translation performance over a baseline. Further, it shows comparable and in some cases better performance as compared to using the previous sentence in terms of both generic and pronoun-focused evaluation. In future work, we plan to investigate translation of other discourse phenomena that may benefit from the use of future context.
The authors are grateful to the anonymous reviewers for their helpful comments and feedback and to George Foster for fruitful discussions. This work is supported by a Google Faculty Research Award to G.H. It is further supported by the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE) (www.massive.org.au).
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1304-1313, New Orleans, Louisiana, USA. Association for Computational Linguistics.
Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012.  $\mathrm{WIT}^3$ : Web inventory of transcribed and translated talks. In Proceedings of the  $16^{th}$  Conference of the European Association for Machine Translation (EAMT), pages 261-268, Trento, Italy.
Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (Short Papers), pages 176-181. Association for Computational Linguistics.
Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of the Student Research Workshop at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 1-10, Avignon, France. Association for Computational Linguistics.
Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In Proceedings of the Seventh International Workshop on Spoken Language Translation (IWSLT), pages 283-289.
Sebastien Jean, Stanislas Lauly, Orhan First, and Kyunghyun Cho. 2017. Does neural machine
<sup>13</sup>It should be noted that the cataphora test set is extracted based on the existence of cataphoric-pairs in the English-side, which may have biased the evaluation when English was in the target.
14 Attention is average of the per-head attention weights.
5975