SlowGuess/NAACL_2021_OCR huggingface dataset:
We use no pre-existing document-level parallel data to train the neural networks of DocRepair, Bayes DocReranker, and our methods, although we use a small amount of document-level parallel data as the development set to tune hyperparameters in the methods that combine multiple models. Instead, document-level information is fed to the models via the round-trip augmented data (DocRepair) or language models (Bayes DocReranker and our methods).
Hyper-parameters We tune the models' hyperparameters based on BLEU score on the development set in the evaluation with BLEU, while we tune these hyper-parameters in the evaluation of contrastive tests by maximizing the coefficient of D-LM under the constraint that it does not deteriorate BLEU compared to the SentTransformer.
For beam search to produce  $B$ -best outputs in Bayes DocReranker and our C-AWARE Rerank, we
Although we can train DocTransformer only on pseudo document-level parallel data generated by back-translation, we confirmed in preliminary experiments that the resulting model exhibited poor performance.
4https://github.com/google/
Models	para only	monolingual data
6M	15M	30M
SentTransformer (w/ BT)	32.36	32.32	32.40	32.40
Shallow Fusion	n/a	32.39	32.56	32.52
basielines
DocTransformer (w/ BT)	32.50	32.36	31.88	31.59
DocRepair	n/a	32.13	32.36	32.35
Bayes DocReranker	n/a	32.80*	33.58**	33.75**
w/o context	n/a	32.53	33.44**	33.67**
proposed
C-AWARE Rerank	n/a	32.74*	33.01**	32.93*
C-AWARE Beam	n/a	32.26	32.28	32.27
Cond. Shallow Fusion	n/a	32.38	32.55	32.55
use a beam size of  $B = 20$ . For document-level beam search of Bayes DocReranker, we use a beam size  $B' = 5$ . For beam search of SentTransformer, DocTransformer, C-AWARE beam, and shallow fusion, we use a beam size of  $B = 4$ .
3.3 Document-level Language models
The architecture of the document-level LM is the decoder part of a Transformer. The number of decoder blocks is 12. The model size is 768 with 12 attention heads, and the inner layer of the feedforward networks has 3072 units. We use position embeddings to represent position information.
As described in § 2.1, when training the language models, a special control symbol  $\langle /s\rangle$  is inserted at every sentence boundary. Each training mini-batch contains text spans each of which is a randomly sampled fragment of a document with a maximum span length of  $W = 384$ . Text spans are batched such that about 32,000 tokens are in a training batch.
3.4 Evaluation methods
The existing automatic metrics are not adequate to evaluate gains from additional contexts (Bawden et al., 2018; Läubli et al., 2018; Müller et al., 2018; Voita et al., 2019b; Sugiyama and Yoshinaga, 2019). We thus adopt a contrastive test set (Voita et al., 2019b) to evaluate the model's ability to capture contextual information in translation, in addition to the evaluation by BLEU scores (Papineni et al., 2002) to confirm that the methods do not sacrifice general translation performance. BLEU is computed using multi-bleu.perl from the Moses Toolkit after decoding the subword repre
Table 2: Test set BLEU scores. \* and \*\* indicate that gains from SentTransformer in the same column are statistically significant  $(p <   0.05$  and  $p <   0.01)$  by bootstrap resampling with 1000 samples, respectively.
Models	deixis	lex.c	ell.infl	ell.vp
SentTransformer	50.0	45.9	53.2	27.0
w/ BT	50.0	45.9	51.6	26.8
basielines
Doc-Transformer	50.0	45.9	56.0	57.2
w/ BT	50.0	45.9	64.4	68.2
DocRepair	89.1	75.8	82.2	67.2
Bayes DocReranker	65.2	72.2	59.6	44.6
proposed
C-SCORE	86.9	94.9	78.2	77.0
Cond. Shallow Fusion	54.7	55.3	53.4	32.4
D-LM PMI(c(y),y)	96.8	97.8	75.8	90.6
p(y|c(y))	89.7	95.7	77.4	81.6
Table 3: Results on contrastive test sets.
sensation of the models' outputs into words using SentencePiece.
The contrastive test set consists of contrastive questions for context-aware NMT models to answer. Each question has a source sentence  $\pmb{x}$ , a source context  $c^{(x)}$ , a target context  $c^{(y)}$ , and translation candidates  $\mathcal{V} = \{y^1, \dots, y^M\}$ . Models must answer with a candidate  $\hat{y} \in \mathcal{V}$  which would be the most appropriate translation of  $\pmb{x}$ , i.e.
$$
\hat {y} = \operatorname * {a r g   m a x} _ {y \in \mathcal {Y}} p (\boldsymbol {y} | \boldsymbol {x}, \boldsymbol {c} ^ {(x)}, \boldsymbol {c} ^ {(\boldsymbol {y})})
$$
The test sets consist of 6000 examples in total.
4.1 General translation performance measured by BLEU scores
Table 2 lists the performance of the models in terms of BLEU scores. Bayes DocReranker and our C-AWARE Rerank consistently outperformed the baseline SentTransformer, even when it used data augmentation by back-translation, while the other methods are just comparable to the baseline. Although Bayes DocReranker performed the best among all the models, the comparison to Bayes DocReranker without context information (using  $p_{\mathrm{S - LM}}(\pmb {y}_i)$  instead of  $p_{\mathrm{D - LM}}(\pmb {y}_i|\pmb {y}_{< i})$ ) reveals that most of the improvement is not obtained by the use of contexts. Back-translation did not contribute to BLEU possibly because the original parallel data is already large and there was little room for improvement with additional pseudo data.
4.2 Results on contrastive test sets
Tables 3 lists evaluation results (accuracy) of the contrastive tests with models using 30M monolingual data. The highest scores on each column
(a) PMI
(b) PMI with rand. context
(c) Cond. prob.
Figure 1: Source-target correlation of contextual PMI (a, b) and conditional probability (c, d), calculated based on the correct context (a, c) and wrong context that is randomly chosen from the dataset (b, d). The dataset is a subset of the training data from the English-Russian parallel corpus. Plots are for 4166 sentence pairs in the dataset.
(d) Cond. prob. (rand. context)
are in bold, and additionally, the higher one of the two D-LM-based scores is shown in bold. The contrastive test include four test sets: deixis is for person deixis, lex.c is for lexical cohesion, ell.infl is for inflection of Russian nouns caused by ellipsis in the source sentence, and ell.vp is for verb ellipsis in English text which is not allowed in Russian. Although the contrastive test is targeted at context-aware NMT models, it is possible to answer the contrastive questions by arg maxy PMI  $(c^{(y)}, y)$  or arg maxy  $p(y|c^{(y)})$ . Scores obtained by these two objectives are also reported in the table in addition to the scores obtained by SentTransformer.
Our C-SCORE outperforms all the context-aware models other than DocRepair. The performance of C-SCORE is slightly worse than DocRepair for deixis (2.2 points) and ell.infl (4.0 points), while achieving large improvements for lex.c (19.1 points) and ell.vp (9.8 points) over DocRepair.
D-LM only objectives achieve higher scores than C-SCORE, except for ell.infl. This is not surprising because the choices in the tests are guaranteed to be valid as translation for the source sentences if given some appropriate context, so the questions can be solved without translation. This result still indicates that the D-LM scores give good hints for tackling contextual ambiguities. The advantage of C-SCORE over the SentTransformer is demonstrated by the excellent performance of D-LM in capturing contexts in translation.
4.3 On translation efficiency
The inference speed depends mainly on the model size and beam size. In our experiments on a single Titan Xp GPU, SentTransformer decoded the fastest at 66 sents/sec, followed by DocTransformer that ran in 40 sents/sec. DocRepair ran in about 28 sents/sec, slightly slower because it decodes in
two passes. C-AWARE Rerank and Bayes DocRe-ranker were about 4.3 sents/sec and 7.7 sents/sec respectively. We expect that these models would be accelerated by using a language model with a better cache mechanism (e.g. TransformerXL (Dai et al., 2019)). C-AWARE Beam ran in about 13 sents/sec.6 We leave thorough analysis on speed/performance trade-offs to future work.
4.4 PMI correlation analysis
In § 4.2 we have confirmed the effectiveness of PMI as a measure of a valid translation given context using contrastive tests. To gain a deeper insight into how well PMI conveys semantic connections between the current sentence and its context, we analyze the correlation of PMI between source and target sentences.
PMI correlation between source and target
The main result we show in this section is that the PMI of the source and target correlate well. This is important because this supports the idea that PMI is a language-independent measure of the connection between the current sentence and its context.
Although we have discussed only target-side  $\mathrm{PMI}(\pmb{c}^{(y)},\pmb{y})$  defined by Eq. 4, we can compute the source-side  $\mathrm{PMI}(\pmb{c}^{(x)},\pmb{x})$  in the same way. Given a document-level parallel corpus, we measure a correlation between  $\mathrm{PMI}(\pmb{c}^{(x)},\pmb{x})$  and  $\mathrm{PMI}(\pmb{c}^{(y)},\pmb{y})$  for each sentence pair  $(\pmb{x},\pmb{y})$  in the corpus.
Figure 1a shows the PMI correlation for about
Note that the running time of NMT decoding also depends on the degree of parallelism, and for C-AWARE Beam, decoding multiple sentences in parallel is less trivial since it demands that all the previous sentences in the document are translated by the time it starts to translate the current one. In our experiments, assuming a practical scenario where a large number of users input their documents for translation, we translate multiple documents in parallel so that multiple sentences from different documents can be translated in parallel.
Figure 2: Correlation of contextual PMI between the source sentences (from the training data) and the outputs of some models (SentTransformer, C-AWARE beam without  $T$ -scaling, and C-AWARE beam with  $T$ -scaling of  $T = 4$ ).
4000 sentence pairs taken from the dev data. The pairs of PMI values are computed using English and Russian language models trained on the training data. We observe a clear correlation between source and target, which agrees with the intuition that if the target sentence matches well in the context, so does the source sentence. What is also obvious in Figure 1a is that most of the points lay in the first quadrant where both the source and target contextual PMI is greater than 0, which is explained by the simple intuition that most sentences should have positive co-occurrence relation with their contexts. This behavior is lost when computing the contextual PMI using an incorrect context  $\tilde{c}$  randomly chosen in the dataset as shown in Figure 1b.
The effectiveness of PMI as a measure of the valid translation of the current sentence given context is further emphasized when compared to the conditional probability  $p(\pmb{y}|\pmb{c}^{(y)})$ , which could be an alternative measure of how suitable  $\pmb{y}$  is in the context as described in § 2.2.4. Figure 1c and 1d are the conditional probability version of Figure 1a and 1b:  $(p(\pmb{x}|\pmb{c}^{(x)}), p(\pmb{y}|\pmb{c}^{(y)}))$  for each sentence pair  $(x, y)$  in the same dataset are plotted in Figure 1c and the same tuples but with random contexts are plotted in Figure 1d. Unlike the contextual PMI correlation, conditional probability correlation remains high even when we give wrong contexts. This is because the conditional probability of a sentence is highly affected by how frequently the sentence is observed regardless of context; if the source sentence is written with common expressions, then so is the target sentence and they are likely to be observed regardless of the context.
Analysis of the model outputs
PMI correlation gives us a good explanation of how C-AWARE beam without  $T$ -scaling fails. We plot the PMI correlation between the source sentences and their translations obtained with NMT models
(Figure 2). We can find some outliers in the bottom right area of the plot for C-AWARE beam without  $T$ -scaling, which is the cause of the low correlation coefficient  $R = 0.610 < R_{\mathrm{src-ref}} = 0.695$ . This result suggests that C-AWARE beam without  $T$ -scaling chooses some tokens based on excessively high token-wise PMI, which breaks some translations resulting in the low BLEU. Translation of the SentTransformer shows a higher correlation with the source texts than the reference translation (Figure 1a). One possible explanation for this is alignment errors in the corpus: although worse than the reference translations in quality, outputs of SentTransformer are considered to be perfectly aligned to the source sentences. C-AWARE beam with  $T$ -scaling ( $T = 4$ ) seems to solve this issue and achieves the highest PMI correlation  $R = 0.740$ .
The effectiveness of incorporating context into translation was shown in earlier literature on document-level NMT (Tiedemann and Scherrer, 2017; Bawden et al., 2018) using the single encoder architecture. Multi-encoder architectures were explored to better capture contextual information (Wang et al., 2017; Tu et al., 2018; Jean et al., 2017; Miculicich et al., 2018; Voita et al., 2018; Bawden et al., 2018; Maruf and Haffari, 2018; Maruf et al., 2019; Kang et al., 2020; Zhang et al., 2020). However, since parallel data is often constructed by picking up reliable sentential alignments from comparable documents, document-level sentence-aligned parallel data for training these document-level NMT models are expensive to obtain and available in only a few domains and language pairs (Sugiyama and Yoshinaga, 2019).
Recent studies have therefore started to focus on modeling contexts using document-level monolingual data. The current approaches are grouped into three categories: data augmentation via back
translation (Sugiyama and Yoshinaga, 2019), a post-editing model (Voita et al., 2019a), and modeling document-level fluency via document-level LMs (Stahlberg et al., 2019; Yu et al., 2020; Jean and Cho, 2020). In what follows, we review these approaches in detail.
Sugiyama and Yoshinaga (2019) reported that the data augmentation by back-translation (Sennrich et al., 2016) enhances a document-level NMT model with a single encoder architecture in low-resource settings. However, we have obtained limited improvements in our settings (Table 2 and Table 3). Moreover, this approach is expensive since it learns a document-level NMT model from a massive amount of pseudo parallel data.
Voita et al. (2019a) proposed DocRepair, a context-aware post-editing model that corrects outputs of a sentence-level NMT model. Because DocRepair ignores the confidence of the first-stage sentence-level translation and possible alternative translations, it can miscorrect outputs of the sentence-level NMT model when they are irregular but correct. Moreover, when we change the target sentence-level NMT model, the accompanying post-editing model must be trained from its outputs. Our approaches, on the other hand, attempt a more "soft" revision, taking into account the output probabilities, i.e., confidence of the sentence-level NMT, and can perform context-aware decoding with any sentence-level NMT model, reusing a pre-trained document-level LM.
Stahlberg et al. (2019) and Yu et al. (2020) utilize a document-level LM to model document-level fluency of outputs; these approaches are similar to shallow fusion (Gulcehre et al., 2015)<sup>7</sup> with document-level LM ( $\S$  2.2.4), although they perform a document-level reranking of translation hypotheses generated for individual source sentences by using sentence-level NMT. In particular, Yu's formulation has a probabilistic foundation like our approaches, and additionally utilizes a backward translation model. Although their formulation brings a significant improvement in BLEU (Table 2), the score is not obtained by better document-level
translation; the comparable BLEU score of the nocontext version of the method (Table 2) and the results of the contrastive tests (Table 3) reveal that the improvement is mostly due to the context-agnostic language model prior and the backward translation model. As we have discussed in  $\S 2.2.4$ , document-level LM scores prefer tokens which frequently appear regardless of context and are unlikely to lead to better document-level translation. Moreover, their method requires training a back-translation model corresponding to the target sentence-level NMT model.
Finally, we noticed that Jean and Cho (2020) (which appeared after the preprint version of this paper (Sugiyama and Yoshinaga, 2020) $^{8}$  had been submitted) have reached a formulation that is very similar to the one presented in this paper by reformulating a noisy channel model of Bayes DocReranker (Yu et al., 2020). Concrete differences between our work and theirs include the fact that we conducted thorough analysis on the performance of different decoding strategies (not only beam search but also reranking). We also interpreted the subtraction of LM scores as point-wise mutual information and analyzed it by observing PMI correlation between source and target PMI to deepen the understanding of the formulation.
We present an approach to context-aware NMT based on PMI between the context and the current sentence. We first provide the formulation of the objective, C-SCORE, and the computation process of the C-SCORE using a sentence-level translation model and a document-level language model. We investigate two search methods, reranking and beam search, and evaluate the methods for English-Russian translation. We also provide some analysis and visualization to better understand the nature of PMI between the context and the current sentence.
We plan to design context-aware BLEU using PMI for evaluating context-aware NMT models. We will evaluate our method on non-autoregressive NMT (Gu et al., 2017). We will release all code and data to promote the reproducibility of results.
Our work is also related to shallow fusion (Gulcehre et al., 2015), in which token-wise probabilities output by an NMT model and a sentence-level LM are combined to be used as translation scores in decoding. The theoretical background of shallow fusion and our C-SCORE are different: in shallow fusion, the LM is intended to promote fluency of translations, whereas in our C-SCORE, we use the probability ratio of two LM probabilities which only provides contextual difference and fluency is still left to the translation model.
This preprint is submitted to and rejected from EMNLP 2020; the interested reader may refer to this paper for experiments on other language pairs such as English to French and English to Japanese translation.
http://www.tkl.iis.u-tokyo.ac.jp/\~sugi/NAACL2021/
We thank anonymous reviewers for their valuable comments. We also thank Joshua Tanner for proofreading this paper. We also thank Masato Neishi for technical advice on implementations of neural machine translation. The research was supported by NII CRIS collaborative research program operated by NII CRIS and LINE Corporation.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the third International Conference on Learning Representations (ICLR).
Rachel Bawden, Rico Senrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1304-1313, New Orleans, Louisiana.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2017. Non-autoregressive neural machine translation. In Proceedings of the fifth International Conference for Learning Representations (ICLR).
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. Computing Research Repository, arXiv:1503.03535.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR.
Sebastien Jean and Kyunghyun Cho. 2020. Log-linear reformulation of the noisy channel model for document-level neural machine translation. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pages 95-101, Online.
Sebastien Jean, Stanislas Lauly, Orhan First, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135.
Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020. Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2242-2254, Online.
Samuel Läubli, Rico Senrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium.
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275-1284, Melbourne, Australia.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3092-3102, Minneapolis, Minnesota.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954, Brussels, Belgium.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 61-72, Brussels, Belgium.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany.