Venue: NAACL 2021

# Context-aware Decoder for Neural Machine Translation

using a Target-side Document-Level Language Model

 Amane Sugiyama

The University of Tokyo1 sugi@tkl.iis.u-tokyo.ac.jp &Naoki Yoshinaga

Institute of Industrial Science,

The University of Tokyo ynaga@iis.u-tokyo.ac.jp

Footnote 1: Currently at Mitsubishi UFI Morgan Stanley Securities Association for Computational Linguistics: Human Language Technologies, pages 5781–5791

###### Abstract

Although many end-to-end context-aware neural machine translation models have been proposed to incorporate inter-sentential contexts in translation, these models can be trained only in domains where parallel documents with sentential alignments exist. We therefore present a simple method to perform context-aware decoding with any pre-trained sentence-level translation model by using a document-level language model. Our context-aware decoder is built upon sentence-level parallel data and target-side document-level monolingual data. From a theoretical viewpoint, our core contribution is the novel representation of contextual information using point-wise mutual information between context and the current sentence. We demonstrate the effectiveness of our method on English to Russian translation, by evaluating with bleu and contrastive tests for context-aware translation.

## 1 Introduction

Neural machine translation (nmt) has typically been explored in sentence-level translation settings. Such sentence-level nmt models inevitably suffer from ambiguities when a source sentence has multiple plausible interpretations. Examples of such ambiguities include anaphora, ellipsis, and lexical coherence Voita et al. (2019b); although resolving these ambiguities has only a minor impact on the translation performance measured by bleu scores Papinieni et al. (2002), they are vital in smoothly reading the translated documents.

To address this issue, context-aware nmt models which incorporate document-level information in translation have recently been explored Jean et al. (2017); Wang et al. (2017); Tiedemann and Scherrer (2017); Maruf and Haffari (2018); Voita et al. (2018); Bawden et al. (2018); Miculicich et al. (2018); Maruf et al. (2019); Voita et al. (2019b); Yu et al. (2020); Kang et al. (2020); Zhang et al. (2020). Most of these models are end-to-end models that require document-level parallel data with sentential alignments for training. However, this data is available in only a few domains Sugiyama and Yoshinaga (2019). Researchers have therefore started to utilize target-side monolingual data to construct auxiliary models which help a sentence-level nmt model perform context-aware translation Voita et al. (2019a); Stahlberg et al. (2019); Yu et al. (2020).

In this study, we propose a simple yet effective approach to context-aware nmt using two primitive components, a sentence-level nmt model and a document-level language model (lm). We can independently train the two components on common sentence-level parallel data and document-level monolingual data, respectively, without using document-level parallel data. Our approach thereby makes it possible to perform context-aware translation with any pre-trained sentence-level nmt model, using a pre-trained document-level lm.

To give a probabilistic foundation to this combination of two independent models, we exploit the probabilistic nature of nmt decoding. When generating a sequence, a left-to-right decoder outputs a categorical probability distribution over the vocabulary at every time step. The decoder assigns higher probabilities to the tokens that would be more suitable at that step. Therefore, when multiple valid translations are possible for the source sentence, the decoder just gives a higher probability to the translation that is plausible without considering contexts. We thus adjust the probability distributions in a context-aware manner using a target-side document-level lm which models inter-sentential dependencies in the target-side document.

We evaluate our methods on English to Russian translations with the OpenSubtitles2018 corpus Lison et al. (2018) in terms of the bleu scores and contrastive discourse test sets Voita et al. (2019b). Experimental results confirm that

our method achieved comparable performance with existing context-aware nmt models that require either document-level parallel data (Zhang et al., 2018; Sugiyama and Yoshinaga, 2019) or more than one additional model (Voita et al., 2019a; Yu et al., 2020) for capturing contexts in translation.

The contributions of this paper are as follows:

* We theoretically derived c-score, a score to **qualify context-aware translation without the need for document-level parallel data**.
* Two formulations with c-score turn any **pre-trained sentence-level nmt model into a context-aware model**, if it generates \(n\)-best outputs or performs left-to-right decoding.
* A comparison between our approach and shallow fusion (Gulcehre et al., 2015) reveals that our approach **redformulates shallow fusion while adding a probabilistic foundation**.

## 2 Context-aware Decoding using Document-level Language Model

In this section, assuming a sentence-level encoder-decoder model (Bahdanau et al., 2015; Vaswani et al., 2017), we first derive _context-aware score_ (c-score for short), a context-aware objective function of outputs to be maximized in decoding. We then describe how to compute the c-score using the decoder with a document-level language model (d-lm) (SS 2.1). We finally detail how to perform context-aware decoding based on c-score (SS 2.2).

### c-score: objective function for context-aware nmt decoding

Let us consider the problem of finding a translation \({\mathbf{y}}\) of a source sentence \({\mathbf{x}}\) in a document. The target-side context sentence(s) preceding \({\mathbf{y}}\), \({\mathbf{c^{({\mathbf{y}}})}}\), are to be given by the past translations. We formulate context-aware translation conditioned on \({\mathbf{c^{({\mathbf{y}}})}}\) as the maximization of the conditional probability \(p({\mathbf{y}}|{\mathbf{x}},{\mathbf{c^{({\mathbf{y}}})}})\),

\[\hat{{\mathbf{y}}} =\arg\max_{{\mathbf{y}}}\log p({\mathbf{y}}|{\mathbf{x}},{\mathbf{c^{({\mathbf{y}}}) }})\] \[=\arg\max_{{\mathbf{y}}}\log\frac{p({\mathbf{c^{({\mathbf{y}}})}}|{\mathbf{x}},{ \mathbf{y}})p({\mathbf{y}}|{\mathbf{x}})}{p({\mathbf{c^{({\mathbf{y}}})}}|{\mathbf{x}})}\] \[=\arg\max_{{\mathbf{y}}}\log p({\mathbf{c^{({\mathbf{y}}})}}|{\mathbf{x}},{\mathbf{y}} )p({\mathbf{y}}|{\mathbf{x}}). \tag{1}\]

Assuming that \({\mathbf{x}}\) and \({\mathbf{y}}\) are semantically similar, we make the following approximation,

\[p({\mathbf{c^{({\mathbf{y}}})}}|{\mathbf{y}},{\mathbf{x}})\approx p({\mathbf{c^{({\mathbf{ y}}})}}|{\mathbf{y}}). \tag{2}\]

From Eq. 1 and Eq. 2, we obtain

\[\hat{{\mathbf{y}}} \approx\arg\max_{{\mathbf{y}}}\log p({\mathbf{c^{({\mathbf{y}}})}}|{\mathbf{y}})p ({\mathbf{y}}|{\mathbf{x}})\] \[=\arg\max_{y}\log\frac{p({\mathbf{c^{({\mathbf{y}}})}},{\mathbf{y}})}{p({\mathbf{ c^{({\mathbf{y}}})}})p({\mathbf{y}})}p({\mathbf{y}}|{\mathbf{x}})\] \[=\arg\max_{{\mathbf{y}}}\text{c-score}({\mathbf{y}};{\mathbf{x}},{\mathbf{c^{({\bm {y}}})}})\]

where

\[\text{c-score}({\mathbf{y}};{\mathbf{x}},{\mathbf{c^{({\mathbf{y}}})}})=\log p({\mathbf{y}}|{\mathbf{ x}})+\textsc{pmi}({\mathbf{c^{({\mathbf{y}}})}},{\mathbf{y}}) \tag{3}\]

\[\textsc{pmi}({\mathbf{c^{({\mathbf{y}}})}},{\mathbf{y}})=\log\frac{p({\mathbf{c^{({\mathbf{y}}})}}, {\mathbf{y}})}{p({\mathbf{c^{({\mathbf{y}}})}})p({\mathbf{y}})}=\log\frac{p({\mathbf{y}}|{\mathbf{c^{( {\mathbf{y}}})}})}{p({\mathbf{y}})} \tag{4}\]

\[\textsc{pmi}({\mathbf{c^{({\mathbf{y}}})}},{\mathbf{y}})\] is the point-wise mutual information of \[{\mathbf{c^{({\mathbf{y}}})}}\] and \[{\mathbf{y}}\] which represents the degree of co-occurrence of \[{\mathbf{y}}\] and \[{\mathbf{c^{({\mathbf{y}}})}}\] Given \[{\mathbf{x}}\] , \[{\mathbf{y}}\] and \[{\mathbf{c^{({\mathbf{y}}})}}\] we can evaluate the c-score by computing the two terms in Eq. 3 using a sentence-level nmt (s-nmt) and a document-level lm (d-lm), respectively.

NotationsWe first introduce some notation to explain the computation in Eq. 3 and Eq. 4 using (auto-regressive) neural sequence generation models in nmt and lm. For a sequence \({\mathbf{s}}\) (\(|{\mathbf{s}}|\geq 0\)) and token \(w\), a neural sequence generation model parameterized by \(\theta\) can compute the log probability that \(w\) follows \({\mathbf{s}}\), which we denote by \(\log p_{\theta}(w|{\mathbf{s}}))\):

\[\log p_{\theta}(w~{}\mathrm{follows}~{}{\mathbf{s}})=\log\frac{p_{\theta}({\mathbf{s}} \cdot w)}{p_{\theta}({\mathbf{s}})}=\log p_{\theta}(w|{\mathbf{s}})\]

where "\(\cdot\)" denotes sequence concatenation. Applying this auto-regressively, for any sequence \({\mathbf{s}}^{(1)}\) (\(|{\mathbf{s}}^{(1)}|\geq 0\)) and \({\mathbf{s}}^{(2)}\) (\(|{\mathbf{s}}^{(2)}|\geq 1\)), the probability that \({\mathbf{s}}^{(2)}\) follows \({\mathbf{s}}^{(1)}\) is thereby computed as:

\[\log p_{\theta}({\mathbf{s}}^{(2)}~{}\mathrm{follows}~{}{\mathbf{s}}^{(1)})\] \[=\log p_{\theta}({\mathbf{s}}^{(2)}|{\mathbf{s}}^{(1)})=\sum_{t=1}^{|{\bm {s}}^{(2)}|}\log p_{\theta}({\mathbf{s}}^{(2)}_{t}|{\mathbf{s}}^{(1)}\cdot{\mathbf{s}}^{(2 )}_{<t}), \tag{5}\]

where \({\mathbf{s}}^{(2)}_{<t}=[s_{1},\dots,s_{t-1}]\).

\(p({\mathbf{y}}|{\mathbf{x}})\) computed by sentence-level nmtComputing \(\log p({\mathbf{y}}|{\mathbf{x}})\) using an s-nmt is straightforward. Suppose \({\mathbf{y}}\) to be a sequence of raw tokens, \({\mathbf{y}}=[y_{1},\dots,y_{T}]\). Then \(\log p({\mathbf{y}}|{\mathbf{x}})\) is computed by

\[\log p({\mathbf{y}}|{\mathbf{x}})=\log p_{\textsc{ss-nmt}}(\tilde{{\mathbf{y}}};{\mathbf{x}}) \tag{6}\]

where \(\tilde{{\mathbf{y}}}=[y_{1},\dots,y_{T},\mbox{$</$\!\mathord{\circ}>$}]\) and \(\mbox{$</$\!\mathord{\circ}>$}\) is a special token to indicate the end of sentence 

Pmi computed by document-level lmTo compute the components of \(\textsc{pmi}(c^{(y)},\mathbf{y})\), \(p(\mathbf{y})\) and \(p(\mathbf{y}|c^{(\mathbf{y})})\), we use a document-level language model (d-lm) which can handle long text spans containing multiple sentences.

We generate training examples for d-lm from a document as follows. We assume d-lm explicitly models sentence boundaries. We first insert the special token 
 into every sentence boundary including the start and end of the document. With this preprocessing, all the sentences start immediately after an 
 token and end immediately before an 
 token. We then sample text spans from the document using a sliding window, where the start and end of the span do not have to match sentence boundaries. The sliding window's size is larger than the stride size, so adjacent spans may overlap. The resulting sequence is fed to the d-lm for training. Note that 
 for d-lm indicates sentence boundaries, in other words, both the start and end of the sequence.

Using d-lm, \(p(\mathbf{y})\) is computed by

\[p(\mathbf{y})=p_{\text{\sc D-lm}}(\tilde{\mathbf{y}}|\texttt{
}). \tag{7}\]

where \(\tilde{\mathbf{y}}=[y_{1},\ldots,y_{T},\texttt{
}]\).

To compute \(p(\mathbf{y}|c^{(\mathbf{y})})\), we first obtain the context sequence \(\tilde{\mathbf{c}}^{(\mathbf{y})}\) by concatenating all the sentences in \(\mathbf{c}^{(\mathbf{y})}\) with 
. We then compute the conditional probability \(p(\mathbf{y}|c^{(\mathbf{y})})\) by

\[p(\mathbf{y}|c^{(\mathbf{y})})=p_{\text{\sc D-lm}}(\tilde{\mathbf{y}}|\tilde{\mathbf{c}}^{(\bm {y})}) \tag{8}\]

where \(\tilde{\mathbf{y}}=[y_{1},\ldots,y_{T},\texttt{
}]\).

Let us explain why we use the boundary-aware d-lm rather than boundary-agnostic d-lm.1

Footnote 1: We cannot rely on punctuations to know sentence boundaries, since they can be omitted in some domains.

Firstly, boundary-agnostic lms cannot compute the probability that a sentence is closed with a certain length, namely, Eq. 7 cannot be computed. Secondly, they also cannot compute \(p(\mathbf{y}|c^{(\mathbf{y})})\) correctly. For example, suppose the context \(c^{(\mathbf{y})}\) is "he's my friend" (with the punctuation ''.'' omitted), and the current target sentence \(\mathbf{y}\) is "he's nice." In this case, Eq. 8 is computed by

\[p(\mathbf{y}|c^{(\mathbf{y})})=p_{\text{\sc D-lm}}([\text{he's,nice}]||[\text{he's,my,friend}]).\]

However, this estimation of \(p(\mathbf{y}|c^{(\mathbf{y})})\) can underestimate the actual \(p(\mathbf{y}|c^{(\mathbf{y})})\) because Eq. 8 inevitably gives significant probabilities to other \(\mathbf{y}\) such as "'s father" as well, since "He's my friend's father" is fluent as a sequence. This behavior is unsuitable for \(\mathbf{y}\),2 since "'s father" is not a complete sentence.

Footnote 2: Strictly speaking, we assume \(\mathbf{y}\) to be a realization of a random variable \(Y\) which is a sentence sampled from the space of an infinitely large document.

### Searching for the optimal solution

Searching for the optimal output \(\mathbf{y}\) that maximizes the c-score is not trivial since there are \(\mathcal{O}(V^{T})\) candidate sequences where \(V\) is the vocabulary size and \(T\) is the maximum length of sequences to be searched. We investigate two approaches to obtain approximate solutions: reranking (SS 2.2.1) and context-aware beam search (SS 2.2.2).

#### 2.2.1 Reranking with c-score

We first generate \(B\) hypotheses of the translation \(\mathcal{H}_{B}=\{\mathbf{y}^{1},\ldots,\mathbf{y}^{B}\}\) with beam search of beam size \(B\) using the sentence-level nmt model. We then choose the one that maximizes the c-score.

\[\hat{y}=\arg\max_{\mathbf{y}\in\mathcal{H}_{B}}\mbox{\sc C-score}(\mathbf{y};\mathbf{x}, \mathbf{c}^{(\mathbf{y})}) \tag{9}\]

An issue with reranking is that we need to set \(B\) to a large value when the diversity of models' outputs is limited Yu et al. (2020), which increases the cost of decoding. We therefore attempt to integrate c-score into the decoding with beam search.

#### 2.2.2 Context-aware beam search

Context-aware beam search (c-aware beam) is beam search that is extended to work with c-score. c-score (Eq. 3) can be decomposed into token-wise c-scores (Eq. 5 through Eq. 8).

\[\begin{split}\textrm{c-score}(\mathbf{y};\mathbf{x},\mathbf{c}^{(\mathbf{y})})& =\log p(\mathbf{y}|\mathbf{x})+\textsc{pmi}(\mathbf{c}^{(\mathbf{y})},\mathbf{y})\\ &=\sum_{t=1}^{T+1}\mbox{\sc c-score}_{w}(\tilde{y}_{t}|\tilde{\bm {y}}_{<t})\end{split} \tag{10}\]

where

\[\begin{split}\textrm{\sc-score}_{w}(\tilde{y}_{t}|\tilde{\mathbf{y}}_{< t})=&\log p_{\text{\sc S-nmt}}(\tilde{y}_{t}|\tilde{\mathbf{y}}_{<t};x)\\ &+\log\frac{p_{\text{\sc D-lm}}(\tilde{y}_{t}|\tilde{\mathbf{c}}^{( \mathbf{y})}\cdot\tilde{\mathbf{y}}_{<t})}{p_{\text{\sc D-lm}}(\tilde{y}_{t}|\texttt{</ s>}\cdot\tilde{\mathbf{y}}_{<t})}\end{split} \tag{11}\]

By this decomposition, c-score\({}_{w}\) is conditioned on the partial sequence generated by time step \(t\). We can therefore apply beam search to generate sequences in an auto-regressive manner.

The first term of Eq. 11 represents the translation probability for the \(t\)-th token. The second term can 

be interpreted as pmi between the \(t\)-th token and the context, that is, how consistent the \(t\)-th token is with the context. Compared to the reranking approach, c-aware beam can be considered to maximize the c-score more directly in the sense that disambiguation and token selection based on the context are performed at every step in beam search. Thus c-aware beam will more space-efficiently consider diverse hypotheses with the same beam size \(B\) than c-aware rerank.

#### 2.2.3 Smoothing probabilities for pmi

In our preliminary experiments, we observe that the original c-aware beam significantly improves contrastive tests but deteriorates bleu at the same time. By analyzing contextual pmi correlation between source and target texts, we find the pmi term in the c-score sometimes takes an excessively large value against the translation probability term, which destroys the c-score. This is understood intuitively by the fact that the calculation of pmi includes subtraction of log probability, and log probability may take a very small negative value to represent a probability close to zero.

To alleviate this problem, we adopt a smoothing method for probabilities. For simplicity, in this paper, we only present the temperature scaling (\(T\)-scaling, for short) Guo et al. (2017). \(T\)-scaling replaces \(p_{y=w}\) by

\[\bar{p}_{y=w}=\frac{p^{1/T}_{y=w}}{\sum_{w^{\prime}}p^{1/T}_{y=w^{\prime}}} \tag{12}\]

where \(T\) is a hyper-parameter. \(T=1\) is equivalent to no smoothing. We choose \(T\) from \([1,\infty)\) to flatten the probability distribution. \(T\)-scaling is applied to both the numerator and denominator using the same \(T\).

#### 2.2.4 On the relation to shallow fusion

Shallow fusion Gulcehre et al. (2015) is a method to integrate probability distribution outputs obtained by nmt and lm at sentence level to form a new translation objective that is expected to promote fluency of translations. The original shallow fusion score is computed using a sentence-level nmt (s-nmt) and language model (s-lm). The token-wise formula of the computation is

\[\log p(y_{t})=\log p_{\text{s-nmt}}(y_{t};\mathbf{x})+\beta\log p_{\text{s-lm}}( y_{t}), \tag{13}\]

where \(\beta\) is a hyper-parameter. In our notation with the document-level lm, this is written as

\[\log p(y_{t})=\log p_{\text{s-nmt}}(\tilde{y}_{t}|\tilde{\mathbf{y} }_{<t};\mathbf{x})\\ +\beta\log p_{\text{s-LM}}(\tilde{y}_{t}|\nicefrac{{<}}{{\mathrm{ s>}}}\cdot\tilde{\mathbf{y}}_{<t}). \tag{14}\]

A natural extension of this objective to the context-aware scenario should be

\[p(y_{t}|\mathbf{c^{(y)}})=\log p_{\text{nmt}}(\tilde{y}_{t}|\tilde{ \mathbf{y}}_{<t};x)\\ +\beta\log p_{\text{D-LM}}(\tilde{y}_{t}|\tilde{\mathbf{c}^{(y)}} \cdot\tilde{\mathbf{y}}_{<t}), \tag{15}\]

where context \(\tilde{c}^{(\mathbf{y})}\) is integrated into the condition. We call this _conditional (document-level) shallow fusion_. Obviously, this is what we obtain from Eq. 11 by ignoring the discount of the unconditional lm probability \(p_{\text{D-lm}}(\tilde{y}_{t}|\nicefrac{{<}}{{\mathrm{s>}}}\cdot\tilde{y}_{< t})\).

Due to the absence of discounting with the unconditional lm, conditional shallow fusion would prefer tokens which frequently occur regardless of the context. It is also worth noting that, when the context is empty, conditional shallow fusion falls back to the original shallow fusion, whereas our c-score falls back to sentence-level nmt. Therefore, we view c-score as a reformulation of shallow fusion for context-aware translation.

## 3 Experimental Setup

We evaluate our methods on English to Russian translation, in terms of bleu scores Papineni et al. (2002) and contrastive tests Voita et al. (2019).

### Datasets and preprocessing

We use the OpenSubtitles2018 corpus Lison et al. (2018) for parallel and monolingual data. Following the criteria for document segmentation and filtering on sentence pairs presented by Voita et al. (2019), we build monolingual and parallel data as follows. To build monolingual data, we add document boundary information into each document such that they consist of contiguous subtitle sentences from the same movie and the timestamp difference of any two adjacent sentences is no more than seven seconds. To build parallel data, we pick subtitle pairs where the time overlap between the source and target language subtitles is at least 0.9 (to reduce alignment errors). For the training of multi-encoder nmt models, document boundary information is added to the parallel data based on the source-side timestamps as with the monolingual data. Prior to building the Russian data, we 

remove the movies from which the contrastive test sets (SS 3.4) were made.

We perform punctuation normalization, tokenization, and truecasing on the source and target texts using Moses toolkit v4.0.3 We then encode the texts into subwords using SentencePiece (v0.1.81)4 with unigram lm. The subword vocabularies are of 16,000 tokens and trained for each language. The statistics of the datasets are listed in Table 1.

Footnote 3: [http://www.statmt.org/moses/](http://www.statmt.org/moses/)

Footnote 4: [https://github.com/google/sentenceepiece](https://github.com/google/sentenceepiece)

### Models

We compare our methods to one sentence-level translation model (**SentTransformer**) Vaswani et al. (2017) and three context-aware translation models: Document transformer Zhang et al. (2018), DocRepair Voita et al. (2019), and Bayes Document Reranker Yu et al. (2020). All the context-aware models use the previous three sentences as context.

**Document Transformer** (**DocTransformer**, for short) is a multi-encoder document-level nmt model which takes source-side context as an auxiliary input and can be thus trained from document-level parallel data. We follow Zhang et al. (2018)'s configuration for DocTransformer.

**DocRepair** is a sequence-to-sequence post-editing model. It repairs document-level inconsistencies in a text, each sentence of which has been translated separately by a sentence-level nmt model. DocRepair is trained on a pseudo parallel data made by pairing a monolingual corpus and its round-trip translations obtained using a back-translation model and a forward-translation model.

**Bayes Document Reranker** (hereafter, **Bayes DocReranker**) performs document-level translation on a document containing \(D\) sentences in the following steps. First, it produces \(B\)-best translations for each sentence in the document and then produces a lattice of width \(B\) and depth \(D\), where each node corresponds to a candidate sentence. It then performs document-level beam search of beam size \(B^{\prime}\) on the lattice using the following score:

\[\begin{split}&\mathrm{Score}(\mathbf{y}_{i};\mathbf{y}_{<i},\mathbf{x}_{ i})=\\ & p_{\text{\tiny D-LM}}(\mathbf{y}_{i}|\mathbf{y}_{<i})+\mathrm{Score}( \mathbf{y}_{i-1};\mathbf{y}_{<i-1},\mathbf{x}_{i-1})\\ &+\lambda_{1}p_{\textsc{NMT}}(\mathbf{y}_{i}|\mathbf{x}_{i})+\lambda_{2} p_{\textsc{BACK-nmt}}(\mathbf{x}_{i}|\mathbf{y}_{i})+\lambda_{3}|\mathbf{y}_{i}| \end{split} \tag{16}\]

Note that this document-level beam search is equivalent to the reranking procedure (SS 2.2.1) when \(B^{\prime}=1\). Therefore, the essential difference between Bayes DocReranker and our c-score reranking is the score function.

SentTransformer, the post-editing model of DocRepair, and the back-translation models are based on the same configuration of Transformer base (see Vaswani et al. (2017) for hyperparameter settings). The SentTransformer is trained using the 5.8M sentence pairs and is also used as the sentence-level nmt model in DocRepair, Bayes DocReranker, and our methods. For the training of DocTransformer, we use the 5.8M sentence pairs with document-level source context, which share the target-side sentences with the training data of SentTransformer. Consequently, scores obtained from the model are for reference.5 We also evaluate DocTransformer and SentTransformer using back-translation (bt) Sennrich et al. (2016) with the same monolingual data as the other models.

Footnote 5: Although we can train DocTransformer only on pseudo document-level parallel data generated by back-translation, we confirmed in preliminary experiments that the resulting model exhibited poor performance.

We use no pre-existing document-level parallel data to train the neural networks of DocRepair, Bayes DocReranker, and our methods, although we use a small amount of document-level parallel data as the development set to tune hyperparameters in the methods that combine multiple models. Instead, document-level information is fed to the models via the round-trip augmented data (DocRepair) or language models (Bayes DocReranker and our methods).

Hyper-parametersWe tune the models' hyper-parameters based on bleu score on the development set in the evaluation with bleu, while we tune these hyper-parameters in the evaluation of contrastive tests by maximizing the coefficient of d-lm under the constraint that it does not deteriorate BLEU compared to the SentTransformer.

For beam search to produce \(B\)-best outputs in Bayes DocReranker and our c-aware Rerank, we

\begin{table}
\begin{tabular}{l r l r r r r} \hline \hline  & \multicolumn{2}{c}{**Train**} & \multicolumn{2}{c}{**Dev.**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-7}  & src & trg & (mono) & src & trig & (mono) & src & trig \\ \hline \# sentences & 5.8M & 30M & 6.0k & 23k & 15.5k \\ avg. \# tokens & 9.9 & 9.4 & 8.5 & 10.1 & 9.6 & 8.9 & 9.8 & 9.1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the parallel and monolingual data.

 

use a beam size of \(B=20\). For document-level beam search of Bayes DocReranker, we use a beam size \(B^{\prime}=5\). For beam search of SentTransformer, DocTransformer, c-aware beam, and shallow fusion, we use a beam size of \(B=4\).

### Document-level Language models

The architecture of the document-level lm is the decoder part of a Transformer. The number of decoder blocks is \(12\). The model size is \(768\) with \(12\) attention heads, and the inner layer of the feed-forward networks has \(3072\) units. We use position embeddings to represent position information.

As described in SS 2.1, when training the language models, a special control symbol 
 is inserted at every sentence boundary. Each training mini-batch contains text spans each of which is a randomly sampled fragment of a document with a maximum span length of \(W=384\). Text spans are batched such that about 32,000 tokens are in a training batch.

### Evaluation methods

The existing automatic metrics are not adequate to evaluate gains from additional contexts Bawden et al. (2018); Laubli et al. (2018); Muller et al. (2018); Voita et al. (2019b); Sugiyama and Yoshinaga (2019). We thus adopt a contrastive test set Voita et al. (2019b) to evaluate the model's ability to capture contextual information in translation, in addition to the evaluation by bleu scores Papineni et al. (2002) to confirm that the methods do not sacrifice general translation performance. bleu is computed using multi-bleu.perl from the Moses Toolkit after decoding the subword representation of the models' outputs into words using SentencePiece.

The contrastive test set consists of contrastive questions for context-aware nmt models to answer. Each question has a source sentence \(\mathbf{x}\), a source context \(\mathbf{c}^{(\mathbf{x})}\), a target context \(\mathbf{c}^{(\mathbf{y})}\), and translation candidates \(\mathcal{Y}=\{y^{1},\ldots,y^{M}\}\). Models must answer with a candidate \(\hat{y}\in\mathcal{Y}\) which would be the most appropriate translation of \(\mathbf{x}\), i.e.

\[\hat{y}=\arg\max_{y\in\mathcal{Y}}p(\mathbf{y}|\mathbf{x},\mathbf{c}^{(x)},\mathbf{c}^{(\mathbf{ y})})\]

The test sets consist of 6000 examples in total.

## 4 Results and Analysis

### General translation performance measured by bleu scores

Table 2 lists the performance of the models in terms of bleu scores. Bayes DocReranker and our c-aware Rerank consistently outperformed the baseline SentTransformer, even when it used data augmentation by back-translation, while the other methods are just comparable to the baseline. Although Bayes DocReranker performed the best among all the models, the comparison to Bayes DocReranker without context information (using \(p_{\text{S-LM}}(\mathbf{y}_{i})\) instead of \(p_{\text{D-LM}}(\mathbf{y}_{i}|\mathbf{y}_{<i})\)) reveals that most of the improvement is not obtained by the use of contexts. Back-translation did not contribute to bleu possibly because the original parallel data is already large and there was little room for improvement with additional pseudo data.

### Results on contrastive test sets

Tables 3 lists evaluation results (accuracy) of the contrastive tests with models using 30M monolingual data. The highest scores on each column

\begin{table}
\begin{tabular}{l l r r r} \hline \hline
**Models** & \multicolumn{1}{r}{**para**} & \multicolumn{3}{c}{**monolingual data**} \\  & **only** & 6M & 15M & 30M \\ \hline SentTransformer (w brt) & 32.36 & 32.32 & 32.40 & 32.40 \\ Shallow Fusion & n/a & 32.39 & 32.56 & 32.52 \\ \hline \hline \multicolumn{4}{l}{_baselines_} \\ DocTransformer (w brt) & 32.50 & 32.36 & 31.88 & 31.59 \\ DocRepair & n/a & 32.13 & 32.36 & 32.35 \\ Bayes DocReranker & n/a & **32.80** & **33.58*** & **33.75*** \\ \multicolumn{4}{l}{w/o context} \\ \multicolumn{4}{l}{_proposed_} \\ \multicolumn{4}{l}{c-aware Rerank} & n/a & 32.74* & 33.01** & 32.93* \\ c-aware Beam & n/a & 32.26 & 32.28 & 32.27 \\ Cond. Shallow Fusion & n/a & 32.38 & 32.55 & 32.55 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Test set bleu scores. ‘*’ and ‘**’ indicate that gains from SentTransformer in the same column are statistically significant (\(p<0.05\) and \(p<0.01\)) by bootstrap resampling with 1000 samples, respectively.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Models** & \multicolumn{1}{r}{**deixis**} & \multicolumn{1}{r}{**lex.ć ell.infilf**} & \multicolumn{1}{r}{**ell.vp**} \\ \hline SentTransformer & 50.0 & 45.9 & 53.2 & 27.0 \\   w/bt & 50.0 & 45.9 & 51.6 & 26.8 \\ \hline \hline \multicolumn{4}{l}{_baselines_} \\ DocTransformer & 50.0 & 45.9 & 56.0 & 57.2 \\   w/bt & 50.0 & 45.9 & 64.4 & 68.2 \\ DocRepair & **89.1** & 75.8 & **82.2** & 67.2 \\ Bayes DocReranker & 65.2 & 72.2 & 59.6 & 44.6 \\ \multicolumn{4}{l}{_proposed_} \\ \multicolumn{4}{l}{c-score} & 86.9 & **94.9** & 78.2 & **77.0** \\ Cond. Shallow Fusion & 54.7 & 55.3 & 53.4 & 32.4 \\ \hline d-lm & pm\((\mathbf{c}^{(\mathbf{y})},y)\) & **96.8** & **97.8** & 75.8 & **90.6** \\  & \(p(\mathbf{y}|\mathbf{c}^{(\mathbf{y})})\) & 89.7 & 95.7 & **77.4** & 81.6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on contrastive test sets.

 

are in bold, and additionally, the higher one of the two d-lm-based scores is shown in bold. The contrastive test include four test sets: _deixis_ is for person deixis, _lex.c_ is for lexical cohesion, _ell.infl_ is for inflection of Russian nouns caused by ellipsis in the source sentence, and _ell.vp_ is for verb ellipsis in English text which is not allowed in Russian. Although the contrastive test is targeted at context-aware nmt models, it is possible to answer the contrastive questions by \(\arg\max_{y}\text{pmi}(e^{(\mathbf{y})},y)\) or \(\arg\max_{y}p(\mathbf{y}|\mathbf{c^{(\mathbf{y})}})\). Scores obtained by these two objectives are also reported in the table in addition to the scores obtained by SentTransformer.

Our c-score outperforms all the context-aware models other than DocRepair. The performance of c-score is slightly worse than DocRepair for _deixis_ (2.2 points) and _ell.infl_ (4.0 points), while achieving large improvements for _lex.c_ (19.1 points) and _ell.vp_ (9.8 points) over DocRepair.

d-lm _only_objectives achieve higher scores than c-score, except for _ell.infl_. This is not surprising because the choices in the tests are guaranteed to be valid as translation for the source sentences if given some appropriate context, so the questions can be solved without translation. This result still indicates that the d-lm scores give good hints for tackling contextual ambiguities. The advantage of c-score over the SentTransformer is demonstrated by the excellent performance of d-lm in capturing contexts in translation.

### On translation efficiency

The inference speed depends mainly on the model size and beam size. In our experiments on a single TITAN Xp GPU, SentTransformer decoded the fastest at 66 sents/sec, followed by DocTransformer that ran in 40 sents/sec. DocRepair ran in about 28 sents/sec, slightly slower because it decodes in two passes. c-aware Rerank and Bayes DocReranker were about 4.3 sents/sec and 7.7 sents/sec respectively. We expect that these models would be accelerated by using a language model with a better cache mechanism (e.g. TransformerXL Dai et al. (2019)). c-aware Beam ran in about 13 sents/sec.6 We leave thorough analysis on speed/performance trade-offs to future work.

Footnote 6: Note that the running time of nmtt decoding also depends on the degree of parallelism, and for c-aware Beam, decoding multiple sentences in parallel is less trivial since it demands that all the previous sentences in the document are translated by the time it starts to translate the current one. In our experiments, assuming a practical scenario where a large number of users input their documents for translation, we translate multiple documents in parallel so that multiple sentences from different documents can be translated in parallel.

### pmi correlation analysis

In SS 4.2 we have confirmed the effectiveness of pmi as a measure of a valid translation given context using contrastive tests. To gain a deeper insight into how well pmi conveys semantic connections between the current sentence and its context, we analyze the correlation of pmi between source and target sentences.

**pmi correlation between source and target**

The main result we show in this section is that the pmi of the source and target correlate well. This is important because this supports the idea that pmi is a language-independent measure of the connection between the current sentence and its context.

Although we have discussed only target-side pmi\((e^{(\mathbf{y})},\mathbf{y})\) defined by Eq. 4, we can compute the source-side pmi\((e^{(\mathbf{x})},\mathbf{x})\) in the same way. Given a document-level parallel corpus, we measure a correlation between pmi\((e^{(\mathbf{x})},\mathbf{x})\) and pmi\((e^{(\mathbf{y})},\mathbf{y})\) for each sentence pair \((\mathbf{x},\mathbf{y})\) in the corpus.

Figure 0(a) shows the pmi correlation for about

Figure 1: Source-target correlation of contextual pmi (a, b) and conditional probability (c, d), calculated based on the correct context (a, c) and wrong context that is randomly chosen from the dataset (b, d). The dataset is a subset of the training data from the English-Russian parallel corpus. Plots are for \(4166\) sentence pairs in the dataset.

 

\(4000\) sentence pairs taken from the dev data. The pairs of pmi values are computed using English and Russian language models trained on the training data. We observe a clear correlation between source and target, which agrees with the intuition that if the target sentence matches well in the context, so does the source sentence. What is also obvious in Figure 1a is that most of the points lay in the first quadrant where both the source and target contextual pmi is greater than \(0\), which is explained by the simple intuition that most sentences should have positive co-occurrence relation with their contexts. This behavior is lost when computing the contextual pmi using an incorrect context \(\tilde{c}\) randomly chosen in the dataset as shown in Figure 1b.

The effectiveness of pmi as a measure of the valid translation of the current sentence given context is further emphasized when compared to the conditional probability \(p(\mathbf{y}|\mathbf{c^{(\mathbf{y})}})\), which could be an alternative measure of how suitable \(\mathbf{y}\) is in the context as described in SS 2.2.4. Figure 1c and 1d are the conditional probability version of Figure 1a and 1b: \((p(\mathbf{x}|\mathbf{c^{(\mathbf{x})}}),p(\mathbf{y}|\mathbf{c^{(\mathbf{y})}}))\) for each sentence pair \((x,y)\) in the same dataset are plotted in Figure 1c and the same tuples but with random contexts are plotted in Figure 1d. Unlike the contextual pmi correlation, conditional probability correlation remains high even when we give wrong contexts. This is because the conditional probability of a sentence is highly affected by how frequently the sentence is observed regardless of context; if the source sentence is written with common expressions, then so is the target sentence and they are likely to be observed regardless of the context.

### Analysis of the model outputs

pmi correlation gives us a good explanation of how c-aware beam without \(T\)-scaling fails. We plot the pmi correlation between the source sentences and their translations obtained with nmt models (Figure 2). We can find some outliers in the bottom right area of the plot for c-aware beam without \(T\)-scaling, which is the cause of the low correlation coefficient \(R=0.610<R_{\text{src-ref}}=0.695\). This result suggests that c-aware beam without \(T\)-scaling chooses some tokens based on excessively high token-wise pmi, which breaks some translations resulting in the low bleu. Translation of the SentTransformer shows a higher correlation with the source texts than the reference translation (Figure 1a). One possible explanation for this is alignment errors in the corpus: although worse than the reference translations in quality, outputs of SentTransformer are considered to be perfectly aligned to the source sentences. c-aware beam with \(T\)-scaling (\(T=4\)) seems to solve this issue and achieves the highest pmi correlation \(R=0.740\).

## 5 Related Work

The effectiveness of incorporating context into translation was shown in earlier literature on document-level nmt[38], swapden2018using the single encoder architecture. Multi-encoder architectures were explored to better capture contextual information [26, 27, 28, 29, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. However, since parallel data is often constructed by picking up reliable sentential alignments from comparable documents, document-level sentence-aligned parallel data for training these document-level nmt models are expensive to obtain and available in only a few domains and language pairs [20].

Recent studies have therefore started to focus on modeling contexts using document-level monolingual data. The current approaches are grouped into three categories: data augmentation via back

Figure 2: Correlation of contextual pmi between the source sentences (from the training data) and the outputs of some models (SentTransformer, c-aware beam without \(T\)-scaling, and c-aware beam with \(T\)-scaling of \(T=4\)).

 

translation (Sugiyama and Yoshinaga, 2019), a post-editing model (Voita et al., 2019), and modeling document-level fluency via document-level lms(Stahlberg et al., 2019; Yu et al., 2020; Jean and Cho, 2020). In what follows, we review these approaches in detail.

Sugiyama and Yoshinaga (2019) reported that the data augmentation by back-translation (Sennrich et al., 2016) enhances a document-level nmt model with a single encoder architecture in low-resource settings. However, we have obtained limited improvements in our settings (Table 2 and Table 3). Moreover, this approach is expensive since it learns a document-level nmt model from a massive amount of pseudo parallel data.

Voita et al. (2019) proposed DocRepair, a context-aware post-editing model that corrects outputs of a sentence-level nmt model. Because DocRepair ignores the confidence of the first-stage sentence-level translation and possible alternative translations, it can miscorrect outputs of the sentence-level nmt model when they are irregular but correct. Moreover, when we change the target sentence-level nmt model, the accompanying post-editing model must be trained from its outputs. Our approaches, on the other hand, attempt a more "soft" revision, taking into account the output probabilities, i.e., confidence of the sentence-level nmt, and can perform context-aware decoding with any sentence-level nmt model, reusing a pre-trained document-level lm.

Stahlberg et al. (2019) and Yu et al. (2020) utilize a document-level lm to model document-level fluency of outputs; these approaches are similar to shallow fusion (Gulcehre et al., 2015)7 with document-level lm (SS 2.2.4), although they perform a document-level reranking of translation hypotheses generated for individual source sentences by using sentence-level nmt. In particular, Yu's formulation has a probabilistic foundation like our approaches, and additionally utilizes a backward translation model. Although their formulation brings a significant improvement in bleu (Table 2), the score is not obtained by better document-level translation; the comparable bleu score of the no-context version of the method (Table 2) and the results of the contrastive tests (Table 3) reveal that the improvement is mostly due to the context-agnostic language model prior and the backward translation model. As we have discussed in SS 2.2.4, document-level lm scores prefer tokens which frequently appear regardless of context and are unlikely to lead to better document-level translation. Moreover, their method requires training a back-translation model corresponding to the target sentence-level nmt model.

Footnote 7: Our work is also related to shallow fusion (Gulcehre et al., 2015), in which token-wise probabilities output by an nmt model and a sentence-level lm are combined to be used as translation scores in decoding. The theoretical background of shallow fusion and our c-score are different: in shallow fusion, the lm is intended to promote fluency of translations, whereas in our c-score, we use the probability ratio of two lm probabilities which only provides contextual difference and fluency is still left to the translation model.

Finally, we noticed that Jean and Cho (2020) (which appeared after the preprint version of this paper (Sugiyama and Yoshinaga, 2020)8 had been submitted) have reached a formulation that is very similar to the one presented in this paper by reformulating a noisy channel model of Bayes DocRepranker (Yu et al., 2020). Concrete differences between our work and theirs include the fact that we conducted thorough analysis on the performance of different decoding strategies (not only beam search but also reranking). We also interpreted the subtraction of lm scores as point-wise mutual information and analyzed it by observing pmi correlation between source and target pmi to deepen the understanding of the formulation.

Footnote 8: This preprint is submitted to and rejected from EMNLP 2020; the interested reader may refer to this paper for experiments on other language pairs such as English to French and English to Japanese translation.

## 6 Conclusions

We present an approach to context-aware nmt based on pmi between the context and the current sentence. We first provide the formulation of the objective, c-score, and the computation process of the c-score using a sentence-level translation model and a document-level language model. We investigate two search methods, reranking and beam search, and evaluate the methods for English-Russian translation. We also provide some analysis and visualization to better understand the nature of pmi between the context and the current sentence.

We plan to design context-aware bleu using pmi for evaluating context-aware nmt models. We will evaluate our method on non-autoregressive nmt(Gu et al., 2017). We will release all code and data to promote the reproducibility of results.9 



## Acknowledgements

We thank anonymous reviewers for their valuable comments. We also thank Joshua Tanner for proofreading this paper. We also thank Masato Neishi for technical advice on implementations of neural machine translation. The research was supported by NII CRIS collaborative research program operated by NIH CRIS and LINE Corporation.

## References

* D. Bahdanau, K. Cho, and Y. Bengio (2015)Neural machine translation by jointly learning to align and translate. In Proceedings of the third International Conference on Learning Representations (ICLR), Cited by: SS3.4.
* R. Bawden, R. Sennrich, A. Birch, and B. Haddow (2018)Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1304-1313. External Links: Link, Document Cited by: SS1, SS2.
* Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov (2019)Transformer-XL: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 2978-2988. External Links: Link, Document Cited by: SS1, SS2.
* J. Gu, J. Bradbury, C. Xiong, V. O.K. Li, and R. Socher (2017)Non-autoregressive neural machine translation. In Proceedings of the fifth International Conference for Learning Representations (ICLR), Cited by: SS1, SS2.
* C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio (2015)On using monolingual corpora in neural machine translation. Computing Research RepositoryarXiv:1503.03535. External Links: Link, 1503.03535 Cited by: SS1, SS2.
* C. Guo, G. Pleiss, Yu Sun, and K. Q. Weinberger (2017)On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70, pp. 1321-1330. Note: PMLR External Links: ISBN 978-3-642-431X1133131 Cited by: SS1, SS2.
* S. Jean and K. Cho (2020)Loglinear reformulation of the noisy channel model for document-level neural machine translation. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, Online, Cited by: SS1, SS2.
* S. Jean, S. Lauly, O. Firat, and K. Cho (2017)Does neural machine translation benefit from larger context?. arXiv preprint arXiv:1704.05135. Cited by: SS1, SS2.
* X. Kang, Y. Zhao, J. Zhang, and C. Zong (2020)Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2242-2254. External Links: Link, Document Cited by: SS1, SS2.
* Samuel Laubli, R. Sennrich, and M. Volk (2018)Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 4791-4796. External Links: Link, Document Cited by: SS1, SS2.
* Pierre Lison, J. Tiedemann, and M. Kouylekov (2018)OpenSubtitles2018: statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, European Language Resources Association (ELRA), Cited by: SS1, SS2.
* Sameen Maruf and G. Haffari (2018)Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 1275-1284. External Links: Link, Document Cited by: SS1, SS2.
* S. Maruf, A. F. T. Martins, and G. Haffari (2019)Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, USA, pp. 3092-3102. External Links: Link, Document Cited by: SS1, SS2.
* Lesly Michelich, D. Ram, N. Pappas, and J. Henderson (2018)Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 2947-2954. External Links: Link, Document Cited by: SS1, SS2.
* Mathias Muller, A. Rios, E. Voita, and R. Sennrich (2018)A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium, pp. 61-72. External Links: Link, Document Cited by: SS1, SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: Link, Document Cited by: SS1, SS2.
* Rico Sennrich, B. Haddow, and A. Birch (2016)Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 86-96. External Links: Link, Document Cited by: SS1, SS2.



* Stahlberg et al. (2019) Felix Stahlberg, Danielle Saunders, Adria de Gispert, and Bill Byrne. 2019. CUED@WMT19:EWc&LMs. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 364-373, Florence, Italy.
* Sugiyama and Yoshinaga (2019) Amane Sugiyama and Naoki Yoshinaga. 2019. Data augmentation using back-translation for context-aware neural machine translation. In _Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)_, pages 35-44, Hong Kong, China.
* Sugiyama and Yoshinaga (2020) Amane Sugiyama and Naoki Yoshinaga. 2020. Context-aware decoder for neural machine translation using a target-side document-level language model. _Computing Research Repository_, arXiv:2010.12827.
* Tiedemann and Scherrer (2017) Jorg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In _Proceedings of the Third Workshop on Discourse in Machine Translation_, pages 82-92, Copenhagen, Denmark.
* Tu et al. (2018) Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2018. Learning to remember translation history with a continuous cache. _Transactions of the Association for Computational Linguistics_, 6:407-420.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30, page 6000-6010. Curran Associates, Inc.
* Voita et al. (2019a) Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 877-886, Hong Kong, China.
* Voita et al. (2019b) Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1198-1212, Florence, Italy.
* Voita et al. (2018) Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1264-1274, Melbourne, Australia.
* Wang et al. (2017) Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2826-2831, Copenhagen, Denmark.
* Yu et al. (2020) Lei Yu, Laurent Sarfran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, and Chris Dyer. 2020. Better document-level machine translation with Bayes' rule. _Transactions of the Association for Computational Linguistics_, 8:346-360.
* Zhang et al. (2018) Jiacheng Zhang, Huambo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 533-542, Brussels, Belgium.
* Zhang et al. (2020) Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1081-1087, Online.