SlowGuess/NAACL_2021_OCR huggingface dataset:
Amane Sugiyama
The University of Tokyo*
sugi@tkl.iis.u-tokyo.ac.jp
Naoki Yoshinaga
Institute of Industrial Science,
ynaga@iis.u-tokyo.ac.jp
Although many end-to-end context-aware neural machine translation models have been proposed to incorporate inter-sentential contexts in translation, these models can be trained only in domains where parallel documents with sentential alignments exist. We therefore present a simple method to perform context-aware decoding with any pre-trained sentence-level translation model by using a document-level language model. Our context-aware decoder is built upon sentence-level parallel data and target-side document-level monolingual data. From a theoretical viewpoint, our core contribution is the novel representation of contextual information using point-wise mutual information between context and the current sentence. We demonstrate the effectiveness of our method on English to Russian translation, by evaluating with BLEU and contrastive tests for context-aware translation.
Neural machine translation (NMT) has typically been explored in sentence-level translation settings. Such sentence-level NMT models inevitably suffer from ambiguities when a source sentence has multiple plausible interpretations. Examples of such ambiguities include anaphora, ellipsis, and lexical coherence (Voita et al., 2019b); although resolving these ambiguities has only a minor impact on the translation performance measured by BLEU scores (Papineni et al., 2002), they are vital in smoothly reading the translated documents.
To address this issue, context-aware NMT models which incorporate document-level information in translation have recently been explored (Jean et al., 2017; Wang et al., 2017; Tiedemann and Scherrer, 2017; Maruf and Haffari, 2018; Voita et al., 2018; Bawden et al., 2018; Miculicich et al., 2018; Maruf et al., 2019; Voita et al., 2019b; Yu et al., 2020;
Kang et al., 2020; Zhang et al., 2020). Most of these models are end-to-end models that require document-level parallel data with sentential alignments for training. However, this data is available in only a few domains (Sugiyama and Yoshinaga, 2019). Researchers have therefore started to utilize target-side monolingual data to construct auxiliary models which help a sentence-level NMT model perform context-aware translation (Voita et al., 2019a; Stahlberg et al., 2019; Yu et al., 2020).
In this study, we propose a simple yet effective approach to context-aware NMT using two primitive components, a sentence-level NMT model and a document-level language model (LM). We can independently train the two components on common sentence-level parallel data and document-level monolingual data, respectively, without using document-level parallel data. Our approach thereby makes it possible to perform context-aware translation with any pre-trained sentence-level NMT model, using a pre-trained document-level LM.
To give a probabilistic foundation to this combination of two independent models, we exploit the probabilistic nature of NMT decoding. When generating a sequence, a left-to-right decoder outputs a categorical probability distribution over the vocabulary at every time step. The decoder assigns higher probabilities to the tokens that would be more suitable at that step. Therefore, when multiple valid translations are possible for the source sentence, the decoder just gives a higher probability to the translation that is plausible without considering contexts. We thus adjust the probability distributions in a context-aware manner using a target-side document-level LM which models inter-sentential dependencies in the target-side document.
We evaluate our methods on English to Russian translations with the OpenSubtitles2018 corpus (Lison et al., 2018) in terms of the BLEU scores and contrastive discourse test sets (Voita et al., 2019b). Experimental results confirm that
*Currently at Mitsubishi UFJ Morgan Stanley Securities
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5781-5791 June 6-11, 2021. ©2021 Association for Computational Linguistics
our method achieved comparable performance with existing context-aware NMT models that require either document-level parallel data (Zhang et al., 2018; Sugiyama and Yoshinaga, 2019) or more than one additional model (Voita et al., 2019a; Yu et al., 2020) for capturing contexts in translation.
- We theoretically derived C-SCORE, a score to qualify context-aware translation without the need for document-level parallel data.
- Two formulations with C-SCORE turn any pre-trained sentence-level NMT model into a context-aware model, if it generates  $n$ -best outputs or performs left-to-right decoding.
- A comparison between our approach and shallow fusion (Gulcehre et al., 2015) reveals that our approach reformulates shallow fusion while adding a probabilistic foundation.
2 Context-aware Decoding using Document-level Language Model
In this section, assuming a sentence-level encoder-decoder model (Bahdanau et al., 2015; Vaswani et al., 2017), we first derive context-aware score (C-SCORE for short), a context-aware objective function of outputs to be maximized in decoding. We then describe how to compute the C-SCORE using the decoder with a document-level language model (D-LM) (§ 2.1). We finally detail how to perform context-aware decoding based on C-SCORE (§ 2.2).
2.1 C-SCORE: objective function for context-aware NMT decoding
Let us consider the problem of finding a translation  $\pmb{y}$  of a source sentence  $\pmb{x}$  in a document. The target-side context sentence(s) preceding  $\pmb{y}$ ,  $\pmb{c}^{(y)}$ , are to be given by the past translations. We formulate context-aware translation conditioned on  $\pmb{c}^{(y)}$  as the maximization of the conditional probability  $p(\pmb{y}|\pmb{x},\pmb{c}^{(y)})$ ,
$$
\begin{array}{l} \hat {\boldsymbol {y}} = \arg \max _ {\boldsymbol {y}} \log p (\boldsymbol {y} | \boldsymbol {x}, \boldsymbol {c} ^ {(y)}) \\ = \arg \max _ {\boldsymbol {y}} \log \frac {p (\boldsymbol {c} ^ {(\boldsymbol {y})} | \boldsymbol {x} , \boldsymbol {y}) p (\boldsymbol {y} | \boldsymbol {x})}{p (\boldsymbol {c} ^ {(\boldsymbol {y})} | \boldsymbol {x})} \\ = \arg \max  _ {\boldsymbol {y}} \log p (\boldsymbol {c} ^ {(\boldsymbol {y})} | \boldsymbol {x}, \boldsymbol {y}) p (\boldsymbol {y} | \boldsymbol {x}). \quad (1) \\ \end{array}
$$
Assuming that  $\pmb{x}$  and  $\pmb{y}$  are semantically similar, we make the following approximation,
$$
p (\boldsymbol {c} ^ {(y)} | \boldsymbol {y}, \boldsymbol {x}) \approx p (\boldsymbol {c} ^ {(y)} | \boldsymbol {y}). \tag {2}
$$
From Eq. 1 and Eq. 2, we obtain
$$
\begin{array}{l} \hat {\pmb {y}} \approx \underset {\pmb {y}} {\arg \max} \log p (\pmb {c} ^ {(\pmb {y})} | \pmb {y}) p (\pmb {y} | \pmb {x}) \\ = \arg \max  _ {y} \log \frac {p (\boldsymbol {c} ^ {(\boldsymbol {y})} , \boldsymbol {y})}{p (\boldsymbol {c} ^ {(\boldsymbol {y})}) p (\boldsymbol {y})} p (\boldsymbol {y} | \boldsymbol {x}) \\ = \arg \max  _ {\boldsymbol {y}} \operatorname {C - S C O R E} (\boldsymbol {y}; \boldsymbol {x}, \boldsymbol {c} ^ {(y)}) \\ \end{array}
$$
$$
\operatorname {C} - \operatorname {S C O R E} (\boldsymbol {y}; \boldsymbol {x}, \boldsymbol {c} ^ {(y)}) = \log p (\boldsymbol {y} | \boldsymbol {x}) + \operatorname {P M I} (\boldsymbol {c} ^ {(y)}, \boldsymbol {y}) \tag {3}
$$
$$
\operatorname {P M I} \left(\boldsymbol {c} ^ {(y)}, \boldsymbol {y}\right) = \log \frac {p \left(\boldsymbol {c} ^ {(y)} , \boldsymbol {y}\right)}{p \left(\boldsymbol {c} ^ {(y)}\right) p (\boldsymbol {y})} = \log \frac {p (\boldsymbol {y} \mid \boldsymbol {c} ^ {(y)})}{p (\boldsymbol {y})} \tag {4}
$$
$\mathrm{PMI}(\pmb {c}^{(y)},\pmb {y})$  is the point-wise mutual information of  $\pmb{c}^{(y)}$  and  $\pmb{y}$  which represents the degree of cooccurrence of  $\pmb{y}$  and  $c^{(y)}$  .Given  $\pmb{x},\pmb{y}$  and  $c^{(y)}$  , we can evaluate the C-SCORE by computing the two terms in Eq. 3 using a sentence-level NMT (S-NMT) and a document-level LM (D-LM), respectively.
Notations We first introduce some notation to explain the computation in Eq. 3 and Eq. 4 using (auto-regressive) neural sequence generation models in NMT and LM. For a sequence  $s$  ( $|s| \geq 0$ ) and token  $w$ , a neural sequence generation model parameterized by  $\theta$  can compute the log probability that  $w$  follows  $s$ , which we denote by  $\log p_{\theta}(w|s)$ :
$$
\log p _ {\theta} (w \mathrm {f o l o w s} \boldsymbol {s}) = \log \frac {p _ {\theta} (\boldsymbol {s} \cdot w)}{p _ {\theta} (\boldsymbol {s})} = \log p _ {\theta} (w | \boldsymbol {s})
$$
where “.” denotes sequence concatenation. Applying this auto-regressively, for any sequence  $s^{(1)}$  ( $|s^{(1)}| \geq 0$ ) and  $s^{(2)}$  ( $|s^{(2)}| \geq 1$ ), the probability that  $s^{(2)}$  follows  $s^{(1)}$  is thereby computed as:
$$
\begin{array}{l} \log p _ {\theta} \left(\boldsymbol {s} ^ {(2)} \text {f o l l o w s} \boldsymbol {s} ^ {(1)}\right) \\ = \log p _ {\theta} (\boldsymbol {s} ^ {(2)} | \boldsymbol {s} ^ {(1)}) = \sum_ {t = 1} ^ {| \boldsymbol {s} ^ {(2)} |} \log p _ {\theta} (\boldsymbol {s} _ {t} ^ {(2)} | \boldsymbol {s} ^ {(1)} \cdot \boldsymbol {s} _ {<   t} ^ {(2)}), \\ \end{array}
$$
$$
\text {w h e r e} \boldsymbol {s} _ {<   t} ^ {(2)} = \left[ s _ {1}, \dots , s _ {t - 1} \right]. \tag {5}
$$
$p(\boldsymbol{y}|\boldsymbol{x})$  computed by sentence-level NMT Computing  $\log p(\boldsymbol{y}|\boldsymbol{x})$  using an S-NMT is straightforward. Suppose  $\boldsymbol{y}$  to be a sequence of raw tokens,  $\boldsymbol{y} = [y_1,\dots,y_T]$ . Then  $\log p(\boldsymbol{y}|\boldsymbol{x})$  is computed by
$$
\log p (\boldsymbol {y} | \boldsymbol {x}) = \log p _ {\mathrm {S - N M T}} (\tilde {\boldsymbol {y}}; \boldsymbol {x}) \tag {6}
$$
where  $\tilde{\pmb{y}} = [y_1,\dots ,y_T, <   / s > ]$  and  $<  / s>$  is a special token to indicate the end of sentence.
PM computed by document-level LM To compute the components of  $\mathrm{PM}(\pmb {c}^{(y)},\pmb {y})$ $p(\pmb {y})$  and  $p(\pmb {y}|c^{(y)})$  , we use a document-level language model (D-LM) which can handle long text spans containing multiple sentences.
We generate training examples for D-LM from a document as follows. We assume D-LM explicitly models sentence boundaries. We first insert the special token  $\langle /s\rangle$  into every sentence boundary including the start and end of the document. With this preprocessing, all the sentences start immediately after an  $\langle /s\rangle$  token and end immediately before an  $\langle /s\rangle$  token. We then sample text spans from the document using a sliding window, where the start and end of the span do not have to match sentence boundaries. The sliding window's size is larger than the stride size, so adjacent spans may overlap. The resulting sequence is fed to the D-LM for training. Note that  $\langle /s\rangle$  for D-LM indicates sentence boundaries, in other words, both the start and end of the sequence.
Using D-LM,  $p(\pmb{y})$  is computed by
$$
p (\boldsymbol {y}) = p _ {\mathrm {D} - \mathrm {L M}} \left(\tilde {\boldsymbol {y}} | <   / s >\right). \tag {7}
$$
where  $\tilde{\pmb{y}} = [y_1,\dots ,y_T, <   / s > ]$
To compute  $p(\boldsymbol{y}|\boldsymbol{c}^{(\boldsymbol{y})})$ , we first obtain the context sequence  $\tilde{\boldsymbol{c}}^{(\boldsymbol{y})}$  by concatenating all the sentences in  $\boldsymbol{c}^{(\boldsymbol{y})}$  with  $< / s>$ . We then compute the conditional probability  $p(\boldsymbol{y}|\boldsymbol{c}^{(\boldsymbol{y})})$  by
$$
p (\boldsymbol {y} | \boldsymbol {c} ^ {(y)}) = p _ {\mathrm {D - L M}} (\tilde {\boldsymbol {y}} | \tilde {\boldsymbol {c}} ^ {(y)}) \tag {8}
$$
Let us explain why we use the boundary-aware D-LM rather than boundary-agnostic D-LM.
Firstly, boundary-agnostic LMs cannot compute the probability that a sentence is closed with a certain length, namely, Eq. 7 cannot be computed. Secondly, they also cannot compute  $p(\pmb{y}|\pmb{c}^{(y)})$  correctly. For example, suppose the context  $\pmb{c}^{(y)}$  is "he's my friend" (with the punctuation ". ." omitted), and the current target sentence  $\pmb{y}$  is "he's nice." In this case, Eq. 8 is computed by
$$
p (\boldsymbol {y} | \boldsymbol {c} ^ {(\boldsymbol {y})}) = p _ {\mathrm {D - L M}} ([ \mathrm {h e}, \mathrm {s}, \mathrm {n i c e} ] | [ \mathrm {h e}, \mathrm {s}, \mathrm {m y}, \mathrm {f r i e n d} ]).
$$
However, this estimation of  $p(\pmb{y}|\pmb{c}^{(\pmb{y})})$  can underestimate the actual  $p(\pmb{y}|\pmb{c}^{(\pmb{y})})$  because Eq. 8 inevitably gives significant probabilities to other  $\pmb{y}$  such as "s father" as well, since "He's my friend's father" is
fluent as a sequence. This behavior is unsuitable for  $y$ , since "s father" is not a complete sentence.
2.2 Searching for the optimal solution
Searching for the optimal output  $\mathbf{y}$  that maximizes the C-SCORE is not trivial since there are  $\mathcal{O}(V^T)$  candidate sequences where  $V$  is the vocabulary size and  $T$  is the maximum length of sequences to be searched. We investigate two approaches to obtain approximate solutions: reranking (§ 2.2.1) and context-aware beam search (§ 2.2.2).
2.2.1 Reranking with C-SCORE
We first generate  $B$  hypotheses of the translation  $\mathcal{H}_B = \{\pmb {y}^1,\dots ,\pmb {y}^B\}$  with beam search of beam size  $B$  using the sentence-level NMT model. We then choose the one that maximizes the C-SCORE.
$$
\hat {y} = \arg \max  _ {\boldsymbol {y} \in \mathcal {H} _ {B}} \operatorname {C - S C O R E} (\boldsymbol {y}; \boldsymbol {x}, \boldsymbol {c} ^ {(y)}) \tag {9}
$$
An issue with reranking is that we need to set  $B$  to a large value when the diversity of models' outputs is limited (Yu et al., 2020), which increases the cost of decoding. We therefore attempt to integrate C-SCORE into the decoding with beam search.
2.2.2 Context-aware beam search
Context-aware beam search (C-AWARE beam) is a beam search that is extended to work with C-SCORE. C-SCORE (Eq. 3) can be decomposed into token-wise C-SCOREs (Eq. 5 through Eq. 8).
$$
\begin{array}{l} \operatorname {C - S C O R E} (\boldsymbol {y}; \boldsymbol {x}, \boldsymbol {c} ^ {(y)}) = \log p (\boldsymbol {y} | \boldsymbol {x}) + \operatorname {P M I} (\boldsymbol {c} ^ {(y)}, \boldsymbol {y}) \\ = \sum_ {t = 1} ^ {T + 1} \mathrm {C - S C O R E} _ {w} \left(\tilde {y} _ {t} \mid \tilde {\mathbf {y}} _ {<   t}\right) \tag {10} \\ \end{array}
$$
$$
\begin{array}{l} C - S C O R E _ {w} \left(\tilde {y} _ {t} \mid \tilde {\mathbf {y}} _ {<   t}\right) = \log p _ {S - N M T} \left(\tilde {y} _ {t} \mid \tilde {\mathbf {y}} _ {<   t}; x\right) \\ + \log \frac {p _ {\mathrm {D} - \mathrm {L M}} \left(\tilde {y} _ {t} \mid \tilde {\boldsymbol {c}} ^ {(\boldsymbol {y})} \cdot \tilde {\boldsymbol {y}} _ {<   t}\right)}{p _ {\mathrm {D} - \mathrm {L M}} \left(\tilde {y} _ {t} \mid <   / \mathrm {s} > \cdot \tilde {\boldsymbol {y}} _ {<   t}\right)} \tag {11} \\ \end{array}
$$
By this decomposition,  $\mathrm{C - S C O R E}_{w}$  is conditioned on the partial sequence generated by time step  $t$ . We can therefore apply beam search to generate sequences in an auto-regressive manner.
The first term of Eq. 11 represents the translation probability for the  $t$ -th token. The second term can
${}^{2}$  Strictly speaking,we assume  $y$  to be a realization of a random variable  $Y$  which is a sentence sampled from the space of an infinitely large document.
<sup>1</sup>We cannot rely on punctuations to know sentence boundaries, since they can be omitted in some domains.
be interpreted as PMI between the  $t$ -th token and the context, that is, how consistent the  $t$ -th token is with the context. Compared to the reranking approach, C-AWARE beam can be considered to maximize the C-SCORE more directly in the sense that disambiguation and token selection based on the context are performed at every step in beam search. Thus C-AWARE beam will more space-efficiently consider diverse hypotheses with the same beam size  $B$  than C-AWARE rerank.
2.2.3 Smoothing probabilities for PMI
In our preliminary experiments, we observe that the original C-AWARE beam significantly improves contrastive tests but deteriorates BLEU at the same time. By analyzing contextual PMI correlation between source and target texts, we find the PMI term in the C-SCORE sometimes takes an excessively large value against the translation probability term, which destroys the C-SCORE. This is understood intuitively by the fact that the calculation of PMI includes subtraction of log probability, and log probability may take a very small negative value to represent a probability close to zero.
To alleviate this problem, we adopt a smoothing method for probabilities. For simplicity, in this paper, we only present the temperature scaling  $(T$ -scaling, for short) (Guo et al., 2017).  $T$ -scaling replaces  $p_{y = w}$  by
$$
\bar {p} _ {y = w} = \frac {p _ {y = w} ^ {1 / T}}{\sum_ {w ^ {\prime}} p _ {y = w ^ {\prime}} ^ {1 / T}} \tag {12}
$$
where  $T$  is a hyper-parameter.  $T = 1$  is equivalent to no smoothing. We choose  $T$  from  $[1,\infty)$  to flatten the probability distribution.  $T$ -scaling is applied to both the numerator and denominator using the same  $T$ .
2.2.4 On the relation to shallow fusion
Shallow fusion (Gulcehre et al., 2015) is a method to integrate probability distribution outputs obtained by NMT and LM at sentence level to form a new translation objective that is expected to promote fluency of translations. The original shallow fusion score is computed using a sentence-level NMT (S-NMT) and language model (S-LM). The token-wise formula of the computation is
$$
\log p \left(y _ {t}\right) = \log p _ {\mathrm {S} - \mathrm {N M T}} \left(y _ {t}; \boldsymbol {x}\right) + \beta \log p _ {\mathrm {S} - \mathrm {L M}} \left(y _ {t}\right), \tag {13}
$$
where  $\beta$  is a hyper-parameter. In our notation with the document-level LM, this is written as
$$
\begin{array}{l} \log p (y _ {t}) = \log p _ {\mathrm {S - N M T}} (\tilde {y} _ {t} | \tilde {\boldsymbol {y}} _ {<   t}; \boldsymbol {x}) \\ + \beta \log p _ {\mathrm {S} - \mathrm {L M}} \left(\tilde {y} _ {t} | <   / \mathrm {s} > \cdot \tilde {\mathbf {y}} _ {<   t}\right). \tag {14} \\ \end{array}
$$
A natural extension of this objective to the context-aware scenario should be
$$
\begin{array}{l} p \left(y _ {t} \mid \boldsymbol {c} ^ {(\boldsymbol {y})}\right) = \log p _ {\mathrm {N M T}} \left(\tilde {y} _ {t} \mid \tilde {\boldsymbol {y}} _ {<   t}; x\right) \\ + \beta \log p _ {\mathrm {D} - \mathrm {L M}} \left(\tilde {y} _ {t} \mid \tilde {\boldsymbol {c}} ^ {(\boldsymbol {y})} \cdot \tilde {\boldsymbol {y}} _ {<   t}\right), \tag {15} \\ \end{array}
$$
where context  $\tilde{c}^{(y)}$  is integrated into the condition. We call this conditional (document-level) shallow fusion. Obviously, this is what we obtain from Eq. 11 by ignoring the discount of the unconditional LM probability  $p_{\mathrm{D - LM}}(\tilde{y}_t| < / s > \cdot \tilde{y}_{< t})$ .
Due to the absence of discounting with the unconditional LM, conditional shallow fusion would prefer tokens which frequently occur regardless of the context. It is also worth noting that, when the context is empty, conditional shallow fusion falls back to the original shallow fusion, whereas our C-SCORE falls back to sentence-level NMT. Therefore, we view C-SCORE as a reformulation of shallow fusion for context-aware translation.
We evaluate our methods on English to Russian translation, in terms of BLEU scores (Papineni et al., 2002) and contrastive tests (Voita et al., 2019b).
3.1 Datasets and preprocessing
We use the OpenSubtitles2018 corpus (Lison et al., 2018) for parallel and monolingual data. Following the criteria for document segmentation and filtering on sentence pairs presented by (Voita et al., 2019b), we build monolingual and parallel data as follows. To build monolingual data, we add document boundary information into each document such that they consist of contiguous subtitle sentences from the same movie and the timestamp difference of any two adjacent sentences is no more than seven seconds. To build parallel data, we pick subtitle pairs where the time overlap between the source and target language subtitles is at least 0.9 (to reduce alignment errors). For the training of multi-encoder NMT models, document boundary information is added to the parallel data based on the source-side timestamps as with the monolingual data. Prior to building the Russian data, we
Train	Dev.	Test
src	trg (mono)	src	trg (mono)	src	trg
# sentences	5.8M	30M	6.0k	23k	15.5k
avg. # tokens	9.9	9.4	10.1	9.6	8.9	9.8
Table 1: Statistics of the parallel and monolingual data.
remove the movies from which the contrastive test sets (§ 3.4) were made.
We perform punctuation normalization, tokenization, and truecasing on the source and target texts using Moses toolkit v4.0.<sup>3</sup> We then encode the texts into subwords using SentencePiece (v0.1.81)<sup>4</sup> with unigram LM. The subword vocabularies are of 16,000 tokens and trained for each language. The statistics of the datasets are listed in Table 1.
We compare our methods to one sentence-level translation model (SentTransformer) (Vaswani et al., 2017) and three context-aware translation models: Document transformer (Zhang et al., 2018), DocRepair (Voita et al., 2019a), and Bayes Document Reranker (Yu et al., 2020). All the context-aware models use the previous three sentences as context.
Document Transformer (DocTransformer, for short) is a multi-encoder document-level NMT model which takes source-side context as an auxiliary input and can be thus trained from document-level parallel data. We follow (Zhang et al., 2018)'s configuration for DocTransformer.
DocRepair is a sequence-to-sequence post-editing model. It repairs document-level inconsistencies in a text, each sentence of which has been translated separately by a sentence-level NMT model. DocRepair is trained on a pseudo parallel data made by pairing a monolingual corpus and its round-trip translations obtained using a back-translation model and a forward-translation model.
Bayes Document Reranker (hereafter, Bayes DocReranker) performs document-level translation on a document containing  $D$  sentences in the following steps. First, it produces  $B$ -best translations for each sentence in the document and then produces a lattice of width  $B$  and depth  $D$ , where each node corresponds to a candidate sentence. It
then performs document-level beam search of beam size  $B'$  on the lattice using the following score:
$$
\begin{array}{l} \operatorname {S c o r e} \left(\boldsymbol {y} _ {i}; \boldsymbol {y} _ {<   i}, \boldsymbol {x} _ {i}\right) = \\ p _ {\mathrm {D - L M}} \left(\boldsymbol {y} _ {i} \mid \boldsymbol {y} _ {<   i}\right) + \operatorname {S c o r e} \left(\boldsymbol {y} _ {i - 1}; \boldsymbol {y} _ {<   i - 1}, \boldsymbol {x} _ {i - 1}\right) \\ + \lambda_ {1} p _ {\mathrm {N M T}} (\boldsymbol {y} _ {i} | \boldsymbol {x} _ {i}) + \lambda_ {2} p _ {\text {B A C K - N M T}} (\boldsymbol {x} _ {i} | \boldsymbol {y} _ {i}) + \lambda_ {3} | \boldsymbol {y} _ {i} | \tag {16} \\ \end{array}
$$
Note that this document-level beam search is equivalent to the reranking procedure (§ 2.2.1) when  $B' = 1$ . Therefore, the essential difference between Bayes DocReranker and our C-SCORE reranking is the score function.
SentTransformer, the post-editing model of DocRepair, and the back-translation models are based on the same configuration of Transformer base (see (Vaswani et al., 2017) for hyperparameter settings). The SentTransformer is trained using the 5.8M sentence pairs and is also used as the sentence-level NMT model in DocRepair, Bayes DocReranker, and our methods. For the training of DocTransformer, we use the 5.8M sentence pairs with document-level source context, which share the target-side sentences with the training data of SentTransformer. Consequently, scores obtained from the model are for reference. We also evaluate DocTransformer and SentTransformer using back-translation (BT) (Sennrich et al., 2016) with the same monolingual data as the other models.