SlowGuess/NAACL_2021_OCR huggingface dataset:
Long Zhang $^{1,2,\dagger}$ , Tong Zhang $^{1,2,\dagger}$ , Haibo Zhang $^{3}$ , Baosong Yang $^{3}$ , Wei Ye $^{1,*}$ , Shikun Zhang $^{1}$
$^{1}$  National Engineering Research Center for Software Engineering, Peking University
$^{2}$  School of Software and Microelectronics, Peking University
3 Alibaba Group
{zhanglong418, zhangtong17, wye, zhangsk}@pku.edu.cn
{zhanhui.zhb, yangbaosong.ybs}@alibaba-inc.com
Document-level neural machine translation (NMT) has proven to be of profound value for its effectiveness on capturing contextual information. Nevertheless, existing approaches 1) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process; and 2) feed ground-truth target contexts as extra inputs at the training time, thus facing the problem of exposure bias. We approach these problems with an inspiration from human behavior – human translators ordinarily emerge a translation draft in their mind and progressively revise it according to the reasoning in discourse. To this end, we propose a novel Multi-Hop Transformer (MHT) which offers NMT abilities to explicitly model the human-like draft-editing and reasoning process. Specifically, our model serves the sentence-level translation as a draft and properly refines its representations by attending to multiple antecedent sentences iteratively. Experiments on four widely used document translation tasks demonstrate that our method can significantly improve document-level translation performance and can tackle discourse phenomena, such as coreference error and the problem of polysemy.
Neural machine translation (NMT) employs an end-to-end framework (Sutskever et al., 2014) and has advanced promising results on various sentence-level translation tasks (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017; Wan et al., 2020). However, most of NMT models handle sentences independently, regardless of the linguistic context that may appear outside the current sentence (Tiedemann and Scherrer, 2017a). This makes NMT insufficient to fully resolve the typical context-dependent phenomena problematic,
e.g. coreference (Guillou, 2016), lexical cohesion (Carpuat, 2009), as well as lexical disambiguation (Gonzales et al., 2017).
Recent studies (Tu et al., 2018; Maruf and Haffari, 2018; Maruf et al., 2019; Tan et al., 2019; Kim et al., 2019; Zheng et al., 2020; Chen et al., 2020; Sun et al., 2020; Ma et al., 2020) have proven to be effective on tackling discourse phenomena via feeding NMT with contextual information, e.g. source-side (Wang et al., 2017; Voita et al., 2018; Zhang et al., 2018) or target-side context sentences (Bawden et al., 2018; Miculicich et al., 2018). Despite their successes, these methods simply merge the representations of context sentences together, lacking a mechanism to explicitly characterize the inter-sentence reasoning upon the context. Another shortage in existing document-level NMT is the problem of exposure bias. Most of methods utilized the ground-truth target context for training but the generated translations for inference, leading to inconsistent inputs at training and testing time (Ranzato et al., 2015; Koehn and Knowles, 2017).
Intuitively, human translators tend to acquire useful context information from the reasoning process among sentences, thus figuring out the correct meaning when they encounter ambiguity during translation. Sukhbaatar et al. (2015) and Shen et al. (2017) empirically verified that modeling multi-hop reasoning among sentences benefits to the language understanding task, e.g text comprehension. Voita et al. (2019) showed that document-level NMT model can profit from relative positions with respect to context sentences, which to some extent confirms the importance of the relationship among sentences. Meanwhile, Xia et al. (2017) demonstrated that sentence-level NMT could be improved by a two-pass draft-editing process, of which the second-pass decoder refines the target sentence generated by a first-pass standard decoder.
Accordingly, we propose to improve document
†These authors contributed equally to this work.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3953-3963 June 6-11, 2021. ©2021 Association for Computational Linguistics
level NMT using a novel framework – Multi-Hop Transformer, which imitates draft-editing and reasoning process of human translators. Specifically, we implement an explicit reasoning process by exploiting source and target antecedent sentences with concurrently stacked attention layers, thus performing the progressive refinement on the representations of the current sentence and its translation. Besides, we leverage the draft to present context information on the target side during both training and testing, alleviating the problem of exposure bias.
We conduct experiments on four widely used document translation tasks: English-German and Chinese-English TED, English-Russian Opensubti-ties, as well as English-German Europarl-7 datasets. Experimental results demonstrate that our method significantly outperforms both context-agnostic and context-aware methods. The qualitative analysis confirms the effectiveness of the proposed multihop reasoning mechanism on resolving many linguistic phenomena, such as word sense disambiguation and coreference resolution. Our contributions are mainly in:
- We propose the Multi-Hop Transformer. To the best of our knowledge, this is the first pioneer investigation that introduces multi-hop reasoning into document-level NMT.
- The proposed model takes target context drafts into account at the training time, which devotes to avoid the training-generation discrepancy.
- Our approach significantly improves document-level translation performance on four document-level translation tasks in terms of BLEU scores and solves some context-dependent phenomena, such as coreference error and polysemy.
Transformer NMT is an end-to-end framework to build translation models. Vaswani et al. (2017) propose a new architecture called Transformer which adopts self-attention network for both encoding and decoding. Both its encoder and decoder consist of multiple layers, each of which includes a multi-head self-attention and a feed-forward sublayer. Additionally, each layer of the decoder applies a multi-head cross attention to capture information from the encoder. Transformer has shown
superiority in a variety of NLP tasks. Therefore, we construct our models upon this advanced architecture.
Document-level NMT In order to correctly translate the sentence with discourse phenomena, NMT models need to look beyond the current sentence and integrate contextual sentences as auxiliary inputs. Formally, let  $\mathbf{X} = (\mathbf{x}^1, \mathbf{x}^2, \dots, \mathbf{x}^I)$  be a source-language document composed of  $I$  sentences, where  $\mathbf{x}^i = (x_1^i, x_2^i, \dots, x_N^i)$  denotes the  $i^{th}$  sentence containing  $N$  words. Correspondingly, the target-language document also consists of  $I$  sentences,  $\mathbf{Y} = (\mathbf{y}^1, \mathbf{y}^2, \dots, \mathbf{y}^I)$ , where  $\mathbf{y}^i = (y_1^i, y_2^i, \dots, y_M^i)$  denotes the  $i^{th}$  sentence involving  $M$  words. Document-level NMT incorporates contextual information from both source side and target side to autoregressively generate the best translation result that has highest probability:
$$
P _ {\theta} \left(\mathbf {y} ^ {i} \mid \mathbf {x} ^ {i}\right) = \prod_ {m = 1} ^ {M} P _ {\theta} \left(y _ {m} ^ {i} \mid y _ {<   m} ^ {i}, \mathbf {x} ^ {i}, \mathbf {X} ^ {- i}, \mathbf {Y} ^ {- i}\right) \tag {1}
$$
where  $y_{<m}^{i}$  is the sequence of proceeding tokens before position  $m$ .  $\mathbf{X}^{-i}$  and  $\mathbf{Y}^{-i}$  denote the context sentences of the  $i^{th}$  sentence.
Related Work Several studies have explored multi-input models to leverage the contextual information from source-side (Jean et al., 2017; Kuang and Xiong, 2018) or target-side sentences (Kuang et al., 2018; Miculicich et al., 2018). For the former, Zhang et al. (2018) propose a new encoder to represent document-level context from previous source-side sentences. Tiedemann and Scherrer (2017b) and Junczys-Dowmunt (2019) utilize the concatenation of previous source-side sentences as input, while Voita et al. (2018) make use of gate mechanism to balance the weight between current source sentence and its context. For the latter, Miculicich et al. (2018) propose a hierarchical attention (HAN) framework to capture the target contextual information in the decoder. Bawden et al. (2018), Maruf and Haffari (2018) and Maruf et al. (2019) take both source-side and target-side context into account.
Motivation As seen, both of the existing methods simply introduce the context sentences without explicitly characterizing the inter-sentence reasoning. Intuitively, when humans have difficulty in translation like encountering ambiguity phenomenon, they could acquire more information
Figure 1: Illustration of Multi-Hop Transformer.  $c_s^{i-j}$  and  $c_t^{i-j}$  indicate the  $j^{th}$  previous sentence in the source side and target side respectively.  $d$  denotes the draft of current source sentence. All drafts are generated by a pre-trained sentence-level NMT model. The modules inside dashed box are the proposed multi-hop attention layers, which gradually refine the representation of current sentence. Finally, the context gate  $\alpha$  is used to control the contextual information.
from the contexts sentence by sentence and then perform reasoning to figure out the exact meaning. We attribute that such reasoning process is also beneficial to machine translation task. Recent successes in text comprehension communities have to some extent supported our hypothesis (Hill et al., 2015; Kumar et al., 2016). For example, Sukhbaatar et al. (2015) propose a multi-hop end-to-end memory network, which can renew the query representation with multiple computational steps (which they term "hops"). Dhingra et al. (2016) extend an attention-sum reader to multi-turn reasoning with a gating mechanism. In addition, Shen et al. (2017) introduce multi-hop attention, which used multiple turns to effectively exploit and reason over the relation among queries and documents.
In this paper, we propose to bring the idea of multi-hop into document translation and aim at mimicking the multi-step comprehension and revising process of human translators. Contrast with those models for text comprehension which scan the query and document for multiple passes, our
model iteratively focuses on different context sentences, which captures the inter-sentence reasoning semantics of contextual sentences to incrementally refine the representation of current sentence.
3 Multi-Hop Transformer
With this mind, we propose a novel method called Multi-Hop Transformer, which models the reasoning process among multiple contextual sentences in both source side and target side. The source-side contexts are directly acquired from the document. The target-side contexts, called target-side drafts in this paper, are generated by a sentence-level NMT model. These contexts are fed into the Multi-Hop Transformer with pre-trained encoders. The overall architecture of our proposed model is illustrated in Figure 1, which consists of three components:
- Sentence Encoder: This component contains two pre-trained encoders, one of which is called source-side sentence encoder and the other is called target-side sentence encoder. These encoders generate representations for
source-side contexts and target-side drafts respectively.
- Multi-Hop Encoder: We extend the original Transformer encoder with a novel multi-hop encoder to efficiently perform sentence-by-sentence reasoning on source-side contexts and generate the representation for the current sentence.
- Multi-Hop Decoder: Similarly, a multi-hop decoder is proposed to acquire information from the target-side drafts and models the translation probability distribution.
3.1 Sentence Encoder
We use multi-layer and multi-head self-attention architecture (Vaswani et al., 2017) to obtain the representations for source-side contexts and target-side drafts. Similar to the encoder of Transformer, sentence encoder contains a stack of six identical layers, each of which consists of two sub-layers. The first sub-layer is a multi-head attention  $(Q,K,V)$ , which takes a query  $Q$ , a key  $K$  and a value  $V$  as inputs. The second sub-layer is a fully connected feed-forward network (FFN).
Source-Side Sentence Encoder. This encoder is utilized to generate the representations for source-side contexts, as shown in Figure 1.
For the current sentence  $s = \mathbf{x}^i$  to be translated, we use the previous sentences  $\mathbf{X}^{-i} = (\mathbf{x}^{i - k},\mathbf{x}^{i - k + 1},\dots,\mathbf{x}^{i - 1})$  in the same document as the source-side context, specially denoted as  $c_{s}^{i - k}$ $c_{s}^{i - k + 1}$  ...  $c_{s}^{i - 1}$  for clarity.  $k$  is the context window size. For the  $j^{th}$  context, we obtain the  $A_{c_s^{i - j}}^{(n)}$  which denotes the  $n^{th}$  hidden layer representation of  $c_{s}^{i - j}$  as follows:
$$
A _ {c _ {s} ^ {i - j}} ^ {(n)} = \operatorname {M H A} \left(H _ {c _ {s} ^ {i - j}} ^ {(n - 1)}, H _ {c _ {s} ^ {i - j}} ^ {(n - 1)}, H _ {c _ {s} ^ {i - j}} ^ {(n - 1)}\right) \tag {2}
$$
where  $n = 1,2,\ldots ,6$ . MHA represents the standard Multi-Head Attention function (Vaswani et al., 2017).  $j$  denotes the distance between the context sentence and current sentence.
Target-Side Sentence Encoder. Most existing works use ground-truth target-side contexts as the input of decoder during training (Voita et al., 2019). However, the target contexts at training and testing are drawn from different distributions, leading to the inconsistency between training and testing.
To alleviate this problem, we instead make use of target-side context drafts generated from a pretrained sentence-level translation model. Similar to source-side sentence encoder, this target-side context draft encoder is used to obtain the context representation  $A_{c_t^{i - j}}^{(n)}$  of the  $j^{th}$  target-side draft  $c_t^{i - j}$ . Besides, we obtain a draft translation  $d$  of the current sentence from the pre-trained sentence-level translation model and use a target-side draft encoder to obtain the representation  $A_d^{(n)}$ .
3.2 Multi-Hop Encoder
The multi-hop encoder contains a stack of 6 identical layers, each of which contains the following sub-layers:
Self-Attention Layer. The first sub-layer makes use of multi-head self-attention to encode the information of current source sentence  $s$  and obtains the representation  $A_{s}^{(n)}$ .
Multi-Hop Attention Layer. The second sublayer uses a multi-hop attention to perform sentence-by-sentence reasoning on  $c_{s}$  in sentence order as shown in Figure 1. Each reasoning step, also called a hop, is implemented by a multi-head attention layer. The first hop takes representation  $A_{s}^{(n)}$  as the query and the representation  $A_{c_{s}^{i-k}}^{(n)}$  of the previous  $k^{th}$  sentence as the key and value.
$$
B _ {s ^ {i - k}} ^ {(n)} = \operatorname {M H A} \left(A _ {s} ^ {(n)}, A _ {c _ {s} ^ {i - k}} ^ {(n)}, A _ {c _ {s} ^ {i - k}} ^ {(n)}\right) \tag {3}
$$
The other hops are implemented:
$$
B _ {s ^ {i - j}} ^ {(n)} = \operatorname {M H A} \left(B _ {s ^ {i - j - 1}} ^ {(n)}, A _ {c _ {s} ^ {i - j},} ^ {(n)}, A _ {c _ {s} ^ {i - j}} ^ {(n)}\right) \tag {4}
$$
where  $j = k - 1, k - 2, \ldots, 1$ .  $j$  denotes the distance between the context sentence and current sentence.
Context Gating. The information of current source sentence is crucial in translation while the contextual information is auxiliary. In order to avoid excessive utilization of contextual information, a context gating mechanism (Tu et al., 2017; Yang et al., 2017, 2019) is introduced to dynamically control the weight between context sentences and current sentence:
$$
\alpha = \sigma \left(W _ {a} A _ {s} ^ {(n)} + W _ {b} B _ {s ^ {i - 1}} ^ {(n)}\right), \tag {5}
$$
where  $\sigma$  is the logistic sigmoid function and  $\alpha$  is the context gate.  $W_{a}$  and  $W_{b}$  denote the weight matrices of  $A_{s}^{(n)}$  and  $B_{s^{i - 1}}^{(n)}$ , respectively.
$$
H _ {s} ^ {(n)} = \alpha \odot A _ {s} ^ {(n)} + (1 - \alpha) \odot B _ {s ^ {i - 1}} ^ {(n)} \tag {6}
$$
Finally, we obtain the representation  $Enc_{s} = H_{s}^{(6)}$  as the final output of the multi-hop encoder.
3.3 Multi-Hop Decoder
Similarly, the multi-hop decoder involves a stack of 6 identical layers. Each of them contains five sub-layers.
Self-Attention Layer. The first sub-layer utilizes multi-head self-attention to encode the information of current target sentence  $t$  and obtains the representation  $A_{t}^{(n)}$ .
Draft-Attention Layer. Inspired by Xia et al. (2017), we introduce the complete draft  $d$  translated from current source sentence by a sentence-level NMT. Then this draft representation  $A_{d}^{(n)}$  is encoded by the target-side draft encoder in Section 3.1. The draft attention is achieved by multi-head attention:
$$
F _ {t} ^ {(n)} = \operatorname {M H A} \left(A _ {t} ^ {(n)}, A _ {d} ^ {(n)}, A _ {d} ^ {(n)}\right). \tag {7}
$$
Multi-Hop Attention Layer. Similar to the encoder, a multi-hop reasoning process is performed on the target-side contexts. The target-side drafts are generated from corresponding source sentences by a pre-trained sentence-level NMT model. The first hop takes representation  $F_{t}^{(n)}$  as the query and the representation  $A_{c_{t}^{i - k}}^{(n)}$  of the previous  $k^{th}$  draft as the key and value.
$$
B _ {t ^ {i - k}} ^ {(n)} = \operatorname {M H A} \left(F _ {t} ^ {(n)}, A _ {c _ {t} ^ {i - k}} ^ {(n)}, A _ {c _ {t} ^ {i - k}} ^ {(n)}\right) \tag {8}
$$
The other hops are achieved:
$$
B _ {t ^ {i - j}} ^ {(n)} = \operatorname {M H A} \left(B _ {t ^ {i - j - 1}} ^ {(n)}, A _ {c _ {t} ^ {i - j}} ^ {(n)}, A _ {c _ {t} ^ {i - j}} ^ {(n)}\right) \tag {9}
$$
where  $j = k - 1, k - 2, \ldots, 1$ .  $j$  denotes the distance between the context draft and current target draft.
Context Gating. Same as the multi-hop encoder, the final output of multi-hop decoder is computed as:
$$
G _ {t} ^ {(n)} = \alpha \odot F _ {t} ^ {(n)} + (1 - \alpha) \odot B _ {t ^ {i - 1}} ^ {(n)} \tag {10}
$$
where  $\alpha$  is used to regulate the weight of target-side contextual information.
Encoder-Decoder Attention Layer. Finally, we use an encoder-decoder attention layer to integrate the output of multi-hop encoder  $Enc_{s}$  with the current target representation  $G_{t}^{(n)}$ .
$$
H _ {t} ^ {(n)} = \operatorname {M H A} \left(G _ {t} ^ {(n)}, \operatorname {E n c} _ {s}, \operatorname {E n c} _ {s}\right) \tag {11}
$$
where  $H_{t}^{(n)}$  represents the final representation of decoder.
To evaluate the effectiveness of the proposed MHT, we conduct experiments on four widely used document translation tasks, including the TED Talk (Cettolo et al., 2012) with two language pairs, Opensubtitles (Maruf et al., 2018) and Europarl7 (Maruf et al., 2018). All datasets are tokenized and truecased with the Moses toolkit (Koehn et al., 2007), and split into sub-word units with a joint BPE model (Sennrich et al., 2016) with 30K merge operations. The datasets are described as follows:
- TED Talk (English-German): We use the dataset of IWSLT 2017 MT English-German track for training, which contains transcripts of TED talks aligned at sentence level. dev2010 is used for development and tst2016-2017 for evaluation. Statistically, there are 0.21M sentences in the training set, 9K sentences in the development set, and 2.3K sentences in the test set.
- TED Talk (Chinese-English): We use the corpus consisting of 0.2M sentence pairs extracted from IWSLT 2014 and 2015 Chinese-English track for training. dev2010 involves 0.8K sentences for development and tst2010-2013 contains 5.5K sentences for test.
- Opensubtitles (English-Russian): We make use of the parallel corpus from Maruf et al. (2018). The training set includes 0.3M sentence pairs. There are 6K sentence pairs in development set, and 9K in test set.
- Europarl7 (English-German): The raw Europarl v7 corpus (Koehn, 2005) contains SPEAKER and LANGUAGE tags where the latter indicates the language the speaker was actually using. We process the raw data and extract the parallel corpus as same as Maruf
Method	TED	Opensubtitles	Europarl7	Params	AVG
En → De	Zh → En	En → Ru	En → De
Transformer*	24.55	18.36	19.46	30.18	50M	23.14
CA-Transformer†	25.04	18.77	20.21	30.67	72M	23.67
(Maruf et al., 2018)†	-	-	19.13◇	26.49◇	-	-
CA-HAN†	25.70	18.79	20.08	26.61	70M	22.79
(Maruf et al., 2019)†	24.62◇	-	-	-	54M◇	-
CADec†	26.08	19.01	19.46	30.36	91M	23.98
MHT (Ours)†	26.22	19.52	20.46	31.25	80M	24.36
Table 1: BLEU scores on TED Talk, Opensubtitles and Europarl7 tasks.  $\star$  mark indicates context-agnostic NMT models and  $\dagger$  mark indicates context-aware NMT models. AVG indicates the average BLEU scores on test sets.  $\diamond$  denotes that the value is reported by the corresponding paper. Our MHT model achieves better performance than both context-agnostic and context-aware strong baseline on four examined tasks. The significance tests are conducted for testing the robustness of approaches, and the results are statistically significant with  $p < 0.05$ .
et al. (2018). 0.1M sentence pairs are used for training, 3K sentence pairs for development, and 5K sentence pairs for evaluation.
We compare our model against four NMT systems as follows:
- Transformer: The state-of-the-art context-agnostic NMT model (Vaswani et al., 2017).
- CA-Transformer: A context-aware transformer model (CA-Transformer) with an additional context encoder to incorporate document contextual information into model (Zhang et al., 2018).
- CA-HAN: A context-aware hierarchical attention networks (CA-HAN) which integrate document contextual information from both source side and target side (Miculicich et al., 2018).
- CADec: A two-pass machine translation model (Context-Aware Decoder, CADec) which first produces a draft translation of the current sentence, then corrects it using context (Voita et al., 2019).
Our model is implemented on the open-source toolkit Thumt (Zhang et al., 2017). Adam optimizer (Kingma and Ba, 2014) is applied with an initial learning rate 0.1. The size of hidden dimension and feed-forward layer are set to 512 and 2048 respectively. Encoder and decoder have 6 layers with 8 heads multi-head attention. Dropout is 0.1
and batch size is set to 4096. Beam size is 4 for inference. Translation quality is evaluated by the traditional metric BLEU (Papineni et al., 2002) on tokenized text. Context window size is set to 3, consistent with the experiments in Section 5.2.
To initialize the source-side sentence encoder in Section 3.1, a sentence-level NMT model is trained from source language to target language using the corresponding datasets without additional corpus. The encoder of this trained model is used to initialize the source-side context encoder. Also, we utilize the trained model to translate the source-side sentences and obtain the target-side drafts. Similarly, we train a sentence-level model from target language to source language to initialize the target-side encoders in Section 3.1. In order to reduce the computational overhead, we share the parameters among the sentence encoders on the same side. The settings of these two sentence-level NMT models are consistent with our baseline Transformer model.
Table 1 summarizes the BLEU scores of different systems on four tasks. As seen, our baseline and re-implemented existing methods outperform the reported results on the same data, which we believe makes the evaluation convincing.
Clearly, our model MHT significantly improves translation quality in terms of BLEU on these tasks, and obtains the best average results that gain 0.38, 0.69 and 1.57 BLEU points over CADec, CA-Transformer and CA-HAN respectively. These results demonstrate the universality and effectiveness of the proposed approach. Moreover, without in
Figure 2: The performance of the MHT model on TED (En-De) and TED (Zh-En) translation task using different context window sizes.