SlowGuess/ACL_2020_OCR huggingface dataset:
Jian Yang $^{1}$ , Shuming Ma $^{2}$ , Dongdong Zhang $^{2}$ , Zhoujun Li $^{1}$ , Ming Zhou $^{2}$
<sup>1</sup>State Key Lab of Software Development Environment, Beihang University
$^{2}$ Microsoft Research Asia
{jiaya, lizj}  $@$  buaa.edu.cn; {shumma, dozhang, mingzhou}  $@$  microsoft.com;
Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only depend on the source text to generate translation. Inspired by the success of template-based and syntax-based approaches in other fields, we propose to use extracted templates from tree structures as soft target templates to guide the translation procedure. In order to learn the syntactic structure of the target sentences, we adopt the constituency-based parse tree to generate candidate templates. We incorporate the template information into the encoder-decoder framework to jointly utilize the templates and source text. Experiments show that our model significantly outperforms the baseline models on four benchmarks and demonstrate the effectiveness of soft target templates.
Recently, neural machine translation (NMT) (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) has achieved significant progress. Some advanced models (Chatterjee et al., 2016; Niehues et al., 2016; Junczys-Dowmunt and Grundkiewicz, 2017; Geng et al., 2018; Zhou et al., 2019a) predict the ultimate translation by multi-pass generation conditioned on the previous text such as CMLMs (Ghazvininejad et al., 2019), ABD-NMT (Zhang et al., 2018), SynST (Akoury et al., 2019), and Deliberation Network (Xia et al., 2017).
Inspired by these works and the successful application of templates for other intriguing tasks, including semantic parsing (Dong and Lapata, 2018), summarization (Cao et al., 2018; Wang et al., 2019a), question answering (Duan et al., 2017;
Figure 1: An example of template guided translation results. S denotes subject and VP denotes verb phrase.
Pandey et al., 2018), and other text generation tasks (Wiseman et al., 2018; Guu et al., 2018), we assume the candidate templates of the target sentences can guide the sentence translation process. We denote these templates extracted from the constituency-based parse tree as soft templates, which consist of tags and target words. The templates are soft because no explicit paradigms are inaugurated to build new translation from them, and the target tokens could be modified.
In order to effectively use the templates, we introduce soft template-based neural machine translation (ST-NMT), which can use source text and soft templates to predict the final translation. Our approach can be split into two phases. In the first phase, a standard Transformer model is trained to predict soft target templates by using source text and templates extracted from the constituency-based parse tree. Secondly, we use two encoders, including a soft target template encoder and a source language encoder to encode source text and templates and generate the final translation. As shown in Figure 1, given the source text "我喜欢打篮球" and the target template "S like to VP", the final translation "I like to play basketball" is generated by two encoders. In this work, the templates play a part in guiding, and some target tokens in
*Contribution during internship at Microsoft Research Asia.
5979
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5979-5989 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Overview of our ST-NMT. Given the source text and the soft target template predicted by the  $P_{\theta_{X \to Y}}$ , the source language Transformer encoder and target template Transformer encoder maps two sequences  $X = (x_1, x_2, x_3, x_4, x_5)$  and  $T = (t_1, y_2', t_3, t_4, t_5)$  into hidden states  $Z^X$  and  $Z^T$ .  $x_i$  denotes the source word,  $t_i$  denotes the template tag and  $y_i$  denotes the target word.  $y_i'$  also denotes the target word but it can be modified to the other target words. The ultimate translation  $Y$  is generated by a Transformer decoder which incorporates the context  $Z^X$  and  $Z^Y$  in the second phase.
the template could also be modified.
In order to prove the effectiveness of our approach, we conduct main experiments on the popular benchmarks, including IWSLT14 German-English translation task, WMT14 English-German translation task, LDC Chinese-English translation task, and ASPEC Japanese-Chinese translation task. Experiments show that our approach achieves significant improvement compared to the baselines, which demonstrates the soft target templates can provide a positive influence for guiding translation procedure effectively. Our approach can be used for diverse scale data sets, different styles, and multiple language pairs.
2 Our Approach
Our model first reads the source language sequence  $X = (x_{1}, x_{2}, x_{3}, \ldots, x_{n})$  in the conventional way by a source language Transformer encoder and generates the template sequence  $T = (t_{1}, t_{2}, t_{3}, \ldots, t_{m})$  by a template Transformer decoder. As shown in Figure 2, our model uses a source language Transformer encoder and a template Transformer encoder, which encodes the source language sequence  $X$  and the template sequence  $T$  separately. We deploy the target language decoder to generate the final translation. In this section, we present the details of the proposed template-based approach. Our method mainly in-
cludes two phases: (1) The training data is constructed by the constituency-based parse tree. Then, we adopt a standard Transformer to convert the source text to the soft target template for the next generation. (2) Based on the source text and the predicted soft target template, we utilize two encoders to encode two sequences into hidden states separately and a target language decoder to generate the ultimate translation.
2.1 Soft Template Prediction
In this procedure, we model the  $P_{\theta_{X \to T}}(T|X)$  to predict soft target templates on top of the constructed training data  $D_{X,T}$ . To construct  $D_{X,T}$ , we use a constituency-based parser to parse the target sequence and get a tree structure. Then, we prune nodes which are deeper than the specific depth and recover the left leaf nodes to the ordered template sequence. Through these operations, we gain the parallel training data  $D_{X,T}$  and train a standard Transformer model  $P_{\theta_{X \to T}}(T|X)$  to predict the soft target template.
The constituency-based parse tree could reveal the structure information of the whole sentence which utilizes the constituency grammar to distinguish terminal and non-terminal nodes. More specifically, the interior nodes are labeled by non-terminal categories which belong to the set of non-terminal tokens  $S$ , while the leaf nodes are labeled
Figure 3: The constituency-based parse tree of the example sentence. Given the target sentence and definite depth of the tree, we gain the sub-tree by pruning the nodes deeper than 4 in this case. Then, the sub-tree can be converted to the soft target template "There are NP VP" from left to right.
by terminal categories  $V$ .  $S = \{\mathrm{S},\mathrm{VP},\mathrm{NP},\ldots ,$  ASBR} and  $V$  is the vocabulary set of the target language. For example, the sentence "There are some people running" could be expressed as Figure 3. In this case, the non-terminal tokens consist of  $S_0 = \{\mathrm{S,NP,VP,EX,VBP,NP,DT,NNS,}$  VBG} and the terminal tokens are composed of  $V_{0}$ $=$  {There, are some, people, running}. Our template  $T = \{t_1,t_2,t_3,t_4\}$  is the ordered sequence which is composed of non-terminal tokens and terminal tokens. In this case,  $t_1 =$  There,  $t_2 =$  are,  $t_3 = \mathrm{VP}$  and  $t_4 = \mathrm{NP}$ . Our template extraction aims to extract the sub-tree of the specific depth and use these nonterminal and terminal tokens locating at the leaf node of sub-tree.
In order to predict the soft target templates, we train a standard Transformer model given the training data of the source text and extracted templates. The Transformer model reads the source text and predicts the soft target templates using beam search. Then, we select the top-K results of the beam search as templates.
The depth of the tree is a trade-off. In Figure 3, One special case is that when the depth equals 1, the template only has one symbol "S". The template "S" cannot provide any useful information. Another special case is that when depth is greater
than 6, the template "There are some people running" only has terminal tokens. The template only contains target words, which cannot provide any additional information. When the depth equals 4, the template is "There are NP VP". The template contains sentence syntactic and structural information, which is suitable for our method.
With the Transformer model  $P_{\theta_{X \to T}}(T|X)$ , we need to construct the pseudo training data  $D_{X, T, Y}$  instead of directly using extracted templates by soft template prediction. Given the source text  $X$ , we use  $P_{\theta_{X \to T}}(T|X)$  to predict the top-1 soft target template  $T$  with beam search. Therefore, we get the triple training data  $D_{X, T, Y} = \{X^{(i)}, T^{(i)}, Y^{(i)}\}_{i=1}^{N}$  which is prepared for the next phase.
2.2 Machine Translation via Soft Templates
The triple training data  $D_{X,T,Y}$  is used to model the probability  $P_{(X,T)\to Y}$  from the two sequences to the ultimate translation. Our approach could generate the target sentence  $Y$ , given the source sequence  $X$  and template  $T$ .
Formulation In formula, we could model the whole procedure on top of the  $P_{\theta_{X \to T}}(T|X)$  and  $P_{\theta_{(X,T) \to Y}}(Y|X,T)$ .
$$
P (Y | X) = P _ {\theta_ {X \rightarrow T}} (T | X) P _ {\theta_ {(X, T) \rightarrow Y}} (Y | X, T) \tag {1}
$$
where  $\theta_{X\to T}$  and  $\theta_{(X,T)\to Y}$  are the parameters for the first and the second phase.
The source language Transformer encoder and the soft template Transformer encoder maps the input sequence  $X$  and the template  $T$  composed of target language words and tags to the hidden states. Then, a Transformer decoder interacting with two encoders generates the final translation  $Y$ , described by the Equation 1.
Encoder In the second phase, our template Transformer encoder and the source language Transformer encoder are stacked by blocks which contain self-attention layers with residual connections, layer normalization and fully connected feedforward network (FFN). Therefore, the hidden states of the source language Transformer encoder and the template Transformer encoder are calculated by:
$$
h _ {l} = \text {T r a n s f o r m e r B l o c k} \left(h _ {l - 1}\right) \tag {2}
$$
5981
where  $h_l = h_l^X$  for the source language Transformer encoder and  $h_l = h_l^T$  for the template Transformer encoder.  $N$  is the number of layers and  $l \in [1, N]$ .
Decoder Based on the hidden states  $h_l^X$  and  $h_l^T$ , the target language Transformer decoder use the encoder-decoder multi-head attention to jointly use the source language and template information to generate the ultimate translation  $Y$ . Besides, the target sequence decoder uses multi-head attention to obtain the representations of target language decoder with the parameters  $(W_X^Q, W_X^K, W_X^V)$  and  $(W_T^Q, W_T^K, W_T^V)$  for different encoders.
In each attention head, the input sequence  $X = (x_{1},\ldots ,x_{m})$  and the template  $T = (t_{1},\dots ,t_{n})$  can be mapped into  $Z^{X} = (z_{1}^{X},z_{2}^{X},\dots ,z_{m}^{X})$  and  $Z^{T} = (z_{1}^{T},z_{2}^{T},\dots ,z_{n}^{T})$  using the source language Transformer encoder and the template Transformer encoder.
On top of the  $Z_{X}$  and  $Z_{T}$ , the decoder separately calculate the multi-head attention with source sentence context  $X = (x_{1},\ldots ,x_{m})$  and target template sentence  $T = (t_{1},\dots ,t_{n})$ , then our model obtain two hidden states  $Z^{X,Y}$  and  $Z^{T,Y}$  by attention with source context and template context. Here, We incorporate the  $Z^{X,Y}$  containing source language information and  $Z^{X,Y}$  including template information in a reasonable way:
$$
Z = \beta Z ^ {X, Y} + (1 - \beta) Z ^ {T, Y} \tag {3}
$$
where  $\beta$  is the parameter to control the degree of incorporation between source text and template.
In order to effectively incorporate source and template information, we calculate the parameter  $\beta$  as below:
$$
\beta = \sigma \left(W _ {Y} Z ^ {X, Y} + U _ {T} Z ^ {X, T}\right) \tag {4}
$$
where  $Z_{Y}$  is the decoder hidden state and  $W_{Y}$  and  $U_{T}$  are parameter matrices.  $\sigma$  is the sigmoid activation function.
Similar to the conventional NMT, in order to make the model predict the target sequence, we use maximum likelihood estimation (MLE) loss function to update the model parameter by maximizing the log likelihood of translation over training set  $D$ . When we train the  $P_{\theta_{X \to Y}}$  without the template Transformer encoder, we only need to optimize the following loss function:
$$
L _ {\theta_ {X \rightarrow Y}} (D) = \sum_ {X, Y \in D} \log P _ {\theta_ {X \rightarrow Y}} (Y | X) \tag {5}
$$
where  $\theta_{X\to Y}$  are the parameters of the source language Transformer encoder and the target language Transformer decoder.
When we train the  $P_{\theta_{(X,T)\to Y}}$  with the template Transformer encoder, the loss function could be calculated by:
$$
L _ {\theta_ {(X, T) \rightarrow Y}} (D) = \sum_ {X, Y \in D} \log P _ {\theta_ {(X, T) \rightarrow Y}} (Y | X, T) \tag {6}
$$
where  $\theta_{(X,T)\to Y}$  are the parameters of the source language Transformer encoder, template language Transformer encoder and target language Transformer decoder.
To balance the two objectives, our model is trained on  $L_{\theta_{X \to Y}}(D)$  objective for the  $\alpha \%$  iterations, and trained on  $L_{\theta_{(X,T) \to Y}}(D)$  objective for the  $(1 - \alpha) \%$  interations. Therefore, this procedure is equivalent to the following formula:
$$
L _ {\theta} (D) = \alpha L _ {\theta_ {X \rightarrow Y}} (D) + (1 - \alpha) L _ {\theta_ {(X, T) \rightarrow Y}} (D) \tag {7}
$$
where  $\alpha$  is a scaling factor accounting for the difference in magnitude between  $L_{\theta_{X\to Y}}(D)$  and  $L_{\theta_{(X,T)\to Y}}(D)$ .
In practice, we find optimizing these two objectives can make training procedure easier and get a higher BLEU score since there exist a few low-quality templates to influence the translation quality. Through optimizing two objectives simultaneously, we can reduce the effect of some low-quality templates and improve the stability of our model.
We conducted experiments on four benchmarks, including LDC Chinese-English, WMT14 English-German, IWSLT14 German-English, and ASPEC Japanese-Chinese translation tasks. By conducting experiments on these four benchmarks, these settings prove that our approach is suitable for diverse situations: (1) These four benchmarks provide a wide coverage of both scale and genres. They vary from small scale to large scale (2) We use the different domains, which include news, science, and talk domain. (3) We also conduct the experiments
5982
on different language pairs, including the German-English translation task, the English-German translation task, the Chinese-English translation task, and the Japanese-Chinese translation task.
In order to verify the effectiveness of our method, we conduct experiments on four benchmarks. WMT14 and LDC datasets are from the news domain. IWSLT14 dataset is from TED talk. ASPEC dataset is from a scientific paper excerpt corpus.
LDC Chinese-English We use a subset from LDC corpus<sup>1</sup> which has nearly 1.4M sentences originally. The training set is selected from the LDC corpus that consists of 1.2M sentence pairs after dropping the low-quality sentence pairs of which the length is more than 2. We used the NIST 2006 dataset as the validation set for evaluating performance in the training procedure, and NIST 2003, 2005, 2008 and 2012 as test sets, which all have 4 English references for each Chinese sentence.
IWSLT14 German-English This dataset contains 16K training sequence pairs. We randomly sample  $5\%$  of the training data as valid test. Besides, we merge the multiple testsets dev2010, dev2012, tst2010, tst2011, tst2012 for testing.
WMT14 English-German The training data consists of 4.5M sentence pairs. The validation set is devtest2014, and the test set is newtest2014.
ASPEC Japanese-Chinese We use 0.67M sentence pairs from ASPEC Japanese-Chinese corpus (Nakazawa et al., 2016)  ${}^{2}$  . We use the devtest as the development data, which contains 2090 sentences, and the test data contains 2107 sentences with a single reference per source sentence.
3.2 Preprocessing and Training Details
LDC Chinese-English The base Transformer model is used for this task, which includes 6 layers, each layer of which has the hidden dimensions of 512, feedforward dimensions of 2048, and 8 attention heads. We use Moses (Koehn et al., 2007) to tokenize English sentences and our in-house tool to tokenize Chinese sentences. We use Byte Pair Encoding (BPE) (Sennrich et al., 2016) to encode
sentences using a shared vocabulary of 40K symbols.
IWSLT14 German-English We adopt the small setup of the Transformer model. The model has 6 layers with the embedding size of 512, a feedforward size of 1024, and 4 attention heads. In order to prevent overfitting, we use a dropout of 0.3, a  $l_{2}$  weight decay of  $10^{-4}$ , and a label smoothing of 0.1. We use BPE to encode sentences with a shared vocabulary of 10K symbols.
WMT14 English-German We use the big setting of Transformer (Vaswani et al., 2017), in which both the encoder and the decoder have 6 layers, with the embedding size of 1024, feedforward size of 4096, and 16 attention heads. The dropout rate is fixed as 0.3. We adopt Adam (Kingma and Ba, 2015) optimizer with a learning rate 0.1 of the similar learning rate schedule as Transformer (Vaswani et al., 2017). We set the batch size as 6000 and the update frequency as 16 on 8 GPUs for updating parameters (Ott et al., 2018) to imitate 128 GPUs. The datasets are encoded by BPE with a shared vocabulary (Sennrich et al., 2016) of 40K symbols.
ASPEC Japanese-Chinese We use the base setting of Transformer the same to the Chinese-English translation task. Following the similar learning rate schedule (Vaswani et al., 2017), we set the learning rate as 0.1. Chinese and Japanese sentences are tokenized with our in-house tools and encoded by BPE with a shared vocabulary of 10K symbols.
We evaluate the performance of the translation results. The evaluation metric is BLEU (Papineni et al., 2002). For the Chinese-English and German-English translation tasks, we use case-insensitive tokenized BLEU scores. For the English-German translation task, we use case-sensitive tokenized BLEU scores for evaluation. All the experiments last for 150 epochs and use Stanford parser to generate templates (Manning et al., 2014).
For all translation tasks, we use the checkpoint, which has the best valid performance on the valid set. For different test sets, we adapt the beam size and the length penalty to get better performance. In order to avoid the difference of the tokenizer for Chinese translation result evaluation, we adopt the character-level BLEU for testing. Checkpoint averaging is not used, except notification.
$^{1}$ LDC2002E17, LDC2002E18, LDC2003E07, LDC2003E14, LDC2005E83, LDC2005T06, LDC2005T10, LDC2006E17, LDC2006E26, LDC2006E34, LDC2006E85, LDC2006E92, LDC2006T06, LDC2004T08, LDC2005T10  $^{2}$ http://orchid.kuee.kyoto-u.ac.jp/ASPEC/
5983
Zh → En	MT06	MT03	MT05	MT08	MT12	Avg.
ConvS2S (Gehring et al., 2017)	39.98	42.25	41.22	33.43	32.21	37.28
GNMT (Wu et al., 2016)	40.53	42.88	42.73	33.97	32.55	38.03
Transformer (our implementation)	43.60	45.80	44.52	36.62	34.60	40.39
ST-NMT (our proposed)	44.69	46.56	46.04	37.53	35.99	41.53
We compare our approach with two types of baselines including one-pass baselines and multi-pass baselines.
One-pass Baselines: ConvS2S (Gehring et al., 2017) is a strong CNN-based baseline. We report the results referring to the paper of convolutional sequence to sequence model (ConvS2S). RNMT+ (Chen et al., 2018) is a state-of-the-art RNN-based NMT model. GNMT (Wu et al., 2016) is the typical encoder-decoder framework. We use the similar setting for all experiments. Transformer (Vaswani et al., 2017) is a strong baseline which has the state-of-the-art performance. We reimplement this baseline. LightConv and DynamicConv (Wu et al., 2019) are simpler but effective baselines. We directly report the results in the paper.
Multi-pass Baselines: Deliberation network (Xia et al., 2017) and SoftPrototype (Wang et al., 2019b) generates and polishes the raw text by a two-pass manner. SB-NMT (Zhou et al., 2019a) is a synchronous bidirectional neural machine translation which predicts its outputs using two direction simultaneously. ABD-NMT (Zhang et al., 2018) is an encoder-decoder NMT framework with the forward and backward decoder. By considering the agreement of both directions left-to-right (L2R) and right-to-left (R2L), Rerank-NMT (Liu et al., 2016) rescores all candidates. SBSG (Zhou et al., 2019b) is a synchronous bidirectional sequence generation model which predicts its translation from both sides to the middle simultaneously. Insertion Transformer (Stern et al., 2019) is a non-monotonic method which predicts the translation
Table 1: Evaluation results on  $\mathrm{Zh} \rightarrow$  En translation task with BLEU% metric. The "Avg." column means the averaged result of all NIST test sets except NIST2006. The result of our model is statistically significant compared to the other baselines ( $p < 0.01$ ).
De → En	BLEU
GNMT (Wu et al., 2016)	31.44
RNMT+ (Chen et al., 2018)	34.51
ConvS2S (Gehring et al., 2017)	30.41
LightConv (Wu et al., 2019)	34.80
DynamicConv (Wu et al., 2019)	35.20
Rerank-NMT (Liu et al., 2016)	34.82
Transformer (our implementation)	34.43
ST-NMT (our proposed)	35.24
Table 2: BLEU-4 scores  $(\%)$  on IWSLT14 De→Entask. The result of our model is statistically significant compared to the other baselines  $(p <   0.05)$
by inserting method.
3.5 Results
For the IWSLT14 German-English machine translation task, we present the results of the ST-NMT and other strong baselines in Table 2. We compare our method with other various methods, including GNMT, RNMT+, convS2S, LightConv, DynamicConv, and the Transformer model with the small setting. The Rerank-NMT model gets 34.82 BLEU by using the two-pass results, including left-to-right (L2R) and right-to-left (R2L), and selects the best candidates. As shown in Table 2, our model also significantly outperforms others and gains an improvement of 0.81 BLEU points than a strong Transformer baseline model. Moreover, our method outperforms the GNMT by 3.80 BLEU points, ConvS2S by 4.83 BLEU, LightConv by 0.44 BLEU, Dynamic by 0.04 BLEU and Rerank-NMT by 0.42 BLEU.
We secondly evaluate our method on the LDC Chinese-English translation task. The evaluation results on all NIST test sets against baselines are listed in Table 1. Our ST-NMT beats the other
<sup>3</sup>https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Translation/GNMT
4https://github.com/pytorch/fairseq
5984
En → De	BLEU
GNMT (Wu et al., 2016)	24.61
ConvS2S (Gehring et al., 2017)	25.16
Transformer (Vaswani et al., 2017)	28.40
RNMT+ (Chen et al., 2018)	28.49
Rerank-NMT (Liu et al., 2016)	27.81
ABD-NMT (Liu et al., 2016)	28.22
Deliberation Network (Xia et al., 2017)	29.11
SoftPrototype (Wang et al., 2019b)	29.46
SB-NMT (Zhou et al., 2019a)	29.21
SBSG (Zhou et al., 2019b)	27.45
Insertion Transformer (Stern et al., 2019)	27.41
Transformer (our implementation)	29.25
ST-NMT (our proposed)	29.68
Table 3: BLEU-4 scores  $(\%)$  on WMT14 En  $\rightarrow$  De task. The result of our model is statistically significant compared to the other baselines  $(p < 0.05)$ .
Ja → Zh	BLEU
GNMT (Wu et al., 2016)	49.12
ConvS2S (Gehring et al., 2017)	50.32
Transformer (our implementation)	52.02
ST-NMT (our proposed)	52.84