SlowGuess/ACL_2020_OCR huggingface dataset:
Hongfei Xu $^{1,2}$  Josef van Genabith $^{1,2*}$  Deyi Xiong $^{3}$  Qiuhui Liu $^{4}$  Jingyi Zhang $^{2}$
hfxunlp@foxmail.com, Josef.Van_Genabith@dfki.de, dyxiong@tju.edu.cn,
liuqhano@foxmail.com, Jingyi.Zhang@dfki.de
The Transformer translation model (Vaswani et al., 2017) based on a multi-head attention mechanism can be computed effectively in parallel and has significantly pushed forward the performance of Neural Machine Translation (NMT). Though intuitively the attentional network can connect distant words via shorter network paths than RNNs, empirical analysis demonstrates that it still has difficulty in fully capturing long-distance dependencies (Tang et al., 2018). Considering that modeling phrases instead of words has significantly improved the Statistical Machine Translation (SMT) approach through the use of larger translation blocks ("phrases") and its reordering ability, modeling NMT at phrase level is an intuitive proposal to help the model capture long-distance relationships. In this paper, we first propose an attentive phrase representation generation mechanism which is able to generate phrase representations from corresponding token representations. In addition, we incorporate the generated phrase representations into the Transformer translation model to enhance its ability to capture long-distance relationships. In our experiments, we obtain significant improvements on the WMT 14 English-German and English-French tasks on top of the strong Transformer baseline, which shows the effectiveness of our approach. Our approach helps Transformer Base models perform at the level of Transformer Big models, and even significantly better for long sentences, but with substantially fewer parameters and training steps. The fact that phrase representations help even in the big setting further supports our conjecture that they make a valuable contribution to long-distance relations.
NMT is a new approach to machine translation that has achieved great success in the last a few
years (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Compared to plain SMT (Brown et al., 1993; Koehn et al., 2003; Chiang, 2005), a neural language model decoder (Sutskever et al., 2014) is better at long-distance re-ordering, and attention mechanisms (Bahdanau et al., 2015; Vaswani et al., 2017) have been proven effective in modeling long-distance dependencies, while these two issues were both challenging for SMT.
The Transformer (Vaswani et al., 2017), which has outperformed previous RNN/CNN based translation models (Bahdanau et al., 2015; Gehring et al., 2017), is based on multi-layer multi-head attention networks and can be trained in parallel very efficiently. Though attentional networks can connect distant words via shorter network paths than RNNs, empirical results show that its ability in capturing long-range dependencies does not significantly outperform RNNs, and it is still a problem for the Transformer to fully model long-distance dependencies (Tang et al., 2018).
Using phrases instead of words enables conventional SMT to condition on a wider range of context, and results in better performance in reordering and modeling long-distance dependencies. It is intuitive to let the NMT model additionally condition on phrase level representations to capture long-distance dependencies better, but there are two main issues which prevent NMT from directly using phrases:
- There are more phrases than tokens, and the phrase table is much larger than the word vocabulary, which is not affordable for NMT;
- Distribution over phrases is much sparser than that over words, which may lead to data sparsity and hurt the performance of NMT.
Instead of using phrases directly in NMT, in
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 386-396 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
this work, we address the issues above with the following contributions:
- To address the large phrase table issue, we propose an attentive feature extraction model and generate phrase representation based on token representations. Our model first summarizes the representation of a given token sequence with mean or max-over-time pooling, then computes the attention weight of each token based on the token representation and the summarized representation, and generates the phrase representation by a weighted combination of token representations;
- To help the Transformer translation model better model long-distance dependencies, we let both encoder layers and decoder layers of the Transformer attend the phrase representation sequence which is shorter than the token sequence, in addition to the original token representation. Since the phrase representations are produced and attended at each encoder layer, the encoding of each layer is also enhanced with phrase-level attention computation;
- To the best of our knowledge, our work is the first to model phrase representations and incorporating them into the Transformer.
Our approach empirically brings about significant and consistent improvements over the strong Transformer model (both base and big settings). We conducted experiments on the WMT 14 English-German and English-French news translation task, and obtained  $+1.29$  and  $+1.37$  BLEU improvements respectively on top of the strong Transformer Base baseline, which demonstrates the effectiveness of our approach. Our approach helps Transformer Base models perform at the level of Transformer Big models, and even significantly better for long sentences, but with substantially fewer parameters and training steps. It also shows effectiveness with the Transformer Big setting. We also conducted length analysis with our approach, and the results show how our approach improves long-distance dependency capturing, which supports our conjecture that phrase representation sequences can help the model capture long-distance relations better.
In this section, we first review previous work which utilizes phrases in recurrent sequence-to
sequence models, then give a brief introduction to the stronger Transformer translation model that our work is based on.
2.1 Utilizing Phrases in RNN-based NMT
Most previous work focuses on utilizing phrases from SMT in NMT to address its coverage (Tu et al., 2016) problem.
Dahlmann et al. (2017) suggested that SMT usually performs better in translating rare words and profits from using phrasal translations, even though NMT achieves better overall translation quality. They introduced a hybrid search algorithm for attention-based NMT which extended the beam search of NMT with phrase translations from SMT. Wang et al. (2017a) proposed that while NMT generally produces fluent but often inadequate translations, SMT yields adequate translations though less fluent. They incorporate SMT into NMT through utilizing recommendations from SMT in each decoding step of NMT to address the coverage issue and the unknown word issue of NMT. Wang et al. (2017b) suggested that phrases play a vital role in machine translation, and proposed to translate phrases in NMT by integrating target phrases from an SMT system with a phrase memory given that it is hard to integrate phrases into NMT which reads and generates sentences in a token-by-token way. The phrase memory is provided by the SMT model which dynamically picks relevant phrases with the partial translation from the NMT decoder in each decoding step.
2.2 The Transformer Translation Model
Our research is based on the Transformer translation model (Vaswani et al., 2017) shown in Figure 1, which significantly outperforms the previous recurrent sequence-to-sequence approach and can be efficiently computed in parallel.
The Transformer includes an encoder and a decoder. Both encoder and decoder are a stack of 6 layers. Besides the embedding matrix and positional embedding matrix in both encoder and decoder, the decoder also has a softmax classifier layer to produce translated tokens. The weights of the softmax classifier are normally tied to the target embedding matrix.
Both encoder layers and decoder layers make use of the multi-head attention mechanism. The multi-head attention mechanism calculates attention results of given queries on corresponding keys
Figure 1: The Transformer Translation Model. Residual connection and Layer normalization are omitted for simplicity.
and values. It first projects queries, keys and values with 3 independent linear transformations, then splits the transformed key, query and value embeddings into several chunks of  $d_{k}$  dimension vectors, each chunk is called a head, $^{1}$  and scaled dot-product attention is independently applied in each head:
$$
A t t n (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V \tag {1}
$$
where  $Q, K$  and  $V$  stand for the query vectors, key vectors and value vectors. Finally, the network concatenates the outputs of all heads and transforms it into the target space with another linear layer. The self-attention network uses the query sequence also as the key sequence and the value sequence in computation, while the cross-attention feeds another vector sequence to attend as queries and values.
Comparing the computation of the attentional network with RNNs, it is obvious that the attention computation connects distant words with a shorter network path, and intuitively it should perform better in capturing long-distance dependencies. However, empirical results show that its ability in modeling long-range dependencies does not significantly outperform RNNs.
2.3 Comparison with Previous Works
Compared to previous works using RNN-based NMT (He et al., 2016; Wang et al., 2017a,b; Dahlmann et al., 2017), our proposed approach is based on the Transformer model, with the following further important differences:
- Our approach aims to improve the long-distance dependency modeling ability of NMT instead of coverage (Tu et al., 2016);
- Our approach does not require to train an SMT system or to extract aligned phrase translation from the training corpus, which makes it efficient and avoids suffering from potential error propagation from the SMT system. The phrase representation learning model is a neural model, and is deeply integrated in the translation model, and the whole neural model is end-to-end trainable;
- We iteratively and dynamically generate phrase representations with token vectors. Previous work does not use SMT phrases in this way.
In more recent work, Wang et al. (2019) augment self attention with structural position representations to model the latent structure of the input sentence; Hao et al. (2019) propose multi-granularity self-attention which performs phrase-level attention with several attention heads.
3 Transformer with Phrase Representation
For the segmentation of phrases, given that N-gram phrases are effective for tensor libraries, we first try to cut a token sequence into a phrase sequence with a fixed phrase length which varies with the sequence length. We pad the last phrase in case it does not have sufficient tokens, thus we can transform the whole sequence into a tensor.
The N-gram phrase segmentation is efficient and simple, and we suggest the drawbacks of such "casual" segmentation boundaries can be alleviated with self-attention computation across the whole sequence and the attention mechanism applied in the generation of phrase representation which values tokens differently to a large extent, given that
2We implement this as: ntok = max(min(8, seql/6), 3), where ntok and seql stand for the number of tokens in each phrase and the length of a sentence respectively.
${}^{1}{d}_{k}$  is 64 for both the Transformer Base and the Transformer Big,and the numbers of heads for them are 8 and 16 respectively.
Algorithm 1 Extracting Phrases from a Parse Tree. Input: A parse tree  $T$ , maximum tokens allowed in a phrase  $n$ ; Output: Extracted phrase sequence  $S$ .
1: while T is not empty do
2: Initialize a phrase sequence  $p = [],$  maximum tokens allowed in this phrase  $mt = n$
3: Find the largest sub-tree  $ST$  with  $nst$  tokens  $(nst < n)$  and depth  $dst$  from the right side of  $T$ ;
4: Add the token sequence in  $ST$  into  $p$ ;
5: Remove  $ST$  from  $T$ ;
6: while  $mt > 0$  do
7: Find the adjacent sub-tree STA of depth dst with nsta tokens from the right side of  $T$ ;
8: if STA exists and  $nsta \leq mt$  then
9: Insert the token sequence of STA to the beginning of  $p$ ;
10: Remove STA from  $T$
11:  $mt = mt - nsta;$
12: else
13: Break;
14: end if
15: end while
16: Append  $p$  to  $S$ ;
17: end while
18: Reverse  $S$ ;
19: return  $S$
neural models have been proven good at learning competitively effective representations with gate or attention mechanism even without modeling linguistic structures (Cho et al., 2014; Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017; Devlin et al., 2019).
In our experiments we also explore phrases extracted from the Stanford Parser (Socher et al., 2013) as as an alternative to our simple segmentation strategy. The maximum number of tokens allowed is consistent with the simple segmentation approach, and we try to use the tokens from the largest sub-tree that complies with the maximum token limitation or from several adjacent sub-trees of the same depth as a phrase for efficiency. Our algorithm to extract phrases from parse trees is shown in Algorithm 1.
To efficiently parallelize parser-based phrases of various length in a batch of data, we pad short phrases to the same length of the longest phrases in the batch of sentences, thus a batch of sequences of
phrases can be saved into a tensor. But significantly more “<pad>” tokens will be introduced, and the model is slightly slower than the simple approach.
3.1 Attentive Phrase Representation Generation
Merging several token vectors into one is very likely to incur information loss, and introducing an importance evaluation mechanism is better than treating tokens equally. To highlight the most important features in a segmented phrase chunk, we introduce an attentive phrase representation generation model to value tokens differently according to their importance in the phrase. The model first roughly extracts features from all tokens into a vector, then assigns a score to each token by comparing each token vector with the extracted feature vector, and produces the weighted accumulation of all token vectors according to their scores.
Phrase representations are generated in every encoder layer, for the  $k_{th}$  encoder layer, we generate phrase representation  $R_{e_{phrase}}^{k}$  from its input representation. Assume the phrase contains  $m$  tokens  $\{t_1,\dots,t_m\}$ , and  $\{R_{e_{t_1}}^k,R_{e_{t_2}}^k,\dots,R_{e_{t_m}}^k\}$  are the corresponding input vectors to the encoder layer, we first generate a summary representation by:
$$
R _ {e _ {a l l}} ^ {k} = F _ {g l a n c e} \left(R _ {e _ {t _ {1}}} ^ {k},..., R _ {e _ {t _ {m}}} ^ {k}\right) \tag {2}
$$
where  $F_{\text{glance}}$  is a function to extract features of the vector sequence into a fixed-dimension vector; We explore both element-wise mean operation and max-over-time pooling operation in our work.
After the summarized representation is produced, we calculate a score for each token in the phrase, the score of the  $i_{th}$  token  $s_i^k$  is calculated as:
$$
\mathrm {s} _ {i} ^ {k} = W _ {2} ^ {k} \sigma \left(W _ {1} ^ {k} \left[ R _ {e _ {t _ {i}}} ^ {k} \mid R _ {e _ {\text {a l l}}} ^ {k} \right] + b _ {1} ^ {k}\right) + b _ {2} ^ {k} \tag {3}
$$
where  $\sigma$  is the sigmoid activation function, and  $|\cdot |$  means concatenation of vectors. The rationale for designing this approach is further explained below.
Then we normalize the score vector to weights with the softmax function, and the probability of the  $i_{th}$  token  $p_i^k$  is:
$$
\mathrm {p} _ {i} ^ {k} = \frac {e ^ {\mathrm {s} _ {i} ^ {k}}}{\sum_ {i = 1} ^ {m} e ^ {\mathrm {s} _ {i} ^ {k}}} \tag {4}
$$
Finally, the representation of the phrase in the  $k_{th}$  encoder layer  $R_{e\text{phrase}}^k$  is generated by a weighted combination of all vectors:
Figure 2: The Encoder/Decoder Layer of the Transformer Model with Phrase Representation. Residual connection and Layer normalization are omitted for simplicity.
$$
R _ {e _ {p h r a s e}} ^ {k} = \sum_ {i = 1} ^ {m} p _ {i} ^ {k} R _ {e _ {t i}} ^ {k} \tag {5}
$$
The representation of the phrase sequence can be computed efficiently in parallel. Each encoder layer will produce a vector sequence as the phrase representation. We do not use the multi-head attention in the computation of the phrase-representation attention because of two reasons:
- The multi-head attention calculates weights through dot-product, we suggest that a 2-layer neural network might be more powerful at semantic level feature extraction, and it is less likely to be affected by positional embeddings which are likely to vote up adjacent vectors;
- Though we employ a 2-layer neural network, it only has one linear transformation and a vector to calculate attention weights, which contains fewer parameters than the multi-head attention model that has 4 linear transformations.
Recent studies show that different encoder layers capture linguistic properties of different levels (Peters et al., 2018), and aggregating layers is of profound value to better fuse semantic information (Shen et al., 2018; Dou et al., 2018; Wang et al., 2018; Dou et al., 2019). We assume that different decoder layers may value different levels of information i.e. the representation of different encoder layers differently, thus we weighted combined phrase representations from every encoder layer for each decoder layer with the Transparent Attention (TA) mechanism (Bapna et al., 2018).
For the decoder layer  $j$ , the phrase representation  $R_{d_{phrase}}^j$  fed into that layer is calculated by:
$$
R _ {d _ {p h r a s e}} ^ {j} = \sum_ {i = 0} ^ {d} w _ {i} ^ {j} R _ {e _ {p h r a s e}} ^ {i} \tag {6}
$$
where  $w_{i}^{j}$  are softmax normalized parameters trained jointly with the full model to learn the importance of encoder layers for the  $j_{th}$  decoder layer.  $d$  is the number of encoder layers, and 0 corresponds to the embedding layer.
3.2 Incorporating Phrase Representation into NMT
After the phrase representation sequence for each encoder layer and decoder layer is calculated with the approach described above, we propose an attentive combination network to incorporate the phrase representation for each layer into the Transformer translation model to aid it modeling long-distance dependencies. The attentive combination network is inserted in each encoder layer and each decoder layer to bring in information from the phrase representation. The structures of the encoder layer and the decoder layer of the Transformer model with phrase representation are shown in Figure 2.
For an encoder layer, the new computation order is: cross-attention to phrases  $\rightarrow$  self-attention over tokens  $\rightarrow$  feed-forward neural network to process collected features, while for a decoder layer it is: self-attention over decoded tokens  $\rightarrow$  cross-attention to source phrases  $\rightarrow$  cross-attention to source tokens  $\rightarrow$  feed-forward neural network to process collected features. Compared to the computation order of the standard Transformer, the new computation order performs additional attending at
phrase level before attending source token representations at token level. We conjecture that attending at phrase level should be easier than at token level, and attention results at phrase level may aid the attention computation at the token-level.
For a given input sequence  $x$  and a phrase vector sequence  $R_{phrase}$ , the attentive combination network first attends the phrase representation sequence and computes the attention output outphrase as follows:
$$
o u t _ {p h r a s e} = \mathrm {A t t n} _ {\mathrm {M H}} (x, R _ {p h r a s e}) \qquad (7)
$$
where  $\mathrm{Att}\mathrm{n}_{\mathrm{MH}}$  is a multi-head cross-attention network with  $x$  as keys and  $R_{phrase}$  as corresponding queries and values.
The attention result is then combined again with the original input sequence  $x$  with a 2-layer neural network which aims to make up for potential information loss in the phrase representation with the original token representation:
$$
o u t = W _ {4} \sigma \left(W _ {3} [ x | o u t _ {p h r a s e} ] + b _ {3}\right) + b _ {4} \tag {8}
$$
We also employ a residual connection around the attentive combination layer, followed by layer normalization to stabilize the training.
Since the phrase representation is produced inside the Transformer model and utilized as the input of layers, and all related computations are differentiable, the attentive phrase representation model is simply trained as part of the whole model through backpropagation effectively.
To compare with Vaswani et al. (2017), we conducted our experiments on the WMT 14 English to German and English to French news translation tasks.
We implemented our approaches based on the Neutron implementation (Xu and Liu, 2019) of the Transformer translation model. We applied joint Byte-Pair Encoding (BPE) (Sennrich et al., 2016) with  $32k$  merge operations on both data sets to address the unknown word problem. We only kept sentences with a maximum of 256 subword tokens for training. Training sets were randomly shuffled in every training epoch. The concatenation of newstest 2012 and newstest 2013 was used for
Models	En-De	En-Fr
Transformer Base +PR	27.38	39.34
28.67†	40.71†
Transformer Big +PR	28.49	41.36
29.60†	42.45†
Table 1: Results on WMT 14 En-De and En-Fr.
validation and newstest 2014 as test sets for both tasks.
The number of warm-up steps was set to  $8k$ , and each training batch contained at least  $25k$  target tokens. Our experiments run on 2 GTX 1080 Ti GPUs, and a large batch size was achieved through gradient accumulation. We used a dropout of 0.1 for all experiments except for the Transformer Big on the En-De task which was 0.3. The training steps for Transformer Base and Transformer Big were  $100k$  and  $300k$  respectively following Vaswani et al. (2017). The other settings were the same as (Vaswani et al., 2017) except that we did not bind the embedding between the encoder and the decoder for efficiency.
We used a beam size of 4 for decoding, and evaluated tokenized case-sensitive  $\mathrm{BLEU}^3$  with the averaged model of the last 5 checkpoints for Transformer Base and 20 checkpoints for Transformer Big saved with an interval of 1, 500 training steps (Vaswani et al., 2017). We also conducted significance tests (Koehn, 2004).
We applied our approach to both the Transformer Base setting and the Transformer Big setting, and conducted experiments on both tasks to validate the effectiveness of our approach. Since parsing a large training set (specifically, the En-Fr dataset) is slow, we did not use phrases from parse results in this experiment (reported in Table 1). Results are shown in Table 1.  $\dagger$  indicates  $p < 0.01$  compared to the baseline for the significance test.
Table 1 shows that modeling phrase representation can bring consistent and significant improvements on both tasks, and benefit both the Transformer Base model and the stronger Transformer Big model. “+PR” is the Transformer with Phrase Representation, corresponding to the
3https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/ generic/multi-bleu.perl
Models	BLEU	Δ	Para. (M)	Time
Train	Decode
Transformer Base	27.38	88.1	1.00x	1.00x
+Mean	27.99	0.61	1.64x	1.45x
+Max	28.13	0.75	129.0	1.60x	1.40x
+Max+Attn	28.52	1.14	1.74x	1.52x
+Max+Attn+TA	28.67	1.29	173.0	1.75x	1.53x
+Max+Attn+TA+Parsing Phrase	28.76	1.38	1.83x	1.60x
Transformer Big	28.49	1.11	264.1	7.73x	2.68x
Table 2: Ablation Study.  $\Delta$  indicates the BLEU improvements compared to the Transformer Base. Time represents the time consumption compared to the Transformer Base (in training and decoding). The Transformer Big consumes 3 times training steps of the Transformer Base.
$+\mathrm{Max} + \mathrm{Attn} + \mathrm{TA}$  setting in Table 2.