SlowGuess/ACL_2020_OCR huggingface dataset:
Emanuele Bugliarello*  
University of Copenhagen  
emanuele@di.ku.dk
Naoaki Okazaki Tokyo Institute of Technology okazaki@c.titech.ac.jp
Most neural machine translation models only rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. In this work, we investigate different approaches to incorporate syntactic knowledge in the Transformer model and also propose a novel, parameter-free, dependency-aware self-attention mechanism that improves its translation quality, especially for long sentences and in low-resource scenarios. We show the efficacy of each approach on WMT English  $\leftrightarrow$  German and English  $\rightarrow$  Turkish, and WAT English  $\rightarrow$  Japanese translation tasks.
Research in neural machine translation (NMT) has mostly exploited corpora consisting of pairs of parallel sentences, with the assumption that a model can automatically learn prior linguistic knowledge via an attention mechanism (Luong et al., 2015). However, Shi et al. (2006) found that these models still fail to capture deep structural details, and several studies (Sennrich and Haddow, 2016; Eriguchi et al., 2017; Chen et al., 2017, 2018) have shown that syntactic information has the potential to improve these models. Nevertheless, the majority of syntax-aware NMT models are based on recurrent neural networks (RNNs; Elman 1990), with only a few recent studies that have investigated methods for the Transformer model (Vaswani et al., 2017).
Wu et al. (2018) evaluated an approach to incorporate syntax in NMT with a Transformer model, which not only required three encoders and two decoders, but also target-side dependency relations (precluding its use to low-resource target languages). Zhang et al. (2019) integrate source-side syntax by concatenating the intermediate representations of a dependency parser to word embeddings.
In contrast to ours, this approach does not allow to learn sub-word units at the source side, requiring a larger vocabulary to minimize out-of-vocabulary words. Saunders et al. (2018) interleave words with syntax representations which results in longer sequences – requiring gradient accumulation for effective training – while only leading to  $+0.5$  BLEU on WAT Ja-En when using ensembles of Transformers. Finally, Currey and Heafield (2019) propose two simple data augmentation techniques to incorporate source-side syntax: one that works well on low-resource data, and one that achieves a high score on a large-scale task. Our approach, on the other hand, performs equally well in both settings.
While these studies improve the translation quality of the Transformer, they do not exploit its properties. In response, we propose to explicitly enhance the its self-attention mechanism (a core component of this architecture) to include syntactic information without compromising its flexibility. Recent studies have, in fact, shown that self-attention networks benefit from modeling local contexts by reducing the dispersion of the attention distribution (Shaw et al., 2018; Yang et al., 2018, 2019), and that they might not capture the inherent syntactic structure of languages as well as recurrent models, especially in low-resource settings (Tran et al., 2018; Tang et al., 2018). Here, we present parent-scaled self-attention (PASCAL): a novel, parameter-free local attention mechanism that lets the model focus on the dependency parent of each token when encoding the source sentence. Our method is simple yet effective, improving translation quality with no additional parameter or computational overhead.
- introducing PASCAL: an effective parameter-free local self-attention mechanism to incorporate source-side syntax into Transformers;
- adapting LISA (Strubell et al., 2018) to subword representations and applying it to NMT;
*Work done while at Tokyo Institute of Technology.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1618-1627 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Parent-Scaled Self-Attention (PASCAL) head for the input sequence "The monkey eats a banana".
- similar to concurrent work (Pham et al., 2019), we find that modeling linguistic knowledge into the self-attention mechanism leads to better translations than other approaches.
Our extensive experiments on standard  $\mathrm{En} \leftrightarrow \mathrm{De}$ ,  $\mathrm{En} \rightarrow \mathrm{Tr}$  and  $\mathrm{En} \rightarrow \mathrm{Ja}$  translation tasks also show that (a) approaches to embed syntax in RNNs do not always transfer to the Transformer, and (b) PASCAL consistently exhibits significant improvements in translation quality, especially for long sentences.
In order to design a neural network that is efficient to train and that exploits syntactic information while producing high-quality translations, we base our model on the Transformer architecture (Vaswani et al., 2017) and upgrade its encoder with parent-scaled self-attention (PASCAL) heads at layer  $l_{s}$ . PASCAL heads enforce contextualization from the syntactic dependencies of each source token, and, in practice, we replace standard self-attention heads with PASCAL ones in the first layer as its inputs are word embeddings that lack any contextual information. Our PASCAL sub-layer has the same number  $H$  of attention heads as other layers.
Source syntax Similar to previous work, instead of just providing sequences of tokens, we supply the encoder with dependency relations given by an external parser. Our approach explicitly exploits sub-word units, which enable open-vocabulary translation: after generating sub-word units, we compute the middle position of each word in terms of number of tokens. For instance, if a word in position 4 is split into three tokens, now in positions 6, 7 and 8, its middle position is 7. We then map each sub-word of a given word to the middle position of its parent. For the root word, we define its parent to be itself, resulting in a parse that is a directed
graph. The input to our encoder is a sequence of  $T$  tokens and the absolute positions of their parents.
2.1 Parent-Scaled Self-Attention
Figure 1 shows our parent-scaled self-attention sublayer. Here, for a sequence of length  $T$ , the input to each head is a matrix  $\mathbf{X} \in \mathbb{R}^{T \times d_{model}}$  of token embeddings and a vector  $\mathbf{p} \in \mathbb{R}^T$  whose  $t$ -th entry  $p_t$  is the middle position of the  $t$ -th token's dependency parent. Following Vaswani et al. (2017), in each attention head  $h$ , we compute three vectors (called query, key and value) for each token, resulting in the three matrices  $\mathbf{K}^h \in \mathbb{R}^{T \times d}$ ,  $\mathbf{Q}^h \in \mathbb{R}^{T \times d}$ , and  $\mathbf{V}^h \in \mathbb{R}^{T \times d}$  for the whole sequence, where  $d = d_{model} / H$ . We then compute dot products between each query and all the keys, giving scores of how much focus to place on other parts of the input when encoding a token at a given position. The scores are divided by  $\sqrt{d}$  to alleviate the vanishing gradient problem arising if dot products are large:
$$
\mathbf {S} ^ {h} = \mathbf {Q} ^ {h} \mathbf {K} ^ {h ^ {\top}} / \sqrt {d}. \tag {1}
$$
Our main contribution is in weighing the scores of the token at position  $t$ ,  $\mathbf{s}_t$ , by the distance of each token from the position of  $t$ 's dependency parent:
$$
n _ {t j} ^ {h} = s _ {t j} ^ {h} d _ {t j} ^ {p}, \quad \text {f o r} j = 1, \dots , T, \tag {2}
$$
where  $\mathbf{n}_t^h$  is the  $t$ -th row of the matrix  $\mathbf{N}^h \in \mathbb{R}^{T \times T}$  representing scores normalized by the proximity to  $t$ 's parent.  $d_{tj}^p = dist(p_t, j)$  is the  $(t, j)^{th}$  entry of the matrix  $\mathbf{D}^p \in \mathbb{R}^{T \times T}$  containing, for each row  $\mathbf{d}_t$ , the distances of every token  $j$  from the middle position of token  $t$ 's dependency parent  $p_t$ . In this paper, we compute this distance as the value of the probability density of a normal distribution centered at  $p_t$  and with variance  $\sigma^2$ ,  $\mathcal{N}(p_t, \sigma^2)$ :
$$
\operatorname {d i s t} \left(p _ {t}, j\right) = f _ {\mathcal {N}} \left(j \mid p _ {t}, \sigma^ {2}\right) = \frac {1}{\sqrt {2 \pi \sigma^ {2}}} e ^ {- \frac {\left(j - p _ {t}\right) ^ {2}}{2 \sigma^ {2}}}. \tag {3}
$$
Finally, we apply a softmax function to yield a distribution of weights for each token over all the tokens in the sentence, and multiply the resulting matrix with the value matrix  $\mathbf{V}^h$ , obtaining the final representations  $\mathbf{M}^h$  for PASCAL head  $h$ .
One of the major strengths of our proposal is being parameter-free: no additional parameter is required to train our PASCAL sub-layer as  $\mathbf{D}^p$  is obtained by computing a distance function that only depends on the vector of tokens' parent positions and can be evaluated using fast matrix operations.
Parent ignoring Due to the lack of parallel corpora with gold-standard parses, we rely on noisy annotations from an external parser. However, the performance of syntactic parsers drops abruptly when evaluated on out-of-domain data (Dredze et al., 2007). To prevent our model from overfitting to noisy dependencies, we introduce a regularization technique for the PASCAL sub-layer: parent ignoring. In a similar vein as dropout (Srivastava et al., 2014), we disregard information during the training phase. Here, we ignore the position of the parent of a given token by randomly setting each row of  $\mathbf{D}^p$  to  $\mathbf{1} \in \mathbb{R}^T$  with some probability  $q$ .
Gaussian weighing function The choice of weighing each score by a Gaussian probability density is motivated by two of its properties. First, its bell-shaped curve: It allows us to focus most of the probability density at the mean of the distribution, which we set to the middle position of the sub-word units of the dependency parent of each token. In our experiments, we find that most words in the vocabularies are not split into sub-words, hence allowing PASCAL to mostly focus on the actual parent. In addition, non-negligible weights are placed on the neighbors of the parent token, allowing the attention mechanism to also attend to them. This could be useful, for instance, to learn idiomatic expressions such as prepositional verbs in English. The second property of Gaussian-like distributions that we exploit is their support: While most of the weight is placed in a small window of tokens around the mean of the distribution, all the values in the sequence are actually multiplied by non-zero factors; allowing a token  $j$  farther away from the parent of token  $t$ ,  $p_t$ , to still play a role in the representation of  $t$  if its score  $s_{tj}^h$  is high.
PASCAL can be seen as an extension of the local attention mechanism of Luong et al. (2015), with the alignment now guided by syntactic information.
Yang et al. (2018) proposed a method to learn a Gaussian bias that is added to, instead of multiplied by, the original attention distribution. As we will see next, our model significantly outperforms this.
Data We evaluate the efficacy of our approach on standard, large-scale benchmarks and on low-resource scenarios, where the Transformer was shown to induce poorer syntax. Following Bastings et al. (2017), we use News Commentary v11 (NC11) with En-De and De-En tasks to simulate low resources and test multiple source languages. To compare with previous work, we train our models on WMT16 En-De and WAT En-Ja tasks, removing sentences in incorrect languages from WMT16 data sets. For a thorough comparison with concurrent work, we also evaluate on the largescale WMT17 En-De and low-resource WMT18 En-Tr tasks. We rely on Stanford CoreNLP (Manning et al., 2014) to parse source sentences.
Training We implement our models in PyTorch on top of the Fairseq toolkit. Hyperparameters, including the number of PASCAL heads, that achieved the highest validation BLEU (Papineni et al., 2002) score were selected via a small grid search.
We report previous results in syntax-aware NMT for completeness, and train a Transformer model as a strong, standard baseline. We also investigate the following syntax-aware Transformer approaches:
- +PASCAL: The model presented in §2. The variance of the normal distribution was set to 1 (i.e., an effective window size of 3) as  $99.99\%$  of the source words in our training sets are at most split into 7 sub-words units.
- +LISA: We adapt LISA (Strubell et al., 2018) to NMT and sub-word units by defining the parent of a given token as its first sub-word (which represents the root of the parent word).
- +MULTI-TASK: Our implementation of the multi-task approach by Currey and Heafield (2019) where a standard Transformer learns to both parse and translate source sentences.
- +S&H: Following Sennrich and Haddow (2016), we introduce syntactic information in the form of dependency labels in the embedding matrix of the Transformer encoder.
For a detailed description, see Appendix A.
$^{2}$ https://github.com/e-bug/pascal.
Figure 2: Analysis by sentence length:  $\Delta$ BLEU with the Transformer (above) and percentage of data (below).
Method	NC11
En-De	NC11
De-En	WMT18
En-Tr	WMT16
En-De	WMT17
En-De	WAT
En-Ja [B]	En-Ja [R]
Eriguchi et al. (2016)	34.9	81.58
Bastings et al. (2017)	16.1
Hashimoto and Tsuruoka (2017)	39.4	82.83
Bisk and Tran (2018)	30.3	24.3
SE+SD-NMT†(Wu et al., 2018)	24.7	36.4	81.83
SE+SD-Transformer†(Wu et al., 2018)	26.2
Mixed Enc. (Currey and Heafield, 2019)	9.6	31.9	26.0
Multi-Task (Currey and Heafield, 2019)	10.6	29.6	23.4
Transformer	25.0	26.6	13.1	33.0	25.5	43.1	83.46
+ PASCAL	25.9↑	27.4↑	14.0↑	33.9↑	26.1↑	44.0↑	85.21↑
+ LISA	25.3	27.1	13.6	33.6	25.7	43.2	83.51
+ MULTI-TASK	24.8	26.7	14.0	32.4	24.6	42.7	84.18
+ S&H	25.5	26.8	13.0	31.9	25.1	42.8	83.88
Table 1: Test BLEU (and RIBES for En-Ja) scores on small-scale (left) and large-scale (right) data sets. Models that also require target-side syntax information are marked with  $\dagger$ , while  $\uparrow$  indicates statistical significance ( $p < 0.01$ ) against the Transformer baseline via bootstrap re-sampling (Koehn, 2004).
Table 1 presents the main results of our experiments. Clearly, the base Transformer outperforms previous syntax-aware RNN-based approaches, proving it to be a strong baseline in our experiments. The table shows that the simple approach of Sennrich and Haddow (2016) does not lead to notable advantages when applied to the embeddings of the Transformer model. We also see that the multi-task approach benefits from better parameterization, but it only attains comparable performance with the baseline on most tasks. On the other hand, LISA, which embeds syntax in a self-attention head, leads to modest but consistent gains across all tasks, proving that it is also useful for NMT. Finally, PASCAL outperforms all other methods, with consistent gains over the Transformer baseline independently of the source language and corpus size: It gains up to  $+0.9$  BLEU points on most tasks and a substantial  $+1.75$  in RIBES (Isozaki et al., 2010), a metric with stronger correlation with hu
man judgments than BLEU in  $\mathrm{En}\leftrightarrow \mathrm{Ja}$  translations. On WMT17, our slim model compares favorably to other methods, achieving the highest BLEU score across all source-side syntax-aware approaches. $^3$
Overall, our model achieves substantial gains given the grammatically rigorous structure of English and German. Not only do we expect performance gains to further increase on less rigorous sources and with better parses (Zhang et al., 2019), but also higher robustness to noisier syntax trees obtained from back-translated with parent ignoring.
Performance by sentence length As shown in Figure 2, our model is particularly useful when translating long sentences, obtaining more than  $+2$  BLEU points when translating long sentences in all low-resource experiments, and  $+3.5$  BLEU points on the distant En-Ja pair. However, only a few sentences  $(1\%)$  in the evaluation datasets are long.
<sup>3</sup>Note that modest improvements in this task should not be surprising as Transformers learn better syntactic relationships from larger data sets (Raganato and Tiedemann, 2018).
SRC
BASE
OURS	In a cooling experiment, only a tendency agreed
冷却実験のは,わざかな傾向が一致,Thed
冷却実験のは傾向のひ一致,Thed
SRC
BASE
OURS	Of course I don't hate you
Natuürlich hasste ich dich nicht
Natuürlich hasse ich dich nicht
SRC
BASE
OURS	What are those people fighting for?
Was sind die Menschen, für die kämpfen?
Wofür kämpfen diese Menschen?
Qualitative performance Table 2 presents examples where our model correctly translated the source sentence while the Transformer baseline made a syntactic error. For instance, in the first example, the Transformer misinterprets the adverb "only" as an adjective of "tendency:" the word "only" is an adverb modifying the verb "agreed." In the second example, "don't" is incorrectly translated to the past tense instead of present.
PASCAL layer When we introduced our model, we motivated our design choice of placing PASCAL heads in the first layer in order to enrich the representations of words from their isolated embeddings by introducing contextualization from their parents. We ran an ablation study on the NC11 data in order to verify our hypothesis. As shown in Table 3a, the performance of our model on the validation sets is lower when placing Pascal heads in upper layers; a trend that we also observed with the LISA mechanism. These results corroborate the findings of Raganato and Tiedemann (2018) who noticed that, in the first layer, more attention heads solely focus on the word to be translated itself rather than its context. We can then deduce that enforcing syntactic dependencies in the first layer effectively leads to better word representations, which further enhance the translation accuracy of the Transformer model. Investigating the performance of multiple syntax-aware layers is left as future work.
Gaussian variance Another design choice we made was the variance of the Gaussian weighing function. We set it to 1 in our experiments motivated by the statistics of our datasets, where the vast majority of words is at most split into a few tokens after applying BPE. Table 3b corroborates our choice, showing higher BLEU scores on the NC11 validation sets when the variance equals 1. Here, "parent-only" is the case where weights are only placed to the middle token (i.e. the parent).
Table 2: Example of correct translation by PASCAL.
Layer	En-De	De-En
1	23.2	24.6
2	22.5	20.1
3	22.5	23.8
4	22.6	23.8
5	22.9	23.8
6	22.4	23.9
(a)
Variance	En-De	De-En
Parent-only	22.5	22.4
1	23.2	24.6
4	22.7	24.3
9	22.8	24.3
16	22.7	24.4
25	22.8	24.1
Table 3: Validation BLEU as a function of PASCAL layer (a) and Gaussian's variance (b) on NC11 data.
Sensitivity to hyperparameters Due to the large computational cost required to train Transformer models, we only searched hyperparameters in a small grid. In order to estimate the sensitivity of the proposed approach to hyperparameters, we trained the NC11 De-En model with the hyperparameters of the En-De one. In fact, despite being trained on the same data set, we find that more PASCAL heads help when German (which has a higher syntactic complexity than English) is used as the source language. In this test, we only find  $-0.2$  BLEU points with respect to the score listed in Table 1, showing that our general approach is effective regardless of extensive fine-tuning.
Additional analyses are reported in Appendix B.
This study provides a thorough investigation of approaches to induce syntactic knowledge into self-attention networks. Through extensive evaluations on various translation tasks, we find that approaches effective for RNNs do not necessarily transfer to Transformers (e.g. +S&H). Conversely, dependency-aware self-attention mechanisms (LISA and PASCAL) best embed syntax, for all corpus sizes, with PASCAL consistently outperforming other all approaches. Our results show that exploiting core components of the Transformer to embed linguistic knowledge leads to higher and consistent gains than previous approaches.
We are grateful to the anonymous reviewers, Desmond Elliott and the CoAStaL NLP group for their constructive feedback. The research results have been achieved by "Research and Development of Deep Learning Technology for Advanced Multilingual Speech Translation," the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph Convolutional Encoders for Syntax-aware Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967. Association for Computational Linguistics.
Yonatan Bisk and Ke Tran. 2018. Inducing Grammars with and for Neural Machine Translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 25-35, Melbourne, Australia. Association for Computational Linguistics.
Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1936-1945. Association for Computational Linguistics.
Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. 2018. Syntax-directed attention for neural machine translation. In AAAI Conference on Artificial Intelligence.
Anna Currey and Kenneth Heafield. 2019. Incorporating source syntax into transformer-based neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 24-33, Florence, Italy. Association for Computational Linguistics.
Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, João Graca, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 1051-1055, Prague, Czech Republic. Association for Computational Linguistics.
Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science, 14(2):179-211.
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-Sequence Attentional Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 823-833, Berlin, Germany. Association for Computational Linguistics.
Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to Parse and Translate Improves Neural Machine Translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 72-78. Association for Computational Linguistics.