SlowGuess/ACL_2020_OCR huggingface dataset:
Kazuma Hashimoto and Yoshimasa Tsuruoka. 2017. Neural Machine Translation with Source-Side Latent Graph Parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 125-135. Association for Computational Linguistics.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944-952, Cambridge, MA. Association for Computational Linguistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL '07, pages 177-180, Stroudsburg, PA, USA. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
Thuong Hai Pham, Dominik Machacek, and Ondrej Bojar. 2019. Promoting the knowledge of source syntax in transformer nmt is not needed. Computación y Sistemas, 23(3).
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium. Association for Computational Linguistics.
Danielle Saunders, Felix Stahlberg, Adrià de Gispert, and Bill Byrne. 2018. Multi-representation ensembles and delayed SGD updates improve syntax-based NMT. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 319-325, Melbourne, Australia. Association for Computational Linguistics.
Rico Sennrich and Barry Haddow. 2016. Linguistic Input Features Improve Neural Machine Translation. In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, pages 83-91, Berlin, Germany. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Lei Shi, Cheng Niu, Ming Zhou, and Jianfeng Gao. 2006. A DOM tree alignment model for mining parallel data from the web. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 489-496, Sydney, Australia. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-Informed Self-Attention for Semantic Role Labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027-5038. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translation architectures. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, pages 4263-4272, Brussels, Belgium. Association for Computational Linguistics.
Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731-4736, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for Neural Machine Translation. CoRR, abs/1803.07416.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Shuangzhi Wu, Dongdong Zhang, Zhirui Zhang, Nan Yang, Mu Li, and Ming Zhou. 2018. Dependency-to-dependency neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 2132-2141.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Baosong Yang, Jian Li, Derek F. Wong, Lidia S. Chao, Xing Wang, and Zhaopeng Tu. 2019. Context-aware self-attention networks. In AAAI Conference on Artificial Intelligence.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4449-4458, Brussels, Belgium. Association for Computational Linguistics.
Meishan Zhang, Zhenghua Li, Guohong Fu, and Min Zhang. 2019. Syntax-enhanced neural machine translation with syntax-aware word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1151-1161, Minneapolis, Minnesota. Association for Computational Linguistics.
Corpus	Train	Filtered Train	Valid	Test
NC11 En-De	238,843	233,483	2,169	2,999
WMT18 En-Tr	207,373	3,000	3,007
WMT16 En-De	4,500,962	4,281,379	2,169	2,999
WMT17 En-De	5,852,458	2,999	3,004
WAT En-Ja	3,008,500	1,790	1,812
Table 4: Number of sentences in each data set.
Data preparation We follow the same preprocessing steps as Vaswani et al. (2017). Unless otherwise specified, we first tokenize the data with Moses (Koehn et al., 2007) and remove sentences longer than 80 tokens in either source or target side.
Following Bastings et al. (2017), we train on News Commentary v11 (NC11) data set with English  $\rightarrow$  German (En-De) and German  $\rightarrow$  English (De-En) tasks so as to simulate low-resource cases and to evaluate the performance of our models for different source languages. We also train on the full WMT16 data set for En-De, using newstest2015 and newstest2016 as validation and test sets, respectively, in each of these experiments. Moreover, we notice that these data sets contain sentences in different languages and use langdetect<sup>4</sup> to remove sentences in incorrect languages.
We also train our models on WMT18 English  $\rightarrow$  Turkish (En-Tr) as a standard lowresource scenario. Models are evaluated on newstest2016 and tested on newstest2017.
Previous studies on syntax-aware NMT have commonly been conducted on the WMT16 En-De and WAT English  $\rightarrow$  Japanese (En-Ja) tasks, while concurrent approaches are evaluated on the WMT17 En-De task. In order to provide a generic and comprehensive evaluation of our proposed approach on large-scale data, we also train our models on the latter tasks. We follow the WAT18 preprocessing steps for experiments on En-Ja but use Cabocha to tokenize target sentences. On WMT17, we use newstest2016 and newstest2017 as validation and test sets, respectively.
Table 4 lists the final sizes of each data set.
Baselines We evaluate the impact of syntactic information with the following approaches:
- Transformer: We train a base Transformer
Figure 3: Weights of normal probability density with  $\sigma^2 = 1$  and the means at positions 5 (left) or 4.5 (right).
model as a strong, standard baseline using the hyperparameters in the latest Tensor2Tensor (Vaswani et al., 2018) version (3).
- +S&H: Following Senrich and Haddow (2016), we introduce syntactic information in the form of dependency labels in the embedding matrix of the Transformer encoder. More specifically, each token is associated with its dependency label which is first embedded into a vector representation of size 10 and then used to replace the last 10 embedding dimensions of the token embedding, ensuring a final size that matches the original one.
- +MULTI-TASK: Our implementation of the multi-task approach by Currey and Heafield (2019) where a standard Transformer learns to both parse and translate source sentences. Each source sentence is first duplicated and associated its linearized parse as target sequence. To distinguish between the two tasks, a special tag indicating the desired task is prepended and appended to each source sentence. Finally, parsing and translation training data is shuffled together.
- +LISA: We adapt Linguistically-Informed Self-Attention (LISA; Strubell et al. 2018) to NMT. In one attention head  $h$ ,  $\mathbf{Q}^h$  and  $\mathbf{K}^h$  are computed through a feed-forward layer and the key-query dot product to obtain attention weights is replaced by a bi-affine operator  $\mathbf{U}$ . These attention weights are further supervised to attend to each token's parent by interpreting each row  $t$  as the distribution over possible parents for token  $t$ . Here, we extend the authors' approach to BPE by defining the parent of a given token as its first sub-word unit (which represents the root of the parent word). The model is trained to maximize the joint probability of translations and parent positions.
$^{4}$ https://pypi.org/project/langdetect.  $^{5}$ http://lotus.kuee.kyoto-u.ac.jp/WAT/WAT2018/baseline/dataPreparationJE.html.  $^{6}$ https://taku910.github.io/cabocha/.
Component	NC11 En-De	NC11 De-En	WMT18 En-Tr	WMT16 En-De	WMT17 En-De	WAT En-Ja
Transformer	22.6	23.8	12.6	29.0	31.5	42.2
+ data filtering	22.8 (+0.2)	24.0 (+0.2)	28.7 (-0.3)
+ PASCAL	23.0 (+0.2)	24.6 (+0.6)	13.6 (+1.0)	29.2 (+0.5)	31.6 (+0.1)	43.5 (+1.3)
+ parent ignoring	23.2 (+0.2)	13.7 (+0.1)	32.1 (+0.6)
Table 5: Validation BLEU when incrementally adding each component used by our best-performing models.
Corpus	Transformer	+PASCAL
NC11	En-De	4,134.1	4,188.8
NC11	De-En	4,276.6	4,177.4
WMT18	En-Tr	3,559.7	3,621.1
WMT16	En-De	23,186.3	23,358.8
WMT17	En-De	23,604.1	24,083.6
WAT	En-Ja	23,005.8	23,073.0
Table 6: Training times (in seconds) for the Transformer baseline and Transformer+PASCAL on each data set. PASCAL adds negligible overhead.
Corpus	lr	(β1,β2)	hP	q
NC11	En-De	0.0007	(0.9, 0.997)	2	0.4
NC11	De-En	0.0007	(0.9, 0.997)	8	0.0
WMT18	En-Tr	0.0007	(0.9, 0.980)	7	0.3
WMT16	En-De	0.0007	(0.9, 0.980)	5	0.0
WMT17	En-De	0.0007	(0.9, 0.997)	7	0.3
WAT	En-Ja	0.0007	(0.9, 0.997)	7	0.0
Table 7: Hyperparameters for the reported models.  $lr$  denotes the maximum learning rate,  $(\beta_{1},\beta_{2})$  are Adam's decay rates,  $h_P$  is the number of PASCAL heads, and  $q$  is the parent ignoring probability.
Training details All experiments are based on the base Transformer architecture and optimized following the learning schedule of Vaswani et al. (2017) with 8,000 warm-up steps. Similarly, we use label smoothing  $\epsilon_{ls} = 0.1$  (Szegedy et al., 2016) during training and employ beam search with a beam size of 4 and length penalty  $\alpha = 0.6$  (Wu et al., 2016) at inference time. We use a batch size of  $32K$  tokens and run experiments on a cluster of 4 machines, each having 4 Nvidia P100 GPUs. See Table 6 for the training times of each experiment.
For each model, we run a small grid search over the hyperparameters and select the ones giving the highest BLEU scores on validation sets (Table 7).
We use the SACREBLEU (Post, 2018) tool to compute case-sensitive BLEU scores. When evaluating En-Ja translations, we follow the procedure employed at WAT by computing BLEU scores after tokenizing target sentences using KyTea.
Following Vaswani et al. (2017), we train Transformer-based models for  $100K$  steps on large-scale data. On small-scale data, we train for  $20K$  steps and use a dropout probability  $P_{drop} = 0.3$  as they let the Transformer baseline achieve higher performance on this size of data. For instance, in WMT18 En-Tr, our baseline outperforms the one in Currey and Heafield (2019) by +3.5 BLEU.
Multiplication vs. addition In Equation (2), we calculated the weighing scores by multiplying the self-attention scores by the distance to the parent token. Multiplication is, in fact, the standard way to weight values (e.g., the gating mechanism of LSTMs and GRUs). In our case, it introduces sparseness in the attention scores for non-parent tokens. Moreover, it weights gradients in backpropagation: Let  $x$  and  $y$  be the attention score and dependency weight, respectively. Consider a loss  $l = f(z)$  where  $z = xy$  and  $dl/dx = df(z)/dz*y$ . The attention score receive gradients more on dependent pairs (larger  $y$ ) than non-dependent ones (smaller  $y$ ), which is sound for dependency information. In contrast, addition cannot obtain such an effect because it does not affect gradients:  $dl/dx = df(z)/dz$  when  $z = x + y$ . For completeness, we trained our best NC11 models replacing multiplication by addition. We find that BLEU scores still improve upon the baseline, meaning that our approach is robust, but find them to be slightly lower  $(-0.2)$  than with multiplication.
Ablation We introduced different techniques to improve neural machine translation with syntax information. Table 5 lists the contribution of each technique, in an incremental fashion, whenever they were used by the models reported in Table 1.
While removing sentences whose languages do not match the translation task can lead to better performance (NC11), the precision of the detection tool assumes a major role at large scale. In WMT16, langdetect removes more than  $200K$  sentences and leads to performance losses. It would
$^{7}$ Signature: BLEU+c.mixed+#.1+s.exp+tok.13a+v.1.2.12.
<sup>8</sup>http://www.phontron.com/kytea/.
also drop  $19K$  pairs on the clean WAT En-Ja data.
The proposed PASCAL mechanism is the component that most improves the performance of the models, achieving up to  $+1.0$  and  $+1.3$  BLEU on the distant En-Tr and En-Ja pairs, respectively. With the exception of NC11 En-De, we find parent ignoring useful on the noisier WMT18 En-Tr and WMT17 En-De datasets. In the former, low-resource case, the benefits of parent ignoring are minimal, but it proves fundamental on the largescale WMT17 data, where it leads to significant gains when paired with the PASCAL mechanism.[9]
Finally, looking at the number of PASCAL heads in Table 7, we notice that most models rely on a large number of syntax-aware heads. Raganato and Tiedemann (2018) found that only a few attention heads per layer encoded a significant amount of syntactic dependencies. Our study shows that the Transformer model can be improved by having more attention heads learn syntactic dependencies.
Note that this ablation is obtained by stripping away each component from the best performing models and hence only seeing  $+0.1$  for PASCAL on WMT17 En-De does not mean that PASCAL is not helpful in this task but rather that combining it with parent ignoring gives better performance (our second best model achieved  $+0.5$  by using PASCAL only).