SlowGuess/ACL_2020_OCR huggingface dataset:
The En-Fr task used a larger dataset ( $\sim 36M$  sentence pairs) and achieved a higher baseline BLEU than the En-De task, we suggest significant improvements obtained by our approach on the En-Fr task with the Transformer Big supports the effectiveness of our approach in challenging settings.
We also conducted a Transformer Base based ablation study on the WMT 14 En-De task to assess the influence of phrase representation, attention mechanism in phrase representation generation, transparent attention and phrases from parser output on performance. Results are shown in Table 2.
“+Mean” and “+Max” are only using elementwise mean operation and max-over-time pooling to generate an initial rough phrase representation of a given token sequence. “+Attn” indicates generating phrase representations with our attentive approach, on top of the max-over-time pooling as  $F_{\text{glance}}$  in Equation 2. “+TA” indicates use of the Transparent Attention mechanism to fuse information generated from every encoder layer for different decoder layers, otherwise only outputs of the last encoder layer are fed into all decoder layers. “+Parse” means using phrases extracted from parse results with Algorithm 1.
Table 2 shows that introducing phrase representation can significantly improve the strong Transformer Base baseline, even only with a simple element-wise mean operation over token repre
sentations brings about a  $+0.61$  BLEU improvement  $(p < 0.01)$ . Summarizing representations with max-over-time pooling performs slightly better than with the element-wise mean operation. Our attentive phrase representation generation approach can bring further improvements over the max-over-time pooling approach. Though utilizing phrases from the parser can make use of linguistic knowledge and obtains most improvements, our simple and effective segmenting approach performs competitively, and we interpret these comparisons to show the positive effects of collapsing token sequences into shorter phrase sequences on the modeling of long-distance dependencies.
Though a significant amount of parameters are introduced for incorporating phrase representation into the Transformer model, our approach ("+Max+Attn+TA") improved the performance of the Transformer Base model by +1.29 BLEU on the WMT 14 En-De news task, and the proposed Transformer model with phrase representation still performs competitively compared to the Transformer Big model with only about half the number of parameters and  $1/3$  of the training steps. Thus, we suggest our improvements are not only because of introducing parameters, but also due to the modeling and utilization of phrase representation.
4.4 Length Analysis
To analyze the effects of our phrase representation approach on performance with increasing input length, we conducted a length analysis on the news test set of the WMT 14 En-De task. Following Bahdanau et al. (2015) and Tu et al. (2016), we grouped sentences of similar lengths together and computed BLEU scores of Transformers and Transformers
This only introduces an additional  $7*6$  parameter matrix, which does not show significant influence in view of the amount of parameters.
Figure 3: BLEU scores with respect to various input sentence lengths.
with phrase representations for each group. Results are shown in Figure 3.
Figure 3 shows that our approach incorporating phrase representation into the Transformer significantly improves its performance in all length groups, and longer sentences show significantly more improvements than shorter sentences. In the Transformer Base setting, our approach improved the group with sentences of more than 45 tokens by +1.72 BLEU, almost twice of the improvements for sentences with less than 15 tokens which was +0.93 BLEU.
The effects of incorporating phrase representations into the Transformer is more significant especially when compared to the Transformer Big which has about twice the number of parameters than our approach and consumes 3 times the training steps. According to Tang et al. (2018), the number of attention heads in Transformers impacts their ability to capture long-distance dependencies, and specifically, many-headed multi-head attention is essential for modeling long-distance phenomena with only self-attention. The Transformer Big model with twice the number of heads in the multi-head attention network compared to those in the Transformer Base model, should be better at capturing long-distance dependencies. However, comparing with the Transformer Base, the improvement of the Transformer Big on long sentences (+1.20 BLEU for sentences with more than 45 tokens) was similar to that on short sentences (+1.14 BLEU for sentences with no more than 15 tokens), while our approach to model phrases in the Transformer model even brings significantly  $(p < 0.01)$  more improvements (+1.72 BLEU) on the performance of longer sentences with the Transformer Base setting (8 heads) than the Transformer Big with 16
Figure 4: Subject-Verb Agreement Analysis. X-axis and y-axis represent subject-verb distance in words and the accuracy respectively.
heads  $(+1.20$  BLEU).
The length analysis result is consistent with our conjecture to some extent given that there are likely to be more long-distance dependencies in longer source sentences. We suggest that phrase sequences which are shorter than corresponding token sequences can help the model capture long-distance dependencies better, and modeling phrase representations for the Transformer can enhance its performance on long sequences.
4.5 Subject-Verb Agreement Analysis
Intuitively, in translating longer sentences we should encounter more long-distance dependencies than in short sentences. To verify whether our method can improve the capability of the NMT model to capture long-distance dependencies, we also conducted a linguistically-informed verb-subject agreement analysis on the Lingeval97 dataset (Sennrich, 2017) following Tang et al. (2018).
In German, subjects and verbs must agree with one another in grammatical number and person. In Lingeval97, each contrastive translation pair consists of a correct reference translation, and a contrastive example that has been minimally modified to introduce one translation error. The accuracy of a model is the number of times it assigns a higher score to the reference translation than to the contrastive one, relative to the total number of predictions. Results are shown in Figure 4.
Figure 4 shows that our approach can improve the accuracy of long-distance subject-verb dependencies, especially for cases where there are more than 10 tokens between the verb and the corresponding subject when comparing the "Base+PR"
with the "Transformer Big".
Considering that the strong Transformer translation model still has difficulty in fully capturing long-distance dependencies (Tang et al., 2018), and that using a shorter phrase sequence (in addition to the original token sequence) is an intuitive approach to help the model capture long-distance features, in this paper, we first propose an attention mechanism to generate phrase representations by merging corresponding token representations. In addition, we incorporate the generated phrase representations into the Transformer translation model to help it capture long-distance relationships. We obtained statistically significant improvements on the WMT 14 English-German and English-French tasks over the strong Transformer baseline, which demonstrates the effectiveness of our approach. Our further analysis shows that the Transformer with phrase representation empirically improves its performance especially in long-distance dependency learning.
We thank anonymous reviewers for their insightful comments and helpful advice. Hongfei Xu acknowledges the support of China Scholarship Council ([2018]3101, 201807040056). Deyi Xiong is supported by the National Natural Science Foundation of China (Grant No. 61861130364), the Natural Science Foundation of Tianjin (Grant No. 19JCZDJC31400) and the Royal Society (London) (NAF\R1\180122). Hongfei Xu, Josef van Genabith and Jingyi Zhang are supported by the German Federal Ministry of Education and Research (BMBF) under the funding code 01IW17001 (Deeplee).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ankur Bapna, Mia Chen, Orhan First, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
3028-3033. Association for Computational Linguistics.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 263-270, Ann Arbor, Michigan. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Leonard Dahlmann, Evgeny Matusov, Pavel Petrushkov, and Shahram Khadivi. 2017. Neural machine translation leveraging phrase-based models in a hybrid search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1411-1420, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4253-4262, Brussels, Belgium. Association for Computational Linguistics.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation with routing-by-agreement. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, pages 86-93.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine
Learning Research, pages 1243-1252, International Convention Centre, Sydney, Australia. PMLR.
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019. Multi-granularity self-attention for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 887-897, Hong Kong, China. Association for Computational Linguistics.
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with SMT features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 151-157.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 127-133.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376-382, Valencia, Spain. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725. Association for Computational Linguistics.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu. 2018. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers), pages 1294-1303, New Orleans, Louisiana. Association for Computational Linguistics.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455-465, Sofia, Bulgaria. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4263–4272, Brussels, Belgium. Association for Computational Linguistics.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-85, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. Multi-layer representation fusion for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3015-3026, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. 2017a. Neural machine translation advised by statistical machine translation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pages 3330-3336.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019. Self-attention with structural position representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1403–1409, Hong Kong, China. Association for Computational Linguistics.
Xing Wang, Zhaopeng Tu, Deyi Xiong, and Min Zhang. 2017b. Translating phrases in neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1421-1431, Copenhagen, Denmark. Association for Computational Linguistics.
Hongfei Xu and Qiuhui Liu. 2019. Neutron: An Implementation of the Transformer Translation Model and its Variants. arXiv preprint arXiv:1903.07402.