SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4: Effects of encoder/decoder depth on the WMT 14 En-De task. The decoding time is for the test set of 3,003 sentences with a beam size of 4.
Layer	Encoder	Decoder
Acc	Δ	Acc	Δ	Self attention	-Cross attention
Acc	Δ	Acc	Δ
0	40.48	14.04
1	41.29	0.81	37.42	23.38	25.56	-11.86	20.40	-17.02
2	43.00	1.71	68.77	31.35	62.01	-6.76	40.67	-28.10
3	44.07	1.07
4	45.86	1.79
5	46.54	0.68
6	47.46	0.92
7	48.92	1.46
8	49.58	0.66
9	50.24	0.66
10	50.35	0.11
Table 5: Word accuracy analysis on Transformer with 10 encoder and 2 decoder layers on the WMT 14 En-De task.
Transformer with 10 encoder layers and 2 decoder layers is 2.32 times as fast as the 6-layer Transformer while achieving a slightly higher BLEU.
Can we use more than 12 encoder layers with a shallow decoder to benefit both translation quality and inference speed? Table 4 shows that the 18-4 model  ${}^{4}$  brings about +1.42 BLEU improvements over the strong baseline, while being 1.38 times as fast in decoding. Comparing the 18-4 model to the 8-4 model, the time cost for using 10 more encoder layers only increases 1 second for translating the test set, suggesting that autoregressive decoding
speed is quite insensitive to the encoder depth.
Our results show that using more encoder layers with fewer but sufficient decoder layers can significantly boost the decoding speed with small gains in translation quality, and that a good choice in the distribution of encoder and decoder layers (18-4) can result in slightly faster decoding and a substantial increase in translation quality, which is simple but effective and valuable for back-translation (Sennrich et al., 2016a) and production applications.
We present the word accuracy analysis results of the 10 encoder layer - 2 decoder layer Transformer on the En-De task in Table 5. Comparing Table 5 with Table 1, we find that: 1) The differences in improvements (1.71 vs. 0.11) brought by individual layers of the 10-layer encoder are larger than those of the 6-layer encoder (1.90 vs. 0.87), indi
4 A full grid search over configurations is tedious and expensive. We take inspiration from Table 4 where going from 5 to 4 decoder layers brings about the biggest relative jump in translation quality. We explored a few configurations and find that using more than 18 encoder layers can still bring improvements, but the gains are relatively small.
Depth	En-De	En-Fr	Cs-En
Encoder	Decoder
6	27.96	40.13	28.69
10	2	28.47	40.49	28.87
18	4	\( 29.38^{\dagger} \)	\( 40.90^{\dagger} \)	\( 29.75^{\dagger} \)
Table 6: Verification of deep encoder and shallow decoder on WMT En-De, En-Fr and Cs-En tasks.  $\dagger$  indicates significance at  $p < 0.01$ .
cating that there might now be some "lazy" layers in the 10-layer encoder; 2) Decreasing the depth of the decoder removes "lazy" decoder layers in the 6-layer decoder and makes decoder layers rely more on the source "encoding" (by comparing the effects of skipping the self-attention sub-layer and cross-attention sub-layer on performance).
4.3 Verification of Deep Encoder and Shallow Decoder on other Language Pairs
To investigate how a deep encoder with a shallow decoder will perform in other tasks, we conducted experiments on the WMT 14 English-French and WMT 15 Czech-English news translation tasks in addition to the WMT 14 English-German task. Results on newstest 2014 (En-De/Fr) and 2015 (Cs-En) respectively are shown in Table 6.
Table 6 shows that the 10-2 model consistently achieves higher BLEU scores than the 6-layer model, and the 18-4 model consistently leads to significant improvements in all 3 tasks.
Analysis of NMT Models. Belinkov et al. (2020) analyze the representations learned by NMT models at various levels of granularity and evaluate their quality through relevant extrinsic properties. Li et al. (2019a) analyze the word alignment quality in NMT and the effect of alignment errors on translation errors. They demonstrate that NMT captures word alignment much better for those words mostly contributed from the source than those from the target. Voita et al. (2019b) evaluate the contribution of individual attention heads to the overall performance of the model and analyze the roles played by them in the encoder. Yang et al. (2019) propose a word reordering detection task to quantify how well the word order information is learned by Self-Attention Networks and RNN, and reveal that although recurrence structure makes the model more universally effective on learning word order,
learning objectives matter more in the downstream tasks such as machine translation. Tsai et al. (2019) regard attention as applying a kernel smoother over the inputs with the kernel scores being the similarities between inputs, and analyze individual components of the Transformer's attention with the new formulation via the lens of the kernel. Tang et al. (2019) find that encoder hidden states outperform word embeddings significantly in word sense disambiguation. He et al. (2019) measure the word importance by attributing the NMT output to every input word and reveal that words of certain syntactic categories have higher importance while the categories vary across language pairs. Voita et al. (2019a) use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers. Early work by Bisazza and Tump (2018) performs a fine-grained analysis of how various source-side morphological features are captured at different levels of the NMT encoder. While they are unable to find any correlation between the accuracy of source morphology encoding and translation quality, they discover that morphological features are only captured in context and only to the extent that they are directly transferable to the target words, and suggest encoder layers are "lazy". Our analysis offers an explanation for their results as the translation already starts at the source embedding layer, and possibly source embeddings already represent linguistic features of their translations.
Analysis of BERT. BERT (Devlin et al., 2019) uses the Transformer encoder, and analysis of BERT may provide valuable references for analyzing the Transformer. Jawahar et al. (2019) provide support that BERT networks capture structural information, and perform a series of experiments to unpack the elements of English language structure learned by BERT. Tenney et al. (2019) employ the edge probing task suite, and find that BERT represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Pires et al. (2019) present a large number of probing experiments, and show that Multilingual-BERT's robust ability to generalize cross-lingually is underpinned by a multilingual representation.
Accelerating Decoding. Zhang et al. (2018a) propose average attention as an alternative to the self-attention network in the Transformer decoder to accelerate decoding. Wu et al. (2019) introduce lightweight convolution and dynamic convolutions. The number of operations required by their approach scales linearly in the input length, whereas self-attention is quadratic. Zhang et al. (2018b) apply cube pruning to neural machine translation to speed up translation. Zhang et al. (2018c) propose to adopt an n-gram suffix-based equivalence function into beam search decoding, which obtains similar translation quality with a smaller beam size, making NMT decoding more efficient. Non-Autoregressive Translation (NAT) (Gu et al., 2018; Libovicky and Helcl, 2018; Wei et al., 2019; Shao et al., 2019; Li et al., 2019b; Wang et al., 2019; Guo et al., 2019) enables parallelized decoding, while there is still a significant quality drop compared to traditional autoregressive beam search, our findings on using more encoder layers might also be adapted to NAT. Recently, and independently of our work, Kasai et al. (2021) compare the performance and speed between a 12-layer encoder 1-layer decoder case with NAT approaches, and show that a one-layer autoregressive decoder yields state-of-the-art accuracy with comparable latency to strong non-autoregressive models. Our work explains why using a deep encoder with a shallow decoder is feasible, and we show that substantial increases in decoding speed are possible with small gains in translation quality, and that for some configurations (e.g., 18-4) significant translation quality increases with modest increases in decoding speed are possible.
We propose approaches for the analysis of word translation accuracy of Transformer layers to investigate how translation is performed. To measure word translation accuracy, our approach trains a linear projection layer that bridges representations from the frozen pre-trained analyzed layer and the frozen pre-trained classifier. While analyzing encoder layers, our approach additionally learns a weight vector to merge multiple attention matrices into one, and transforms the source "encoding" to the target shape by multiplying the merged alignment matrix. Both the linear projection layer and the weight vector are trained on the frozen Transformer. This is minimally invasive, and training the
new parameters does not account for the findings reported. For the analysis of decoder layers, we additionally analyze the effects of the source context and the decoding history in word prediction through bypassing the corresponding cross- and self-attention sub-layers. Our findings motivate and explain the benefits of trading decoder for encoder layers in our approach and that of Kasai et al. (2021).
Our analysis is the first to reveal the existence of target translations performed by encoder layers (including the source embedding layer). We show that increasing encoder depth while removing decoder layers can lead to significant BLEU improvements while boosting the decoding speed.
We thank anonymous reviewers for their insightful comments. Hongfei Xu acknowledges the support of China Scholarship Council ([2018]3101, 201807040056). Josef van Genabith is supported by the German Federal Ministry of Education and Research (BMBF) under funding code 01IW20010 (CORA4NLP). Deyi Xiong is supported by the National Natural Science Foundation of China (Grant No. 61861130364), the Natural Science Foundation of Tianjin (Grant No. 19JCZDJC31400) and the Royal Society (London) (NAF\R1\180122).
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. Computational Linguistics, 46(1):1-52.
Arianna Bisazza and Clara Tump. 2018. The lazy encoder: A fine-grained analysis of the role of morphology in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2871-2876, Brussels, Belgium. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252, International Convention Centre, Sydney, Australia. PMLR.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Nonautoregressive neural machine translation. In International Conference on Learning Representations.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019. Non-autoregressive neural machine translation with enhanced decoder input. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 3723-3730. AAAI Press.
Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019. Towards understanding neural machine translation with word importance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 953-962, Hong Kong, China. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Int. Res., 61(1):907-926.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. 2021. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In International Conference on Learning Representations.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019a. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293-1303, Florence, Italy. Association for Computational Linguistics.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019b. Hint-based training for non-autoregressive machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5708-5713, Hong Kong, China. Association for Computational Linguistics.
Jindrich Libovicky and Jindrich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. 2019. Retrieving sequential information for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3013-3024, Florence, Italy. Association for Computational Linguistics.
Gongbo Tang, Rico Senrich, and Joakim Nivre. 2019. Encoders help you disambiguate word senses in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1429-1435, Hong Kong, China. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4344-4353, Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 4396-4406, Hong Kong, China. Association for Computational Linguistics.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019b. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. arXiv preprint arXiv:2003.12298.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 5377-5384. AAAI Press.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312, Florence, Italy. Association for Computational Linguistics.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
Hongfei Xu and Qiuhui Liu. 2019. Neutron: An Implementation of the Transformer Translation Model and its Variants. arXiv preprint arXiv:1903.07402.
Hongfei Xu, Qiuhui Liu, Josef van Genabith, Deyi Xiong, and Jingyi Zhang. 2020. Lipschitz constrained parameter initialization for deep transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 397-402, Online. Association for Computational Linguistics.
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019. Assessing the ability of self-attention networks to learn word order. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3635-3644, Florence, Italy. Association for Computational Linguistics.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018a. Accelerating neural transformer via an average attention network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1789-1798, Melbourne, Australia. Association for Computational Linguistics.
Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359-361, Brussels, Belgium. Association for Computational Linguistics.
Wen Zhang, Liang Huang, Yang Feng, Lei Shen, and Qun Liu. 2018b. Speeding up neural machine translation decoding by cube pruning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4284-4294, Brussels, Belgium. Association for Computational Linguistics.
Zhisong Zhang, Rui Wang, Masao Utiyama, Eiichiro Sumita, and Hai Zhao. 2018c. Exploring recombination for efficient decoding of neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4785-4790, Brussels, Belgium. Association for Computational Linguistics.