SlowGuess/ACL_2020_OCR huggingface dataset:
We first evaluate 36-layer, 54-layer and 72-layer Msc nets on IWSLT tasks. Table 1 summarizes the architecture. As shown in Table 2, applying Msc to the vanilla Transformer with 6 layers slightly increases translation quality by  $+0.26 \sim +0.37$  BLEU  $(1 \rightarrow 2)$ . When the depth is increasing to 36, we use relatively larger dropout rate of 0.3 and achieve substantially improvements  $(+1.4 \sim +1.8$  BLEU) over its shallow counterparts (③ v.s. ②). After that, we continue deepening the encoders in order, however, our extremely deep models (72 layers, ⑤) suffer from overfitting issue on the small IWSLT corpora, which cannot be solved by simply enlarging the dropout rate. We seek to solve this issue by applying L2 regularization to the weights of encoders with greatly increased depth. Results show that this works for deeper encoders (⑥).
We also report the inference speed in Table 2 (the last column). As expected, the speed decreases
#	Model	Param.	En→De	De→En	En→Fr	ΔTrain/ΔDec
1	small, 6 layers, dpa = dpr = 0.1	10.5M	27.23	32.73	41.19	17/1800
2	w/ Msc	15.6M	27.49	33.10	41.53	17/1736
3	Msc, 36 layers, dpa = dpr = 0.3	43.3M	29.04	34.86	42.90	23/1498
4	w/ 54 layers	60.0M	29.32	35.16	43.62	27/1412
5	w/ 72 layers	76.6M	-	-	-	-
6	w/ 72 layers, λl2 = 10-5	76.6M	29.67	35.81	44.15	30/1340
Table 2: BLEU scores  $[\% ]$  of IWSLT translation tasks.  $\Delta$  Train/  $\Delta$  Dec: training time (hours)/decoding time (tokens per second) with a batch size of 32 and a beam size of 5. Dropout is applied to the residual connection  $(dp_{r})$  and attention weights  $(dp_{a})$ . We apply L2 regularization to the weights of deeper encoders with  $\lambda_{l_2} = 10^{-5}$ , which is only applied to the IWSLT tasks as the corpora are smaller and thus more regularization is required.
Model (small, 36 layers)	BLEU
Bapna et al. (2018)	28.09
Wang et al. (2019a)	28.63
Msc	29.04
Model (small, 72 layers)
Bapna et al. (2018)	failed
Wang et al. (2019a)	28.34
Msc	29.67
with the depth of Msc increasing, which is consistent with observation of Wang et al. (2019a). Compared to the baseline, Msc (72 layers) reduces decoding speed by  $26\%$ . We leave further investigation on this issue to future work.
For fair comparisons, we implement existing methods (Bapna et al., 2018; Wang et al., 2019a) on the same vanilla Transformer backbone. We separately list the results of 36-layer and 72-layer encoders on the IWSLT14 En→De task in Table 3. The method of Bapna et al. (2018) fail to train a very deep architecture while the method of Wang et al. (2019a) is exposed a degradation phenomenon  $(28.63\rightarrow 28.34)$ . In contrast, Msc in both 36-layer and 72-layer cases outperform these methods. This suggests that our extremely deep models can easily bring improvements on translation quality from greatly increased depth, producing results substantially better than existing systems.
Table 4 lists the results on the WMT14 En→De translation task and the comparison with the current state-of-the-art systems. The architectures  $(N\times M)$  of the 18-layer, 36-layer and 48-layer encoders
Table 3: Comparison with existing methods on IWSLT14 En→De translation. For a fair comparison, we implemented all methods on the same Transformer backbone as well as model settings.
Model	Param.	BLEU
Vaswani et al. (2017)	213M	28.4
Bapna et al. (2018)	137M	28.0
Dou et al. (2018)	356M	29.2
He et al. (2018)	‡210M	29.0
Wang et al. (2019a)	137M	29.3
Zhang et al. (2019a)	560M	29.62
Wu et al. (2019)	‡268M	29.9
TRANSFORMER (base)	63M	27.44
MSC, 6 layers (base)	73M	27.68
MSC, 36 layers (base)	215M	29.71
MSC, 48 layers (base)	272M	30.19
TRANSFORMER (big)	211M	28.86
MSC, 6 layers (big)	286M	29.17
MSC, 18 layers (big)	512M	30.56
Table 4: BLEU scores [%] on WMT14 En→De translation. ‡ denotes an estimate value.
are set as  $6 \times 3$ ,  $6 \times 6$  and  $6 \times 8$  respectively. We can see that incorporating our MSC into the shallow base/big contributes to  $+0.24 / +0.31$  BLEU  $(27.44 \rightarrow 27.68 / 28.86 \rightarrow 29.17)$  improvements under the same depth. When the depth grows, MSC demonstrates promising improvements of  $+1.39 \sim +2.51$  BLEU points over its shallow counterparts. It is worth noting that deep MSC with the base setting significantly outperforms the shallow one with the big setting  $(29.17 \rightarrow 30.19)$ , though both of them have around the same number of parameters. Compared to existing models, our MSC outperforms the transparent model (Bapna et al., 2018)  $(+2.2$  BLEU) and the DLCL model  $(+0.9$  BLEU) (Wang et al., 2019a), two recent approaches for deep encoding. Compared to both the depth scaled model (Zhang et al., 2019a) and the current
(a) Plain Network
(b) Msc Network
SOTA (Wu et al., 2019), our MSC achieves better performance with identical or less parameters.
Analysis of Degradation. We examine 36-layer and 72-layer plain and MSC nets, respectively. For plain networks, we simply stack dozens of layers. As we can see from Figure 2(a), the plain nets suffer from the degradation problem, which is not caused by overfitting, as they exhibit lower training BLEU. In contrast, the 72-layer MSC exhibits higher training BLEU than the 36-layer counterpart and is generalizable to the validation data. This indicates that our MSC can be more easily optimized with greatly increased depth.
Analysis of Handling Complicated Semantics. Although our MSC can enjoy improvements of BLEU score from increased depth, what does the models benefit from which is still implicit. To better understand this, we show the performance of deep MSC nets in handling sentences with complicated semantics. We assume that complicated sentences are difficult to fit with high prediction losses. Then we propose to use the modified prediction losses to identify these sentences:
$$
\begin{array}{l} s (\mathbf {x}, \mathbf {y}) = \mathbb {E} \left[ - \log P (\mathbf {y} | \mathbf {x}; \boldsymbol {\Theta}) \right] \tag {14} \\ + \operatorname {S t d} \left[ - \log P (\mathbf {y} | \mathbf {x}; \boldsymbol {\Theta}) \right], \\ \end{array}
$$
where  $\mathbb{E}\big[-\log P(\mathbf{y}|\mathbf{x};\boldsymbol {\Theta})\big]$  is approximated by:
$$
\begin{array}{l} \mathbb {E} \left[ - \log P (\mathbf {y} | \mathbf {x}; \boldsymbol {\Theta}) \right] \\ \approx \frac {1}{K} \sum_ {k = 1} ^ {K} - \log P (\mathbf {y} | \mathbf {x}; \boldsymbol {\Theta} ^ {(k)}), \tag {15} \\ \end{array}
$$
where  $\{\Theta^{(k)}\}_{k = 1}^{K}$  indicates model parameters for the last  $K$  ( $K = 20$ ) checkpoints. Std[\cdot] is the
Figure 3: Comparison between plain nets and MSC nets on fine-grained test sets with increasing translation difficulty from "Simple" to "Challenging". Improvements (BLEU [%]) of translation quality over the 6-layer plain net. Higher is better. The results of this baseline are enclosed in the parentheses.
Figure 2: Illustration of the degradation problem on IWSLT14  $\mathrm{En} \rightarrow \mathrm{De}$  task. We randomly select 3K sentence pairs from our training data for evaluation. For a fair comparison, we implemented all models on the same Transformer backbone as well as model settings.
(a) 6-layer
Figure 4: Visualization of the attention weights from the top-most layer of the decoder for both shallow and deep MSC nets.
(b) 72-layer
standard deviation of prediction loss of sentence y given sentence x, and the introduction of which aims to prevent training oscillations from affecting complicated sentences identification.
We adopt a shallow plain net (small, 6 layers) to assign the prediction loss  $s(\mathbf{x},\mathbf{y})$  to each sentence pair. Further, we split the IWSLT  $\mathrm{En} \rightarrow \mathrm{De}$  test set into 4 equal parts according to the prediction losses, which are pre-defined to have "Simple", "Ordinary", "Difficult" and "Challenging" translation difficulties, respectively. Results on these fine-grained test sets are shown in Figure 3. First of all, all methods yield minor BLEU improvements over the baseline on the first sub-set that containing sentences with little difficulties to be translated. However, when the translation difficulty increases, the improvements of the deep MSC nets are expanded to around 2 BLEU. These results indicate that our MSC framework deals with sentences which are difficult to be translated well.
4The fine-grained test sets are publicly available at https://github.com/pemywei/MSC-NMT/tree/master/IWSLT_En2De_Split_Test.
Figure 5: Gradient norm (y-axis) of each encoder layer in 72-layer Msc over the fist 10k training steps. "Li" denotes the  $i$ -th encoder layer. The Msc framework helps balance the gradient norm between top and bottom layers during training
We also visualize the attention weights from the top-most layer of the decoder of both shallow and deep MSC nets in Figure 4. As shown in Figure 4(a), when generating the next token of "tun", the shallow MSC attends to diverse tokens, such as "to", "that", ":" and "eos", which causes the generation of "eos" and the phrase "be able to" is mistakenly untranslated. Remarkably, the deep MSC (Figure 4(b)) mostly focuses on the source tokens "be", "able" and "to", and translates this complicated sentence successfully. More cases can be found in Appendix C. This kind of cases show the advantages of constructing extremely deep models for translating semantic-complicated sentences.
Analysis of Error Propagation. To understand the propagation process of training signals, we collect the gradient norm of each encoder layer during training. Results in Figure 5 show that with the MSC framework each layer enjoys a certain value of gradient for parameter update, and the error signals traverse along the depth of the model without hindrance. MSC helps balance the gradient norm between top and bottom layers in deep models.
Ablation Study. We conduct ablation study to investigate the performance of each component of our model. The results are reported in Table 5: (1) We use simple element-wise addition for feature fusion instead of using a gated combination as introduced in Section 3.2. This method achieves a 29.45 BLEU, which is lower than the best result. We additionally modify the implementation of the contextual collaboration cell  $\mathcal{Q}(\cdot)$  as  $\mathrm{FFN}(\cdot)$ , which shows that the performance is reduced by 0.5 BLEU. (2) Removing CXT-ENC ATTENTION and/or contextual collaboration makes the BLEU score drop by  $\sim 0.7$ , which suggests that multiscale
Model	BELU
Msc, 72 layers	29.67
- feature fusion with addition	29.45
- implement Q(·) in Eq. (9) as FFN(·)	29.17
- remove CXT-ENC ATTENTION	28.99
- remove contextual collaboration	28.94
Msc, 18 layers (emb=512, ffn=1024)	29.08
Msc, 36 layers (emb=512, ffn=1024)	29.41
Table 5: Ablation study on IWSLT14 En→De task.
collaboration helps in constructing extremely deep models. (3) Considering that the deep Msc introduces more parameters, we also train another two Msc models with about the same or double number of parameters: with 18/36 layers, embedding size 512 and FFN layer dimension 1024. These models underperform the deeper 72-layer model, which shows that the number of parameters is not the key to the improvement.
Researchers have constructed deep NMT models that use linear connections to reduce the gradient propagation length inside the topology (Zhou et al., 2016; Wang et al., 2017; Zhang et al., 2018b) or read-write operations on stacked layers of memories (Meng et al., 2015). Such work has been conducted on the basis of the conventional RNN architectures and may not be fully applicable to the advanced Transformer.
Recently, Bapna et al. (2018) introduced a transparent network into NMT models to ease the optimization of models with deeper encoders. To improve gradient flow they let each decoder layer find an unique weighted combination of all encoder layer outputs, instead of just the top encoder layer. Wang et al. (2019a) found that adopting the proper use of layer normalization helps to learn deep encoders. A method was further proposed to combine layers and encourage gradient flow by simple shortcut connections. Zhang et al. (2019a) introduced a depth-scaled initialization to improve norm preservation and proposed a merged attention sublayer to avoid the computational overhead for deep models. Researchers have also explored growing NMT models in two stages (Wu et al., 2019), in which shallow encoders and decoders are trained in the first stage and subsequently held constant, when another set of shallow layers are stacked on the top. In concurrent work, Xu et al. (2019) studied the effect
of the computation order of residual connection and layer normalization, and proposed an parameter initialization method with Lipschitz restrictions to ensure the convergence of deep Transformers. Our method significantly differs from these methods, solving the problem by associating the decoder with the encoder with multi-granular dependencies in different space-scales.
Exploiting deep representations have been studied to strengthen feature propagation and encourage feature reuse in NMT (Shen et al., 2018; Dou et al., 2018, 2019; Wang et al., 2019b). All of these works mainly attend the decoder to the final output of the encoder stack, we instead coordinate the encoder and the decoder at earlier stage.
In this paper, we propose a multisacle collaborative framework to ease the training of extremely deep NMT models. Specifically, instead of the top-most representation of the encoder stack, we attend the decoder to multi-granular source information with different space-scales. We have shown that the proposed approach boosts the training of very deep models and can bring improvements on translation quality from greatly increased depth. Experiments on various language pairs show that the MSC achieves prominent improvements over strong baselines as well as previous deep models.
In the future, we would like to extend our model to extremely large datasets, such as WMT'14 English-to-French with about 36M sentence-pairs. And the deeper MSC model results in high computational overhead, to address this issue, we would like to apply the average attention network (Zhang et al., 2018a) to our deep MSC models.
We would like to thank the anonymous reviewers for the helpful comments. We also thank Xingxing Zhang, Luxi Xing and Kaixin Wu for their instructive suggestions and invaluable help. This work is supported by the National Key Research and Development Program (Grant No.2017YFB0803301).
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. Character-level language modeling with deeper self-attention. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pages 3159-3166.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. In arXiv:1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Ankur Bapna, Mia Chen, Orhan First, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028-3033, Brussels, Belgium. Association for Computational Linguistics.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2016. Hierarchical multiscale recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4253-4262, Brussels, Belgium. Association for Computational Linguistics.
Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation with routing-by-agreement. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, pages 86-93.
Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers), pages 4052-4059, Minneapolis, Minnesota. Association for Computational Linguistics.
Salah El Hihi and Yoshua Bengio. 1996. Hierarchical recurrent neural networks for long-term dependencies. In Advances in Neural Information Processing Systems 8, NIPS, Denver, CO, USA, November 27-30, 1995, pages 493-499.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In arXiv:1705.03122.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems 31, pages 7944-7954.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 4700-4708.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. 2014. A clockwork rnn. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014.
Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL 2004, Barcelona, Spain.
Yang Liu. 2019. Fine-tune BERT for extractive summarization. In arXiv:1903.10318.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Fandong Meng, Zhengdong Lu, Zhaopeng Tu, Hang Li, and Qun Liu. 2015. A deep memory-based architecture for sequence-to-sequence learning. In arXiv:1506.06442.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Belgium, Brussels. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In OpenAI Blog.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Jürgen Schmidhuber. 1992. Learning complex, extended sequences using the principle of history compression. In Neural Computation, pages 234-242.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational