SlowGuess/ACL_2020_OCR huggingface dataset:
Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu. 2018. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1294-1303, New Orleans, Louisiana. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Mingxuan Wang, Zhengdong Lu, Jie Zhou, and Qun Liu. 2017. Deep neural machine translation with linear associative unit. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 136-145, Vancouver, Canada. Association for Computational Linguistics.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019a. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy. Association for Computational Linguistics.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019b. Exploiting sentential context for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6197-6203, Florence, Italy. Association for Computational Linguistics.
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Depth growing for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5558-5563, Florence, Italy. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, and Klaus Macherey. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. In arXiv:1609.08144.
Hongfei Xu, Qiuhui Liu, Josef van Genabith, and Jingyi Zhang. 2019. Why deep transformers are difficult to converge? from computation order to lipschitz restricted parameter initialization. In arXiv:1911.03179.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. 2018. Deep layer aggregation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2403-2412.
Biao Zhang, Ivan Titov, and Rico Sennrich. 2019a. Improving deep transformer with depth-scaled initialization and merged attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 897-908, Hong Kong, China. Association for Computational Linguistics.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018a. Accelerating neural transformer via an average attention network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1789-1798, Melbourne, Australia. Association for Computational Linguistics.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018b. Neural machine translation with deep attention. In IEEE Trans. Pattern Anal. Mach. Intell., pages 154-163.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019b. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep recurrent models with fast-forward connections for neural machine translation. Transactions of the Association for Computational Linguistics, 4:371-383.
A Abstractive Summarization
We further verify the effectiveness of MSC on text summarization. Automatic text summarization produces a concise and fluent summary conveying the key information in the input (e.g., a news article). We focus on abstractive summarization, a generation task aims to generate the summary of a document with rewriting. We use the
Model	R-1	R-2	R-L
Extractive Summarization
Lead3	40.38	17.61	36.59
HIBERTM	42.37	19.95	38.83
Liu (2019)	43.25	20.24	39.63
Abstractive Summarization
PGNet	39.53	17.28	36.38
Bottom-Up	41.22	18.68	38.34
S2S-ELMo	41.56	18.94	38.47
TRANSFORMER	40.28	17.87	37.25
HIERTRANS (36 L)	41.22	18.97	38.45
MSC (36 L)	41.96	19.50	39.07
Table 6: Results on CNNDM summarization using ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). "36L" is short for "36-layer encoder".
non-anonymized version of the CNN/DailyMail (CNNDM) dataset (See et al., 2017) for evaluation. We preprocessed the dataset using the scripts from the authors of See et al. (2017), and the resulting dataset contains 287,226 documents with summaries for training, 13,368 for validation and 11,490 for test.
We still adopt the Transformer (Vaswani et al., 2017) as our backbone, with a embedding size of 512 and FFN layer dimension of 1024. We train our model on the training set for 30 epochs, and also use label smoothing with rate of 0.1. We set batch size to 32, and maximum length to 768. During decoding, we use beam search with beam size of 5. The input document is truncated to the first 640 tokens. We remove duplicated trigrams in beam search, and tweak the maximum summary length on the development set (Paulus et al., 2018; Edunov et al., 2019). We use the F1 version of ROUGE (Lin, 2004) as the evaluation metric.
In Table 6, we compare MSC (36 layers) against the baseline and several state-of-the-art models on CNN/DailyMail, with extractive models in the top block and abstractive models in the bottom block. Lead3 is a baseline which simply selects the first three sentences as the summary. HIBERT $_M$  (Zhang et al., 2019b) adds the large open-domain unlabeled data to pre-train hierarchical transformer encoders and fine-tune on the extractive summarization task. We also include in Table 6 the best reported extractive summarization result taken
from (Liu, 2019) on the dataset. PGNet (See et al., 2017), Bottom-Up (Gehrmann et al., 2018) and S2S-ELMo (Edunov et al., 2019) are all sequence to sequence learning based models with copy and coverage modeling, bottom-up content selecting and pre-trained ELMo representations augmenting. We also implemented two baselines. One is the standard 6-layer Transformer model. We can see that the deep MSC leads to a  $+1.8$  ROUGE improvement over TRANSFORMER. The other baseline is the hierarchical transformer summarization model (HIERTRANS), which invokes both a sentence-level and a document-level transformer encoders, as well as a standard transformer decoder. Note the settings for both encoders are the same (each of them have  $L = 18$ ,  $\mathrm{emb} = 512$ ,  $\mathrm{ffn} = 1024$ , head  $= 8$ ). The deep MSC outperforms HIERTRANS by 0.5 to 0.7 ROUGE with the same depth of encoders.
B Derivations of Block-Scale Collaboration
In pre-norm Transformer, a general transformation can be formulated as:
$$
\mathrm {H} ^ {l} = \mathcal {F} \left(\mathrm {H} ^ {l - 1}; \boldsymbol {\Theta} ^ {l}\right) + \mathrm {H} ^ {l - 1}, \tag {16}
$$
where  $\mathrm{H}^{l - 1}$  and  $\mathrm{H}^l$  are the input and output of the  $l$ -th layer. For the Block-Scale Collaboration framework, there are two channels for passing error gradients from the prediction loss  $\mathcal{L}$  to encoder layers, which are from the top-most layers of the whole encoder stack  $\mathrm{H}_e^{N,M_N}$  (identical to  $\mathrm{B}_e^N$ ) and the current bock  $\mathrm{H}_e^{n,M_n}$  (identical to  $\mathrm{B}_e^n$ ), respectively. From the chain rule of back propagation we can obtain:
$$
\frac {\partial \mathcal {L}}{\partial \mathrm {H} _ {e} ^ {n , l}} = \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {N}} \times \frac {\partial \mathrm {B} _ {e} ^ {N}}{\partial \mathrm {H} _ {e} ^ {n , l}} + \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {n}} \times \frac {\partial \mathrm {B} _ {e} ^ {n}}{\partial \mathrm {H} _ {e} ^ {n , l}}. \tag {17}
$$
We can recursively use Eq. (16) to formulate that
$$
\begin{array}{l} \mathrm {B} _ {e} ^ {N} = \mathrm {H} _ {e} ^ {N, M _ {N}} \\ = \mathrm {H} _ {e} ^ {n, l} + \sum_ {k = l + 1} ^ {M _ {n}} \mathcal {F} \left(\mathrm {H} _ {e} ^ {n, k - 1}; \boldsymbol {\Theta} _ {e} ^ {n, k}\right) \tag {18} \\ + \sum_ {i = n + 1} ^ {N} \sum_ {j = 1} ^ {M _ {i}} \mathcal {F} (\mathrm {H} _ {e} ^ {i, j - 1}; \Theta_ {e} ^ {i, j}), \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {B} _ {e} ^ {N} = \mathrm {H} _ {e} ^ {n, M _ {n}} \\ = \mathrm {H} _ {e} ^ {n, l} + \sum_ {k = l + 1} ^ {M _ {n}} \mathcal {F} \left(\mathrm {H} _ {e} ^ {n, k - 1}; \Theta_ {e} ^ {n, k}\right), \tag {19} \\ \end{array}
$$
$^{5}$ https://github.com/abisee/cnn-dailymail
respectively. In this way, the derivations of  $\mathrm{B}_e^N$  and  $\mathrm{B}_e^n$  with respect to  $\mathrm{H}_e^{n,l}$  can be calculated as:
$$
\begin{array}{l} \frac {\partial \mathrm {B} _ {e} ^ {N}}{\partial \mathrm {H} _ {e} ^ {n , l}} = 1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathcal {F} (\mathrm {H} _ {e} ^ {n , k - 1} ; \boldsymbol {\Theta} _ {e} ^ {n , k})}{\partial \mathrm {H} _ {e} ^ {n , l}} \\ + \sum_ {i = n + 1} ^ {N} \sum_ {j = 1} ^ {M _ {i}} \frac {\partial \mathcal {F} \left(\mathrm {H} _ {e} ^ {i , j - 1} ; \boldsymbol {\Theta} _ {e} ^ {i , j}\right)}{\partial \mathrm {H} _ {e} ^ {n , l}}, \tag {20} \\ \end{array}
$$
$$
\frac {\partial \mathrm {B} _ {e} ^ {n}}{\partial \mathrm {H} _ {e} ^ {n , l}} = 1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathcal {F} \left(\mathrm {H} _ {e} ^ {n , k - 1} ; \boldsymbol {\Theta} _ {e} ^ {n , k}\right)}{\partial \mathrm {H} _ {e} ^ {n , l}}.
$$
Finally, we can put Eq. (20) into Eq. (17) and obtain:
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial \mathrm {H} _ {e} ^ {n , l}} \\ = \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {N}} \times \frac {\partial \mathrm {B} _ {e} ^ {N}}{\partial \mathrm {H} _ {e} ^ {n , l}} + \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {n}} \times \frac {\partial \mathrm {B} _ {e} ^ {n}}{\partial \mathrm {H} _ {e} ^ {n , l}} \\ = \underbrace {\frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {N}} \times (1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathrm {H} _ {e} ^ {n , k}}{\partial \mathrm {H} _ {e} ^ {n , l}} + \sum_ {i = n + 1} ^ {N} \sum_ {j = 1} ^ {M _ {i}} \frac {\partial \mathrm {H} _ {e} ^ {i , j}}{\partial \mathrm {H} _ {e} ^ {n , l}})} _ {(a)} \\ + \underbrace {\frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {n}} \times (1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathrm {H} _ {e} ^ {n , k}}{\partial \mathrm {H} _ {e} ^ {n , l}})} _ {(b)}. \\ \end{array}
$$