SlowGuess/ACL_2020_OCR huggingface dataset:
Xiangpeng Wei $^{1,2*}$ , Heng Yu $^{3\dagger}$ , Yue Hu $^{1,2\dagger}$ , Yue Zhang $^{4,5}$ , Rongxiang Weng $^{3}$ , Weihua Luo $^{3}$
$^{1}$ Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
$^{2}$ School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
<sup>3</sup>Machine Intelligence Technology Lab, Alibaba Group, Hangzhou, China
$^{4}$ School of Engineering, Westlake University, Hangzhou, China
$^{5}$ Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China
{weixiangpeng,huyue}@iei.ac.cn yue.zhang@wias.org.cn
{yuheng.yh, wengrx, weihua.luowh}@alibaba-inc.com
Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neural networks can be more effective but are difficult to train. In this paper, we present a MultiScale Collaborative (Msc) framework to ease the training of NMT models that are substantially deeper than those used previously. We explicitly boost the gradient backpropagation from top to bottom levels by introducing a block-scale collaboration mechanism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration. We provide empirical evidence showing that the Msc nets are easy to optimize and can obtain improvements of translation quality from considerably increased depth. On IWSLT translation tasks with three translation directions, our extremely deep models (with 72-layer encoders) surpass strong baselines by  $+2.2\sim +3.1$  BLEU points. In addition, our deep Msc achieves a BLEU score of 30.56 on WMT14 English  $\rightarrow$  German task that significantly outperforms state-of-the-art deep NMT models.
Neural machine translation (NMT) directly models the entire translation process using a large neural network and has gained rapid progress in recent years (Sutskever et al., 2014; Sennrich et al., 2016). The structure of NMT models has evolved quickly, such as RNN-based (Wu et al., 2016), CNN-based (Gehring et al., 2017) and attention-based (Vaswani et al., 2017) systems. All of these models follow the encoder-decoder framework with attention (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) paradigm.
Deep neural networks have revolutionized the state-of-the-art in various communities, from computer vision to natural language processing. However, training deep neural networks has been always a challenging problem. To encourage gradient flow and error propagation, researchers in the field of computer vision have proposed various approaches, such as residual connections (He et al., 2016), densely connected networks (Huang et al., 2017) and deep layer aggregation (Yu et al., 2018). In natural language processing, constructing deep architectures has shown effectiveness in language modeling, question answering, text classification and natural language inference (Peters et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019; Devlin et al., 2019). However, among existing NMT models, most of them are generally equipped with 4-8 encoder and decoder layers (Wu et al., 2016; Vaswani et al., 2017). Deep neural network has been explored relatively little in NMT.
Recent evidence (Bapna et al., 2018; Wang et al., 2019a) shows that model depth is indeed of importance to NMT, but a degradation problem has been exposed: by simply stacking more layers, the translation quality gets saturated and then degrades rapidly. To address this problem, Bapna et al. (2018) proposed a transparent attention mechanism to ease the optimization of the models with deeper encoders. Wang et al. (2019a) continued this line of research but construct a much deeper encoder for Transformer by adopting the pre-norm method that establishes a direct way to propagate error gradients from the top layer to bottom levels, and passing the combination of previous layers to the next. While notable gains have been reported over shallow models, the improvements of translation quality are limited when the model depth is beyond 20. In addition, degeneration of translation quality is still observed when the depth is beyond 30. As a result, two questions arise naturally: How
*Work done at Alibaba Group.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 414-426 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
to break the limitation of depth in NMT models? and How to fully utilize the deeper structure to further improve the translation quality?
In this paper, we address the degradation problem by proposing a MultiScale Collaborative (Msc) framework for constructing NMT models with very deep encoders. In particular, the encoder and decoder of our model have the same number of blocks, each consisting of one or several stacked layers. Instead of relying on the whole encoder stack directly learns a desired representation, we let each encoder block learn a fine-grained representation and enhance it by encoding spatial dependences using a bottom-up network. For coordination, we attend each block of the decoder to both the corresponding representation of the encoder and the contextual representation with spatial dependences. This not only shortens the path of error propagation, but also helps to prevent the lower level information from being forgotten or diluted.
We conduct extensive experiments on WMT and IWSLT translation tasks, covering three translation directions with varying data conditions. On IWSLT translation tasks, we show that:
- While models with traditional stacking architecture exhibit worse performance on both training and validation data when depth increases, our framework is easy to optimize.
- The deep MSC nets (with 72-layer encoders) bring great improvements on translation quality from increased depth, producing results that substantially better than existing systems.
On the WMT14 English  $\rightarrow$  German task, we obtain improved results by deep Msc networks with a depth of 48 layers, outperforming strong baselines by  $+2.5$  BLEU points, and also defeat state-of-the-art deep NMT models (Wu et al., 2019; Zhang et al., 2019a) with identical or less parameters.
Given a bilingual sentence pair  $(\mathbf{x},\mathbf{y})$ , an NMT model learns a set of parameters  $\Theta$  by maximizing the log-likelihood  $P(\mathbf{y}|\mathbf{x};\Theta)$ , which is typically
decomposed into the product of the conditional probability of each target word:  $P(\mathbf{y}|\mathbf{x};\Theta) = \prod_{t=1}^{T_{\mathrm{y}}}P(\mathbf{y}_{t}|\mathbf{y}_{<t},\mathbf{x};\Theta)$ , where  $T_{\mathrm{y}}$  is the length of sentence  $\mathbf{y}$ ,  $\mathbf{y}_{<t}$  is the partial translation that contains the target tokens before position  $t$ . An encoder-decoder framework is commonly adopted to model the conditional probability  $P(\mathbf{y}|\mathbf{x};\Theta)$ , in which the encoder and decoder can be implemented as RNN (Wu et al., 2016), CNN (Gehring et al., 2017), or Self-Attention network (Vaswani et al., 2017). Despite variant types of NMT architectures, multiple-layer encoder and decoder are generally employed to perform the translation task, and residual connections (He et al., 2016) are naturally introduced among layers, as  $\mathrm{H}^l = \mathrm{LAYER}(\mathrm{H}^{l-1};\Theta^l) + \mathrm{H}^{l-1}$ , where  $\mathrm{H}^l$  is the output of the  $l$ -th layer,  $\mathrm{LAYER}(\cdot)$  is the layer function and  $\Theta^l$  be the parameters.
We take the state-of-the-art Transformer as our baseline model. Specifically, the encoder consists of a stack of  $L$  identical layers, each of which comprises two subcomponents: a self-attention mechanism followed by a feed-forward network. Layer normalization (Ba et al., 2016) is applied to the input of each subcomponent (i.e., pre-norm) and a residual skip connection (He et al., 2016) adds each subcomponent's input to its output. Formally,
$$
\begin{array}{l} \mathrm {O} _ {e} ^ {l} = \operatorname {A T T N} \left(\mathrm {Q} _ {e} ^ {l}, \mathrm {K} _ {e} ^ {l}, \mathrm {V} _ {e} ^ {l}; \boldsymbol {\Theta} _ {e} ^ {l}\right) + \mathrm {H} _ {e} ^ {l - 1}, \tag {1} \\ \mathrm {H} _ {e} ^ {l} = \operatorname {F N N} \left(\mathrm {L N} \left(\mathrm {O} _ {e} ^ {l}\right); \boldsymbol {\Theta} _ {e} ^ {l}\right) + \mathrm {O} _ {e} ^ {l}, \\ \end{array}
$$
where  $\mathrm{LN}(\cdot)$ ,  $\mathrm{ATTN}(\cdot)$  and  $\mathrm{FFN}(\cdot)$  are layer normalization, attention mechanism, and feed-forward networks with ReLU activation in between, respectively.  $\{\mathrm{Q}_e^l,\mathrm{K}_e^l,\mathrm{V}_e^l\}$  are query, key and value vectors that are transformed from the normalized  $(l - 1)$ -th encoder layer  $\mathrm{LN}(\mathrm{H}_e^{l - 1})$
The decoder is similar in structure to the encoder except that it includes a standard attention mechanism after each self-attention network, which attends to the output of the encoder stack  $\mathrm{H}_e^L$ :
$$
\begin{array}{l} \mathrm {O} _ {d} ^ {l} = \mathrm {A T T N} \left(\mathrm {Q} _ {d} ^ {l}, \mathrm {K} _ {d} ^ {l}, \mathrm {V} _ {d} ^ {l}; \boldsymbol {\Theta} _ {d} ^ {l}\right) + \mathrm {H} _ {d} ^ {l - 1}, \\ \mathrm {S} _ {d} ^ {l} = \operatorname {A T T N} \left(\mathrm {L N} \left(\mathrm {O} _ {d} ^ {l}\right), \mathrm {K} _ {e} ^ {L}, \mathrm {V} _ {e} ^ {L}; \boldsymbol {\Theta} _ {d} ^ {l}\right) + \mathrm {O} _ {d} ^ {l}, \tag {2} \\ \mathrm {H} _ {d} ^ {l} = \mathrm {F N N} (\mathrm {L N} (\mathrm {S} _ {d} ^ {l}); \boldsymbol {\Theta} _ {d} ^ {l}) + \mathrm {C} _ {d} ^ {l}, \\ \end{array}
$$
where  $\{\mathrm{Q}_d^l,\mathrm{K}_d^l,\mathrm{V}_d^l\}$  are transformed from the normalized  $(l - 1)$ -th decoder layer  $\mathrm{LN}(\mathrm{H}_d^{l - 1})$  and  $\{\mathrm{K}_e^L,\mathrm{V}_e^L\}$  are transformed from the top layer of the encoder. The top layer of the decoder  $\mathrm{H}_d^L$  is used to generate the final output sequence. In the
<sup>1</sup>In our scenario, we mainly study the depth of encoders. The reason is similar in (Wang et al., 2019a): 1) encoders have a greater impact on performance than decoders; 2) increasing the depth of the decoder will significantly increase the complexity of inference.
$^{2}$ Msc not only performs well on NMT but also is generalizable to other sequence-to-sequence generation tasks, such as abstractive summarization that is introduced in Appendix A.
(a) Overall framework.
(b) Detailed illustration of the  $n$ -th block.
Figure 1: Illustration of Multiscale Collaborative Deep NMT Model.  $N$  is the number of encoder and decoder blocks. The  $n$ -th block of the encoder consists of  $M_{n}$  layers, while each decoder block only contains one layer.
following sections, we simplify the equations as
$$
\mathrm {H} _ {e} ^ {l} = \mathcal {F} \left(\mathrm {H} _ {e} ^ {l - 1}; \boldsymbol {\Theta} _ {e} ^ {l}\right) + \mathrm {H} _ {e} ^ {l - 1}, \tag {3}
$$
$$
\mathrm {H} _ {d} ^ {l} = \mathcal {G} (\mathrm {H} _ {d} ^ {l - 1}, \mathrm {H} _ {e} ^ {L}; \boldsymbol {\Theta} _ {d} ^ {l}) + \mathrm {H} _ {d} ^ {l - 1},
$$
for the encoder and decoder, respectively.
As discussed by Wang et al. (2019a), applying layer normalization to the input of each subcomponent is the key to learning deep encoders, as it establishes a direct way to pass gradient from the top-most layer to bottom layers:
$$
\frac {\partial \mathcal {L}}{\partial \mathrm {H} _ {e} ^ {l}} = \frac {\partial \mathcal {L}}{\partial \mathrm {H} _ {e} ^ {L}} \times (1 + \sum_ {j = l} ^ {L - 1} \frac {\partial \mathcal {F} (\mathrm {H} _ {e} ^ {j} ; \boldsymbol {\Theta} _ {e} ^ {j + 1})}{\partial \mathrm {H} _ {e} ^ {l}}), \tag {4}
$$
where  $\mathcal{L}$  is the cross entropy loss. However, as pointed out by Wang et al. (2019a) that it can be difficult to deepen the encoder for better translation quality. We argue that as the right-most term in Eq. (4) approaches 0 for the lower levels of the encoder, the parameters of which cannot be sufficiently trained using the error gradient  $\frac{\partial\mathcal{L}}{\partial\mathrm{H}_e^L}$  only. To solve this problem, we propose a novel approach to shorten the path of error propagation from  $\mathcal{L}$  to bottom layers of the encoder.
3 Multiscale Collaborative Deep Model
In this section, we introduce the details of the proposed approach, a MultiScale Collaborative (Msc) framework for constructing extremely deep NMT models. The framework of our method consists of
two main components shown in Figure 1(a). First, a block-scale collaboration mechanism establishes shortcut connections from the lower levels of the encoder to the decoder (as described in 3.1), which is the key to training very deep NMT models. We give explanation by seeing the gradient propagation process. Second, we further enhance source representations with spatial dependencies by contextual collaboration, which is discussed in Section 3.2.
3.1 Block-Scale Collaboration
An intuitive extension of naive stacking of layers is to group few stacked layers into a block. We suppose that the encoder and decoder of our model have the same number of blocks (i.e.,  $N$ ). Each block of the encoder has  $M_{n}$  ( $n \in \{1,2,\dots,N\}$ ) identical layers, while each decoder block contains one layer. Thus, we can adjust the value of each  $M_{n}$  flexibly to increase the depth of the encoder. Formally, for the  $n$ -th block of the encoder:
$$
\mathrm {B} _ {e} ^ {n} = \operatorname {B L O C K} _ {e} \left(\mathrm {B} _ {e} ^ {n - 1}\right), \tag {5}
$$
where  $\mathrm{BLOCK}_e(\cdot)$  is the block function, in which the layer function  $\mathcal{F}(\cdot)$  is iterated  $M_{n}$  times, i.e.
$$
\mathrm {B} _ {e} ^ {n} = \mathrm {H} _ {e} ^ {n, M _ {n}},
$$
$$
\mathrm {H} _ {e} ^ {n, l} = \mathcal {F} \left(\mathrm {H} _ {e} ^ {n, l - 1}; \Theta_ {e} ^ {n, l}\right) + \mathrm {H} _ {e} ^ {n, l - 1}, \tag {6}
$$
$$
\mathrm {H} _ {e} ^ {n, 0} = \mathrm {B} _ {e} ^ {n - 1},
$$
where  $l\in \{1,2,\dots,M_n\}$ ,  $\mathrm{H}_e^{n,l}$  and  $\Theta_e^{n,l}$  are the representation and parameters of the  $l$ -th layer in
the  $n$ -th block, respectively. The decoder works in a similar way but the layer function  $\mathcal{G}(\cdot)$  is iterated only once in each block,
$$
\begin{array}{l} \mathrm {B} _ {d} ^ {n} = \operatorname {B L O C K} _ {d} \left(\mathrm {B} _ {d} ^ {n - 1}, \mathrm {B} _ {e} ^ {n}\right) \tag {7} \\ = \mathcal {G} \left(\mathrm {B} _ {d} ^ {n - 1}, \mathrm {B} _ {e} ^ {n}; \boldsymbol {\Theta} _ {d} ^ {n}\right) + \mathrm {B} _ {d} ^ {n - 1}. \\ \end{array}
$$
Each block of the decoder attends to the corresponding encoder block. He et al. (2018) proposed a model that learns the hidden representations in two corresponding encoder and decoder layers as the same semantic level through layer-wise coordination and parameter sharing. Inspired by this, we focus on efficiently training extremely deep NMT models through directly attending decoder to the lower-level layers of the encoder, rather than only to the final representation of the encoder stack.
The proposed block-scale collaboration (BSC) mechanism can effectively boost gradient propagation from prediction loss to lower level encoder layers. For explaining this, see again Eq. (4), which explains the error back-propagation of pre-norm Transformer. Formally, we let  $\mathcal{L}$  be the prediction loss. The differential of  $\mathcal{L}$  with respect to the  $l$ -th layer in the  $n$ -th block  $\mathrm{H}_e^{n,l}$  can be calculated as:3
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial \mathrm {H} _ {e} ^ {n , l}} \\ = \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {N}} \times \frac {\partial \mathrm {B} _ {e} ^ {N}}{\partial \mathrm {H} _ {e} ^ {n , l}} + \frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {n}} \times \frac {\partial \mathrm {B} _ {e} ^ {n}}{\partial \mathrm {H} _ {e} ^ {n , l}} \\ = \underbrace {\frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {N}} \times (1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathrm {H} _ {e} ^ {n , k}}{\partial \mathrm {H} _ {e} ^ {n , l}} + \sum_ {i = n + 1} ^ {N} \sum_ {j = 1} ^ {M _ {i}} \frac {\partial \mathrm {H} _ {e} ^ {i , j}}{\partial \mathrm {H} _ {e} ^ {n , l}})} _ {(a)} \\ + \underbrace {\frac {\partial \mathcal {L}}{\partial \mathrm {B} _ {e} ^ {n}} \times \left(1 + \sum_ {k = l + 1} ^ {M _ {n}} \frac {\partial \mathrm {H} _ {e} ^ {n , k}}{\partial \mathrm {H} _ {e} ^ {n , l}}\right)} _ {(b)}, \tag {8} \\ \end{array}
$$
where term  $(a)$  is equal to Eq. (4). In addition to the straightforward path  $\frac{\partial\mathcal{L}}{\partial\mathrm{B}_e^N}$  for parameter update from the top-most layer to lower ones, Eq. (8) also provides a complementary way to directly pass error gradient  $\frac{\partial\mathcal{L}}{\partial\mathrm{B}_e^n}$  from top to bottom in the current block. Another benefit is that Bsc shortens the length of gradient pass chain (i.e.,  $M_{n}\ll L$
3.2 Contextual Collaboration
To model long-term spatial dependencies and reuse global representations, we define a GRU (Cho et al.,
2014) cell  $\mathcal{Q}(\mathbf{c},\bar{\mathbf{x}})$ , which maps a hidden state  $\mathbf{c}$  and an additional input  $\bar{\mathbf{x}}$  into a new hidden state:
$$
\begin{array}{l} \mathrm {C} ^ {n} = \mathcal {Q} \left(\mathrm {C} ^ {n - 1}, \mathrm {B} _ {e} ^ {n}\right), n \in [ 1, N ] \tag {9} \\ \mathrm {C} ^ {0} = \mathcal {E} _ {e}, \\ \end{array}
$$
where  $\mathcal{E}_e$  is the embedding matrix of the source input  $\mathbf{x}$ . The new state  $C^n$  can be fused with each layer of the subsequent blocks in both the encoder and the decoder. Formally,  $\mathrm{B}_e^n$  in Eq.(5) can be re-calculated in the following way:
$$
\begin{array}{l} \mathrm {B} _ {e} ^ {n} = \mathrm {H} _ {e} ^ {n, M _ {n}}, \\ \mathrm {H} _ {e} ^ {n, l} = \mathcal {F} \left(\mathrm {H} _ {e} ^ {n, l - 1}, \mathrm {C} ^ {n - 1}; \boldsymbol {\Theta} _ {e} ^ {n, l}\right) + \mathrm {H} _ {e} ^ {n, l - 1}, \tag {10} \\ \mathrm {H} _ {e} ^ {n, 0} = \mathrm {B} _ {e} ^ {n - 1}. \\ \end{array}
$$
Similarly, for decoder, we have
$$
\begin{array}{l} \mathrm {B} _ {d} ^ {n} = \mathrm {B L O C K} _ {d} \left(\mathrm {B} _ {d} ^ {n - 1}, \mathrm {B} _ {e} ^ {n}\right) \tag {11} \\ = \mathcal {G} (\mathrm {B} _ {d} ^ {n - 1}, \mathrm {B} _ {e} ^ {n}, \mathrm {C} ^ {n}; \boldsymbol {\Theta} _ {d} ^ {n}) + \mathrm {B} _ {d} ^ {n - 1}. \\ \end{array}
$$
The above design is inspired by multiscale RNNs (MRNN) (Schmidhuber, 1992; El Hihi and Bengio, 1996; Koutnik et al., 2014; Chung et al., 2016), which encode temporal dependencies with different timescales. Unlike MRNN, our MSC enables each decoder block to attend to multi-granular source information with different space-scales, which helps to prevent the lower level information from being forgotten or diluted.
Feature Fusion: We fuse the contextual representation with each layer of the encoder and decoder through attention. A detailed illustration of our algorithm is shown in Figure 1(b). In particular, the  $l$ -th layer of the  $n$ -th encoder block  $\mathcal{F}(\cdot; \Theta_e^{n,l})$ ,  $l \in [1, M_n]$  and  $n \in [1, N]$ ,
$$
\begin{array}{l} \mathrm {O} _ {e} ^ {n, l} \\ = g _ {e} \odot \mathrm {A T T N} _ {h} \left(\mathrm {H} _ {e} ^ {n, l - 1}, \mathrm {H} _ {e} ^ {n, l - 1}, \mathrm {H} _ {e} ^ {n, l - 1}; \Theta_ {e} ^ {n, l}\right) \\ + \left(1 - g _ {e}\right) \odot \mathrm {A T T N} _ {c} \left(\mathrm {H} _ {e} ^ {n, l - 1}, \mathrm {C} ^ {n - 1}, \mathrm {C} ^ {n - 1}; \Theta_ {e} ^ {n, l}\right) \\ + \mathrm {H} _ {e} ^ {n, l - 1}, \\ \end{array}
$$
$$
g _ {e} = \sigma \left(W _ {1} \mathrm {A T T N} _ {h} (\cdot) + W _ {2} \mathrm {A T T N} _ {c} (\cdot) + b\right), \tag {12}
$$
where  $g_{e}$  is a gate unit,  $\mathrm{ATTN}_h(\cdot)$  and  $\mathrm{ATTN}_c(\cdot)$  are attention models (see Eq. (1)) with different parameters.  $\mathrm{O}_e^{n,l}$  is further processed by  $\mathrm{FFN}(\cdot)$  to output the representation  $\mathrm{H}_e^{n,l}$ . Symmetrically, in the decoder,  $\mathrm{S}_d^n$  in Eq. (2) can be calculated as
$$
\begin{array}{l} \mathrm {S} _ {d} ^ {n} = g _ {d} \odot \mathrm {A T T N} _ {h} \left(\mathrm {O} _ {d} ^ {n}, \mathrm {B} _ {e} ^ {n}, \mathrm {B} _ {e} ^ {n}; \Theta_ {d} ^ {n}\right) \\ + \left(1 - g _ {d}\right) \odot \operatorname {A T T N} _ {c} \left(\mathrm {O} _ {d} ^ {n}, \mathrm {C} ^ {n}, \mathrm {C} ^ {n}; \Theta_ {d} ^ {n}\right) \tag {13} \\ + \mathrm {O} _ {d} ^ {l} \\ \end{array}
$$
3For a detailed derivation, we refer the reader to Appendix B.
where  $\mathrm{O}_d^n$  is the output of the self-attention sublayer defined in Eq. (2).  $g_{d}$  is another gate unit.
We first evaluate the proposed method on IWSLT14 English  $\leftrightarrow$  German (En  $\leftrightarrow$  De) and IWSLT17 English  $\rightarrow$  French (En  $\rightarrow$  Fr) benchmarks. To make the results more convincing, we also experiment on a larger WMT14 English  $\rightarrow$  German (En  $\rightarrow$  De) dataset.
Dataset. The dataset for IWSLT14  $\mathrm{En} \leftrightarrow \mathrm{De}$  are as in Ranzato et al. (2016), with  $160k$  sentence pairs for training and 7584 sentence pairs for validation. The concatenated validation sets are used as the test set (dev2010, dev2012, tst2010, tst2011, tst2012). For  $\mathrm{En} \rightarrow \mathrm{Fr}$ , there are  $236k$  sentence pairs for training and 10263 for validation. The concatenated validation sets are used as the test set (dev2010, tst2010, tst2011, tst2012, tst2013, tst2014, tst2015). For all IWSLT translation tasks, we use a joint source and target vocabulary with  $10k$  byte-pair-encoding (BPE) types (Sennrich et al., 2016). For the WMT14  $\mathrm{En} \rightarrow \mathrm{De}$  task, the training corpus is identical to previous work (Vaswani et al., 2017; Wang et al., 2019a), which consists of about 4.5 million sentence pairs. All the data are tokenized using the script tokenizer.pl of Moses (Koehn et al., 2007) and segmented into subword symbols using jointly BPE with  $32k$  merge operations. The shared source-target vocabulary contains about  $37k$  BPE tokens. We use newstest2013 as the development set and newstest2014 as the test set. Following previous work, we evaluate IWSLT tasks with tokenized case-insensitive BLEU and report tokenized case-sensitive BLEU (Papineni et al., 2002) for WMT14  $\mathrm{En} \rightarrow \mathrm{De}$ .
Model Settings. For IWSLT, the model configuration is transformer_iwslt, representing a small model with embedding size 256 and FFN layer dimension 512. We train all models using the Adam optimizer  $(\beta_{1} / \beta_{2} = 0.9 / 0.98)$  with adaptive learning rate schedule (warm-up step with 4K for shallow models, 8K for deep models) as in (Vaswani et al., 2017) and label smoothing of 0.1. Sentence pairs containing  $16\mathrm{K}\sim 32\mathrm{K}$  tokens are grouped into one batch. Unless otherwise stated, we train small models with 15K maximum steps,
Depth	36-layer	54-layer	72-layer
dec. (N)	6	6	6
enc. (N×M)	6×6	6×9	6×12
Table 1: Deep architectures of MSC on IWSLT tasks. We simply set  $M_{1} = \dots = M_{N} = M$
and decode sentences using beam search with a beam size of 5 and length penalty of 1.0.
For WMT14 En  $\rightarrow$  De, the model configuration is transformer_base/big, with a embedding size of 512/1024 and a FFN layer dimension of 2048/4096. Experiments on WMT are conducted on 8 P100 GPUs. Following Ott et al. (2018), we accumulate the gradient 8 iterations and then update to simulate a 64-GPU environment with a batch-size of 65K tokens per step. The Adam optimizer  $(\beta_{1} / \beta_{2} = 0.9 / 0.98$  for base,  $\beta_{1} / \beta_{2} = 0.9 / 0.998)$  for big) and the warm-up strategy (8K steps for base, 16K steps for big) are also adopted. We use relatively larger batch size and dropout rate for deeper and bigger models for better convergence. The transformer_base/big is updated for 100K/300K steps. For evaluation, we average the last 5/20 checkpoints for base/big, each of which is saved at the end of an epoch. Beam search is adopted with a width of 4 and a length penalty of 0.6. We use multi-bleu.perl to evaluate both IWSLT and WMT tasks for fair comparison with previous work.