Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
Authors: Jungo Kasai
Nikolaos Pappas
Hao Peng
James Cross
Noah Smith
Venue: ICLR 2021 Poster (2021)
Keywords: Machine Translation
Sequence Modeling
Natural Language Processing

Abstract: Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs.  In contrast to the latter, where generation is sequential, the former allows generation to be parallelized across target token positions. Some of the latest non-autoregressive models have achieved impressive translation quality-speed tradeoffs compared to autoregressive baselines. In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study autoregressive models with encoders and decoders of varied depths. Our extensive experiments show that given a sufficiently deep encoder, a single-layer autoregressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. Our results establish a new protocol for future research toward fast, accurate machine translation. Our code is available at https://github.com/jungokasai/deep-shallow.

Published as a conference paper at ICLR 2021
DEEP ENCODER, SHALLOW DECODER: REEVALUATING NON- NON-AUTOREGRESSIVVE MACHINE
TRANSLATION
Jungo *asai
Nikolaos Pappas Hao Peng James Cross*
Noah A. Smith
Paul G. Allen School of Computer Science & Engineering, University of Washington
Facebook AI AAllen Institute for Al (jkasai, , mpappas, hapeng, nasmi t.h thheess washingtoon.eei jcrosse fb. com
ABSTRACT
Much recent effort has been invested conreettreeressvvee neural machine trans- lation, which appears to be an efficient alternative to state-of-the-arr aftt autoregres- sive machine translation on modern GPUs. contrast to the latter, where gen- eration is sequential, the former allows generation to be parallelized across target token positions. Some of the latest non -atrrrrrrrrreevvee models have achieved im- pressive translation quality- spplit tradeoffs compared to autoregressive baselines. In this work, we reexamine this tradeoff and argue that autoregressive baselines can be substantially sped up without loss in accuracy. Specifically, we study au- toregressive models with encoders and decoders of varied depths. Our extensive experiments show that given sufficiently deep encoder, a single-layer autore- gressive decoder can substantially outperform strong non-autoregressive models with comparable inference speed. We show that the speed disadvantage for au- toregressive baselines compared to non-autoregressive methods has been overes- timated in three aspects: suboptimal layer allocation, insufficient speed measure- ment, and lack of knowledge distillation. Our results establish new protocol for future research toward fast, accurate machine translation. Our code is available at https://gitse com/ http:/jnnnkaaaadeeepeae shalloo
INTRODUCTION
Fast, accurate machine translation is fundamental goal with a wide range of applications both in research and production. State-of-the-arr neural machine translation systems generate translations autoregressively where words are predicted one-by-one conditioned on all previous words (Kalch- brenner& & Blunsom 2013; Sutskever et al. 2014 Bahdanau et al. 2015 Wu al., 2016 Vaswani et al. 2017). This sequential property limits parallclization, since multiple tokens in each sentence cannot be generated in parallel. A flurry of recent work developed ways to (partially) parallelize the decoder with autoreerrgeeevee machine translation (NAR: Gu et al. (2018), thereby speeding up decoding during inference. NAR tends to suffer in translation quality because parallel decoding assumes conditional independence between the output tokens and prevents the model from properly capturing the highly multimodal distribution of target translations (Gu et al. 2018).
Recent work proposed methods to mitigate this multimodality issue, including iterative refinement (e.g.. Lee et al.] 2018 Ghazvininejad et al., 2019), and modeling with latent variables (e.g.. Ma et al. 2019 Shu et al. 2020). These approaches modify the decoder transformer to find a balance between decoding parallelism and translation quality. In this work, however, we adopt a different speed-quality tradeoff. Recent work by Kim et al. (2019) in autoregressive machine translation (AR) suggests that better speed-quality tradeoffs can be achieved by having different depths in the encoder and the decoder. Here, we make formal argument in favor of deep encoder, shallow decoder configurations and empirically demonstrate better speed-quality tradeoff's for the AR baselines.
Work partially done Facchook AI.

---

We provide extensive speed-quality comparisons between iterative NAR models and AR models with varying numbers of encoder and decoder layers. In particular, we use two types of speed measures for translation and discuss their relation to computational complexity. The two measures reflect two different application scenarios: feeding one sentence at a time, and feeding as many words possible into the GPU memory. The first scenario is designed to simulate, for example, instantaneous machine translation that translates text (or even speech) input from users. This is Published as a conference paper at ICLR 2021
where current NAR models shine-we can make full use parallelism across decoding positions in GPU. For this reason, much prior work in NAR only measures speed using this metric (e.g., Gu al., 2018 2019b] Kasai et al., 2020 Li et al., 2020). The second scenario aims at a situation where we want translate large amount of text as quickly as possible. In this case, we see that AR models run faster than NAR models by a large margin. Computation at each time step is large enough to exploit parallelism in a GPU, which cancels out the benefit from parallel NAR decoding. Further, AR models can cache all previous hidden states (Ott et al. 2019) and compute each step in linear time complexity with respect to the sequence length. In contrast, NAR models necessitate fresh run of quadratic self and cross attention in every decoding iteration.
Interestingly, using deep encoder and a shallow decoder in NAR models fails retain the original translation accuracy by using 6 layers each ($4.1). This suggests that departure from AR decoding necessitates more capacity in the decoder; the strategy is effective specifically for AR models. In particular, our analysis demonstrates that an NAR decoder requires more layers to learn target word ordering (5 In summary, our contributions are the following:
We challenge three conventional assumptions in NAR evaluation: suboptimal layer alloca- tion, lack of distillation for AR baselines, and insufficiently general speed measures.
We provide a complexity analysis and identify an optimal layer allocation strategy that leads to better speed-quality tradeoffs, namely a deep-shallow configuration.
We perform extensive analyses and head-to-heat comparisons of AR and strong NAR mod- els on seven standard translation directions. We demonstrate that the accuracy gap between the two model families is much wider than previously thought and that NAR models are unable to capture target word order well without sufficiently deep decoders.
2 REEVALUATING NON-AUTOREGRESSIVE MACHINE TRANSLATION
We critically examine in this section the evaluation practices and assumptions that are widely held in the non-autoregressive neural machine translation (NAR) literature (e.g.. Gu et al. 2018 Ghazvininejad et al., 2019 Kasai al., 2020). In particular, focus three aspects: speed measurement ( (.21 layer allocation ($2.) and knowledge distillation ( 2.3
2.1
SPEED MEASURES
One major benefit of NAR models over AR ones is their ability to generate text in parallel. Current research on measuring speed has focused solely on the setting of translating one sentence at a time where full parallelization is trivial with a single GPU. However, we argue that this speed measure is not realistic in some scenarios because the GPU memory is finite and the GPU unit in such setting is underused. To address this issue, we use two translation speed metrics to measure inference speed:
S₁ measures speed when translating one sentence at a time. This metric used standard practice and aligns with applications like instantaneous machine translation that translates text imput from users immediately.
Smax measures speed when translating in mini-batches as large the hardware allows. This corresponds to scenarios where one wants to translate a large amount of text given in advance. For instance, such large-batch machine translation is implemented in the Google cloud service
For all models, both metrics measure wall-clock time from when the weights are loaded until the last sentence is translated. We report speedups relative to an AR baseline with a 6-layer encoder and a 6-layer decoder following prior work (Gu al. 2018 Li et al. 2020 Kasai et al. 2020
https://cloud, http:///...ggg....momoaaaaaaaaaa http:///...../rraaaaaaaa/////aaa http://aaaaaaaaeeeoo////a//aaaaa
antion
2

---

Published as a conference paper at ICLR 2021
2.2 LAYER ALLOCATION
Current evaluation practice in the NAR literature uses an equal number of layers for the encoder and decoder both AR baselines and NAR models. However, previous studies in AR machine translation suggest that this allocation strategy leads to suboptimal speed-quality tradeoff (Barone) at. Kim et al. 2019). These findings have several implications for evaluating NAR methods. We first discuss the strategy of deep encoder, shallow decoder ($2.2.T). and provide a theoretical analysis of the speed-quality tradeoff in the context of NAR evaluation ( (....2) Our analyses are verified empirically in the next section 3
2.2.1 DEEP ENCODER, SHALLOW DECODER
In line with prior work on deep encoders or shallow decoders (Barone al.) 2017; Wang et al. 2019a; Kim et al. 2019), we depart from the convention to allocate an equal number of layers on both sides and explore pairing deep encoder with shallow decoder for both AR and NAR methods. Here, we study the impact of such architectures and systematically compare AR and NAR [ methods As we will show in $ an AR model with a deep-shallow configuration retains translation accuracy, but can substantially reduce decoding time. This is because at inference time, the encoder accounts for smaller part the overhead since its computation can be easily parallelized over source positions; the other hand, the speedup gains from lightweight decoder substantial.
By Layer
Full Model
Enc.
AR Dec.
E-D O(TN2) O(EN² + DN O(N2) O(TN) O(EN DN2)
NAR Dec.
AR E-1 OENN' + +1.N9)
NAR E-D OEEN" + DTN*) M(EN+N9) OEEN DTN)
Total Operations O(N*)
(NN))
Time Complex. O(N)
Table I: Analysis of transformers. Time complex. indicates time complexity when full paralleliza- tion is assumed. N: source/target length; E: encoder depth; D: decoder depth; T: # NAR iterations.
2.2.2 COMPLEXITY ANALYSIS
This section analyzes the complexities of transformer-based encoders, autoregressive and non- autoregressive decoders. We focus on two key properties: (1) the total amount of operations and (2) time complexity when full parallclization is assumed (Harris) 2007).
Notation For simplicity let us assume the source and target text have the same length N. T is the number of iterations in an iterative NAR method (typically T' < N). Let E and D denote the numbers of encoder and decoder layers.
Table breaks down the comparison. AR and NAR models use the same encoder architecture. There are several interesting distinctions between AR and NAR decoders. First, although their total amounts of operations are both quadratic in sequence length, an NAR decoder with T decoding iterations needs T times more computation. Second, an AR decoder has time complexity quadratic in sequence length. This contrasts with the linear time complexity of an NAR decoder, which is the powerhouse of its S speedup ($4.1) This is because the attention computation can be readily parallelized across target positions in NAR decoders.
By such comparisons make the following key observations:
(a) For both AR and NAR models, the time complexity dominated by decoders. When T < N. an NAR model has an advantage over its AR counterpart with the same layers. (b) Innocuous as it may seem, the constant T contributes major computational cost in terms of the total operations of NAR models. Empirically. T needs to be at least to perform competitively to AR models (Ghazvininejad et al. 2019 Kasai et al. 2020).
(a) suggests that can significantly speed up S decoding by using shallower decoders, while increasing the encoder depth only results in a mild slowdown. As we will show later in the experi- ments, AR decoders much more robust to using fewer layers than NAR decoders. For example,
Note that Kim et al. 2019) proposed other methods to optimize CPU decoding of AR models, but we do apply them, ensure fair comparisons between AR and NAR models.
3

---

Published as a conference paper at ICLR 2021
AR E-1 can decode much faster than AR E-D and comparably to NAR E-D, while retaining the accuracy of AR E-D. From (b), one may expect a different trend in Smax from S₁: in large mini- AR model can make use of the GPU's compute units, since
batch decoding, happens over the instances in a mini-batch. In other words, under the Smax evaluation where the GPU running close to its maximum flop/s, NAR can actually be slower since it needs more operations due to its iterative decoding. This is confirmed by our experiments ($4.1)
parallelization
2.3 KNOWLEEDGE DISTILLATION
Most NAR models rely on sequence-level knowledge distillation (Hinton al., 2015 Kim & Rush 2016 to achieve reasonable speed-quality tradeoff, where NAR models are trained on output translations from (larger) AR model. Nevertheless, standard practice in this area assumes that knowledge distillation is not required for the AR baseline. Here, we aim for fair evaluation by applying distillation to both model families; we depart from previous practice where NAR models trained with distillation
compared with AR models trained without (Ran et al., 2021: Sun al., 2019] Shu et al., 2020] Zhou et al. 2020 Saharia et al.] (2020) with few exceptions (Gaazvinin- jad al., 2019:; Kasai et al.) 2220). Our analysis ($5) demonstrates that AR models also benefit from knowledge distillation and that the accuracy gap between AR and NAR models is wider than
previously established.
EXPERIMENTS
We compare NAR and AR models with different layer allocation strategies standard machine translation datasets of varying languages and sizes. Our results show that deep-shallow AR models provide better speed-quality tradeoff than NAR models.
3.1 BASELINES AND COMPARISON
Prior work has proposed various approaches to non-autoregressive machine translation (NAR). These methods must seek a balance between speed and quality: the more decoding parallelization is introduced into model, the more the output quality deteriorates due to a conditional independence assumption. Some of the existing NAR models rescore the candidates with external autoregressive models (Sun et al.] 2019 Li et al. 2020),, or apply reordering modules (Ran et al. (021). We mainly
compare with two iterative NAR models (Ghazvininejad et al. 2019: Kasai et al. 2020) because of
their strong performance withou relying on any external system:
CMLM (Ghazvininejad et al., 2019) predicts randomly masked target tokens given ob- served ones well the source. At inference time, it first predicts all target words non- autoregressively, and then iteratively masks and predicts the words that the model least confident about. Following previous practice (Ghazvininejad et al. 2019 2020b), we de- code 5 candidate lengths in parallel (length beam) with T = or T 10 iterations. DisCo (Kasai al.] (020) predicts every target token given arbitrary subset of the rest of the target tokens. Following Kasai et al. 2020), we use their parallel easy-first inference, and set the maximum number of iterations 10 and the length beam size 5.
Knowledge Distillation We apply sequence-level knowledge distillation (Hinton et al. 2015 Kim & Rush 2016) when training both NAR and AR models ($2.3) For the teacher models, we use left-to-right AR transformer models: transformer-large for EN-DE, EN-ZH, and EN-FR, and transformer- base for EN-RO (Ghazvininejad et 2019 Kasai et al. 2020
3.2 EXPERIMENTAL SETUP
We experiment with 7 translation directions from four datasets of various training data sizes:
WMT14 EN-DE (4.5M pairs, Bojar et al. 2014), WMT16 EN-RO (610K, Bojar et al. 2016). WMT17 EN-ZH (20M, Bojar et al. 2017). and WMT14 EN-ER (36M, EN-FR only). These datasets are all encoded into BPE subwords (Sennrich et al. 2016). We follow the preprocessing
and data splits of previous work (EN- RDD Vaswani et al. 2017 EN-RO: Lee al. 2018 EN-ZH:
Hassan et al. 2018 Wu et al. 2019: EN-FR: Gehring et al. 2017). Following previous practice, we evaluate EN-ZH performance, and BLEU (Papineni et al. 2002)
use SacreBLEE BL (Pst 2018)
4

---

Published as a conference paper at ICLR 2021
for others For all autoregressive models, apply beam search with size 5 and length penalty 1.0. All models are implemented using fairsec (Ott et al. 2019). S₁ and Smax wall-clock time speedups ($2) are evaluated on the same single Nvidia VIOO GPU with 16GB memory. We apply half-precision training and inference (Micikevicius et al. 2018 Ott et al. 2019). speeds up NAR models' Smax by 30+%, but not S1, in line with previous observations (Kim et al. 2019).
Hyperparameters We follow the hyperparameters of the base sized transformer (Vaswani et al, 2017): 8 attention heads, 512 model dimensions, and 2,048 hidden dimensions for both the encoder and decoder. For each model and dataset, the dropout rate is tuned from [0.1, 0.2, 0.3] based on development BLEU performance. The EN-FR models are trained for 500K updates, while others for 300K (Kasai et al. 2020). Dev. BLEU is measured after each epoch, and we average the best checkpoints to obtain the final model (Vaswani et al. 2017). See the appendix for further details.
4 RESULTS AND DISCUSSION
We provide in-depth results comparing performance and speedup across AR and NAR models.
X NAR: CMLM T=4
NAR: CMLM T=10
NAR: DisCo
AR (C) EN +RO S
(A) EN MNN-DDE S1
(B) DE SEN S1
6-6
28
6-6
12-1
31.5
6-6
12-1
6-6
34.5
BLEU
27.5
6-6
6-6
31
12-1
27
6-1
6-6
30.5
6-1
12-1
33.5
33
6-1 6:0
12-1
6-6
12-1
26.5
12-1
30
32.5
12-1
12:
26
12-1
29.5
12-1
32
1x
3x
5x
1x
3x
5X
7x
1x
3x
5x
7x
(D) RO St
(F) EN DE Smax
(F) RO- Smax
34.5
6-6
12-1
28
6-6
12-
34.5
6-6
12-
8-1
6-1
BLEU
34
27.5
6-6
6-6
666
6-1
6-6
33.5
6-66-6
27
12-1
33.5
6-6
33
12-1
12-
26.5
6-6
33
32.5
12-1
26
32.5
$12
3x
5x
0.5x
1x
1.5x
0.5x
1x Speedup
1.5x
2x
Speedup
Speedup
Figure BLEU and speed comparisons with varying numbers of encoder and decoder layers on the test data. 12-1 denotes 12 encoder layers and 1 decoder layer. AR deep-shallow (12-1) finds balanced middle ground in the tradcoff. Knowledge distillation is applied to all models ($3.)) See Table in Appendix for more results.
4.1 DEEP ENCODER, SHALLOW DECODER
Fig. I shows translation speed-quality tradcoff curves of CMLM, DisCo, and AR models on WMT14 EN-DE and WMT16 EN-RO test data. For each model we plot the results of configurations with varying encoder and decoder depths. For brevity, denote by E-D model with E-layer encoder and a D-layer decoder. All speedups are measured relative to the AR 6-6 baseline ($2).
Firstly, under the 6-6 configuration, the AR model outperforms both CMLM and DisCo by a siderable margin in BLEU, but it achieves the slowest S (see Fig. IA-D). Using single-layer decoder, AR 6-1 gains substantial S speedup (2.6x for EN-DE and 2.9x for RO-EN), but this comes at cost of BLEU: 28.34 vs. 27.39 for EN-DE, and 34.57 vs. 34.31 for RO-EN, AR 12-1
SScreBBLER REU hash: Bl. http://w....i.eooo......ea.mm/.o htt.://////.....eeeeiiooooo sh+version..t.... 1.3.7.
7tok
5

---

Published as a conference paper at ICLR 2021
lands a balanced middle ground: it yields similar BLEU to AR 6-6, but its S₁ is more than 2.5 times faster. Notably, AR 12-1 achieves even faster S₁ than that of the CMLM 6-6 model with 10 it- erations. In contrast, NAR 12-1 models generally suffer in BLEU compared to the 6-6 configuration; e.g., 26.75 (DisCo 12-1) vs. 27.35 (DisCo 6-6) in EN-DE.
Interestingly, all NAR models achieve slower Smax than the AR 6-6 baseline (DisCo 6-6: 0.3x; CMLM 6-6 7=10: 0.1x in RO-EN). This is consistent with our complexity analysis in $2.2.2 where found that with the same layer allocation, iterative NAR models need more total com- putation than the AR counterpart. AR 12-1 still gains a considerable speedup over AR 6-6 (2.0x RO-EN). These results suggest that current NAR models have little advantage when translating a large amount of text given in advance, and one should clarify this distinction when discussing translation speed. See Table in the appendix for full results from all four directions.
Model
WMT17 EN-ZH
WMT17 ZH-EN
WMT14 EN-FR
T
E-D
BLEU
S₁
Smax
BLEU
S₁
Smax
BLEU
S
Smax
CMLM
4
6-6
33.58
3.5x
0.2x
22.56
3.8x
0.2x
40.21
3.8x
0.2x
CMLM
10
6-6
34.24
1.5x
0.1x
23.76
1.7x
0.1x
40.55
1.7x
0.1x
DisCo
6-6
34.63
2.5x
0.2x
23.83
2.6x
0.2x
40.60
3.6x
0.2x
AR Deep-Shallowe
12-1
34.71
2.7x
1.7x
24.22
2.9x
1.8x
42.04
2.8x
1.9x
AR
6-6
35.06
1.0x
1.0x
24.19
1.0x
1.0x
41.98
1.0x
1.0x
Dist. Teacher
6-6
35.01
24.65
42.03
Table 2: Test BLEU and speed comparisons with varying numbers of encoder (E) and decoder (D)
layers on large bitext. Best performance is bolded.
Table 2 presents results from large bitext experiments, EN++ZH and EN-FR. We observe similar
trends: AR deep-shallow achieves similar BLEU to AR 6-6 while boosting both Sj and Smux speed substantially. For EN+ZH, AR deep-shallow has more S speedup than DisCo (2.7x vs. 2.5x in EN-ZH, 2.9 vs. 2.6 in ZH-EN). Particularly noteworthy is its performance in EN-FR: 42.04 BLBU, 1.4 point improvement over the best NAR model. These results illustrate that the strategy of having a deep encoder and shallow decoder remains effective in large bitext settings, when the model has learn potentially more complex distributions from more samples.
Lastly, Table Table3ccompan coompress AR deep-shallow to recent iteration-based NAR results, All NAR models use the 6-6 configuration with the base size except that Imputer (Saharia al. 2020) uses 12 self- attention layers over the concatenated source and target. Overall, our AR deep-shallow models outperform most NAR models, with the only exception being EN-RO where underperforms Imputer by 0.6 BLEU points. However, each iteration takes strictly more time in the Imputer model than in CMLM or DisCo, since requires a fresh run of 12-layer self attention over a concatenation of input and output sequences. As we saw in Fig. [ AR deep-shallow yields comparable S₁ to CMLM 6-6 with iterations, which would be about twice as fast as Imputer with itcrations.
4.2 CONSTRAINED VIEWS
In this section, we present two controlled experiments compare NAR and AR models thoroughly.
S₁ Speed Constraint From $4.1 we see that compared to NAR models, AR deep-shallow yields a better translation speed-quality dalance-desppte being slightly slower in S₁ on some of the datasets, it achieves better BLEU across the board. To confirm this result, we further compare an AR deep- shallow model against two NAR models, controlling for S, speed. More specifically, we experiment with NAR models of varying encoder depths, and pair each with as many decoder layers as possible until it reaches AR 12-1's S) speed. Fig. 2 (left) shows the results. For CMLM T44, CMLM 10= and DisCo. the best configurations of 1 12-layer encoders were paired up with 12, 4, and 9 decoder layers, respectively. All NAR models improve performance as the encoder becomes deeper and surpass the scores of the 6-6 baselines (shown squares along 6). Nonetheless, there is still large BLEU gap from AR 12-1. This illustrates that the two NAR models are not able to match AR deep-shallow's accuracy under the same S speed budget.

---

Published as a conference paper at ICLR 2021
WMT14 EN-DE DE
WMT16 EN-RO RO
WMT17 ZN-ZH
Model
+DE
T
+EN
T
ROO
T
+EN
T
+ZH
T
+EN
T
CMLM
25.9
4
29.9
32.5
4
33.2
32.6
21.9
27.0
10
31.0
10
33.1
10
33.3
10
33.2
10
23.2
10
LevT
27.3
>7
33.3
>7
DisCo
27.3
4.8
31.3
4.2
33.2
3.3
33.2
3.1
34.6
5.4
23.8
5.9
SMART
27.0
4
30.9
4
33.4
22.6
27.6
10
31.3
10
34.1
10
23.8
10
Imputer
28.0
4
31.0
34.3
34.1
28.2
8
31.3
34.4
8
34.1
8
6-6
28.3
N
31.8
N
34.6
N
34.6
N
35.1
N
24.2
N
AR 12-1
28.3
N
31.8
N
33.8
N
34.8
N
34.7
N
24.2
N
Teacher
28.6
N
31.7
N
34.6
34.6
N
35.0
N
24.7
N
Table 3: Test BLEU comparisons with iterative NAR methods. T indicates the average # iterations. CMLM: Ghazvininejad et al. (2019); LevT: Gu et al.
(2019b); DisCo: Kasai al. (2020); SMART:
Ghazvinmeejad et al. (2020b); Imputer: (Saharia al. (2020). Best performance is bolded.
Layer Constraint We can speed up autoregressive translation (AR) by developing a model with a deep encoder and one-layer decoder. Here we thoroughly compare layer allocation strategies. Shown in Fig. (middle) are results of NAR and AR methods under the constraint 12 transformer layers in total. NAR models perform well when the decoder and encoder are balanced with slight tendency to deep encoders. On the other hand, the AR models perform consistently well with 4 or more encoder layers. This confirms that using deep encoders and shallow decoders is more effective in AR models than NAR ones. Note that the number of parameters in each layer allocation differs since decoder layer contains 30% more parameters than encoder layer, due to cross attention.
Decoder Layers
10
28.5
CMLM T=4
AR 12-T
CMLM T=
28
5x
CMLM 6.6 T T=4
28.0
DisCo
27
Relative Spoodup
4x
CMLM 6-6 T
DisCo 6.6
BLBB
BLEU
AR12-1
27.5
10 6-6
26
3x
DisCo 6-6
CMLM T=4 CMLM T=10
2x
27.0
25
DisCo
26.5
6-6
24
AR
12
mano
# Encoder Layens
Enooder Layers
# Sentences per Batch
varying depths of the en- coder under the S, speed constraint AR 12-1 m. Middle: varying allocation of total of 12 transformer layers over the encoder and decoder. Right: varying inference batch sizes.
Figure 2: WMT14 EN-DE test results under various conditions. Left:
5 FURTHER ANALYSIS
Speedup and Batch Size When decoding with large batches, NAR models can be slower than their AR counterpart ($4.1). Here we further study this effect. Fig. 2 (right) plots the relative speedups of different models" decoding with varying numbers of sentences per batch up to the hardware limit ("max." " 12.1). The speedup by NAR models diminishes as the batch size grows: they have similar decoding speed to AR 6-6 with batch size 50, and become slower with larger batch sizes. In contrast, the speedup from AR deep-shin shap-shallow decreases much more gradually.
Decoder Depth and Reordering Words From earlier results decoders than AR models to perform well. We hypothesize that one reason is that NAR decoders need to learn adjust to diverging word order between the source and the target: AR decoder
that NAR models need deeper

---

Published as a conference paper at ICLR 2021
Model
E-D
Orig.
Reorder
A
Model
E-D
Raw
Dist.
A
CMLM, T 10
6-6
27.4
31.7
4.3
CMLM, T=
6-6
22.3
25,9
3.6
CMLM, T 10
12-1
26.3
31.0
4.7
CMLM, T= 10
6-6
24.6
27.0
2.4
Imputer, T
12
24.7
27.9
3.2
DisCo
6-6
27.4
31.0
3.6
Imputer, 8
12
25.0
27.9
2.9
DisCo
12-1
26.8
31.6
4.8
DisCo
6-6
24.8
27.4
2.6
AR
6-6
28.3
32.6
4.3
AR Deep-Shallow
12-1
26.9
28.3
1.4
AR Deep-Shallle
12-1
28.3
32.6
4.3
AR
6-6
27.4
28.3
0.9
Table 4: Left: WMT14 EN-DE test results in BLEU using reordered English input. Right: WMT14 EN-DE test results in BLEU that analyze the effects of distillation in fast translation methods. All distillation data are obtained from a transformer large. E: encoder depth; D: de- coder depth; T: iterations. Imputer (Saharia et al. 2020) uses 12 self-attention layers over the concatenated source and target, instead of the encoder-decoder architecture.
takes input all preceding tokens and explicitly learns conditional distribution, while an NAR decoder needs to learn target word ordering from scratch.
To test this hypothesis, conduct the following controlled experiment in EN-DE translation. We choose German because of its divergence in word order from English. We first run the fast align all bitext data (including the test set), and disable the NULL word
tool (Dyer al. 2013 feature to ensure that every English word is aligned to exactly one German word. We then shuffle the English words according to the order of their aligned German words. When multiple English words are aligned to the same German word, we keep the original English order. Finally, we apply the same BPE operations as the original data, and train and evaluate various models on the new reordered data. Table 4 (left) shows the results. AR gains the same improvement regardless of the layer configuration; contrast, NAR 12-1 benefits more than NAR 6-6. This result supports our hypothesis that word reordering is one reason why NAR models need deeper decoder.
Effects Distillation We applied sequence-level knowledge distillation (Kim Rush 2016) to all models. Here we analyze its effects over the WMT14 EN-DE test data (Table 4 right). An AR transformer large model is used as the teacher model. All models benefit from distillation as indicated by positive A, including the AR models Many recent works only compare NAR models trained with distillation to AR models trained without. Our finding shows that that AR models with distillation be additional baseline for future NAR research. AR deep-shallow deteriorates much less on the raw data compared to the iterative NAR methods, suggesting that the strategy of speeding up AR models is better suited to modeling raw, complex data than the NAR methods.
AR 6-6
AR Deep-Shallow 12-1
(A) RO-EN
(B) Z11 >EN
(C) EN-
26
34
24
22
34
20
32
18
30
16
14
28
28
1-10 11-20 21-30 31-40 40+
1-10
11-20
31-40 40+
1-10
11-20 21-30 30--3031-400 40+ Reference Length
Reference Length
Reference Length
Figure 3: Test BLEU and target length comparisons for the AR 6-6 and deep-shallow 12-1 models.
https::/ //gtthub com/ clap// fast Align
While the same distillation data are used in Ghazvininejad al. (2019) ,they report smaller improvement
from distillation in BLEU. There are several potential reasons: we tuned the dropout rate for each model on
the validation data and averaged the checkpoints that achieved the top 5 BLEU scores ($3.2) We note that our
results are in line with the previous observations (Kim et al. 2019 /hou et al. 2020 Kasai et al. 2020 where similar improvement is gained by distilling AR transformer large model to base onc.
8

---

Published as a conference paper at ICLR 2021
Breakdown by Sentence Length Fig. iillstrates the relation between BLEU scores and reference translation lengths. We observe almost identical patterns between AR 6-6 and deep-shallow models, suggesting that they perform similarly regardless of the translation length.
Can we reduce the decoder further? We saw that an autoregressive model with a single-layer decoder and a sufficiently deep encoder can retain the accuracy of the baseline with 6 layers each. One may ask whether we can make the decoder even more compact. Our preliminary experiments showed that we can remove the feed-forward module from the decoder without hurting performance. This increases the S₁ speed by 10%. We leave further exploration to future work.
Length Candidates and Smax for NAR Following the original works (Ghazvinieejad et al. 2019 Kasai et al. 2020), we fixed the number of length candidates (i.c.. the length beam size) to 5 for all NAR models, but smaller beam size can speed up Smax by allowing more sentences be fed in a batch. Indeed, we found that NAR models can improve their Smax by reducing the beam size at the expense of some accuracy drop. For example, we observed loss of 0.5 BLEU points in EN-DE when decreasing the length beam size from 5 to 1. Nonetheless, NAR 6-6 models with beam size still resulted in 0.6-0.9x Smax compared to the AR 6-6 baseline.
FURTHER RELATED WORK
Non-autoregreesive Machine Translation In addition to the work already discussed, several other
works proposed to iteratively refine (or insert) output predictions (Mansimov et al.) 2019
Stern
ctt al. 2019 Gu et al. 2019a Chan et al.) 2019 2020 Li ct al.) 2020 Guo et al. 2020).
Other
approaches include adding a light autoregressive module to parallel decoding (Kaiser et al.
2018
Sun et al. 2019; Ran et al.) 2021), partially decoding autoregressively (Stern et al. 2018
2019).
rescoring output candidates autoregressively (e.g.. Gu al.
2018), mimicking hidden states of an
autoregressive teacher (Li et al. (2019), training with different objectives than vanilla cross- eross-entropy 2019b Shao et al.
(Libovický & Helel 2018 Wang
2020 Tu al. 2020 Saharia al. 20200]
Ghazvininejad et al. 2020a), reordering input sentences Ran et al., (022). training on additional
data from an autoregressive model (Zhou & Keung 2020). and modeling with latent variables (Ma) ct al. 2019 Shu et al. 2020). The approach of adding a light autoregressive module is closest to our method, but note that we pack all non-autoregressive computation into the encoder.
Optimizing Autoregressive Transformer Prior work has suggested various ways to optimize au- toregressive transformers for fast inference. For example, Kim et al. (2019) considered shallow decoders and layer tying (Dabre & Fujita 2019; DDehghani et al. 2019) on the transformer decoder and found that it sped up inference on CPUs, but not on a GPU, which was our focus. Kim et al. (019) also explored concurrent streams where multiple batches are fed at once to make better use of a GPU. Shi & Knight (017)) proposed a vocabulary reduction method to speed up the last softmax computation, [Senellart al. (2018) also adopted vocabulary reduction and explored "fat decoder, thin encoder" on RaNeb models. ZZhang et al.] (2018) used dynamic programming in an aver- age attention network to accelerate inference. [Wu et al. (2019) developed model with dynamic convolutions and compared its speed and accuracy with non-autoregressive models. Other works proposed methods to reduce attention computation in autoregressive transformers (Kitaev et al., 2020 Katharopoulos et al.) 2020:Chelbba Chelba et al. 2020; [Png et al. (02T). Some of these methods can be used orthogonally to further facilitate fast inference in transformer, but our goal is to fairly reex- amine the speed-quality tradeoff between autoregressive and non non-autoregressive approaches under the same conditions.
CONCLUSION AND FUTURE WORK
We presented theoretical and empirical studies to demonstrate that autoregressive neural machine translation can be dramatically sped up by a simple layer allocation strategy: deep encoder, shal- low decoder. Compared to strong non-autoregressive models, deep-shallow autoregressive models achieve substantial improvement in translation quality with comparable inference speed. Our results suggest that layer allocation, knowledge distillation, and speed measurement are important aspects that future work generally, a model with a deep encoder and a shallow decoder can be used for any sequence-to- sequence task, including large-scale pretraining (Lewis et al. 2020 Liu al. 2020
aooreerrrere machine translation should take into consideration. More

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Ofir Press, and Tim Dettmers as well as the anonymous reviewers for their helpful feedback and discussions this work. This research in part funded by the Funai Overseas Scholarship to Jungo Kasai. Nikolaos Pappas was supported by the Swiss National Science Foundation under the project UNISON, grant number P400P2.183911
We thank Luke Zettlemoyer, Marcin
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015. URL wtttts://aaxii...rrttte./ abs/
1409. 0473
Antonio Valerio Miceli Barone, Jindřich Helel, Rico Sennrich, Barry Haddow, and Alexandra Birch. Deep architectures for neural machine translation. In Proc. of WMT, 2017. URL https://
arxiv.org/ale www.//gg/abaai..eee
17677076331
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Lev- eling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop statistical machine translation. In Proc. of WMT, 2014. URLhttps
www. aclweb. org/ ant W14 3302
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Ne- gri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 con- ference on machine translation. In Proc. of WMT, 2016. URL https: /ttp://wwww... www.aclweb org/ anthology/ 2
Ondłej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang. Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Mat- teo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on machine translation (WMT17). In Proc. of WMT, 2017. URL https:
7 7/www. a www.. org/ antho /11774/17-4777
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Genera- tive insertion-basee based modeling for sequences, 2019. URL http://xxxvv.rrr.. wtt.oggg//bs/9/9//
01 604
William Chan, Mitchell Stem, Jamie Ryan Kiros, and Jakob Uszkoreit. An empirical study of generation order for machine translation. In Proc. of EMNLP, 2020. bttps:::///axx org/abs/ 1910. 13437
Ciprian Chelba, Mia Chen, Ankur Bapna, and Noam Shazeer. Faster transformer decodnn N-gram masked self-attention, 2020. URL /ttt///tsse///att...o//a///////a
04589
Raj Dabre and Atsushi Fujita. Recurrent stacking of layers for compact neural machine translation models. In Proc. of AAAI, 2019. /tt.
05353
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz. Universal trans- formers. In Proc. of ICLR, 2019. URL httgo/////taee/aaaaaooor////////
03819
Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameteriizttion of ibm model 2. In Proc. of NAACL, 2013. UuUnttssr/bboons......eessaooooo www.swwwwh..nnnnceccoo.moo chwhuneauhuuuuuuaaaaaa
nnasmi hhasmith/pppers/dyye thap:ppppppeeeraarraataauueaaaa smi hww.aauuneaattmt.....aaaa1..1...
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proc. of ICML., 2017. https: pttepps://axxivoor///bbb.
1705. 03122
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke S. Zettlemoyer. Mask-predict: Parallel
decoding of conditional masked language models. In Proc. of EMNLP, 2019. URL https:
w/arxiv.org/aaa 904. 09324
10

---

Published as a conference paper at ICLR 2021
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy.
Aligned cross URL https:
entropy for non-autoorerressive machine translation. In Proc. of ICML, 2020a.
//22avv.cro/aab///2044.0000 01655
Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. Semi-autooregressive training improves 0877
mask-predict decoding, 2020b. URL
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In Proc. of ICLR, 2018. URL https://arxiv
org/ hos/abs/1711 02281
Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion- decoding with automatically inferred gen- 0910000000/990000000
eration order. TACL, 2019a. URL
Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein transformer. In Proc. of NeurIPS, 2019b. URL wttptttss///iiiive wrw./orrg..ooo abs 905. 11006
Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequenee model for non- autoregressive neural machine translation. In Proc. of ACL, July 2020. URL https://ww/
aclweb. www.ee..rgganttooooyy/oooo http://220.ccl11-11. acl-main.336
Mark Harris. Optimizing parallel reduction in CUDA, 2007. URL https://developee
download wwwiia.
pdf
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mengnan Li, Shujie Liu, Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity on automatic Chinese to English news translation, 2018. URL https::t/sset http://axxvvo.ogaaa..
1803. 05567
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In Proc. NeurlPS Deep Learning and Representation Learning Workshop, 2015. URL https:
02531
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Proc. of ICLR, 2017. URL BolLhtttp///acxi.orrr
01462
Eukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Proc. of ICML, 2018.
/ttp////ww....r///aa////000000 03382
Nal Kalchbrenner and Phil Blunsom. decurrent translation models. In Proc. of EMNLP, 2013. httts:/tts:////tt
ac http://wwwwww.b...oo////aaooro//
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine translation with disentangled context transformer. In Proc. of ICML, 2020. URL https://
arxiv. www.l/.orgaabb///000 05136
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proc. of ICML, 2020. URL
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proc. of EMNLP.
2016.
Young Jin Kim, Marcin Junczys-Dommunt Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Ro- man Grundkiewiez, and Nikolay Bogoychev. From research to production and back: Ludicrously fast neural machine translation. In Proc. of WNGT, 2019, URL BRLhttts:///wwt aclweb. org/ anthology/DD13 5632
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proc. of ICLR, 2020. URL http::httt//aaaaet wtt.or/a/abbs...oaaa////// 04451

---

Published as a conference paper at ICLR 2021
Jason D. Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregresssive neural sequence modeling by iterative refinement. In Proc. of EMNLP, 2018. URL https://arxxx
org/abs 1802. 06901
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. In Proc. of ACL, 2020. URL
https://arxit wwtp:/wwwwwvoooo/ooop//ooo wwt.:/s//////oooooo/oooo
13461
Xiaoya Li. Yuxian Meng, Arianna Yuan, Fei Wu, and Jiwei Li. LAVA NAT: A non-autoregrressvv translation model with look-around decoding and vocabulary attention, 2020. URL https:
//arxiv.org/ wttp:///waaab//.......3 03084
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. Hint-based training for non-autoreggressive machine translation. Proc. of EMNLP. 2019. URL httpp:://axxvv.
Jindřich Libovický and Jindřich Helel. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proc. of EMNLP, 2018. URL BRLhttts://aaee
org/abs/1811. 04719
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. TACL, 2020. [tRlhttts:///rrxt.
wrw. abs////////// / 2001 08210
Xuezhe Ma, Chumting Zhou, Xian Li. Graham Neubig, and Eduard H. Hovy. FlowSeq: Non- autoregressive conditional sequence generation with generative flow. Proc. of EMNLP, 2019.
URL httpt/tttaaa.i....o..om http://wwwwvvoora///////////// www.rg/ab//999... 02480
Elman Mansimov, Alex Wang, and Kyunghyun Cho. A generalized framework of sequence gen- eration with application to undirected sequence models, 2019. URL https:://rxiv..ort
abs /ess/1955.. 12790
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In Proc. of ICLR, 2018. URL //axiiv..//ii//////a///i/
03740
Myle Out, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, 2018. htRp::t:: httppps://arxi...o wwg/ abs 11t./1806..000 00187
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL Demon- strations, 2019.
01038
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Proc. of ACL, 2002. URL https: www aclweb.
orc/aanthcLogy/P02-DO000
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In Proc. of ICLR, 2021. URL http::://gpenrevveee.ccc
Forum?id-gttKTaVee.
Matt Post. A call for clarity in reporting BLEU scores. In Proc. of WMT, 2018. URL https:
7/www. ac www.wwwwbbb....m. org/ www..orgggggggggg.oooooo /M18 6319
Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proc. of EACL, 2017. URL URR.hntp:::/aaxii.or/////a
05859
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-autorrerressvve neural machine translation decoding with reordering information. In Proc. of AAAI, 2021. URL https: /ttp:///axxvvvaa abs 1911 02215
12

---

Published as a conference paper at ICLR 2021
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autorregressive machine translation with latent alignments. In Proc. of EMNLP, 2020. URL httpp://aaxa org/abs 2004. 07437
Jean Senellart, Dakun Zhang, Bo Wang. Guillaume Klein, Jean-Pierre Ramatchandirin, Josep Crego, and Alexander Rush. OpenNMT system description for WNMT 2018: 800 words/sec on
single-core CPU. In Proc. of WNG, 2018. URLhttps::/ a acl www.ww.www.b........... wwt.aalcee..rggg/annoooo...... w18-2715
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016. URL hattps://ww..cccneboooo///.aaaas
P16 P16-1162
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressivve neural machine translation. In Proc. of AAAI, 2020.
URL htttt:///aaaiiiti wrg. abs abs/1911. 09320
Xing Shi and Kevin Knight. Speeding up neural machine translation decoding by shrinking run-time vocabulary. In Proc. of ACL, 2017. URL. httt:://xingshh: me vee/aata/pd:
ACL2017short.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non- autoregressive neural machine translation with deterministic inference using delta posterior. In Proc. of AAAI, 2020. WRLhhttss///axxtt wwwp./wwww//w//////////// /ttr://ab/198..771111
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep au- toregressive models. In Proc. of NeurIPS, 2018. URL
031
Mitchell Stem,, William Chan, Jamie Ryan Kiros, and Jakob Uszkoreit. Insertion transformer: flexible sequence generation via insertion operations. In Proc. of ICML, 2019. URL https:
Zhiqing Sun. Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. In Proc. of NeurlPS, 2019. URL
abs/ 1tt./1910.1155
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Proc. of NewrlPS, 2014.
3215
Lifu Tu, Richard Yuanzhe Pang. Sam Wiseman, and Kevin Gimpel. ENGINE: Energy-based in- ference networks for non-autoregreessive machine translation. In Proc. of ACL, 2020. URL
https://arxxv. wwtp://ggg//ab////ooo/ooosss 00850
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurlPS, 2017. URL
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proc. of ACI, 2019a. URL htRps:
//arxiv.or//aba.. 906. 01787
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In Proc. of AAAI, 2019b. URL https://
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR, 2019. URL https:tppp./ hotttttpps///aaa //apxii/////..../////////// orr/ abs/ 901 10430)
13

---

Published as a conference paper at ICLR 2021
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws. Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's ral machine translation system: Bridging the gap between human and machine translation, 2016.
URL http: wrg.orggg//aooo awtp/ab///aa/ 1609. 08144
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelcrating neural transformer via an average attention network. Proc. ACL, 2018. URL
Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation in non- autoregressive machine translation. In Proc. of ICLR, 2020. URL https://arxiv..ot
abbs/1911.02
Jiawei Zhou and Phillip Keung. Improving non-autoregresssive neural machine translation with monolingual data. In Proc. ACI, 2020.
A APPENDIX
A.1
RESULTS
Model
WMT14 EN-DE
WMT16 EN-RO
E-D
+DE
S
Smax
ENN
S₁
Smax
+RO
S
Smax
+EN
S
Smax
4
6-6
26.7
4.4x
0.3x
30.8
4.1x
0.3x
33.0
3.6x
0.3x
33.3
3.5x
0.2x
6-6
27.4
1.9x
0.2x
31.2
1.8x
0.2x
33.3
1.6x
0.1x
33.7
1.5x
0.1x
CMLM
4
12-1
24.7
7.6x
0.4x
29.4
6.9x
0.4x
31.9
6.6x
0.3x
32.7
5.9x
0.3x
10
12-1
26.3
4.3x
0.2x
30.3
4.0x
0.2x
32.4
3.5x
0.1x
33.0
3.4x
0.2x
6-6
27.4
3.6x
0.3x
31.3
3.6x
0.3x
33.2
4.0x
0.2x
33.3
4.1x
0.2x
DisCo
12-1
26.8
6.4x
0.4x
30.6
6.2x
0.4x
32.6
6.0x
0.3x
32.6
6.3x
0.3x
6-6
28.3
1.0x
1.0x
31.8
1.0x
1.0x
34.6
1.0x
1.0x
34.6
1.0x
1.0x
AR
6-1
27.4
2.7x
1.4x
30.8
2.6x
1.5x
33.2
3.0x
2.0x
34.3
2.9x
2.0x
12-1
28.3
2.5x
1.4x
31.8
2.5x
1.4x
33.8
2.9x
2.0x
34.8
2.9x
2.0x
Table 5: Test BLEU and speed comparisons with varying numbers of encoder (E) and decoder (D)
layers.
Table provides comparisons of speed and quality in the WMT14 EN-DE and WMT16 EN-RO datasets.
A.2 HYPERPARAMETERS AND SETTING
All of our models implemented in fairseg (Ott et al. 2019] and trained with 16 Telsa V100 GPUs CUDA 10.1, and cuDNN 7.6.3. We used mixed precision and distributed training over 16 GPUs interconnected by Infiniband (Micikevieius et al. 2018 Out et et. 20188) Apart from EN+*ZH where we used separate BPE operations, we tie all embeddings (Press & Wolf 2017 Inan et al. 2017).
We generally follow the hyperparameters chosen in Vaswani et al. (011)): Ghazvininejad et al.
2019); Kasai et al. 2020) regardless of the numbers of encoding and decoding layers Specifically, we list the hyperparameters in Table for easy replication. All other hyperparamter options are left as default values in fairseg.
'We use their
and https:
h
14

---

Published as a conference paper at ICLR 2021
Hyperparameter
Value
Hyperaarameter
Value
label smoothing
0.1
label smoothing
0.1
# max tokens
4096
# max tokens
8192
dropout rate
[0.1, 0.2,
0.3]
dropout rate
[0.1, 0.2, 0.3]
encoder embedding dim
512
encoder embedding dim
512
encoder ffn dim # encoder attn heads decoder embedding dim decoder ffn dim # decoder attn heads
2048
encoder fin dim
2048
# encoder attn heads decoder embedding dim
512
512
2048
decoder ffn dim # decoder attn heads max source positions
2048
max source positions
10000
10000
max target positions
10000
max target positions
10000
Adam Irate
5 10
Adam Irate
5 10
Adam 31
0.9
Adam 31
Adam Bg
0.98
Adam B3
0.999
Ir-scheduler
inverse square
Ir-scheduler
inverse square
warm-up Ir
1 10-7
warm-up Ir
10
# warmup updates max updates length penalty
4000
# warmup updates
10000
300K, 500K (EN-FR)
max updates
300K, 500K (EN-FR)
1.0
Table 6: Autoregressive (left) and non-autoregressivve (right) fai rsee hyperparameters and set- ting.
A.3 SAMPLE OUTPUTS
Here we provide translation outputs randomly sampled from the validation data in ZH-EN, We not find qualitative difference between AR 6-6 and deep-shallow 12-1 models.
15

---

Published as a conference paper at ICLR 2021
Source
Reference
AR 6-6
AR Deep-Shallow 12-1
The previous list is not exhaustive, of course One million people, tenth of the entire Canadian population, were dependent on government relief.
The list above certainly incomplete
The list above is certainly not complete One million people, one tenth Canada's total population. depend on government aid for their survival.
Ëλハ 1 R.
One million people, one tenth of Canada's population, live on government aid.
Women entrepreneurs were also assisted through series of training courses in small-business management, first conducted in January 1994 Amman New
Women entrepreneurs were also assisted through series of training courses on small-scale enterprise management, which began in Jamuary 1994 Amman New Camp.
Women entrepreneurs have also been assisted through series of training courses small enterprise management, which began at Amman New Camp in January 1994,
w 19
Camp.
# デズ dhr F1Z2
Indeed, the investment in both saving lives and reining in terror needs to be sustained over the medium-term in order to effectively turn the tide in this continuing tragedy.
In fact, the investment made in saving lives and controlling terror must be maintained in the medium term in order to effectively reverse the momentum of this continuing tragedy.
Indeed, investment in saving lives and controlling terror must be maintained in the medium term in order to effectively reverse the momentum of this continuing tragedy.
PJ IT (
A A XE
X ı
In this complex and convulsed world that is supposedly pluralistic. free and democratic, the United States is trying deny my country, Cuba, its rightful place as a sovereign nation. It is as if two centuries of relations between the two countries meant nothing.
I am afraid that this complex and volatile world is pluralistic, democratic and free, in which the United States is trying deprive my country of its rightful place as a sovereign State, just two hundred years of relations between the
This complex and volatile world, which is feared. pluralistic. democratic and free, the United States is seeking deprive our country its rightful place as sovereign
T. R
x A
$
State, as if two hundred years of relations between the two countries were not a matter.
two countries are not worth it.
Table 7: Sample translation outputs from the ZH- validation data.
16


Reviews:
[{"review_id": "geaf4XK0gAv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Traditional NTM is done with a large encoder and large autoregressive (AR) decoder. Due to the sequential nature of the AR decoder, inference can be slow due to lack of parallelism (unless done at very large batch sizes). Non-Autoregressive (NAR) models have been proposed to alleviate this problem, but all NAR approaches trade off some translation quality for speed gains. In this paper, the authors claim that an alternative to NAR is to speed up standard AR decoding by reallocating network weights and layers to the (easily parallelizable) encoder, and making the decoder a single layer. They claim that this matches the speed of NAR models while keeping the performance of traditional AR models, making it a better choice in the design space than any NAR models. Comparisons are made to CMLM and DisCo NAR models to justify these claims with experimental evidence.\n\nPros:\n\n1. The results are fairly compelling. A 12-layer encoder + 1-layer decoder model matches the 6/6 model in translation quality, and, as promised, speedups are comparable to the speedups obtained from NAR methods.\n\n2. The point made about knowledge distillation is also valuable. Knowledge distillation is valuable tool for speeding up models, and it is just as applicable to AR speedups as NAR speedups.\n\nCons:\n\n1. In my view, the comparison of S_{max} is misleading. According to the authors, it \"measures speed when translating in mini-batches as large as the hardware allows. This is closer to practical scenarios where one wants to translate a large amount of text\". However, this is emphatically *not* the purpose of NAR models, which are *specifically* for the low-batch-size case. In addition, I disagree that this \"is closer to practical scenarios\"; many user queries are in fact short sentences or phrases, rather than many paragraph articles. Both use cases exist, and comparing NAR vs AR models on S_{max} is comparing a slightly wrong metric.\n\nThis could be improved by instead plotting speed as a function of batch size, rather than choosing arbitrarily the two extremes. A curve of batch size vs speedup is valuable, as done in Fig 2.\n\n2. I found that Section 2.2.1 and 2.2.2, formal derivations of runtime and number of operations, were unnecessary. To me, these formalisms made the paper harder to understand and parse, rather than easier.\n\nThis could be improved by reducing these to key points: (a) encoders are parallelizable (b) T is a significant factor in NAR models, and is around 4-10 for good performance (c) AR decoders are more robust to using fewer layers than NAR decoders, etc.\n\n3. Figure 1 is confusing.\n\nI think this could be improved by converting it into a table with a \"NAR\" section and an \"AR\" section, and each row being a model.  \n\nRecommendation: Accept. Although I had some criticisms of the paper, they pertained mostly to clarity, style, and precise definition of chosen metrics; the key results of the paper are solid. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_lRJNs7Ocon", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes deep encoder and shallow decoder models for auto-regressive NMT. They compare rigorously to NAR models. They also study three factors: layer allocation, speed measurement and knowledge distillation. They include that with a 12E-D1 model they obtain significant speed-up and can outperform the standard 6-6 AR model and almost always beat the NAR model in terms of quality. They also show that NAR models need deep decoders because they need to handle reordering.\n\nReasons for score:\nI scored this paper a 9. I think this is an important paper which establishes very strong AR baselines for future NAR work in the field. They correctly point out the three issues with the comparisons that many NAR papers make. They conduct various meaningful ablation studies and validate their various hypothesis properly. They also show that certain factors like knowledge distillation need to be applied to both AR and NAR systems. Finally, they advocate for reporting both S_1 and S_max when comparing speed gains.\n\nCons:\n- One issue I had with the presentation of the results was the selection of different formats and language pairs for different experiments. For example, table 2, 3 and 4 report on different subsets of language-pairs. Same with the figures. This might raise questions of whether the authors are randomly subselecting or selecting favorable subsets. I would have liked to see all experiments done on all LPs.\n\n\nMinor comments: \n- Section 2.1: S_max - \"This is closer to practical scenarios where one wants to translate a large amount of text.\" - this is a very subjective statement and I would tone this down. \n- Section 2.2.2: \"Denote respectively by E and D the numbers of encoder and decoder layers.\" -- please fix grammar\n\nMissing citations:\n- Section 1: Along with Sutskever, Bahdanau and Vaswani. please also cite https://www.aclweb.org/anthology/D13-1176.pdf and Wu et al. 2016 (https://arxiv.org/abs/1609.08144) when you mention state-of-the-art NMT.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "O2dElHid61D", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors advocate for fair comparison between autoregressive (AR) and non-autoregressive models (NAR) in non-autoregressive machine translation (NAT) research. They highlight three main aspects where the comparison has not been fair so far in the literature - suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation. They perform extensive comparisons between AR and NAR models in these 3 aspects and report interesting results. \n\nPros:\n- The paper is very well written and easy to follow. Limitations of previous works have been well explained, and the motivation behind their work is clearly evident. The experiments are rigorous.\n- The paper shows that autoregressive models with deep encoders and shallow decoders are comparable and sometimes outperform current state of the art iterative semi-autoregressive models in terms of quality-latency tradeoff. I believe these findings can be valuable to the non-autoregressive MT and machine translation community in general.\n\nCons / Suggestions:\n\n- When doing the S_{max} speed comparisons, I am assuming that the maximum amount of tokens fitting in the GPU memory also depends on the length beam. I will suggest the authors to clarify this, and possibly include an experiment showing what speed-quality tradeoff iterative models achieve with different values of length beam. \n- In Section 2.3, the authors should point out that some previous works like Mask-Predict (Ghazvininejad et al., 2019) do report AR baselines with knowledge distillation. \n- Unless I inferred the results incorrectly, Mask-Predict finds almost no improvement in AR baselines when using distilled data, while in this paper, the authors find an improvement of around 1 BLEU point. Can the authors comment on why this could be the case? Can this  be because of variations in the distilled data used, or some difference in model training. I believe that 1 BLEU point is too big a difference to be accounted for as a noise signal. \n\nOverall, this paper proposes a different direction for research in fast and accurate machine translation, and points out the current shortcomings of research in non-autoregressive machine translation in details. I find findings by this paper very interesting and potentially valuable to the concerned research community, hence I recommend its acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CX5GaFPC29J", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\n\nIn this paper, the authors present a throughout study on the deep-shallow architecture for autoregresstive machine translation model (AR) and non-autoregressive model (NAR). Authors compare the two types of models from speed measures, layer allocation and the usage of knowledge distillation.\nMany results in this paper are interesting and meaningful:\n(1)\tDeep-shallow architecture (in this paper, 12-1) can achieve good BLEU performances across different tasks and speed up inference.\n(2)\tNAR is not as fast as AR when using S_{max} evaluation (i.e., the inference in batch model, where all GPU memory should be utilized).\n(3)\tKnowledge distillation is helpful for both AR and NAR. \n\nThe experiment results are solid and reproducible. \nThe paper is well written. \n\n** Significance **\nAlthough this is a well-written paper with solid results, I still have the following concerns:\n1.\tTo speed up inference, one can easy get to use a shallow decoder to reduce inference time, since the decoding process in AT takes most of the time. The results do not give me so much surprise. \n2.\tWhy the 12-1 architecture can achieve better results than 6-6 architecture on AR, but worse results on NAR? (see Figure 1 (B)(C)) What roles do encoder and decoder play in different modes (i.e., AR and NAR?) The following terms might be helpful to you:\na.\tThe comparison of their training/validation curves to verify whether the improvement is due to better training/generalization.\nb.\tIn Figure 1, the results are mainly obtained on 6-1, 12-1, 6-6. What if we try more combinations?\ni.\tFix the encoder layer as 6/12 and vary the decoder layers {1,2,4};\nii.\tFix the decoder layer as {1,2,4} and try more the encoder layers?\nIn this way, we can see how encoder/decoder layer effect the results (speed and quality).\n3.\tWhat are the differences between the output of 6-6 and 12-1? Any human analysis? I think you can briefly summarize the discovery in the paper with examples attached in the appendix. BLEU is important but you should evaluate your model from more perspectives. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work demonstrates that autoregressive (AR) models for machine translation can can be competitive with their non-autoregressive (NAR) counterparts in terms of practicality. This is a timely observation, given the flurry of recent work on NAR models, whose primary benefit is often cited to be fast inference.

It was argued that the results are not surprising -- if this is the case, I still think this work merits acceptance because its thesis runs counter to the direction the field as a whole seems to be moving in, and the results are convincing. That said, I agree with the authors that the observation that some encoder and decoder layers are interchangeable, is not self-evident (i.e. it _is_ surprising). This is of course subjective to some degree, so I am making a judgement call here. The work also has value in that it draws attention to some practices regarding evaluation in NAR machine translation literature that could be improved and made more fair (specifically regarding comparison with AR models).

There were some concerns about whether these models should be evaluated in the small-batch or large-batch setting. The authors have updated their manuscript in response, and it now explicitly discusses both settings. The authors have also run more experiments and added several additional results requested by reviewers to the manuscript.

All things considered, I am inclined to follow the majority and recommend acceptance.
Decision: Accept (Poster)