Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

1 INTRODUCTION . Fast , accurate machine translation is a fundamental goal with a wide range of applications both in research and production . State-of-the-art neural machine translation systems generate translations autoregressively where words are predicted one-by-one conditioned on all previous words ( Kalchbrenner & Blunsom , 2013 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ; Wu et al. , 2016 ; Vaswani et al. , 2017 ) . This sequential property limits parallelization , since multiple tokens in each sentence can not be generated in parallel . A flurry of recent work developed ways to ( partially ) parallelize the decoder with non-autoregressive machine translation ( NAR ; Gu et al. , 2018 ) , thereby speeding up decoding during inference . NAR tends to suffer in translation quality because parallel decoding assumes conditional independence between the output tokens and prevents the model from properly capturing the highly multimodal distribution of target translations ( Gu et al. , 2018 ) . Recent work proposed methods to mitigate this multimodality issue , including iterative refinement ( e.g. , Lee et al. , 2018 ; Ghazvininejad et al. , 2019 ) , and modeling with latent variables ( e.g. , Ma et al. , 2019 ; Shu et al. , 2020 ) . These approaches modify the decoder transformer to find a balance between decoding parallelism and translation quality . In this work , however , we adopt a different speed-quality tradeoff . Recent work by Kim et al . ( 2019 ) in autoregressive machine translation ( AR ) suggests that better speed-quality tradeoffs can be achieved by having different depths in the encoder and the decoder . Here , we make a formal argument in favor of deep encoder , shallow decoder configurations and empirically demonstrate better speed-quality tradeoffs for the AR baselines . ∗ Work partially done at Facebook AI . We provide extensive speed-quality comparisons between iterative NAR models and AR models with varying numbers of encoder and decoder layers . In particular , we use two types of speed measures for translation and discuss their relation to computational complexity . The two measures reflect two different application scenarios : feeding one sentence at a time , and feeding as many words as possible into the GPU memory . The first scenario is designed to simulate , for example , instantaneous machine translation that translates text ( or even speech ) input from users . This is where current NAR models shine—we can make full use of parallelism across decoding positions in a GPU . For this reason , much prior work in NAR only measures speed using this metric ( e.g. , Gu et al. , 2018 ; 2019b ; Kasai et al. , 2020 ; Li et al. , 2020 ) . The second scenario aims at a situation where we want to translate a large amount of text as quickly as possible . In this case , we see that AR models run faster than NAR models by a large margin . Computation at each time step is large enough to exploit parallelism in a GPU , which cancels out the benefit from parallel NAR decoding . Further , AR models can cache all previous hidden states ( Ott et al. , 2019 ) and compute each step in linear time complexity with respect to the sequence length . In contrast , NAR models necessitate a fresh run of quadratic self and cross attention in every decoding iteration . Interestingly , using a deep encoder and a shallow decoder in NAR models fails to retain the original translation accuracy by using 6 layers each ( §4.1 ) . This suggests that departure from AR decoding necessitates more capacity in the decoder ; the strategy is effective specifically for AR models . In particular , our analysis demonstrates that an NAR decoder requires more layers to learn target word ordering ( §5 ) . In summary , our contributions are the following : • We challenge three conventional assumptions in NAR evaluation : suboptimal layer allocation , lack of distillation for AR baselines , and insufficiently general speed measures . • We provide a complexity analysis and identify an optimal layer allocation strategy that leads to better speed-quality tradeoffs , namely a deep-shallow configuration . • We perform extensive analyses and head-to-head comparisons of AR and strong NAR models on seven standard translation directions . We demonstrate that the accuracy gap between the two model families is much wider than previously thought and that NAR models are unable to capture target word order well without sufficiently deep decoders . 2 REEVALUATING NON-AUTOREGRESSIVE MACHINE TRANSLATION . We critically examine in this section the evaluation practices and assumptions that are widely held in the non-autoregressive neural machine translation ( NAR ) literature ( e.g. , Gu et al. , 2018 ; Ghazvininejad et al. , 2019 ; Kasai et al. , 2020 ) . In particular , we focus on three aspects : speed measurement ( §2.1 ) , layer allocation ( §2.2 ) , and knowledge distillation ( §2.3 ) . 2.1 SPEED MEASURES . One major benefit of NAR models over AR ones is their ability to generate text in parallel . Current research on measuring speed has focused solely on the setting of translating one sentence at a time where full parallelization is trivial with a single GPU . However , we argue that this speed measure is not realistic in some scenarios because the GPU memory is finite and the GPU unit in such a setting is underused . To address this issue , we use two translation speed metrics to measure inference speed : • S1 measures speed when translating one sentence at a time . This metric is used in standard practice and aligns with applications like instantaneous machine translation that translates text input from users immediately . • Smax measures speed when translating in mini-batches as large as the hardware allows . This corresponds to scenarios where one wants to translate a large amount of text given in advance . For instance , such large-batch machine translation is implemented in the Google cloud service.1 For all models , both metrics measure wall-clock time from when the weights are loaded until the last sentence is translated . We report speedups relative to an AR baseline with a 6-layer encoder and a 6-layer decoder following prior work ( Gu et al. , 2018 ; Li et al. , 2020 ; Kasai et al. , 2020 ) . 1https : //cloud.google.com/translate/docs/advanced/batch-translation . 2.2 LAYER ALLOCATION . Current evaluation practice in the NAR literature uses an equal number of layers for the encoder and decoder both in AR baselines and NAR models . However , previous studies in AR machine translation suggest that this allocation strategy leads to a suboptimal speed-quality tradeoff ( Barone et al. , 2017 ; Kim et al. , 2019 ) . These findings have several implications for evaluating NAR methods . We first discuss the strategy of deep encoder , shallow decoder ( §2.2.1 ) , and provide a theoretical analysis of the speed-quality tradeoff in the context of NAR evaluation ( §2.2.2 ) . Our analyses are verified empirically in the next section ( §3 ) . 2.2.1 DEEP ENCODER , SHALLOW DECODER . In line with prior work on deep encoders or shallow decoders ( Barone et al. , 2017 ; Wang et al. , 2019a ; Kim et al. , 2019 ) , we depart from the convention to allocate an equal number of layers on both sides and explore pairing a deep encoder with a shallow decoder for both AR and NAR methods . Here , we study the impact of such architectures and systematically compare AR and NAR methods.2 As we will show in §3 , an AR model with a deep-shallow configuration retains translation accuracy , but can substantially reduce decoding time . This is because at inference time , the encoder accounts for a smaller part of the overhead since its computation can be easily parallelized over source positions ; on the other hand , the speedup gains from a lightweight decoder are substantial . By Layer Full Model Enc . AR Dec. NAR Dec. AR E-D AR E-1 NAR E-D Total Operations O ( N2 ) O ( N2 ) O ( TN2 ) O ( EN2 +DN2 ) O ( EN2 + 1 ⋅N2 ) O ( EN2 +DTN2 ) Time Complex . O ( N ) O ( N2 ) O ( TN ) O ( EN +DN2 ) O ( EN +N2 ) O ( EN +DTN ) Table 1 : Analysis of transformers . Time complex . indicates time complexity when full parallelization is assumed . N : source/target length ; E : encoder depth ; D : decoder depth ; T : # NAR iterations . 2.2.2 COMPLEXITY ANALYSIS . This section analyzes the complexities of transformer-based encoders , autoregressive and nonautoregressive decoders . We focus on two key properties : ( 1 ) the total amount of operations and ( 2 ) time complexity when full parallelization is assumed ( Harris , 2007 ) . Notation For simplicity let us assume the source and target text have the same length N . T is the number of iterations in an iterative NAR method ( typically T < N ) . Let E and D denote the numbers of encoder and decoder layers . Table 1 breaks down the comparison . AR and NAR models use the same encoder architecture . There are several interesting distinctions between AR and NAR decoders . First , although their total amounts of operations are both quadratic in sequence length , an NAR decoder with T decoding iterations needs T times more computation . Second , an AR decoder has time complexity quadratic in sequence length . This contrasts with the linear time complexity of an NAR decoder , which is the powerhouse of its S1 speedup ( §4.1 ) . This is because the attention computation can be readily parallelized across target positions in NAR decoders . By such comparisons we make the following key observations : ( a ) For both AR and NAR models , the time complexity is dominated by decoders . When T < N , an NAR model has an advantage over its AR counterpart with the same layers . ( b ) Innocuous as it may seem , the constant T contributes major computational cost in terms of the total operations of NAR models . Empirically , T needs to be at least 4 to perform competitively to AR models ( Ghazvininejad et al. , 2019 ; Kasai et al. , 2020 ) . ( a ) suggests that one can significantly speed up S1 decoding by using shallower decoders , while increasing the encoder depth only results in a mild slowdown . As we will show later in the experiments , AR decoders are much more robust to using fewer layers than NAR decoders . For example , 2Note that Kim et al . ( 2019 ) proposed other methods to optimize CPU decoding of AR models , but we do not apply them , to ensure fair comparisons between AR and NAR models . AR E-1 can decode much faster than AR E-D and comparably to NAR E-D , while retaining the accuracy of AR E-D. From ( b ) , one may expect a different trend in Smax from S1 : in large minibatch decoding , an AR model can make use of the GPU ’ s compute units , since now parallelization happens over the instances in a mini-batch . In other words , under the Smax evaluation where the GPU is running close to its maximum flop/s , NAR can actually be slower since it needs more operations due to its iterative decoding . This is confirmed by our experiments ( §4.1 ) .

Summary: Traditional NTM is done with a large encoder and large autoregressive (AR) decoder. Due to the sequential nature of the AR decoder, inference can be slow due to lack of parallelism (unless done at very large batch sizes). Non-Autoregressive (NAR) models have been proposed to alleviate this problem, but all NAR approaches trade off some translation quality for speed gains. In this paper, the authors claim that an alternative to NAR is to speed up standard AR decoding by reallocating network weights and layers to the (easily parallelizable) encoder, and making the decoder a single layer. They claim that this matches the speed of NAR models while keeping the performance of traditional AR models, making it a better choice in the design space than any NAR models. Comparisons are made to CMLM and DisCo NAR models to justify these claims with experimental evidence.