Hybrid-Regressive Neural Machine Translation

1 INTRODUCTION . Although autoregressive translation ( AT ) has become the de facto standard for Neural Machine Translation ( Bahdanau et al. , 2015 ) , its nature of generating target sentences sequentially ( e.g. , from left to right ) makes it challenging to respond quickly in a production environment . One straightforward solution is the non-autoregressive translation ( NAT ) ( Gu et al. , 2017 ) , which predicts the entire target sequence in one shot . However , such one-pass NAT models lack dependencies between target words and still struggles to produce smooth translations , despite many efforts developed ( Ma et al. , 2019 ; Guo et al. , 2019a ; Wang et al. , 2019b ; Shao et al. , 2019 ; Sun et al. , 2019 ) . Recent studies show that extending one-pass NAT to multi-pass NAT , so-called iterative refinement ( IR-NAT ) , is expected to break the performance bottleneck ( Lee et al. , 2018 ; Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Guo et al. , 2020 ; Kasai et al. , 2020a ) . Unlike onepass NAT , which outputs the prediction immediately , IR-NAT takes the translation hypothesis from the previous iteration as a reference and regularly polishes the new translation until achieving the predefined iteration count I or no changes appear in the translation . Compared with AT , IR-NAT with I=10 runs 2-5 times faster with a considerable translation accuracy , as reported by Guo et al . ( 2020 ) . However , we highlight that the fast decoding of IR-NAT heavily relies on small batch size and GPU , which is rarely mentioned in prior studies 1 . Without loss of generality , we take Mask-Predict ( MP ) 1Unfortunately , such a decoding setting is not common in practice . NMT systems deployed on GPUs tend to use larger batches to increase translation throughput , while the batch size of 1 is used more frequently in offline systems running on CPUs . e.g. , smartphones . ( Ghazvininejad et al. , 2019 ) as an example , a typical IR-NAT paradigm based on the conditional masked language model . Figure 1 illustrates that when the batch exceeds 8 , MP ( I=10 ) is already running slower than AT , and the situation is even worse on CPU . Further analysis shows that the increase in batch size leads to the efficiency degradation of parallel computing in NAT models 2 . To tackle this problem , we first design a synthetic experiment to understand the relationship between target context and iteration times . We mask some proportion tokens on the translation generated by a pretrained AT and take it as the decoder input of the pretrained MP . Then we surprisingly found that even masking 70 % AT hypothesis , and the remaining target context can help MP ( I=1 ) to compete with the standard MP ( I=10 ) ( Figure 2 ) . This result confirms that decoding with multiple iterations in NAT is unnecessary when providing a good ( partial ) reference hypothesis . Inspired by this , we propose a two-stage translation prototype——Hybrid-Regressive Translation ( HRT ) . After encoding , HRT first uses an autoregressive decoder ( called Skip-AT ) to produce a discontinuous translation hypothesis . Concretely , at decoding step i , the SKip-AT decoder immediately predicts the ( i + k ) -th token yi+k without generating yi+1 , . . . , yi+k−1 , where k is a hyperparameter and k > 1 . Then , a non-autoregressive decoder like MP ( called Skip-MP ) predicts previously skipped tokens with one iteration according to the deterministic context provided by Skip-AT . Since both Skip-AT and Skip-MP share the same model parameters , HRT does not increase parameters significantly . To train HRT effectively and efficiently , we further propose joint training guided by curriculum learning and mixed distillation . Experimental results on WMT En↔Ro and En↔De show that HRT is far superior to existing IR-NATs and achieves comparable or even better accuracy than the original AT 3 with a consistent 50 % decoding speedup on varying batch sizes and devices ( GPU , CPU ) . 2 BACKGROUND . Given a source sentence x = { x1 , x2 , . . . , xM } and a target sentence y = { y1 , y2 , . . . , yN } , there are several ways to model P ( y|x ) : Autoregressive translation ( AT ) is the dominant approach in NMT , which decomposes P ( y|x ) by chain rules : P ( y|x ) = N∏ t=1 P ( yt|x , y < t ) ( 1 ) where y < t denotes the generated prefix translation before time step t. However , the existence of y < t requires the model must wait for yt−1 to be produced before predicting yt , which hinders the possibility of parallel computation along with time step . Non-autoregressive translation ( NAT ) is first proposed by Gu et al . ( 2017 ) , allowing the model to generate all target tokens simultaneously . NAT replaces y < t with target-independent input z and rewrites Eq . 1 as : P ( y|x ) = P ( N |x ) N∏ t=1 P ( yt|x , z ) ( 2 ) In Gu et al . ( 2017 ) , they monotonically copy the source embedding as z according to a fertility model . Subsequently , the researchers developed more advanced methods to enhance z , such as adversarial source embedding ( Guo et al. , 2019a ) , reordered source sentence ( Ran et al. , 2019 ) , latent variables ( Ma et al. , 2019 ; Shu et al. , 2019 ) etc , but there still is a huge performance gap between AT and NAT . Iterative refinement based non-autoregressive translation ( IR-NAT ) extends the traditional onepass NAT by introducing the multi-pass decoding mechanism ( Lee et al. , 2018 ; Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Guo et al. , 2020 ; Kasai et al. , 2020a ) . IR-NAT applies a conversion function 2Early experiment shows that when the batch size increases from 1 to 32 , the latency of AT is reduced by 22 times , while MP ( I=10 ) only reduces by four times . Latency is measured by the average time of translating a sentence on a constant test set . See Appendix A for details . 3Thanks to the proposed training algorithm , a single HRT model can support both hybrid-regressive decoding and autoregressive decoding at inference . Here , the AT model refers to the autoregressive teacher model that generates the distillation data . F on the deterministic hypothesis of previous iteration y′ as the alternative to z . Common implementations of F include identity ( Lee et al. , 2018 ) , random masking ( Ghazvininejad et al. , 2019 ) or random deletion ( Gu et al. , 2019 ) etc . Thus , we can predict y by : P ( y|x ) = N ′∏ t=1 P ( y′m ( t ) |x , F ( y′ ) ) ( 3 ) where N ′ is the number of refined tokens in F ( y′ ) , m ( t ) is the real position of t-th refined token in y′ . In this way , the generation process of IR-NAT is simple : first , the NAT model produces an inaccurate translation as the initial hypothesis , and then iteratively refines it until converge or reaching the maximum number of iterations . Mask-Predict ( MP ) is a typical instance of IR-NAT , trained by a conditional masked language model objective like BERT ( Devlin et al. , 2019 ) . In this work , we use MP as the representation of IR-NAT due to its excellent performance and simplification . In MP , F randomly masks some tokens over the sequence in training but selects those predicted tokens with low confidences at inference . 3 IS ITERATIVE REFINEMENT ALL YOU NEED ? . As mentioned earlier , IR-NAT with multiple iterations slows down severely in some cases . It is natural to think of reducing iterations to alleviate it . This section starts from synthetic experiments on WMT ’ 16 En→Ro and WMT ’ 14 En→De to verify the assumption that a sufficiently good decoder input can help reduce iterations . Here we construct the “ good ” decoder input from the translation hypothesis produced by an AT model . Models We use the official MP models released by Ghazvininejad et al . ( 2019 ) 4 . Since the authors did not publish their AT baselines , we use the same data to retrain AT models with the standard Transformer-Base configuration ( Vaswani et al. , 2017 ) and obtain comparable performance with theirs ( see Appendix B for more details ) . Decoding AT models decode with beam sizes of 5 on both tasks . Then , we replace a certain percentage of AT translation tokens with < mask > and use it as input to the MP model ( see below for replacement strategy ) . Unlike the standard MP model that uses a large beam size ( e.g. , 5 ) and iterates several times ( e.g. , 10 ) , the MP model used here only iterates once with beam size 1 . We substitute all input < mask > with MP ’ s predictions to obtain the final translation . We report casesensitive tokenized BLEU score by multi-bleu.perl . Mask Strategy We tested 4 strategies to mask AT translations : Head , Tail , Random and Chunk . Given the masking rate pmask and the translation length N , the number of masked tokens is Nmask=max ( 1 , bN×pmaskc ) . Then Head/Tail always masks the first/last Nmask tokens , while Random masks the translation randomly . Chunk is slightly different from the above strategies . It first divides the target sentence into C chunks , where C = Ceil ( N/k ) and k is the chunk size . Then in each chunk , we retain the first token , but mask other k-1 tokens . Thus , the actual masking rate in Chunk is 1-1/k instead of pmask . To exclude randomness , we ran Random three times with different seeds and report the average results . 3.1 RESULTS . The experimental results are illustrated in Figure 2 , where we can see that : A balanced bidirectional context is critical . Compared with Tail and Head , it is obvious that Rand and Chunk both have better performance . We attribute it to the benefit of the bidirectional context in Rand and Chunk ( Devlin et al. , 2019 ) , because Tail and Head can only provide unidirectional context ( i.e. , prefix or suffix ) . In addition , compare Chunk with Random , we find that Chunk is moderately but consistently superior to Random , even if more tokens are masked . For instance , on the WMT En-De task , when the chunk size is 4 ( the masking rate is 75 % ) , the BLEU score of Chunk is 27.03 , which is +0.3 BLEU higher than that of Random with the masking rate of 4https : //github.com/facebookresearch/Mask-Predict 70 % . Because the difference between Chunk and Random lies only in the distribution of < mask > , this experiment indicates that making < mask > uniformly on sequence is better than random 5 . Small beams and one iteration are sufficient . Compared with the standard MP with the beam size of 5 and 10 iterations , it is interesting to find that even if only 30 % -40 % of the AT translations are exposed , our MP using greedy search and one iteration can achieve quite comparable performance .

Summary: This paper proposes a hybrid-regressive machine translation (HRT) approach—combining autoregressive (AT) and non-autoregressive (NAT) translation paradigms: it first uses an AT model to generate a “gappy” sketch (every other token in a sentence), and then applies a NAT model to fill in the gaps with a single pass. As a result the AT part latency is roughly reduced by half compared to a full AT baseline. The AT and NAT models share a majority part of the parameters and can be trained jointly with a carefully designed curriculum learning procedure. Experiments on several MT benchmarks show that the proposed approach achieves speedup over the full AT baseline with comparable translation quality.