Non-Autoregressive Models are Better Multilingual Translators

1 INTRODUCTION . Neural machine translation ( NMT ) is the current state-of-the-art approach ( Bahdanau et al. , 2014 ; Sutskever et al. , 2014 ) for machine translation in both academia ( Bojar et al. , 2017 ) and industry ( Hassan et al. , 2018 ) . Recent works ( Firat et al. , 2016 ; Johnson et al. , 2017 ; Aharoni et al. , 2019 ; Lin et al. , 2020 ) extend the approach to support multilingual translation , i.e . training a single model that can translate across multiple language directions . Multilingual models are appealing for several reasons . First , they can reduce the online translation service number , enabling simpler deployment ( Arivazhagan et al. , 2019 ) when plenty of translation directions are required . Additionally , multilingual training makes it possible to transfer knowledge from high-resource languages to low-resource ones , thus improving the translation quality of low-resource directions ( Zoph et al. , 2016 ; Johnson et al. , 2017 ; Wang & Neubig , 2019 ) . However , most multilingual NMT systems are built upon the autoregressive architecture , which translates from left to right and thus is not efficient enough in terms of translation speed . Such efficiency problem is more serious in multilingual setting because all translation directions suffer from this slow inference speed . A straightforward solution to improve the multilingual translation efficiency is to develop multilingual non-autoregressive translation ( NAT ) . NAT generates translation outputs in parallel ( Gu et al. , 2018 ) , which leads to significantly faster translation speed . Thanks to the recent progress of NAT ( Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Deng & Rush , 2020 ) , current state-of-the-art NAT models have achieved comparable BLEU scores ( Li et al. , 2018 ; Wei et al. , 2019 ; Qian et al. , 2020 ) with their auto-regressive counterparts . Among them , the glancing transformer ( GLAT ) proposed by Qian et al . ( 2020 ) is a representative work , which even outperforms many strong autoregressive translation systems in BLEU score on German-English translation task of WMT21 ( Qian et al. , 2021 ) . In this paper , we argue that multilingual NAT models is not only superior in efficiency , but also can achieve better multilingual translation accuracy , due to its capability of generating high quality codeswitched translations ( Lin et al. , 2020 ; Yang et al. , 2020 ; Jose et al. , 2020 ) . In particular , we propose switch-GLAT , a carefully designed multilingual version of GLAT , which can outperform multilingual Transformer in both speed and translation quality . Generally , the main idea of switch-GLAT is to employ a code-switch decoder , which can generate contextual code-switched translations ( instead of using dictionary for replacement ) for a given source sentence , and then perform code-switch back-translation to boost the multilingual translation performance . In more details , the glancing sampling module in GLAT enables switch-GLAT to generate partial translation based on others . Then combined with its multilingual and non-autoregressive characteristics , switch-GLAT could output Encoder H ParallelDecoder Glancing Compute Distance H ' ParallelDecoder Sample words Replace Input H Compute Loss code-switched translations with the employment of token-level language tags instead of sentencelevel ones in auto-regressive models . Ultimately , switch-GLAT can generate contextual translated words at arbitrary positions of the target sentence in arbitrary languages . This greatly improves the multilingual translation performance when we reverse the pairs of source to contextual code-switched target sentences , for training in a back-translation fashion ( so called code-switch back-translation ) . We conduct extensive experiments on 3 merged translation datasets : WMT with four language pairs ( both close languages and distant ones ) and WMT with 10 language pairs . switch-GLAT shows consistent improvements over autoregressive multilingual baselines on all datasets , validating that switch-GLAT can achieve better multilingual translation performance simultaneously with a faster decoding speed . We further evaluated the cross-lingual representations through word induction and sentence retrieval tasks . The results demonstrated the proposed code-switch back-translation benefits better-aligned cross-lingual representations . 2 BACKGROUND . Multilingual Neural Machine Translation ( MNMT ) Given a source sentence X = { x1 , x2 , ... , xM } with length M and its target sentence Y = { y1 , y2 , ... , yN } with length N , MNMT leverages the standard bilingual neural machine translation models and extends the source and target inputs respectively with a source and target language token src and tgt . This results in X ′ = { src , x1 , x2 , ... , xM } and Y ′ = { tgt , y1 , y2 , ... , yN } . MNMT is generally modeled from X ′ to Y ′ with Transformer ( Vaswani et al. , 2017 ) . Transformer consists of stacked encoder and decoder layers , which are jointly trained to maximize the conditional probability of Y ′ given X ′ : P ( Y ′|X ′ ) = N∑ i=1 logP ( yi|y < i , X ′ , tgt ; θ ) ( 1 ) where θ are the trainable model parameters . Glancing Transformer prposed by Qian et al . ( 2020 ) is a NAT architecture which achieves top results in machine translation with 8x ∼ 15x speedup . It performs two-pass decoding in training but is still fully auto-regressive in inference . In the first decoding pass , given the encoder Fe and decoder Fd , H0d = { h01 , h02 , ... , h0N } is the decoder input either gathered from the encoder output using soft copy ( Wei et al. , 2019 ) or full mask ( Ghazvininejad et al. , 2019 ) , and then Y is predicted as : Ŷ = Fd ( H 0 d , Fe ( X ; θ ) ; θ ) ( 2 ) where θ are the trainable model parameters . Then glancing transformer ( GLAT ) adopts glancing sampling strategy to sample a subset of Y according to its distance with Ŷ , thus resulting inGS ( Y , Ŷ ) . Finally , GLAT predicts the target sequence Y based on this subset and source sentence X in the second decoding pass as follows : LGLAT = − ∑ yt∈GS ( Y , Ŷ ) logP ( yt|GS ( Y , Ŷ ) , X ; θ ) ( 3 ) where H0d is updated accordingly and GS ( Y , Ŷ ) denotes the remaining subset of Y after removing the sampled tokens . The overall architecture of GLAT is shown in Figure 1 . The twice decoding in training makes GLAT capable of predicting some partial translations based on others . Then combined with multilingual and non-autoregressive characteristics , GLAT has the potential to generate contextual code-switched outputs . In the next section , we will show how to modify the GLAT architecture to achieve this . 3 PROPOSED METHOD : switch-GLAT This section will detail the proposed switch-GLAT . The training objective can be factorized into two parts . One aims to make the model have multilingual translation ability , and the other to achieve better-aligned cross-lingual representations to boost translation performance , respectively denoted as Lmulti and Lcsbt . The overall objective can be formulated as follows : L = Lmulti + λ · Lcsbt ( 4 ) where λ plays the role of a “ temperature ” to schedule the importance of code-switch back-translation . As the training progresses , λ is gradually increased , which allows for more complex data instances to be involved , i.e . code-switched translations incorporating more languages and higher switching ratios . This process encourages model to align similar words from different languages into the same vector space , thus boosting machine translation performance . The overall architecture is shown in Figure 2 . 3.1 CODE-SWITCH DECODER . switch-GLAT leverages the general parallel decoder of GLAT and extends it into a code-switch decoder by employing the token-level language tag . The code-switch decoder first gains multilingual translation ability through multilingual training , and then it can generate code-switched translations in arbitrary languages with the help of token-level language tag . Specifically , given a multilingual corpora D = { Dl } Ll=1 consisting of L language pairs , the loss Lmulti is then defined as : Lmulti = ∑ Dl∈D ∑ ( Xlj , Y l j ) ∈D l { Ltag ( Y lj |Xlj ; θM ) + Lllen ( j ) } Lllen ( j ) = −P ( Llj ) log P̂ ( Llj | [ Fe ( Xlj ; θM ) ; Esrc ; Etgt ] ; θM ) ( 5 ) where Dl = { ( X lj , Y lj ) } Nl j=1 is a parallel corpus with size Nl and L l len ( j ) is the length prediction loss of the j-th pair . P ( Llj ) is the real length distribution of target sentence , and P̂ ( L l j ) is the predicted one based on the concatenation of encoder output as well as source and target language embeddings . θM are trainable model parameters . Ltag is the accordingly updated GLAT training loss ( Equation 3 ) incorporating the source and target language tags : Ltag ( Y l j |Xlj ; θM ) = − ∑ yt∈GS ( Y lj , Ŷ l j ) logP ( yt|GS ( Y lj , Ŷ lj ) , Xlj , src , tgt ; θM ) Ŷ lj = Fd ( H̃ 0 d , Fe ( X l j , src ; θ ) , tgt ; θM ) ( 6 ) To involve the indicative language tag , we add it to the first layer input and final layer output at each position of both encoder and decoder as follows : f̃0i = f 0 i + Esrc ; f̃ K i = f K i + Esrc h̃0j = h 0 j + Etgt ; h̃ K j = h K j + Etgt ( 7 ) where f0i denotes the first encoder layer input at position i and h 0 j denotes the first decoder layer input at position j . Correspondingly , fKi and h K j denote the last layer output . src and tgt are respectively the source and target individual tags , while Esrc and Etgt are their corresponding representations . The overall prediction process of pair ( X lj , Y l j ) is illustrated in the left module of Figure 2 . Through the multilingual training process , switch-GLAT can translate between different languages using the indicative language tags , of which the decoder is called code-switch decoder . It has the ability to generate contextual translated words in arbitrary languages due to its token-level characteristics . 3.2 CODE-SWITCH BACK-TRANSLATION . Thanks to the code-switch decoder , we can perform code-switch back-translation ( CSBT ) , which is critical because it encourages model to align the produced words and the original ones into the same vector space according to their similar context information . Better-aligned cross-lingual representations benefit better translation performance . Specifically , a subset DS = { ( Xi , Yi , lsrci , l tgt i ) } Si=1 of size S is first sampled , where lsrci and l tgt i are respectively source and target languages of i-th pair . Then , Yi is masked with a given rate PM , leading to Ỹi . Subsequently , the masked positions of Ỹi can be decoded into a third randomly sampled language by leveraging the token-level language tag . Thus , the final decoded sequence Ŷi consists of contextual tokens from mixed languages , which will in turn be taken as the source side input and the original source sentence as the target side input . This process results in a code-switch back-translation corpus DC = { ( Ŷi , Xi ) } Si=1 , which is illustrated in the middle module of Figure 2 . The dynamically generated code-switch translations can thus augment data instance distribution to enhance model training as illustrated in the right module of Figure 2 . Loss Lcsbt is defined as : Lcsbt = ∑ ( Ŷi , Xi ) ∈DC Ltag ( Xi|Ŷi ; θM ) ( 8 ) As the training continues , the masked rate PM and number of mixed languages are gradually increased . Specifically , the value of PM is iterated from 0.1 to 0.5 with step size 0.1 every 10 epochs . In the first iteration of PM , the number of mixed languages is set to 1 . Afterwards , it will be increased to one-third of the total . Through this process , abundant code-switched sentences can be generated , which helps to learn better-aligned cross-lingual representations .

Summary: This paper extends non-autoregressive NMT models, particularly the GLAT model, from bilingual translation to multilingual translation. To enable such extension, the authors proposed token-level language tags for the decoder, code-switch decoder paired with back-translation, and also scheduled training between the traditional MLE loss and the BT loss. Experiments on several multilingual tasks based on WMT corpus show that the proposed model obtains better supervised translation quality than its auto-regressive counterparts and delivers better cross-lingual representations.