Non-Autoregressive Models are Better Multilingual Translators

1 INTRODUCTION . Neural machine translation ( NMT ) is the current state-of-the-art approach ( Bahdanau et al. , 2014 ; Sutskever et al. , 2014 ) for machine translation in both academia ( Bojar et al. , 2017 ) and industry ( Hassan et al. , 2018 ) . Recent works ( Firat et al. , 2016 ; Johnson et al. , 2017 ; Aharoni et al. , 2019 ; Lin et al. , 2020 ) extend the approach to support multilingual translation , i.e . training a single model that can translate across multiple language directions . Multilingual models are appealing for several reasons . First , they can reduce the online translation service number , enabling simpler deployment ( Arivazhagan et al. , 2019 ) when plenty of translation directions are required . Additionally , multilingual training makes it possible to transfer knowledge from high-resource languages to low-resource ones , thus improving the translation quality of low-resource directions ( Zoph et al. , 2016 ; Johnson et al. , 2017 ; Wang & Neubig , 2019 ) . However , most multilingual NMT systems are built upon the autoregressive architecture , which translates from left to right and thus is not efficient enough in terms of translation speed . Such efficiency problem is more serious in multilingual setting because all translation directions suffer from this slow inference speed . A straightforward solution to improve the multilingual translation efficiency is to develop multilingual non-autoregressive translation ( NAT ) . NAT generates translation outputs in parallel ( Gu et al. , 2018 ) , which leads to significantly faster translation speed . Thanks to the recent progress of NAT ( Ghazvininejad et al. , 2019 ; Gu et al. , 2019 ; Deng & Rush , 2020 ) , current state-of-the-art NAT models have achieved comparable BLEU scores ( Li et al. , 2018 ; Wei et al. , 2019 ; Qian et al. , 2020 ) with their auto-regressive counterparts . Among them , the glancing transformer ( GLAT ) proposed by Qian et al . ( 2020 ) is a representative work , which even outperforms many strong autoregressive translation systems in BLEU score on German-English translation task of WMT21 ( Qian et al. , 2021 ) . In this paper , we argue that multilingual NAT models is not only superior in efficiency , but also can achieve better multilingual translation accuracy , due to its capability of generating high quality codeswitched translations ( Lin et al. , 2020 ; Yang et al. , 2020 ; Jose et al. , 2020 ) . In particular , we propose switch-GLAT , a carefully designed multilingual version of GLAT , which can outperform multilingual Transformer in both speed and translation quality . Generally , the main idea of switch-GLAT is to employ a code-switch decoder , which can generate contextual code-switched translations ( instead of using dictionary for replacement ) for a given source sentence , and then perform code-switch back-translation to boost the multilingual translation performance . In more details , the glancing sampling module in GLAT enables switch-GLAT to generate partial translation based on others . Then combined with its multilingual and non-autoregressive characteristics , switch-GLAT could output Encoder H ParallelDecoder Glancing Compute Distance H ' ParallelDecoder Sample words Replace Input H Compute Loss code-switched translations with the employment of token-level language tags instead of sentencelevel ones in auto-regressive models . Ultimately , switch-GLAT can generate contextual translated words at arbitrary positions of the target sentence in arbitrary languages . This greatly improves the multilingual translation performance when we reverse the pairs of source to contextual code-switched target sentences , for training in a back-translation fashion ( so called code-switch back-translation ) . We conduct extensive experiments on 3 merged translation datasets : WMT with four language pairs ( both close languages and distant ones ) and WMT with 10 language pairs . switch-GLAT shows consistent improvements over autoregressive multilingual baselines on all datasets , validating that switch-GLAT can achieve better multilingual translation performance simultaneously with a faster decoding speed . We further evaluated the cross-lingual representations through word induction and sentence retrieval tasks . The results demonstrated the proposed code-switch back-translation benefits better-aligned cross-lingual representations . 2 BACKGROUND . Multilingual Neural Machine Translation ( MNMT ) Given a source sentence X = { x1 , x2 , ... , xM } with length M and its target sentence Y = { y1 , y2 , ... , yN } with length N , MNMT leverages the standard bilingual neural machine translation models and extends the source and target inputs respectively with a source and target language token src and tgt . This results in X ′ = { src , x1 , x2 , ... , xM } and Y ′ = { tgt , y1 , y2 , ... , yN } . MNMT is generally modeled from X ′ to Y ′ with Transformer ( Vaswani et al. , 2017 ) . Transformer consists of stacked encoder and decoder layers , which are jointly trained to maximize the conditional probability of Y ′ given X ′ : P ( Y ′|X ′ ) = N∑ i=1 logP ( yi|y < i , X ′ , tgt ; θ ) ( 1 ) where θ are the trainable model parameters . Glancing Transformer prposed by Qian et al . ( 2020 ) is a NAT architecture which achieves top results in machine translation with 8x ∼ 15x speedup . It performs two-pass decoding in training but is still fully auto-regressive in inference . In the first decoding pass , given the encoder Fe and decoder Fd , H0d = { h01 , h02 , ... , h0N } is the decoder input either gathered from the encoder output using soft copy ( Wei et al. , 2019 ) or full mask ( Ghazvininejad et al. , 2019 ) , and then Y is predicted as : Ŷ = Fd ( H 0 d , Fe ( X ; θ ) ; θ ) ( 2 ) where θ are the trainable model parameters . Then glancing transformer ( GLAT ) adopts glancing sampling strategy to sample a subset of Y according to its distance with Ŷ , thus resulting inGS ( Y , Ŷ ) . Finally , GLAT predicts the target sequence Y based on this subset and source sentence X in the second decoding pass as follows : LGLAT = − ∑ yt∈GS ( Y , Ŷ ) logP ( yt|GS ( Y , Ŷ ) , X ; θ ) ( 3 ) where H0d is updated accordingly and GS ( Y , Ŷ ) denotes the remaining subset of Y after removing the sampled tokens . The overall architecture of GLAT is shown in Figure 1 . The twice decoding in training makes GLAT capable of predicting some partial translations based on others . Then combined with multilingual and non-autoregressive characteristics , GLAT has the potential to generate contextual code-switched outputs . In the next section , we will show how to modify the GLAT architecture to achieve this . 3 PROPOSED METHOD : switch-GLAT This section will detail the proposed switch-GLAT . The training objective can be factorized into two parts . One aims to make the model have multilingual translation ability , and the other to achieve better-aligned cross-lingual representations to boost translation performance , respectively denoted as Lmulti and Lcsbt . The overall objective can be formulated as follows : L = Lmulti + λ · Lcsbt ( 4 ) where λ plays the role of a “ temperature ” to schedule the importance of code-switch back-translation . As the training progresses , λ is gradually increased , which allows for more complex data instances to be involved , i.e . code-switched translations incorporating more languages and higher switching ratios . This process encourages model to align similar words from different languages into the same vector space , thus boosting machine translation performance . The overall architecture is shown in Figure 2 . 3.1 CODE-SWITCH DECODER . switch-GLAT leverages the general parallel decoder of GLAT and extends it into a code-switch decoder by employing the token-level language tag . The code-switch decoder first gains multilingual translation ability through multilingual training , and then it can generate code-switched translations in arbitrary languages with the help of token-level language tag . Specifically , given a multilingual corpora D = { Dl } Ll=1 consisting of L language pairs , the loss Lmulti is then defined as : Lmulti = ∑ Dl∈D ∑ ( Xlj , Y l j ) ∈D l { Ltag ( Y lj |Xlj ; θM ) + Lllen ( j ) } Lllen ( j ) = −P ( Llj ) log P̂ ( Llj | [ Fe ( Xlj ; θM ) ; Esrc ; Etgt ] ; θM ) ( 5 ) where Dl = { ( X lj , Y lj ) } Nl j=1 is a parallel corpus with size Nl and L l len ( j ) is the length prediction loss of the j-th pair . P ( Llj ) is the real length distribution of target sentence , and P̂ ( L l j ) is the predicted one based on the concatenation of encoder output as well as source and target language embeddings . θM are trainable model parameters . Ltag is the accordingly updated GLAT training loss ( Equation 3 ) incorporating the source and target language tags : Ltag ( Y l j |Xlj ; θM ) = − ∑ yt∈GS ( Y lj , Ŷ l j ) logP ( yt|GS ( Y lj , Ŷ lj ) , Xlj , src , tgt ; θM ) Ŷ lj = Fd ( H̃ 0 d , Fe ( X l j , src ; θ ) , tgt ; θM ) ( 6 ) To involve the indicative language tag , we add it to the first layer input and final layer output at each position of both encoder and decoder as follows : f̃0i = f 0 i + Esrc ; f̃ K i = f K i + Esrc h̃0j = h 0 j + Etgt ; h̃ K j = h K j + Etgt ( 7 ) where f0i denotes the first encoder layer input at position i and h 0 j denotes the first decoder layer input at position j . Correspondingly , fKi and h K j denote the last layer output . src and tgt are respectively the source and target individual tags , while Esrc and Etgt are their corresponding representations . The overall prediction process of pair ( X lj , Y l j ) is illustrated in the left module of Figure 2 . Through the multilingual training process , switch-GLAT can translate between different languages using the indicative language tags , of which the decoder is called code-switch decoder . It has the ability to generate contextual translated words in arbitrary languages due to its token-level characteristics . 3.2 CODE-SWITCH BACK-TRANSLATION . Thanks to the code-switch decoder , we can perform code-switch back-translation ( CSBT ) , which is critical because it encourages model to align the produced words and the original ones into the same vector space according to their similar context information . Better-aligned cross-lingual representations benefit better translation performance . Specifically , a subset DS = { ( Xi , Yi , lsrci , l tgt i ) } Si=1 of size S is first sampled , where lsrci and l tgt i are respectively source and target languages of i-th pair . Then , Yi is masked with a given rate PM , leading to Ỹi . Subsequently , the masked positions of Ỹi can be decoded into a third randomly sampled language by leveraging the token-level language tag . Thus , the final decoded sequence Ŷi consists of contextual tokens from mixed languages , which will in turn be taken as the source side input and the original source sentence as the target side input . This process results in a code-switch back-translation corpus DC = { ( Ŷi , Xi ) } Si=1 , which is illustrated in the middle module of Figure 2 . The dynamically generated code-switch translations can thus augment data instance distribution to enhance model training as illustrated in the right module of Figure 2 . Loss Lcsbt is defined as : Lcsbt = ∑ ( Ŷi , Xi ) ∈DC Ltag ( Xi|Ŷi ; θM ) ( 8 ) As the training continues , the masked rate PM and number of mixed languages are gradually increased . Specifically , the value of PM is iterated from 0.1 to 0.5 with step size 0.1 every 10 epochs . In the first iteration of PM , the number of mixed languages is set to 1 . Afterwards , it will be increased to one-third of the total . Through this process , abundant code-switched sentences can be generated , which helps to learn better-aligned cross-lingual representations .

Summary: This paper describes experiments in training a non-autoregressive Transformer (glancing Transformer, or GLAT) in a multilingual MT setting. The use of a non-autoregressive model, plus token-level language tags, enables a code-switching strategy where a source sentence in a given language can generate a target sentence containing words from multiple specified languages. This is used to augment the multilingual training corpus with back-translated data consisting of code-switched synthetic source sentences paired with original monolingual target sentences. A multilingual GLAT model trained with this strategy is shown to outperform many baselines including monolingual GLAT and several autoregressive multilingual models. Ablation studies demonstrate that code-switched back-translation is essential for achieving good performance.