Year: 2021

Abstract:
Multilingual machine translation aims to develop a single model for multiple language directions. However, existing multilingual models based on Transformer are limited in terms of both translation performance and inference speed. In this paper, we propose switch-GLAT, a non-autoregressive multilingual machine translation model with a code-switch decoder. It can generate contextual codeswitched translations for a given source sentence, and perform code-switch backtranslation, greatly boosting multilingual translation performance. In addition, its inference is highly efficient thanks to its parallel decoder. Experiments show that our proposed switch-GLAT outperform the multilingual Transformer with as much as 1.16 BLEU improvement and 6.6x faster decoding speed in inference.

heading: 1 INTRODUCTION
text: Neural machine translation (NMT) is the current state-of-the-art approach (Bahdanau et al., 2014; Sutskever et al., 2014) for machine translation in both academia (Bojar et al., 2017) and industry (Hassan et al., 2018). Recent works (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Lin et al., 2020) extend the approach to support multilingual translation, i.e. training a single model that can translate across multiple language directions. Multilingual models are appealing for several reasons. First, they can reduce the online translation service number, enabling simpler deployment (Arivazhagan et al., 2019) when plenty of translation directions are required. Additionally, multilingual training makes it possible to transfer knowledge from high-resource languages to low-resource ones, thus improving the translation quality of low-resource directions (Zoph et al., 2016; Johnson et al., 2017; Wang & Neubig, 2019).
However, most multilingual NMT systems are built upon the autoregressive architecture, which translates from left to right and thus is not efficient enough in terms of translation speed. Such efficiency problem is more serious in multilingual setting because all translation directions suffer from this slow inference speed.
A straightforward solution to improve the multilingual translation efficiency is to develop multilingual non-autoregressive translation (NAT). NAT generates translation outputs in parallel (Gu et al., 2018), which leads to significantly faster translation speed. Thanks to the recent progress of NAT (Ghazvininejad et al., 2019; Gu et al., 2019; Deng & Rush, 2020), current state-of-the-art NAT models have achieved comparable BLEU scores (Li et al., 2018; Wei et al., 2019; Qian et al., 2020) with their auto-regressive counterparts. Among them, the glancing transformer (GLAT) proposed by Qian et al. (2020) is a representative work, which even outperforms many strong autoregressive translation systems in BLEU score on German-English translation task of WMT21 (Qian et al., 2021).
In this paper, we argue that multilingual NAT models is not only superior in efficiency, but also can achieve better multilingual translation accuracy, due to its capability of generating high quality codeswitched translations (Lin et al., 2020; Yang et al., 2020; Jose et al., 2020). In particular, we propose switch-GLAT, a carefully designed multilingual version of GLAT, which can outperform multilingual Transformer in both speed and translation quality. Generally, the main idea of switch-GLAT is to employ a code-switch decoder, which can generate contextual code-switched translations (instead of using dictionary for replacement) for a given source sentence, and then perform code-switch back-translation to boost the multilingual translation performance. In more details, the glancing sampling module in GLAT enables switch-GLAT to generate partial translation based on others. Then combined with its multilingual and non-autoregressive characteristics, switch-GLAT could output
Encoder H ParallelDecoder Glancing
Compute Distance
H' ParallelDecoder
Sample words
Replace Input H
Compute Loss
code-switched translations with the employment of token-level language tags instead of sentencelevel ones in auto-regressive models. Ultimately, switch-GLAT can generate contextual translated words at arbitrary positions of the target sentence in arbitrary languages. This greatly improves the multilingual translation performance when we reverse the pairs of source to contextual code-switched target sentences, for training in a back-translation fashion (so called code-switch back-translation).
We conduct extensive experiments on 3 merged translation datasets: WMT with four language pairs (both close languages and distant ones) and WMT with 10 language pairs. switch-GLAT shows consistent improvements over autoregressive multilingual baselines on all datasets, validating that switch-GLAT can achieve better multilingual translation performance simultaneously with a faster decoding speed. We further evaluated the cross-lingual representations through word induction and sentence retrieval tasks. The results demonstrated the proposed code-switch back-translation benefits better-aligned cross-lingual representations.

heading: 2 BACKGROUND
text: Multilingual Neural Machine Translation (MNMT) Given a source sentence X = {x1, x2, ..., xM} with length M and its target sentence Y = {y1, y2, ..., yN} with length N , MNMT leverages the standard bilingual neural machine translation models and extends the source and target inputs respectively with a source and target language token src and tgt. This results in X ′ = {src, x1, x2, ..., xM} and Y ′ = {tgt, y1, y2, ..., yN}. MNMT is generally modeled from X ′ to Y ′ with Transformer (Vaswani et al., 2017). Transformer consists of stacked encoder and decoder layers, which are jointly trained to maximize the conditional probability of Y ′ given X ′:
P (Y ′|X ′) = N∑ i=1 logP (yi|y<i, X ′, tgt; θ) (1)
where θ are the trainable model parameters.
Glancing Transformer prposed by Qian et al. (2020) is a NAT architecture which achieves top results in machine translation with 8x ∼ 15x speedup. It performs two-pass decoding in training but is still fully auto-regressive in inference. In the first decoding pass, given the encoder Fe and decoder Fd, H0d = {h01, h02, ..., h0N} is the decoder input either gathered from the encoder output using soft copy (Wei et al., 2019) or full mask (Ghazvininejad et al., 2019), and then Y is predicted as:
Ŷ = Fd(H 0 d , Fe(X; θ); θ) (2)
where θ are the trainable model parameters. Then glancing transformer (GLAT) adopts glancing sampling strategy to sample a subset of Y according to its distance with Ŷ , thus resulting inGS(Y, Ŷ ). Finally, GLAT predicts the target sequence Y based on this subset and source sentence X in the second decoding pass as follows:
LGLAT = − ∑
yt∈GS(Y,Ŷ )
logP (yt|GS(Y, Ŷ ), X; θ) (3)
where H0d is updated accordingly and GS(Y, Ŷ ) denotes the remaining subset of Y after removing the sampled tokens. The overall architecture of GLAT is shown in Figure 1.
The twice decoding in training makes GLAT capable of predicting some partial translations based on others. Then combined with multilingual and non-autoregressive characteristics, GLAT has the potential to generate contextual code-switched outputs. In the next section, we will show how to modify the GLAT architecture to achieve this.
3 PROPOSED METHOD: switch-GLAT
This section will detail the proposed switch-GLAT. The training objective can be factorized into two parts. One aims to make the model have multilingual translation ability, and the other to achieve better-aligned cross-lingual representations to boost translation performance, respectively denoted as Lmulti and Lcsbt. The overall objective can be formulated as follows:
L = Lmulti + λ · Lcsbt (4)
where λ plays the role of a “temperature” to schedule the importance of code-switch back-translation. As the training progresses, λ is gradually increased, which allows for more complex data instances to be involved, i.e. code-switched translations incorporating more languages and higher switching ratios. This process encourages model to align similar words from different languages into the same vector space, thus boosting machine translation performance. The overall architecture is shown in Figure 2.

heading: 3.1 CODE-SWITCH DECODER
text: switch-GLAT leverages the general parallel decoder of GLAT and extends it into a code-switch decoder by employing the token-level language tag. The code-switch decoder first gains multilingual translation ability through multilingual training, and then it can generate code-switched translations in arbitrary languages with the help of token-level language tag.
Specifically, given a multilingual corpora D = {Dl}Ll=1 consisting of L language pairs, the loss Lmulti is then defined as:
Lmulti = ∑ Dl∈D ∑ (Xlj ,Y l j )∈D l {Ltag(Y lj |Xlj ; θM ) + Lllen(j)}
Lllen(j) = −P (Llj) log P̂ (Llj |[Fe(Xlj ; θM );Esrc;Etgt]; θM )
(5)
where Dl = {(X lj , Y lj )} Nl j=1 is a parallel corpus with size Nl and L l len(j) is the length prediction loss of the j-th pair. P (Llj) is the real length distribution of target sentence, and P̂ (L l j) is the predicted one based on the concatenation of encoder output as well as source and target language embeddings.
θM are trainable model parameters. Ltag is the accordingly updated GLAT training loss (Equation 3) incorporating the source and target language tags:
Ltag(Y l j |Xlj ; θM ) = − ∑ yt∈GS(Y lj ,Ŷ l j ) logP (yt|GS(Y lj , Ŷ lj ), Xlj , src, tgt; θM )
Ŷ lj = Fd(H̃ 0 d , Fe(X l j , src; θ), tgt; θM )
(6)
To involve the indicative language tag, we add it to the first layer input and final layer output at each position of both encoder and decoder as follows:
f̃0i = f 0 i + Esrc; f̃ K i = f K i + Esrc
h̃0j = h 0 j + Etgt; h̃ K j = h K j + Etgt
(7)
where f0i denotes the first encoder layer input at position i and h 0 j denotes the first decoder layer input at position j. Correspondingly, fKi and h K j denote the last layer output. src and tgt are respectively the source and target individual tags, while Esrc and Etgt are their corresponding representations. The overall prediction process of pair (X lj , Y l j ) is illustrated in the left module of Figure 2.
Through the multilingual training process, switch-GLAT can translate between different languages using the indicative language tags, of which the decoder is called code-switch decoder. It has the ability to generate contextual translated words in arbitrary languages due to its token-level characteristics.

heading: 3.2 CODE-SWITCH BACK-TRANSLATION
text: Thanks to the code-switch decoder, we can perform code-switch back-translation (CSBT), which is critical because it encourages model to align the produced words and the original ones into the same vector space according to their similar context information. Better-aligned cross-lingual representations benefit better translation performance.
Specifically, a subset DS = {(Xi, Yi, lsrci , l tgt i )}Si=1 of size S is first sampled, where lsrci and l tgt i are respectively source and target languages of i-th pair. Then, Yi is masked with a given rate PM , leading to Ỹi. Subsequently, the masked positions of Ỹi can be decoded into a third randomly sampled language by leveraging the token-level language tag. Thus, the final decoded sequence Ŷi consists of contextual tokens from mixed languages, which will in turn be taken as the source side input and the original source sentence as the target side input. This process results in a code-switch back-translation corpus DC = {(Ŷi, Xi)}Si=1, which is illustrated in the middle module of Figure 2. The dynamically generated code-switch translations can thus augment data instance distribution to enhance model training as illustrated in the right module of Figure 2. Loss Lcsbt is defined as:
Lcsbt = ∑
(Ŷi,Xi)∈DC
Ltag(Xi|Ŷi; θM ) (8)
As the training continues, the masked rate PM and number of mixed languages are gradually increased. Specifically, the value of PM is iterated from 0.1 to 0.5 with step size 0.1 every 10 epochs. In the first iteration of PM , the number of mixed languages is set to 1. Afterwards, it will be increased to one-third of the total. Through this process, abundant code-switched sentences can be generated, which helps to learn better-aligned cross-lingual representations.

heading: 3.3 SCHEDULED CODE-SWITCH TRAINING
text: The influence of multilingual training and code-switch back-translation is balanced by λ. As the training progresses, λ is increased and more complicated data instances are involved, which means that code-switch back-translation becomes more important. We use the step function to evolve the value of λ:
λ(t) = f(t) + λ(0), f(t) =
{ 0, t < E,
1, t ≥ E (9)
where λ(t) is the importance value at step t and E is a pre-defined changing point to incorporate code-switched translations. λ(0) is the starting value and we set it to 0. As the training goes on, the code-switch back-translation and model training are iterated, leading to well-aligned cross-lingual representations and improved machine translation performance.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS
text: To better measure the effects of multilingualism at different levels, we test our proposed switch-GLAT on the following three merged datasets: (1) WMT-EDF: We collect 4 language pairs from WMT-14 English (En) ↔ German (De) and English (En) ↔ French (Fr). All three languages belong to Indo-European language family and are relatively close on linguistics. (2) WMT-EFZ: We also collect 4 language pairs from WMT-14 English (En)↔ French (Fr) and WMT-17 English (En)↔ Chinese (Zh), which are distant languages on linguistics and their relationships are more difficult to learn. (3) WMT-many: We also gather 10 language pairs from WMT-14 English (En)↔ German (De), English (En)↔ French (Fr), WMT-16 English (En)↔ Russian (Ru), English (En)↔ Romanian (Ro) and WMT-17 English (En)↔ Chinese (Zh) to test switch-GLAT on more diverse language pairs.

heading: 4.2 BASELINE MODELS
text: We compare switch-GLAT against the following representative multilingual baselines: (1) MTransformer: We set a multilingual transformer baseline, which uses a sentence-level language tag on both source and target sides. (2) CLSR: Zhang et al. (2021) present CLSR that learns to route between language-specific or shared pathways from the data itself to obtain the sharing structure. (3) Adapter: Bapna et al. (2019) propose to inject a lightweight adapter layer for each language in MNMT to extract some language-specific feature. We reimplement Adapter since its code is not available currently. (4) MNAT: We also set a multilingual NAT baseline (Gu et al., 2018) modeled by adding token-level language tags to the vanilla NAT. All model settings are the same as switch-GLAT.
Besides, we also report the performance of two typical bilingual models: (1) Transformer: We use the code released by Vaswani et al. (2017) to implement Transformer. (2) GLAT: GLAT is the backbone of our switch-GLAT and it achieves comparable results to Transformer.
To better analyze the influence of different components in our switch-GLAT, we also conduct ablation tests as follows: (1) switch-GLAT-w/o-glancing: switch-GLAT removes glancing sampling strategy in training. (2) switch-GLAT-w/o-CSBT: switch-GLAT is trained without code-switch back-translation, which means the model is only trained on the golden parallel corpora. (3) switchGLAT-w/o-CCS: switch-GLAT constructs the code-switched data by simply replacing aligned words using a bilingual dictionary instead of contextualized code-switching.

heading: Bilingual models

heading: Multilingual models

heading: 4.3 TRAINING AND INFERENCE DETAILS
text: We use 6 layers for encoder and parallel decoder. The model hidden size dmodel and feed-forward hidden size dff are set to 512 and 2048 respectively. The number of attention head is set to 8. All the datasets are tokenized and segmented into subword units using BPE encodings (Sennrich et al., 2016). The vocabulary size is set to 75k for WMT-EDF/EFZ and 95k for WMT-many.

heading: Bilingual models

heading: Multilingual models
text: Following the previous works (Gu et al., 2018; Li et al., 2020; Ma et al., 2019), the sequence-level knowledge distillation is used for all datasets, and the Transformer-base architecture (Vaswani et al., 2017) is employed as the teacher model.
The changing point E is set to 300, 000 steps and sampling number S is set to 300, 000 for each pair. The mini-batch size is set to 64k tokens and the maximum training step is 800, 000. The model is trained with 8 NVIDIA Tesla V100 GPU cards. We follow the default parameters of Adam optimizer (Kingma & Ba, 2014) and learning rate schedule in Vaswani et al. (2017). Dropout annealing strategy (Rennie et al., 2015) is applied to stable training and the initialized dropout rate is set to 0.3. In training, data from different language pairs are sampled according to a multinomial distribution rebalanced by a temperature of 0.3 (Conneau et al., 2019).
In inference, following Qian et al. (2020), switch-GLAT performs one-iteration parallel decoding. Moreover, we also leverage the common practice of noise parallel decoding (Gu et al., 2018; Ma et al., 2019) and the number of length reranking candidate is set to 5.

heading: 4.4 MAIN RESULTS

heading: 4.4.1 RESULTS ON WMT-EDF/EFZ
text: The left and right halves of Table 1 respectively show the results of WMT-EDF and WMT-EFZ.
switch-GLAT outperforms all multilingual competitors on the average score in both settings. Specifically, in WMT-EDF scenario, our model achieves the best average BLEU score of 33.47 among all multilingual competitors, especially exceeding multilingual Transformer with a fairly significant margin of 0.46 point. Moreover, it performs better than CLSR and Adapter with an improvements of 0.99 and 1.06 points respectively. The same phenomena can be observed in WMT-EFZ scenario where the language pairs are more distant on linguistics. Our interpretation is that switch-GLAT has the ability to produce context-dependent code-switched translations with the code-switch decoder.
The augmented code-switched translations can help model align the generated tokens and the original ones into the same vector space due to their similar context information, thus boosting machine translation performance (Mikolov et al., 2013).
switch-GLAT can largely speed up decoding. In WMT-EDF scenario, the decoding speed tested on Fr-En pair is reported in Table 1, which is evaluated in the throughput of each model (full speed values are illustrated in Figure 3 (a) and Figure 3 (b)). Table 1 shows that the decoding speed of switch-GLAT is 6.6 times faster than multilingual transformer. Additionally, Figure 3 (a) and (b) also illustrate that the throughput of our model is significantly larger than multilingual transformer on all directions. These validate that the switch-GLAT can greatly improve decoding efficiency thanks to its parallel decoder. Note that the length predictor is jointly trained incorporating source and target language embeddings, thus leading to different length preference in different models.
switch-GLAT performs better than GLAT on average. In both scenarios, switch-GLAT achieves higher accuracy in 3 out of 4 directions than GLAT, especially X-to-En directions. These indicate that multilingual training makes it possible to transfer knowledge so that performance of X-to-En direction is enhanced due to the augmented data in English on the target side.

heading: 4.4.2 RESULTS ON WMT-MANY
text: Table 2 shows the results on the gathered WMT dataset from 10 language pairs.
switch-GLAT performs better than multilingual baselines in most cases. It is primarily because switch-GLAT is able to produce diverse contextual code-switch translations, which encourage the model to automatically align the generated word vectors and the original ones due to their similar context information. Well-aligned representations are beneficial for improving machine translation performance. Notably, Adapter performs much worse than multilingual Transformer on En-Ro and En-Ru directions as the original paper interprets that Adapter do not perform well in low-resource settings (the reverse directions are enhanced by the English data from other language pairs).
switch-GLAT performs slightly worse than GLAT. In this scenario, the failure of switch-GLAT may be owing to the limitation of model capacity. However, switch-GLAT outperforms GLAT when the model translates from a foreign language to English. One interpretation is that all the parallel corpora including English on either source or target side can lead to a better parameter estimation of the English decoder (Firat et al., 2016).
switch-GLAT can enlarge the power of efficient decoding. The decoding speed in this setting are illustrated in Figure 3 (c), showing that the throughput of our model is much larger than multilingual transformer on 10 directions. It demonstrates that switch-GLAT can enlarge the power of decoding efficiency since all translation directions can be accelerated within a single model.

heading: 4.5 ABLATION STUDY
text: The last three lines in Table 1 and Table 2 show the results of ablation study.
Glancing sampling strategy plays an important role in switch-GLAT. After removing the glancing sampling strategy, the performance drops the most. It validates that glancing sampling is critical in training an advanced parallel multilingual translation model, which makes it suitable to generate high-quality code-switched translations.
Code-switch back-translation promotes multilingual translation. Excluding code-switch backtranslation or employing code-switched data constructed by dictionary substitution both harms translation performance, of which the former hurts more. It validates that applying code-switched data can significantly increase multilingual translation performance, especially those generated by incorporating context information instead of dictionary replacing.

heading: 5 ANALYSIS
text: In this section, we will make a rigorous analysis to reveal the reasons why our switch-GLAT can achieve better translation performance than multilingual transformer. All tasks are tested in WMTEDF scenario.

heading: 5.1 VISUALIZATION
text: There is much smaller gap in the cross-lingual representations learned by our switch-GLAT. To intuitively illustrate how well the cross-lingual representations are aligned, we plot a two-dimensional projection of representations learned by four models in Figure 4. Here multilingual models are trained on the merged WMT-EDF dataset, while bilingual ones on the English → German part. We use the t-SNE algorithm (van der Maaten Laurens & Hinton, 2008) to perform the projection. English and German words are gold word pairs from Open Multilingual WordNet (OMW) datasets (Bond & Foster, 2013). English words are displayed in blue color and German words in red. We find there is much more overlapping between blue and red areas in Figure 4 (d), validating that switch-GLAT can produce better-aligned cross-lingual representations than other models.

heading: 5.2 CROSS-LINGUAL WORD INDUCTION
text: The cross-lingual representations are aligned well at word level. We assess the cross-lingual word induction performance to see how well the similar words from different languages are close to each other in the learned vector space (Gouws et al., 2015). Specifically, golden word pairs are extracted from the OMW datasets (Bond & Foster, 2013), resulting in 10, 585 En-De and 12, 511 EnFr pairs. Top-1 accuracy is shown in the left half of Table 3. It shows that switch-GLAT significantly outperforms all autoregressive multilingual competitors on average, indicating that cross-lingual representations in our model are aligned well at word level.

heading: 5.3 PARALLEL SENTENCE RETRIEVAL
text: The cross-lingual representations are aligned well at sentence level. To further assess the quality of the cross-lingual representations at sentence level, we employ parallel sentence retrieval (PSR) task. PSR aims to extract parallel sentences from a comparable corpus between two languages. Following the settings of XTREME (Hu et al., 2020), we use the Tatoeba dataset (Artetxe & Schwenk, 2019). Cosine similarity is leveraged to search the nearest neighbour and the search accuracy of each model is reported in the right half of Table 3. It can be seen that switch-GLAT achieves the highest accuracy
among all competitors, validating that better contextualized embeddings are obtained in switch-GLAT as the sentence embeddings are calculated with regarding all tokens in it.

heading: 6 RELATED WORK

heading: 6.1 MULTILINGUAL NEURAL MACHINE TRANSLATION
text: Early works extend the NMT model proposed by Bahdanau et al. (2014) to a multilingual version based either on LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Cho et al., 2014). Dong et al. (2015) propose a one-to-many translation model (from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baseline. Zoph & Knight (2016) present a multi-source NMT which significantly outperforms a single-source model. Firat et al. (2016) propose a many-to-many model involving 10 language pairs by leveraging separate encoder and decoder per language as well as a sharing attention mechanism. Further, Ha et al. (2016) and Johnson et al. (2017) propose to train a multilingual model with a shared encoder-decoder-attention network across all languages to perform many-to-many translation.
Recently, Transformer (Vaswani et al., 2017) is applied as the backbone of MNMT by many works. The Multi-Distillation (Tan et al., 2019) is trained to simultaneously fit the training data and match the outputs of individual bilingual models through knowledge distillation. Bapna et al. (2019) present to inject a language-specific adaptation layer into a pretrained model to keep the language-specific features. Aharoni et al. (2019) propose a multilingual NMT model involving up to 102 languages which achieves good zero-shot performance.
All the above MNMT systems are based on autoregressive architectures which follow a sequential generation process and are limited to slow decoding speed.

heading: 6.2 NON-AUTOREGRESSIVE NEURAL MACHINE TRANSLATION
text: Recently, NAT has attracted much attention due to its ability of parallel decoding. A fully nonautoregressive model is first proposed by Gu et al. (2018), based on which latent variables are introduced to deal with dependencies among target words (Ma et al., 2019; Bao et al., 2019; Ran et al., 2021). Other branches of fully non-autoregressive systems either try to transfer knowledge from autoregressive models (Li et al., 2018; Wei et al., 2019; Guo et al., 2020; Sun & Yang, 2020), or apply different training objectives (Libovickỳ & Helcl, 2018; Shao et al., 2020; Ghazvininejad et al., 2020; Qian et al., 2020).
Besides, some works build NAT systems with structured prediction to model inter-dependencies among output words. Sun et al. (2019) propose to incorporate a structured inference module into the non-autoregressive models. Deng & Rush (2020) present a Markov transformer to perform cascaded decoding. Additionally, serveral semi-autoregressive models are also constructed to refine the outputs with multi-pass iterative decoding (Lee et al., 2018; Gu et al., 2019; Ghazvininejad et al., 2019; Kasai et al., 2020).
All these works are limited to bilingual translation tasks. To our best knowledge, switch-GLAT is the first one to perform multilingual neural machine translation based on a non-autoregressive architecture.

heading: 7 CONCLUSION
text: This paper proposes switch-GLAT, a non-autoregressive multilingual neural machine translation model. It incorporates a switch-decoder, which can produce contextualized code-switched sentences and perform code-switch back-translation. Through this process, the multilingual translation performance and cross-lingual representations can both be improved. Besides, its parallel decoder enables a highly efficient inference. Experiments on three datasets suggest that switch-GLAT is superior to multilingual transformer in both effectiveness and efficiency. Further analyses also demonstrate the effectiveness of the learned contextual cross-lingual representations.

heading: A APPENDIX

heading: A.1 COMPARING WITH VANILLA BACK-TRANSLATION
text: We conduct ablation study of using vanilla back-translation (switch-GLAT-with-vanilla BT) in Table 4. It shows that though it outperforms model that do not use any augmentation strategy (switch-GLATw/o-CSBT), it can further improved by leveraging code-switch back-translation, which validated that our proposed code-switch back-translation is favorable.

heading: A.2 APPLYING CSBT TO AUTOREGRESSIVE MODEL
text: We apply our proposed code-switch back-translation (CSBT) to multilingual transformer, and the results are shown in Table 5. It shows that the CSBT also boosts the performance of multilingual Transformer. Though M-Transformer-CSBT performs slightly better than our switch-GLAT, our model could significantly improve inference efficiency.

heading: A.3 TRANSFER ABILITY
text: To validate that our model can improve the performance of low-resource languages, we train our switch-GLAT on the subset of TED dataset that are merged from English (En)-Spanish (Es), EnglishFrench (Fr) and English-Portuguese (Pt) pairs. The results are shown in the following table. The size of En-Es and En-Fr are about four times that of En-Pt. The results show that our switch-GLAT can significantly improve the performance of low-resource languages with as much as 20.14 and 1.18 BLEU points respectively compared with GLAT and Transformer.
To further validated the zero-shot ability of our model, we test the model (trained on WMT-many) on the test set of German-French in WMT 2019. Table 7 shows that zero-shot abilities of switch-GLAT and multilingual Transformer are both poor, but applying self pivot to these two models can achieve good performance.

heading: A.4 RESULTS WITH A BIG MODEL
text: As capacity is crucial for multilingual models, we conducted experiments to show how switch-GLAT performance compares as model size grows. Since producing KD data is time-consuming, we train big model on the KD data by base model. The results are shown in Table 8. It shows that in a multi-direction scenario, GLAT performs better than switch-GLAT when trained on a base model, but increasing model size achieves opposite results, demonstrating that model capacity is quite critical for multilingual models.

heading: A.5 APPENDED VISUALIZATION
text: We also visualize the representations learned by CLSR since it is also a representative multilingual baseline. Again, we find there is much more overlapping between blue and red areas in Figure 5 (d), validating that switch-GLAT can produce better-aligned cross-lingual representations than other models.

heading: A.6 HYPER-PARAMETER SELECTION
text: We tune the training step of changing lambda (100000 to max steps with step size 100000), sample number S (100000 to max data size with step size 200000) and initialized dropout rate (0.1 to 0.5
with step size 0.1)with grid search on the validation set of WMT-EDF, and found that the values stated in the paper performed best.

heading: A.7 CASE STUDY
text: To gain an insight on how well our model can generate code-switched sentences, we provide some examples in Table 9. It shows that constructing code-switched sentences by dictionary replacement may cause errors. For example, pisar is a verb while the original word step is a noun, and using a verb is not appropriate.

Review summaries:
The paper proposed a novel method for the multilingual non-autoregressive machine translation (NAT) model. The key idea is to modify the decoder module of glancing Transformer (GLAT), making it generate the contextual code-switched translations, then use them to perform code-switch back-translation. To incorporate with the generation of contextual code-switch translations, the authors proposed to add token-level language tags to the first layer and the final output layer at each position. The empirical results show that: 1) the inference speed is faster than the autoregressive baseline model (M-Transformer). 2) the translation performance outperforms the NAT baselines in terms of WMT-EDF, WMT-EFZ benchmarks. Although the performance is slightly worse than the GLAT model on the WMT-Many benchmark, the author claim that the reason is due to the limitation of model capacity. 3) Two cross-lingual experiments and the visualization of representation prove that the proposed method has better cross-lingual capability compared to other multilingual models. 

This paper describes experiments in training a non-autoregressive Transformer (glancing Transformer, or GLAT) in a multilingual MT setting. The use of a non-autoregressive model, plus token-level language tags, enables a code-switching strategy where a source sentence in a given language can generate a target sentence containing words from multiple specified languages. This is used to augment the multilingual training corpus with back-translated data consisting of code-switched synthetic source sentences paired with original monolingual target sentences. A multilingual GLAT model trained with this strategy is shown to outperform many baselines including monolingual GLAT and several autoregressive multilingual models. Ablation studies demonstrate that code-switched back-translation is essential for achieving good performance. 

This paper extends non-autoregressive NMT models, particularly the GLAT model, from bilingual translation to multilingual translation. To enable such extension, the authors proposed token-level language tags for the decoder, code-switch decoder paired with back-translation, and also scheduled training between the traditional MLE loss and the BT loss. Experiments on several multilingual tasks based on WMT corpus show that the proposed model obtains better supervised translation quality than its auto-regressive counterparts and delivers better cross-lingual representations.