switch-GLAT: Multilingual Parallel Machine Translation Via Code-Switch Decoder
Authors: Zhenqiao Song
Hao Zhou
Lihua Qian
Jingjing Xu
Shanbo Cheng
Mingxuan Wang
Lei Li
Venue: ICLR 2022 Poster (2022)
Keywords: multilingual non-autoregressive machine translation
contextualized code-switching
back-translation

Abstract: Multilingual machine translation aims to develop a single model for multiple language directions. However, existing multilingual models based on Transformer are limited in terms of both translation performance and inference speed. In this paper, we propose switch-GLAT, a non-autoregressive multilingual machine translation model with a code-switch decoder. It can generate contextual code-switched translations for a given source sentence, and perform code-switch back-translation, greatly boosting multilingual translation performance. In addition, its inference is highly efficient thanks to its parallel decoder. Experiments show that our proposed switch-GLAT outperform the multilingual Transformer with as much as 0.74 BLEU improvement and 6.2x faster decoding speed in inference.


Published as a conference paper at ICLR 2022
switch-GLAT: MULTILINGUAL PARALLEL MACHINE TRANSLATION VIA CODE-SWITCH DECODER
Zhenqiao Song', Hao Zhou"! Lihua Qian', Jingjing Xu', Shanbo Cheng', Mingxuan Wang's Lei Li
ByteDDnce Al Lab, Shanghai, China University of California, Santa Barbara songghengiaa zhouhao.. nip, gianl ihua @byt com, lile i @ucsb. edu (chengshanbo, wangmingxuan. 89) @bytedance. com, j jingjingxuéppe edu. cn
ABSTRACT
Multilingual machine translation aims to develop a single model for multiple language directions. However, existing multilingual models based on Transformer are limited in terms both translation performance and inference speed. In this paper, propose switch-GLAT,, non-autoregressive multilingual machine translation model with a code-switch decoder. It can generate contextual code- switched translations for given source sentence, and perform code-switch back- translation, greatly boosting multilingual translation performance. In addition, its inference is highly efficient thanks to its parallel decoder. Experiments show that our proposed switch-GLAT outperform the multilingual Transformer with as much as 0.74 BLEU improvement and 6.2x faster decoding speed in inference.
INTRODUCTION
Neural machine translation (NMT) is the current state-of-the-arr approach (Bahdanau et al. 2014 Sutskever al. 2014) for machine translation in both academia (Bojar et al. 2017) and industry (Has-
san et al.) 2018). Recent works (Firat et al.) 2016 Johnson et al. 2017| Aharoni et al. 2019 Lin et al. 2020) extend the approach to support multilingual translation, i.e. training a single model that can translate across multiple language directions. Multilingual models are appealing for several reasons.
First, they can reduce the online translation service number, enabling simpler deployment (Arivazha gan et al. 2019) when plenty of translation directions are required. Additionally, multilingual training makes possible to transfer knowledge from high-resource languages to low-resource ones, thus improving the translation quality of low-resource directions ZZoph al. 2016: Johnson et al. 2017]
Wang & Neubig 2019).
However, most multilingual NMT systems are built upon the autoregressive architecture, which translates from left to right and thus is not efficient enough in terms translation speed. Such efficiency problem is more serious in multilingual setting because all translation directions suffer from this slow inference speed.
straightforward solution to improve the multilingual translation efficiency is develop multi- lingual non-autoregressive translation (NAT). NAT generates translation outputs in parallel (Gu
al. 2018). which leads significantly faster translation speed. Thanks to the recent progress of NAT Ghazvininejad et al. 2019:; Gu et al. 2019 Deng & Rush 2020), current state-of-the-ar NAT
models have achieved comparable BLEU scores (Li et al. 2018 Wei al. 2019 Qian al. 2020
with their retto-regressive counterparts. Among them, the glancing transformer (GLAT) proposed by Qian et al. (2020) is a representative work, which even outperforms many strong autoregressive
translation systems in BLEU score on German-English translation task of WMT21 (Qian et al. 2021).
In this paper, we argue that multilingual NAT models is not only superior in efficiency, but also can achieve better multilingual translation accuracy, due to its capability of generating high quality code- switched translations (Lin et al.) 2020 Yang et al. 22000] Jose et al. 2020)). In particular, we propose switch-GLAT, carefully designed multilingual version of GLAT, which can outperform multilingual Transformer in both speed and translation quality. Generally, the main idea of switch-GLAY GLAT is to
"Corresponding author.

---

Published as a conference paper at ICLR 2022
katpe
H'
hg
Sumple
str,2) wurds
Replace Inpor
Decoder
attention
Glancing
D2
Glancing Sampligg
ho
Compuse distunce
Die
Puraliel Decoder
91
92
93
94
9s
Encoder
hg
hy
hy
hs
very
Figure I: The overall architecture of GLAT. The left half shows the training process of GLAT while the right half details the glancing sampling strategy.
employ a code-switch decoder, which can generate contextual code-switched translations (instead of using dictionary for replacement) for given source sentence, and then perform code-switch back-translation to boost the multilingual translation performance. In more details, the glancing sampling module in GLAT enables switch-GLA combined with its multilingual and non- oon-autoregressiee characteristics, switch-GLAV could output code-swite switched translations with the employment of token-level language tags instead of sentence- level ones in auto- regressieee models. Ultimately, switch-Gl. GLAT can generate contextual translated words at arbitrary positions of the target sentence arbitrary languages. This greatly improves multilingual translation performance when we reverse the pairs of source to contextual code-switched target sentences, for training in a tock-translttion fashion (so called code-switch back bocc-trasslatio).
generate partial translation based on others. Then
We conduct extensive experiments on 3 merged translation datasets: WMT with four language pairs (both close languages and distant ones) and WMT with 10 language pairs. switch-GLAT shows consistent improvements over autoregressive multilingual baselines all datasets, validating that switch-GLAT can achieve better multilingual translation performance simultaneously with a faster decoding speed. We further evaluated the cross-lingual representations through word induction and sentence retrieval tasks. The results demonstrated the proposed code-switch back-translation benefits better-aligned cross-lingual representations.
2 BACKGROUND
Multilingual Neural Machine Translation (MNMT) Given a source sentence X (21,42, FM) with length M and its target sentence Y {yr, ... VNN with length N, MNMT leverages the standard bilingual neural machine translation models and extends the source and target inputs respectively with a source and target language token src and tgt. This results in X' {sre, I2, ..., ZMM and Y' {tgt, V1, U/. J.
MNMT generally modeled from X' to Y' with Transformer (Vaswani et al. 2017). Transformer consists of stacked encoder and decoder layers, which jointly trained to maximize the conditional probability of Y" given X':
N
P(Y Σ
log P(yrlees ttggt)))
where 0 are the trainable model parameters.
Glancing Transformer prposed by Qian et al. 2020) is NAT architecture which achieves top results in machine translation with 8x ~ 15x speedup. It performs two-pass decoding in training but is still fully non-autoregressive in inference. In the first decoding pass, given the encoder F, and decoder Fa, HD [44, h%) is the decoder input either gathered from the encoder output using soft copy (Wei et 2019 or full mask (Ghazvininejad al. 2019), and then Y is predicted as:
Ý Fa(H2, Fx(:0)):0)
(2)
where 0 are the trainable model parameters. Then glancing transformer (GLAT) adopts glancing sampling strategy sample subset of Y according its distance with Y, thus resulting in GS Ý).
2

---

Published as a conference paper at ICLR 2022
Token-level
Code-Switch
Code-Switch
Language Eng
Decoder
Iranslation
kaize
schr
chat
lindo
Gat
very
Decoder
Decader
Decoder
attention
attention
Glaocing Sampling
500 D
De
DC
De
De
Ls
Encader
Encoder
Encoder
The
VITY
CHil
DNC
ChIL
maY
lindo
E
Fm
FHI
Figure 2: The overall architecture of switch-GLAT. The left module shows that GLAT is extended multilingual version with the token-level language tag, the middle one describes how the code- switched sentence is produced through the code-switch decoder, and the right module introduces the code-switch back-translation process, in which the generated code-switched target and its paired source is swapped to enhance training.
Finally, GLAT predicts the target sequence based on this subset and source sentence X in the second decoding pass follows:
LGLAT
Σ
log P(y. (Y)
EGS(Y)
where H. updated accordingly and CY Y) denotes remaining subset of Y after removing the sampled tokens. The overall architecture of GLAT is shown in Figure
The twice decoding in training makes GLAAT capable of predicting some partial translations based others. Then combined with multilingual and non-autoregressive characteristics, GLAT has the potential to generate contextual code-switched outputs. In the next section, we will show how to modify the GLAT architecture achieve this.
3 PROPOSED METHOD: switch-GLAY
This section will detail the proposed switch-GLAT. The training objective can be factorized into two parts. One aims make the model have multilingual translation ability, and the other achieve better-aligned cross-lingual representations to boost translation performance, respectively denoted as Laula and Leabt. The overall objective can be formulated as follows:
L Limulti + A Lesbs
(4)
where À plays the role of "temperature" to schedule the importance of code-switch back-translation. As the training progresses, À is gradually increased, which allows for more complex data instances to be involved, i.c. code-switched translations incorporating more languages and higher switching ratios. This process encourages model to align similar words from different languages into the same vector space, thus boosting machine translation performance. The overall architecture is shown in Figure
3.1 CODE-SWITCH DECODER
switch-GLAT leverages the general parallel decoder of GLAT and extends it into code-switch decoder by employing the token-level language tag. The code-switch decoder first gains multilingual translation ability through multilingual training, and then it can generate code-switched translations in arbitrary languages with the help of token-level language tag.
Specifically, given multilingual corpora D {ID*) HD consisting of L language pairs, the loss
Lmuits then defined as:
Lmnulti
Σ
Σ
[Iass (Y) + Il)
DIED (XS,
(5)
(la(()) -P(()) log P(L)) 0Mr); Emci Engl 0xs)
3

---

Published as a conference paper at ICLR 2022
where D' ((X,, y,)) is parallel corpus with size Nj and Lien() (j) is the length prediction loss of the j-th pair. P(Lÿ) is the real length distribution of target sentence, and P(L)) is the predicted one based the concatenation of encoder output as well source and target language embeddings. I are trainable model parameters, Liag is the accordingly updated GLAT training loss (Equation 3)
incorporating the source and target language tags:
Leag (Y3) (x)
Σ
log P( GB(G) Yj), Xj,, sre, tgt; Oax)
ME W.EGSO
(6)
vJ F& F. (X,,, 0), Ow)
To involve the indicative language tag, we add it to the first layer input and final layer output each position of both encoder and decoder as follows:
J 1 Farci fi Si + Fare
k h3 Fagt; h h + Fagt
where S denotes the first encoder layer input position i and hy denotes the first decoder layer input at position j. Correspondingly, f.K and 1. denote the last layer output. src and Igt are respectively the source and target individual tags, while Earr. and Eig are their corresponding representations. The overall prediction process of pair (X), YJ) is illustrated in the left module of Figure
Through the multilingual training process, Switch-GLAT can translate between different languages using the indicative language tags, of which the decoder is called code-switch decoder. It has the ability to generate contextual translated words arbitrary languages due to its token-level characteristics.
3.2 CODE-SWITCH
Thanks to the code-switch decoder, we can perform code-switch fack-translation (CSBT), which is critical because it encourages model align the produced words and the original ones into the same vector space according to their similar context information. Better-aligned cross-lingual representations benefit better translation performance.
Specifically, subset D3 ((x,, 23re, 1 ) of size S is first sampled, where 15 and g. are respectively source and target languages of i-th pair. Then, Y; is masked with a given rate Par, leading to Y, Subsequently, the masked positions of Y, can be decoded into a third randomly sampled language by leveraging the token-level language tag. Thus, the final decoded sequence Y; consists of contextual tokens from mixed languages, which will in turn be taken as the source side input and the original source sentence as the target side input. This process results in a code-switch back-translation corpus De X.) which is illustrated in the middle module of Figure
The dynamically generated code-switch translations can augment data instance distribution to enhance model training as illustrated in the right module of Figure 2 Loss Lesht is defined as:
Leabt
Σ
(8)
(FG,)))EDC
As the training continues, the masked rate PMr and number of mixed languages are gradually increased. Specifically, the value of Pas is iterated from 0.1 to 0.5 with step size 0.1 every 10 epochs. In the first iteration of Par the number of mixed languages is set to 1. Afterwards, it will be increased to one-third of the total. Through this process, abundant code-switched sentences can be generated, which helps to learn better-aligned tross-lingual representations.
3.3 SCHEDULED CODE-SWITCH TRAINING
The influence of multilingual training and code-switch back-translation is balanced by X. As the training progresses, À is increased and more complicated data instances are involved, which means that code-switch back-translation becomes more important. We use the step function to evolve the
value of X:
(0, << E,
X(t) = f(t) + A(O), S(1)
(9)
1
t E
4

---

Published as a conference paper at ICLR 2022
where ((t is the importance value step and E is pre-defined changing point to incorporate code-switched translations. A(0) is the starting value and we set it to 0. As the training goes on, the code-switch back-translation and model training iterated, leading to well-aligned cross-lingual representations and improved machine translation performance.
4 EXPERIMENTS
4.1 DATASETS
To better measure the effects multilingualissm at different levels, we test our proposed switch-GLAT on the following three merged datasets: (1) WMT-EDF: We collect 4 language pairs from WMT-14
English (En) < German (De) and English (En) + French (Fr). All three languages belong to
Indo Indo-European language family and are relatively close linguisties. (2) WMT-EFZ: We also collect
4 language pairs from WMT-14 English (En) <> French (Fr) and WMT-17 English (En) ++ Chinese (Zh), which are distant languages linguistics and their relationships are more difficult to learn. (3)
WMT-many: We also gather 10 language pairs from WMT-14 English (En) <> German (De), English (En) French (Fr), WMT-16 English (En) + Russian (Ru), English (En) ++ Romanian (Ro) and
WMT-17 English (En) + Chinese (Zh) to test switch-GLAT on more diverse language pairs.
4.2 BASELINE MODELS
We compare switch-GLAT against the following representative multilingual baselines: (1) M- Transformer: We a multilingual transformer baseline, which also uses token-level language tags on both source and target sides. (2) CLSR: Zhang et al. (2021) present CLSR that learns to route between language-specific
shared pathways from the data itself to obtain the sharing structure. (3) Adapter: Bapna et al. (2019) propose to inject a lightweight adapter layer for each language in MNMT to extract some kanguage-spectnn
features. We reimplement Adapter since its code is not available currently. (4) MNAT: We also set multilingual NAT baseline (Gu et al. (2018) modeled by adding token-level language tags to the vanilla NAT. All model settings are the same as switch-GLAT.. Besides, also report the performance of two typical bilingual models: (1) Transformer: We use the code relcased by Vaswani et al. (017)) to implement Transformer. (2) GLAT: GLAT is the backbone of our switch-GLAT and it achieves comparable results Transformer. To better analyze the influence of different components in our sWitch-GLAT, we also conduct ablation tests as follows: (1) - switch-GLAN removes glancing sampling strategy in training. (2) www.th-GLAT--ee -AW.-L/--TT---TTTTTT switch-GLAY is trained without code-switch back-translation. which means the model is only trained the golden parallel corpora. (3) Crricc-GLLT--Wo-- switch-GLAT constructs the code-switched data by simply replacing aligned words using bilingual dictionary instead of contextualized code-switching.
WMT-EDF
WMT-EFZ
Models
En-De
De-Ei
Ea-Fr
Avg
Speed
En-Fir
Fi-En
En-Zh
Zh-En
Avg
Bilingual models
Transformer
27.77
31.55
38.80
37.35
33.86
1.3x
38.80
37.35
23.60
24.05
30.95
GLAT
26.09
30.53
38.62
34.44
32.42
6.3
38.62
34.44
21.05
22.89
Mubrilinguol models
modills
M TrTrmssoomer
25.84
31.57
38.52
36.03
32.99
1.0%
38.06
35.07
20.76
22.19
29.02
CLSR
23.51
31.29
31.67
35.62
20.23
21.13
28.59
Adapier
22.19
29.56
40.72
35.88
32.08
0.9x
40.13
35.02
19.87
21.29
29.08
MNAT
13.82
21.89
2T.331
25.28
21.311
5.99
19.46
20.15
7.89
F:35
13.17
switeh-GLAT
25.27
31.29
40.81
36.00
33.34
6.2x
40.54
36.48
19.47
22.55
29.76
w/a glancing
17.76
23.28
25.91
29.15
24.02
6.0%
21.28
21.97
8.62
8.02
14.97
-wo CSBT
24.29
29.03
36.45
34.09
30.07
6.2
35.17
33.65
18.32
20.37
26.87
w/o CCS
21.72
29.51
37.32
34.17
31.43
6.1xx
35.96
33.61
18.45
20.83
27.21
Table Translation performance (BLEU) on WMT-EDEFEFZ! Avg means the average BLEU score.
We updated all results in camera ready version using multi bleu,perl script of Moses for evaluation because we misused -Ic command in the original submission. Though the average BLEU scores were 0.0 ~ 0.8 points lower than before, the overall conclusions changed.
5

---

Published as a conference paper at ICLR 2022
WMT-many
Models
En. De
De En
En-FF
Fr-En
En Ro
Ro. En
En-Ru
Ru-En
En-Zh
Zh. En
Avg
Bilingual models
Transformer
27.77
31 31.55
38, 80
37.35
33.01
33.59
28. 28.22
29.89
23.60
24.05
30,78
GLAT
26.09
30.53
38.62
34.44
31.83
32.39
25.42
28.13
21.05
22.89
29.20
Multitingual models
M-Transformer
23.14
20.38
35.19
34.07
34.36
35.26
24.63
29. 14
16.63
20.17
28.
CLSR
21.57
30.01
34.07
32.72
30.61
35.29
19.13
30.27
16.28
20.19
27.01
Adapter
23.26
29.87
38,79
40.03
30.42
32.01
22.87
26.38
18.29
20.31
28,22
MNAA
8.33
13.86
1:2.07
19.21
12.89
"21.36
7..66
16743
3.85
6,00
12.16
30.49
39.47
36.30
24.16
28.33
-wlo glancing
6.83
16.22
11.08
20.33
13.57
23.34
7.07
18.25
7.31
6.74
13.13
CSBT
22.48
27 91
33.35
29.89
32.41
22.43
26.48
13.33
18.32
26.01
www CCS
22.25
28.13
34.01
33.62
30.16
33.0T
21.68
26.53
14.92
19.07
26.34
Table Translation performance (BLEU) on WMT-many². Avg means the average score.
Inference spend on WMT-EDF
Inference speed WKT.
Inference speed on WMT-many
M-Trunaformar
Throughout (sentencesis)
WiTranafomm av I--CLAT
M-Trnnafomor awichhCLAT
switch-CAT
26
20
28
20
15.
15
Throughtut
Throughout
10-
10
de-en
tn-de
freen
en-fr
frin
zh-en
67r2h
(a) WMT-EDF
(b) WMT-EFZ
(c) WMT-many
Figure 3: Inference speed evaluated on throughputs.
4.3 TRAINING AND INFERENCE DETAILS
We use 6 layers for encoder and parallel decoder. The model hidden size dmode and feed-forward All the hidden size f are 512 and 2048 respectively. The number attention head is datasets are tokenized and segmented into subword units using BPE encodings (Sennrich et al. 2016). The vocabulary size is set 85k for WMT-EDF and 95k for WMT-EFZ/many. The changing point E is set to 300, 000 steps and sampling number S is set to 300, 000 for each pair. The mini-batch 200, 000. The model trained with NVIDIA Tesla V100 GPU cards. We follow the default parameters of Adam optimizer (Kingma & Ba 2014) and learning rate schedule in Vaswani et al. (2017). Dropout annealing strategy (Rennic al. (015)) is applied to stable training and the initialized dropout rate is set to 0.3. In training, data from different language pairs are sampled according to a multinomial distribution rebalanced by a temperature of 0.3 (Conneau al. 2019).
size set to 64k tokens and the maximum training step
Following the previous works (Gu et al. 2018 Li et al. 2020 Ma et al. 2019). the sequence-lever knowledge distillation is used for all datasets, and the Transformer-bass architecture (Vaswani et al. 2017) is employed as the teacher model. In inference, following Qian et al. (202)), switch-GLAA performs one-iteration parallel decoding. Moreover, we also leverage the common practice of noise parallel decoding (Gu et al. 2018) and the number of length reranking candidate is set to 5.
4.4
MAIN RESULTS
4.4.1 RESULTS ON WMT-EDF/EFZ
The left and right halves of Table respectively show the results of WMT-EDF and WMT-EFZ.
We updated all results in camera ready version using multi bleu,prrl script of Moses for evaluation because we misused -Ic command in the original submission. Though the average BLEU scores were 0.0 ~ 0.5 points lower than before, the overall conclusions changed.

---

Published as a conference paper at ICLR 2022
switch-GLAT outperforms all multilingual competitors on the average score in both settings. Specifically, in WMT-EDF scenario, our model achieves the best average BLEU score of 33.34 among all multilingual competitors, especially exceeding multilingual Transformer with fairly significant margin of 0.35 point. Moreover, it performs better than CLSR and Adapter with an improvements of 1.33 and 1.26 points respectively. The same phenomena can be observed in WMT-EFZ scenario where the language pairs are more distant on linguistics, Our interpretation is that switch-GLAT has the ability to produce context-dependent code-switched translations with the code-switch decoder. The augmented code-switched translations can help model align the generated tokens and the original ones into the same vector space due to their similar context information, thus boosting machine translation performance (Mikolov et al. 2013).
sceeaaio
switch-GLAT can largely speed up decoding. In WMT-EDF scenario, the decoding speed tested
on Fr-En pair is reported in Table 1 which is evaluated in the throughput of each model (full speed
values are illustrated in Figure 3] (a) and Figure (b)). Table shows that the decoding speed of
switch-GLAT is 6.2 times faster than multilingual transformer. Additionally, Figure (a) and (b) also
illustrate that the throughput of our model is significantly larger than multilingual transformer on all directions. These validate that the switch-GLAY can greatly improve decoding efficiency thanks to its parallel decoder. Note that the length predictor is jointly trained incorporating source and target language embeddings, thus leading to different length preference in different models.
switch-GLAT performs better than GLAT on average. In both scenarios, switch-GLAT achieves higher performance than GLAT on average, especially X-to-En directions. These indicate that multilingual training makes it possible to transfer knowledge so that performance of X-to-En direction enhanced due to the augmented data in English on the target side.
4.4.2 RESULTS ON WMT-MANY
Table shows the results on the gathered WMT dataset from 10 language pairs.
switch-GLTT performs better than multilingual baselines in most cases. It is primarily because switch-GLAT is able to produce diverse contextual code-switched translations, which encourage the model to automatically align the generated word vectors and the original ones due their similar context information. Well-aligned representations are beneficial for improving machine translation performance. Notably, Adapter performs much worse than multilingual Transformer on En-Ro and En-Ru directions as the original paper interprets that Adapter do not perform well in low-resource settings (the reverse directions are enhanced by the English data from other language pairs).
switch-GLAT performs slightly worse than GLAT. In this scenario, the failure of switch-GLAT may be owing to the limitation of model capacity. However, -LAth achieves better or comparable performance with GLAT when the model translates from a foreign language to English. One interpretation that all the parallel corpora including English on either source or target side can lead to better parameter estimation of the English decoder (Firat et al.) 2016).
switch-GLAT can enlarge the power of eflicient decoding. The decoding speed in this setting are illustrated in Figure (c), showing that the throughput of our model is much larger than multilingual transformer on 10 directions. It demonstrates that switch-GLAT can enlarge the power of decoding efficiency since all translation directions can be accelerated within single model.
4.5 ABLATION STUDY
The last three lines in Table I and Table 2 show the results of ablation study.
Glancing sampling strategy plays an important role in swich-GLAT. After removing the glanc- ing sampling strategy, the performance drops most. It validates that glancing sampling is critical in training an advanced parallel multilingual translation model, which makes it suitable to generate high-quality code-switched translations.
Code-switch back-translation promotes multilingual translation. Excluding code-switch back- translation or employing code-switched data constructed by dictionary substitution both harms translation performance, of which the former hurts more. It validates that applying code-switched data can significantly increase multilingual translation performance, especially those generated by incorporating context information instead of dictionary replacing.
7

---

Published as a conference paper at ICLR 2022
(a) Transformer
(b) GLAT
(c) M-Transformer
(d) switch-GLAT
Figure 4: Representations learned by (a) Transformer Vaswani et al. 2017), (b) GLAT
QQaa et al.
2020 (c) M- d--ransformer and (d) switch-GLAT, projected to 2D.
Word Induction
Sentence Retrieval
Models
En-De
De-En
En-Fr
Fr-En
Avg
En-De
De-En
En-Fr
Fr-En
Avg
M- Transformer
30.2
31.7
32.7
38.9
33.3
23.6
22.3
24.5
23.8
23.5
CLSS
24.9
25.8
37.5
36.2
31.1
19.6
18.3
26.8
27.9
23.2
Adapter
32.8
34.5
40.1
42.8
37.5
34.6
28.7
26.9
35.2
31.4
MNAT
24.1
23.3
32.4
33.6
28.3
13.8
11..6
18.2
16.3
16.5
switch-GLAT
33.8
36.2
41.9
46.3
39.5
34.8
36.3
36.9
37.2
36.3
w/o glancing
24.3
24.6
32.9
33.7
28.9
18.2
22.8
19.2
20.3
20.1
w/o CSBT
28.2
30.1
31.9
37.8
32.0
17.6
21.4
23.3
25.3
21.8
CCS
30.5
29.7
33.6
35.8
32.4
18.8
22.9
22.7
21.6
21.5
Table Results of quality analyses. Avg means the average accuracy.
ANALYSIS
In this section, will make rigorous analysis reveal the reasons why switch-GLAT can achieve better translation performance than multilingual transformer. All tasks are tested in WMT- EDF scenario.
5.1 VISUALIZATION
There is much smaller gap in the cross-lingual representations learned by our switch-GLAT. To intuitively illustrate how well the cross-lingual representations are aligned, we plot two-dimensional projection of representations learned by four models in Figure Here multilingual models are trained on the merged WMT-EDF dataset, while bilingual ones the English - German part. We use the t-SNE algorithm (van der Maaten Laurens & Hinton 2008 to perform the projection. English and German words gold word pairs from Open Multilingual WordNet (OMW) datasets (Bond & Foster 2013). English words are displayed in blue color and German words in red. We find there is much overlapping between blue and red areas in Figure (d), validating that wwich-GLAT can produce better-aligned cross-lingual representations than other models.
5.2 CROSS-LINGUAL WORD INDUCTION
The cross-lingual representations are aligned well at word level. We assess the cross-lingual word induction performance to see how well the similar words from different languages are close to each other in the learned vector space (Gouws al. 2015). Specifically, golden word pairs are extracted from the OMW datasets (Bond & Foster 2013 resulting in 10, 585 En-De and 12, 511 En- Fr pairs. Top-1 accuracy shown the left half of Table 3 It shows that switch-GLAT significantly outperforms all autoregressive multilingual competitors on average, indicating that cross-lingg lingual representations in model are aligned well word level.
8

---

Published as a conference paper at ICLR 2022
5.3 PARALLEL SENTENCE RETRIEVAL
The cross-lingual representations are aligned well at sentence level. To further assess the quality of the cross-lingual representations sentence level, we employ parallel sentence retrieval (PSR) task. PSR aims to extract parallel sentences from a comparable corpus between two languages. Following the settings of XTREME (Hu et al. 2020), we use the Tatoeba dataset (Artetxe & Schwenk 2019). Cosine similarity is leveraged to search the nearest neighbour and the search accuracy of cach model reported in the right half of Table 3 can be seen that switch- achieves the highest accuracy among all competitors, validating that better contextualized embeddings are obtained in switch-GLAT
as the sentence embeddings are calculated with regarding all tokens in it.
RELATED WORK
6.1 MULTILINGUAL NEURAL MACHINE TRANSLATION
Early works extend the NMT model proposed by Bahdanau et al. (014) to a multilingual version based either on LSTM (Hochreiter & Schmidhuber 1997) or GRU (Cho et al. 2014). [ong al.) (2013) propose a one-to-many translation model (from English into 4 languages) by adding a dedicated decoder per target language. showing improvements over strong single-pal pair baseline. Zoph & Knight (2016) present a multi-source NMT which significantly outperforms single-souree model. Firat et al. (2016) propose many-to-many model involving 10 language pairs by leveraging separate encoder and decoder per language well sharing attention mechanism. Further, Ha et al. (2016) and Johnson et al. (2017) propose to train a multilingual model with a shared encoder-decodler-ateeen network across all languages to perform many-to-many translation. Recently, Transformer (Vaswani ct al. 2017) is applied as the backbone of MNMT by many works. The culti-Distiillaton (Tan et al. 2019 is trained to simultaneously fit the training data and match the outputs of individual bilingual models through knowledge distillation. Baanaa et al.] (2019) present to inject a -pefcfc adaptation layer into a pretrained model keep the language-specific features. Aharoni et al. (2019) propose a multilingual NMT model involving up to 102 languages which achieves good zero-shot performance. All the above MNMT systems are based on autoregressive architectures which follow a sequential generation process and are limited to slow decoding speed.
6.2 NON-AUTORFGGRESSIVE NEURAL MACHINE TRANSLATION
Recently, NAT has attracted much attention due to its ability of parallel decoding. A fully non- autoregressive model is first proposed by Gu et al. (2018). based on which latent variables are introduced to deal with dependencies among target words (Ma al. 2019 Bao et al. 2019 Ran et al.) 2021). Other branches of fully autorrrrreeee systems either try to transfer knowledge from
autoregressive models (Li et al. 20188] Wei et al. 2019 Guo et al. 2020 Sun & Yang 2020), or apply
different training objectives (Libovický & Helel 2018 Shao al. 2020 Ghazvininejad et al. 2020
Qian et al. 2020). Besides, some works build NAT systems with structured prediction to model inter-
dependenctes among output words. Sun al. (019)) propose to incorporate a structured inference module into the won-autoreeressive models. Deng & Rush (2020) present a Markov transformer to perform cascaded decoding. Additionally, serveral semi-autoregressive models are also constructed to refine the outputs with multi-pass iterative decoding (Lee et al. 20188 Gu et al., 2019 Ghazvininejad al., 2019 Kasai al. (2020). All these works are limited to bilingual translation tasks. To our best knowledge, switch-GLAT is the first one to perform multilingual neural machine translation based on a non-autoregressive architecture.
7 CONCLUSION
This paper proposes switch-GLAT, non-autoregressive multilingual neural machine translation model. It incorporates a switch-decoder, which can produce contextualized code-switched sentences and perform code-switch back-translation. Through this process, the multilingual translation perfor- mance and cross pooss-lingual representations can both be improved. Besides, its parallel decoder enables highly efficient inference. Experiments on three datasets suggest that switch-GLAT is superior to multilingual transformer in both effectiveness and efficiency. Further analyses also demonstrate the effectiveness of the learned contextual cross-lingual representations.

---

Published as a conference paper at ICLR 2022
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat, Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3884-3884, 2019.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05013 2019.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 2XXiv:100.04777 2014.
Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. Non-autoregressive transformer by position learning. arXiv preprint arXiv:1911.1667, 2019.
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint 19X.:::999908478.. 2019.
Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, pp. 169-214, 2017.
Francis Bond and Ryan Foster. Linking and extending an open multilingual wordnet. Proceedings of the 51th Annual Meeting of the Association for Compulational Linguistics, pp. 1352-1362, 2013.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation leamning at scale. arXiv preprint wwXiv:191..22/11 2019.
Yuntian Deng and Alexander Rush. Cascaded text generation with markov transformers. Advances in Neural Information Processing Systems, 33, 2020.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 1723-1732, 2015.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 866-875, 2016.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeutlemoye. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.--ICNIIP), pp. 6112-6121, 2019.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non- auooerrtreeeeesivve machine translation. In International Conference on Machine Learning, pp. 3515-3523. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representa- tions without word alignments. In Proceedings of the 32nd International Conference on Machine Learning, pp. 748-756, 2015.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressivve neural machine translation. In International Conference Learning Representations, 2018.
Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein transformer.
arXiv preprint
arXiv: 1905. 1006, 2019.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7839-7846, 2020.
Thanh-Le Ha, Jan Niehues, and Alexander Waibel. Toward multilingual neural machine translation with universal encoder and decoder. arXiv preprint arXiv: 1et:::1111099999 1.04798, 2016.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmun William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating looss-lnguaa generalisation. In International Conference on Machine Learning, pp. 4411-44211 PMLR, 2020.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Google's 's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2017.
Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, and John McCrae. A survey of current datasets for code-switching research. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), pp. 131-141. IEEE, 2020.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine trans- lation with disentangled context transformer. In International Conference on Machine Learning. pp. 5144-5155. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic aortr neural sequence modeling by iterative refinement. arXiv preprint 10xiv:1802.06901, 2018.
Xiaoya Li, Yuxian Meng, Arianna Yuan, Fei Wu, and Jiwei Li. Lava nat: A non-autoregressive transla- tion model with look-around decoding and vocabulary attention. arXiv preprint arXiv:2002.03084, 2020.
Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie- Yan Liu. Hint-based training for non-autoregressive translation. EMNLP-IJCNLP, 2018.
Jindfich Libovický and Jindfich Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3016-302) 2018
Zehui Lin, Xiao Pan, Mingxuan Wang. Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. Pre-training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 2649-2663, 2020.
11

---

Published as a conference paper at ICLR 2022
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNLP-IJCNIPP), pp. 4273-4283, 2019.
Tomas Mikolov, Quoe V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:
2013.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural machine translation. arXiv preprint WiXiv:2008.07905 2020.
Lihua Qian, Yi Zhou, Zaixiang Zheng, Yaoming Zhu, Zehui Lin, Jiangtao Feng, Shanbo Cheng, Lei Li, Mingxuan Wang, and Hao Zhou. The volctrans glat system: Non-autoregressive translation meets wm121. arXiv preprint arXiv:2109. Xi142209..1147,, 2021.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non non-autoregressivv neural machine translation decoding with reordering information. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13727-13735, 2021.
Steven J Rennie, Pierre L Dognin, Xiaodong Cui, and Vaibhava Goel. Annealed dropout trained maxout networks for improved Ivesr. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5181-5185, IEEE, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, 1725, 2016.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of- ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 198-205, 2020.
Zhiqing Sun and Yiming Yang. An em approach to non-autoregresssvve conditional sequence genera- tion. In International Conference on Machine Learning, pp. 9249-9258. PMLR, 2020.
Zhiqing Sun, Zhuohan Li, Haoqing Wang. Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32:3016-3026, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-31 3112, 2014.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv e-prints, pp. 1902, 2019.
van der Maaten Laurens and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, ((Nov))2579-266) 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Xinyi Wang and Graham Neubig. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1823-5828, 2019.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1304-1312, 2019.
Zhen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. Csp: Code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2624-2636, 2020.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-speeific capacity for multilingual translation. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2022
Barret Zoph and Kevin Knight. Multi-source neural translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 30-34, 2016.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1568-1575, 2016.
A APPENDIX
A.1 COMPARING WITH VANILLA BACK-TRANSLATION
We conduct ablation study of using vanilla back-translation Table It shows that though the model using vanilla transsatnll outperforms the one not using any augmentation strategy Saaech-GLATTwwO-SSSS it can be further improved by leveraging code-switch trecssransl,,oon, validating that our proposed code-switch -t-aanslattion favorable.
(wwicc--GLAATwwih--vnillll BT) in
WMT-EFZ
Models
En-Fr
Fr-En
En-Zh
Zh-En
Avg
switch-GLAA GAT
40.54
36.48
19.47
22.55
29.76
with vanilla BT
38.68
35.73
19.23
21.69
28.83
w/o CSBT
35.17
33.65
18.32
20.37
26.87
Table 4: Ablation study of using vanilla back-translation WMT-EFZ.
A.2 APPLYING CSBT TO AUTOREGRESSIVE MODEL
We also apply our proposed code-switch back-translation (CSBT) to multilingual transformer, and the results are shown in Table 5 It shows that CSBT also boosts the performance of multilingual Transformer. Though -.Transformer-with--C model could significantly improve inference efficiency.
performs slightly better than our switch-GLAT. our
WMT-many
Models
En De
De En
En.FF
Fr-En
Fn Ro
Ro Fin
Ru
Ru-Ei
En-Zh
Zh. En
Avg
M-Transformer
23.
29.38
35.19
34.07
34.36
35.26
24.63
29.14
16.63
20.17
28. 19
switch-GLAA
24.18
30.49
39.47
36.30
31.93
32.40
24.16
28.33
16.25
21.23
28.47
w/o CSBT
22.48
27.91
31.62
33.55
29.89
32.41
22.43
26.48
15.33
18.32
26.04
Table 5: Translation performance of applying code-switch back-translation multilingual Trans- former.
A.3 TRANSFER ABILITY
To validate that our model can improve the performance of low-resource languages, we train our switch-GLAY on the subset of TED dataset that is merged from English (En)-Spanish (Es), English- French (Fr) and English-Portuguess, (Pt) pairs. En-Es and En-Fr pairs have about four times as much data as En-Pt pair. The results are shown in Table 6 It can be seen that wwitch-GLAY can significantly improve the performance of low-resource languages (Pt-En) with as much as 19.91 and 0.95 BLEU points respectively compared with GLAT and Transformer.
To further validate the zero-shot ability of our model, we evaluated it (trained on WMT-many) on the test of WMT 2019 German -rench.. Table ( shows that the zero-shot ability of switch-Gl.. GLAT and multilingual Transformer are both poor, but applying self pivot to these two models can achieve good performance.
13

---

Published as a conference paper at ICLR 2022
TED
Models
En-Es
Es-En
En-Fr
Fr-En
En-Pt
Pt-En
Avg
Transformer
34.62
38.52
37.78
36.34
27.79
30.94
34.33
GLAT
34.87
37.89
36.64
36.58
10.08
11.98
28.00
switch-GL SEA--GLLTTTTTTSSSS
34.03
38.27
35.62
36.79
21.18
30.54
32.74
switch-GLAT
34.86
38.69
36.04
37.92
23.58
31.89
33.83
Table 6: Translation performance of TED subsets.
WMT-many
Models
German-French
French-German
Avg
zero-shot 2.86
3.00
2.93
M- Transformer
self pivot
30.38
21.87
26.13
zero-shot 2.09
2.58
2.34
switch-ILAT
pivot 27.86
19.54
23.70
Table 7: Zero-shot performance of our switch-GLAT. Self pivot on German-French denotes first German to English and then English to French. So does French-German.
A.4 RESULTS WITH A BIG MODEL.
As capacity is crucial for multilingual models, we conduct experiments to show how switch-GLAT performance compares as model size grows. We adopt a standard Transformer- large architec- ture (Vaswani et al., (017). which employs 6-layer encoder, 6-layer decoder and 1024 dimension on 16 heads. The results are shown in Table g shows that in a multi-direction scenario, GLAT performs better than wiith--GLTT when trained base model, but increasing model size brings opposite results, demonstrating that model capacity quite critical for multilingual models.
A.S APPENDED VISUALIZATION
We also visualize the representations learned by CLSR since it is also representative multilingual baseline. Again, we find there much more overlapping between blue and red areas in Figure 5 (d), validating that witchh-GLT can produce better-aligned cross-lingual representations than other models.
A.6 HYPER-PARAMETER SELECTION
tune the training step of changing lambda (100000 to max steps with step size 100000), sample number S (100000 to max data size with step size 200000) and initialized dropout rate (0.1 to 0.5
(a) Transformer
(b) GLAT
(c) CLSR
(d) Switch-GLAT
Figure 5: Representations learned by (a) Transformer, (b) GLAT, (c) CLSR and (d) switch-GLAT. projected 2D.
14

---

Published as a conference paper at ICLR 2022
WMT-many
Models
En. De
De: En
En--F
Fr-Ei
En. Ro
Ro: En
En Ru
Ru-Ei
En 2h
7hh Em
Avg
Base models
GLAT
26.09
30.53
38.62
31.41
31.83
32.59
25.42
28.13
21.05 05
22.89 89
29. 20
switch-GLAT
24.18
30.49
39.47
36.30
31.93
32.
24.16
28.33
16.25
21 21.23
28.47
Large models
GLAT
26.56
31.18
39.15
35.03
31.70
31.55
27.01
29.40
19.50
21.61
20.32
SwHch-GLAT
25.69
31.87
41.86
36.33
33.02
34.16
25.41
29.85
18.39
21.82
29.84
Table Results of different model size WMTTmany.
with step size 0. with grid search on the validation set of WMT-EDF, and found that the values described in the paper performed best.
PM
switch-GLAT
switch-GLLATwwo-CCCS
0.0
The longest journey begins with the first step.
The longest journey begins with the first step-
0.1
The longest voyage hegins with the first step.
The longest journey begins with the first étape.
0.3
Le plus long viaje begins with the first step.
The longest journey empezar with the premier pisar.
0.5
Die längste Reise empieza con the first step.
The höchste viajar empezar with das first step.
Table 9: Some examples of generated code-switched sentences respectively by our switch-GLAT and dictionary replacement. French, Spanish and German are represented in red, purple and yellow respectively.
A.7 CASE STUDY
To gain insight on how well model generate code-switched sentences, we provide some examples in Table 9 It shows that constructing code-switched sentences by dictionary replacement may cause errors. For example, pixar is verb while the original word step is noun, and using verb here is not appropriate.
15


Reviews:
[{"review_id": "rq0otnAp5y9", "reviewer": "Reviewer_cdKo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Novel application of NAT to a multilingual setting, with an interesting new strategy for code-switched back-translation.\n- Impressive gains over a fairly broad selection of baselines, including strong auto-regressive models.\n- Potential to be a significant breakthrough in increasing both speed and accuracy of multilingual models.\n\nWeaknesses:\n- The capacity used is quite small for multilingual models. It certainly seems sub-optimal for the WMT-many setting, where both bilingual baselines are starting to pull away from the multilingual models. Since capacity is such a crucial parameter for multilingual models, experiments to show how this new technique compares as models grow would have made the paper more convincing.\n- There are two new things in this paper: the application of NAT to a multilingual setting, and the use of code-switched back-translation. It would have been interesting to tease them apart using NAT architectures other than GLAT: how well do other recent NAT models work on multilingual data; how much do they benefit from code-switched back-translation (since that idea should apply to any NAT model)? The NAT baseline used in the paper is not competitive.\n- There are many hyper-parameters involved in the setup. No information is supplied about how carefully they were tuned, or how they would generalize to new settings.\n- Some problems with clarity, see details below.\n\nDetails:\n- This is hard to understand: “\tSuch efficiency problem is more serious in multilingual setting because serving many translation directions in one model always leads to a higher throughput practically.”\n- This statement should be qualified: “NAT generates translation outputs in parallel (Gu et al., 2018), which leads to significantly faster translation speed.” See: Kasai, Jungo, et al. \"Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation.\" arXiv preprint arXiv:2006.10369 (2020).\n- s3.2 isn’t very clear. The code-switched generation (middle panel in figure 2) seems to have abandoned the GLAT training idea, and just be the standard CMLM algorithm with language tags added to the masks to get code-switched output?\n- s4.2 How exactly does switch-GLAT-w/o-glancing work?\n- 4.4.2 \"switch-GLAT is superior to all multilingual baselines on the average BLEU score.” - not really, it’s basically tied with Adapter and M-Transformer.\n", "summary_of_the_paper": "This paper describes experiments in training a non-autoregressive Transformer (glancing Transformer, or GLAT) in a multilingual MT setting. The use of a non-autoregressive model, plus token-level language tags, enables a code-switching strategy where a source sentence in a given language can generate a target sentence containing words from multiple specified languages. This is used to augment the multilingual training corpus with back-translated data consisting of code-switched synthetic source sentences paired with original monolingual target sentences. A multilingual GLAT model trained with this strategy is shown to outperform many baselines including monolingual GLAT and several autoregressive multilingual models. Ablation studies demonstrate that code-switched back-translation is essential for achieving good performance.\n", "summary_of_the_review": "The first application of non-autoregressive transformers to the important problem of multilingual MT, with a novel idea (code-switched back-translation) specific to that application. Results are generally impressive, but the experiments are missing crucial model-scaling data points, and they miss the opportunity to evaluate code-switched back-translation independent from the GLAT architecture the authors favour.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YcjPSY0XdGd", "reviewer": "Reviewer_ip8L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: \n1. The paper presents a multilingual solution of NAT. The method works with GLAT and provides good insights on how to learn a good cross-lingual representation for NAT.\n\n\n2. The proposed method speeds up the decoding process of the multilingual Transformer (M-Transformer). The generation quality of the proposed method is close to the bilingual autoregressive models (Transformer).\n\n\n3. The author conducted comprehensive analytical studies on the improvement of the proposed method. It is clear to see how well the learned representations are aligned from the cross-lingual perspective. Besides, the word induction and parallel sentence retrieval task support the alignment ability in word-level and sentence-level, respectively.\n\n**Weaknesses**: \n1. The proposed method builds on the top of the GLAT method, which limits its generality. According to the ablation study, the performance dropped a lot without using the glancing mechanism (w/o glancing in Table 1 and 2).\n\n\n2. Although I can understand that how the proposed method works, the description text of Figure 2 seems to be problematic. In Section 3.1, 3.2, the authors said that the code-switch decoder and code-switch back-translation process are illustrated in the left and middle modules, respectively. It is quite confusing that why they do not match the upper annotations shown in Figure 2.\n\n\n3. There may be an error of mathematical symbol in Section 3.3. What does $L_{CS}$ refer to? Should it be $L_{bt}$?\n\n\n4.    Missed one result of representation visualization. I understand that the adapter cannot learn cross-lingual alignment well because it models language-specific capability. Since CLSR can learn the shared pathway and language-specific capability together, why the authors did not choose to compare it in Section 5.1?\n\n\n5.    The explanation that why switch-GLAT performs slightly worse than GLAT does not support by the experiments. The authors claim that it “may be owing to the limitation of model capacity”, will a bigger model or a balanced training set be helpful to alleviate this problem? It should be verified by additional experiments.\n\n\n6. In Section 3.2, the authors said that “In the ﬁrst iteration of $P_M$, the number of mixed languages is set to 1. Afterward, it will be increased to one-third of the total.” There are four languages in WMT-EDF and WMT-EFZ training sets, does it mean that the number of mixed languages is always set to 1 (4*1/3~=1) during the training process? In this case, how the cross-lingual alignment is learned?\n", "summary_of_the_paper": "The paper proposed a novel method for the multilingual non-autoregressive machine translation (NAT) model. The key idea is to modify the decoder module of glancing Transformer (GLAT), making it generate the contextual code-switched translations, then use them to perform code-switch back-translation. To incorporate with the generation of contextual code-switch translations, the authors proposed to add token-level language tags to the first layer and the final output layer at each position. The empirical results show that: 1) the inference speed is faster than the autoregressive baseline model (M-Transformer). 2) the translation performance outperforms the NAT baselines in terms of WMT-EDF, WMT-EFZ benchmarks. Although the performance is slightly worse than the GLAT model on the WMT-Many benchmark, the author claim that the reason is due to the limitation of model capacity. 3) Two cross-lingual experiments and the visualization of representation prove that the proposed method has better cross-lingual capability compared to other multilingual models. ", "summary_of_the_review": "The paper is well-written and contributes a novel approach for multilingual translation in NAT. The design of the code-switch back-translation is reasonable for boosting the performance of multilingual translation and the empirical analysis is convincing. Experiments show that switch-GLAT is better than other NAT/multilingual baselines in limited benchmarks and has close performance compared to bilingual AT baseline.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OswwiStw5VM", "reviewer": "Reviewer_np9D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe authors examined the effectiveness of non-autoregressive NMT on multilingual translation, and obtained encouraging supervised translation performance on several data conditions.\n\n\nWeaknesses:\n\n* Some claims and proposals are not well supported by the current experiments;\n* Experimental details are not always clear to me;\n* Model comparison might be unfair and misleading, and more comparisons are required.\n\n\nComments:\n\n1)\tFirstly, at the core of the adaptation of the GLAT model to multilingual translation is the newly proposed code-switch decoder. This decoder uses token-level target language tag to produce code-switch translations. But the necessity of such design is not tested. Why use code-switch decoder not the vanilla GLAT decoder? A comparison to a vanilla multilingual GLAT should be given.\n2)\tAlso, the authors propose code-switch back-translation to improve model’s performance, but don’t prove why such specific BT is favorable or necessary. What if using the vanilla BT to the NMT model? For example, this can be achieved via random online back-translation proposed in [1].\n3)\tThe authors show that the proposed model improves cross-lingual representation compared to auto-regressive NMT models, but results in Table 3 show that BT delivers a large impact on such improvement without which switch-GLAT performs much worse. Then a question would be: what if applying BT to auto-regressive baselines as well (such as the above mentioned random online BT)?\n4)\t Some experimental details are very unclear! How did you train your baseline models, CLSR and Adapter? What’s the hyperparameters? For CLSR, what’s your budget constraint? How did you set it? In addition, how did you train the MNAT baseline? Did you apply the recently introduced optimization techniques as in [2]?\n5)\tYou mentioned in page 5 that knowledge distillation is used. What’s your teacher model, bilingual Transformer base or multilingual Transformer base? Did you also apply the KD to all baselines (autoregressive and non-autoregressive)? I would assume that the authors used bilingual models for KD and only applied it to the NAT models. In such a case, claiming multilingual NAT “outperforms” multilingual AT is non-convincing.\n6)\tAs explained in the introduction, multilingual NMT benefits low-resource translation, which also enables zero-shot translation. Neither are explored in the current experiments. WMT also offers low-resource languages. Would switch-GLAT benefit these languages in a multilingual setup? What about switch-GLAT’s zero-shot ability?\n\n[1] Zhang et al., 2020 Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation\n[2] Gu et al., 2020 Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade\n", "summary_of_the_paper": "This paper extends non-autoregressive NMT models, particularly the GLAT model, from bilingual translation to multilingual translation. To enable such extension, the authors proposed token-level language tags for the decoder, code-switch decoder paired with back-translation, and also scheduled training between the traditional MLE loss and the BT loss. Experiments on several multilingual tasks based on WMT corpus show that the proposed model obtains better supervised translation quality than its auto-regressive counterparts and delivers better cross-lingual representations.", "summary_of_the_review": "In short, this paper explores an interesting direction: applying non-autoregressive models to multilingual translation. But the experiments suffer from unclarity, unfair comparison and over claims. The paper could be largely improved with more analysis and comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes several innovations for machine translation. The reviewers had several questions about the claims that were made and the authors addressed these and also acknowledged that some of their formulations (e.g. 'better') would need to be qualified. Overall, there are several interesting ideas that have been put together in a sensible way, but the story is not super consistent.

The detailed exchanges between the reviewers are authors are commendable!
Decision: Accept (Poster)