Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?
Authors: Leiyu Pan
Supryadi
Deyi Xiong
Venue: EMNLP 2023 Findings (2023)
Keywords: robustness
transfer learning
multilingual neural machine translation

Abstract: Robustness, the ability of models to maintain performance in the face of perturbations, is critical for developing reliable NLP systems. Recent studies have shown promising results in improving the robustness of models through adversarial training and data augmentation. However, in machine translation, most of these studies have focused on bilingual machine translation with a single translation direction. In this paper, we investigate the transferability of robustness across different languages in multilingual neural machine translation. We propose a robustness transfer analysis protocol and conduct a series of experiments. In particular, we use character-, word-, and multi-level noises to attack the specific translation direction of the multilingual neural machine translation model and evaluate the robustness of other translation directions. Our findings demonstrate that the robustness gained in one translation direction can indeed transfer to other translation directions. Additionally, we empirically find scenarios where robustness to character-level noise and word-level noise is more likely to transfer.

Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?
Leiyu Pan, Supryadi and Deyi Xiong"
College of Intelligence and Computing. Tianjin University, Tianjin, China (lypan, supryadi, dyxiongJetju.edu.c
Abstract
pairs in MNMT are more vulnerable perturba- tions than high-resource language pairs, which is observed our preliminary experiments.
Robustness, the ability of models maintain performance in the face of perturbations, is crit- ical for developing reliable NLP systems. Re- cent studies have shown promising results in improving the robustness of models through ad- versarial training and data augmentation. How- ever, in machine translation, most of these stud- ies have focused on bilingual machine trans- lation with single translation direction. In this paper, we investigate the transferability of robustness across different languages multi- lingual neural machine translation. We propose a robustness transfer analysis protecol and con- duct a series experiments. particular, we character-. word-, and multi-leve noises to attack the specific translation direction of the multilingual neural machine translation model and evaluate the robustness of other translation directions. Our findings demonstrate that the robustness gained in translation direction can indeed transfer to other translation direc- tions. Additionally, we empirically find scenar- ios where robustness character-level noise and word-level noise is more likely to transfer.
In the context of NLP, transfer learning has proven highly effective in various tasks, includ- ing sentiment analysis (Liu al.. 2019), named entity recognition (Lee al., 2018), and question answering (Chung et al., 2018). Existing research demonstrates that transfer learning is also present in MNMT models (Lakew et al., 2018), e.g., knowl- edge transfer from high-resource language transla- tion to low-resource language translation. While previous research has shed light the effective- ness of knowledge transfer within different lan- guage pairs, the robustness transfer across diverse translation directions remains an open question.
In this work, we aim to investigate whether and how robustness is transferable across languages in MNMT. To achieve this purpose, we attack the inputs for particular translation direction and then observe the difference performance of other translation directions on the noise test set. The model that we use a standard multilingual trans- lation model which shares the Transformer encoder- decoder between language pairs. The adversarial attack is conducted in both one-to-many and many- to-one translation tasks. Meanwhile, guages from the same and different language fam- ilies for our experiments. For each language pair, we generate types of noises (character-, word-, and multi-level noise) in the source language.
1 Introduction
select lan-
The research interest in multilingual neural ma- chine translation (MNMT) has been increasing in recent years (Zhang et al., 2020; Yigezu et al., 2021: Costa-jussà et al., 2022). This is partially due to the capability of MNMT in translating mul- tiple language pairs, including both high- and low- resource languages, with a single model, easing the deployment machine translation systems. De- spite this capability, MNMT, like bilingual neu- ral machine translation, is still vulnerable towards domain shift (Müller et al., 2020), perturbations and noises in words and sentences (Belinkov and Bisk, 2018). Furthermore, low-resource language
After attacking the MNMT model with different noises in the source language, we evaluate the trans- lation quality of the language pair that is attacked as well as that of other language pairs that are not attacked in terms of BLEU. We are especially in- terested in the impact of the adversarial attack to an MNMT model on language pairs that are not attacked.
To summarize, the contributions in this paper are:
"Corresponding author.

---

Train
Test
en
character nobs sevee-eellle
en
Multilingual Transformer
Trained Multilingul Transformer
61
on
Cs
en
ds
character
It
It
multilavel
Figure I: The proposed robustness transfer analysis protocol. Slashed cells indicate languages being attacked in black-box way.
We propose robustness transfer analysis pro- tocol to investigate the robustness transfer phe- nomenon in MNMT models.
via similar word insertion and replacement (Alzan- tot et al., 2018) and Differently, white-box attacks are usually gradient-based (Cheng al., 2019).
We have empirically found that the attack with character/word/multileeeee noise increases the robustness of the MNMT model.
Recent research has attempted to address the challenging task of enhancing the robustness MNMT models. Zhou et al. (2021) propose ro- bust optimization methods to tackle the issue of imbalanced data distribution between high- and low- low-resource languages. Grosso et al. (2022) at- tempt to address variations in data distributions across different domains by introducing robust do- main adaptation techniques. Different from the above works, our work focuses on the robustness of MNMT models towards noise and the transfer- ability of robustness across languages in MNMT models.
We have further observed that robustness can transfer across languages. The robustness of the character-level noise is more likely to transfer across related languages while the ro- bustness of word-level noise is more likely to transfer across distant languages.
2 Related Work
MNMT supports one-to-mane many, many-to-one, and many-to-many translation via partial or full param- eter sharing (Ha et al., 2016; Firat et al., 2016; Johnson et al., 2017). Massively MNMT has been explored, where single model supports large number of languages, including high- and low- resource languages (Aharoni al., 2019). In the context of massively MNMT, M2M-100, a non- english-centric MNMT model is proposed (Fan et al., 2021), which is capable of translating 100 language pairs. The successor to M2M-100, NLLB- 200, hugely extends the number of languages to 200 via model scaling (NLLB Team et al., 2022). NMT is vulnerable towards both naturally oc- curring and synthetic noise, e.g., human typos, character swaps (Belinkov and Bisk, 2018). Many works have been done to improve the robustness of NMT, including black-box approaches (Belinkov and Bisk, 2018; Niu et al., 2020; Zhang et al., 2021) and white-box methods (Cheng et al., 2018). Black-box attack methods include both character- and word-level attacks (Karpukhin al., 2019). Character-level attacks are obtained usually by ran- dom character insertion, deletion, replacement and swapping, and word-level attacks can be performed
3 Robustness Transfer Analysis Protocol
In order explore the phenomenon of robustness transfer across translation directions, we propose robustmess transfer analysis protocol. As shown in Figure 1, the protocol consists of phases, a training phase and testing phase.
During the training phase, we conduct black- box attacks on the source side within specific translation direction in multilingual training cor- pus. This entails carrying out character-level, word- level, or multi-level black-box attacks. Specifically, character-level black-bor attacks involve random character insertion, deletion, replacement, and adja- cent character swap operations. Word-level black- box attacks consist of random word swapping, dele- tion, insertion and replacement operations. For word replacement, we use pre-trained word em- beddings to find words semantically related to the target word for replacement. The combination of character-level and word-level black-box methods constitutes the multi-level black-bon approach. It is important to note that while we attuck the source data in a particular translation direction, the target

---

data in the same translation direction remains un- changed, as well as the data in other translation directions. Lastly, the combined multilingual data are fed into the multilingual Transformer for train- ing.
paug library (Ma, 2019). For character-level black- box attacks, we set the proportion of attacked words to the total number of words in each sentence to 0.1. The coverage rate of the four character- attack operations per sentence is 25%. Regarding word- level black-box attacks, the proportion of attacked words in each sentence was also set to 0.1 of all words. Words insertion and substitution operations were language-specific, where pretrained Glo Ve word embeddings were used for English' while pretrained fasttext word embeddings for other lan- guages. Specifically, we first extracted the top-k words with word embeddings similar to the target word, and then randomly chose one word from them as the insertion replacement word. Sim- ilarly, the coverage of the four word-level attack operations per sentence is 25%. In the case of multi-level black-box attacks, character-level and word-level attack operations were combined. The coverage of eight attack operations per sentence was set to 12.5%.
During the testing phase, we conduct a black-box attack on the source side of all translation directions on the multilingual test corpus. To maintain con- sistency in the distribution of the training and test data, we apply the same black-box attack opera- tions to the test corpus as we do on the training corpus. The attacked source data are then fed into the multilingual Transformer, which is trained in the previous phase. We measure the translation per- formance of the translation directions that are not attacked during training. Additionally, compare the translation performance of the model in this training setting with the translation performance of the model under other training settings. For instance, this can involve comparing it to the per- formance of a model trained on a clean corpus on the same test set, or to the performance of a model trained to perform other levels of black-box attacks on the same test set.
Model We used the multilingual transformer wslt de en model provided by the fairseq frame- work (Ott et al., 2019) as our multilingual machine translation model. The hyperparameters we used to train the model are shown in Appendix B.
4 Experiments
With the robustness transfer analysis protocol, we conducted extensive experiments to investigate whether and how robustness transfers across lan- guages in multilingual NMT.
4,2 Is Robustness Transferable across Languages?
In the one-to-many and many-to-one experimental setups, we conducted separate experiments to study robustness transfer phenomenon. The results of the one-to-many experimental setup are shown in Ta- ble 1 while those of the many-to-one experimental setup shown in Table 2 and Table 3.
4.1 Settings
Data We used the publicly available TED TALKS (Qi et al., 2018) and News Commentary as the datasets for training the multilingual translation model. To address the issue that Japanese-Fngle parallel data in News Commentary are limited, we incorporated the KFTT dataset (Neubig, 201 I) as additional resource. Our experiments covered both one-to-many and many-to-one settings with the dataset. In the many-to-one experimental setup, we specifically selected language pairs that include source languages from the same or different lan- guage familics, allowing us to study the robust- ness transfer across different types of language pairs. We used the sentencepiece toolkit (Kudo and Richardson, 2018) to learn shared vocabulary for all language pairs. The vocabulary size was set to 30,000.
One-to-Many Translation
In the one-to-many experimental setup, we intro- duced noise into the English-French translation direction while keeping the corpus of other transla- tion directions unchanged. The results reveal that the performance of the non-attacked translation directions also improves on the test set with the added noise. For example, the model trained with character-level attacks on the English-French trans- lation direction demonstrates better performance on the test set generated by character-level attacks compared to the model trained on clean corpus (9.6 for clean corpus vs. 12.2 for character-level at-
Attack To implement character-level werd-level, and multi-level black-box attack, we utilized the nl-
hatp:///nlp.stanffoo http:///lpsstaaooooooddddd/uuuue http:///ssstteteecccoccc///ppaae

---

Training dataset
Test dataset
en-fr
en-ja
en-ar
en-de
clean corpus
43.1
14.6
17.3
28.7
character-level attack
28.0
9.6
9.3
16.6
clean corpus
word-level attack
31.3
11.3
11.3
18.8
multi-level attack
29.2
10.2
10.2
17.7
clean corpus
42.1
14.8
17.2
28.5
character-level attack
38.8
12.2((27.1%)
13.1(140.9%)
22.2(133.7%)
character-level attack
word-level attack
31.7
11.5
11.2
19.2
multi-level attack
34.4
11.6
12.2
20.5
clean corpus
41.6
14.2
16.7
28.0
character-level attack
30.5
10.2
9.5
17.3
word-level attack
word-level attack
35.9
11.7(13.55)
12.3(18.8%)
20.3(18.0%)
multi-level attack
32.8
10.5
10.5
18.4
clean corpus
42.1
14.9
17.2
28.5
character-level attack
37.4
12.0
12.8
21.8
lulti-level attack
word-level attack
35.8
11.9
12.4
20.8
multi-level attack
36.2
12.1(718.6%)
12.5(122.5%)
21.1(119.2%)
Table 1: BLEU scores for models with different training settings on different test sets using the TED TALKS
dataset in the one-to-many The bolded scores represent the optimal performance on each test set for specific language direction. The attacked translation direction is en-fr.
tack in English-Japanese translation direction). is worth noting that models trained with black-box at- tacks also exhibit enhancements in performance on test sets that differ from the training data distribu- tion. For example, models trained with word-level attacks on the English-Frent Frenchh translation direction perform better on test sets generated by character- level attacks than models trained on clean cor- pus. This be attributed the similarity be- tween the character-level and wevd-level black box attack operations, as both involve insertions, re- placements, deletions and swaps, thus resulting in mutual benefits. It is important to acknowledge that the one-to-many experimental setup implies that the source languages are all the same. Although we only attack the source language for specific trans- lation direction, the encoder might have learned denoised representation of the source language during training. Thus, the difficulty of robustness transfer the one-to-many experimental setup is relatively lower. Instead, it is more valuable to ex- plore the phenomenon of robustness transfer across translation directions in a many-to-on scenario.
ily. We incorporated noise into the French- French-English translation direction while keeping the corpus other translation directions intact. The results the experiments are presented in Table Com- pared to the source languages being the same lan- guage, the source languages being different lan- guages causes the robustness to be more difficult to transfer across translation directions. It can be observed that the percentage increase in BLEU scores in the weebbbeiinntckkeee translation direc- tions is not as high as that in the one-to-many ex- perimental setting. However, is undeniable that the not-being-attacked translation directions still show significant performance improvement on the noisy test set. This is an even stronger evidence that robustness transfer across translation direc- tions. Similarly, observe that the robustness of models trained with the character-level attack can also transfer to handle word-level noise other translation directions, and vice versa.
For improving the reliability of our findings, we changed the attacked source language in the same target language setting. Specifically, we switched the attack dircction from French-English to German-Engliss and compared the performances across different attack direction cases. On the character-level noise test set, as depicted Fig- ure 2, we observe that the model's robustness to character-level noise can still transfer across trans-
Many-to-One Translation: Source Languages from the Same Language Family
In the many-to-one experimental setup, we initially conducted robust transfer experiments where the source languages belong to the same language fam- ily, specifically the Indo-Eur Europpa language fam-

---

Training dataset
Test dataset
fr-en
de-en
es-en
it-en
clean corpus
41.9
36.5
42.2
39.2
character-level attack
24.2
23.8
27.7
25.8
clean corpus
word-level attack
27.1
25.3
29.4
27.0
multi-level attack
24.8
24.0
28.1
25.8
clean corpus
41.5
36.7
42.5
39.2
character-level attack
37.9
26.2(110.
31.2()12.6%)
29.1(112.8%)
character-level attack
word-level attack
29.8
25.7
29.9
27.5
multi-level attack
33.2
25.8
30.2
27.8
clean corpus
40.3
36.6
42.4
38.7
character-level attack
28.1
24.4
28.2
26.0
word-level attack
word-level attack
34.8
26.9(16.3%)
32.3(19.9%%
29.2(18.1%)
multi-level attack
30.8
25.3
29.8
27.4
clean corpus
41.0
36.5
42.
38.9
character-level attack
35.8
26.0
30.4
28.1
lulti-level attack
word-level attack
34.8
27.0
32.1
29.4
multi-level attack
35.1
26.2(19.2%)
30.8(19.6%)
28.2(19.3%)
Table 2: BLEU scores for models with different training settings on different test sets using the TED TALKS dataset in the many-to-one case. The bolded scores represent the optimal performance on each set for specific language direction. The source languages are from the same language family. The attacked translation direction is fr-en.
frin chanddaelee
charsda--enn sttack
Pich vortlond phoct
do mull love ottcc
34
Pron vont levd stoes
on na ottack
36
28
28
26
24-
frinn
de-on
es
fn
frion
de-en
cs c5
it
translation direction
translation direction
Figure 2: Comparison of the performance of attacking the fr-en translation direction and attacking the de-en translation direction the character-level noise test for various training settings.
Figure 3: Comparison of the performance of attacking the fr-en translation direction and attacking the de-en translation direction on the word-level noise test set for various training settings.
lation directions even when the attacked transla- tion direction is altered. However, the magnitude transfer is relatively smaller than that observed with the attack on the French-Englise translation di- rection. We presume that this is due to the greater linguistic distance between German and French, Spanish and Italian languages. They are from dif- ferent language branches. French, Spanish and Italian are the Romance language branch, while German is in the Germanic language branch. On the wevdl noise test set, as shown in Figure 3,
similarly we observe that word-level attacks on the German-English translation direction enhance the performance of other translation directions on the word-level noise test set. This provides ad- ditional evidence of the transferability of model robustness to word-level noise across translation directions. It is also noteworthy that the model robustness transfer degree for both word-level
tacks are essentially the same. The BLEU scores of Spanish-English English and Kalian-Engllan translation directions improve by roughly the same amount.

---

Training dataset
Test dataset
fr-en
ja-en
ar-en
zh-en
clean corpus
41.1
16.8
31.9
21.1
character-level attack
22.5
14.8
22.8
20.3
clean corpus
word-level attack
25.8
11.3
23.1
15.2
multi-level attack
23.4
12.5
22.5
17.3
clean corpus
40.5
17.1
32.6
21.6
character-level attack
37.1
15.6(15.4%)
25.3(+10.7%)
21.0(13.4%)
character-level attack
word-level attack
29.2
12.2
23.8
16.4
multi-level attack
32.4
13.5
24.2
18.0
clean corpus
39.9
17.3
32.3
21.4
character-level attack
28.1
15.5
24.3
20.7
word-level attack
word-level attack
34.3
13.9(123.0%)
26.1(113.0%)
17.8(117.1%)
multi-level attack
30.4
14.4
25.0
19.0
clean corpus
40.3
17.1
32.8
21.6
character-level attack
35.8
15.8
25.4
20.9
multi-level attack
word-level attack
34.0
13.5
26.3
17.9
multi-level attack
34.5
14.4(115.2%)
25.7(114.2%)
19.3()11.6%)
Table 3: BLEU scores for models with different training settings on different test sets using the TED TALKS dataset in the many-to-one case. The bolded scores represent the optimal performance on each set for specific language direction. The source languages are from different language families. The attacked translation direction is fr-en.
Many-fo-One Translation: Source Languages from Different Language Families
same language family was used for analysis. Firstly, selected a seed sentence with the same mean- ing for each source language. Next, we generated four character-level noisy sentences for each souree language's 's seed sentence using the character-level black-box attack method. These four sentences correspond to four character-level attack opera- tions, including random character insertion, ran- dom character deletion, random character replace- ment, and random adjacent character swapping. Subsequently,
In the many-to-one experimental setup, we con- ducted robustness transfer experiments where the source languages are of different language families. In particular, French belongs to the Indo-European family, Japanese is in the Altaic family, Arabic comes from the Semitic family and Chincse is in the Sino-Tibetan family. We attacked the French- English translation direction, keeping the corpus in the other directions unchanged. Results are pre- sented in Table 3. We observe that the model's robustness to noise at all levels can also transfer across translation directions within this experimen- tal setup. On the other hand, the robustness of the models to both character-level and word-level noise can also be transferred to each other and si- multaneously across translation directions. It is noteworthy that the degree of robustness transfer across translation directions does not decrease sig- nificantly despite decrease in source language similarity, which is a surprising finding.
combined the four noisy tences with the original seed sentences without noise from each souree language. We chose two models for comparison: one trained on the clean corpus and the other trained on character-level attacks on the French-English translation direc- tion. We then fed the prepared sentences into each of these models. Figure 4 visualizes dimension- reduced representations of the encoder outputs of these models using PCA. We can observe that the representations of noisy sentences learned by the model trained on a clean corpus is more dis- persed, whereas those Icarned by the model trained with character-level attacks on the French- Fanch-Enggiih translation direction is more compact. This indi- cates that training with black-box attacks enhances the model's 's resilience to interference and reduces the susceptibility to biases caused by minor pertur-
Many-to-One Translation: In-depth Analysis and Visualization
Furthermore. we delved into the reasons behind the transfer of robustness across translation direc- tions. An example an experimental setup with many-to-om and the source languages from the

---

desn deecorpa
menicc yhar
duan metioce phore
Figure 4: PCA-based Cemension-reducee representa- tions of the encoder outputs of sentences with character- level noise. The encoders are from a model trained on a clean corpus and a model trained on character-level attacks respectively.
Figure 5: PCA-based dimension-reduueed representa- tions of the encoder outputs of sentences with word- level noise. The encoders are from model trained on clean corpus and model trained on word-level attacks respectively.
bations in the input. Moreover, the models trained with character-level attacks are more aligned in terms of noisy semantic representations across dif- ferent translation directions. This alignment could be crucial factor contributing to the transfer of robustness across translation directions. Through- the model training process, the model needs learn the correspondence between the source language and the target language. The noise gen- erated by the black-box attacks compels the model to capture alignment information semantic units to better handle the noise. This semantic alignment assists the model in effectively dealing with noisy data in other translation directions.
of the noise semantic representations throughout training facilitates the transfer of model robustness towards noise across translation directions. the end we conducted additional experiments to investigate the effect of shared encoder on ro- bustness transfer. The results and conclusions are presented in the Appendix A.
4.3 Robustness to Character-Level Noise Tends Transfer across Related Languages
We further investigated the robustness transfer lated to character-level noise across languages. As depicted in Table 2, evident that the model trained with character-level attacks on the French- English translation direction exhibits greater en- hancement in BLEU scores on the character-level noise test set compared to the model trained with word-level attacks tested on the word-level noise test set. This finding suggests a higher likelihood of robustness transfer for character-level noise across languages within this experimental setup.
Similarly, we employed word-level black-box attack methods to generate four word-level noisy sentences for each source language's seed sentence. Each of these four sentences corresponds to four word-level attack operations, including random word swapping, random word deletion, insertion and replacement operations using word embed- dings with words similar to the target word. We fed them into models trained on the clean corpus, as well as models trained on word-level attacks for specific translation directions. Figure 5 dis- plays dimension-reduce representations of the en- coder outputs of these two models using PCA. The same conclusion can be observed for the represen- tations of word-level noisy sentences as for those of character-level noisy sentences. This further substantiates the presence of robustness transfer in multilingual NMT. The continuous alignment
Let's ss revisit the results in the experimental set- tings involving many-to-one scenarios where the source languages are from different language fam- ilies, depicted in Table 3. We observe that the previously proposed conclusion does not hold in these cases. We hypothesize that the conclusion is applicable only related languages, e.g., lan- guages from the same language family. We can also validate the applicability of this conclusion using the results obtained from our one-to-many

---

DE-EN
sentence without noise reference translation
Wir hatten Angst, aber wir wollten trotzdem zur Schule gehen.
We were scared, but still, school was where we wanted to be.
sentence with character-level noise hypothesis translation (model trained with character-level attacks)
Wir thatten Angst, aber wir woJliten trotzdem Schule gehen.
We fear that, but we still wanted to go school.
senience with werd-level noise
Wir hatten Angst, aber gehürte wollten trotzdem wirkungssgeschichte Schule gehen.
hypothesis translation
We were scared, but wanted to go to real school anyway.
(model trained with word-level attacks)
Table 4: Translation examples of related languages (from German to English).
To provide more intuitive illustration of this finding, case study has been conducted. Table presents translation examples from German English. It is evident that when the source and target languages are closely related, inputting sen- tences with character-level noise into the model trained with character-level attacks yields superior translation quality compared to inputting sentences with word-level noise into the model trained with word-level attacks.
Test set with character-levil lereleel-lleeeeeeeee nabi
Test set with word evel nosse
BLEU Inpoeepeee
8-
direction
We speculate that the reason for this may be due to the close sharing of commonality in vocab- ulary, grammar structures, and language features among related languages. The model can capture character-level similarities between different lan- guages, making it easy for the model's robustness against character-level noise to transfer between related languages.
Figure BLEU improvements on the test set with character//word-levee noise for models trained on character-/word-leewe attacks on the French-Englisg translation direction over models trained on the clean corpus. The dataset used is News Commentary. The source languages are from the same language family.
experiments, as presented in Table 1. When the source language is English for all translation direc- tions, which increases the similarity in translation directions, the experimental outcomes also indi- cate a greater likelihood robustness transfer for character-level noise across languages.
4.4 Robustness to Word-Level Noise Tends to Transfer across Distant Languages
On the other hand, also further investigated the transfer of robustness towards word-level noise across languages. Table can observed that the models trained with word-level attacks on the French-Engliän translation direction gain a greater improvement in BLEU on the word-level noise test set in the experimental setup where the source languages from different language families. Ad- ditionally, we also note that models trained with word-level attacks outperform models trained with character-level attacks on the not-being- attacked translation directions the multi-level noise test set. This suggests that, in this experimental setup, the robustness of the model towards word-level noise is more likely to transfer to other translation directions than towards character-level noisc.
The aforementioned experimental conclusions are drawn from experiments conducted on the TED TALKS dataset. In order to ensure the reliability of the findings, we also conducted experiments on the News Commentary dataset. These experiments were carried out in a many-to-one setting where all source languages are from the same language family. During training, we attacked the French- English translation direction. Results are shown in Figure 6. It can be observed that the model trained with character-level attacks exhibits simi- larly larger improvement in BLEU on the character- level noise test set. This further emphasizes that the robustness of the model towards character-level noise is more likely to transfer across related lan- guages.
We have observed similar experimental phe- nomenon in cases where the source languages are in the same language family. When the attacked

---

ZH-EE R i2 A P [K] #
sentence without noise reference translation
T T バ
Now, to explain this need to tell you a very disturbing fact.
sentence with character-level noise
3 T 1 7
मे off 1K it A Hit A1 *
hypothesis translation (model trained with character-leves atlacks)
To explain this, I'm going to talk about everything that's disturbing.
sentence with word-level noise
i il
Hi #
hypothesis translation
To explain this, I'm going tell you one thing that's very troubling.
(model trained with word-level attacks)
Table 5: Translation examples of distant languages (from Chinese English).
evidence that the robustness of the model towards word-level noise is more likely to transfer across dissimilar translation directions. It is worth not- ing that the transfer degree may be slightly weaker in Appanese-Engliie translation direction it is different domain of data from those of other trans- lation directions. Nevertheless, robustness is still able to transfer across languages despite domain shift.
12
with
tren
ja transtation oirection
ar
2hh oh
We also provide examples of zh-en translation direction to demonstrate this observation in Table 5. It can be seen that the model trained on word- level noise attack behaves better when dealing with word-level noise.
Figure 7: BLEU improvements on the test set with character/wword-levw noise for models trained on character//word-level attacks on the Englih translation direction over models trained on the clean corpus. The dataset used News Commentary and KFTT. The source languages are from different lan- guage families.
We assume that the reason for this that in the case of distant languages, compared to character- level alignment, the model may be more inclined to perform semantic alignment at the word level. Therefore, robustness to word-level noise is more likely to transfer across languages.
translation direction is German- Engiiih,, the robust- ness of the model towards word-level noise is more likely to transfer across translation directions. This observation leads us to speculate that the robust- ness of the model to word-level noise is more likely to transfer across distant languages, which is sup- ported by the fact that German is in different language branch from the source languages of the other three translation directions.
Conclusions
In this paper, we have conducted extensive and in-depth investigation on robustness transfer in mul- tilingual neural machine translation models with the proposed robustness transfer analysis protocol. Our empirical findings demonstrate that robustness can indeed transfer across languages. Furthermore, we find that robustness towards character-level noise is more likely to transfer across related lan- guage while robustness towards word-level noise is more likely transfer across distant languages.
To further validate this, we also conducted exper- iments on the News Commentary dataset. In partic- ular, the KFTT dataset was chosen as the training data for the Eeranese-Englisee translation direction to compensate for the lack of Eapaaese--ngliing par- allel data in the News Commentary dataset. Results are shown in Figure 7. We observe that, apart from the attacked translation direction, the BLEU im- provement on the word-level noise test set is greater than that on the character-level noise test set for all other translation directions. This provides stronger
Limitations
Findings in this paper are derived from empirical experiments while underlying reasons for these findings are yet to be further investigated. In addi- tion, noises in this study are synthetic. Naturally occurring noise encompasses a broader range

---

variations, necessitating further investigation into the transfer patterns of robustness. Finally, other numerous factors potentially influence the transfer of robustness across translation directions. This paper focuses on a subset of such factors, including language distance and noise types, leaving room for the exploration of additional influential factors in the future.
translation. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 1756-1766 176, Mel- bourne, Australia. Association for Computational Linguistics.
Yu-An Chung, Hung- Yi Lee, and James Glass, 2018. Supervised and unsupervised transfer learning for question answering. In Proceedings of the 2018 Con- ference of the North American Chupter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long Papers), pages New Orleans. Louisiana, Association for Computational Linguistics.
Ethies Statement
This study adheres to the ethical guidelines set forth by our institution and follows the principles out- lined in the ACM Code of Ethics and Professional Conduct. All datasets used in our experiments are publicly available.
Marta R. Costa-jussà, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. 2022. Interpreting gender bias in neu- ral machine tramslation: Multilingual architecture matters. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11): 1 1855-11863.
Acknowledgments
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky. Sergey Edunov, Michael Auli, and Ar- mand Joulin. 2021. Beyond english-centrie multilin- gual machine translation. Journal of Machine Learn- ing Research, 22(107)::1-45
The present research was supported by the Nat- ural Science Foundation of Xinjiang Uygur Au- tonomous Region (No. 2022D01D43) and the Key Research and Development Program of Yunnan Province (No. A08203..0800044.. We would like to thank the anonymous reviewers for their insight- ful comments.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi- way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 866-875, San Diego, California. Association for Computational Linguistics.
References
Roce Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Volume (Long and Short Papers). pages 3874-384 3884, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Mathieu Grosso, Alexis Mathey, Pirashanth Ratnamo- gan, William Vanhuffel, and Michael Fotso. 2022. Robust domain adaptation for pre-trained multilin- gual neural machine translation models. In Proceed- ings of the Massively Multilingual Natural Language Understanding Workshop (MMNLU-22), pages 1-11, Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial ex- amples. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896.
Thanh-Le Ha, Jan Niehues, and Alex Waibel. 2016. Toward multilingual neural machine translation with universal encoder and decoder. In Proceedings of the 13th International Conference on Spoken Language Translation, Seattle, Washington D.C. International Workshop on Spoken Language Translation.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine transla- tion. In International Conference on Learning Rep- resentations.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wa, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg. Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly ad- versarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 4324-4333, Florence, Italy, Associa- tion for Computational Linguistics.
Yong Cheng. Zhaopeng Tu, Fandong Meng. Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic

---

noise improves robustness to natural noise in machine translation. In Proceedings of the Sth Workshop on Noisy User-generated Text (W-NUT 2019), pages 42- 47.
Safyyaa Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations) pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece:
A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Surafel M. Lakew, Aliia Erofceva, Mattco Negri, Mar- cello Federico, and Marco Turchi. 2018. Transfer learning multilingual neural machine translation with dynamic vocabulary. In Proceedings of the 15th International Conference Spoken Language Trans- lation, pages 54-61, 61, Brussels. International Confer- ence on Spoken Language Translation.
Ye Qi. Devendra Sachan, Matthieu Felix, Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neu- ral machine translation? Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Short Papers). pages 529-535, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Proceedings of the 2018
Ji Young Lee, Franck Demoncourt, and Peter Szolovits. 2018. Transfer learning for named-entity recognition with neural networks. In Proceedings of the Eleventh Language Resources
Mesay Gemeda Yigezu, Michael Melese Woldeyohan- nis, and Atnafu Lambebo Tonja. 2021. Multilin- gual neural machine translation for low resourced languages: Ometo-english In 2021 International Conference on Information and Communication Tech- nology for Development for Africa (ICT4DA), pages 89-94.
International Conference and Evaluation (LREC 2018), Miyazaki, Japan. Eu- ropean Language Resources Association (ELRA).
Ruijun Liu, Yuqian Shi, Changjiang Ji, and Ming Jia. 2019. A survey of sentiment analysis based on trans- fer learning. IEEE Access, 7:85401-85412.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
Edward Ma. 2019.
NLP augmentation.
Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Associa- tion for Machine Translation in the Americas (Volune 1: Research Track), pages 151-164 Virtual. Associa- tion for Machine Translation in the Americas.
Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. 2021. Crafting adversarial examples for neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1967-1977, Online. Association for Computational Linguistics.
Graham Neubig. 1. The Kyoto free translation task. http://wwww.hhnnrrn.comm///
Xing Niu, Prashant Mathur, Georgiana Dinu, and Yaser Al-Onaizan. 2020. Evaluating robustness to input Pro-
perturbations for neural machine translation. ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8538- 8544, Online. Association for Computational Lin- guistics.
Chunting Zhou, Daniel Levy, Xian Li, Marjan Ghazvininejad, and Graham Neubig. 2021. Distri- butionally robust multilingual machine translation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5664 1664,,677, Online and Punta Cana, Dominican Re- public, Association for Computational Linguistics.
Marta R. Costa-jussà NLLB Team, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,

---

Training dataset
Test dataset
fr-en
de-en
es-en
it-en
clean corpus
41.6
36.4
42.1
38.8
character-level attack
23.7
23.3
26.6
25.2
clean corpus
word-level attack
27.2
24.7
28.5
26.6
multi-level attack
24.8
23.8
27.2
25.2
clean corpus
41.1
36.8
42.3
38.8
character-level attack
37.7
23.0(U1.3%%
26.8(10.7%)
24.9(11.2%)
character-level attack
word-level attack
29.3
24.1
29.1
26.4
multi-level attack
32.8
23.3
27.4
25.3
clean corpus
40.8
36.8
42.2
38.8
character-level attack
28.1
23.7
27.1
24.8
word-level attack
word-level attack
35.3
24.8(10.4%)
29.4(13.2%)
26.3(41.1%) 1%)
multi-level attack
31.2
23.7
27.7
25.0
clean corpus
40.5
36.7
42.2
38.5
character-level attack
35.8
23.6
27.0
24.6
multi-level attack
word-level attack
34.4
24.8
29.2
26.2
multi-level attack
34.7
24.0(10.8%)
27.7(11.8%)
25. 25.1((0.4%)
Table 6: BLEU scores for models with different training settings on different test using the TED TALKS dataset in the many-to-one case. The holded scores represent the optimal performance on each test set for specific language direction. The source languages are from the same language family. The attacked translation direction is fr-en. Encoders not shared across source languages.
Additional Experiments: No Shared Encoder across Source Languages
We conducted additional experiments demon- strate the significance of the shared encoder in en- abling robustness transfer. The experimental results are shown in Table 4. It can be observed that when source languages do not share the same encoder, robustness barely transfers across translation direc- tions.
B Model Hyperparameters
Model Hyperparameters
Value
adam-betas
(0.9, 0.98)
0.0005
warmup-updates
4000
warmup-init-Ir
le-07
label-smoothing
0.1
dropout
0.3
weight-decay
0.0001
max-tokens
4000
update-freq
8
Table Model Hyperparameters.


Reviews:
[{"review_id": "bQzh03o4Kf", "reviewer": "Reviewer_xN8k", "summary": "", "questions": "A. What's the potential application of this paper? Could it provide insights regarding how to enhance the translation robustness of LLMs?\n\nB. Could you provide some cases in section 4.3 and 4.4?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and suggestions, and they are exceedingly helpful for us to improve our paper. We will earnestly incorporate them into the revised version of our paper. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n> Although adversarial attacks enhance the robustness of MNMT models, the sacrifice on clean test set seems not trivial (e.g., 1~2 BLEU score on the en-fr translation task in Table1). It may be helpful to also investigate the trade-off between robustness and clean-performances of MNMT models.\n\nThank you for this constructive comment. We do notice that when attacking a certain translation direction, the performance of that direction on the clean corpus decreases significantly, and we will continue our research to improve the robustness of the model while maintaining its performance on the clean corpus. However, we are also delighted to note that the noise attack on one translation direction does not seem to significantly affect the performance of other translation directions on the clean corpus, and even the performance of other translation directions on both the clean corpus and the noise test set is improved at the same time.\n\n> What's the potential application of this paper? Could it provide insights regarding how to enhance the translation robustness of LLMs?\n\nThe main motivation of our work is to try to find the robustness patterns and mechanisms of multilingual machine translation, which finally helps the development of efficient approaches to improving translation robustness across multiple languages in multilingual machine translation. We find that when training a multilingual neural machine translation (MNMT) model using a black-box attack method for a single translation direction, translation robustness is also improved for all or related translation directions. With this finding, it is hoped that it will (1) improve the robustness of translations with optimized attacking efforts and (2) improve translation robustness of languages (e.g., low-resource languages) where noisy datasets are not easy to collect with noisy data from languages where those datasets are available. We believe that our work has implications for the translation robustness of multilingual LLMs as they also share the same model for all languages. For example, where we might be able to improve robustness across all languages by adding high-quality noise data only to high-resource languages. We'd like to provide such new results on LLMs in the next version following your great suggestion.\n\n> Could you provide some cases in section 4.3 and 4.4?\n\nIn order to illustrate the issue more intuitively, a case study has been conducted following your suggestion. First, some examples of de-en translation directions are provided in the following Table 1 to demonstrate the conclusion in Section 4.3 that the robustness of the model to character-level noise transfers more easily between translation directions than robustness to word-level noise among related languages. It can be observed from the data in Table 1 that the models trained on character-level noise attacks perform better when dealing with character-level noise.\n\n|                                                              | DE-EN                                                        |\n| ------------------------------------------------------------ | ------------------------------------------------------------ |\n| sentence without noise                                       | Wir hatten Angst, aber wir wollten trotzdem zur Schule gehen. |\n| reference translation                                        | We were scared, but still, school was where we wanted to be. |\n| sentence with character-level noise                          | Wir thatten Angst, aber wir woJllten trotzdem zur Schule gehen. |\n| translation results of feeding sentences with character-level noise into the model trained with the character-level noise attack | We fear that, but we still wanted to go to school.           |\n| sentence with word-level noise                               | Wir hatten Angst, aber gehörte wollten trotzdem wirkungsgeschichte Schule gehen. |\n| translation results of feeding sentences with word-level noise into the model trained with the word-level noise attack | We were scared, but we wanted to go to real school anyway .  |\n\nTable 1. Examples of translations with source languages being related languages\n\nIn addition, we also provide examples of zh-en translation direction to demonstrate the conclusions in Section 4.4. Between languages that are distant from each other, the robustness of the model to word-level noise is more likely to transfer between translation directions than the robustness to character-level noise. It can be observed from the data in Table 2 that in this case the model trained on word-level noise attack behaves better when dealing with word-level noise.\n\n|                                                              | ZH-EN                                                        |\n| ------------------------------------------------------------ | ------------------------------------------------------------ |\n| sentence without noise                                       | 为了 解释 这个 ，我 要说 一件 很 让 人 困扰 的 事实          |\n| reference translation                                        | Now , to explain this , I need to tell you a very disturbing fact . |\n| sentence with character-level noise                          | 为了 解释 这个 ， 我 要说 t件 很 让 人 困扰 的 事            |\n| translation results of feeding sentences with character-level noise into the model trained with the character-level noise attack | To explain this, I'm going to talk about everything that's disturbing . |\n| sentence with word-level noise                               | 为了 解释 这个 ， 要说 一件 让 很 人 困扰 事实               |\n| translation results of feeding sentences with word-level noise into the model trained with the word-level noise attack | To explain this, I'm going to tell you one thing that's very troubling. |\n\nTable 2. Examples of translations with source languages being distant languages", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is clear.\n2. The proposed experiments illustrate some insightful findings.", "reasons_to_reject": "1. Although adversarial attacks enhance the robustness of MNMT models, the sacrifice on clean test set seems not trivial (e.g., 1~2 BLEU score on the en-fr translation task in Table1). It may be helpful to also investigate the trade-off between robustness and clean-performances of MNMT models.\n\n2. Some cases would be helpful when explaining the findings in section 4.3 and 4.4, i.e., why character-level attack is more useful for enhancing MNMT model robustness in related languages?", "questions_to_authors": "A. What's the potential application of this paper? Could it provide insights regarding how to enhance the translation robustness of LLMs?\n\nB. Could you provide some cases in section 4.3 and 4.4?", "paper_topic_and_main_contributions": "This paper systematically investigates whether the robustness of multilingual machine translation (MNMT) models can be transferred between different languages, via crafting different adversarial examples towards various language pairs to attack MNMT models. Experimental results demonstrate that robustness of MNMT models can transfer across different languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MVgXDzRDUa", "reviewer": "Reviewer_USAS", "summary": "", "questions": "- Line 288, what does \"In order to avoid experimental chance,\" mean? Please change a more precise expression.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your valuable insights and suggestions. We have taken meticulous care to integrate these insights into the revised version of our paper. For ease of your reference, we present your comments first, followed by our detailed responses addressing each point raised.\n\n> Many figures in the paper lead to similar conclusions (e.g., Figure 2 vs. Figure 3, Figure 4 vs. Figure 5), which take up lots of space in the paper and make the paper look verbose. It would be better to move some of them to the appendix.\n\nThank you for this feedback. Figure 3 and Figure 5 visualize results when attacking the MNMT model with word-level attack. As the results are relatively similar to the character-level attack, we will consider moving Figure 3 and Figure 5 to the appendix following your suggestion.\n\n> Figure 4,5 look a little confusing and need more detailed descriptions.\n\nFigures 4 and 5 visualize the noise representations of a model trained on a clean corpus and a model trained on noise attack. The primary objective is to assess the alignment of noise representations across different languages. Taking Figure 4 as an example, we started by selecting semantically equivalent seed sentences for each source language to maximize the similarity of their semantic representations. Subsequently, based on these seed sentences, we employed the four character-level black-box attack methods in our paper to generate four distinct character-level noisy sentences, each corresponding to a specific attack method. Next, we combined noisy sentences with clean sentences and fed them into two models: one trained on the clean corpus and the other trained on character-level attacks on the French-English translation direction. Subsequently, we applied PCA dimensionality reduction to the outputs of the encoders of both models and visualize the reduced representations of the noisy input. Figure 5 essentially involves a similar procedure. The difference lies in the fact that the four noisy sentences correspond to four types of word-level attack methods, using models trained on clean corpus and models trained with word-level attacks in the specific direction. We'll make these more clear in the next version with detailed descriptions as you suggested.\n\n> For most translation directions, the robustness transferred from EN-FR is not significant enough (still lags behind the results on clean corpus by ≈10 BLEU scores). Although the authors use growth rates to make the benefits look more obvious in Tables 1,2,3, the transfer effects are still limited.\n\nThank you for this question. The approach employed in this paper merely illustrates the transferability of robustness across different translation directions, leaving substantial room for further improvements in the effectiveness of robustness transfer that can be achieved. In our upcoming research, we will delve deeper into enhancing the transfer of robustness across various translation directions.\n\n> I would like to see more results based on multilingual pre-trained models, e.g., mBART (Liu et al., 2020), mRASP2 (Pan et al., 2021).\n\nOur findings have implications for machine translation with either solely multilingual LLMs or models enhanced by multilingual LLMs. We'd like to extend our framework to investigate the robustness transfer across languages with multilingual LLMs in our future work.\n\n> Line 288, what does \"In order to avoid experimental chance,\" mean? Please change a more precise expression.\n\nWe would like to explain about additional experiments for supporting our findings. We will change this sentence into \"For improving the reliability of our findings\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors conduct a series of analyses on the core research question, from shallow to deep, providing sufficient (some even redundant) evidence for their claim.\n- The findings may motivate more future work on the robustness of the multilingual encoders (known to be vulnerable as QE models).", "reasons_to_reject": "- Many figures in the paper lead to similar conclusions (e.g., Figure 2 vs. Figure 3, Figure 4 vs. Figure 5), which take up lots of space in the paper and make the paper look verbose. It would be better to move some of them to the appendix.\n- Figure 4,5 look a little confusing and need more detailed descriptions.\n- For most translation directions, the robustness transferred from EN-FR is not significant enough (still lags behind the results on clean corpus by ≈10 BLEU scores). Although the authors use growth rates to make the benefits look more obvious in Tables 1,2,3, the transfer effects are still limited.\n- I would like to see more results based on multilingual pre-trained models, e.g., mBART (Liu et al., 2020), mRASP2 (Pan et al., 2021).\n\n\n\n1. Liu et al. Multilingual Denoising Pre-training for Neural Machine Translation. TACL 2020.\n2. Pan et al. Contrastive Learning for Many-to-many Multilingual Neural Machine Translation. ACL 2021.", "questions_to_authors": "- Line 288, what does \"In order to avoid experimental chance,\" mean? Please change a more precise expression.", "paper_topic_and_main_contributions": "This paper aims to study whether robustness can be transferred across different languages in multilingual neural machine translation (MNMT) and the answer may be yes. The authors design several attack experiments for one-to-many and many-to-one MNMT models and obtain consistent conclusions. These findings are somewhat helpful for understanding the robustness transfer in MNMT models and multilingual encoder models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 075, interested -> interested in\n2. Line 472, not-being-atttacked -> not-being-attacked\n"}, "extra_text": {}}, {"review_id": "fOpWWQsTbr", "reviewer": "Reviewer_SHa5", "summary": "", "questions": "A. Do you have a hypothesis that explains why the degree of robustness transfer does not seem to rely on source language similarity?\n\nB. How do you combine the pretrained GloVe / fasttext word embeddings that are used for word-level attacks with your model, which has a different vocabulary than GloVe? Do you only use the subset which matches? If so, what percentage of the total vocab is this?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your insightful comments and suggestions. We sincerely appreciate your time in reviewing the paper, and our point-to-point responses to your comments are given below.\n\n> Apart from the observation that the authors investigate robustness in a multilingual setting, and previous work considered a bilingual setting, it does not become clear how this work fits in with the broader field. Black-box methods and white-box methods are mentioned, but not explained. No attempt is made to put the experimental results in context with earlier findings of the field. The paper of the quality would improve with some more discussion: which findings are in agreement with earlier work? Which disagree, and what are the possible reasons? From the paper, it's also not clear how the types of noise that you consider are (dis)similar to earlier work.\n\nThank you for the advice. Previous studies have shown that black-box attacks can indeed improve the robustness of machine translation models[1], which is consistent with our observations. In addition, the attacks used in this paper include three types, including character-level[2], word-level, and multi-level attacks. The character-level attacks include random character insertion, deletion, replacement, and adjacent character swap. The word-level attacks include random word swapping, deletion[3], and similar word insertion and replacement[4]. The multi-level attacks combine the first two attack methods. We will include the above in more details in the Related Work section of our paper. For comparison to previous findings, it might be difficult as we conduct experiments in the context of multilingual machine translation and try to find robustness patterns across multiple translation directions while previous works focus on bilingual machine translation where there is only one translation direction. This makes a direct comparison to previous findings lack of foundation. \n\n[1]Belinkov Y, Bisk Y. Synthetic and Natural Noise Both Break Neural Machine Translation[C]//International Conference on Learning Representations. 2018.\n\n[2]Karpukhin V, Levy O, Eisenstein J, et al. Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation[C]//Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019). 2019: 42-47.\n\n[3]Wei J, Zou K. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019: 6382-6388.\n\n[4]Alzantot M, Sharma Y, Elgohary A, et al. Generating Natural Language Adversarial Examples[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 2890-2896.\n\n> Most of the findings are unsurprising, which is of course not necessarily an issue. Unfortunately, for the surprising findings, there is no further investigation. For instance, comparing Table 2 and Table 3, the authors find that the degree of robustness transfer does not seem to rely on source language similarity, which is a highly surprising finding. I think the paper would benefit a lot from digging into this deeper.\n\nThanks for this comment. We would like to point out, however, that the statement that robustness transfer does not depend on the similarity of the source language is not actually the view expressed in this paper. It will be specified in a later answer to your question.\n\n> It is not clear what data is used exactly. Authors mention TED Talks and News Commentary, but it is not clear which experiments use which dataset. Maybe they are combined? It is also not clear which test set is used. TED Talks, News Commentary, or the combination thereof?\n\nFor 4.2 experiments, we used the TED Talks dataset for training. In 4.3 experiments, we used News Commentary dataset. For 4.4 experiments, we used News Commentary dataset for fr-en, ar-en, and zh-en translation directions and KFTT dataset for ja-en translation. We will explain more clearly about datasets used in the Experiment Settings part of our revised paper.\n\n> It is unclear how to interpret \"bold\" in the Table; this is not mentioned anywhere and it's not as simple as \"best score\".\n\nIn fact, the bolded scores represent the optimal performance on each test dataset within a specific language direction. For instance, as shown in Table 2 for the fr-en translation, the highest BLEU score for the clean corpus dataset is attained when the model is trained with the corresponding clean corpus, resulting in the bolded value of 41.9. Further elucidation regarding the significance of these bolded scores in the table will be meticulously provided in our revised paper.\n\n> The differences in Figure 4 and Figure 5 are very subtle, and based on these plots I would not conclude that representations learned using noisy data is more compact compared to representations trained on clean data. It would be easy to quantify this claim by calculating representational differences, which could be used as additional evidence.\n\nThank you for your feedback. We have chosen models trained with character-level noise attacks and models trained on clean corpus. We input sentences containing character-level noise into the models and measure the representation differences in the encoder outputs by calculating the Euclidean distance between the output vectors of the model's encoder for each pair of languages. The average representation difference between each pair of languages is then computed. The same procedure is applied for word-level noise to calculate the Euclidean distance between the noise representations. After conducting these calculations, the resulting data is presented in the table below.\n\n|                             | models trained with character-level noise attacks | models trained with word-level noise attacks | models trained with clean data |\n| --------------------------- | ------------------------------------------------- | -------------------------------------------- | ------------------------------ |\n| character-level noise input | 109.4941                                          | -                                            | 113.6249                       |\n| word-level noise input      | -                                                 | 113.3749                                     | 115.7528                       |\n\nFrom the table, we can conclude that when we input noise sentences from each language into the corresponding noise attack trained models, the noise representations produced by different languages are indeed more compact.\n\n> The BLEU signature is missing.\n\nWe use sacrebleu to calculate the BLEU score. The details will be added to the revised paper.\n\n> Do you have a hypothesis that explains why the degree of robustness transfer does not seem to rely on source language similarity?\n\nThank you for your question. However, we would like to clarify a potential misunderstanding of the point our work aims to convey. Our findings do not suggest that robustness transfer is independent of the similarity between source languages. On the contrary, a comparison between Figure 6 and Figure 7 in the paper reveals that both character-level and word-level noise robustness exhibit greater improvements under experimental settings where the source languages belong to the same language family.\n\nOur intended message is that, among similar languages, the robustness of a model to character-level noise is more likely to be transferred between translation directions than the robustness to word-level noise. Conversely, across distant language pairs, the model's robustness to word-level noise is more likely to be transferred between translation directions than the robustness to character-level noise. This phenomenon could be attributed to the degree of alignment between languages. Similar languages might share a greater portion of vocabulary, grammatical structures, and linguistic traits, enabling the model to capture similarities at the character level across different languages. Hence, the model's robustness to character-level noise might be more readily transferrable between languages. In the case of distant languages with greater dissimilarity, the model might rely more on word-level alignment to fulfill translation tasks. Consequently, the robustness to word-level noise is more prone to transfer across distant languages.\n\n> How do you combine the pretrained GloVe / fasttext word embeddings that are used for word-level attacks with your model, which has a different vocabulary than GloVe? Do you only use the subset which matches? If so, what percentage of the total vocab is this?\n\nThank you for this question. Word embeddings are employed to identify semantically related words for word-level attack (replacement or insertion). GloVe word embeddings are utilized for English texts, whereas fastText word embeddings specific to each language are employed for texts in other languages. It's worth noting that these embeddings are not combined. The aforementioned word replacement and insertion attack method is implemented based on the nlpaug library. We first find the top-k words with similar word embeddings as the target word and then randomly select 1 word from them as the word for replacement or insertion.\n\n> Line 078: \"For summary\" -> \"To summarize\"\n>\n> Figures 2 and 3 are hard to read, and I would argue that this type of Figure is not well-suited for the message you're trying to convey. A Table would be a better fit. If you insist on a Figure, I would go for a radar chart.\n>\n> Line 214: it would be a nice courtesy to the reader to summarise model hyperparameters here (or in Appendix).\n\nThank you for pointing out these issues. We also agree that radar chart is better to represent the data rather than line plot. We will consider changing the figure into radar chart. Regarding model hyperparameters, we basically used the default parameters of the multilingual_transformer_iwslt_de_en model provided by the fairseq framework. The parameters that need to be additionally pointed out to be set in the training script are shown in the following table. These hyperparameter settings will be added to the appendix of the next version.\n\n| hyperparameter      | value |\n| --------------- | ----------- |\n| adam-betas      | (0.9, 0.98) |\n| lr              | 0.0005      |\n| warmup-updates  | 4000        |\n| warmup-init-lr  | 1e-07       |\n| label-smoothing | 0.1         |\n| dropout         | 0.3         |\n| weight-decay    | 0.0001      |\n| max-tokens      | 4000        |\n| update-freq     | 8           |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The investigation of robustness transferability has not been explored before.\n* The results clearly indicate that robustness is transferable. This finding can be used by follow-up work that aims to improve MNMT.", "reasons_to_reject": "* The connection to related work is insufficient.\n\nApart from the observation that the authors investigate robustness in a multilingual setting, and previous work considered a bilingual setting, it does not become clear how this work fits in with the broader field. Black-box methods and white-box methods are mentioned, but not explained. No attempt is made to put the experimental results in context with earlier findings of the field. The paper of the quality would improve with some more discussion: which findings are in agreement with earlier work? Which disagree, and what are the possible reasons? From the paper, it's also not clear how the types of noise that you consider are (dis)similar to earlier work.\n\n* The findings lack depth\n\nMost of the findings are unsurprising, which is of course not necessarily an issue. Unfortunately, for the surprising findings, there is no further investigation. For instance, comparing Table 2 and Table 3, the authors find that the degree of robustness transfer does not seem to rely on source language similarity, which is a highly surprising finding. I think the paper would benefit a lot from digging into this deeper.\n\n* Important information is missing. Presentation could use improvement.\n\nIt is not clear what data is used exactly. Authors mention TED Talks and News Commentary, but it is not clear which experiments use which dataset. Maybe they are combined? It is also not clear which test set is used. TED Talks, News Commentary, or the combination thereof?\n\nIt is unclear how to interpret \"bold\" in the Table; this is not mentioned anywhere and it's not as simple as \"best score\".\n\nThe differences in Figure 4 and Figure 5 are very subtle, and based on these plots I would not conclude that representations learned using noisy data is more compact compared to representations trained on clean data. It would be easy to quantify this claim by calculating representational differences, which could be used as additional evidence.\n\nThe BLEU signature is missing.", "questions_to_authors": "A. Do you have a hypothesis that explains why the degree of robustness transfer does not seem to rely on source language similarity?\n\nB. How do you combine the pretrained GloVe / fasttext word embeddings that are used for word-level attacks with your model, which has a different vocabulary than GloVe? Do you only use the subset which matches? If so, what percentage of the total vocab is this?", "paper_topic_and_main_contributions": "This analysis paper investigates robustness in the context of multilingual machine translation. In particular, the authors investigate the effect of different types of noise (insertion of character-level noise / word-level noise / the combination thereof) on model performance.\n\nThe results seem to indicate that (unsurprisingly) robustness transfers, where the magnitude of transfer is relatively high for one-to-many models and relatively low for many-to-one models. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "* Line 094 (full parameter-sharing): Johnson et al. Google’s multilingual neural machine translation system: enabling zero-shot translation. In: TACL 2017"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 078: \"For summary\" -> \"To summarize\"\n* Figures 2 and 3 are hard to read, and I would argue that this type of Figure is not well-suited for the message you're trying to convey. A Table would be a better fit. If you insist on a Figure, I would go for a radar chart.\n* Line 214: it would be a nice courtesy to the reader to summarise model hyperparameters here (or in Appendix)."}}]

Author rebuttal:

Meta-review: This paper studies whether robustness to input noise, as conferred by “attacks” involving noised input during training, can be transferred across languages in multilingual NMT. Experimental results indicate that this can indeed be the case, and further analyses characterize the conditions under which it occurs.

Reviewers felt that the study was novel and well motivated, and were mostly convinced by the findings and analyses indicating positive results. They noted some downsides, including a tradeoff between robustness and quality (which is however confined to the language under attack), some problems with the clarity and conciseness of the presentation, and a lack of further analysis to account for some of the experimental results.

This is a solid contribution. As the authors point out, a big benefit to their findings is that robustness to a high-resource language can be transferred to a lower-resource language without needing to explicitly attack the data in that language, which might be problematic for various reasons (including the potential to reduce output quality that might already be marginal). The paper has the potential to generate interesting follow-up work, in particular to try to solve the robustness / quality tradeoff for languages subjected to attacks.
Decision: Accept-Findings