Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance
Authors: Shaomu Tan
Christof Monz
Venue: EMNLP 2023 Main (2023)
Keywords: Zero-shot Machine Translation
Multilingual Machine Translation
TL;DR: An comprehensive analysis of zero-shot performance in Multilingual Machine Translation

Abstract: Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing but often suffers from poor zero-shot (ZS) translation qualities. While prior work has explored the causes of overall low zero-shot translation qualities, our work introduces a fresh perspective: the presence of significant variations in zero-shot performance. This suggests that MNMT does not uniformly exhibit poor zero-shot capability; instead, certain translation directions yield reasonable results. Through systematic experimentation, spanning 1,560 language directions across 40 languages, we identify three key factors contributing to high variations in ZS NMT performance: 1) target-side translation quality, 2) vocabulary overlap, and 3) linguistic properties. Our findings highlight that the target side translation quality is the most influential factor, with vocabulary overlap consistently impacting zero-shot capabilities. Additionally, linguistic properties, such as language family and writing system, play a role, particularly with smaller models. Furthermore, we suggest that the off-target issue is a symptom of inadequate performance, emphasizing that zero-shot translation challenges extend beyond addressing the off-target problem. To support future research, we release the data and models as a benchmark for the study of ZS NMT.

Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance
Shaomu Tan
Christof Monz
Language Technology Lab University of Amsterdam (s. tan, m.monz) nl
Abstract
1.0
En-centric
Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing but of- len suffers from poor zero-shot (7S) translation qualities. While prior work has explored the causes of overall low zero-shot translation qual- ities, work introduces fresh perspective: the presence of significant variations in zero- shot performance. This suggests that MNMT does not uniformly exhibit poor zero-shot pability; instead, certain translation directions yield reasonable results. Through systematic experimentation, spanning 1,560 language di- rections across 40 languages, we identify three key factors contributing to high variations in 7S NMT performance: 1) target-side translation quality, 2) vocabulary overlap, and 3) linguistic properties. Our findings highlight that the tar- get side translation quality is the most influen- tial factor, with vocabulary overlap consistently impacting zero-shot capabilities. Additionally, linguistic properties, such as language family and writing system, play role, particularly with smaller models. Furthermore, we suggest that the off tar-ta issue is symptom of inade- quate performance, emphasizing that zero-shot translation challenges extend beyond address- ing the tar-tt problem. To support future research, we release the data and models benchmark for the study of 7S NMT.!
Z5 condition-1
25-condition-2
Perertage
0.2
0.0
20
50
Spbleuu
Figure 1: SpBleu Distribution for English-centric and zero-shot directions. The y-axis denotes the percent- age of performance surpassing the value on the x-axis. Condition- 1 refers resource-nie rich ZS directions where the source and target language share linguistic proper- ties, while condition-2 includes other ZS directions.
In contrast, our work introduces a fresh per- spective within zero-shot NMT: the presence of high variations in the zero-shot performance. This phenomenon suggests that certain ZS translation directions can closely match supervised counter- parts, while others exhibit substantial performance gaps. We recognize this phenomenon holds for both English-centric systems and systems going be- yond English-centric data (e.g.: m2m-100 100 models), This raises the question: which factors contribute to variations in the zero-shot translation quality?
1 Introduction
Multilingual Neural Machine Translation (MNMT) has shown great potential in transferring knowledge across languages, but often struggles achieve satisfactory performance in zero-shot (ZS) transla- tions. Prior efforts have been focused on investigal- ing causes of overall poor zero-shot performance, such as the impact of model capacity (Zhang et al., 2020). initialization (Chen et al., 2022; Gu et al., 2019; Tang et al., 2021; Wang et al., 2021), and how model forgets language labels can affect ZS perfor- mance (Wu al., 2021; Raganato et al., 2021).
Through systematic and comprehensive exper- imentation involving 1,560 language directions spanning 40 languages, we identify three key fac- tors contributing to pronounced variations in zero- shot NMT performance: 1) target side translation capacity, 2) vocabulary overlap, and 3) linguistic properties. More importantly, eral regardless of the resource level of languages and hold consistently across various evaluation met- rics, spanning word, sub-word, character, and rep- resentation levels. Drawing from our findings, we offer potential insights to enhance zero-shot NMT.
findings are gen-

---

Our investigation begins by assessing the impact of supervised translation capability on scot performance variations. This achieved by decom- posing the unseen ZS direction (Sre->Tgt) into two seen supervised directions using pivot language English. For instance, Ste-Tgt can be decom- posed into two seen supervised directions: Sre-+En (source side translation) and En->Tgt (target side translation) for English-centric systems. Our find- ings show the target side translation quality signifi- cantly impacts ZS performance and can explain the variations the most. Surprisingly, the source side translation quality has a very limited impact.
2 Related Work
2.1 MNMT corpus
Current MNMT studies mainly utilize two types of datasets: English-centric (Arivazhagan et al., 2019b; Yang et al., 2021), which is by far the most common approach, and, more rarely, non-English- centric (Fan et al., 2021; Costa-jussà et al., 2022). English-centric datasets rely on bitext where En- glish is either the source or target language, while Non-English-centrie ones sample from all available language pairs, resulting in much larger number of non-English directions. For instance, OPUS100 dataset contains 100 languages with 99 language pairs for training, while the M2M-100 dataset com- prises 100 languages covering 1,100 language pairs (2,200 translation directions).
Moreover, our analysis demonstrates the substan- tial impact of linguistic properties, i.e., language family and writing system, in elucidating the varia- tions in zero-shot performance. Figure highlights this conclusion by showing the much stronger ZS performance of resource-rich ZS directions with similar linguistic properties compared to other di- rections. Intriguingly, our investigation also shows that the impacts of linguistic properties are more pronounced for smaller models. This suggests that larger models place less reliance on linguistic simi- larity when engaging in ZS translation, expanding more insights upon prior research about the impact of model capacity on ZS NMT (Zhang et al., 2020). Furthermore, we found the language pair with higher vocabulary overlap consistently yields bet- ter zero-shot capabilities, suggesting a promising future aspect to improve the ZS NMT. In addition, while Zhang et al. (2020) asserts the off-target issue as primary cause that impairs the zero-shot capa- bility, we conclude that the off-target issue is more likely to be a symptom of poor zero-shot transla- tion qualities rather than the root cause. This is evident by small off-ttareet rates (smaller than 5%) not necessarily resulting in high Z.S capabilities.
Non-Ennglih--eentiic approaches primarily en- hance the translation quality in non-English di- rections by incorporating additional data. How- ever, constructing such datasets is challenging due to data scarcity in non-English language pairs. In addition, training becomes more computation- ally expensive as more data included compared to English-cente centric approaches. Furthermore, Fan et al. (2021) demonstrates that English-centric ap- proaches can match performance to non- -on-English- centric settings in supervised directions using only 26% of the entire data collection. This suggests that the data boost between non-English pairs has limited impact on the supervised directions, high- lighting the promise of improving the zero-shot performance of English-centric systems. There- fore, in this study, we focus on the English-centric setting as it offers a practical solution by avoid- ing extensive data collection efforts for numerous language pairs.
2.2 Understanding Zero-shot NMT
Lastly, we argue that prior rescarch on zero-shot NMT limited by focusing only on the 1% of all possible ZS combinations (Aharoni et al.. 2019; Pan et al., 2021; Tang et al., 2021) or prioritizing resource-rich language pairs (Yang et al., 2021; Ra- ganato et al., 2021; Chen et al., 2022; Zhang et al., 2020; Qu and Watanabe, 2022). To overcome these limitations, create the EC40 MNMT dataset for training purposes and utilize multi-parallel test sets for fair and comprehensive evaluations. Our dataset is the first of its kind considering real-world data distribution and diverse linguistic characteristics, serving as a benchmark to study ZS NMT.
Previous studies have primarily focused on inves- tigating the main causes of overall poor zero-shot performance, such as the impact of model capacity, initialization, and the off-target issue on zero-shot translation. Zhang et al. (2020) found that increas- ing the modeling capacity improves zero-shot trans- lation and enhances overall robustness. In addition, Wu et al. (2021) shows the same MNMT system with different language tag strategies performs sig- nificantly different on zero-shot directions while retaining the same performance on supervised di- rections. Furthermore, Gu et al. (2019); Tang et al. (2021); Wang et al. (2021) suggest model initializa-

---

tion impacts zero-shot translation quality, Lastly, Gu et al. (2019) demonstrates MNMT systems are likely to capture spurious correlations and indicates this tendency can result in poor zero-shot perfor- mance. This is also reflected in the work indicating MNMT models are prone to forget language labels (Wu et al., 2021; Raganato et al., 2021).
To this end, we introduce the EC40 dataset to ad- dress these limitations. The EC40 dataset uses and expands OPUS (Tiedemann, 2012) and consists of over 66 million bilingual sentences, encompass- ing 40 non-English languages from five language families with diverse writing systems. To main- tain consistency and make further analysis more comprchensive, carefully balanced the dataset across resources and languages by strictly maintain- ing each resouree group containing five language families and each family consists of eight represen- tative languages.
Attention is also paid to examining the relation- ship between off-target translation and zero-shot performance. Off-target translation refers to the issue where an MNMT model incorrectly trans- lates into a different language (Arivazhagan et al., 2019a). Zhang et al. (2020) identifies the off- target problem as a significant factor contributing to inferior zero-shot performance. Furthermore, several studies (Gu and Feng, 2022; Pan et al., 2021) have observed zero-shot performance im- provements when the off-target rate drops.
EC40 covers wide spectrum of resouree avail- ability, ranging from High(5M) to Medium( IM), Low(100K), and In total, there are 80 English-centric directions for training and 1,640 directions (including all super- vised and ZS directions) for evaluation. To the best of our knowledge, EC40 is the first of its kind for MNMT, serving as a benchmark to study the zero- shot NMT. For more details, see Appendix A.1. As for evaluation, we specifically chose Ntrex- 128 (Federmann et al., 2022) and Flores-200 (Costa-jusså et al., 2022) as our validation and test datasets, respectively, because of their unique multi- parallel characteristics. We combine the Flores200 dev and devtest sets to create our test set. We do not include any zero-shot pairs in the validation set. These datasets provide multiple parallel trans- lations for the same source text, allowing for more fair evaluation and analysis.
resources.
Our work complements prior studies in two key aspects: 1) Unlike previous analyses that focus on limited zero-shot directions, we examine a broader range of language pairs to gain a more comprehen- sive understanding of zero-shot NMT. 2) We aim investigate the reasons behind the variations in zero-shot performance among different language pairs and provide insights for improving the zero- shot NMT systems across diverse languages.
3 Experiment
3.1 EC40 Dataset
Current MNMT datasets pose significant chal- lenges for analyzing and studying zero-shot trans- lation behavior. We identify key shortcomings in existing datasets: 1) These datasets are limited in the quantity of training sentences. For instance, the OPUS100 (Zhang et al., 2020) dataset covers 100 languages but is capped to maximum of 1 million parallel sentences for any language pair. 2) Datasets like PC32 (Lin et al., 2020) fail ac- curately reflect the real-world distribution of data, with high-resource languages like Frrench and Ger- man disproportionnately represented by 40 million and 4 million sentences, respectively. 3) Linguistic diversity, a critical factor, is often overlooked in datasets such as Europarl (Koehn, 2005) and Mul- GUN (Chen and Eiscle, 2012). 4) Lastly, systematic zero-shot NMT evaluations are rarely found in ex- isting MNMT datasets, either missing entirely or covering less than 1% of possible zero-shot com- binations (Aharoni et al., 2019; Pan et al., 2021; Tang et al., 2021).
3.2 Experimental Setups
Pre- Pre-processing To handle data in various lan- guages and writing systems, we carefully apply data pre-processing before the experiments. Fol- lowing similar steps as prior studies (Fan et al., 2021; Baziotis et al., 2020; Pan et al., 2021), our dataset is first normalized on punctuation and then tokenized by using the Moses tokenizer. In ad- dition, we filtered pairs whose length ratios are over 1.5 and performed de-duplication after all pre-processing steps. All cleaning steps were per- formed on the OPUS corpus, and EC40 was con- structed by sampling from this cleaned dataset.
We then learn 64k joint subword vocabulary us- ing SentencePiece (Kudo and Richardson, 2018). Following Fan et al. (2021); Arivazhagan et al. (2019b), we performed temperature sampling (T 5) for learning SentencePiece subwords to over-
chttps://git http:::///gttuub.cmm//eeeeee

---

Sucrebieu
Chriet
SpBleu
Comet
Fn-+X
X-FFn
Ene-X
78
Ra-xX
X-
Enex
25
En-xX
X-4Fin
En--X
73
En-XX
X
EnneX
78
Averaged Performance
mTg
25.3
49.9
20.5
30.6
78.3
54.7
mBart50
22.7
29.5
26.1
6.6
46.8
53.9
50.3
23.5
29.6
32,6
31.0
80.2
80.1
80.1
58.8
n lTr-e
23.6
28.7
26.1
7.0
47.6
53.3
90.4
25.3
30,5
31.8
31.1
10.1
79.2
79.0
79.1
59,5
Coefficient Variation (CV)
mT-big
0.45
0.39
0.43
0.93
0.24
0.22
0.24
0.52
041
0.37
0.39
0.85
0.11
0.14
0.13
0.21
mBart50
0.48
0.44
0.80
0.25
0.22
0.24
0.48
0.36
0.39
0.74
0.13
0.23
mT large
0.46
0.38
0.43
0.83
0.25
0.23
0.24
0.47
0.41
0.38
0.39
0.77
0.12
0.15
0.13
0.23
Table 1: Average performance scores and coefficient of variation on English-centric and Zero-shot (ZS) directions. The table includes three metrics: Sacrebleu, Chrf++, SpBleu, and Comet. The best performance scores (higher means better) are highlighted in bold depending values before rounding, while the highest CV scores in the coefficient of variation section (higher means more variability) are underlined to highlight high variations.
come possible drawbacks of overrepresenting high- resource languages, which is also aligned with that in the training phase.
While we acknowledge that metrics like Sacre- bleu may have limitations when comparing transla- tion quality across language pairs, we believe that consistent findings across all these metrics provide more reliable and robust evaluation results across languages with diverse linguistic properties. For Comet scores, we evaluate the supported 35/41 lan- guages. As for beam search, we use the beam size of 5 and a length penalty of 1.0.
Models Prior research has suggested that zero- shot performance can be influenced by both model capacity (Zhang al., 2020) and decoder pre- training (Gu et al., 2019; Lin et al., 2020; Wang al., 2021). To provide an extensive analysis, we conducted experiments using three different mod- els: Transformer-big, Transformer-larg (Vaswani et al., 2017) 10, and fine-tuned mBART50. Addition- ally, we evaluated m2m- 100 models directly in our evaluations without any fine-tuning
4 Variations in Zero-Shot NMT
Table presents the overall performance of three models for both English-centric and zero-shot di- rections on four metrics. It is evident that all mod- els exhibit a substantial performance gap between the supervised and zero-shot directions. Specif- ically, for our best model, the zero-shot perfor- mances of Sacrebleu and SpBleu less than one- third compared to their performance in supervised directions, which highlights the challenging nature of zero-shot translation. In addition, compare the results of mT-big and mT-larre, we observe that in- creasing the model size can benefit zero-shot trans- lation, which aligns with previous research (Zhang et al.. 2020). Furthermore, we show that while the mBART50 fine-tuning approach shows superior performance in Sre -En directions, consistently lags behind in En and zero-shot directions.
Training All training and inference in this work use Fairseq (Ott et al., 2019). For Transformer mod- els, we follow Vaswani et al. (2017) using Adam (Kingma and Ba, 2014) optimizer with ,11 0.9, $2 = 0.98, As suggested by Johnson et al. (2017); Wu al. (2021), we prepend target language tags the source side, c.g.: '<2de>' denotes translating into German. Moreover, we follow mBART50 MNMT fine-tuning parammee settings (Tang et al., 2021) in our experiments. More training and model specifications can be found in Appendix A.2.
10-9, warmup steps 4,000.
Evaluation We ensure a comprehensive analy- sis by employing multiple evaluation metrics, aim- ing for a holistic assessment of our experiments. Specifically, we utilize four evaluation metrics across various levels: 1) Chrf++: character level (Popović, 2017). 2) SentencePieceelluu (SpBleu): tokenized sub-word level (Goyal et al., 2022), 3) Sacrebleu: detokenized word level (Post, 2018)).
Does pre-training matter? Gu et al. (2019); Tang et al. (2021); Wang et al. (2021) have shown that pre-trained seq2seq language models can help alleviate the issue of forgetting language IDs of- observed in Transformer models trained from scratch, leading improvements in zero-shot per- formance. However, our results show an interest- ing finding: When the MNMT model size matches that of the pre-trained model, the benefits of pre-
and 4) COMET: representation level (Rei et al., 2020). For the Comet score, we use the wmm22- comet-da model(Rei et al., 2022).

---

Sacrebleu
Chri++
which are determined by the training settings of m2m100. The results in Table 2 yield two impor- tant observations. Firstly, significant performance gaps exist between supervised and zero-shot direc- tions, suggesting that the challenges of zero-shot translation persist even in Non-Engiish-cenrree sys- tems. More importantly, our finding of consider- able variations in zero-shot NMT also holds for non-Engliih--centric systems.
En-gent.
Sup.
7S
En-cent.
Sup.
zs
Averaged Performance
m'T-large
27.7
11.9
6.2
52.5
34.1
23.2
m2m100-419m
24.5
15.7
49.2
39.3
25.8
m2m100-1.26
29.0
18.9
52,9
42.8
29.5
Coeflicient
Variation (CV)
mTT-arge-418m
0.36
0.86
0.96
0.20
0.38
0.50
m2m100-419m
0.45
1.05
0.27
0.36
0.57
m2m100-1.22
0.43
0.57
0.25
0.33
0.52
Table 2: Evaluation of m2m 100 models on our bench- mark. We consider English-centric, supervised and zero- shot directions this table according to m22110. This means many "supervised" directions actually unseen for our m Tla-re system.
Factors in the Zero-Shot Variations
We investigate factors that might contribute to vari- ations in zero-shot directions: 1) English transla- tion capability 2) Vocabulary overlap 3) Linguistic properties 4) Off-target issues. For consistency, use our best model (m'T-large) in the following analyses, unless mentioned otherwise. For simplic- ity, we denote the zero-shot direction as Src >Tgt throughout the following discussion, where Sre and Tgt represent the Source and Target language re- spectively. In this chapter, we present all analysis results using SpBleu and provide results based on other metrics in the Appendix for reference.
training zero-shot NMT become less prominent. This result is consistent for both seen and unseen languages regarding mBart50, see Appendix A.4. Our observation aligns with previous claims that the mBART model weights can be easily washed out when fine-tuning with large-scale data on super- vised directions (Liu et al., 2020; Lee et al., 2022).
Quantifying variation We identify the higher variations that exist in zero-shot translation perfor- mance than supervised directions by measuring the Coefficient of Variation (CV- = A (Everitt and Skrondal, 2010). The CV metric is defined as the ratio of the standard deviation a to the mean of performance, which is more useful than purely using standard deviation when comparing groups with vastly different mean values.
Target
Med
B-Low
Avg
En
37.53
38, 30.00
27:11
18.47
30.50
Source
High
36:32
16.50
17:25
0.45
3.77
11.38
Med
36.26
16.48
17.01
9.79
4.18
11.61
Low
30.97
13:12
13.61
8.70
3.25
9.17
11.07
14:18
14.97
7.72
3.70
10.12
Table 3: Resource-level analysis based on SpBleu (we provide analyses based on other metrics in appendix A.5. I). We include both English-centric (shaded blue) and zero-shot (shaded red) directions. Avg-tAvg de- notes averaged zero-shot SpBleu score.
As shown in Table 1, we find substantially higher CV scores in the zero-shot directions compared to the supervised ones, with an average increase of around 100% across all models and metrics. This observation highlights that zero-shot performance is much more prone to variations compared to the performance of supervised directions. This raises the question: What factors contribute to the signifi- cant variations observed in zero-shot performance?
5.1 English translation capability
We first hypothesize that data size, which is known to play a crucial role in supervised training, may also impact zero-shot capabilities. We categorize data resource levels into four classes and examine their performance among each other as shown in Table 3. English-centre results are also included for comparison. Our findings indicate that the re- source level of the target language has stronger effect on zero-shot translations compared that of the source side. This is evident from the larger drop in zero-shot performance (from 14.18 to 3.70) observed when the target data size decreases, as opposed to the source side (from 1 11.38 to 8.41).
Exploring the Role of Non-English- Sys- tems Training with non-English language pairs has shown promise in improving zero-shot perfor- mance (Fan et al., 2021). To delve deeper into this aspect, we evaluate m2m (00 models directly without further finetuning set because our goal is to investigate whether the high variations in the zero-shot performance phe- nomenon hold for Non-English-cenrric models.
benchmark test
Our analysis consists of English-centrr (54), supervised (546), and zhro-shot (860) directions,

---

Setup To further quantify this observation, we conducted correlation and regression analyses, see Table 4, following Lauscher et al. (2020) to analyze the effect of data size and English-centrie perfor- mance. Specifically, we calculate both Pearson and Spearman for correlation, and use Mcan absolute error (MAE) and root mean square error (RMSE) for regression. We use data size after temperature sampling in Src->En and En En->Tgt directions, as well as the corresponding performances as features.
and modify it as follows: Overlap Verd Vral
Vryal where Vsre and Vrgt represent the vocabularies of the source (Sre) and target (Tgt) languages, respec- tively, This measurement quantifies the proportion of subwords in the target language that is shared with the source language the zero-shot transla- tion direction.
Correlation En->Tgt performance
25
R=-0. 1.80
30
Results Three key observations can be made from these results: 1) The factors on the target side consistently exhibit stronger correlations with the zero-shot performance, reinforcing our conclu- sions from the resource-level analysis in Table 3. 2) The English-centric performance feature demon- strates greater R- R-square compared to the data size. This conclusion can guide future work to augment out-of-English translation qualities, we further ex- pand it in the section 6. 3) We also observe that the correlation alone does not provide a compre- hensive explanation for the underlying variations observed in zero-shot performance by visualizing the correlation (Figure 2).
25
perto siroaieee spalee
20
50
Lnt spbleu score
Figure 2: Correlation between En->Tgt SpBleu and zero-shot (All++Tgt) SpBleu. Each faded blue point denotes the performance of a single zero-shot direction based on SpBleu. R=0.68 indicates the Pearson correla- tion coefficient (see Table 4 for more details).
Setup We first investigate the correlation be- tween the vocabulary overlap and zero-shot perfor- mance. As noted by Philippy et al. (2023), vocabu- lary overlap alone is often considered insufficient to fully explain transfer in multilingual systems. We share this view, particularly in the context of multilingual translation, where relying solely on vocabulary overlap to predict zero-shot translation quality presents challenges. Hence, we incorporate the extent of the vocabulary overlap factor into our regression analysis with English translation perfor- mance in section 5.1.
Features
Data-size' Sre_size
Enscentric perf. En->Tg
Metrics
Tgt_size
Sre->En
Pearson
0.16
0.53
0.41
0.68
Comelation
Spearman
0.41
R-square
32.54%
61.34%
Regression
MAF
5.47
4,70
RMSE
6.62
5.59
Table 4: Analysis of zero-shot performance considering data size and English-centric performance based on Sp- Bleu. Data-size is after the temperature sampling it represents the actual size of the training set.
5.2 The importance of Vocabulary Overlap
Results As shown in Table 5. The results indicate that considering the degree of overlap between the source and target languages further contributes explaining the variations in zero-shot performance. Importantly, this pattern holds true across differ- ent model capacities, and it shows more consistent results than linguistic features such as script and family. We recognize this conclusion can promote to future investigation on how to improve the zero- shot NMT. For example, encouraging greater cross- lingual transfer via better vocabulary sharing by leveraging multilingual dictionaries, or implicitly learning multilingual word alignments via multi- source translation, we leave them to future work.
Vocabulary overlap between languages is often con- sidered to measure potential linguistic connections such as word order (Tran and Bisazza, 2019), mak- ing more basic measure of similarity in sur- face forms compared to other linguistic measure- ments such as language family and typology dis- tance (Philippy et al., 2023). Stap et al. (2023) also identifies vocabulary overlap as one of the most important predictors for cross-lingual transfer in MNMT. In our study, we investigate the impact of vocabulary sharing on zero-shot NMT.
We build upon the mcasurement of vocabulary overlap proposed by Wang and Neubig (2019)

---

ID
Features
Rquuue
MAE
RMSE
resource levels. Additionally, we observe that the language family feature exhibits relatively weaker significance as shown in Appendix A.5.3. To fur- ther quantify the effect of these linguistic properties on ZS NMT, we conduct a regression analysis, see Table 5. Our findings highlight their critical roles in explaining the variations of zero-shot performance.
mT-big
1
En performance 45.63%
4.03
4.88
2
Vocab-Sim
63.42%
3.70
4.77
3
Linguistic-features 81.17%
3.42
4.37
mT--laree
4
En_ perfforaacce 61.34%
4.70
5.59
5
4 Vocab-Sim 79.75%
3.76
4.84
6
Linguistic-feature 81.75%
3.67
4.75
Tgt resource
cLow
Low
Mod
High
Table 5: Prediction of Zero-Shot Performance using En- Centric performance, vocabulary overlap, and linguistic properties. We present the result based on SpBleu in this table.
If Sre
and Tgi in the same I Language Family
No
2.12
4.82
9.77
9.43
Yes
3.14
7.69"
13.16
12.88" *
If Src
and Tgt use the same Writing System
No
1.58
3.97
9.31
8.67
Yes
3.21
8.13"
11.71
12.68
5.3 The impact of Linguistic Properties
represents p <= 0.05
Previous work on cross- cross-lingual transfer of NLU tasks, such as NER and POS tagging, highlights that transfer is more successful for languages with high lexical overlap and typological similarity (Pires et al., 2019) and when languages more syntactically or phonologically similar (Lauscher et al., 2020). In the context of multilingual machine translation, although limited to only validating four ZS directions, Aharoni et al. (2019) has em- pirically demonstrated that the zero-shot capability between close language pairs can benefit more than distant ones when incorporating more languages. Accordingly, we further extend this line of inves- tigation by examining linguistic factors that may impact zero-shot performance MNMT. We mea- sure the role of two representative linguistic prop- ertics, namely language family and writing system, in determining the zero-shot performance. The spe- cific information on linguistic properties of each language can be found in Appendix A.I.
Table 6: The impact of linguistic properties on zero-shot performance (we mTT-large and SpBleu here for example). We conduct Welch's 's t-test to validate if one group significantly better than another. The detailed table, including the impact of X resource, can be found in Appendix A.5.3.
Furthermore, our analysis reveals interesting findings regarding the effect of linguistic properties considering the model size. As shown in Table 5, we observed that the contribution of linguistic fea- tures is more pronounced for the smaller model, i.c., mT-big. While the larger model tends to place more emphasis on English-centric performance. This suggests that smaller models are more susceptible to the influence of linguistic features, potentially due to their limited capacity and generalization ability. In contrast, larger models exhibit better generalization capabilities, allowing them to rely less on specific linguistic properties.
Setup To examine the impact of linguistic prop- erties on zero-shot performance, we specifically evaluate the performance in cases where: 1) source and target language belong to the same or different family and 2) source and target language use the same or different writing system. This direct com- parison allows to assess how linguistic similari- ties between languages influence the effectiveness of zero-shot translation.
5.4 The role of Off- Translations
Previous work (Gu and Feng, 2022; Pan et al.. 2021) have demonstrated consistent trend, where stronger MNMT systems generally exhibit lower off-target rates and simultaneously achieve better zero-shot BLEU scores. To further investigate this, we analyze the relationship between off-target rates and different levels of zero-shot performance.
Setup We adopt the off-target rate measurement from Yang et al. (2021) and Costa-jussà et al. (2022) using fasttext (Joulin et al., 2016) to detect if sentence is translated into the correct language.
Results To provide fine-grained analysis, examine this phenomenon across different resource levels for the target languages, as shown in Table 6. The results reveal a significant increase in zero-shot performance when the source and target languages share the same writing system, irrespective of the
While Zhang et al. (2020) identifies the tff-target issue as a crucial factor that contributes
Results

---

ysis reveals that, especially with smaller models, distant pairs exhibit notably lower zero-shot per- formance compared to closer ones. Consequently, enhancing zero-shot performance for distant pairs is key strategy to improve overall capability. An unexplored avenue for consideration involves multi- source training (Sun et al., 2022) using Romaniza- tion (Amrhein and Sennrich, 2020), with gradual reduction in the impact of Romanized language.
R= 0.08
30
(Spbee
25
20
23 pe 25
15
10
Offitarget rate
Encourage cross-lingual transfer via vocabulary sharing Furthermore, we have consistently ob- served that vocabulary overlap plays significant role in explaining zero-shot variation. Encouraging greater cross-lingual transfer and knowledge shar- ing via better vocabulary sharing has the potential to enhance zero-shot translations. Previous studies (Wu and Monz, 2023; Maurya al., 2023) have shown promising results improving multilingual translations by augmenting multilingual vocabulary sharing. Additionally, cross-lingual pre-training methods utilizing multi-paralle dictionaries have demonstrated improvements in word alignment and translation quality (Ji al., 2020; Pan et al., 2021).
Figure 3: Correlation between ofr-target rate and zero- shot performance (SpBleu). R represents the Spearman correlation coefficient. We focus on directions where the off-target rate is considerably low (less than 5%). Results based on other metrics can be found in A.5.5.
poor zero-shot results. However, our analysis. as illustrated in Figure 3, suggests that the reasons for poor zero-shot performance go beyond just the off-target issue. Even when the off-target rate is very low, e.g., less than 5% of sentences being off- target, we still observe wide variation in zero-shot performance, ranging from very poor (0.1 SpBleu) to relatively good (34.6 SpBleu) scores. Based on these findings, conclude that the off-target issue is more likely to be a symptom of poor zero-shot translation rather than the root cause. This em- phasizes that translating into the correct language cannot guarantee decent performance.
Conclusion
In this work, we introduce a fresh perspective within zero-shot NMT: the presence of high varia- tions in the zero-shot performance. We recognize that our investigation of high variations in zero-shot performance adds an important layer of insight the discourse surrounding zero-shot NMT, which provides an additional perspective than understand- ing the root causes of overall poor performance in zero-shot scenarios.
6 From Causes to Potential Remedies
In this section, summarize our findings and offer insights, building upon the previous observations.
Enhance target side translation We identified that the quality of target side translation (En-+Tgh strongly influences the overall zero-shot perfor- mance in an English-centric system. To this end, future research should explore more reliable ap- proaches enhance the target side translation capa- bility. One practical promising direction is the use of trecsstranlttiin (Sennrich et al., 2016) focus- ing more on improving Engllshh translations. Similarly, approaches like multilingual regulariza- tion, sampling, and denoising are worth exploring to boost the zero translation directions.
We first show the target side translation qual- ity significantly impacts zero-shot performance the most while the source side has a limited im- pact. Furthermore, we conclude higher vocabulary overlap consistently yields better zero-shot perfor- mance, indicating a promising futture aspect to im- prove zero-shot NMT. Moreover, linguistic features can significantly affect ZS variations in the perfor- mance, especially for smaller models. Additionally, we emphasize that zero-shot translation challenges extend beyond addressing the off aff-target problem.
We release the EC-40 MNMT dataset and model checkpoints for future studies, which serve as a benchmark study zero-shot NMT. In the future. we aim to investigate zero-shot NMT from other views, such as analyzing the discrepancy on the representation level.
Focus more on distant pairs We recognize that distant language pairs constitute a significant per-
centage of all zero-shot directions, with 61% in- volving different scripts and 81 % involving differ- ent language families in our evaluations. Our anal-

---

Limitations
References
Roee Aharoni. Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- rional Länguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884.
One limitation of this study is the over- representation of Indo-European languages in our dataset, including languages in Germanic, Ro- mance, and Slavic sub-families. This could result in non-Indo- Europeann languages being less repre- sentative in our analysis. Additionally, due to data scarcity, we were only able to include million parallel sentences for high-resource languages. As a result, the difference in data size between high and medium-resource languages is relatively small compared to the difference between medium and low-resource languages (which is ten times). To address these limitations, we plan to expand the EC40 dataset in the future, incorporating more non- Indo-European languages and increasing the data size for high-resource languages.
Chantal Amrhein and Rico Sennrich. 2020. On roman- ization for model transfer between scripts in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2461-2460
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. 2019a. The missing ingredient in zero- shot neural machine translation. arXiv preprint 1rXiv:1933.0701..
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019b. Massively multilingual neu- ral machine translation in the wild: Findings and challenges. arXiv preprint arXiv: 1 1907.05019.
Broader Impact
We collected new multilingual dataset (EC40) from OPUS, which holds potential implications for the field of multilingual machine translation. The EC40 dataset encompasses a diverse range of lan- guages and language pairs, offering researchers and developers an expanded pool of data for training and evaluating translation models. It also serves as a benchmark for enabling fair comparisons and fos- tering advancements in multilingual translation re- search. Recognizing the inherent risks of mistrans- lation in machine translation data, we have made ef- forts to prioritize the incorporation of high-quality data, such as the MultiUN (Chen and Eisele, 2012) dataset (translated documents from the United Na- tions), to enhance the accuracy and reliability of the EC40 dataset. By sharing the EC40 dataset, we aim to contribute to the promotion of transparency and responsible use of machine translation data, fa- cilitating collaboration and driving further progress in multilingual machine translation research.
Christos Baziotis, Barry Haddow, and Alexandra Birch, 2020. Language model prior for low-resoure neural machine translation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7622-7634.
Guanhua Chen, Shuming Ma. Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang. and Furu Wei. 2022. Towards making the most of cross- lingull transfer for zero-shot neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 142-
Chen and Andreas Fisele. 2012. Multiun v2: Un documents with multilingual alignments. In LREC. pages 2500-2504.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centere machine translation. arXiv preprint arXiv:2207.0460
A Aeknowledgments
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online (v2020.3). Zenodo.
This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project number VI...12..000. We would like to thank Di Wu, Yan Meng. David Stap, and Baohao Liao for their useful comments and discussions. We would also like to thank the reviewers for their feedback.
Brian S Everitt and Anders Skrondal. 2010. The cam- bridge dictionary of statistics.
Angela Fan, Shruti Bhosale, Holger Schwenk. Zhiyi Ma. Ahmed Ki-Kish Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multi- lingual machine translation. The Journal of Machine Leurning Research, (1)):4839-4888

---

Christian Federmann, Tom Koemi, and Ying Xin. 2022. Ntrex-128 news test references for mt evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21-24.
En-Shiun Lee, Sarubi Thillainathan, Shravan Nayak. Surangika Ranathunga, David Adelani, Ruisi Su, and Arya D McCarthy. 2022. Pre-trained multilin- gual requence-to-sequeeee
models: A hope for low- resource language translation? In Findings of the As- sociation for Computational Linguisties: ACL 2022, pages 58-67.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores- 101 evaluation benchmark for low-resource and multilingual chine translation. Transactions of the Association for Computational Linguistics, 10:522-532
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng. Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2649- 2663.
Jiatao Gu, Yong Wang. Kyunghyun Cho, and Victor OK Li. 2019. Improved gero-shot neural machine trans- lation via ignoring spurious correlations. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1258-122 -128..
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov. Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Shuhao Gu and Yang Feng. 2022. Improving zero- shot multilingual translation with universal represen- tations and cross- mapping. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 6492-6504.
Kaushal Kumar Maurya, Rahul Kejriwal, Maunen- dra Sankar Desarkar, and Anoop Kunchukuttan. 2023, Utilizing lexical similarity to enable zero-shot ma- chine translation for extremely low-resource lan- guages. arXiv preprint 205.022305.05214
Baijun Ji, Zhirui Zhang, Xiangyu Duan, Min Zhang. Boxing Chen, and Weihua Luo. 2020. Cross-lingual pre-training based transfer for zero-shot neural ma- chine translation. Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 115-122.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- for Computational Linguistics (Demonstrations), pages 48-53.
Melvin Johnson, Mike Schuster, Quoe V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas. Martin Wattenberg, Greg Corrado. et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In P Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 244-258.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv: 1612.0361.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. CoRR, abs/1412.6980.
Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Towards common understanding of con- tributing factors for cross-lingual transfer in multi- lingual language models: A review. arXiv preprint arXiv:2305. 230..230..6688.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86.
Telmo Pires, Eva Schlinger. and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001.
Taku Kudo and John Richardson. 2018 Sentencepiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. EMNLP 2018, page 66.
Maja Popović. 2017. chrf++:
words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 4483-4499,
Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference Machine Translation: Research Pupers, pages 186- 191.

---

Zhi Qu and Taro Watanabe. 2022. Adapting to non- centered languages for zero-shot multilingual transla- tion. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 5251- 5265.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Alessandro Raganato, Raúl Vázquez, Mathias Creutz, and Förg Tiedemann. 2021. An empirical investi- gation of word alignment supervision for zero-shot multilingual neural machine translation. In Proceed- ings of the 2021 Conference Empirical Methods in Natural Language Processing. The Association for Computational Linguistics.
Weizhi Wang, Zhirui Zhang, Yichao Du, Boxing Chen, Jun Xie, and Weihua Luo. 2021. Rethinking zero- shot neural machine translation: From a perspective of latent variables. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4321-4327.
Xinyi Wang and Graham Neubig. 2019. Target condi- tioned sampling: Optimizing data selection for multi- lingual neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5823-5828.
Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Sev- enth Conference on Machine Translation (WMT), pages 578-585.
Di Wu and Christof Monz. 2023. Beyond shared vocab- ulary: Increasing representational word similarities across languages for multilingual machine translation. arXiv preprint arXiv:2305. 230.:9199..
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt eval- uation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702.
Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. 2021. Language tags matter for zero-shot neural machine translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001-3007.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In 54th Annual Meeting of the Association for Computational Linguistics, pages 86-96. Association for Computational Linguistics (ACL).
Yilin Yang. Akiko Eriguchi. Alexandre Muzio, Prasad Tadepalli, Stefan Lee, and Hany Hassan. 2021. Im- proving multilingual translation by representation and gradient regularization. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7266-7277777
David Stap, Vlad Niculae, and Christof Monz. 2023. Viewing knowledge transfer in multilingual machine translation through a representational lens. arXiv preprint arx: 1550.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639.
Simeng Sun, Angela Fan, James Cross, Vishrav Chaud- hary, Chau Tran, Philipp Koehn, and Francisco Guzmán. 2022. Alternative imput signals trans- fer multilingual machine translation. Proceed- ings of the 60th Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pages 5291-5305
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450-3466.
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in opus. In Proceedings of the Fighth Inter- national Conference on Language Resources and Evaluation (LREC'12. 12), pages 2214-2218.
Ke Tran and Arianna Bisazza. 2019. Zero-shot depen- dency parsing with pre-trained multilingual sentence representations. EMNL-P-JJCNLP 2019, page 281.

---

A Appendices
or2ncd
on2lov Loxden
high2on
alozžen
A.1 Dataset Statistics
80
We list the details of the EC40 dataset in Table
7. Overall, EC40 is an English-centric multilin- gual machine translation dataset containing over 66 million sentences including languages (to- gether with English). EC40 is more profound the total number of languages and in the balance of lan- guage family and writing systems. Specifically, for each language family, we include 8 representative languages across different resources.
40
20
Figure 4: perplexity curves on English-centric directions on test-set.
12h
Moreover, set the number of sentences the same for each resource level, e.g.: all High- resource Languages have 5M sentences. Note: we list precise numbers in the table instead of approximate ones, for instance, 5M denotes ex- actly 5,000,000 number of sentences after pre- processing. We use ISO 639-13 in this table. We follow Flores-200 to label the writing system classes and use WALS (Dryer and Haspelmath, 2013) label the language family for languages in our dataset.
250
50
Figure 5: perplexity curves zero-shot directions our test-set. h, m, e denote High. Medium, Low, and extremely-Low resource levels, respectively.
A.3 Validation of Spurious Correlation
A.2
Training and Model specification
To ensure that our model does not inadvertently capture spurious correlations during training, we conduct validation process by visualizing the per- plexity curves for both English-centric and zero- shot directions proposed by (Gu et al., 2019). 64k
Encoder layers
Decoder layers
Vocab size
Model
Num
Emb
big
arFllagge
418M
12
1024
4096
16
mBart50
610M
12
1024
4096
250k
It important to note that these curves are solely used for visualization purposes and are not used criteria for early stopping. Our early stopping cri- terion for training is based solely on the validation perplexity, and we only consider English-centric directions in this regard.
Table Model specification
We also show the model specifications of mTransformer-big mTransformer--ne mBart50 Fine-tuning in the Table 8. It is worth noting that mBart50 utilizes vocabulary larger than our trained- from- from--cratcch scratch models. Furthermore, we adopt Vaswani et (2017) to set up the learning rate as 5e-4 with 4000 warmup steps and label smoothing of 0.1.
and
In Figure and Figure 5, we present the perplex- ity curves for English-centric and zero-shot direc- tions, respectively. We observe that the perplexities for the zero-shot directions gradually decrease dur- ing training, indicating that the model is learning and improving its translation performance on those directions. Importantly, no significant overfitting patterns are observed in the zero-shot perplexity curves. Instead, the decreasing perplexities on zero- shot directions suggest that the model is effectively learning the underlying patterns and generalizing its translation capabilities to unseen language pairs.
To keep the consistency of learning Sentence- Piece vocabulary, we also used temperature sam- pling (T 5) for training all models. We trained all models (including mBart50 FT) with NVIDIA A6000 GPUs for maximum of 200k updates. For larger models, we set the total max tokens as 215,040 using gradient accumulation to stimulate the large batch-size training in Tang et al. (2021).
Phttps://en ptitts::/ee.wwikipeee wwkipedia. hwtppeedii...g/iiiiiaeeeee wtt//wkii/s/......

---

Germanic
Romance
Slavic
Indo Aryan
Afro-Asizas Asiasic
ISQ
Language
Sciptt
I50
Language
Script
ISO
Langunge
Scrige
ISO
Language
Scrigt
ISC
Langunge
Scripe
French
Lain
Russian
Cyrillic
Hindi
Devanggari
Arabic
(5m)
nl
Dunch
Latin
Spanish
Latin
Cocch
Latin
bn
Bengali
Bengali
he
Hobrew
Hebrew
Swedish
pl
Devanagun
(1m)
da
Danish
Latin
Portugaese
Latin
hg
Rulgarian
Cyrillic
Marathi
Devanagrri
hu
Hansa"
Latin
Low
x
Afrikanns
Latin
Romanian
Latin
uk
Ukrninian
Cyrillic
sd
Sindhi
Arabic
d
Tigrisya
Ethiopic
(100k)
Lusembourgish
Latin
Oceitan
Latin
Sethian
Cinjarni
Desanagun
Ambaric
Ishiopic
clow
00
Norwegian
Latin
2el
Asturion
Latin
be
Belarusion
Cyrillic
mc
Nepali
Devanaari
kab
Kabyle"
Latin
ic
Teclandic
Catalan
Latin
bs
Bosnian
Undu
Arsiii
Somali
Latin
Table Details of Our EC40 Multilingual Machine Translation Dataset. Numbers in the table represent number of sentences, for example, 5m denotes exactly 5,000,000 number of sentences. Two exceptions are Hausa and Kabyle, where their data-size are 334k (334,000) and 18k (18,448) respectively.
A.4 mBART50 Performance comparison
We show performance comparison results on English-centrie and ZS directions in Figure 6, 7, and 8 categorized by both seen and unseen lan- guages. It is clear that fine-tuned mBart50 out- performs the mTT-large model on most of X->En directions, but lags behind in En-xX and zero-shot directions.
Figure Performance comparison between fine-tuned m0 and mT-llrge on X2En directions.
Furthermore, our conclusions are more com- prehensive and reliable due to two factors: I) compare to Tang et al. (2021), our evaluation set encompasses multi-parallel sentences, allowing for performance assessment across various lan- guage pairs, including low-resource directions. 2) compare to Wang et al. (2021), we employ the Transformer- model, enabling fairer compar- ison mBART50 in terms of model size.
Figure Performance comparison between fine-tuned mBart50 and m'T-large on ZS directions. Note that we show average ZS sacrebleu scores this Figure, e.g: column denotes All de (All ZS directions translating from and into german).
other metrics (Sacrebleu, Chrf++, Comet) in this section.
Figure 6: Performance comparison between fine-tuned mBart50 and miTllaree on En2X directions.
A.5.1 Resource-level Analysis
A.5 Additional Experiment Results based on other metrics
Table 9, 10, and 11 show the Resource level anal- ysis of the miTranfformrr-lagge model for both English-centric and Zero-shot directions across three different metrics. Combined with Table 3, we verify that our conclusions section 5.1 are consistent across all four metrics.
We ensure a comprehensive analysis by employing multiple evaluation metrics, aiming for a holistic assessment of our experiments. In the paper, we have already shown the results based on the SpBlcu, thus, we provide results for all analyses based on

---

Bacrablcu
Features
Data-size"
En-centric perf.
Target
Metries
Sie_size
TgLsize
Ste-En
En-+Igt
High
Ava
23.83
Correlation
Pearson
0.12
0.54
0,34
0.74
Source
High
32:73
11.41
12.09
0.82
2.36
7.32
Spearman
0.15
0.59
0.35
0.72
Med
32.05
11.91
11:30
6.12
2.52
0.12
R squuare
34.47%
64.50%
Regression
MAE
8.12
6,96
644
21
7.72
15
4.76
2:18
5.70
RMSE
10.07
8.42
Avg
26.69
10.09
10.42
5.32
2.50
2,02
Table 13: Analysis of zero-shot performance consider- ing data size and English-centric performance based on Chrf++.
Table 9: Resource- Baaeed Translation Performance Anal- ysis of m'l-large based on Sacrebleu. We include both English-centric and zero-shot directions.
Features
Data-size'
En-centric perf.
Chift
Metrics
Sre_size
Tgt_size
Sre-FFn
En-oTyr
Targel
Med
Low
e-Low
Avg
Corelltton
Pearson
0.25
0,43
0.53
0.66
En
53.65
54.85
43:68
38
47.5
Spearman
0.31
0.47
0.46
0.60
So.ne
High
08.33
21
22.35
16.22
15.85
26.
Regression
R-square
MAE
4,40%
11.08
69.53
7.35
10.53%
LOW
51.99
30:17
31
19.39
14.55
23.05
RMSF
13.67
9.04
e-L.ww
45-46
23.53
21.07
15:66
23.00
Table 14: Analysis zero-shot performance consider- ing data size and English-centric performance based Comet.
Table 10: Resource-Based Translation Performance Analysis of mTT-large based on Chrf++. We include both English-centre and zero-shot directions.
A.5.3 The effect of Linguistic properties
Comeet Target
We investigate how zero-shot performances change if the source and target languages are linguistically more similar, considering language family and writ- ing system. Table 15 demonstrates the fine-grained analysis regarding the effect of linguistic properties on the ZS translation quality.
Avg
En
63.58
82:80
78.77
70.02
78.82
Souce
High
84-48
68.37
68.87
81:28
50.50
69.53
67.41
50.57
60.74
63.57
54.20
46.08
56.49
e-Low
11
50.1
18
00.31
53.22
43.90
$3.92
Avg
75:40
63.78
65.29
37.07
47.55
28.05
Table 11: Reseurce-Based Translation Performance Analysis of mT-larre based on Comet. We include both English-centrie and zero-shot directions.
clow
Mad
Med
High
sume Languge Finily
5,05
7.67
2.12
4.32
9.43
Yes
9.15"
K.74"
9,62
3.14"
7,69
13.16"
12.8"
H
V tee the tome Wrinig sjarm
A.5.2 The impact of data and English-centric performance
No
4.56
5.10
6.78
6.57
1.58
3.97
9.31
8.0
9.15
12AR
Wetch's Sear "p s- 0.000 "0.00 o <- D.0, '00) i s- 5.0 m. denetee D105 S
Table 12, 13, and 14 show the impact of data and English-centric performance of the mTransformer- large model across three different metrics. Com- bined with Table 4, we verify that our conclusions in section 5.1 are consistent across all four metrics.
Table 15: The impact linguistic properties on zero- shot performance. To investigate it in depth, analyze it in fine- tie-grained levels by observing different resource levels We also conducted Welch's t-test to validate if one group is significantly better than another.
A.5.4 Overall Correlation Analysis using all factors
Features
Data-size Sre_size
En-centric perf. En->Tgr
Metrics
Tgt_size
Sre-EBn
Pearson
0.15
0.52
0.40
0.67
Our findings hold consistently across various evalu- ation metrics, spanning word, sub-word, character, and representation levels. For analyses in the see- tion 5.2 and 5.3, we show the additional results that based on Sacrebleu, Chrf++, Comet in Table 16.
Comelation
Spcarman
0.18
0.62
0.38
0.69
R-square
30.93%
60.97%
Regression
MAE
4.12
3.46
RMSE
4.96
4,16
Table 12: Analysis of zero-shot performance consider- ing data size and English-centric performance based on Sacrebleu.
A.5.5 The role Off-Target Issue
We utilized SpBleu in Section 5.4 to align with the setup employed by Zhang et al. (2020). We

---

ID
Features
R-square
MAE
RMSE
Sacrebleu (mT-large)
1
En performance 60.97%
3.46
4.16
2
Vocab-Sim 79.09%
2.85
3.62
3
Linguistie-features 81.27%
2.80
3.54
Chrf++ (mTT-lrrge)
4
En performancc 64.50%
6.96
8.42
5
4 Vocab-Sim 77.22%
6.87
8.49
6
Linguistic-feature 84.45%
5.74
7.09
Comet (m'T-large)
7
En_ performacce 69.53%
11.08
13.67
8
7 Vocab-Sim 77.22%
6.87
8.49
9
Linguistic-feature 79.51%
8.30
Table 16: Prediction of Zero-Shot Performance using En En-Centric performance, vocabulary overlap, and lin- guistic properties.
further provide the correlation results (correlation coefficient) based on Chrf++ and Sacrebleu below. Note that all experimental setups are the same, the only difference is the change of MT metric, for more details please check the setup descriptions in Section 5.4.
Sacrebleu
Chrf++
SpBleu
Spearman -0.07
-0.07
-0.08
Pearson -0.06
-0.03
-0.07
Table 17: Correlation coefficient between zero-shot per- formance and off-target rate when focusing on direc- tions where the off-target rate considerably low (less than 5%).
A.6 Performance for all directions
Figure 9 and 10 illustrates the specific results of the mrransforrer-llagg model all 1,560 zero-shot directions using four metrics.

---

Figure 9: Zero-shot performance of inTransformer-large on 1560 directions for Sacrebleu and Chrf++
Figure 10: Zero-shot performance of m on 1560 directions for SpBleu and Comet


Reviews:
[{"review_id": "zchM3QPrtP", "reviewer": "Reviewer_jc8P", "summary": "", "questions": "-) Page 3 lines 238-240: What does this sentence mean?\n\n-) Page 4 lines 266-269: Is the cleaning step performed before or after selecting the data in the EC40 dataset? If it is performed after the creation of the dataset, how can you guarantee that the data distribution is not modified?\n\n-) What are the numerator and denominator in the CV equation?\n\n-) Page 7, why do the authors use RMSE in Table 6 but only R-square and MAE in Table 4? What is the advantage of adding RMSE in Table 6?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n(1) Addresses an interesting and important issue for the machine translation community by providing in-depth study and extensive analyses;\n\n(2) The experimental setting is grounded and the results confirm the claims being made;\n\n(3) The main findings of the paper are summarized and possible future directions are identified;\n\n(4) The novel dataset is an important contribution.\n\nWe have responded to your comments below and updated our revision accordingly. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into the paper.\n\n---\n\n> **Reasons To Reject 1, 2, and 4.** Some of the key concepts used in the paper are not properly introduced. Some relevant results are reported in the Appendix. Can this part be moved in the Appendix?\n\nThank you for offering this guidance. We have already improved the presentation of them accordingly in our revision. We really appreciate your feedback and we believe this strengthened our paper. \n\n---\n\n> **Reasons To Reject 3.** The vocabulary overlap part is an important contribution to the paper, but its setting is not clear. In Section 5.2.2 lines 509-513, it is not clear what correlation is computed and it seems to refer only to the source language. This part should be clarified and improved.\n\nWe thank you for the feedback. As outlined in line 495, in the whole paper, we measure vocabulary overlap in the same way, i.e., the extent of shared subwords between the source and target languages.\n\nWhen solely studying whether a higher overlap correlates with improved ZS performance (lines 503-509), we fixed the target language. e.g., the target language is German, the source side is other 39 languages; then we compute the correlation score using these 39 overlap scores and corresponding 39 ZS performances. We report the average correlation scores (for all 40 target languages) in lines 518--522.\n\nWhen incorporating the vocabulary overlap feature with English translation capability (En_performance) in the regression experiment (Table 6), we use the overlap directly in the regression analysis.\n\n---\n\n> **Reasons To Reject 5.** The MT evaluation are all string-matching metrics. If the use of COMET could make the findings more robust.\n\nThank you for this suggestion. We have conducted the COMET evaluations and ran all analyses based on COMET's supported languages (35/41 including English). **In sum, the findings based on COMET for all sections are consistent with our previous analyses. and we believe this makes our paper more solid.** Below we provide the main evidence, and we have already added them in the revision.\n\n#### Table 10: Resource-level analysis based on Comet score. This result is consistent with Table 3 in page 5. Our main conclusion in this part remains the same: the resource level of the target language has a stronger effect on ZS translation qualities.\n|       |      | Target     |       |       |       |       |       |\n|------ |------|:----------:|-------|-------|-------|-------|-------|\n|       |      |    En      | High  |  Med  |  Low  | e-Low |  Avg  |\n| Source|  En  |     -      | 0.84  | 0.83  | 0.79  | 0.70  | 0.79  |\n|       | High |    0.84    | 0.68  | 0.70  | 0.61  | 0.51  | 0.63  |\n|       |  Med |    0.81    | 0.66  | 0.67  | 0.60  | 0.50  | 0.61  |\n|       |  Low |    0.77    | 0.62  | 0.64  | 0.54  | 0.46  | 0.56  |\n|       |e-Low |    0.72    | 0.58  | 0.60  | 0.53  | 0.44  | 0.54  |\n|       |  Avg |    0.78    | 0.64  | 0.65  | 0.57  | 0.48  | 0.59  |\n\n#### Table 11: Analysis of zero-shot performance considering data size and English-centric performance based on Comet score. The result is consistent with Table 4 on page 6. Our main conclusion in this part remains the same: Out-of-English translation quality correlates with zero-shot performance the most. We also observed the data-size feature can even harder to explain the zero-shot variations (with low R-square and high MAE, and RMSE scores). \n\n|                 |                | Src_size  | Tgt_size | Src->en          | en->Tgt  |\n|-----------------|----------------|-----------|----------|------------------|----------|\n| Correlation     | Pearson        | 0.25      | 0.43     | 0.53             | 0.66     |\n|                 | Spearman       | 0.31      | 0.47     | 0.46             | 0.60     |\n\n\n|                    |                    |  Data-size | En-centric perf. |\n|-----------------|----------------|----------------|----------------------|\n| Regression      | R-square       | 4.40%     | 69.53%   |      \n|                 | MAE            | 11.08     | 7.35     |          \n|                 | RMSE           | 13.67     | 9.04     |        \n\n#### Table 12: Prediction of ZS Performance using En-Centric performance, vocabulary overlap, and linguistic properties. We present the result based on the mT-large and the Comet score in this table. This result is consistent with Table 6 on page 7. Our main conclusion in this section remains the same: Vocabulary overlap and linguistic features can further explain the ZS variations. \n|   ID  |          Features          | R-square |   MAE   |   RMSE  |\n|-------|---------------------------|----------|---------|---------|\n|   4   |      En_performance       | 69.53%   | 11.08   | 13.67   |\n|   5   |      4 + Vocab-Sim        | 77.22%   | 6.87    | 8.49    |\n|   6   | 5 + Linguistic-features   | 79.51%   | 6.63    | 8.30    |\n\n---\n\n> **Question 1.** Page 3 lines 238-240: What does this sentence mean?\n\nThis sentence means for each resource, we control the number of languages and the number of training sentences the same.\n\n* Are the number of languages in each resource level the same? \n    * Yes, 10 languages for each resource.\n* Are the number of languages in each language family the same?\n    * Yes, 8 languages for each family.\n* Are the number of training sentences in each resource level the same?\n    * Yes, e.g.: 5M for high-resource, and 1M for Medium-resource, and etc.\n\n---\n\n> **Question 2.** Page 4 lines 266-269: Is the cleaning step performed before or after selecting the data in the EC40 dataset? If it is performed after the creation of the dataset, how can you guarantee that the data distribution is not modified?\n\nThe cleaning step is before selecting the data in the EC40 dataset. This means all numbers of training sentences reported in Table 7 (page 12) are after the cleaning, therefore our models trained using the exact numbers we reported in Table 7. Please also check the steps we list below.\n\nDownload the data from OPUS -> Do the data cleaning step -> Sample data from the cleaned set -> Learn vocabulary and train models\n\n---\n\n> **Question 3.** What are the numerator and denominator in the CV equation?\n\nThe Coefficient of Variation $CV = \\frac{\\sigma}{\\mu}$ is defined as the ratio of the standard deviation (numerator: $\\sigma$) to the mean (denominator: $\\mu$) of zero-shot performance.\n\n---\n\n> **Question 4.** Page 7, why do the authors use RMSE in Table 6 but only R-square and MAE in Table 4? What is the advantage of adding RMSE in Table 6?\n\nWe add RMSE to our experiments instead of only using MAE mainly because of two points: \n* Sensitivity to Large Errors: RMSE gives more weight to larger errors due to the squaring operation.\n* Outlier Impact: Because RMSE squares the differences, it is more sensitive to outliers than MAE. \n\nTherefore, we incorporated two metrics in Table 6 for a more comprehensive analysis. We appreciate your advice and we further present the RMSE for Table 4 (see below) and updated this in the revision to make the presentation more consistent.\n\n|              |                     | Data-size | En-centric perf. | \n|--------------|---------------------|-----------|----------|\n|Regression    | R-square            | 32.54%    | 61.34%   |\n|              | MAE                 | 5.47      | 4.70     |\n|              | RMSE                | 6.62      | 5.59     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an interesting and important issue for the machine translation community. In particular, it tries to better understand zero-shot translation by running an extensive set of experiments covering a large number of languages and conditions.\n\nThe experiments take into account several factors trying to limit biases and situations that can alter the results and the findings. The overall experimental setting is grounded. The results confirm the claims being made.\n\nI appreciate Section 5.4 where the main findings of the paper are summarized and possible future directions are identified.\n\nThe novel dataset for training purposes is an important contribution of this paper. ", "reasons_to_reject": "The weaknesses of the paper are:\n\n1) Although the relevant papers are cited, some of the key concepts used in the paper are not properly introduced. IMHO, this reduces the target audience of the paper only to experts in multilingual neural machine translation and makes the paper difficult to follow and understand. For instance:\n1.1) “issue of spurious correlation”: this is mentioned in Section 2.2 and re-used in several parts of the paper. \n1.2) “off target issue”: this is mentioned for the first time at the end of the introduction.\n1.3) “correlation and regression analysis”: this is mentioned in Section 5.1 and used in the remaining sections.\n1.4) “sentence level off target rate”: used in Section 5.3.\nAll these concepts should be properly introduced in the paper.\n\n2) The appendix can be used to report “anonymised related work (see above), preprocessing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper” (see the call for papers) and the paper should be self-contained. In this paper, some relevant results are reported in the Appendix:\n2.1) Section 4 “Does pre-training matter?”\n2.2) Section 4 “Quantifying variation”\n2.3) Section 5.2.1” “Linguistic Properties”\nThis makes the paper difficult to follow. \n\n3) The vocabulary overlap part is an important contribution to the paper, but its setting is not clear. In the abstract, it is mentioned the vocabulary overlap between source and target languages, but in Section 5.2.2 lines 509-513 it is not clear what correlation is computed and it seems to refer only to the source language. This part should be clarified and improved.\n\n4) Some of the results are somehow expected, in particular, performance improvements when source and target languages adopt the same writing script. Can this part be moved in the Appendix using the left space for improving other parts?\n\n5) The MT evaluation is based on three metrics: Bleu, Chrf++, and SpBleu. These are all string-matching metrics. I was wondering if the use of COMET could make the findings more robust. Not sure, if COMET covers all the languages involved in this study. If not, it can be used only for a subset of language directions.", "questions_to_authors": "-) Page 3 lines 238-240: What does this sentence mean?\n\n-) Page 4 lines 266-269: Is the cleaning step performed before or after selecting the data in the EC40 dataset? If it is performed after the creation of the dataset, how can you guarantee that the data distribution is not modified?\n\n-) What are the numerator and denominator in the CV equation?\n\n-) Page 7, why do the authors use RMSE in Table 6 but only R-square and MAE in Table 4? What is the advantage of adding RMSE in Table 6?\n\n", "paper_topic_and_main_contributions": "This paper presents an in-depth study and analysis of zero-shot neural machine translation (ZS-NMT). Starting from the assumption that knowledge sharing is the basement for ZS-NMT, this paper runs an extensive analysis with the aim of better understanding zero-shot translation and its limit and benefit. This study reveals that there are high variations in performance across different language directions. The reasons for this can be ascribed to the role of English-centric translation capability, vocabulary overlap between source and target languages, and linguistic properties. The analysis shows that the first factor is the most important one. To have a fair analysis, a novel dataset for training purposes is proposed and released. This dataset balances data distribution and considers linguistic characteristics of diverse representations of languages.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "All the references are appropriate.\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 7 line 533: “model capacities. And it shows” dot should be replaced with a comma.\n\nThe paper is quite dense with several important concepts that are not fully introduced (see issue 1 in the weaknesses section) or relevant results for the paper that are in the Appendix (see issue 2 in the weaknesses section). To favor the understanding and reading of the paper, a drastic decision has to be made by moving some experiments to the Appendix obtaining more space for improving the remaining parts. A good candidate can be Section 5.3."}}, {"review_id": "1zuFbnYO2s", "reviewer": "Reviewer_2RvR", "summary": "", "questions": "Please see 'Reasons to reject'", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n1. Conduct a large multilingual translation dataset.\n2. Explore the influence of factors on the performance of zero-shot translation.\n3. Conduct extensive experiments.\n\nWe have responded to your comments below and updated our revision accordingly. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into the paper.\n\n---\n\n> **Reasons To Reject 1.** What is the difference between Sacrebleu and SpBleu in the experiments? And are their scores in different translation directions comparable?\n\nThank you for the question. To address your question, it's important to note that our paper does not involve cross-metric comparisons. Secondly, our approach focuses on providing a comprehensive analysis, incorporating various metrics to ensure a thorough evaluation. Specifically, we incorporate the following metrics: \n* ChrF++ (character level)\n* SpBleu (tokenized sub-word level)\n* SacreBleu (detokenized word level)\n* COMET (representation level)\n\nOur intention in employing these multiple metrics is to ensure a holistic assessment of our experiments. Notably, we maintain consistency in our findings across all these metrics, which lends robustness to our conclusions. To delve further into the analysis based on COMET, we kindly direct you to our responses in the context of *\"Reasons To Reject 5\"* where more detailed insights are provided, specifically addressing the concerns raised by reviewer jc8P. \n\n---\n\n> **Reasons To Reject 2.** Why EC40 is balanced, while EC40's scale is from 50K to 5M and the Script of most languages is Latin?\n\nThank you for raising this point. We pointed out that EC40 is balanced across resources and languages (lines 218--221) instead of the number of training sentences among the whole dataset or type of scripts. Please check Sec.A.1 for more details in the Appendix. Specifically, the EC40 is carefully balanced across various scales (from lines 234-240):\n\n1. For each resource, the number of languages is the same (10 languages).\n2. For each resource, the number of training sentences is balanced (line 238-240), e.g.: 5M for all High-resource languages)\n3. For each language family, we incorporate 8 languages (lines 236-238).\n\n---\n\n> **Reasons To Reject 3.** Why the m2m-100 model could be evaluated directly, without any fine-tuning? No gap between itself train set and EC40?\n\nThank you for your question. **Our goal in introducing the m2m-100 models is to investigate whether the high variations in the ZS performance phenomenon hold across models.** Specifically, m2m-100 is a trained multilingual translation model using 7.5B parallel sentences covering 2200 training directions, whereas our EC40 contains only 66M sentences. Moreover, m2m-100 is a strong translation model achieving SOTA results in multiple translation directions including both supervised and zero-shot ones [1].\n\n**An additional noteworthy point is the alignment of domain factors.** Our validation and test sets encompass data from diverse domains, a characteristic shared with the m2m-100 model's training data, which includes content from multiple domains such as CCMatrix. This feature contributes to a smaller domain mismatch between the m2m-100 model and our test set, further justifying our direct evaluation approach.\n\n---\n\n> **Reasons To Reject 4.** What is referred to in the CV metric for zero-shot in Table 1? And why the highest score is noted by underline?\n\n\nThank you for your question. **In the context of Table 1, the reference is to the Coefficient of Variation ($CV = \\frac{\\sigma}{\\mu}$), which serves as a metric used to quantify the magnitude of variation in zero-shot translation performances when compared to supervised translation directions.** The purpose of utilizing this metric is outlined in lines 349--355 of the paper. Specifically focusing on Table 1, the Coefficient of Variation (CV) scores are computed for various machine translation (MT) metrics, namely Sacrebleu, Chrf++, and Spbleu. These scores are calculated under different settings, such as language pairs denoted as En→X.\n\n\nNotably, in Table 1, the highest CV score for a particular metric and setting is indicated by being underlined. This visual emphasis (underline) draws attention to the translation scenarios where the variation between zero-shot directions is more pronounced than supervised ones, helping to highlight significant findings of high variations.\n\n\n\n---\n\n> **Reasons To Reject 5.** What is the inner relationship among these aspects (English-centric capacity, vocabulary overlap, linguistic properties, and off-target issue)\n\nThank you for this question. **While investigating correlations between these factors could indeed provide valuable insights, the primary objective of our study is to uncover their relationships with our target variable: zero-shot translation ability.** The main focus of the paper is to shed light on how these factors interpret high variations in zero-shot translation performance.\n\nMoreover, we discuss one intuitively strong relationship here: One of the intuitive connections lies between vocabulary overlap and linguistic properties. For instance, languages that share the same writing system might exhibit a higher vocabulary overlap due to lexical similarities. This part is explicitly explored in the paper (lines 382--488), highlighting that vocabulary overlap serves as a more fundamental indicator of surface-level similarity compared to more intricate linguistic metrics such as language family or typology distance.\n\n---\n\n> **Reasons To Reject 6.** Why switch the metric to 'Sacrebleu' in Sec.5.4?\n\nThank you for your question. I assume you mean why use Sacrebleu in Sec.5.3 instead of Sec.5.4? Please let us know if this is not true. \n\nWe utilized Sacrebleu in Sec. 5.3 to align with the setup employed by [2] (the paper hypothesized the off-target issue is the root cause of the low ZS performance). we further provide the correlation results (correlation coefficient) based on Chrf++ and Spbleu below. Note that all experimental setups are the same, the only difference is the change of MT metric, for more details please check the setup descriptions in Sec.5.4.\n\n|           | Sacrebleu | Chrf++ | SpBleu|\n|-----------|-----------|--------| ------|\n| Spearman  |  -0.07    | -0.07  | -0.08 |\n| Pearson   |  -0.06    | -0.03  | -0.07 |\n\n---\n\n\nReferences:\n\n[1] Fan, Angela, et al. \"Beyond english-centric multilingual machine translation.\"\n\n[2] Zhang, Biao, et al. \"Improving massively multilingual neural machine translation and zero-shot translation.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Authors conduct a large multilingual translation dataset, containing 66M sentence pairs in total.\n2. This paper tries to explore the influence of three aspects on the performance of zero-shot translation.\n3. This paper conducts expensive experiments to prove their conclusion.", "reasons_to_reject": "Although the authors conduct expensive experiments, some questions about it are as follows:\n \n1. What is the difference between Sacrebleu and SpBleu in the experiments? And are their scores in different translation directions comparable? \n2. Why EC40 is balanced, while EC40's scale is from 50K to 5M and the Script of most languages is Latin?\n3. Why the m2m-100 model could be evaluated directly, without any fine-tuning? No gap between itself train set and EC40?\n4. What is referred to in the CV metric for zero-shot in Table 1? And why the highest score is noted by underline?\n5. What is the inner relationship among these aspects (English-centric capacity, vocabulary overlap, linguistic properties, and off-target issue)\n6. Why switch the metric to 'Sacrebleu' in Sec.5.4?", "questions_to_authors": "Please see 'Reasons to reject'", "paper_topic_and_main_contributions": "This paper conducts a multilingual dataset (EC40), which contains 66M sentence pairs in total over 40 languages, by expanding the OPUS dataset.\nFurther, the authors adopt EC40 as the train set to explore the influence of three factors (English-centric translation capacity, vocabulary overlap, and linguistic properties) on the zero-shot translation.\nBesides, the authors", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uGUIEuhr21", "reviewer": "Reviewer_3cCd", "summary": "", "questions": "- Why do translations from/to medium-resource languages have better quality than that from/to high-resource languages according to Table 3? \n- While the paper notes that high variation is not the sole cause of low average zero-shot performance, the relationship between these factors and the root cause of low performance is not discussed in depth. Could the authors provide further insights into the underlying causes of low performance and how they relate to the factors of variation?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n1. Design extensive and comprehensive experiments to delve into the phenomenon of high variations in zero-shot MT.\n2. The conclusions are general regardless of the resource level of languages.\n3. A new dataset is suitable for academic research.\n4. Propose a new point of view that the off-target problem is a symptom.\n\nWe have responded to your comments below and updated our revision accordingly. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into the paper.\n\n> **Reasons To Reject 1.** The three factors of variations are intuitive and have already been observed in previous works. No novel phenomena were found.\n\nWe respectfully hold a different perspective on this matter. To the best of our knowledge, this paper represents the first attempt to comprehensively and systematically study in depth the impact of these three factors on zero-shot translation qualities. **Our approach distinctly sets us apart from previous research, and we invite you to assess the recent relevant works that we have collated for comparison (all these references have also been introduced in our paper).**\n\n| Paper                                                    | Task              | Descriptions                                                                                              |\n| -------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------- |\n| Zhang et al. (2020) | Zero-Shot NMT     | Investigates how modeling capacity affects ZS performance                                                                   |\n| Wu et al. (2021); Raganato et al., (2021); Wang et al. (2021) | Zero-Shot NMT     | MNMT models are prone to forget the language label information  |\n| Chen et al. (2022); Gu et al. (2019); Tang et al. (2021); Wang et al. (2021) | Zero-Shot NMT     | Examines the impact of model initialization on ZS performance|\n| Pan et al., 2021; Ji et al., (2020); Wu et al. (2021); | Zero-Shot NMT  | Recognize the inconsistency of semantic representation in different languages |\n| Aharoni et al. (2019) | Zero-Shot NMT     | Suggests that adding more languages in an MNMT system improves zero-shot performance more for close language pairs |\n| Pires et al., (2019)  | Cross-lingual NLU | Concludes that transfer is more successful for languages with high lexical overlap and typological similarity |\n| Lauscher et al., (2020) | Cross-lingual NLU | Concludes that transfer is more successful for syntactically or phonologically similar languages |\n\n**We would like to reiterate the novelty and contribution of our research concerning the three factors, and we invite you to substantiate your claims with references for a more profound discussion.**\n\n1. Our identification of the considerable influence of out-of-English translation quality on zero-shot variation is novel, supported by detailed insights provided in lines 566 to 574. We further provide insight into how to use this finding to improve the zero-shot NMT in Section 5.4.\n2. We found that performance in the close language pairs (with same linguistic properties) is significantly better than distant pairs regardless of resource level, and this phenomenon is especially pronounced for smaller models. We further stress the importance of focusing efforts on distant language pairs rather than exclusively concentrating on resource-rich pairs, as prior ZS-NMT research has done (lines 575--586).\n3. We have consistently observed that vocabulary overlap plays a significant role in explaining zero-shot variations, encouraging greater cross-lingual transfer and knowledge sharing via better vocabulary sharing has the potential to enhance zero-shot translations (lines 587--600).\n\n\n---\n\n\n> **Question 1.** Why do translations from/to medium-resource languages have better quality than that from/to high-resource languages according to Table 3?\n\nThank you for raising this point. This phenomenon is observed in other En-centric MNMT experiments, as evidenced by the studies conducted by [1] and [2]. \n\nFurthermore, our own experiments have shown that certain medium-resource languages, like Afrikaans, demonstrate the highest performance across all metrics. This exceptional performance of Afrikaans contributes significantly to raising the overall average score within the medium-resource group. This perspective is consistent with earlier research, which highlights Afrikaans' extensive lexical and syntactic borrowings from English, as well as its close linguistic affiliations with multiple neighboring languages of English [3].\n\n---\n> **Reasons To Reject 2 and Question 2** High variation is not the cause for low average zero-shot performance. Their relationship and the root cause of low performance are not discussed in depth. While the paper notes that high variation is not the sole cause of low average zero-shot performance, the relationship between these factors and the root cause of low performance is not discussed in depth. Could the authors provide further insights into the underlying causes of low performance and how they relate to the factors of variation?\n\nThank you for your thoughtful feedback. We acknowledge the universally recognized challenge of zero-shot performance consistently falling behind supervised performance in the broad domain of deep learning. The community as a whole is deeply engaged in understanding this issue and actively works to enhance it. In the context of Multilingual Machine Translation, **recent research has focused on seeking explanations for the causes of overall low ZS performance and trying to enhance it.** Please refer to the literature and descriptions we demonstrated in *Reasons To Reject 1* and Sec.2.2 in the paper.\n\n**In contrast, our study introduces a fresh perspective within zero-shot NMT: the presence of high variations.** This implies that ZS directions don't uniformly suffer from low performance; instead, some directions exhibit decent results. Specifically, it is evidenced by certain ZS directions exhibit promising performance, with approximately 70% of ZS-condition-1 achieving SpBleu scores surpassing 20 (see Figure 1 on page 1). In stark contrast, merely 10% of the remaining directions achieve comparable SpBleu scores. This intriguing phenomenon prompted us to further understand the underlying factors driving this extensive variation in zero-shot NMT.\n\nWe recognize that our investigation of high variations in zero-shot performance adds an important layer of insight to the discourse surrounding zero-shot NMT, which provides an additional perspective than understanding the root causes of overall poor performance in zero-shot scenarios. Drawing from our findings, we have identified three key factors responsible for driving these variations and have subsequently offered insights to enhance zero-shot NMT performance. Of note, our conclusions emphasize the significance of resource-lean and distant language pairs as a focus for improving zero-shot performance. Crucially, this insight aligns with, rather than contradicts, existing efforts aimed at improving performance in resource-rich language pairs, while also addressing the aspect of language relatedness that has often been overlooked.\n\n\n---\n\nReferences:\n\n[1] Zhang, Biao, et al. \"Improving massively multilingual neural machine translation and zero-shot translation.\"\n\n[2] Yang, Yilin, et al. \"Improving multilingual translation by representation and gradient regularization.\"\n\n[3] Zhou, Zhong, and Alex Waibel. \"Family of origin and family of choice: Massively parallel lexiconized iterative pretraining for severely low resource text-based translation.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper delves into the phenomenon of high variations in zero-shot translation performance, and design extensive and comprehensive experiments to demonstrate the factors of the phenomenon. The conclusions are general regardless of the resource level of the source and target language.\n- A new multilingual translation dataset is released for community which is suitable for academic research.\n- The authors propose a new point of view that the off-target problem is a symptom rather than the root cause since directions with low off-target rate may also have poor performance.", "reasons_to_reject": "- The three factors of variations are intuitive and have already been obeserved in previous works. No novel phenomena were found.\n- High variation is not the cause for low average zero-shot performance. Their relationship and the root cause of low performance are not discussed in depth.", "questions_to_authors": "- Why do translations from/to medium-resource languages have better quality than that from/to high-resource languages according to Table 3? \n- While the paper notes that high variation is not the sole cause of low average zero-shot performance, the relationship between these factors and the root cause of low performance is not discussed in depth. Could the authors provide further insights into the underlying causes of low performance and how they relate to the factors of variation?", "paper_topic_and_main_contributions": "This paper focuses on the phenomenon of variations in zero-shot multilingual neural machine translation performance, and investigate the factors behind the phenominon via extensive experiments. Since the current MNMT datasets have limited training sentences and exhibit high levels of imbalance, the authors first introduce a new MNMT dataset EC40 which contains 40 non-English languages with different linguistic properties and all directions for further research. Experiments show that zero-shot directions have much lower performance and higher variations. To explain the high zero-shot variations, the authors identify three key factor: English-centric translation capacity, vocabulary overlap and linguistic properties, Among which the capacity of out-of-English translation serves as the most important one. Moreover, the authors argue that the off-target problem is a symptom of inadequate zero-shot performance rather than the root cause. Based the observations. the authors finally give some potential remedies to enhance the zero-shot performance of MNMT.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"bleu\" -> \"blue\" in the caption of Table 3\n- Use vector graphics for figures"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the phenomenon of zero-shot neural machine translation (ZS-NMT) performance variations across different language directions and seeks to understand the underlying factors. It introduces a new dataset, EC40, and conducts extensive experiments involving 40 languages, focusing on the influence of English-centric translation capacity, vocabulary overlap, and linguistic properties. 

Pros:
		
* Novel Dataset and In-Depth Analysis: The introduction of the EC40 dataset is highlighted as an important contribution to the paper. The extensive experiments conducted on this dataset offer valuable insights into ZS-NMT performance variations.
		
* Thorough Investigation: The paper explores the factors contributing to ZS-NMT performance variations comprehensively, considering English-centric translation capacity, vocabulary overlap, and linguistic properties. This in-depth analysis provides a holistic understanding of the issue.
		
* Identification of Off-Target Issue: The paper suggests that the off-target issue is a symptom rather than the root cause of inadequate ZS-NMT performance, offering a novel perspective on the problem.


Cons:
		
* Concept Clarification: Reviewer 1 points out that some key concepts used in the paper, such as "spurious correlation," "off-target issue," "correlation and regression analysis," and "sentence-level off-target rate," are not properly introduced, making the paper challenging to follow for non-experts.
		
* Metric Choice: Reviewer 2 questions the use of string-matching metrics (Bleu, Chrf++, and SpBleu) and suggests considering the use of COMET, although it's noted that COMET may not cover all the languages involved in the study.
		
* Lack of Novel Phenomena: Reviewer 3 notes that while the paper delves into ZS-NMT variations, the three identified factors are intuitive and have been observed in previous works. There is a need for more exploration of the relationship between high variation and low average ZS performance.

In summary, the paper presents a comprehensive analysis of ZS-NMT performance variations, supported by a new dataset, EC40. While it makes valuable contributions to the field, addressing the need for concept clarification, handling appendix content, and further exploring the relationship between high variation and low performance could enhance the paper's quality and impact.
Decision: Accept-Main