Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens
Authors: David Stap
Vlad Niculae
Christof Monz
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual machine translation
cross-lingual knowledge transfer

Abstract: We argue that translation quality alone is not a sufficient metric for measuring knowledge transfer in multilingual neural machine translation. To support this claim, we introduce Representational Transfer Potential (RTP), which measures representational similarities between languages. We show that RTP can measure both positive and negative transfer (interference), and find that RTP is strongly correlated with changes in translation quality, indicating that transfer \textit{does} occur. Furthermore, we investigate data and language characteristics that are relevant for transfer, and find that multi-parallel overlap is an important yet under-explored feature. Based on this, we develop a novel training scheme, which uses an auxiliary similarity loss that encourages representations to be more invariant across languages by taking advantage of multi-parallel data. We show that our method yields increased translation quality for low- and mid-resource languages across multiple data and model setups.\footnote{We will release our code upon acceptance.}

Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lens
David Stap Vlad Niculae Christof Monz
Language Technology Lab University of Amsterdam
dd.sta,, V. v.nicula,, c. c.monz @uva.nn nl
Abstract
In the context of transferring knowledge from parent translation model to child model, some rescarch has shown that quality improvements are larger when using a closely related parent (Zoph et al., 2016), while others found that unrelated lan- guage pairs can work even better (Koemi and Bojar, 2018). Another finding is that an English-centric model benefits most from positive transfer for direc- tions into English, while improvement in the other directions modest (Arivazhagan et al., 2019).
We argue that translation quality alone is not sufficient metric for measuring knowledge transfer in multilingual neural machine transla- tion. support this claim, we introduce Rep- resentational Transfer Potential (RTP), which measures representational similarities between languages. We show that RTP can measure both positive and negative transfer (interfer- ence), and find that RTP is strongly correlated with changes in translation quality, indicating that transfer does occur. Furthermore, we in- vestigate data and language characteristics that are relevant for transfer, and find that multi- parallel overlap is an important yet under- explored feature. Based on this, we develop novel training scheme, which iliary similarity loss that encourages represen- tations be more invariant across languages by taking advantage of multi-parallel data. We show that our method yields increased trans- lation quality for low- and mid-resource lan- guages across multiple data and model setups.
One of the most striking observations in the lit- erature is that the improvements of many-to-one mNMT can be explained to a large extent by the increased amount target data (Fan al., 2021), rather than by cross-lingual knowledge transfer.
aux-
Understanding cross- -onnull knowledge trans- fer in the context of mNMT is an under-explored research direction (Hupkes al., 2023). Despite some existing studies that have examined mNMT representations, none have yet connected these rep- resentations to knowledge transfer. For instance, when translating "voiture" in French and "Auto" in German to "car" in English, one would expect that the cross-attention context vectors for French- English and German-English would be similar. However, Johnson al. (2017) show that cluster- ing occurs on the sentence level rather than the word level. Even identical sentences in various languages do not occupy the same position in the representation space (Escolano et al., 2022), and encoder representations are dependent the target language (Kudugunta et al., 2019) instead of source meaning.
1 Introduction
Multilingual neural machine translation (mNMT) (Ha et al., 2016; Johnson et al., 2017) can support multiple translation directions in single model, with low-resource languages benefiting most and high-resource languages degrading in quality (Ari- vazhagan et al., 2019). However, there is a large discrepancy among low-resource languages, with some languages benefitiing a lot, while others see relatively little improvement. Conflicting findings have emerged in cross cross-lingual knowledge transfer research, leaving the underlying causes for this dis- crepancy unclear. For example, some studies have found that token overlap can be leveraged to in- crease translation performance (Patil et al., 2022; Wu and Monz, 2023), while others have found that token overlap is unimportant for cross- cross-lingual trans- fer (K et al., 2020; Conneau et al., 2020).
In this paper, we investigate the relationship between cross-lingual transfer and cross-attention similarities between languages, which we formalise as Representational Transfer Potential (RTP). This allows reason about knowledge transfer in a way translation quality (BLEU) is unable to cap- ture. We investigate cross-attentin because it acts as bottleneck between the encoder (mostly respon-

---

sible for representing the source sentence) and the decoder. We find that RTP can be used quantify positive, as well as negative transfer (also known as interference). Furthermore, we show that these similarities correlate with improvements in trans- lation quality, indicating that there is knowledge transfer, and the improved translation quality is not only due to the increased data on the target side.
OPUS-100 (Zhang et al., 2020) due to its high translation quality and inclusion of relatively large portions of explicit multi-parallel data, which is an important characteristic for our analysis. We train a many-to-many model on all language pairs that contain English in the source or target, resulting in 1 116 translation directions. To ensure compara- ble results, we apply joint subword segmentation (Sennrich et al., 2016) and use vocabulary size of 32K. We also train and evaluate bilingual baselines using the same setup.
Our approach allows us to identify the dataset and language characteristics that are relevant for transfer, such as multi murtillarallll overlap between languages. Based on our findings, we propose a method for training a multilingual translation model using an auxiliary similarity loss that ex- ploits marlli-praallll data, thereby increasing the degree of language invariance across source repre- sentations. Contrary to common perception, sig- nificant amount of multi martilla data exists within parallel datasets such as WMT, making it more abundant than commonly assumed (Freitag et al., 2020). Our method works by alternately feeding parallel and multi murti--arallel batches to model. For multi muutti-araalll batches, minimize an auxiliary similarity loss that encourages context vectors, re- sulting from cross-attention, to be similar. Our re- sults show that this approach leads to increased per- formance for low-resource languages across multi- ple data and model setups.
Additionally we evaluate on the out-of-domain FLORES-101 evaluation benchmark (Goyal et al., 2021). Out-of-domain data helps to assess robust- ness and generalization capabilities, and provides a more realistic measure of how well the system handle diverse and unexpected inputs. This dataset is completely multi-parallel, which is necessary property for our analysis. It consists of a dev (997 sentences) and devtest (1012 sentences) split, both of which we combine to enhance the robustness of our findings. Sentences are extracted from En- glish Wikipedia, and translated to 101 languages by professional translators.
Evaluation We calculate BLEU scores (Papineni et al., 2002) using sacreBLEU (Post, 2018).
Models We train many-to-one and many-to- many Transformer base models (Vaswani et al., 2017). Detailed information about the models and training process can be found in Appendix A.1.
2 Analyzing Transfer in Many-to-Many Models
this section, we aim to delve deeper into the un- derstanding of knowledge transfer across languages in mNMT models, moving beyond the commonly used metric of translation quality as a proxy for transfer. By exploring the relationship between transfer and hidden representations in multilin- gual model, we aim gain insight into why certain languages benefit more from multilingual training (as discussed in Section 3). Furthermore, we aim to develop training strategies that can increase repre- sentational similarity and thus enhance knowledge transfer (as outlined in Section 4).
Results For evaluation on TED we used tok- enized BLEU to be comparable with Neubig and Hu (2018) and Aharoni et al. (2019). Table 1 shows that our many-to-one and many-to-many models obtains comparable or better BLEU scores for X- X-=Eggishh directions.
2.2 (Dis Javantadvss of Multilingual Training
Having validated that our model meets strong base- lines, we will use the FLORES 101 (Goyal et al., 2021) evaluation dataset for our subsequent analy- ses. X- results are summarized in Table 2. In general, low-resource and mid-resource languages benefit (+5 and +4.5 BLEU), and high-resource language scores are weakened (-0.7 BLEU) com- pared bilingual baselines. Similar to previous findings (Johnson et al., 2017) we find that a many- to-many setup outperforms a many-to-one setup.
2.1 Experimental Setup
Data To investigate the relationship between transfer and representation in multilingual machine translation, we conduct our experiments on the TED Talks corpus (Qi et al., 2018). The corpus comprises parallel data from 59 languages and is chosen over other large parallel corpora such as
urefs:l [aseemised]] elfooo/ok:111 L3 mmmocheesp//eriion222. P full results on TED, see Table 4

---

Be-En 4.5K
Az-En 5.9K
GI-En 10K
Sk-En 61K
De-En 167K
It-En 203K
He-En 211K
Ar-En 213K
Avg.
Train dataset size
109K
Neubig Hu (2018) (many-to-one)
18.3
11.7
29.1
28.3
21.6
Aharoni (2019) (many-to-many
21.7
12.8
30.7
29.5
33.0
35.1
33.2
28.3
28.04
Ours (many-to-one
23.8
14.3
34,9
33,4
36.3
38.5
36.5
31.3
31.1
Ours (many-to-man) manyy)
24.9
15.2
36.0
34.2
37.5
39.8
37.3
32.6
32.2
Table I: X-En test BLEU (tokenized) TED Talks corpus for language pairs from Aharoni al. (2019).
low (<10K) 12 languages
mid (10K-150K)
high (>150K)
representations of our multilingual model, and how this affects translation quality and transfer. Unlike previous studies that have focused on the investiga- tion of hidden encoder and decoder representations (Kudugunta et al., 2019), we concentrate on cross- attention, which connects the encoder and decoder. To investigate the degree of language invariance, sample semantically equivalent sentence triples S from dataset D, where S {-+, x² y). Here, x² and x² are sentences that originate from two different non-English source languages and E' while the language the target sentence L" is al- ways English. We then measure the average cosine similarity of the cross-attention vectors of all sen- tences in l and l' at different decoding time steps t:
23 languages
17 languages
bi
1.2
12.3
18.6
m2o
8.8 (18/012
14.64(20/3)
13.0
m2m
(12/0)
16.8-(21/0)
17.9*(9/55)
Table X X->En BLEU on FLORES-101 for bilingual (bi). many-to-one (m2o) and many-to-man (m2m) models. Re- sults are bucketed by number of training examples in TED. * (n//m) denote the fraction scores in bucket the are sig- nificantly better (x) or worse (m) to the bilingual baseline, according to bootstrap resampling.
Low-resource BLEU scores have a large standard deviation (#6,9), indicating that some languages benefit much more than others.
2.3 Representational View on Transfer
To further investigate the differences between multi- lingual and bilingual models, will now focus on understanding the underlying mechanics of knowl- edge transfer. Using translation quality alone as a measure of knowledge transfer inadequate, as differences in translation quality can have various causes, such as target data distribution (Fan et al., 2021). Therefore, in the following experiments, we aim to gain deeper insight into the mechanisms behind knowledge transfer in multilingual models, focusing on the many-to-many model, which pro- duced the highest translation scores.
Σ Σ c(xx(())))) x,(x .x))) SED-
Xsim
(1)
where c is the cosine similarity, xe(-,-) -) is the con- text vector, i.e., the result of encoder decoder cross- attention at decoding time step t, and D is subset of 'D that consists of sentence triples in source lan- guages l and f', and target language eT (English). We use FLORES- 101, consisting of 2,009 multi- parallel sentences. As we need multiple source sentences and a single target sentence, our analy- sis focuses on many-to-one directions. We only consider cross-attention within the final decoder layer in this analysis, and leave extensions to non- English target languages to future work.
encoder-decoder
When translating two semantically equivalent sentences from different source languages to the same target language, if the context vectors produced by the cross-attention mechanism are (almost) identical for every decoding timestep, the resulting translations will be the same. However, the reverse may hold true; it is possible for distinct context vectors to produce the same output, and these variations may correspond to specific aspects of the target language. The question of whether source language invariance is a desirable or even necessary trait for an mNMT model remains unresolved.
The resulting similarity matrix is displayed in Figure for eight languages. An xsim similarity value indicates that the encoder representations are identical for all decoding time steps, i.e., the representations are language invariant. Conversely, low similarity suggests that the representations dissimilar average, indicating that they are far from being language invariant. From the matrix, we can observe several patterns. High-resource languages tend have relatively high similarity with other high-resource languages. For instance, the similarity between French and Portuguese,
Language invariance Our goal is to determine the degree of language invariance in the encoder

---

X
2
g
20
026
ja (203K)
0.76
0.74
0.76
0.75
0.69
0.76
0.61
0
15
fr (190K)
0.76
0.78
0.86
0.84
0.69
0.82
0.59
BLEU
10
о me
fv
(131K)
0.74
0.78
0.78
0.77
0.67
0.59
Oft
pt (52K)
Cho
OOO
Oe
Do.
gl (10K)
0.75
0.84
0.77
0.86
0.68
0.81
0.58
Chu
m
-5
(5.8K)
0.89
0.69
0.67
0.69
0.68
0.69
0.61
-0.4
-02
0.0
0.2
Representational Transfer Potential
(5.6K)
0.76
0.82
0.81
0.69
0.59
Figure 2: The x-axis represents Representational Trans- fer Potentials (RTP), which measure the total transfer potential for language (as detailed in Equation 2), on FLORES-100 101. The y-axis illustrates the differ- ence in BLEU scores (multilingual BLEU bilingual BLEU) on FLORES-I01. The size of the dots indicates the bilingual BLEU score. The correlation coefficient (Spearman's p) is .77 and it is statistically significant (p < 0.001). The trend illustrates that a higher RTP value is positively associated with changes in transla- tion performance in multilingual setting.
(5.0K)
Figure 1: Average cosine similarities between context vectors (see Equation 1) for different source language combinations into English. Train data size is shown between brackets. The higher the similarity, the higher the degree of language invariance.
(s, pt. en). is 0.86, and between Greek and French, (cl fr. en). is 0.78. Furthermore, we find that some low-resource languages, such Galician (gl) and Bosnian (bs), have high similarities with high-resource languages. These languages benefit greatly from multilingual modeling, as evidenced by an increase of 16.8 and 20.1 BLEU points, respectively, compared to their bilingual scores. Other low-resource languages, such as Tamil (ta), do not have high similarities with high-resource languages. These languages do not benefit much from transfer, demonstrated by small increase of only 3.6 BLEU points in the case of Tamil. A full version of the similarity matrix can be found Appendix B.3.
ing into English, which can be thought of as an up- per bound for the potential transfer between and 8. (B (") is then weighted by the average repre- sentational similarity between & and l' when trans- lating into English, xsim .en) (see Liquation 1), RTP thus shows to what extent languages act as donor, i.c., benefiting other languages, or recipi- ent, i.e., benefiting from other languages. Positive transfer can occur when a language has better translation performance than f, which increases the weighted RTP score. Negative transfer can oceur when language é has worse translation per- formance than f, which decreases the score. It is important to note that RTP not score of lan- guage isolation, but rather a score language dataset in the context of other language datasets. Thus, RTP depends on the languages involved and the available resources in dataset.
Connecting representations to BLEU We quantify the potential for knowledge transfer into language E L from other languages e E L {(}, by connecting context vector similarity and transla- tion quality. To the best of our knowledge, this is the first approach that quantifies transfer at the rep- resentational level. We define the Representational Transfer Potential (RTP) as follows:
In Figure 2, we plot the resulting RTP scores on the x-axis, and the changes in BLEU scores in multilingual model versus a bilingual model on the y-axis. We observe strongly positive and sig- nificant correlation (p .77, 0.001), where higher RTP score implies increased translation performance, and a lower RTP score implies lower translation performance. Consider Hebrew (he), which has high similarities with lower perform- ing languages, and smaller similarities with better
AB(E,))
RTP
ssim (e)) AB(E,
(2)
where El( is the difference in bilingual BLEU scores between the languages when translat-

---

performing languages. Therefore, RTP can cor- rectly predict that Hebrew does not benefit from the multilingual setup, which is evidenced by its negative RTP score (-339) and decreased BLEU score 3.7). On the other hand, Bosnian (bs) has a relatively high RTP score of .28, meaning it is sim- ilar to languages with stronger translation quality. Bosnian is the language that benefits most from the multilingual setup (+200.1 BLEU). This means that the resulting differences in translation quality are due to knowledge transfer as captured by the RTP score, and not as a side effect of increased target data size. However, it is worth mentioning that this trend is not perfect and can only explain part of the transfer. For instance, Galician and Finnish have similar RTP scores (.23 and .21) but the increase in translation quality for Galician is far greater: 16.8 3.8 for Finnish. The discrepancies in RTP scores warrant further investigation (see next Section).
similarities xsim .)) defined in Equation 1.
3.1 Data Features and Linguistic Features Dataset size: The difference in training data size for two languages may serve as predictor for transfer. It is likely that low-resource language would benefit from a high-resource language. Let Sy denote the number of parallel sentences to En- glish for language L, and Sy be defined similarly for language E", We then compute the ratio of the smaller value to the larger value as follows:
S(EP) = min( Se) max S
(3)
Since xsim symmetric, we design features that are also symmetric, when applicable.
Vocabulary occupancy: We calculate the difference in vocabulary occupancy for and & The fraction of the vocabulary that is used by a language captures information about how well the subwords are optimized for that language. Let Va be the set of unique subwords in vocabulary V that are present in the training data St of language e. The vocabulary occupancy is then computed as: Vel/IV]. Ve defined similarly. The vocabulary occupancy ratio between l and is defined as:
To ensure the validity and generalizability of our RTP analysis findings beyond single test dataset (FLORES-IP 101), we incorporate an additional test dataset, NTREX-122 128 (Federmann et al., 2022). It consists of 1997 multi-parallel sentences in 128 languages. For NTREX-122
again observe strongly positive correlation (p= .73. < 0.001) between RTP and translation quality, further es- tablishing their relationship. See Appendix B.1 Figure for the corresponding plot. Additionally, the mean absolute RTP deviation per language on FLORES-10 101 and NTREX- 128 is 0.008, and the correlation is extremely robust ((p .99, p 0.001)). These results provide further evidence that RTP scores are consistent across different test sets, rather than being an artifact of a specific dataset.
min Mnin(VVil/VV] (Vll//VV VVeV max Mnax//V//IVV.
Vieee
(4)
Source subword overlap: We measure the simi- larity between the (subword) vocabularies of lan- guage l and language E.. This is calculated by taking the ratio of the number of subwords that are common to both languages (VV n Ver) and the total number of unique subwords in both languages (VV Vyrl) according to the following equation:
We also perform ablations on RTP and compare to linguistic baselines, which are detailed in Ap- pendix B. We conclude that RTP has far better correlation with translation quality compared to ablations and linguistic baselines.
Ven Vel Ve Vel
Oseote,e)
(5)
Finally, we show that RTP can be used to pick suitable auxiliary transfer languages. We find that training& language with its top 5 RTP contribu- tors leads to substantially better results of up to 6.8 BLEU, compared to training with its bottom 5 contributors. More results are in Appendix B.2.
We also investigated the use of sraquency-weightee subwords, which produced similar results. Multi- overlap: We are interested to see how generating identical target sentences (in En- glish) affects transfer. To calculate this, take the ratio of the number of multi muulti-arrallll sentences shared by the languages & and & denoted Sg n Se, to the total number of training sentences in both languages (Sp U St):
3 Analyzing Causes for Transfer
Next, we investigate characteristics that are relevant for transfer. Our objective is to use dataset and linguistic features to predict the representational
Se n Se
Sshared(t,e))
(6)
Se U St

---

Target n-gram overlap: We also measure the sim- ilarity between the generated target n-grams for the languages. This is similar to the (weighted) source subword overlap but applied to the target side. Let S(r,7)) be the set of aligned training sentence pairs of language & with pivot language ! (English is taken as pivot here). The (weighted) target subword overlap is then defined as:
Regressor
LR
MLP
GB
baseline (noise)
0.061
0.061
0.061
dataset size
0.052
0.052
0.046
datuset
vocabulary occupancy
0.041
0.041
0.035
multi malti-parallll overlap
0.047
0.042
0.034
source subword overlap
0.040
0.036
0.031
target subword overlap
0.050
0.046
0.042
geographic distance
0.062
0.053
0.049
linguistic
genetic distance
0.054
0.053
0.049
inventory distance
0.062
0.061
0,055
syntactic distance
0.051
0.050
0,050
Oigite,) ΣΣ n-g(S (p,)))) n-g(S((,)))) n,
phonological distance
0.061
0.061
0.052
all data
0.031
0.029
0.021
(7)
all linguistic
0.049
0.043
0.034
where n-g(-) is the n-gram count in sentence. We have also experimented with higher-order n-grams, and found similar results as unigram, thus we only report the results for unigram.
all data + all linguistic
0.028
0.025
0.016
Table 3: Mean absolute error (MAE) scores averaged over language pairs for transfer prediction, i.e., predict- ing (e, (similarity scores between languages and e when translating into English, see Equation 1) using data features and linguistic features (Section 3.1). Best scores per regressor in bold and per feature class underlined.
Linguistic features: We adopt five linguistic fea- tures, as described in Lin et al. (2019): geo- graphic distance, genetic distance (derived from language descent tree), inventory distance (kNN- based phonological inventory vectors, distinet from phonological distance), syntactic distance, and phonological distance.
3.3 Prediction Results
The results for predicting representational similari- ties are shown in Table 3. First, combined features lead to better MAE scores than single features. Us- ing all dataset features results in better predictions than using all linguistic features, and combining dataset and linguistic features results in best results for all algorithms. Furthermore, all single features have the potential to improve over naïve baseline (random input), indicating that they have least some predictive power.
3.2 Experimental Setup
We treat the prediction of the representational sim- ilaritics xsi,(g,er,") (see Equation 1) when trans- lating into target language !" (English) between source languages l and L' as regression prob- lem. We use the features described in the previ- subsection input variables. To account for variations in feature values across different lan- guage pairs, scale the features between 0 and 1. We consider all 52 source languages. Consider- ing that representational similarities are symmetric, and discarding combinations where = f", the re- sulting number of to be predicted representational similarities is (52-52)-5 52 1326. We use leave- one-out validatio approach, leaving out all similarities for a single language in each round. To evaluate the performance of the model, we use the average (over all language pairs) mean abso- lute error (MAE) as the evaluation metric. Since different machine learning algorithms have differ- ent inductive biases, we train and evaluate three regression models using the scikit-learn library (Pe- dregosa al., 2011 1): linear regression (LR), mul- tilayer perceptron (MLP), and gradient boosting (GB). The detailed hyper-paramett settings used for each model can be found in Appendix A.2.
3.4 Feature Importance
We investigate the importance of features to gain a better understanding of their role in transfer. Linear regression coefficients: Weight coeffi- cients are used as crude measure of feature impor- tance. These coefficients quantify the conditional association between the target xsim and given feature, while holding other features constant. The sign of the coefficients shows the direction of the association, and the magnitude is an indication of the strength of the association. In Figure 3, we can see that multi-paralle overlap, source subword overlap, and vocabulary occupancy have the largest positive weights among the data features, which implies that these features are positively associated with the target variable and have strong influence on the prediction. Furthermore, Genetic and Syn- tactic distance have the highest importance among the linguistic features.
2

---

Parallel datasets contain large number of hidden multi-parallel sentences that remain unused, and resurfacing these improves multilingual translation quality (Freitag and Firat, 2020; Xu et al., 2022). However, these approaches only add multi-parallel data, but do not explicitly exploit properties as part of the learning objective. In con- trast, we describe method that explicitly leverages the characteristics of multi-parallel data.
0.10
coefficients
target n-gram overlap
0.08
muult Seerraa allleel overray
geographic dist
phonologica dist
subword overlap
vocab occupanca
0.06
inventory dist
syntactic dist
genen dist
datasef size
0.04
0.02
We introduce an auxiliary similarity loss that en- courages context vectors be more similar when generating the same target token. When sampling a parallel batch, consisting of a source sentence a and the corresponding target sentence y, we optimize the cross cross-entropy loss as usual. When sampling a multi-parallel batch, consisting of meaning equíva- lent triples {x ,,²,y) (as defined in Section 2.3), such that y * x² and x + x², we optimize a similarity loss function:
0.00
Figure 3: Feature importance for transfer prediction: linear regression sign coefficients. Absolute values are plotted. Black line indicates negative coefficient (e.g.. genetic distance is negative).
Permutation importance: To further understand the importance of each feature, we additionally calculate permutation feature importance scores (Breiman, 2001; Fisher et al., 2019). This method evaluates the decrease in model score when sin- gle feature value randomly shuffled. This model- agnostic procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. The results using permutation fea- ture importance are consistent with the results ob- tained using linear regression coefficients. Specifi- cally, we find that multi-pare caatt-iparallll overlap the most important feature for all three regression models. Source subword overlap also important for ML. and GB, and slightly less for LR. Vocabulary oc- cupancy and dataset size also score relatively high on importance. Genetic distance is consistently the most important linguistic feature among all models. For more details, the permutation feature impor- tance plots can be found in Appendix B.4.
(x = Σ s(x((x ,y), xe(xx,1(200 .)), (8)
where s(.,-) a similarity function and xe(- is the context vector resulting from the cross-attention at decoding timestep t. The goal of minimizing Lxsim is to encourage representations that are in- variant across languages. The final learning objec- tive for multi-parallel batches (x x² ,y) combines minimizing Lxsim and cross-entropy (CCE):
2
( .=? ()) = Afrxsim 2,) +
fcr(r (9)
4.1 Experimental Setup
We follow the setup as described in Section 2. and make the following modifications: 1) we sample parallel and multi- multi-parallel batches in a 1:1 ratio, 2) for the multi-parallel batches, we optimize an auxil- iary cosine similarity loss and set weight À 1. To reduce the impact of a small number of dimen- sions that can dominate similarity metrics, known as rogue dimensions (Timkey and van Schijndel, 2021), we subtract the mean context vector ج from each context vector in the batch before calculating similaritics. If we sample batch where English is not the target, do calculate similarity loss, i.e., X 0. Note, that our method does not require a dataset that fully multi- multi-parallel. The paral- lel dataset consists of all X-+En and En-xX pairs. Its size 10M pairs. The multi murti-parallel dataset
4 Optimising for Representational Invariance
Some features that we have shown to be predic- tive transfer have been used in previous work. Higher vocab overlap leads to more positive trans- fer (Chung et al., 2020; Patil et al., 2022; Sun et al., 2022). Temperature sampling addresses dataset size (imbalance) (Arivazhagan al., 2019). Back- translated data can be used for similar effect (Liao al., 2021). Grouping languages by their linguis- tic similarity outperforms English-centric models (Oncevay et al., 2020; Fan et al., 2021). contrast, there are no such methods for multi multi-parallel parallel data.

---

FLORES-10
TED
low (<10K) 12 languages
mid (10K-150K)
high (>150K)
low (<10K)
mid (10K-150K)
high (>150K)
23 languages
17 languages
12 languages
23 languages
17 languages
many-to-mane
9.7
16.8
17.9
20.5
30.2
31.2
multi-parallel
(.9*(9/0)
16.9*(0/0)
17.7*(0/0)
20.8*(0/0)
30.1
31.0" $1.0*(0/0)
xsim
11.5*(12/0)
17.8-(19/0)
17.4"00/13)
2128
30.7 (04/73
30.4*(0/44)
many-to-one
8.8
11.6
15.0
19.9
27.9
27.2
+ muulti-paallel
8.6T(0/0)
(4.7*(0/0)
14.9*(0/0)
19.7~(0/0)
27 27.6*t0/0)
27 (7.0"(0/0)
Xsim
(0.8*
15.7 (5.7-(1/44)
(4.5*(
22 22.0*
28.8 (3.4/))))
26.6*(0/11)
Table X- BLEU on FLORES-10 101 and TED test for multilingual many-to-many and many-to-one models, compared to including multi-parallel batches during training (multi-parallel) and additionally adding auxiliary similarity loss (+ xsim). =nn/m) denote the fraction of scores in a bucket the are significantly better (n) or worse (m) the bilingual baseline, according to bootstrap resampling.
consists of all x²)) source combinations, with target fixed to English. The size is 5.9M triples. Additionally we perform an ablation experiment where we the similarity loss to 0, investi- gate the role of the loss versus the modified data sampling strategy. Note that we cannot ablate the multi-parallel batching, since the similarity loss requires multi-parallel batches.
a small price in performance. This trend holds for in- and out-of-domail test sets, and different types of multilingual models.
5 Related Work
Analyzing MNMT Investigating representations using Singular Value Canonical Correlation Analy- sis (SVCCA, Raghu et al., 2017) showed that en- coder representations cluster on linguistic similar- ity, and encoder representations are dependent on the target language (Kudugunta et al., 2019). Addi- tionally, the set of most important attention heads are similar across language pairs, which enables language clustering (Kim et al., 2021). Further- more, representations of different languages cluster together when they are semantically related (John- son et al., 2017; Escolano et al., 2019). In particular, visualising cross-and cross-attention per decoding time-step shows that meaning equivalent sentences generally cluster together (Johnson et al., 2017).
4.2 Results
We include results for our method on both in-domain (TED) and out-of-domain test sets (FLORES-101) for both many-to-man as well as many-to-one models. Table 4 shows BLEU scores and comparison to the baselines. Adding multi- parallel batches and our similarity loss yields im- provements for low- and mid-resource languages, both many-to-many and many-to-one models. Including multi-paralle parallel batches without applying a similarity loss leads to scores that are not sta- tistically significantly different from the baseline. Furthermore, many-to-many models have the best performance on all aggregated test score buck- ets. Lowest resource languages benefit most from this approach, with an average BLEU increase of +1.8 and +1.3 (many-to-many: This makes sense, since Lxsim encourages the representations of these languages to be more similar to other languages, most of which have better performance. Mid- resource languages also benefit from adding Lxsim: +1.0 and +0.5 average increase for FLORES- 101 and TED. Higher resource languages suffer from adding the auxiliary loss (-0.5 for FL0RES-101, -0.8 for TED). These results demonstrate that lower- and mid-resource languages improve when explicitly optimizing for language invariance using multi-paralle data. Higher-resoure languages pay
However, the extent of these phenomena has not been quantified per language. Moreover, these studies have primarily focused on representations in isolation, or its relation with linguistic similarity, with less focus on the role of representations knowledge transfer. In contrast, we explicitly connect representations to transfer, which allows for deeper understanding of the impact of transfer translation quality.
6 Conclusion
Previous research has primarily measured knowl- edge transfer in terms of BLEU scores, leaving open the question of whether improvements in translation quality are due to transfer or other fac- tors such as target data distribution. To address

---

this gap, we proposed new measure of knowl- edge transfer, Representational Transfer Potential (RTP), which measures the representational simi- larities between languages. We demonstrated that RTP is capable of measuring both positive and neg- ative transfer (interference). A key finding is that RTP is positively correlated with improved transla- tion quality, indicating that the observed improve- ments in translation quality are result of knowl- edge transfer rather than other factors. Addition- ally, we explored the role of dataset and language characteristics in predicting transfer, and found that multi mulltl-arrlllll overlap is highly predictive for the degree of transfer, yet under-explored in existing literature. We proposed a novel learning objective that explicitly leverages multi-parallel properties, by incorporating auxiliary similarity loss that encourages representations to be invariant across languages. Our results show that higher degree of invariance yields substantial improvements in translation quality in low- and mid-resource lan- guages.
for low-resource languages. Our method of explic- itly aligning representations likely results in less risk for low-resource languages, since the trans- lation quality is improved, and increased risk for high-resource languages.
References
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics, pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguis- tics.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neu- ral machine translation in the wild: findings and challenges. arXi::9907.05019 [cs]. ArXiv: 1907.05019.
Leo Breiman. 2001. Random Forests. Springer, 45:5- 32.
Acknowledgements
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clusteee vocabularies. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536-4546 Online. Association for Computational Linguistics.
This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project numbers VI.C. and VI. 222222.... We thank Ali Araabi, Yan Meng, Shaomu Tan and Di Wu for their helpful sugges- tions and insights.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Emerging cross- lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6022- 6034, Online. Association for Computational Lin- guistics.
Limitations
While our focus is on English-centric many-to- many and many-to-English models, it is impor- tant to note that there has been prior work that has explored non-English-centri setups, such as the studics by Fan et al. (2021) and Freitag and Firat (2020). This may present limitations in the general- izability of our results to other multilingual settings. While our analysis already uses 53 languages, did not measure to what extent our findings hold when using even more languages. Furthermore, our training data size relatively small which may affect model performance. We use TED instead of the larger OPUS-100 dataset, because TED has higher translation quality and consists of partly multi-parallel data.
Carlos Escolano, Marta R. Costa-jusså, and José A. R. Fonollosa. 2019. From bilingual to multilingual neu- machine translation by incremental training. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics: Student Re- search Workshop, pages 236-242, Florence, Italy. Association for Computational Linguistics.
Carlos Escolano, Marta Ruiz Costa-jussà, and José A. R. Fonollosa. 2022. Multilingual Machine Transla- tion: Deep Analysis of Language-Specife Encoder- Decoders. Journal of Artificial Intelligence Re- search, 13:1555-1552.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed KilKisskk, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave,
Broader Impact
In general, machine translation poses potential risks such as mistranslation. This risk is higher

---

Michael Auli, and Armand Joulin. 2021. Be- yond English-centre centric multilingual machine trans- lation. Journal of Machine Learning Research, 22(107):1-48.
Karthikeyan K. Zihan Wang. Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilin- gual BERT: empirical study. In International Conference on Learning Representations, Online.
Christian Federmann, Tom Koemi, and Ying Xin. 2022. NTREX-IBS 128 - News Test References for MT Evalu- ation of 128 Languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation. pages 21-24, Online. Association for Computational Linguistics.
Zae Myung Kim, Laurent Besacier, Vassilina Nikoulina, and Didier Schwab. 2021. Do Mul- tilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads? In Findings of the Association for Compulational Linguistics: ACL-IJCNLP 2021, pages 2832-2841. Online. Association for Computational Linguistics.
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. 2019. All Models are Wrong, but Many are Useful: Learning& a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously. Journal of Machine Learning Research, 20(177):1- 81.
Diederik P. Kingma and Jimmy L. Ba. 2015. Adam: a method for stochastic optimization. In International conference learning representations, San Diego, California.
Tom Koemi and Ondfej Bojar. 2018. Trivial transfer learning for low-resource neural machine translation. In Proceedings of the Third Conference on Machine Translation, pages 244 252, Belgium. Brussels. As- sociation for Computational Linguistics.
Markus Freitag and Orhan Firat. 2020. Complete mul- tilingual neural machine translation. In Proceed- ings of the Fifth Conference on Machine Translation, pages 548-558, Online. Association for Computa- tional Linguisties.
Sneha Kudugunta. Ankur Bapna, Isaac Caswell, and Orhan Firat. 2019. Investigating multilingual NMT representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing. pages 1565-1575, Hong Kong, China. Association for Computational Linguistics.
Markus Freitag. David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krish- nan, Mare' Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The FLORES-10 101 Evaluation Benchmark for Low-Resource and Multilingual Ma- chine Translation. arXiv:2106.03193 [cs]. ArXiv: 2106.03193.
Baohao Liao, Shahram Khadivi, and Sanjika Hewavitharana. 2021. Back-translation for Large-Scale Multilingual Machine Translation. In Proceedings of the Sixth Conference Machine Translation, pages 418-424, Online. Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwami, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neu- big. 2019. Choosing Transfer Languages for Cross- Lingual Learning. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 3 3125-3135, Florence, Italy, Asso- ciation for Computational Linguistics.
Thanh-Lee Ha, Jan Nichues, and Alexander Waibel. 2016. Toward multilingual neural machine transla- tion with universal encoder and decoder. In Proceed- ings of the International Workshop on Spoken Lan- guage Translation (IWSLT), Seattle, Washington.
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe. Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra. Arabella Sinclair, Dennis Ulmer. Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalathari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023. State-of- the-art art generalisation research in NLP: taxonomy and review. ArXiv:2210.0300 [cs].
Graham Neubig and Junjie Hu. 2018 Rapid adapta- tion of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 875-880, Brussels, Belgium. Association for Com- putational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado. Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: en- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020. Bridging linguistic topology and multilingual machine translation with multi-view language repre- sentations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pages 2391-2406 Online. Associa- tion for Computational Linguistics.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, page 311, Philadelphia, Pennsylvania. Association for Computational Lin- guistics.
from overfitting, The Journal of Machine Learning Research, 15(1):1929-155 1958..
Simeng Sun, Angela Fan, James Cross, Vishrav Chaud- hary, Chau Tran, Philipp Koehn, and Francisco Guzmán. 2022. Alternative Input Signals Ease Transfer in Multilingual Machine Translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5291-5305, Dublin, Ireland. Association for Computational Linguisties
Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. 2022. Overlap-base based Vocabulary Generation Im- proves Cross-lingual Transfer Among Related Lan- guages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing, Dublin, Ireland. Associ- ation for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, and Jon Shlens. 2016. Rethinking the inception architecture for computer vision. In IEEE Con- ference Computer Vision and Pattern Recogni- tion (CVPR). pages 2226, Las Vegas, Nevada. IEEE
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexan- dre Passos, David Cournapeau. Matthieu Brucher. Matthieu Perrot, and Edouard Duchesnay. 2011. Scikit-leam: Machine Learning in Python. Journal of Machine Learning Research, 2285522825.2880.
William Timkey and Marten van Schijndel. 2021. All Bark and No Bite: Rogue Dimensions in Trans- former Language Models Obscure Representational Quality. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4527-4546 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polasukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), Long Beach, Califor- nia. Neural Information Processing Systems (NIPS). ArXiv: 1706.03762.
Ofir Press and Lior Wolf. 2017. Using the output em- bedding to improve language models. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguis- tics, pages 157-163. Valencia. Spain. Association for Computational Linguistics.
Di Wu and Christof Monz. 2023. Beyond Shared Vo- cabulary: Increasing Representational Word Simi- larities across Languages for Multilingual Machine Translation. MxXiv22355.14188 [es].
Ye Qi, Devendra Sachan, Matthieu Felix. Sarguna Pad- manabhan, and Graham Neubig. 2018 When and why are pre-trained word embeddings useful for neu- ral machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Ax- sociation for Computational Linguistics, pages 529- 535, New Orleans, Louisiana. Association for Com- putational Linguisties.
Yulin Xu. Zhen Yang, Fandong Meng, and Zhou Jie. 2022. EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, Dublin, Ire- land, Association for Computational Linguistics.
Maithra Raghu, Justin Gilmer. Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: singular vec- tor canonical correlation analysis for deep learn ing dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30, pages 6076-6085. Curran Associates, Inc.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, pages 1715-1725, Berlin, Germany. Asso- ciation for Computational Linguistics.
Barret Zoph, Deniz Yuret, Jonathan May. and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1568-1575, Austin, Texas. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks

---

A Experimental Setup
metric
p
RTP
.77
.001
A.1 Translation Models
only A BLEU
.56
.001
only xsim
0.28
< .05
We the Transformer Base architecture (6 lay- ers, model dimension 512, hidden dimension 2048 and attention heads) and share all parameters be- tween language pairs (Ha et al., 2016; Johnson et al., 2017). We use Adam (Kingma and Ba, 2015) (31 - 0.9. 32- = 0.98 and = 10-9) to optimize a label smoothed (Szegedy et al., 2016) (smooth- 1ng=0.1) cross entropy loss function. To be able make of multilingual data within single system we use larget-language prefix tag to each source sentence (Johnson al., 2017; Ha al., 2016). We tie the weights of the decoder input em- beddings and the decoder softmax layer (Press and Wolf, 2017) and apply a 0.2 dropout rate (Srivas- tava et al., 2014) on the sum of the input- and posi- tional embeddings, the output of each sublayer. the output after the ReLU activation in each feedforward sublayer, and the attention weights.
genetic distance
.11
.30
inventory distance
-14
.30
syntactic distance
.14
.30
phonological distance
-.13
combined distances
-01
.30
Table 5: RTP ablations and linguistic baselines, calcu- lated on FLORES- 101.
20
os
15
no
BLEU
10
#o Ba
OR
6
O
O
O
OO
The resulting model has 93M trainable parameters. We use a batch size of 25k tokens. Following Neu- big and Hu (2018) and Aharoni et al. (2019), we do not use temperature sampling. Models are imple- mented in our open-source translation system. All models we train converge in approximately 2 days of training, using 4x NVIDIA TITAN V (12GB) GPUs.
O
-0,4
-0.2
0.0
0.2
Representational Transfer Potential
Figure 4: The x-axis represents Representational Trans- fer Potentials (RTP), which measure the total transfer potential for a language (as detailed in Equation 2), NTREX- 128. The y-axis illustrates the difference in BLEU scores (multilingual BLEU- bilingual BLEUU on NTREX-122 The size of the dots indicates the bilin- gual BLEU score.
A.2 Regressors
For MLP and GB, we report the average score over 3 random seeds. We do not report STD as it is negligible.
MLP For the multilayer perceptron, we used hidden layer dimensionality 80 using 3 layers. We use the ReLU activation function, Adam (Kingma and Ba, 2015) (31 = 0.9, B2 0.98 and 10 10-9).
Removing the xsim term in RTP gives p .56. and removing A BLEU results in p 0.77. The linguistic features do not correlate with BLEU dif- ference. We conclude that RTP has far better cor- relation with translation quality than linguistic dis- tances and ablations.
GB: For the gradient booster, we used squared error, 0.1 learning rate, and 100 estimators.
Figure 4 shows RTP scores calculated on the NTREX-128 (Federmann et al., 2022) dataset. The trend illustrates that a higher RTP value is posi- tively associated with changes in translation per- formance in multilingual setting. The correlation coefficient (Spearman's 's p) 0.73 and it is statis- tically significant (p < 0.001). Figures 4 and 2 (RTP vs delta BLEU on FLORES- 101) are highly similar, indicating that RTP generalizes to different test sets.
B Additional Results
B.1 Representational Transfer Potential ablations
Table shows ablations on RTP, and linguistic base- lines as described in Lin al. (2019). We calcu- late correlation coefficients (Spearman's p) on the metrics and the difference in BLEU scores (multi- lingual BLEU bilingual BLEU) on FLORES-101.

---

RTP kpp
ICTT min
Ba RTT upp
size
Bn RTP min
size
uk (0.75)
107K
-1
bn (0.59)
3.9K
1
hi (0.65)
178K
-1
(0.58)
195K
2
(0.75)
206K
-2
ta (0.6)
5.1K
2
(0.65)
9.3K
-2
gl (0.58)
9.9K
3
bg (0.73)
172K
my (0.6)
19K
3
(0.62)
5.7K
-3
pt (0.58)
52K
mk (0.72)
249K
mr (0.64)
9.3K
4
(0.62)
7.4K
-4
(0.59)
203K
(0.72)
136K
(0.64)
5.7K
5
hy (0.62)
203K
-5
nb (0.59)
16K
Table 6: RTP top and bottom contributing languages for Belarusian (first two columms) and Bengali (last two columns). Data sizes for the contributing languages into English are shown in columns size. We underline the smallest from the top or bottom, and use this size to subsample the larger one when creating the RTP top and bottom training sets.
B.2 Represenational Transfer Potential: Top 5 vs Bottom 5
For Belarusian and Bengali, find the top 5 and bottom 5 contributors their RTP scores. We then create a training set for both sets, by comparing the top and bottom data sizes and subsampling the largest such that has the same size as the smallest. This information is presented in Table
We then train a many-to-many system on the resulting datasets, after including Belarusian or Bengali and English. Results can be found in Ta- ble 7. We observe large discrepancies in scores for the top and bottom datasets, even though the dataset sizes are identical, for both in-domain (TED) and out-of-domain (FLORES-10) 101) settings. all cases, the model trained on the top 5 RTP con- tributors outperforms the one trained on the bottom 5 contributors. The difference is substantial: Be-En TED with the top RTP contributors scores 16.2 BLEU, whereas the system trained the bottom contributors results in 9.4 BLEU. These findings show that RTP can be used identify suitable aux- iliary transfer languages.
B.3 Encoder Representation Similarity
Figure 5 shows cross attention similaritics between all language combinations.
B.4 Permutation Feature Importances
See Figure 6 for the feature importance box plots.

---

FLORES-I01
TED
Be-Fn
Bn-En
Be-En
Bn-En
bilingual
0.5
0.4
6.1
6.8
mony-to-many (all)
7.0
6.1
24.9
19.3
many-to 1o-many (RTPup))
4.1
16.2
many-to-many (RTP min)
1.6
9.4
BLEU
+3.1
+2.5
+6.8
+6.7
Table 7: Be-En and Bn-En BLEU scores on FLORES-101 and TED. We compare systems trained on bilingual data with many-to-many systems trained all data (all), the top contributors RTP (RTPupp) and bottom 5 contributors to RTP (RTP
ATI
9009
60R91
6109
H109
5
5
1990
1291
Figure 5: Cross-attention similarities for all language combinations. Training data size into English depicted between brackets. (Zoom for better visibility.)

---

parallel everiap
sutrward overtap
genetic dist
vocab eccupancy
Earget e gram overap
dataset
phonological
inventory
0.10
0.55
020
0.5
0.35
0.40
sutti-sraalel
genetic
vocaa eccupancy
sutword averlap
syntactic dist
HD+
grographic dist
+++
phonelogical dit-
ivventory det
Langet n-gram destae
MLP
milti parailee overaa
subward overlap
occupancy
genetic
grographir
syntactic
Inventory dist
+0
phonological dal
+D-
0.0
0.4
0.8
100
Figure 6: Sorted permutation feature importance scores for LR (top), MLP (middle) and GB (bottom).


Reviews:
[{"review_id": "5OReRUbROJ", "reviewer": "Reviewer_sd5X", "summary": "", "questions": "Question A\n- If you exclude xsim from equation (2), what impact does it have on the correlation with the difference in BLEU scores?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n* RTP depends on the performance of bilingual translation models so it is not a robust metric.\n\nRegarding your comment that RTP depends on the performance of bilingual models and is therefore not robust; this is a valid point and it was also one of our concerns, which is why the paper lists RTP results on two different test sets from different domains: FLORES-101 (Wikipedia) and NTREX (News). Note that the bilingual model scores are, of course, different for FLORES-101 and NTREX. In lines 321-336 we discuss that both FLORES-101 and NTRES show a strongly positive correlation between RTP and translation quality: 0.77 and 0.73 (p<0.001), respectively. Furthermore, the mean absolute RTP deviation per language on these test sets is only 0.008, and the correlation is extremely robust (0.99, p<0.001), which empirically demonstrates the robustness of RTP. Finally, note that we train on TED and choose different domains for the test sets, instead of relying on the TED test data. All RTP results are on out-of-domain FLORES-101 or NTREX, *not* on TED, which further helps to ensure robustness of our findings.\n\n* If you exclude xsim from equation (2), what impact does it have on the correlation with the difference in BLEU scores?\n\nRegarding your question about excluding xsim from Equation 2, this is a good point which we had already included in our submission by doing ablations on RTP (we mention this in lines 337-341 and link to Appendix B.1). To summarize, excluding xsim from RTP results in a moderate correlation of 0.56 (p<0.001), versus a strong correlation of 0.77 (p<0.001) when including xsim. We conclude that RTP has far better correlation with translation quality compared to ablations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* They demonstrated the effectiveness of knowledge transfer in low- and middle-resource languages and identified the crucial attribute for knowledge transfer in multilingual translation.", "reasons_to_reject": "* RTP depends on the performance of bilingual translation models so it is not a robust metric.", "questions_to_authors": "Question A\n- If you exclude xsim from equation (2), what impact does it have on the correlation with the difference in BLEU scores?", "paper_topic_and_main_contributions": "The authors investigate the correlation between cross-lingual information flow and cross-attention similarity across different languages. Their proposed novel method, called Representational Transfer Potential (RTP), allows for observing knowledge transfer and inferring facts that BLEU scores cannot capture. A significant discovery is that there is a positive correlation between RTP and improved translation quality, suggesting that the enhancements in translation quality are attributable to knowledge transfer rather than other factors. They also propose a novel learning objective that explicitly utilizes multi-parallel properties.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GU3CVGiuay", "reviewer": "Reviewer_kGrk", "summary": "", "questions": "- TED seems to be a special case with high-quality multi-parallel data. However, it is not obvious whether there exists sufficient multi-parallel data for large-scale datasets like CCMatrix. Even if they do exist, their quality should also be assessed, as it might be attributed to the artifact of the mining strategy and not true multi-parallel data. Have you inspected this for other datasets?\n- In section 2.3, it is stated that \"low similarity suggests that the representations are dissimilar on average, indicating that they are far from being language invariant.\" Based on this, it is not clear why minimizing xsim loss results in invariant representations. Can you provide more detailed explanations?\n- Are there any experiment results about ablating xsim loss and multi-parallel batching?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Two variables in Figure 2 (RTP and BLEU difference) are not fully independent. For example, high-resource languages tend to have low RTP (due to the negative numerator in equation 2) and high BLEU difference (as shown in Table 2). More rigorous justifications should be provided.\n\nRegarding your comment about RTP and BLEU not being fully independent: this is a good observation, and we agree they are not fully independent. (Note that RTP (Equation 2) uses *bilingual* models and their delta BLEU scores, whereas Figure 2 plots differences with *multilingual* model BLEU scores.) However, RTP ablations in Table 5, Appendix B, indicate that using RTP (i.e., xsim + delta bilingual BLEU) leads to a 0.77 correlation coefficient, whereas removing the xsim part leads to a substantially lower correlation of 0.56. This indicates that the correlation between bilingual and multilingual BLEU can explain only part of the correlation, and including xsim leads to a substantially stronger correlation of 0.77.\n\n* The use of xsim loss is not well motivated. For example, in section 2, only RTP (not xsim) is explicitly connected with BLEU, so the motivation should be stated more concretely.\n\nRegarding your comment about the motivation of xsim: we provide motivation at the beginning of Section 4, see lines 494-513. To summarize, we identified features that are relevant for transfer, and point out related works that have used these features to improve mNMT transfer. We found that multi-parallelism is consistently the best predictor for transfer, and so far no explicit learning objectives have been proposed in the literature that tap into the unique characteristics of this type of data. Our method in Section 4 fills this research gap. (The goal of RTP is to investigate transfer in mNMT models by connecting it to translation quality as measured in BLEU.) We’ll make sure to update the text to make this more clear.\n\n* TED seems to be a special case with high-quality multi-parallel data. However, it is not obvious whether there exists sufficient multi-parallel data for large-scale datasets like CCMatrix. Even if they do exist, their quality should also be assessed, as it might be attributed to the artifact of the mining strategy and not true multi-parallel data. Have you inspected this for other datasets?\n\nRegarding your question about high-quality multi-parallel data: related works [1,2] have shown how to extract multi-parallel sentences in large datasets such as WMT. For example, [1] found that when considering 6 languages in WMT, 123M sentences are available in 2 languages, 6.9M in 3 languages, 5.4M in 4 languages, 0.7M in 5 languages, and 10K in all 6 languages (from Table 3 in [1]). Furthermore, there exist several corpora that are multi-parallel by design, such as UN (5 languages, 11M sentence pairs), Europarl (20+ languages, up to 2M sentence pairs), EMEA (20 languages, 1M sentence pairs). Finally, there are ongoing efforts that investigate multi-parallelism (quantity, quality) for larger scale datasets such as CCMatrix.\n\n* In section 2.3, it is stated that \"low similarity suggests that the representations are dissimilar on average, indicating that they are far from being language invariant.\" Based on this, it is not clear why minimizing xsim loss results in invariant representations. Can you provide more detailed explanations?\n\nRegarding your question about why minimizing xsim loss results in invariant representations: L_{xsim} uses a cosine similarity loss which we point out in the experimental setup, Section 4.1 lines 537-387. When the cosine similarity is 1 (x and y are identical), the loss is 0. We’ll make sure to update L_{xsim} Equation 9 and corresponding text to make this more clear and avoid confusion.\n\n* Are there any experiment results about ablating xsim loss and multi-parallel batching?\n\nRegarding your question about ablating xsim loss and multi-parallel batching: we ran additional experiments *with* multi-parallel batching and *without* xsim loss. To summarize, these results are almost identical to the multilingual baseline. For many-to-many, the scores are as follows (baseline / ablation: only multi-parallel / xsim): low-resource: (9.7/9.9/11.5) ; mid-resource (16.8/16.9/17.8); high-resource (17.9/17.7/17.4). The other ablation (only xsim without multi-parallel batching) is not possible since xsim requires multi-parallel batches; we cannot calculate it otherwise. We’ll make sure to include these results in the updated version of our paper.\n\n\n[1] Markus Freitag and Orhan Firat. Complete multilingual neural machine translation. WMT 2020.\n\n[2] Yulin Xu, Zhen Yang, Fandong Meng, and Jie Zhou. EAG: Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a novel representation metric to explain multilingual knowledge transfer.\n- Analysis for the origin of xsim is provided, and the proposed auxiliary loss results in BLEU improvements for low- and mid-resource languages.", "reasons_to_reject": "- Two variables in Figure 2 (RTP and BLEU difference) are not fully independent. For example, high-resource languages tend to have low RTP (due to the negative numerator in equation 2) and high BLEU difference (as shown in Table 2). More rigorous justifications should be provided.\n- The use of xsim loss is not well motivated. For example, in section 2, only RTP (not xsim) is explicitly connected with BLEU, so the motivation should be stated more concretely. \n- Baseline results in Table 4 seem to be the same as the results from section 2. However, the proposed method contains two modifications (different sampling of batches and an additional loss), so ablations for each modification should also be provided to verify the effect of xsim loss.", "questions_to_authors": "- TED seems to be a special case with high-quality multi-parallel data. However, it is not obvious whether there exists sufficient multi-parallel data for large-scale datasets like CCMatrix. Even if they do exist, their quality should also be assessed, as it might be attributed to the artifact of the mining strategy and not true multi-parallel data. Have you inspected this for other datasets?\n- In section 2.3, it is stated that \"low similarity suggests that the representations are dissimilar on average, indicating that they are far from being language invariant.\" Based on this, it is not clear why minimizing xsim loss results in invariant representations. Can you provide more detailed explanations?\n- Are there any experiment results about ablating xsim loss and multi-parallel batching?", "paper_topic_and_main_contributions": "This paper studied knowledge transfer in multilingual translation by analyzing representation similarities between cross-attention outputs. Concretely, it proposes the representational transfer potential (RTP) and analyzes its relation to translation quality. Furthermore, the paper analyzes the origin of transfer and proposes a new auxiliary similarity loss to encourage language-invariant representations and improve translation quality.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "okZjRwQ5bb", "reviewer": "Reviewer_4uHs", "summary": "", "questions": "Q1: In Table 1, the baselines are compared to the proposed Many-to-Many model. I was wondering why the authors didn't provide the results of their Many-to-One model? \n\nQ2: This work is an English-centric proposed model as stated in the Limitation Section,  was there any evaluation on pairs of languages not involving English? I was wondering why the target language is always English? Is it because your model is not yet suited for other target languages?\n\nQ3: While there is a little drop in performance for high-resource languages, I was wondering if the authors thought about adapting the  multi-parallel batch and using it only for  low- and mid-resource pairs and see if there is an impact on high resource languages evaluation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n* Q1: In Table 1, the baselines are compared to the proposed Many-to-Many model. I was wondering why the authors didn't provide the results of their Many-to-One model?\n\nWe only show the many-to-many (m2m) model and not the many-to-one (m2o) model in Table 1 because the m2m model is stronger (this observation is in line with related work [1]), hence we use that model for subsequent experiments and analyses. For completeness, the m2m vs m2o scores for the languages listed in Table 1 are: Be-En (24.9 vs 23.8), Az-en (15.2 vs 14.3) Gl-En (36.0 vs 34.9) Sk-en (34.2 vs 33.4), De-En (37.5 vs 36.3), It-en (39.8 vs 38.5), He-En (37.3 vs 36.5), Ar-en (32.6 vs 31.3), Average (32.2 vs 31.1). We will include the m2o numbers in Table 1.\n\n* Q2: This work is an English-centric proposed model as stated in the Limitation Section, was there any evaluation on pairs of languages not involving English? I was wondering why the target language is always English? Is it because your model is not yet suited for other target languages?\n\nMultilingual NMT models are often English-centric, and since we investigate this model class in our paper we are limited to English on the source side, or English on the target side. (While these models have 0-shot capabilities, results are generally poor compared to English-centric directions, which is why we excluded them.) Additionally, our analysis uses cross-attention similarities, for which we require sentences from two different source languages, and a sentence from a *fixed* target language. The question we want to answer is: how similar are the 2 source representations when translating into the target? We measure this on the token-level. \n\n* Q3: While there is a little drop in performance for high-resource languages, I was wondering if the authors thought about adapting the multi-parallel batch and using it only for low- and mid-resource pairs and see if there is an impact on high resource languages evaluation?\n\nThis is an interesting idea which we also thought about and experimented with, but it led to poor results. We think this is because we do not simply want all representations to be invariant, but we specifically want the “weak” low-resource representations to be similar to the “strong” high-resource representations. This requires multi-parallel batches that mix {low,mid}-resource languages with high-resource languages. In the limit, the {low,mid}-resource sentences are most often combined with high-resource sentences, which draws them closer together. We suspect the high-resource language quality deteriorates a little because their representations are slightly drawn towards the “weak” {low,mid}-resource representations as a result of the xsim loss. We tried to mitigate this using weighting, such that {low,mid}-resource representations are drawn to high-resource but not vice versa, but results did not significantly improve and it complicated our method, so we did not include it in the paper.\n\nFinally, note that we conduct experiments on three (not two) datasets; in addition to TED (for training) and FLORES-101 (evaluation, Wikipedia domain) we also use the NTREX dataset (evaluation, News domain). We used multiple test sets to ensure our method is robust.\n\n[1] Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes and Jeffrey Dean. Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. TACL 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and well written\n- Strong evaluation \n- The proposed method shows significant improvements overall (except highly-resource languages)\n  \n", "reasons_to_reject": "- No evaluation on other target language than English", "questions_to_authors": "Q1: In Table 1, the baselines are compared to the proposed Many-to-Many model. I was wondering why the authors didn't provide the results of their Many-to-One model? \n\nQ2: This work is an English-centric proposed model as stated in the Limitation Section,  was there any evaluation on pairs of languages not involving English? I was wondering why the target language is always English? Is it because your model is not yet suited for other target languages?\n\nQ3: While there is a little drop in performance for high-resource languages, I was wondering if the authors thought about adapting the  multi-parallel batch and using it only for  low- and mid-resource pairs and see if there is an impact on high resource languages evaluation?", "paper_topic_and_main_contributions": "This work addresses mNMT translation quality under the prism of knowledge transfer. \nThe question the authors try to solve is whether the improvements in translation quality are due to knowledge transfer or due to other reasons. \nTo answer this question, this paper proposes a new measure of knowledge transfer called Representational Transfer Potential (RTP) which computes similarities between languages in both positive and negative transfer. \nOne important finding is that RTP correlates with improved translation quality. Also, the authors explored the role of data sets and language characteristics impact in predicting transfer, showing that multi-parallel overlap is the main indicator for the degree of transfer. \nFinally, the authors propose a new learning objective via an auxiliary similarity loss in order to push  representations to be invariant across languages.\nThe experiments were conducted on two data sets that is: TED Talks and Flores-101.\nThe results show the effectiveness of the proposed method for low and mid-resource languages. While the results slightly drop for highly-resource languages. \n\n\n\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that cross-attention similarity in a multilingual NMT system is correlated with translation quality, indicating that higher quality is linked to knowledge transfer. It also demonstrates gains on lower-resourced languages from an auxiliary loss function that encourages representational similarity in multi-way parallel data.

Reviewers were mostly convinced by the analysis, finding the proposed cross-attention similarity metric novel, and the demonstrated link between knowledge transfer and quality credible. They also highlighted the positive results from use of the auxiliary representational loss function. They had reservations about potential lack of generality due to reliance on multi-way parallel data, the fact that the similarity metric depends on parallel data, and experiments on into-English directions only. They also raised questions about a number of smaller technical issues.

This seems to be a nice addition to work on multilingual NMT. The demonstration that knowledge transfer is a key determinant of quality, while not surprising, is a useful and solid contribution. Similarly, the loss function, while roughly in line with previous proposals, is novel in exploiting multi-way parallel data, which recent work has shown to be highly beneficial. The authors did an excellent job of addressing reviewers’ questions, and I don’t have any major concerns about the paper’s technical merits.
Decision: Accept-Findings