Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation
Authors: Di Wu
Christof Monz
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual Machine Translation
Shared Vocabulary
TL;DR: We inject priors of meaning-equivalent words into embedding table to enhance knowledge transfer for multilingual machine translation.

Abstract: Using a shared vocabulary is common practice in Multilingual Neural Machine Translation (MNMT). In addition to its simple design, shared tokens play an important role in positive knowledge transfer, which manifests naturally when the shared tokens refer to similar meanings across languages. However, when words overlap is small, e.g., using different writing systems, transfer is inhibited. In this paper, we propose a re-parameterized method for building embeddings to alleviate this problem. More specifically, we define word-level information transfer pathways via word equivalence classes and rely on graph networks to fuse word embeddings across languages. Our experiments demonstrate the advantages of our approach: 1) the semantics of embeddings are better aligned across languages, 2) our method achieves evident BLEU improvements on high- and low-resource MNMT, and 3) only less than 1.0\% additional trainable parameters are required with a limited increase in computational costs, while the inference time is identical to baselines.

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation
Di Wu Christof Monz Language Technology Lab University of Amsterdam {d. wu, c. mono2)@uva. nl
Abstract
biegyle)
bicyole
Egi bicgyole]
Using vocabulary that is shared across lan- guages is common practice in Multilingual Neural Machine Translation (MNMT). In ad- dition to its simple design, shared tokens play an important role in positive knowledge trans- fer, assuming that shared tokens refer to similar meanings across languages. However, when word overlap is small, especially due to dif- ferent writing systems, transfer is inhibited. In this paper, we define word-level informa- tion transfer pathways via word equivalence classes and rely on graph networks to fuse word embeddings across languages. Our ex- periments demonstrate the advantages of our approach: 1) embeddings of words with similar meanings are better aligned across languages, 2) our method achieves consistent BLEU im- provements of up to 2.3 points for high- and low-resource MNMT, and 3) less than 1.0% ad- ditional trainable parameters are required with a limited increase in computational costs, while inference time remains identical to the baseline. We release the codebase to the community.
station
Q
6
ficts
E Ee[fists]
E [Rede]
Figure I: bicycle and fiets have the same meaning, but use different forms, potentially leading to larger dis- tance 5 between their embeddings (E(-)). Our graph- based module G explicitly reparameterizes the word embeddings (Ea(-) leading to a reduced distance 8'.
lingual word embedding or bilingual lexicon in- duction, i.e., word translation. This point is also held for higher-level representation learning, e-g., multilingual BERT (Devlin et al., 2019), where Pires et al. (2019) show that knowledge transfer is more pronounced between languages with higher lexical overlap. For bilingual machine translation, Aji et al. (2020) investigate knowledge transfer in a parent-child transfer setting² and reveal that word embeddings play an important role, particularly if they are correctly aligned.
All findings above point to the importance of word-level knowledge transfer for downstream sys- tems, no matter whether this transfer is achieved through sharing, mapping, or alignment. To some extent, knowledge transfer in multilingual trans- lation systems (Johnson et al., 2017) can be seen as a special case in the aforementioned bilingual translation setting. where model parameters and vo- cabulary are both shared and trained from scratch, and then knowledge transfer should emerge natu- rally. However, the shared vocabulary, as one of the core designs, has limitations: 1) When languages use different writing systems, there is little word overlap and knowledge sharing suffers. 2) Even if languages use similar writing systems, shared tokens may have completely different meanings in
1 Introduction
Multilingual systems (Johnson et al., 2017; Lam ple and Conneau, 2019) typically use shared vo- cabulary to build the word space uniformly. For instance, in MNMT scenarios, this is achieved by combining all source and target training sentences together and training a shared language-agnostic tokenizer, e.g., BPE (Sennrich et al., 2016), to split the words into tokens.
Such design is simple and scales easily. More- over, shared tokens also encourage positive knowl- edge transfer when they refer to equivalent or sim- ilar meanings. Research on cross-lingual word embeddings (Søgaard et al., 2018; Ruder et al., 2019) shows that exploiting weak supervision sig- nal from identical words remarkably boosts cross-
Normally, it refers to tuning the child model on the pre- trained parent model, while the embedding table could he fully shared, partially shared, or not shared at all.
hhttps://gitee wttp://ww..cmmm/ooe3330000 BeyondShareeWoocabulary

---

different languages, increasing ambiguity. In this paper, we target the first issue of word- level knowledge sharing across languages. As il- lustrated in Figure I, due to different surface forms, bicycle in English and fiets in Dutch have the same meaning but are placed differently in the embed- ding space, unlike the shared word station. Our goal is to pull together embeddings of meaning- equivalent words in different languages, including languages with different writing systems, facilitat- ing knowledge transfer for downstream systems. For simplicity, we choose English as the pivot and encourage words in other languages to move closer to their English counterparts.
some specific words with their cross-lingual equiv- alent counterparts, a downstream cross-lingual part- of-speech tagging task can benefit from its high- resource counterpart. Using cross-lingue word em- bedding (Søgaard et al., 2018; Ruder et al., 2019) can be seen as an extension of this idea at the rep- resentation level. E.g., by fixing and applying pre- trained embeddings to cross-lingual tasks.
For transfer learning in bilingual NMT, Amrhein and Sennrich (2020) show that the child and par- ent models should share substantial part of the vocabulary. For languages with different scripts, vocabulary overlap minimal and transfer suffers.
To alleviate this issue, they leverage romanization tool, which maps characters in various scripts to Latin characters, often based on pronunciation, to increase overlap. However, romamization has some limitations because 1) romanization operates at the character level, sometimes including pronunciation features, and hence benefits are mostly limited to named entities, and 2) the process of romanization is not always reversible.
To this end, we define and mine subword-level knowledge propagation paths and integrate them adjacency
into a graph which is represented matrix. Then, leverage graph networks (Welling and Kipf, 2016) to model information flow. The em- beddings of meaning-equie equivalent words will trans- fer information along with the paths defined in the graph and finally converge into a re-parameterized embedding table, which is used by the MNMT model as usual. At higher level, our approach establishes priors of word equivalence and then in- jects them into the embedding table explicitly encourage knowledge transfer between words in the vocabulary.
Sun et al. (2022) extend surface-level characters normalization and apply it to multilingual transla- tion. Besides romanization signals, they further introduce phonetic and transliterated signals aug- ment training data and unify writing systems. Their approach also shows the merits of larger vocabulary overlaps. However, the aforementioned limitations of romanization and transliteration still hold.
We choose multilingual translation as test bed to investigate the impact of word- baseb knowledge transfer. Several experiments show the advantages of our approach: 1) Our re- parameterized embed- ding table exhibits enhanced multilingual capabili- ties, resulting in consistently improved alignment of word-level semantics across languages, encour- aging word-level knowledge transfer beyond iden- tical surface forms. 2) Our method consistently outperforms the baseline by a substantial margin (up to 2.3 average BLEU points) for high- and low- resource MNMT, which empirically demonstrates the benefits of our re-parameterized embeddings. 3) Our method scales: The extra training time and memory costs are small, while the inference time is exactly the same as benchmark systems. More- over, we demonstrate our method adapts to massive language pairs and large vocabulary.
Compared to previous work, our method is more general and practical. We mine equivalent words at the semantic level and inject priors into embed- dings to improve knowledge transfer. Firstly, it avoids the flaws in romanization- or pronunciation- based approaches. Secondly, the graph- based trans- framework naturally adapts to multilingual sce- narios, where many-to-many equivalences exist.
Reparameterization Framework
In this section, we describe 1) how to build an equivalence graph using pair-wise data and 2) how to re-parameterize the embeddings via our graph- based approach. Note that in the discussion below we use the term 'word' as referring to an actual word or subword if word segmentation approach such as BPE (Sennrich et al., 2016) used.
2 Related Work
3.1 Equivalence Graph Building
Prior work has demonstrated the importance of word-level knowledge transfer. For instance, Gouws and Søgaard (2015) show that by replacing
We define the words with the same meaning in different languages as equivalent words classes,

---

Encoder
Decoder
Lookup
Lookup
Blli
Merge
Fahrrad
Fiets
Figure 2: Illustration of our framework. The left part denotes the subgraphs we build for each language pair, e.g.- EN-DE and EN-NL, which are further merged into multilingual graph. Since we only rely on English-cente centric data, the graph sparse, and only four (of the nine possible) sub-matrices are filled. As shown the right part, the information from the original embeddings (in grey) transfer and converge into the re-paramcterized embeddings (in blue) along the pathways defined in the graph, which are further used by a standard encoder-decoder model. All parameters, including embeddings, are trained from scratch.
e.g., (bicycle, bike, Fahrrad, all mean bike and therefore belong to the same equivalence class, This semantic equivalence involves many-to-many relationships across languages, which we represent in graph follows. For these words, aim transfer information with each other through the graph and make them converge into similar embedding representations.
aligned with each other across D3,t-
A higher ratio is derived from greater pair- wise occurrence in the bitext. This is based on the intuition that when pair of aligned words frequently co-occur, they 1) have higher confidence as equivalent words, and 2) the knowledge sharing between these two will benefit more context during training. The corresponding bilingual equivalence graph G can be induced by filling an adjacency matrix using Fi.5" By element-wise summation of multiple Ge.t, can merge multiple bilingual graphs into a single multilingual graph G:
Given a shared vocabulary V with size [V], we define the corresponding equivalence graph as an adjacency matrix GIV [ VI, Each point Dtj E G is a non-negative real number between 0.0 to 1.0 and denotes the ratio of information transferred from the word by to word i; in V. In our approach, mine equivalent words class via word alignment and define the corresponding transfer ratios base on them.
G
Σ GH
(2)
DDDCD
and further normalize G to guarantee that each sums 1.0.
More formally, let be the entire (multilingual) bitext corpus, Da,: S D denotes subset of parallel data with translation direction from source lan- guage to target language t. We train and extract all of the subword-level alignments for each Dx,, separately. For words v and uy in V, we define an alignment probability from 0y to 0 Dx, as x.t corresponding transfer ratios gi as follows:
Figure 2 illustrates our equivalence graph G for two language pairs, i.e., EN-DE and EN-NL. It is worth noting that, due to only English-centric align- ments being extracted, the spaces of non-English di- rection are usually empty (such as German-Dutel For practicality, in this paper, we choose English as the pivot and define transfer paths based align- ments between English and other languages. It is worth mentioning that alternative approaches, like leveraging multilingual vocabulary (Lample et al., 2018b 8b) or multilingual colexification graphs (Liu et al., 2023), could be used as well.
81 2.
(1)
Σ
where at is the number of times both words are o/

---

3.2
GNN based Messaging Passing
converges to the re-parameterized table. Then, the downstream standard MNMT system looks up corresponding word embeddings from the re- parameterized table. The whole architecture is end- to-end and supervised by the translation objective. It is worth noting that, although the graph we build in Section 3.1 only contains English-centric pathways, knowledge transfer beyond English- centric directions can also be handled through the multi-hop mechanism. E.g.. words in Dutch will transfer information to English counterparts first and then further propagate to other languages, e.g.. German, by a 2-hop mechanism. We empirically evaluate this point in Section 5.2.
Section 3.1, we model the priors of word equiv- alences as a graph. Here, we show how to inject such priors into embeddings via graph networks, thereby re- parameterizing the embedding table.
The graph can be represented matrix GIV|IIVI see Section 3.1. Let E E IR V be the original embedding table containing all [V] original word embeddings, where is the represen- tation dimensionality.
adjacency
Given G and E, we can easily define various information transfers on the graph as matrix opera- tions. E.g., a weighted sum of meaning-cquivalens word embeddings can be defined as E' = G E.
Graph networks (Welling and Kipf, 2016) ex- tend such operations in multilayer neural network fashion. In each layer, non-linear functions and learnable linear projections are also involved to adjust the aggregation messages. Here, the re- parameterized word embeddings derived from the first-layer graph network are defined as follows:
Experiments and Results
In this section, we apply our approach train mul- tilingual translation models under different config- urations.
4.1
Experimental Setup
4.1.1 Datasets
cf pWWie + W2 Σ 9i, ey + b), JEV())
(3)
We conduct experiments on two datasets, the smaller IWSLT14 benchmark and our own large- scale dataset called EC30.
where e; E(us], i.e., the embedding of word ox and W1, W/2 Raxd and b E Rd are learnable pa- rameters, and p a non-linear activation function, such as ReLu (Glorot et al., 2011). (z)) denotes a set of neighbors of i-th node the graph, i.e., the aligned words with 9t, > 0. Respectively, W1 learns the projection for the current word embed- ding e; and W2 learns for each neighbors ())A((i)- Equivalently, can rewrite Equation 3 in matrix fashion follows:
For IWSLT14, we follow the setting of Lin et al. (2021) and collect English-centric language pairs, with size ranging from 89k to 169k. The data pro- cessing script follows Tan al. (2018).
To ensure more diverse and inclusive large- scale evaluation, we used the EC30 dataset de- rived from EC40 (Tan and Monz, 2023), where we excluded the data of 10 super low-resource lan- guages. The EC30 dataset consists of 61 million English-centric bilingual sentences as training data, covering 30 Engllsh languages across wide spectrum of resource availability, ranging from High (5M) to Medium (IM), and Low (100K) re- sources. Each resource group consists of languages from families with multiple writing systems. We choose Ntrex (Federmann et al., 2022) and Flores- 101 (Goyal et al., 2022) as our validation and test datasets, respectively. A more detailed description of datasets is provided in Appendix A.I.
PEW, + GEW, + B).
(4)
To allow the message to pass over multiple hops, we stack multiple graph networks and calculate representations recursively as follows:
Ed+1 = p(E'Wi + GE"W,, +B"), (5) where h is the layer index, i.e. hop, and E is equal to the original embedding table E. The last layer representation E is the final re-parameterized em- bedding table, for the maximum number of hops H, which is then used by the system just like any vanilla embedding table.
We tokenize data with Moses (Koehn al., 2007) and use SentencePiece with BPE (Sennrich et al., 2016) with 30K and 128K merge operations
Figure illustrates the overall architecture. The information from the original embedding table propagates through multi-hop graph networks and
' httpps///gtthbb cwt.oub..cco///eeeeeemm mul hww/prepprr-iwss111.. "https: thub. htm//oogll//eeeencpiicee

---

Model
DE
ES
FA
AR
HE
NL
PL
IT
EN-X
X-EN
AVG
Baseline (Lin al., 2021)
28.1
35.2
16.9
20.9
29.0
30.9
16.4
29.2
25.8
LASS (Lin et al., 2021)
29.8
37.3
17.9
22,9
30.9
33.0
17.9
30.9
27.6
Our Baseline
28.5
36.0
17.4
20.2
27.9
31.5
17.6
29.7
24.4
27.8
26.1
Weighted Sum
29.2
36.7
18.1
20.9
28.5
32.2
18.2
30.5
24.8
28.7
26.8
GraphMerge- lhop
30.2
37.5
19.0
21.7
30.0
33.4
18.8
31.3
25.4
30.0
27.7
GraphMerge-2hog
30.4
37.9
19.0
21.9
30.0
33.7
19.2
31.6
25.5
30.5
28.0
GraphMerge-3hop
30.7
38.2
19.9
22.3
30.1
34.0
19.4
32.2
25.4
31.3
28.4
3-hop Gain
+2.2
+2.2
42.5
42.1
+2.2
+2.5
+1.8
+2.5
+1.0
+3.5
+2.3
Table 1: Results on the IWSLT14 dataset. Following previous work (Lin et al., 2021), we report average out-of- and into-English BLEU scores. For instance, the numbers on DE column are the average of EN-DE DE-EN BLEU scores. EN-X and X-EN denote the average performance on 8 language pairs. We show the results from Lin et al. (2021) in the first block, which learns language-spechin sub-network for MNMT. Also, we report our reproduced baseline results. 3-hop Gain are the gains over the reproduced baseline. best results in each column are in bold. More detailed results can be found in Appendix A.3.
for IWSLT14 and EC30, respectively. For EC30, we employ temperature sampling to select data to train BPE. The temperature aligned with that of the MNMT training phase.
4.2 Results on IWSLT14
Table summarizes the results for IWSLT14. The Weighted Sum represents our most naïve setting, i.e., the left multiplication of a graph matrix over the embedding table as described in Section 3.2. It is worth noting that in this setting, +0.7 aver- age BLEU gains obtained without introducing any extra trainable parameters. For simplicity, we name our graph-basen approach GraphMerge and conduct experiments for 1, 2, and hops.
4.1.2 Training Settings
For IWSLT14, we follow the setting of Lin et al. (2021), using a standard 6-layer encoder 6-layer decoder transformer model with 4 attention heads, 512 embedding dimensions, and 1,024 feedforward dimensions. For EC30, we use Transformer-Bi with 16 attention heads, 1,024 embedding dimen- sions, and 4,096 feedforward dimensions. All mod- els are trained in a many-to-many setting.
It can seen that GraphMerge consistently out- performs the baseline by a substantial margin: 1) the GraphMerge models yield better performance for all language directions, 2) GraphMerge with 3-hop setting achieves the best results with av- erage gain of +2.3 BLEU. The largest gain of +3.5 BLEU can be found for into-English translations. Also, for out-of-Engllnn translation which is of- ten considered the more difficult task, our method obtains an improvement of +1.0 BLEU.
The learning rate 5e-4 with 4,000 warmup steps and a inverse sqrt decay schedule. All dropout rates and label smoothing are set to 0.1. Data from different language pairs are sampled with temperature of 2.0 and 5.0 for IWSLT14 and EC30, respectively. We train all models with early-stopping strategy² and evaluate by using the best checkpoint selected based the loss on the development set. More detailed training settings can be found in Appendix A.2
A noteworthy finding is that as we increase the depth of the GNN from 1-hop to 3-hop, the per- formance consistently improves for almost all lan- guage pairs, resulting in notable gains of 1.6, 1.9, and 2.3 BLEU, respectively. We attribute the pro- gressive improvement to the gradual enhancement of the quality of cross-lingual word embeddings. A more detailed analysis can be found in Section 5.1.
Tokenized BLEU is used as the metric the con- tent. To show a consistent improvement across met- rics, BacreBLEU (Post, 2018) ChrF++ (Popović, 2017), and Comet (Rei et al., 2020) are also re- ported in Appendix A.3 and Appendix A.4.
The results in other metrics, like sacreBI.Es ChrF++, and Comet can also be found in Table 11, where we show that our improvements remain con- sistent across large spectrum of evaluation met- rics.
If not mentioned otherwise, use eftomal? with intersect alignment symmetrization to extract align- ments and build graphs as described Section 3.1.
'Patience is (20, 10). i.... training stops if perfor- the validation does not improve for the last (20, 10) checkpoints, with 1,000 steps between checkpoints. nrefs: Ilcase:mixedleffnoolt htw.:ookk..Blmooaaeeeoppeeeeee (3ukmoothhexpkwersiinn2... Thttps: / thub. com/ comm//obertottlnggefeffeee
"In practice, for the graph G in the naive setting, we add an identity matrix to ensure no loss of information for current words.

---

High
Medium
Low
ALL
Model
EN-X
X-EN
EN-X
X-IN
EN-X
X-EN
EN-X
X-EN
AVG
Baseline (Trans.-Big)
28.7
31.3
31.0
31,4
20.0
25.6
26.5
29.4
28.0
TraphMerge-Ihog
29.5
32.0
31.7
31.8
20.6
27.0
27.3
30.3
28.8
GraphMerge-2h5
29.7
32.2
32.0
32.0
20.9
27.4
27.6
30.5
29.1
GraphMerge-Shopg
29.4
31.8
32.0
31.9
21.0
27.4
27.5
30.4
29.0
2-hopp Gain
+1.0
+0.9
+1.0
+0.6
+0.9
+1.8
+1.1
+1.1
+1.1
Table Large-scale experiments on the EC30 dataset (6IM sentence pairs, 128K shared vocabulary). EN-XX and X-EN denote the average performance of out-of- and into- into-English translation each resource group, respectively. The best results in each column are in bold.
Baseline
1-Hop
2-Hop
Settings
EN-X
X-KN
AVG
12.0
Baseline
24.4
27.8
26.1
1-hop
25.4
30.0
27.7
10.0
1-hop w/o Tie
25.4
28.7
27.0
8.0
2-hop
25.5
30.5
28.0
BLEU
2-hop w/o Tie
25.1
29.6
27.4
3-hop
25.4
31.3
28.4
4.0
3-hop w/o Tie
25.3
29.4
27.4
2-hop
25.5
30.5
28.0
2.0
eftomal + FastAlign
25.4
30.1
27.8
intersect gdfa
25.2
29.9
27.6
H-H
H-M
H4L
M-H
M-M
M-L
L-M L-L L-H
Figure 3: Zero-shot performance EC30 (870 lan- guage directions), grouped by High-, Medium-, and Low-resource.
Table 3: Ablation experiment results on IWSLT14, 1-,
2-, and 3-hop refer to the number of hops in GraphMerge and 'wo tie' indicates tied input and output with original embeddings instead of re- parameterized ones. gdfa is abbreviation of
finul-and.
4.3 Large-scale Multilingual Results
To evaluate the scalability of our approach, we also run experiments on the EC30 dataset which includes 30 language pairs with 61M sentence pairs. Furthermore, also use bigger model (Transformer-Big) and larger vocabulary size (128K), resulting in stronger baseline.
source target language. On average, ap- proach obtains improvements of 1.9 BLEU for all groups compared the baseline.
4.4 Ablation
To investigate the impact of some specific settings in our framework, we conduct ablation experiments on the IWSLT14 dataset in this section.
As shown in Table 2, our method achieves con- sistent improvements across all resource groups over the baseline for English-centric pairs. When extending GraphMerge from 1-hop to 2-hop, fur- ther improvements of up to +1.1 average BL.EU points were achieved, but the improvements started slightly weaken in the 3-hop setting. These improvements are in line with our results for IWSLT14, see Section 4.2. The results for indi- vidual languages can be found in Appendix A.4, where the results in other metrics, like SacreBLEUU, ChrF++, and Comet are also reported and consis- tent improvements are shown as well.
Tied Embeddings. A basic setting for the transformer-based NMT model is to tie the de- coder's input and output embedding (Press and Wolf. 2017: Pappas et al., 2018). Here, two embed- ding tables exist in our GraphMerge model, i.e., the original embedding table and the re-parameterized is better to tie with the
We test which decoder's output embedding. Note that all embed- dings are trained from scratch.
Graph Quality. We use the efiomal alignment tool with the intersect strategy to extract align- ments and construct the corresponding graph. The graphs induced from different alignment strategies may influence downstream results. We evaluate this point and set up experimental groups as fol- lows: 1) Using FastAlign (Dyer et al., 2013) to extract alignments instead of using eftomal, where
We also evaluate the zero-shot translation per- formance EC30 for 870 language directions not involving English. Figure shows our results for 9 resource groups, considering all resource combi- nations (high, medium, low) for source and target. Note that here resource size refers to the amount parallel English data that is available for a given

---

EN++ DE
EN+NL
EN+AR
EN+
Model
Similarity
BLEU
Similarity
BLEU
Similarity
BLEU
Similarity
BLEU
Bascline
0.24
28.5
0.25
31.5
0.23
20.2
0.23
27.9
GraphMerge- phop
0.35
30.2
0.37
33.4
0.32
21.7
0.32
30.0
GraphMerge 2hop
0.42
30.4
0.44
33.7
0.38
21.9
0.38
30.0
GraphMerge-3hop
0.46
30.7
0.48
34.0
0.41
22.4
0.41
30.1
EN+ES
EN++F
EN++P
EN+IT
Model
Similarity
BLEU
Similarity
BLEU
Similarity
BLEU
Similarity
BLEU
Baseline
0.25
36.0
0.22
17.4
0.24
17.6
0.27
29.7
GraphMerge- Thop 0.38
37.5
0.31
19.0
0.35
18.8
0.40
31.3
GraphMerge-2hun 0.45
37.9
0.37
19.0
0.43
19.2
0.48
31.6
GraphMerge-3hop 0.49
38.2
0.40
19.9
0.47
19.4
0.52
32.2
Table 4: English-centric word similarity analysis for each language pair in the IWSLT14 dataset. A high degree of consistency between the similarity of representations and the corresponding BLEU scores can be found.
the latter is often considered to have better perfor- mance (Östling and Tiedemann, 2016). 2) Using grow-diag ding--fnal-andd final-and as the alignment strategy in- stead of intersect, which improves recall but re- duces precision (Koehn, 2009).
We use cosine similarity to evaluate the aver- age distance between the word pairs across all lan- guage pairs. We also consider the isotropy of the space, which can be seen the distribution bias of space (Ethayarajh, 2019). Preserving isotropy is avoid situation where the similarities of words a certain space are all significantly higher than those in other spaces, making the comparison across spaces not fair. The detailed results of the degree of isotropy can be found in Appendix A.6. In short, isotropies are all at high level, i.e., low degree of anisotropy, and therefore not cause for inflated similarities.
Table shows the ablation results on IWSLT14, We report results with different settings: 1-, 2-, 3-hop. By replacing the re-parameterized embed- dings with the original one, BLEU performance drops by [0.7, 0.6, 1.0) on average, respectively, It shows that tying re- re-parameterized embeddings with the decoder's projection function significantly influences performance. The quality of the graph also mildly impacts downstream results: When alignments are noisier, as the case for FastAl- ign and gdfa, performance drops by 0.2 and 0.4, respectively, compared to the 2-hop GraphMerge.
Table 4 shows the analysis results on IWSLT14. is evident that a strong level of consistency ex- ists in our findings: 1) Firstly, as we increase the depth of our GNN model from 1-hop to 3-hop, we observe a progressive increase in the similarities be- tween word representations with similar meanings. This indicates that re-pa parameterized method effectively enhances the cross-linguality of the em- bedding table. 2) Secondly, we observe consistent translation improvements in each direction as the cross lingualin becomes stronger. This supports our hypothesis that improving the multilinguality of the shared embedding table results in greater translation quality.
5 Analysis
5.1 English-Centric Cross-lingual Word Similarity
To verify whether embeddings of words with simi- lar meanings are indeed closer to each other in the representation space, compared to the baseline, we conduct additional experiments as described below.
We utilize bilingual dictionaries from MUSE (Lample et al., 201 2a) as the ground truth and analyze the embedding similarity between word pairs. MUSE contains 110 English-centric bilingual dictionaries and all languages in our ex- periments are included. We limit our comparisons to the word pairs that exist in both our vocabulary of IWSLT14 and MUSE dictionaries. For each language pair, more than 1,000 word pairs exist in the intersection.
5.2 Beyond English-Centric Cross-lingual Word Similarity
As mentioned above, for practicality, we choose En- glish the pivot and only mine equivalent words between English and other languages. Hence, in non-English directions, such as DE-NL, meaning- equivalent relationships are not explicitly incor- porated into the graph (c.g., the empty parts Figure 2). We argue that even this setting, cross-
"tttps: http://hww..mmmeceeoorrrrrrrr/// ground- ground-truth-- wwwpuhhbbiiigguu--diiitiiooo

---

Model
DE+NN
DE++R
DE<HH
NL+AR
NL++H
AR+HE
Baseline
0.29
0.23
0.25
0.24
0.26
0.29
GraphMerge-Ihog 0.36
0.28
0.30
0.30
0.31
0.33
GraphMerge-2hu 0.42
0.32
0.34
0.35
0.35
0.37
GraphMerge-3hop 0.47
0.36
0.38
0.39
0.39
0.41
Table 5: Beyond English-centric word similarity analysis for each language pair in the IWSLTI4 dataset. Consis- tently enhanced multilinguality shows as graph networks go deeper.
Model
EN-DA
DA-EN
EN-AF
AF-EN
Bilingual Baseline
35.3
35.4
31.7
35.9
Bilingual GraphMerge-Shopp
35.6
35.8
33,9
39.6
EC30 Baseline
36.9
39.7
38.4
48.2
EC30 GraphMerge-
38.1
40.0
38.5
50.1
Table 6: Results of bilingual experiments EN-DE and EN-AF language pairs. The evaluation set is the same that of EC30, i... Flores-101. The best results in each column are in bold.
linguality could still be enhanced due to the pivot of English bridge the way of knowledge passing from German Dutch through our multi-hop mecha- nism. We evaluate this point as follows.
transformer-base (6 layers, 512 embeddings dim. 1024 feedforward, 4 attention heads), the back- bone. Also, we shrunk the vocabulary size to 16K considering less data. In Table 6, we compare the bilingual baseline with the GraphMerge-3hor set- ting in language directions, where we also list the performances in these directions from our MMT model trained on EC30 (see Section 4.3).
Non-English-centri dictionaries included in MUSE are limited, therefore we extend MUSE by mapping words paired with the same English words together, e.g., given an EN-DE pair (bike, Fahrrad) and an EN-NL pair {bike. ) we can build new word pair (Fahrrad, fiets) } for DE-NL. For the 8 languages among the IWSLT14 dataset, we build 28 dictionaries for the corresponding non-English- centric language pairs. Each of these dictionaries contains more than 1,000 Soan--Eglish-cettre word pairs except for AR-HE, where 697 pairs are found.
It shows that even in an "extreme" setting (bilin- gual translation), our method still brings clear ben- efits, especially for the low-resource pair (EN-AIF). +2.2 and +3.7 BLEU gains are achieved, respec- tively. When we extend the "extreme" setting 30-language MMT, resulting in a stronger baseline, the improvements remain in these four directions. It is worth noting that in the bilingual setting, only one language is fed to the encoder or decoder. We attribute the improvements here to two potential factors: 1) Monolingual synonyms are modeled better. E.g., bike and bicycle are both linked the pivot words (e.g., cykel in Danish), resulting in enhanced representational similarity via the multi- hop mechanism, and 2) the enhanced cross- -ooss-lingual word similarity, e.g., that of bike and cykel, may also lead to higher accuracy during decoding, We leave the further analysis for future work.
We report analysis results for 6 language pairs among 4 languages, involving the same and dif- ferent writing systems, i.e., DE, NL, AR, and HE in Table 5. One easily that re- parameterized embeddings consistently exhibit higher degree of cross-linguality compared to the baseline, which underlines the generality our approach even if only English-centric equivalence relationships are leveraged. Full results for 8 lan- guages and 28 pairs can be found in Appendix A.7.
5.3 Bilingual Experiments
In addition, also demonstrate applicability for bilingual translation the English-Hebrew track in the WMT 2023 competi- tion. short, clear gains are still observed, even when tested on a large-scale dataset exceeding 30 million sentences. A brief summary of par- tial experiments results for our WMT2023 submis- sion (Wu et al., 2023) can be found in Section A.5.
method's
In this section, we explore whether our method is able to bring benefits for the "extreme" setting of multilingual translation systems, i.e., bilingual translation. We argue that the advantages of better semantic alignments should also apply here.
We pick two language pairs from EC30, i.e. EN- DA (1M) and EN-AF (100K), and conduct experi- ments individually (note that the graph is rebuilt for each as well). We use a rule of thumb setting, i.e.,

---

Model
WPS
Times
practicality for large-scale settings.
Transformer (30K)
201,378
1.00
GraphMerge-Ihog Phop
192,367
1.04
6
Conclusions
GraphMerge-2N1
188,851
1.06
Transformer-Biig (128K)
69,702
1.00
In this paper, we target the shortcomings of solely relying on a shared vocabulary for knowledge trans- fer. Broadly speaking, our approach is to mine word equivalence across languages and inject such priors into the embedding table using graph net- work, thereby significantly improving transfer.
Graph 1hop
43,416
1.61
GraphMerge-2hop
33,912
2.05
Model
Params
Times
Transformer (30K)
62.3M
1.00
Praphheereelhop
63.3M
1.01
GraphMerge-2hop
64.4M
1.02
Transformer-Big (128K)
438.6M
1.00
Our experiments show that our approach results in embeddings with higher degree of multilingual- ity, leading to consistent improvements MNMT. E.g., our approach achieves 2.3 average BLEU gain on IWSLT14, and the improvements still hold even on much larger and more diverse dataset, namely EC30 (61M bitext, 30 language pairs). Also, even for the minimal setting of MMT, i.e., bilingual translation, the performance gains still held for 30+M dataset.
Graph Ihop
442.8M
1.01
GraphMerge-2hop
447.0M
1.02
Table "WPS" is the average word number the model processes per second. We fix "Time" of transformer models to 1.0. 30K and 128K indicate the corresponding vocabulary size.
5.4 Speed and Memory
Compared with the standard transformer model, method introduces extra graph operations. Par- ticularly, a big graph (adjacency) matrix of size 30K 30K is multiplied with the original embed- ding table. An obvious question here whether training latency increases to an unacceptable level. We show training latency in Table 7. All experi- ments here were conducted on a single NVIDIA A6000 GPU card with FP16 optimization. We uti- lized the sparse matrix optimization provided by PyTorch (Paszke al., 2019) for implementation. As shown in Table 7, for the model we used for IWSLT14 14 (30K vocabulary), the extra training la- tency is negligible, with only 4% and 6% for the I I-hop and 2-hop models. Even for a much bigger graph matrix, (128K vocabulary, nearly 16 times bigger adjacency matrix), training latency is lim- ited. The only trainable parameter we introduced here is the dense matrix within the graph networks, which constitutes approximately 2% of all model parameters. Furthermore, due to the sparsity of the adjacency matrix, the additional memory usage is negligible, amounting to less than 1%.
At the same time, our framework remains prac- tical: 1) Through the multi-hop mechanism, the pivot language (English) bridges the way of knowl- edge transfer among non-English language pairs. Therefore, even when only English-centric bitext datasets are available, multilingual transfer can be achieved. 2) A negligible number of additional trainable parameters are required with limited increase in computational costs during training. Meanwhile, the inference latency is exactly as same as benchmark systems by storing and deploying the re-parameterized embeddings for online systems.
Limitations
As the MT system scales up, big vocabulary may be introduced. Extra computing costs may get pro- nounced the vocabulary seales to super big size, e.g., NLLB (Costa-jussà et al., 2022) uses a 256K vocabulary.
Broader Impact
It is worth noting that although the graph matrix can be sizable, the re-parameterized embedding table can be decoupled and stored for online de- ployment, meaning inference latency is exactly the same as for the baseline. Meanwhile, for train- ing, Graph only applies to the embedding table once per training step. other words, the batch size or the model size of the transformer architecture does not matter. Both benefits show
Multilingual translation systems have significant progress recently. However, potential challenges such as mistranslation or off-target issues still exist. Morcover, the fairness problem also raise, i.c., the generation ability is not guaranteed to be fair across languages or demographic features, which may run the risk of exploiting and reinforeing the societal biases (c.g. gender or race bias) that are present in the underlying data.

---

9 Acknowledgement
Hong Kong, China, Association for Computational Linguistics.
We thank the anonymous reviewers for their efforts to make this paper better. We thank our colleagues from the Language Technology Lab at the Univer- sity of Amsterdam, especially Shaomu Tan, Vlad Niculae, and David Stap. for their suggestions on data, code, and writing. We thank Chongyang for its invaluable spiritual support. This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project number VI.C.192.080.
Christian Federmann, Tom Koemi, and Ying Xin. 2022. NTREX- 128 news test references for MT evalua- tion of 128 languages. In Proceedings of the First Workshop Scaling Up Multilingual Evaluation, pages 21-24, Online. Association for Computational Linguisties.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Pro- ceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315- 323. JMLR Workshop and Conference Proceedings.
References
Stephan Gouws and Anders Søgaard. 2015. Simple task- specific bilingual word embeddings. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1396-1390, Denver, Colorado. Association for Computational Linguisties.
Alham Fikri Aji. Nikolay Bogoychev. Kenneth Hcafield, and Rico Sennrich. 2020. In neural machine trans- lation, what does transfer learning transfer? In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7701- 7710, Online. Association for Computational Lin- guistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju. Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-533 538.
Chantal Amrhein and Rico Sennrich. 2020. On Roman- ization for model transfer between scripts in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2461-2469, Online. Association for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's s multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centere machine translation. arXiv preprint vrXiv:2207.04672
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedingy of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris -urlison-Burch. Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer. Ondfej Bojar, Alexandra Constantin, and Evan Herbst. 2007, Moses: Open source toolkit for statistical machine tramslation. In Proceedingy of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions. pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameteriza- tion IBM model 2. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining. In Neural Infor- mation Processing Systems.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIP), pages 55-65,
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018a Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.

---

Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou, 2018b. Word translation without parallel data. In Interna- tional Conference on Learning Representations.
on Empirical Methods Natural Language Process- ing (EMNLP), pages 2685-2702, 2702. Online. Association for Computational Linguistics.
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569- 631.
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 293-305, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Yihong Liu, Haotian Ye. Leonie Weissweiler, and Hin- rich Schütze. 2023, Crosslingual transfer learning for low-resource languages based on multilingual colexi- fication graphs. arXiv preprint arX6x:2305. 12818.
Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictio- nary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 778-788, Melbourne, Australia. Association for Computational Linguistics.
Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with markow chain monte carlo. The Prague Bulletin of Mathematical Linguisties.
Nikolaos Pappas. Lesly Miculicich, and James Hen- derson. 2018. Beyond weight tying: Learning joint input-output embeddings for neural machine trans- lation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 73-83, Brussels, Belgium. Association for Computational Linguistics.
Simeng Sun, Angela Fan, James Cross, Vishrav Chaud- hary, Chau Tran, Philipp Koehn, and Francisco Guzmán. 2022. Alternative input signals ease trans- fer in multilingual machine translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5291-5309 155305, Dublin, Ireland. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Shaomu Tan and Christof Monz. 2023. Towards a bet- ter understanding of variations in zero-shot neural machine translation performance.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1909, Flo- rence, Italy. Association for Computational Linguis- tics.
Xu Tan, Yi Ren. Di He, Tao Qin, Zhou Zhao, and Tic- Yan Liu, 2018 Multilingual neural machine trans- lation with knowledge distillation. International Conference on Learning Representations.
Max Welling and Thomas N Kipf. 2016. Semi- supervised classification with graph convolutional networks. In J. International Conference Learn- ing Representations (ICLR 2017).
Maja Popović. 2017. chrf++:
words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Di Wu, Shaomu Tan, David Stap, Ali Araabi, and Christof Monz. 2023. Uva-mt's participation in the wmt23 general translation shared task.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
A
Appendix
Ofir Press and Lior Wolf. 2017. Using the output bedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 2. Short Papers, pages 157-163, Valnniaa. Spain. Association for Computational Linguistics.
A.1 Dataset Details
The detailed data size of IWSLT14 are as follows: FA (89K), PL (128K), AR (140K), HE (144K), NL (153K), DE (160K). IT (167K), and ES (169K) We list the details of the EC30 dataset in Table 8. Overall, EC30 is an English-centric multilingual machine translation dataset containing 61 million sentences including 31 languages (together with
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation, In Proceedings of the 2020 Conference

---

High
de
nl
es
ru
CS
bn
ar
he
Medium
Sv
da
it
pt
pl
bg
kn
ha mnr
mt
Low lb
af
0C
uk
sd
gu
ti
Table 8: Languages grouped in different resource levels in EC30. We use ISO 639-1 in this table.
Sampled Data (offline)
Full Data (offline)
Full Data (online)
Model
EN-HE
HE-EN
KN-HK
HE-EN
EN-HE
HE-EN
Bilingual Baseline
24.6
31.1
34.1
46.0
MMT Baseline
24.7
31.6
34.1
45.8
33.3
50.3
MMT + GraphMerge 1-hop"
26.2
32.3
34.3
46.2
33.6
50.7
Table 9: Offline and online evaluation results on sampled (2M) and full (34M) training data for the WMT2023 competition. MMT GraphMerge 1-hop means that equip graph-based re- parameterized embedding tables for our MMT baseline. The best BLEU scores in each column are written in bold.
English). EC30 is more profound in the total num- ber of languages and in the balance of language family and writing systems. Specifically, for each language family, we include 6 representative lan- guages across different resources (2 for each re- source level). we categorized the languages into high- medium-, and low-resource groups, each having 5 million, million, and 100,000 bitext examples, respectively.
ments remain consistent across large spectrum of evaluation metries.
A.4 Detailed EC30 Results
We show full English-centric results (60 directions) the EC30 dataset in Table 12. The results are placed from high-, medium-, to low-resource groups.
Also, we show the results in SacreBLEU, CharF++, and Comet on EC30 in Table 13. It shows that our improvements remain consistent across large spectrum of evaluation metrics.
For experiments on EC30, we temperature sampling to select data for training BPE due to it being highly imbalanced (high resource data is 50 times bigger than low ones), the temperature is aligned with that of the MNMT training phase. More specifically, we keep the lowest-resource dataset unchanged and sample data in other bitexts to match the temperature ratio. Then, we merge the data for BPE training and apply BPE to the EC30 corpus.
A.5 Detailed WMT2023 submission
For our WMT 2023 submission (Wu et al., 2023), we make use of all the available data from the constrained track of the shared task for English- Hebrew translation, where 70M parallel data are released. After data preprocessing, we reduced the size of the bitext to 34M. We further normalize and tokenize bitexts using Moses and SentencePiece with 32K vocabulary size.
A.2 Training Settings
For all of our experiments, the training parameters of our methods and baselines range from 62M to 447M. All models are trained on 4 A6000 GPU cards and training times are from 10 hours to days.
For offline evaluation, we conducted experi- ments on full data (34M) and the sampled data (2M). We apply Transformer- Large as the back- bone for the full dataset considering the large scale of bilingual sentences while using Transformer Base for the sampled dataset here. Both bilingual and MMT (two directions within one model) set- tings are tested. We further equip GraphMerge- lhop upon MMT baselines to show the benefits. and the results can be found in Table 9.
A.3 Detailed IWSLT14 Results
We show full results on IWSLT14 in tokenized BLEU in Table 10. An interesting finding is that as graph networks go deeper (from 1-hop to 5-hop), into-English translation consistently gets improved, while the average best one for out-of-English trans- lation GraphMeger with 3-hop setting.
It is easy to see that 1) The performances of the bilingual and MMT models are comparable on both full and sampled datasets. 2) Moreover, the
We also show full results on IWSTT14 in other wide-used metrics, like SacreBLEU, ChrF++, and Comet in Table I1. This shows that our improve-
10 2-layer encoder and decoder, 16 attention heads, 1024 embedding dimensions, and 4096 feed-forward dimensions.

---

Model
EN
DE
EN
ES
EN
FA
EN
AR
EN
HE
EN
NL
EN
PL
EN
IT
baseline
27.9
37.2
14.7
14.1
24.6
31.8
15.3
29.7
GraphMerge-Ihoge
29.2
38.2
15.9
14.6
25.6
33.0
16.1
31.0
GruphMerge-2hop
29.1
38.2
16.1
14.7
25.4
32.8
16.2
31.2
GruphMerge-3hop
28.9
37.9
16.2
14.7
25.2
32.6
16.2
31.3
GraphMerge 4hop
37.9
14.5
25.0
32.8
31.2
GruphMerge-Stog
28.3
37.4
15.6
14.0
24.6
32.3
15.6
30.5
Model
DE EN
- EN
FA
EN
AR -+ EN
HE
EN
NL - EN
PL
EN
EN
Our baseline
29.2
34.9
20.1
26.4
31.2
31,2
19.8
29.6
GraphMerge-Ihop
31.2
36.9
22.1
28.8
34.4
33.8
21.6
31.6
GruphMerge-2hop
37.6
21.9
29.2
34.7
22.1
32.1
GruphMerge-3hop
32.4
38.5
23.6
30.0
35.1
35.4
22.6
33.0
GhophMerge-thop
32.8
38.9
24.1
30.3
35.9
35.8
22.7
33.1
Graphherge-SShp 33.5
39.4
24.3
30.3
36.0
36.2
22.7
33.5
Table 10: Detailed results on the IWSLT14 dataset. We report the BLEU score for each English-centric translation direction.
SacreBLEU
DE
ES
FA
AR
HE
NL
PL
IT
EN-X
X-EN
AVG
baseline
27.6
35.1
16.8
19.5
27.2
30.8
17.1
29.0
24.7
26.0
25.4
GraphMerge- Ihop
29.7
36.9
18.7
21.3
29.4
32.9
18.5
30.6
25.5
29.0
27.2
2rpp
29.7
37.1
18.7
21.1
29.1
33.0
18.5
30.8
25.2
29.3
27.3
GraphMerge-Shop
30.1
37.6
19.5
21.9
29.5
33,5
19.0
31.5
25.3
30.3
27.8
3-hop Gain
+2.6
+2.6
+2.7
+2.4
+2.3
+2.8
+1.9
+2.5
+0.6
+4.3
+2.4
ChrF++
DE
ES
FA
AR
HE
NL
PL
IT
EN-X
X-EN
AVG
baseline
52.1
58.0
40.1
43.7
50.5
54.5
42.1
52.8
50.0
48.4
49.2
GraphMerge- lhop
54.0
60.0
41.8
45.4
52.5
56.5
43.6
54.6
50.6
51.5
51.0
Graphherge-2hop
54.0
60.0
41.8
45.4
52.5
56.5
43.6
54.6
50.6
51.5
51.0
GraphMerge-Shop
54.3
60.3
42.2
45.9
52.9
56.8
43.9
55.0
50.4
52.4
51.4
3-hop Gain
+2.3
+2.3
+2.1
+2.2
+2,44
+2.3
+1.8
+2.2
+0.4
+4.0
+2.2
Comet
DE
ES
FA
ARR
HE
NL
PL
IT
EN-X
X-EN
AVG
baseline
75.5
79.8
74.2
76.2
78.8
78.7
72.9
78.3
78.9
74.7
76.8
GraphMerge- hhop
77.5
81.4
76.2
78.2
80.7
80.8
74.8
80.0
79.9
77.4
78.7
Graphherge-2hop
77.5
81.7
76.1
78.3
80.9
80.9
74.8
80.2
79.8
77.8
78.8
GraphMerge-Shop
77.8
81.8
76.6
78.5
81.2
81.3
75.2
80.5
79.7
78.5
79.1
3-hop Gain
+2.3
+2.1
+2.4
+2.3
+2.4
+2.6
+2.3
+2.2
+0.8
+3.8
+2.3
Table 11:
Detailed results on the IWSLT14 dataset in SacreBLEU, ChrF++, and Comet. Consistent Gains can found for the GraphMerge-3hop setting. Especially, into-English improvements are large.
models equipped with GraphMerge- Ihop achieve consistent improvements in both two directions, especially for sampled data, the gain is evident. Finally, we use the model trained on full data for WMT23's final online evaluation. It shows that even 30+M dataset consisting of two lan- guages only, GraphMerge- Ihop can still achieve +0.3 and +0.4 BLEU improvements for out-of- and into-English translation, which also consistent with offline evaluation. Other strategies we used in this competition can be found in (Wu et al., 2023).
Given the degree of isotropy derived from the stan- dard transformer model as an example:
For a set of word pairs (such as EN-DE), given the degree of isotropy derived from specific model example: 1) for each word z; in the intersection of MUSE EN-DE dictionary and vocabulary, randomly select the target word x 50 times from the whole subword space and aver- age the similarity score between the embedding of x; and xy as I,, 2) then, average J, for all the words æ; in the subsection as the metric LEN DE. For the optimal isotropic space, such metric should be close to zero. Meanwhile, if large difference in isotropy between two spaces exists, the similarity comparison across space is problematic. Table 14 shows the analysis results on IWSLT14. For all of the language pairs, more than 1,000
A.6 Isotropy Definition and Results
Introducing isotropy is to avoid a situation where the similarities of words in a certain space are all significantly higher than those in other spaces mak- ing it difficult to compare results across spaces.

---

Model
DR
ES
RN
haseline
32.1
24.1
41.8
22.8
22.7
25.9
30.9
52.2
19.5
25.8
GruphMerge- thop
GraphMerge- Shop
33.3
21.6
43,3
23,8
24.3
26.6
41.5
32.3
20.3
26.9
GruphMarge-Ba Shop
Model
DE
EN
NL
1--E EN
FR
EN
ES
EN
RU
EN
CS - EN
III EN
BN
EN
AR
EN
IIE
EN
27.9
GmahMarg--hng thop
38.1
28.4
37.6
25.5
28.8
33.6
34.4
28.9
28.4
36.5
GraphMerge Zhop
38.0
28.9
33.3
28.6
366
GrophMerge-Mie Shop
57.8
28.2
37.4
25.5
28.8
33.6
33.5
28.5
28.6
$6.5
Model
EN
SV
EN
DA
EN
PT
EN
BG
EN
KN
EN
MR
EN
MT
EN
36,9
24.6
16.7
36.1
34.9
27.3
48,4
14.4
GraphMarg- Ihop
36.6
37.5
25.4
35.3
17.3
37.8
35,3
27.7
49.7
14.5
GraphMerge- Shop
36.9
36,3
25.9
35.2
17.9
37.9
35.8
27.9
50.1
14.5
GraphMorge-Bi Shop
25.3
17.8
49.9
Model
SV
RN
DA - EN
FF TF-EXN
EN
PT---M RN
Pl.- RN
BG - FN
KN
EN
MR
EN
MT
EN
HA
EN
boseliie
39.5
27.3
23.3
24.6
26.1
48.2
10.2
GruphMerge- thop
40.3
40.2
27.8
40.4
23.7
35.2
24.8
26.5
49,0
10.5
GraphMerge22hog
28.0
49.5
GruphMerge Shop
39.9
40.0
27.9
40.2
23,9
35.6
25.0
26.9
49.5
9.8
Model
EN - AF
EN
LB
EN - RO
EN - OC
EN
UK
EN
SR
EN - SD
EN - GU
EN
TT
EN
AM
25.9
GimphMere--lngg Thop
38.3
16.1
26.2
30.5
22.8
25.1
29.6
4.5
7.8
GaphMaga--llg
26.7
4.7
GruphMerge Shop
38.5
16.5
27.7
30.7
23.4
29.9
4.7
7.8
Model
EN
LB
EN
RO - EN
OC
EN
UK
SR
EN
SD
EN
GU
EN
EN
AM - EN
buseline
48.2
21.8
31.0
35.7
27.3
30.6
1.4
24.4
15.1
20.6
GruphMerge thop
49.7
23.2
32.0
38.1
28.4
32.1
26.1
15.7
22.4
GraphMerge-hogg
50.2
22.4
32.5
39.1
29.2
32.8
1.2
27.0
16.4
22.8
GrahMMrree300
Table 12: Detailed results on the EC30 dataset in 60 directions. The results are placed from high-, medium-, to low-resource groups. We report the BLEU score for each direction.
word pairs exist the intersection of our vocabu- lary and MUSE dictionaries. It is easy to see that our re-parameterized embeddings are consistent with better cross- longulling i.e., the distance be- tween word representations with similar meanings is much smaller. Meanwhile, the isotropies are all high level (low degree), i.e., evaluation risk due space bias excluded as well. other words, similarity comparisons Table are fair.
A.7 Full Results of Beyond English-Centric Word Similarity
In Table 15, we show the full results of non- non-English- centric word similarity analysis for 28 language directions in the IWSLT14 dataset. It is easy to find that, as graph networks go deeper, word similarity consistently gets closer.

---

High
Medium
Low
ALL
SacreBLEU
EN-X
X-EN
EN-X
X-EN
EN-X
X-EN
EN-X
X-EN
AVG
Baseline (Trans.- -Big)
28.4
31.1
29.8
31.2
19.4
25.5
25.9
29.3
27.6
GraphMerge- Ihop
29.1
31.6
30.4
31.5
20.0
26.7
26.5
29.9
28.2
GraphMerge-2hop
29.3
31.9
31.7
20.3
26.8
30.2
28.5
GraphhMerge-Shop
29.2
31.6
30.7
31.7
20.4
27.3
26.8
30.2
28.5
2-hop Gain Gain
+0.9
+0.8
+1.0
+0.5
+0.8
+1.6
+0,9
+1.0
+0.9
High
Medium
Low
ALL
ChrF++
EN-X
X-EN
DN-XX
X-IN
EN-X
X-EN
EN-X
X-IN
AVG
Baseline (Trans.-Big)
52.5
57.1
53,9
56.4
42.8
49.7
49.8
54.4
52.1
GraphMerge-Ine
53.0
57.5
54.4
56.6
43.5
50.6
50.3
54.9
52.6
GraphMerge-2hun
53.3
57.6
54.8
56.6
43.8
50.9
50.6
55.0
52.8
GraphMerge-9hop
53.2
57.5
54.7
56.6
43.8
51.0
50.6
55.0
52.8
2-hop Gain
+0.7
+0.5
+0.9
+0.2
+1.0
+1.2
+0.9
+0.6
+0.8
High
Medium
Low
ALL
Comet
EN -X
X-EN
EN-X
X-EN
EN-X
X-EN
EN-X
X-EN
AVG
Baseline (Trans.-Bigg
82.0
83.4
80.6
79.7
73.3
72.7
78.6
78.6
78.6
GraphMerge- -hop
82.8
83.9
81.2
80.2
74.2
74.0
79.4
79.4
79.4
GraphMerge-2hop
83.1
84.0
81.5
80.2
74.6
74.3
79.8
79.5
79.6
GraphMerge-Shop
83.0
83.9
80.1
74.8
74.5
79.7
79.5
79.6
2-hop Gain
+1.1
+0.7
+0.9
+0.5
+1.4
+1.6
+1.1
+0.9
+1.0
Table 13: Large-scale experiments the EC30 dataset (61M sentence pairs, 128K shared vocabulary) in Sacre- BLEU, ChrF++, and Comet. EN-X and X-EN denote the average performance of out-of- and into-English translation on each resource group, respectively.
Model
EN-DE
EN-ES
EN-FA
EN-AR
EN-HE
EN-NL
EN-PL
EN-IT
Baseline
0.068
0.071
0.074
0.073
0.072
0.069
0.074
0.073
GraphMerge- Ihop
0.001
-0,001
-0.001
-0.001
-0,001
-0.001
-0,001
-0.002
GraphMerge-2hop
0.001
0.001
0.002
0.001
0.001
0.001
0.002
0.001
GraphMerge-Shop
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
Table 14: The degree of isotropy in each space. It is easy find that all of the degrees induces from our spaces
close to 0, i.e., evaluation risk due to space bias is excluded.

---

Model
DE
DE<+R
DE
DE
DE+FA
DE<+L
DE+IT
baseline
0.30
0.23
0.25
0.27
0.25
0.27
0.29
Graph 1hop
0.37
0.28
0.30
0.34
0.30
0.33
0.36
GraphMerge-2hhop
0,43
0.33
0.34
0.40
0.35
0.39
0.42
Graphherge-3hop
0.47
0.36
0.38
0.44
0.38
0.43
0.46
Model
NL+AR
NL+HE
NL+ES
NL+FA
NL++PL
NL+IT
AR+HE
baseline
0,24
0.26
0.28
0.25
0.28
0.30
0.29
GraphMerge- Ihop
0.30
0.31
0.36
0.30
0.35
0.39
0.33
2raphherge-2hop
0.35
0.35
0.35
0.41
0.45
0.38
Mergehhrg
0.39
0.39
0.46
0.39
0.45
0.49
0.41
Model
AR+ES
AR+*F
AR+PL
AR+ITT
HE++ES
HE++FA
HE++PL
baseline
0.26
0.27
0.25
0.27
0.26
0,26
0.27
GraphMerge-
0.32
0.32
0.31
0.33
0.32
0.31
0.31
Graphherge--hhop
0.37
0.36
0.36
0.39
0.36
0.35
0.36
GraphMerge-3hop
0.40
0.40
0.39
0.43
0.40
0.39
0.40
Model
HЕ++I
ES+PP
ES++PL
ES++TT
FA++P
FA++IT
PL++IT
baseline
0.27
0.25
0.28
0.33
0.24
0.25
0.29
Graph Ihop
0.33
0.31
0.36
0.41
0.29
0,31
0.37
Graphherge-2hop
0.38
0.36
0.41
0.48
0.34
0.36
0.42
Graphherge-Shhp
0.42
0.40
0.45
0.52
0.37
0.39
0.47
Table 15: Full results of non- -on-Engliih--entric word similarity analysis on the IWSLT14 dataset for 28 language pairs.


Reviews:
[{"review_id": "5UmdUT3t0F", "reviewer": "Reviewer_8KXe", "summary": "", "questions": "Question A: Your framework only uses old word aligners. There are several established word aligners such as SimAlign and AwesomeAlign that perform much better. Is there a reason that you didn't use them? Because I think it would further improve your performance.\n\nQuestion B: With the current model, the rare words that don't occur in the word alignments won't benefit from this model at all. Do you have any plans or solutions for them?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reading our paper and insightful suggestions. We’re delighted that you pointed out the connections between our paper and [imanigooghari-etal-2021-graph, imani-etal-2022-graph]. We agree on this point and will put them in the related work in our final version. We summarize and answer your questions as follows:\n\n> [Reason to Reject-1]: Why not use Precision@k (commonly 1) to check the quality of Bilingual Lexicon Induction instead of representational similarity?\n\n**[Response-1]:** One reason is that we intend for the analysis to align closely with our motivation, namely, the enhancement of representational word similarity. Additionally, relying solely on high-level Precision@k scores in the context of BLI might not necessarily ensure that the representation distances are close. To illustrate this, consider an extreme scenario where pairs of word embeddings are meticulously arranged along two parallel lines in the space. In such a case, the distance between the parallel lines does not impact the BLI outcome.\n\n> [Reason to Reject-2]: Section 5.2: Adding more pairs that are extracted from word alignments made these results questionable. In a way, the word alignment model might have a bias on the graph model to make specific types of words closer to each other (for example only nouns). Testing the outputs with datasets extracted from the same word aligners is not valid.\n\n**[Response-2]:** We think there might be a misunderstanding here. In our case, the word pairs for testing are still sourced from MUSE, rather than alignments. However, we extended the MUSE to non-English pairs [refer to Lines 488-493]. However, the graph structure remains unchanged, still exclusively based on English-centric alignments. Our analysis shows that our method can also enhance the representational similarities among non-English word pairs. This enhancement is achieved as the pivot (English) facilitates the transfer of knowledge through our multi-hop mechanism.\n\n\n>[Reason to Reject-3]: The experiments don't include baselines that use multilingual language models as encoders. It might be the case that a multilingual LM as the encoder that only uses monolingual data performs better than the proposed method.\n\n**[Response-3]:** Using multilingual LM as an encoder is an interesting setting. However, some issues would arise: 1) As a kind of pre-train + finetune fashion, there is no guarantee that the knowledge learned from the multilingual LM (like multilinguality of embeddings) will not be washed out after finetuning [liu-etal-2020-multilingual-denoising], resulting in situations where pretraining won't work well for high-resource language directions (even hurt the performance sometimes). 2) The model size is bundled with the pretraining model as well. 3) Lastly, we should also mention that the main issue we target, i.e., the lack of explicit knowledge transfer among words in different scripts, still holds for multilingual LMs.\n\n\n> [Question-1]: Your framework only uses old word aligners. There are several established word aligners such as SimAlign and AwesomeAlign that perform much better. Is there a reason that you didn't use them? Because I think it would further improve your performance.\n\n**[Answer-1]:** We totally agree that better alignment quality could further improve the performance. This point has been demonstrated already as our ablation study shows that using Eflomal is slightly better than using FastAlign (see Table 3, page 6). However, we also agree that taking a look at how far our architecture can go is valuable. We will try to include them in our next version.\n\n> [Question-2]: With the current model, the rare words that don't occur in the word alignments won't benefit from this model at all. Do you have any plans or solutions for them?\n\n**[Answer-2]:** This is indeed a good question and we also considered this point throughout the project. 1) Firstly, we should note that \"rare words\" are a general problem in most NLP scenarios, not only a particular flaw in our architecture. 2) Secondly, our method (although not mentioned in the paper) may have the potential to alleviate this issue by manually adding the edges between rare words with those counterparts across languages. 3) Also, relying on technologies in cross-lingual word embeddings and then conducting BLI (or other kinds of retrieval) across languages to build graphs, may have the potential to cover more links among rare words. However, other designs need to be considered since the weights of the graph become hard to define in this case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like this paper and I like to see it accepted at this conference.\n- Although using GNNs over word alignments to improve the quality of representations has been proposed before (imani-etal-2022-graph), using it in an end-to-end fashion in an MNMT model is novel and could be used more in the community.\n- The experiments include several language pairs and show the generalizability of the proposed model. I like the fact that Table 2 has different sections for high, medium, and low resource performance.\n- Including the ablation studies and the additional analysis makes this work more valuable.\n- The authors include as many technical details as possible that facilitate to reproduce the results. They also claim to release the codes upon acceptance of the paper.\n", "reasons_to_reject": "- Table 4, 5: It is established to use Precision@k (commonly 1) to check the quality of Bilingual Lexicon Induction.\nUsing the MUSE dataset with other metrics could be misleading.\nAlthough we know the quality of translations has improved, it could be the case that the nearest neighbors of each word could change during the graph transformation. Looking at average similarity is not a good metric to check this.\n\n- Section 5.2: Adding more pairs that are extracted from word alignments made these results questionable.\nIn a way, the word alignment model might have a bias on the graph model to make specific types of words closer to each other (for example only nouns). Testing the outputs with datasets extracted from the same word aligners is not valid.\n\n- The experiments don't include baselines that use multilingual language models as encoders. It might be the case that a multilingual LM that only uses monolingual data performs better than the proposed method.\n\n", "questions_to_authors": "Question A: Your framework only uses old word aligners. There are several established word aligners such as SimAlign and AwesomeAlign that perform much better. Is there a reason that you didn't use them? Because I think it would further improve your performance.\n\nQuestion B: With the current model, the rare words that don't occur in the word alignments won't benefit from this model at all. Do you have any plans or solutions for them?\n", "paper_topic_and_main_contributions": "This paper presents a method to decrease the embedding distance of similar words and improve the machine translation quality in an end-to-end model.\nThe proposed framework, first extracts lists of word pairs with similar meanings using word alignments.\nA graph network transforms the original embeddings to a new space where similar words are better aligned across languages.\nAn encoder-decoder transformer model then uses the new embeddings for translation, and the whole model is trained end-to-end.\n\nThe experiments include the evaluation of two datasets containing several language pairs ranging from low-resource to high-resource. The results show consistent improvements over baseline when using more graph layer hops.\nThe paper also includes an ablation study over the quality of word alignments and tying encoder and decoder embeddings, as well as other experiments to study the quality of the embeddings and the performance of the proposed model.\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[imanigooghari-etal-2021-graph] and [imani-etal-2022-graph] show methods to improve word alignment quality.\n"}, "extra_text": {}}, {"review_id": "ax7qjTHFl3", "reviewer": "Reviewer_4Nxw", "summary": "", "questions": "I'm moving some suggestions for further experiments in this section, since they do not constitute grounds for rejection and are merely suggestions.\n\nRegarding **B1/**, I would have found it useful to have two other baselines along the Weighted Sum:\n1. One where the re-parametrization only relies on the random walk, with no extra learnable parameters; e.g., $E' = \\left(\\prod_1^k G\\right) E$\n2. One where the reparametrization only relies on extra learnable parameters, e.g $E_k = \\rho(W_k E_{k-1} + b_k)$ and $E_0 = E$\n\nOn a related note, remark that the Weighted Sum baseline is defined as $GE$, instead of $E + GE$ as is done for the $k$-hop variants.This entails that the embedding for e.g. _bike_ is re-parametrized as a weighted sum of the wordpieces it is aligned with; crucially, these might not contain the wordpiece _bike_ itself. \n\n\nRegarding **B2/**: To provide some concrete actionable feedback, this is something that could have been effectively tested by testing two other formulations of $G$, e.g. in the ablation experiments in §4.4:\n1. A variant $G_\\mathrm{equiv.}$ where $G$ is strictly controlled to only contain edges between known equivalent words (e.g., using MUSE dictionaries or wordnet lemmas)\n2. A variant $G_\\mathrm{transl.}$  where $G_{ij}$ is defined simply as the empirical probability of wordpiece $w_j$ occurring in the translation of a sentence containing the wordpiece $w_i$ (regardless of eflomal alignments).\n\nIn practice $G_\\mathrm{equiv.}$ might not outperform $G$ owing to the limited number of edges that such an approach could yield; so such a comparison might call for controlling for the number of edges by pruning the least-weighted edges of $G$; nonetheless, if the argument of the authors stands, then we would expect performances using $G_\\mathrm{equiv.}$ to be better than using $G_\\mathrm{transl.}$.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of our efforts and your in-depth comments, especially in part B. Also, we'd love to hear that it might be useful for your future work. We summarize and answer the questions as follows:\n\n> [Reason to Reject-A1] The authors only report 1 seed (as far as I can tell).\n\n**[Response-A1]:** Due to the limitation of resources, we use a fixed random seed for all of the experiments. However, we agree that multiple runs and significance analysis are great. We will try to include them in our final version.\n\n> [Reason to Reject-A2] The baselines do not include comparable methods, and in particular other embedding alignment approaches; e.g., Aji et al (2020) (cited by the authors)\n\n**[Response-A2]:** We agree that inducing baselines sharing the same motivation is great, however, there is no similar research in multilingual MT scenarios as far as we know. Sun et al. (2022) might be a case, however, as we mentioned in Lines 145-146 and Lines 134-138, the limitations of Romanization and transliteration still hold. We should also mention that Aji et al (2020) is designed for bilingual MT and it relies on the pretraining method, which is somewhat hard to compare with.\n\n> [Reason to Reject-B1] One question that specifically comes to mind is that of the relative importance of the extra embedding weights, vs. that of the implicit random walk in the alignment graph. As it currently stands, the authors do not disentangle these two factors... I would have found it useful to have two other baselines along the Weighted Sum: \n> 1) One where the re-parametrization only relies on the random walk, with no extra learnable parameters; e.g., $E^{k}=(\\prod_{i=1}^{k}G)E$\n> 2) One where the reparametrization only relies on extra learnable parameters, e.g., $E_k=\\rho(W_k E_{k-1} + b_k)$\n\n**[Response-B1]:** This is a really insightful point. We appreciate your interest in addressing these two factors, especially for the random walk setting. It is also efficient since we can calculate $G^*=\\prod_{i=1}^{k} G$ before training. We will try to introduce this part in our final version. For the second point, we are hesitating to claim any potential performance change can be solely attributed to the extra parameter. Because the extra parameters are introduced by the projection function, which works as a part of GNN to adjust aggregation. A separate form, e.g., the one you mentioned in the second part, would lose this capability directly.\n\n\n> [Reason to Reject-B2]:  Another area where I'm uncertain what to make of the authors' claim concerns the question of the semantic similarity of items with high-weighted edges in $G$. As the authors remark (line 195 sq.): [The graph definition] is based on the intuition that when a pair of aligned words frequently co-occur, they 1) have higher confidence as equivalent words, and 2) the knowledge sharing between these two will benefit more context during training.\n\n**[Response-B2]:** We deeply value and appreciate your efforts on the details of our paper. We try to reply to you in the following aspects:\n\n1) Firstly, we wanna explain the second point of intuition a little bit, i.e., _\"the knowledge sharing between these two will benefit more context during training.\"_ Suppose a verb A with the meaning of 'bike/cycle to somewhere' in one language. It could be translated to 'bike', 'cycle', and 'pedal' in English, and the translation of 'bike' and 'cycle' occurs a lot of time while only a few times for 'pedal'. Intuitively, the weight between A and 'pedal' should be low since its influence is low as well. We will modify this point in our final version in case any misleading. **In the following responses, we solely focus on the potential or issue with the design of $G_{equiv}$ and $G_{transl}$ you mentioned.** \n2) Secondly, we appreciate the idea of utilizing MUSE to construct the graph $G_{equiv}$, as also suggested by Reviewer hsEL (albeit from an aesthetic perspective rather than an ablation one). However, it's important to note that this approach may introduce certain potential issues.  E.g., frequency information will be lost in this case, and the thing gets more complex when words in MUSE are split into subwords.\n3) Lastly, your design of $G_{transl}$ is really an interesting point, which makes sense to some extent and gets rid of using an external tool, e.g., alignment or MUSE. One potential issue would be like: it will make the graph dense since the edge now is defined upon the co-occurrence of any word pairs within sentence pairs. Maybe defining the weights using pointwise mutual information and using a threshold to prune part of them is a way. We will try to explore it in the future.\n\n\n> [Reason to Reject-B3]: Lastly, it would have been interesting to verify the usefulness of the proposal on bilingual models. It is unclear to me (at least unproven) that the method actually fares well in smaller settings.\n\n**[Response-B3]:**  Yes, it is an interesting point. We are also curious about this. Intuitively, our method might also work for bilingual translation. We conducted additional experiments to check this point:\n\nDue to the limitation of time, we randomly picked a medium-resource and a low-resource language pair from the WMT30 dataset: EN-DA (1M) and EN-FA (100K), to conduct the experiment individually (Note that the graph is rebuilt for each as well). Unfortunately, we can't search for an optimal backbone in a short time, hence we use a rule of thumb setting, i.e., transformer-base (6 layers, 512 embeddings dim, 1024 feedforward, 4 attention heads). We shrink the vocabulary size to 16K and the batch size to 80K and 40K for medium- and low-resource respectively. We compare the baseline with GraphMerge-3hop setting as follows:\n\n|                | EN-DA | DA-EN | EN-AF | AF-EN |\n|----------------|-------|-------|-------|-------|\n| Bilingual Baseline       | 35.3  | 35.4  | 31.7  | 35.9  |\n| Bilingual GraphMerge-3hop| 35.6  | 35.8  | 33.9  | 39.6  |\n| WMT30 Baseline            | 36.9  | 39.7  | 38.4  | 48.2  |\n| WMT30 GraphMerge-3hop     | 38.1  | 40.0  | 38.5  | 50.1  |\n\nIt shows that in an \"extreme\" setting (bilingual setting), our method still brings some benefits, especially on the low-resource part. We also list the results of our MMT trained on WMT30 in these directions for your reference. It is worth noting that, although MMT seems to be better here, we are hesitating to claim it, since the backbone of bilingual translation may not be optimal (e.g., model size).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach is well thought out; on a personal note, I can also say it is likely to be useful in my future work\n- The experiments demonstrate the effectiveness and efficiency of the approach, with a clear improvement in BLEU scores and a low memory footprint.", "reasons_to_reject": "**A/** there a few methodological points that could be better handled. In particular:\n\n**A1/** the authors only report 1 seed (as far as I can tell)\n\n**A2/** the baselines do not include comparable methods, and in particular other embedding alignment approaches; e.g., Aji et al (2020) (cited by the authors)\n\n**B/** While the method is proven to be effective, I am not completely sold on the narrative around it; i.e., that it is effective for the reasons suggested by the authors. This boils down to a few points of comparison that would be valuable, but are regrettably missing.\n\n**B1/** One question that specifically comes to mind is that of the relative importance of the extra embedding weights, vs. that of the implicit random walk in the alignment graph. As it currently stands, the authors do not disentangle these two factors. Throughout the paper, the authors seem to attribute the success of their approach to the former, but do not control for the latter (though in fairness they do remark that the added parameters correspond to a fraction (2%) of the total embeddings).\n\n**B2/** Another area where I'm uncertain what to make of the authors' claim concerns the question of the semantic similarity of items with high weighted edges in $G$. As the authors remark (line 195 sq.):\n> [The graph definition] _is based on the intuition that when a pair of aligned words frequently co-occur, they 1) have higher confidence as equivalent words, and 2) the knowledge sharing between these two will benefit more context during training._\n\n As such the graph $G$ need not only describe equivalent word classes. Or put another way: it is plausible that the improvements that this method obtains are mostly due to point 2) in the citation above, rather than point 1).\n\n**B3/** Lastly, it would have been interesting to verify the usefulness of the proposal on bilingual models., as such models constitute a useful point of reference. Reading from line 110&mdash;111:\n> _our method adapts to massive language pairs and a large vocabulary._\n\nIt is unclear to me (at least unproven) that the method actually fares well on smaller settings.\n\n**Edit after rebuttal:** The authors' rebuttal provided a supplementary set of results that alleviates my concerns as per this last point.", "questions_to_authors": "I'm moving some suggestions for further experiments in this section, since they do not constitute grounds for rejection and are merely suggestions.\n\nRegarding **B1/**, I would have found it useful to have two other baselines along the Weighted Sum:\n1. One where the re-parametrization only relies on the random walk, with no extra learnable parameters; e.g., $E' = \\left(\\prod_1^k G\\right) E$\n2. One where the reparametrization only relies on extra learnable parameters, e.g $E_k = \\rho(W_k E_{k-1} + b_k)$ and $E_0 = E$\n\nOn a related note, remark that the Weighted Sum baseline is defined as $GE$, instead of $E + GE$ as is done for the $k$-hop variants.This entails that the embedding for e.g. _bike_ is re-parametrized as a weighted sum of the wordpieces it is aligned with; crucially, these might not contain the wordpiece _bike_ itself. \n\n\nRegarding **B2/**: To provide some concrete actionable feedback, this is something that could have been effectively tested by testing two other formulations of $G$, e.g. in the ablation experiments in §4.4:\n1. A variant $G_\\mathrm{equiv.}$ where $G$ is strictly controlled to only contain edges between known equivalent words (e.g., using MUSE dictionaries or wordnet lemmas)\n2. A variant $G_\\mathrm{transl.}$  where $G_{ij}$ is defined simply as the empirical probability of wordpiece $w_j$ occurring in the translation of a sentence containing the wordpiece $w_i$ (regardless of eflomal alignments).\n\nIn practice $G_\\mathrm{equiv.}$ might not outperform $G$ owing to the limited number of edges that such an approach could yield; so such a comparison might call for controlling for the number of edges by pruning the least-weighted edges of $G$; nonetheless, if the argument of the authors stands, then we would expect performances using $G_\\mathrm{equiv.}$ to be better than using $G_\\mathrm{transl.}$.", "paper_topic_and_main_contributions": "This paper proposes to re-parametrize the embedding layer of an MT system such that the embedding of a word in a given language is pulled towards that of its translations in the other language. Their graph-based approach relies on the probability of two words being aligned in the dataset. Extensive experiments and qualitative analyses demonstrate the effectiveness of the approach across two datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uu6ZC7TNFj", "reviewer": "Reviewer_CxMx", "summary": "", "questions": "A. I'm not entirely clear how is the graph transfer ratio $g_{i,j}^{s,t}$ (defined in L191) is used subsequently in Section 3.2? Is it used as initialization of the edge weights?\nB. L421-422: What is the \"original\" embedding referring to here?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts on our paper. We summarize and answer your questions as follows:\n\n> [Reason to Reject-1]: My biggest concern about the paper is its evaluation: the paper only reported tokenized BLEU.\n\n**[Response-1]:** We agree with this point and release the evaluation results in other metrics (SacreBleu, Chrf++, and COMET) for the IWSLT14 dataset here. \n\n| SacreBLEU  | en-de | en-es | en-fa | en-ar | en-he | en-nl | en-pl | en-it |\n|------------|------:|------:|------:|------:|------:|------:|------:|------:|\n| baseline   |  27.6 |  35.1 |  16.8 |  19.5 |  27.2 |  30.8 |  17.1 |  29.0 |\n| GM-1hop    |  29.7 |  36.9 |  18.7 |  21.3 |  29.4 |  32.9 |  18.5 |  30.6 |\n| GM-2hop    |  29.7 |  37.1 |  18.7 |  21.1 |  29.1 |  33.0 |  18.5 |  30.8 |\n| GM-3hop    |  30.2 |  37.7 |  19.5 |  21.9 |  29.5 |  33.6 |  19.0 |  31.5 |\n| 3-hop Gain |   +2.6 |  +2.6 |  +2.7 |   +2.4 |   +2.3 |   +2.8 |   +1.9 |   +2.5 |\n\n| ChrF++     | en-de | en-es | en-fa | en-ar | en-he | en-nl | en-pl | en-it |\n|------------|------:|------:|------:|------:|------:|------:|------:|------:|\n| baseline      |  52.1 |  58.0 |  40.1 |  43.7 |  50.5 |  54.5 |  42.1 |  52.8 |\n| GM-1hop    |  54.0 |  60.0 |  41.8 |  45.4 |  52.5 |  56.5 |  43.6 |  54.6 |\n| GM-2hop    |  54.0 |  60.0 |  41.8 |  45.4 |  52.5 |  56.5 |  43.6 |  54.6 |\n| GM-3hop    |  54.4 |  60.3 |  42.2 |  45.9 |  52.9 |  56.8 |  43.9 |  55.0 |\n| 3-hop Gain |   +2.3  |   +2.3   |   +2.1 |   +2.2   |   +2.4 |   +2.3   |   +1.8  |   +2.2 |\n\n| Comet      | en-de | en-es | en-fa | en-ar | en-he | en-nl | en-pl | en-it |\n|------------|------:|------:|------:|------:|------:|------:|------:|------:|\n| baseline   |  75.5 |  79.8 |  74.2 |  76.2 |  78.8 |  78.7 |  72.9 |  78.3 |\n| GM-1hop    |  77.5 |  81.4 |  76.2 |  78.2 |  80.7 |  80.8 |  74.8 |  80.0 |\n| GM-2hop    |  77.5 |  81.7 |  76.1 |  78.3 |  80.9 |  80.9 |  74.8 |  80.2 |\n| GM-3hop    |  77.8 |  81.9 |  76.6 |  78.5 |  81.2 |  81.3 |  75.2 |  80.5 |\n| 3-hop Gain |   +2.3 |   +2.1 |   +2.4 |   +2.3 |   +2.4 |   +2.6 |   +2.3 |   +2.2 |\n\n**It shows that average of 2+ points improvement over there for all of the metrics**. This shows that our improvements remain consistent across a large spectrum of evaluation metrics. \n\nWe also release the result on WMT30 here:\n\n| SacreBLEU  | EN-X | X-EN | EN-X | X-EN | EN-X | X-EN |\n|------------|:----:|:----:|:----:|:----:|:----:|:----:|\n| baseline   | 28.4 | 31.1 | 29.8 | 31.2 | 19.4 | 25.5 |\n| GM-1hop    | 29.1 | 31.6 | 30.4 | 31.5 | 20.0 | 26.7 |\n| GM-2hop    | 29.3 | 31.9 | 30.8 | 31.7 | 20.3 | 27.1 |\n| GM-3hop    | 29.2 | 31.6 | 30.7 | 31.7 | 20.4 | 27.3 |\n| 2-hop Gain |  +0.9 |  +0.8 |  +1.0 |  +0.5 |  +0.9 |  +1.6 |\n\n| ChrF++     | EN-X | X-EN | EN-X | X-EN | EN-X | X-EN |\n|------------|:----:|:----:|:----:|:----:|:----:|:----:|\n| baseline   | 52.5 | 57.1 | 53.9 | 56.4 | 42.8 | 49.7 |\n| GM-1hop    | 53.0 | 57.5 | 54.4 | 56.6 | 43.5 | 50.6 |\n| GM-2hop    | 53.3 | 57.6 | 54.8 | 56.6 | 43.8 | 50.9 |\n| GM-3hop    | 53.2 | 57.5 | 54.7 | 56.6 | 43.8 | 51.0 |\n| 2-hop Gain |  +0.8 |  +0.5 |  +0.9 |  +0.2 |  +1.0 |  +1.2 |\n\n| COMET      | EN-X | X-EN | EN-X | X-EN | EN-X | X-EN |\n|------------|:----:|:----:|:----:|:----:|:----:|:----:|\n| baseline   | 82.0 | 83.4 | 80.6 | 79.7 | 73.3 | 72.7 |\n| GM-1hop    | 82.8 | 83.9 | 81.2 | 80.2 | 74.2 | 74.0 |\n| GM-2hop    | 83.1 | 84.1 | 81.5 | 80.2 | 74.7 | 74.3 |\n| GM-3hop    | 83.0 | 83.9 | 81.5 | 80.1 | 74.8 | 74.5 |\n| 2-hop Gain |  +1.1 |  +0.7 |  +0.9 |  +0.5 |  +1.4 |  +1.6 |\n\nThese results will also be included in the final version.\n\n> [Reason to Reject-2]: Tokenize BLEU is hard to reproduce. The comparison with Lin et al. (2021) is not suitable.\n\n**[Response-2]:** Firstly, we acknowledge that SacreBleu is great since it unifies measurements. However, here we use the same tokenizer Lin et al. (2021) used in their open-source scripts, therefore the results are already comparable. Secondly, we agree that there is no need to compare with particular scores of previous work as we are not doing competitions, especially when the motivations are different. We list their results here just to show that our reproduced baseline is fine.\n\n>[Reason to Reject-3]: While I understand the importance of the research question, I don't fully agree with the necessity or the benefit of using GNN for that. To me, there are simpler ways to inject such information, such as random substitutions (https://aclanthology.org/2020.emnlp-main.210.pdf) or contrastive learning (https://aclanthology.org/2021.acl-long.21.pdf).\n\n**[Response-3]:** These two papers are somewhat relevant, but we should mention that 1) they both introduce external data in the data augmentation or pre-training phase and 2) they both introduce an external tool like MUSE or a method like contrastive learning. It's hard to say whether they are \"simpler\" or not. Meanwhile, we don't think a mature tool, like GNN, will make the system that complex. However, we agree to include them in our related work in our final version. Thanks for your suggestion.\n\n\n> [Question For the Authors]: How is $g_{i,j}^{s,t}$ used in section 3.2? Is it used as initialization of the edge weights? What is the \"original\" embedding referring to here?\n\n**[Answers]:** We answer your questions one by one as follows:\n1) As we mentioned in Lines 200-204: \"The corresponding bilingual equivalence graph $G_{s,t}$ can be induced by filling an adjacency matrix using $g_{i,j}^{s,t}$. By element-wise summation of multiple $G_{s,t}$, we can merge multiple bilingual graphs into a single multilingual graph $G$\". **Here, $G$ is the graph we used in section 3.2**.\n2) It is used to build the weights of edges in the graph. Given you mentioned \"initialization\" here, in case of potential misunderstanding of the training fashion, we want to highlight that: **the graph is fixed during training**. The transfer pathways and ratio defined in the graph are the priors we want to inject into the embedding table. E.g., “bike” and “fiets” can transfer information with each other hence there is an edge with a weight between them in the graph, resulting in that even though they write in different ways, knowledge can explicitly transfer.\n3) As a re-parameterized method, **note that there are two embedding tables here.** One is original and the other is re-parameterized. The former does not transfer information with GNN, and the latter is the one fed into the transformer. The two embedding tables are trained in an end-to-end way.\n\nPlease refer to our responses to other Reviewers (E.g., Review 4Nxw) for a better understanding of the formalization.\n\n> [Missing Reference]: GNN has been extensively used in neural machine translation. It would be good to at least mention a few.\n\nDue to page limitations, we could only focus on the most relevant related approaches, but we are happy to include additional references in the camera-ready version. However, we should also mention that while the references provided by the reviewer are related, none of them actually focus on applying GNNs in the context of multilingual machine translation, and the motivation differs a lot as well. E.g., document translation [xu-etal-2021-document-graph] and introducing syntax in bilingual translation [bastings-etal-2017-graph,marcheggiani-etal-2018-exploiting] or embeddings [vashishth-etal-2019-incorporating].\n\n\n---\n\nLastly, given the fact that your biggest concern is “BLEU is the only metric”, we tried our best to make it comprehensive, where the main widely-used metrics are all included. Also, as reviewer-hsEL mentioned, **\"using only one metric is weak but that is not a reason to reject it.\"**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research question addressed by this paper is a valid and important one in the context of multilingual translation models.\n2. All experiments showed improvements over the baseline, both in terms of translation quality and word similarity benchmark.\n3. The presentation of the paper is mostly clear and easy to follow.", "reasons_to_reject": "1. My biggest concern of the paper is its evaluation. (1) The paper only reported tokenized BLEU, which has been shown in Post (2018) (https://aclanthology.org/W18-6319.pdf) to be very hard to reproduce. Plus, the findings of WMT 2022 metrics shared task (https://www.statmt.org/wmt22/pdf/2022.wmt-1.2.pdf) also suggested against using BLEU as the only metric for evaluation.\n\nI know the authors probably chose to use the tokenized BLEU in order to keep consistent with Lin et al. (2021), which also reported tokenized BLEU. Yet, I would still strongly recommend using sacreBLEU and also report at least one of BLEURT or COMET. If it is hard to get those numbers for the models in Lin et al. (2021), I would rather drop that baseline.\n\n2. While I understand the importance of the research question, I don't fully agree with the necessity or the benefit to use GNN for that. To me, there are simpler ways to inject such information, such as random substitutions (https://aclanthology.org/2020.emnlp-main.210.pdf) or contrastive learning (https://aclanthology.org/2021.acl-long.21.pdf). Have the authors considered those alternatives? Has there been any effort to verify that GNN is the better method?\n\n3. Lots of important related work is omitted, which I'll elaborate on in the missing references section.", "questions_to_authors": "A. I'm not entirely clear how is the graph transfer ratio $g_{i,j}^{s,t}$ (defined in L191) is used subsequently in Section 3.2? Is it used as initialization of the edge weights?\nB. L421-422: What is the \"original\" embedding referring to here?", "paper_topic_and_main_contributions": "This paper aims to improve multilingual neural machine translation systems by explicitly increasing the representation similarity of lexical translations, even in cases where different writing systems are used. To do this, the paper designed a word embedding re-parametrization mechanism based on the graph neural network (GNN). The high-level idea is to inject the lexical translation information obtained from a word aligner in the form of a word translation graph, thus allowing message-passing across lexical translations.\n\nThe method is evaluated on two different datasets, one being the IWSLT2014 multilingual benchmark also used by some prior work (e.g. LASS by Lin et al. (2021) as cited in the paper), the other being a new WMT-data-based benchmark created by the authors, called WMT30. Results indicate that the proposed method improves the model performance by +2.3 BLEU on average for IWSLT and +1.1 BLEU on WMT30. Further analysis also showed that the graph-based embeddings perform better by word similarity on the MUSE dictionary.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "GNN has been extensively used in neural machine translation. It would be good to at least mention a few:\n- https://aclanthology.org/P19-1320.pdf\n- https://aclanthology.org/N18-2078v2.pdf\n- https://arxiv.org/pdf/2012.03477.pdf\n- https://openreview.net/pdf?id=B1fA3oActQ\n- https://arxiv.org/pdf/1704.04675.pdf\n\n A few other papers that explored incorporating word alignment information into multilingual models:\n- https://aclanthology.org/2020.emnlp-main.210.pdf\n- https://aclanthology.org/2021.acl-long.21.pdf\n\nThis paper be used as a motivation: https://arxiv.org/pdf/2201.00075.pdf", "Typos_Grammar_Style_and_Presentation_Improvements": "- The word \"station\" in Figure 1 appears quite arbitrary to me. I'm not sure what point that part of the figure is trying to make.\n- L247: $\\alpha_i$ is used without introduction\n- L334: missing period\n- Table 2: I would suggest (1) adding M2M/NLLB model as a reference (you don't have to be better -- just to give the readers an idea of where your model stands) (2) including a breakdown of comparing results of non-latin-script languages -- this will reinforce your earlier point that this would enable lexical knowledge sharing across writing systems."}}, {"review_id": "X5s72hh9Nb", "reviewer": "Reviewer_hsEL", "summary": "", "questions": "A. How do these BLEU scores compare with bilingual MT system scores?\n\nB. You use MUSE for evaluating your word embeddings, but couldn't you construct a word similarity graph directly from MUSE?\n\nC. Have you considered building graphs for other relationships besides synonymy? For example, if in French word A is a hyponym of word B, and in English word C is a hyponym of word D, and we know that word A and word C are translations of each other, could the GNN somehow infer that word B and word D are more likely to be translations of each other?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts on our paper. We summarize and answer your questions as follows:\n\n>[Reason to Reject-1]: Semantic similarity of words across languages can be quantified is complicated by a few facts: 1) some words have multiple meanings, and 2) some \"subwords\" may not have meanings at all.\n\n**[Response-1]:** Yes, this is a good point and we also considered these during this project: \n1) Firstly, polysemy is not the only challenge for our method. Actually, it applies to the fact that we only use one embedding to represent the semantics of a word. The standard design of shared vocabulary also has this problem (Lines 65-67) and here we do not focus on this in this paper (lines 68-69). \n2) Secondly, we agree that some “short subwords” may not have explicit meaning. However, the information transfer between two “short subwords” may also help with modeling morphological aspects across languages. \n\n>[Reason to Reject-2]: A separate word-alignment stage is not aesthetic enough.\n\n**[Response-2]:** We accept this point.\n\n>[Weaknesses-3]: BLEU should be computed on untokenized references (e.g., using SacreBLEU), not tokenized references. Another metric like BLEU would be helpful.\n\n**[Response-3]:** We agree with this point and thank you for highlighting that it's not a reason to reject. To make our results comprehensive, we show the evaluation results in different metrics as well, e.g., Bleu, SacreBleu, ChrF++, and COMET. Please consult **Response-1 to Reviewer CxMx**. In short, the improvements are still evident (2+ points) for all of them. We will also include them in our final version.\n\n>[Question-1] How do these BLEU scores compare with bilingual MT system scores?\n\n**[Answer-1]:** Due to the limited time, it is hard for us to search for an optimal architecture for each language pair in our corpus. Instead, we show comparison within a limited setting. For the setting and results, please refer to **Response-B3 to Reviewer 4Nxw**. In short, 1) MMT shows better performance (however, we hesitate to make such a claim that MMT is better than bilingual MT because the backbone may not be optimal), 2) Our method is still able to bring some benefits in a bilingual setting.\n\n\n>[Question-2] You use MUSE for evaluating your word embeddings, but couldn't you construct a word similarity graph directly from MUSE?\n\n**[Answer-2]:** Good point. Using MUSE directly will make the architecture more aesthetic, and to be honest, we considered this point during this project as well. However, using MUSE to build graphs has some potential issues too: 1) MUSE has no frequency information. It will be hard to define the transfer ratio, resulting in losing control of the edge importance. 2) When a word in MUSE is split into multiple subwords, things get more complex. \n\n>[Question-3] Have you considered building graphs for other relationships besides synonymy? For example, if in French word A is a hyponym of word B, and in English word C is a hyponym of word D, and we know that word A and word C are translations of each other, could the GNN somehow infer that word B and word D are more likely to be translations of each other?\n\n**[Answer-3]:** This is an interesting point. However, we are somewhat hesitant about its viability for translation. This could be due to hyponyms occurring at different levels. E.g., A and C are “poodle” in French and English, while B and D mean “dog” and “animal” respectively. Here, B and D should not be considered as translations of each other.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The model is based on an intuitive idea, that word similarity should be transitive, and the improvements are large (2+ BLEU).", "reasons_to_reject": "I have some hesitation about including a word-similarity graph in an MT system, for two reasons.\n\nThe first is that the idea that the semantic similarity of words across languages can be quantified is complicated by a few facts:\n- Because of polysemy, some words have multiple meanings\n- Because of subword segmentation (line 159), some \"words\" do not have meanings at all\n\nThe second may be merely aesthetic; one of the benefits of neural MT over phrase-based MT was that it was a single end-to-end model, without separate stages for things like word alignment. This work brings back a separate word-alignment stage.\n\nNot reasons to reject, but weaknesses:\n- Please state earlier (line 187) what word alignment models are used, since this is part of your model. Also, please state explicitly what training data is used for the word alignment models.\n- BLEU should be computed on untokenized references (e.g., using SacreBLEU), not tokenized references.\n- Another metric like BLEU would be helpful.", "questions_to_authors": "A. How do these BLEU scores compare with bilingual MT system scores?\n\nB. You use MUSE for evaluating your word embeddings, but couldn't you construct a word similarity graph directly from MUSE?\n\nC. Have you considered building graphs for other relationships besides synonymy? For example, if in French word A is a hyponym of word B, and in English word C is a hyponym of word D, and we know that word A and word C are translations of each other, could the GNN somehow infer that word B and word D are more likely to be translations of each other?\n", "paper_topic_and_main_contributions": "This paper improves multilingual MT by (1) constructing a word similarity graph from word-aligned parallel text and (2) inserting a graph neural network after the word embeddings of the MT system. Improvements are large (2+ BLEU).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Tokenization methods in SoTA MT consider subwords with the same orthography as equivalent, which does not make much sense with totally unrelated languages. The paper proposes using semantic equivalence instead of orthographic equivalence. They achieve systematic improvements in translation quality in all major automatic metrics (some of which were added during the discussion period) both in small (IWSLT14) and large (WMT) data setup.

The paper received 4 reviews, 3 of which are very positive about the paper, both in terms of soundness and excitement, one review was very negative. Two of the reviewers raised important issues concerning the evaluation metrics, which were later clarified during the discussion period and the only negative-leaning reviewer increased their score.
Decision: Accept-Main