Code-Switching with Word Senses for Pretraining in Neural Machine Translation
Authors: Vivek Iyer
Edoardo Barba
Alexandra Birch
Jeff Z. Pan
Roberto Navigli
Venue: EMNLP 2023 Findings (2023)
Keywords: Word Sense Disambiguation
Pretraining approaches
Neural Machine Translation
TL;DR: We propose to integrate Word Sense Disambiguation and structured knowledge in the (code-switched) pretraining of multilingual NMT models, leading to performance improvements.

Abstract: Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT), with many state-of-the-art (SOTA) NMT systems struggling to handle polysemous words (Campolungo et al., 2022). The same holds for the NMT pretraining paradigm of denoising synthetic "code-switched" text (Pan et al., 2021; Iyer et al., 2023), where word senses are ignored in the noising stage -- leading to harmful sense biases in the pretraining data that are subsequently inherited by the resulting models. In this work, we introduce Word Sense Pretraining for Neural Machine Translation (WSP-NMT) - an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. Our experiments show significant improvements in overall translation quality. Then, we show the robustness of our approach to scale to various challenging data and resource-scarce scenarios and, finally, report fine-grained accuracy improvements on the DiBiMT disambiguation benchmark. Our studies yield interesting and novel insights into the merits and challenges of integrating word sense information and structured knowledge in multilingual pretraining for NMT.

Code-Switching with Word Senses for Pretraining in Neural Machine Translation
Vivek Iyer'
Edoardo Barba² Alexandra Birch'
Jeff Z. Pan' 'School of Informatics, University of Edinburgh SSapienza NLP Group, Sapienza University of Rome (vivek. iyer, pinzhen cinzhen.chen, a. bibrrch]@dd. ac. uk (barba, navigliJ@diag uniromal..
Roberto Navigli*
Abstract
similar efforts to plug sense information in NMT frameworks (Liu al., 2018; Pu al., 2018). But, since the introduction of the Transformer (Vaswani et al., 2017), the task of disambiguation has largely been left to the attention mechanism (Tang et al.. 2018, 2019).
Lexical ambiguity is significant and per- vasive challenge in Neural Machine Transla- (NMT), with many state-of-the-ar (SOTA) NMT systems struggling handle polysemous words (Campolungo et al., 2022a). The same holds for the NMT pretraining paradigm of de- noising synthetic "code-switched" text (Pan et al., 2021; Iyer et al., 2023), where word senses ignored in the noising stage lead- ing to harmful sense biases in the pretraining data are subsequently inherited by the re- sulting models. In this work, introduce Word Sense Pretraining for Neural Machine Translation (WSP-NMT) an end-to-end ap- proach for pretraining multilingual NMT mod- els leveraging word sense-specific information from Knowledge Bases. Our experiments show significant improvements in overall translation quality. Then, we show the robustness of our approach scale to various challenging data and cesource-scarce scenarios and, finally, re- port fine-grained accuracy improvements on the DiBiMT disambiguation benchmark. Our studies yield interesting and novel insights into the merits and challenges of integrating word sense information and structured knowledge in multilingual pretraining for NMT.
In the last three years, though, many works have challenged the ability of modern-day NMT sys- tems to accurately translate highly polysemous and/or rare word senses (Emelin et al., 2020; Ra- ganato et al., 2020; Campolungo et al., 2022a). For example, Campolungo et al. (2022a) expose ma- jor word sense biases in not just bilingual NMT models like OPUS (Tiedemann and Thottingal, 2020), but also commercial systems like Deepl. and Google Translate", and massively pretrained multilingual models (Tang et al., 2021; Fan et al.. 2021). A likely explanation is that these mod- els capture inherent data biases during pretrain- ing. This particularly holds for the pretraining paradigm of denoising code-switched text most notably, Aligned Augmentation (AA, Pan et al., 2021), where, during the pretraining phase, input sentences are noised by substituting words with their translations from multilingual lexicons, and NMT models are then tasked to reconstruct (or 'de- noise') these sentences. AA and subsequent works (Reid and Artetxe, 2022; Iyer et al., 2023; Jones et al., 2023) show the benefits of code-switched pretraining for high- and low-resource, supervised and unsupervised translation tasks. Despite their success, major limitation these substitution mechanisms is that they are unable to handle lexical ambiguity adequately, given their usage of 'sense- agnostic' translation lcxicons. In fact, in most of these works, substitutions for polysemes are chosen randomly, regardless of context (Pan et al., 2021;
1 Introduction
Lexical ambiguity is long-standing challenge in Machine Translation (Weaver, 1952) due poly- semy being one of the most commonly occurring phenomena natural language. Indeed, thanks to a plethora of context-dependent ambiguities (e.g. the word run could mean run a marathon, run a mill, run for elections etc.), words can convey very distant meanings, which may be translated with entirely different words in the target language. To deal with this challenge, traditional Statistical Ma- chine Translation approaches tried to incorporate Word Sense Disambiguation (WSD) systems MT with mostly positive results (Carpuat and Wu, 2007; Xiong and Zhang, 2014). These were followed by
'httpss//wwN deepl. cwm/en/ hoom/en/traan translator
Phttps://translee gorrraasaaaeegoglleee..... com com 'In this work, we refer to this family of approaches as 'code- switched pretraining' for brevity

---

Source Sentence:
had edge the compettion.
3.
We evaluate the robustness of WSP-NMT scale to various challenging data and resource- constrained scenarios in NMT, and point out its efficacy in low-resource and zero-shot translation tasks.
Baseline Translation (AA): Ha avuto un margine alla concomanta.
Our Transiation (WSP-NMT): Aveva vantaggio sulla concorennza.
Figure 1: Italian translations of a sentence from the DiBiMT disambiguation benchmark (Campolungo al., 2022a) by: a) our main baseline, Aligned Augmentation (AA, Pan et al., 1021), and b) our approach, WSP-NMT. AA mistranslates the ambiguous word edge as margine (border, rim). Due to "sense-pivoted pretraining'. WSP- NMT correctly translates it as vantaggio (advantage).
Finally, we evaluate the disambiguation capa- bilities of our models on the DiBiMT bench- mark, and contribute a fine-grained under- standing of the scenarios where WSP-NMT helps resolve lexical ambiguity in translation.
Reid and Artetxe, 2022: Jones et al., 2023).
2 Related Work
In effort to introduce knowledge grounding the word sense level during pretraining and potentially minimise data errors, enhance conver- gence, and improve performance, we propose the notion of 'sense-pivoted pretraining" - to move code-switched pretraining from the word level to the sense level. Specifically, we propose an ap- proach called Word Sense Pretraining for Neural Machine Translation (WSP-NMT) that first disam- biguates word senses in the input sentence, and then code-switches with sense translations for denoising- based pretraining. Figure provides an intuition of how integrating disambiguation in pretraining helps our model handle ambiguous words better, avoiding defaulting to more frequent senses, and reducing errors in translation.
Multilingual pretraining for NMT The success of multilingual NMT in the latest WMT shared tasks (Akhbardeh et al., 2021; ; Koemi al., 2022) has heightened the research focus on noising func- tions used in the denoising-base training of NMT models. While masking has been successful in scaling to massive models (Tang et al., 2021; Costa-jussà et al., 2022), there has also been parallel strand of research exploring more op- timal noising functions that grant superior per- formance at lower data quantities. In particular, code-switched pretraining (Yang et al., 2020; Lin et al., 2020; Pan et al., 2021) has gained popularity since moves the denoising task from language modelling to machine translation, and induces su- perior cross-lingual transfer. Notably, Pan et al. (2021) proposed Aligned Augmentation that code- switches text using MUSE lexicons, and trained the mRASP2 model yielding SOTA results on a wide variety of translation tasks, beating strong baselines pretrained on more data. Subsequent re- search has tried to extend this: Li et al. (2022) and Reid and Artetxe (2022) combine AA-like lexicon-based code-switching with masking, while Iyer et al. (2023) and Jones et al. (2023) explore optimal code-switching strategies. While effec- tive, none of these works have attempted move code-switching the word sense level. Most of them sample random sense translations when en- countering polysemes, with these errors further propagating during pretraining. Li et al. (2022) attempt to partially circumvent this issue by choos- ing the appropriate sense translation based on the reference sentence, but this trick cannot scale to code-switching (abundant) monolingual data or for code-switching in languages other than the target one. lyer et al. (2023) use pretrained translation and word alignment models to provide empirical gains, but the reliability is low with these black-
multilingual pre-
Indeed, our experiments on using WSP-NMT yield significant gains in multilingual NMT about +1.2 spBLEU and +0.02 COMET22 points over comparable AA baselines in high reshuresdrreed setups. Among other interesting performance trends, we observe that our margin of improvement increases substantially as we move towards low-resource (+3 to +4 spBLEU) and medium-resource BLEU) settings. Lastly, for more fine-grained evaluation, we also compare our models on the DiBiMT disambiguation benchmark (Campolungo et al., 2022a) for Italian and Spanish, and note accu- racy improvements of up to 15% in the challenging task of verb disambiguation.
resource (+5 sp-
Our key novel contributions are, thus, as follows:
1. We show how incorporating WSD in NMT pretraining can outperform the widely used paradigm of lexicon-bart based code-switching.
2. We demonstrate how reliable structured knowledge can be incorporated into the multi- lingual pretraining of NMT models, Icading to error reduction and improved performance.

---

box systems since such techniques lack grounding. In order to handle lexical ambiguity in a princi- pled manner, we propose WSP-NMT to provide knowledge grounding at the word sense level while pretraining multilingual NMT models.
between a pair of KGs, while Moussallem et al. (2019) augment Named Entities with KG Embed- dings, comparing RNNs and Transformers. Lu al. (2019) and Ahmadnia et al. (2020) enforce mono- lingual and bilingual constraints with KG rela- tions and observe small gains for bilingual English- Chinese and English-Spanist pairs respectively. It remains unclear how these constraint-basee meth- ods would scale to the current paradigm of pretrain- ing multilingual NMT models. Relatedly, Hu al. (2022) proposed multilingual pretraining leverag- ing Wikidata, but their sole focus was on Named Entity (NE) translation. They showed that their approach performs comparably overall, while im- proving NE translation accuracy. Our work is com- plementary to theirs and shows how a KG could be used to pretrain multilingual NMT model with generic concept translations, improving over- all NMT as well disambiguation performance.
WSD for Machine Translation In the traditional Statistical Machine Translation paradigm, incor- porating WSD was shown to improve translation quality (Carpuat and Wu, 2007; Chan et al., 2007; Xiong and Zhang. 2014). With the rise of NMT, various techniques were proposed to integrate word sense information. Nguyen al. (2018) simply provided annotations of Korean word senses to an NMT model, while other works then computed sense embeddings using tricks like Neural Bag-of- Words (Choi et al., 2017), bidirectional LSTMs (Liu et al., 2018) and adaptive clustering (Pu et al.. 2018) using these to augment word embeddings in the training of sequence-to-sequenne models (Bahdanau et al., 2015). With the introduction of the Transformer (Vaswani et al.. 2017), Tang et al. (2019) hypothesized that Transformer encoders are quite strong at disambiguation, and used higher- layer encoder reprexentations to report sense classi- fication accuracies of up to 97% on their test sets.
3
Approach
3.1 Definitions
Code-switching! is defined as the phenomenon of shifting between two or more languages in sen- tence. In this work, it refers to the 'noising' func- tion used for synthesising pretraining data. Word Sense Disambiguation the task of linking a word io in a context C' to its most suitable sense in a predefined inventory and can thus be rep- resented as mapping function f(w, C, I) s. In this work, we use BabelNet (Navigli et al., 2021) as reference sense inventory.
However, with the creation of challenging dis- ambiguation benchmarks more recently (Raganato et al., 2019, 2020; Campolungo al., 2022a), the purported capabilities of NMT models have been called into question once again. Indeed, when top- performing open-source and commercial systems were evaluated on the DiBiMT benchmark (Cam- polungo et al., 2022a), it was found that even the best models (i.e. Deepl. and Google Translate) yielded accuracies 50% when translating ambigu- ous words, biasing heavily towards more frequent senses, and vastly underperformed compared to the (then) SOTA WSD system, ESCHER (Barba et al., 2021a) - indicating significant room for improve- ment. While recent works have tried to address this by fine-tuning models with sense information (Campolungo et al., 2022b), explore if we can avoid an extra fine-tuning stage and incorporate WSD information while pretraining itself, so as to yield enhanced off-the-shelf performance.
3.2 Aligned Augmentation
Aligned Augmentation (AA, Pan et al., 2021) is a denoising-based pretraining technique that uses MUSE lexicons* M for code-switching the source sentence. These lexicons provide non-contextual, one-to-one word translations. If a word has mul- tiple possible translations (e.g. polysemes), AA (and subsequent works) randomly choose one gardless of context making this approach 'sense- agnostic'. Moreover, in order to code-switch in multiple languages, Pan et al. (2021) also chain bilingual lexicons together, by pivoting through English, and this causes a further propagation of "sense-agnostic" errors. Nevertheless, AA achieved SOTA results in massively multilingual NMT, so we use it as our primary baseline and aim to verify
Structured Knowledge for Machine Translation Though limited, there have been some scattered efforts to use Knowledge Graphs (KGs) in MT. bilingual NMT, Zhao et al. (2020b) generate synthetic parallel sentences by inducing relations
hhttps://githbb http:tttt..cco/eeeeeoorreeaaaaae

---

Word Translation Extraction
Code-Switched Sentence (AA)
String Search
Dictionary
Noisy due to Randomized Word Sense Assignment, Chainin lexicons etc.
Input Sentence
a) SOTA method: Aligned Augmentation (Sense-A Pretraining)
Multilingual NMT Pretraining
optional
Sense Tranalation Extraction from KG
Generation of Morphological Inflections
Word Sense Disambiguation
Knowledge Graph
Code-Switched Sentence (WSP-MT)
Muttlingual lesicalizations from synsets
Hashmap Lookup
b) Our method: WSP-NMT (Sense-Pivoted Pretraining)
Figure 2: Pipelines contrasting the approach used by the AA and WSP-NMT pretraining algorithms.
the impact of "sensc-pivoted" pretraining by com- paring against similarly trained WSP-NMT models. We also note that while some extensions to AA have been proposed recently, by combining it with masking or other code-switching strategies (see Section 2), these improvements likely also hold for code-switching with sense translations making these techniques complementary to our work.
izations for 520 languages with varying degrees of coverage. Thus, at the end of this stage, all content words will have list of possible translations in the target languages.
3.3.3 Generation of Morphological Inflections
Finally, given that concept lexicalizations in Babel- Net are present as lemmas, we include an additional postprocessing stage to convert target lemmas l of sense 8 into their contextually appropriate mor- phological variant m,- We do this by leveraging MUSE lexicons M. We first preprocess entries in M using lemmatizer L like so: given word trans- lation pairs (Izz 3/j) in M, we create a hashmap where H (z L(y,)) - yy Given H, during code- switching, if the source word is not a lemma we do a lookup for the key H (w;, 4,) and return the corresponding value the contextually appro- priate morphological inflection m, for sense which is then used for code-switching. If the key is unavailable in H. we code-switch w; with 4 itself. This step allows us to generate translations that take into account both word senses and morphol- ogy - thus combining the best of both worlds between 'sense-agnostic" lexicons and canonical- ized Knowledge Graphs storing sense informa- tion. While this postprocessing does improve per- formance, is optional and we show in Section 4.2 that baselines without this stage also yield decent improvements, while minimizing the overhead. We use Trankit (Nguyen et al., 2021) for lemmatiza- tion, which was shown to achieve SOTA results on this task on treebanks from 90 languages?.
3.3 Word Sense Pretraining for NMT
We depict the WSP-NMT pipeline, and compare it with AA, in Figure 2. It involves three stages:
3.3.1 Word Sense Disambiguation (WSD)
To determine the most appropriate translation for word we in sentence, begin by identify- ing its intended meaning in context using WSD systems. To measure the impact of disambigua- tion quality on the final translation performance, we adopt two WSD systems, namely, ESCHER, a SOTA open-innentory WSD model, and AMuSE- WSD (Orlando et al., 2021), faster but still com- petitive, off-the-shelf classification system. We note that, since ESCHER originally only re- leased for English, we also trained models for other languages on the multilingual datasets in XL-WSD (Pasini et al., 2021). We provide further training details in Appendix A.2. As a result of this stage, each content word in the input sentence is linked to specific synset" in BabelNet.
3.3.2 Extracting Translations from BabelNet Given the synset associated with disambiguated word, we retrieve its possible translations in the languages of experimentation by using BabelNet. Indeed, in NabelNet, each synset each language comes with set of lexicalizations that can express the concept represented by that synset. At the mo- ment of writing, BabelNet contains concept lexical-
3,4 Training
For fair comparison, follow the experimental setup proposed by Pan et al. (2021) for training our
'.c KGs representing concepts as canonicalized lemmas. 'https: ///rankit.. htt./rraniit..eattteccossss. meadthedocs.io/ee/LLLe io/en/latest/ performance. html
'A synset is group of semantically equivalent word senses.

---

Monolingual
Parallel
Language Pair
En-Es
En-Fr
En-It
En-Ro
En 7.5M Fr 7,5M It 7.5M En-Es 1.8 En-It 1.7M
Test Set
WMT
WMT
WMT
IMT
Es 7.5M Pt. 7.5M
7.5M
En-Fr 1.8M
En-Ro 365K
Language Pair
En-Pt
En-Hi
En-Fa
(a) Size of Romance language corpora (sentence count)
Monolingual
Parallel
Test Set
FLORES 200
WMT1 FLORES 200
En 20M Hi 6M Fa 6M
En-Hi 1.9M En-Fa 1M
Table Test sets used per language this work.
(b) Size of Indo-Iranian language corpora (sentence count)
Running Time (for 330K sents)
Table I: Monolingual and Parallel Data statistics.
Model
En
It
Fr
Es
Avg
AMUSE-WSD 77.13 76.03 80.35 72.77 76.57
78 mins 180 mins
ESCHER 78.72 79.14 83.94 77.52 79.83
AA and WSP-NMT models. We code-switch par- allel and monolingual sentences (with translations generated using the respective approaches) and shuffle them to create our training dataset D. We input the code-switched sentences to the encoder, while the target sentences are the reference sen- tences for parallel data and the original 'denoised' sentences for monolingual data. We prepend special language token indicating language ID to each input and target sentence. Finally, we train the model using a loss function L that jointly optimizes Contrastive Loss LCON and Cross Entropy Loss LCB- In principle, Contrastive Loss minimizes the semantic representation gap between positive examples (code-switched and reference sentences) and maximizes the distance between negative ex- amples (approximated other random samples in the minibatch). With this in mind, we define L as:
Table 3: Fl-scores of ESCHER and AMuSE WSD on XL--WSD (Pasini al., 2021) for English (En),, Italian (It), French (Fr), Spanish (Es), and the Average (Avg) of all scores. All results are statistically significant (p 0.01 as per McNemar's test (Dietterich, 1998)). For comparing efficiencies, we also report the running time for disambiguating 330K sentences on RTX 3090.
4
Experiments
4.1 Experimental Setup
Table I shows the statistics for our training data. In this work, we primarily experiment with the Ro- mance languages and pretrain multilingual NMT models on the monolingual and parallel data in Table la. following the training setup described previously. We use Portuguese later for zero- shot experiments, so parallel data is provided. We also explore NMT for Indo-Iranian languages in Section 4.3.3, and use the data given in Table 1b to train those models. We note here that since our objective in this work is to explore the util- ity of KGs for sense-pivoted pretraining, our data setups follow similar scale as other works on KG- enhanced NMT (Zhao et al., 2020a; Xie et al., 2022; Hu et al., 2022), and diverges from the massively multilingual paradigm of Pan al. (2021).
C - + |s| LcoN
where: LCE
log Po(y)z),,
(M,)ED
e (((+),&(W)/
LCON = Σ log (D.))ED
For validation, we sample the last 1000 sen- tences from the parallel corpus per pair. As shown in Table 2, for testing, we use either the latest WMT test sets per pair, if available, or FLORES 200 sets (Costa-jusså et al., 2022) otherwise. We evaluate NMT results using spBLEUS (Costa-jussà et al., 2022), chrF++ (Popović, 2017) and COMET22 (Rei al., 2022) metrics, and compute statistical significance using Student's T-test (Student, 1908). We use the Transformer (Vaswani et al., 2017) architecture with encoder and decoder layers
Elabren esim (E()).E(M)/T
Here. E signifies the average pooled encoder representations for the input sentence, while 'sim' computes positive and negative semantic similar- ities between the contrastive examples, denoted by sim and sim respectively. Temperature controls the difficulty of distinguishing between positive and negative samples, and is set 0.1. B denotes the mini-batch in dataset D that (Ir w) be- longs to. Lastly, |s] the average token count of sentence s that balances token-level cross entropy loss and sentence-level contrastive loss.
"nrefs: : 1 casee :aasc/amieeeeeeeeeeee...cc mixemmeff::fmmm tok: flores smooth:
exp version:2. 2. 1
/ec./ec:666 'nrefs: 1 caase missemmixedd :yes Inc: n:: ||sace:: no molversio: 2.

---

WSP-NMT (AMUSE WSDD)
1. And when Federer levelled the match, one-time Brilish No Petchey temia fearrer--faa- the worst.
(ESCHER)
15
WSP-NMT (ESCHER) w/ morph. inflection
S.RLEEU gain wr AA
2. The world was mirando (looking-loodk--miirr--mmii- at him.
1.0
3. GM deber (owes-rowe-ddebe-ddeee $23.4bn, excluding S3.9bn realised Treasury pérdidas sasssss---oss--prrrdddd-ddddddd according to Treasury statement dated Friday.
0.5
0.0
Figure 4: WSP-NMT with Morphological Inflection. Code-switched words are highlighted. a'->a->b~>b' denotes the inflection generation process explained in Section 3,3.3 where a' and b' are source and target words, while and b the corresponding lemmas. a->b is extracted from BabelNet, while the hashmap lookup yields b' given (a', b).
EniEs
Es-En Endfr frên
Enit
En-Ro RoiEn
(a) A gain in SpBLEU
WSP (AMuSE WSD)
0.03
WSP-NMT (ESCHER)
(ESCHER) morph. inflection
COMET22 gain Art
marginally underperforms these gains also ex- tend to individual language pairs (Figure 3). This suggests that minimising errors in pretraining data with 'sense- sense-pivoted pretraining' can enhance overall convergence and yield stronger models.
0.01
0.00
-0.01
En-Es
Es-En
En-Fr
FsEn
En-it
It-En
En-Ro
Ro-En
(b) A gain COMET22
2. Superior disambiguation helps, but cheaper WSD systems work well too. We observe in Fig- ure 3 that ESCHER consistently induces greater gains than AMuSE-WSD across all pairs. This makes sense given the superior disambiguation quality of the former (Table 3). However, we note that AMuSE-WSH WSD, which is about 2.3x faster, also outperforms AA and can be used as effec- tive, but cheaper alternative, suitable for disam- biguating large corpora under budget constraints.
Figure 3: Improvements in the average performance of WSP over AA for each language pair.
for training our models. For tokenization, we use sentencepiece (Kudo and Richardson, 2018). We train the sentencepiece model on concatenated and shuffled data from Table 1, using unigram lan- guage model with vocabulary size of 32,000 and character coverage of 1.0. We replacement ratios of 0.1 and 0.05 for our Romance and Indo- Iranian models respectively. We provide further hyperparameter settings in Appendix A.1.
3. Postprocessing KG lemmas to predict the rele- vant morphological inflection is beneficial. We observe generating morphological inflections by intersecting with MUSE lexicons (Section 3.3.3) yields major boosts across the board. We under- stand this qualitatively by showing examples of such inflections from our dataset, in Figure 4. We observe that this approach helps in ensuring linguistic agreement (such as for tense and num- ber) which, in turn, enhances the quality of code- switching and, thus, pretraining. Our technique can thus be used the gap between KG lemmas and morphologi- cally inflected words in corpora a gap which has proved to be one of the major roadblocks in leveraging structured knowledge NMT so far.
Lastly, in Table 3, we evaluate and report key performance statistics for our two WSD systems, AMuSE-WSD and ESCHER. We shall use these to better interpret our NMT results and comment the observed quality/speed tradeoff in Section 4.2.
4.2
Main Results
We show overall results for the Romance language experiments, averaged across 5 runs, in Table 4. We also report language pair-specific improvements in Figure? 3. We make the following key observations:
effective way to bridge
1. WSP-NMT consistently outperforms AA. We observe that our best WSP-NMT model, i.e. the one using ESCHER for disambiguation and
MUSE lexicons for generating morphological predictions, consistently beats AA models by significant margins across all metrics includ- ing a + +1.2 boost in spBLEU for En-X and +0.7 for X-En. Except for Fr-En where WSP-NMT
4. Higher gains are observed for the lower re- sourced En-Ro pair. WSP-NMT obtains +2
"TThs is possibly because the WordNet that BabelNet uses for French is WOLF (Sagot and Fišer, 2008), an automatically constructed silver quality resource

---

En-X
X-Rn
Baseline
SpBLEU
chrF++
COMET22
spBLEU
chrF++
COMET22
24.385 0.639
47 4 0.647
0.714
24.725 0.577
49.660
0,716 0.006
WSP-NMT (AMuSE- WSD)
24.910 =: 0.336
48.525 0.304
0.723 0.006
24.935 0.313
49.880
0.243
0.724 # 0.005
WSP-NMT (ESCHER)
25,310 = 0.909
48.765 = 0.717
0.728 = 0.013
23.095
49,905 0.705
0.725 = 0.010
WSP-NMT (ESCHER)
25.595 0.279
49.115 0.235
0.731 0.003
25.435
50.130 0.229
0.729 0.002
morph. inllection
Table 4: Overall results of the Romance language experiments with respect spBLEU, chrF++ and COMET22 metries. Mean and standard deviation after 5 runs, initialised with different random seeds (0-4) has been reported. Statistically significant improvements (p < 0.05) over our primary baseline, AA, are highlighted in bold.
spBLEU improvement over AA for the En-Ro pair - which is the lowest resourced in our setup, with 365K parallel sentences (Table 1). This sug- gests approach could be even for medium and low data setups something empirically verify Section 4.3.1.
useful
20
AA (En-X)
5. More significant boosts are obtained for COMET22 than string-based spBLEU or chrF++ metrics. While our best model improves across all the metrics, we observe this in Table for our weaker models. This is probably because the errors made by incorrect word sense trans- lations can sometimes be too subtle to ade- quately penalized by weaker string-based metrics. On the other hand, neural metrics like C0MET22 are more robust and have been trained to detect word-level well sentence-level errors. It has shown high correlations with human evaluation, ranking second in the WMT'22 Metrics shared task (Freitag et al., 2022). We, thus, conclude that statistically significant gains in COMET scores, when averaged across 5 seeds, help in increasing the strength and reliability of our results.
WSP-NMT (En-X)
16
AA (XCEn)
50K
100K
125K
250K
S00K
750K
IM
125M
15M
Parallel data language)
Figure 5: Average En-X and X-En spBLEU scores for AA and WSP-NMT models, with variation in parallel data (in terms maximum sentence count per language)
Baseline
En-Pt
Pt-Fn
AA
2.92 + 0.64
6.88 = 1.56
WSP-NMT (best)
3.60 0.19
8.52 + 0.53
Table 5: Zero-Shot Translation scores (spBLEU) for English- Portuguee (En-Pt) pair. use the best WSP-NMT model from Table 4 and compare with AA. Statistical significance (p=0.04) is highlighted in bold.
resource scenarios (750K-1.5M sentences), sug- gesting AA models become more robust noise when high quantities of parallel data are provided. Based on these findings, we discuss potential appli- cations of our method Section 4.5.
4.3 How well does WSP-NMT extend to more challenging MT scenarios?
Here, we try to evaluate the robustness of our ap- proach to scale to various challenging translation scenarios: a) data scarcity, b) zero-shot translation and c) translation of under-represented languages.
4.3.2 Scaling to Zero-Shot Translation
Continuing our exploration into data scarcity, we now evaluate the performance of WSP-NMT with no parallel data (i.e. zero-shot) in Table 5. We use En-Pt simulated zero-shot pair, following the setup in Table la. We compare our best WSP-NMT model from Table 4 with AA, averaged across 5 runs before. We observe statistically significant gains of about +0.7 and +1 1.64 spBLEU for En-Pt and Pt-En pairs respectively. We conclude that WSP-NMT is strong in a wide range of scenarios
4.3.1 Scaling to data-scarce settings
In Figure 5, we explore how performance varies with the scale of parallel data, while keeping mono- lingual data constant since the latter is usually more abundant. We observe an improvement mar- gin of +3 to +4 spBLEU the low-resource setting (50-100K sentences) that widens to +5 spBLEU in the medium-resource The gap then narrows as we move towards high-
resource setup (250-500K sentences).

---

En-X
X-Fn
4.4 DiBiMT benchmark results
AA
22.79 + 1.063
20.49 = 0.893
Finally, we test our models on the DiBiMT disam- biguation benchmark (Campolungo et al., 2022a) in Figure 6. The DiBiMT test set comprises sen- tences with an ambiguous word, paired with "good" and "bad" translations of this word. Then, given an NMT model's 's hypotheses, it computes accuracy, i.e. the ratio of sentences with "good" translations compared to the sum of "good" and "bad" ones. Hypotheses that fall in neither of these categories are discarded as "MISS' sentences. We compare AA and our best WSP-NMT model on Italian and Spanish, the two Romance languages supported by DiBiMT, and report overall accuracies. In addition, we also include fine-grained scores on disambigua- tion of specific POS tags like nouns and verbs to better understand the capabilities of our models.
WSP-MT (AMuSE-WSD)
22 0.704
19.58 + 0.559
WSP-MT (AMuSE-WSD)
21.83 à: 0.727
19.53 0.72
morph. inflection
WSP-MT (AMuSE-WSD)
22.71
20.23 + 0.906
+ synonyys hypemyms
Table 6: spBLEU scores for Indo-Iranian languages (Hindi and Farsi) to English, averaged across 5 seeds
and can improve performance at varying degrees of resourcedness, including in zero-shot tasks.
4.3.3 Scaling to under-represented languages
Lastly, we explore how WSP-NMT performs for languages under-represented in external resources such as WSD systems, lexicons, and KGs. The is- sues faced here include low coverage, poor quality, and unreliable WSD. To study the impact of these factors (in isolation from data scarcity), we train a multilingual NMT model between English and 2 Indo-Iranial languages (Hindi and Farsi) using the data in Table 1b. We choose these languages since they are under-represeented in BabelNer"I are supported by AMuSE- 'S and have MUSE lex- icons available for training AA basclines.
We observe a small increase in overall accuracy. Looking more closely, we find that while WSP- NMT's accuracy for nouns is comparable to AA, significant improvements are observed for verb dis- ambiguation especially for Italian, where average accuracy increases from 15.34% to 17.65% (15% increase). Prior research has shown that verbs are much harder to disambiguate than nouns since they are highly polysemous (Barba al., 2021a,b; Cam- polungo et al., 2022a) and require much larger con- texts to disambiguate (Wang et al., 2021). Thus, while sense- aannssagmottic AA models are able to disam- biguate nouns, verb disambiguation proves more challenging, which is where sense-pivoted pretrain- ing with WSP-NMT helps. We provide example translations in Appendix A.3 to drive intuition of this further. Finally, we note that WSP-NMT also reduces the average MISS% from 39.3% to 37.2% for Spanish, and 41.2% to 40.2% for Italian. These trends are encouraging, since errors such as word omission, word copying and hallucination are the most common issues and constitute about 50% of all 'MISS' cases (Campolungo et al., 2022a).
In Table 6, we find that integrating morphologi- cal inflection does not yield gains, likely because due to poor coverage we find matches in MUSE lexicons far fewer times (only 18.3% compared to 74% for the Romance data). BabelNet also suffers from poor coverage (45% of disambiguated senses do not have translations in either Hindi or Farsi). To address this, we explore if translations of syn- onyms and hypernyms of a KG concept could be used as substitutes. In practice, we find that, while this does improve the vanilla WSP-NMT model, is unable to beat AA. This is likely rooted in the fact that low-resource disambiguation is an unad- dressed challenge in NLP, primarily because of a lack of resources be it training datasets, reliable WSD models, or even evaluation benchmarks. De- spite these hurdles, we note that, since WSP-NMT is effective in data-scarce settings, creating a small amount of sense-annotated data in these languages might suffice to yield potential gains.
4.5 Applications
We conclude our results section by noting that while WSP-NMT is not ideally suited for either very low-resource (Section 4.3.3) or very resource- rich (Section 4.3.1) settings, it does yield signifi- cant performance gains for low and medium-data settings of well-resourced (e.g. European) lan- guages, as shown in Figure 5. We hypothesize that this can, thus, be quite useful for domain-specine translation in these languages, such as in news,
" http:///wannnnnttrrgssssatsstete 1 wnw.ilaa.nirrmallll. itl.ii/ammmaa tt/amsssewsdd
roi-documentation We are unable to use ESCHER here due to lack of relevant training data in XL-WSD

---

would be a useful direction of future research. The BabeINet KG also has the potential for improving in this regard, allowing for the addition of gender categories to specific concepts.
AA (Spannh)
WSS-NKI (Sparat)
30
AA (Rakka))
(ttalon)
04
We also note that WSP-NMT may not be effective in the translation of under-represented languages. As we note in Section 4.3.3, this is partly rooted in the unavailabilityy disambigua- tion resources and high-quality Knowledge Bases for such languages. However this is quickly chang- ing: i) WSD for low-resource languages has been a very emerging area of research recently (Saced et al., 2021; Bhatia et al., 2022; Boruah, 2022), and, at the same time, ii) BabelNet itself is constantly being updated and scaled up. For example, due to the creation of newer low-resource wordnets, BabelNet expanded from 500 supported languages (v5.1 from Jul 2022) to 520 (v5.2 from Nov 2022) in just 4 months. As the quality of these curated resources improves with community contributions and advances in low-resource NLP/WSD, we an- ticipate that methods such as WSP-NMT, which show to be robust in low-data settings (<IM parallel sents), could be increasingly useful.
ALL
NOUN
vERB
Tage
Figure 6: Accuracy of our AA and WSP-NMT models. in Spanish and Italian, on the DiBiMT benchmark.
healthcare, e-commeree, etc. In such information- centric domains, minimising disambiguation er- rors and improving translation quality with reliable, general-purpose KGs would likely bring value.
5 Conclusion
We propose WSP-NMT, a 'sense-pivoted" algo- rithm leveraging word sense information from KBs for pretraining multilingual NMT models. We ob- serve significant performance gains over the base- line, AA, and note various factors of interest in- cluding the impact of WSD quality and morphol- ogy. We then study how WSP-NMT scales var- ious data and resource-constrained scenarios and, finally, note significant accuracy gains in verb dis- ambiguation on the DiBiMT benchmark. Our re- sults help emphasize the utility and strength of WSP-NMT.
Acknowledgements
This work was funded by UK Research and In- novation (UKRI) under the UK government's Horizon Europe funding guarantee Igrant num- ber 10039436). The authors also gratefully ac- knowledge the support of the PNRR MUR project PE0000013-FAIR
Limitations
The computations described this research were performed using the Baskerville Tier 2 HPC ser- vice https///www.bsskkvilll.....m was funded by the EPSRC and UKRI through the World Class Labs scheme (EP/TO22221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Re- search Computing at the University of Birming- ham.
While our approach attempts to minimize the risk of low-quality translations creeping into the pre- training data by using human-curated structured knowledge. it is possible certain biased translations enter our data nonetheless. For instance, during code-switching, there is possibility of gender bias in the sense translations chosen, as certain gendered translations of a concept that ungendered in the source language are provided in the same word sense in BabelNet. Though we use random sam- pling to mitigate such risks, the propagation of po- tential biases depends heavily on the fairness and di- versity of translations available in BabelNet. While such issues are similar those that would arise in the current paradigm of dietionary-based code- switching, a principled approach to resolve gender based on sentence- context and to choose the appropriate translation from the BabelNet synset
Baskerville
References
Benyamin Ahmadnia, Bonnie J Dorr, and Parisa Kord- jamshidi. 2020. Knowledge graphs effectiveness in neural machine translation improvement. Computer Science, 21.
Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondfej Bojar, Rajen Chatter- jee, Vishrav Chaudhary, Marta R. Costa-jussa,

---

Cristina España-Bonae Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Ro- man Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Hcafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakae
Computational Linguisties: Human Language Tech- nologies, pages 4824 4838, Seattle, United States. Association for Computational Linguisties.
Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Au- guste Tapo, Marco Turchi, Valentin Vydrin, and Mar- cos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1 1-88, Online. Association for Computational Linguis- tics.
Marine Carpuat and Dekai Wu. 2007. Improving statisti- cal machine translation using word sense disambigua- tion. In Proceedings of the 2007 Joint Conference Empirical Methods in Natural Language Process- ing and Computational Natural Language Learning SEMNLP-CONLAS pages 72, Prague, Czech Re- public. Association for Computational Linguistics.
Yee Seng Chan, Hwee Tou Ng. and David Chiang. 2007. Word sense disambiguation improves statistical ma- chine translation. In Proceedings of the 45th annual meeting of the association of computational linguis- rics, pages 33-40.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. 2017. Context-dependent word representation for neural machine translation. Computer Speech Lan- guage, 45:149-160.
Edoardo Barba, Tommaso Pasini, and Roberto Nav- igli. 2021a. ESC: Redesigning WSD with extractive sense comprehension. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages Association for Computational Linguisties.
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi. Janice Lam, Daniel Licht, Jean Maillard. et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.0667
Online.
Thomas G Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7): 1895-
Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2021b. ConSeC: Word sense disambiguation as con- tinuous sense comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1492-1503, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Denis Emelin, Ivan Titov, and Rico Sennrich, 2020. De- tecting word sense disambiguation biases in machine translation for model- model-agnostic adversarial attacks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7635-7653, Online. Association for Computa- tional Linguisties.
Surbhi Bhatia, Ankit Kumar, and Mohammed Mutil- lah Khan. 2022. Role of genetic algorithm in opti- mization of hindi word sense disambiguation. IEEE Access, 10:75693-75707.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal. Mandeep Baines, Onur Celebi, Guillaume Wenzek. Vishrav Chaudhary, et al. 2021. Beyond english-centric multi- lingual machine translation. The Journal of Machine Learning Research, 22( (2(1):4339-444 4886.
Puberun Boruah. 2022. A novel approach to word sense disambiguation for a low-resource morphologically rich language. In 2022 IEFF 6th Conference on Information and Communication Technology (CICT), pages 1-6. IEEE
Niccolò Campolungo, Federico Martelli, Francesco Saina, and Roberto Navigli. 2022a. DiBiMT: A novel benchmark for measuring Word Sense Disam- biguation biases in Machine Translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 4331 14352, Dublin, Ireland. Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In Proceedings of the Seventh Conference Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Niccolò Campolungo, Tommaso Pasini, Denis Emelin, and Roberto Navigli. 2022h. Reducing disambigua- tion biases in NMT by leveraging explicit word sense information. In Proceedings of the 2022 Conference of the North American Chapter of the Association for
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangee embedding shar- ing.

---

Junjie Hu, Hiroaki Hayashi, Kyunghyun Cho, and Gra- ham Neubig. 2022. DEEP: DEnoising entity pre- training for neural machine translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers), pages 1753-1766, Dublin, Ireland. Association for Computational I Linguistics.
Diego Moussallem, Axel-Cyrille Ngonga Ngomo, Paul Buitelaar, and Mihael Arcan. 2019. Utilizing knowl- edge graphs for neural machine translation augmen- tation. In Proceedings of the 10th International Con- ference on Knowledge Capture, K-CAP '19, page 139-146, New York, NY, USA, Association for Com- puting Machinery.
Vivek Iyer, Arturo Oncevay, and Alexandra Birch. 2023, Exploring enhanced soie-switched noising for pre- training in neural machine translation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 954 968.
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. 2021. Ten years of BabelNet: A survey. In ICCAI, pages 4559-4567.
Alex Jones, Isaac Caswell, Ishank Saxena, and Orhan Firat. 2023. Bilex rx: Lexical data augmentation for massively multilingual machine translation. arXiv preprint arXiv:2303./520
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Vey- sch, and Thien Huu Nguyen. 2021. Trankit: A light- weight transformer-basee ural language processing. In Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguisties: System Demon- strations, pages 80-90, Online. Association for Com- putational Linguistics.
toolkit for multilingual nat-
Tom Koemi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann. Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45.
Quang- Phungg-PuccCaaa Nguyen, Anh-Dung Vo, Joon-Choul Shin, and Cheol-Young Ock. 2018. Effect of word sense disambiguation neural machine translation: A case study in korean. IEEE Access, 6:38512-38523,
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Riccardo Orlando, Simone Conia, Fabrizio Brignone, Francesco Cecconi, and Roberto Navigli. 2021. AMuSE-WSD: An all-in-one multilingual system for easy Word Sense Disambiguation. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 298-307, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Pengfei Li, Liangyou Li, Meng Zhang, Minghao Wu, and Qun Liu. 2022. Universal conditional masked language traininin for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( VVolume 1: Long Papers), pages 6379-6399 Dublin, Ireland. Association for Computational Linguisties.
Xiao Pan, Mingxuan Wang. Liwei Wu, and Lei Li. 2021. Contrastive learning for many many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 244-258, Online. Asso- ciation for Computational Linguistics.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2649- 2663, Online. Association for Computational Lin- guistics.
Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. XI-wsd: An extra-large and cross- lingual evaluation framework for word sense disam- biguation. Proceedings of the AAAI Conference Artificial Intelligence, 35(15):13648 31113666666666... 13656.
Frederick Liu, Han Lu, and Graham Neubig. 2018. Han- dling homographs in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Volume 1 (Long Papers), pages 1336-1345, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Maja Popović. 2017. chif++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612-618.
Xiao Pu, Nikolaos Pappas, James Henderson, and An- drei Popescu-Belis. 2018 Integrating weakly super- vised word sense disambiguation into neural machine translation. Transactions of the Association for Com- putettioal Linguistics, 6:635-649 649.
Yu Lu, Jiajun Zhang, and Chengqing Zong. 2019. Ex- ploiting knowledge graph in neural machine transla- tion. In Machine Translation: 14th China Workshop, CWMT 2018, Wuyishan, China, October 25-26, 2018, Proceedings 14, pages 27-38. Springer.
Alessandro Raganato, Yves Scherrer, and Jörg Tiede- mann. 2019. The MuCoW test suite at WMT

---

2019: Automatically harvested multilingual con- trastive word sense disambiguation test sets for ma- chine translation. In Proceedings of the Fourth Con- ference on Machine Translation (Volume 2: Shared Task Papers. Day 1), pages 470-480.
for Computational Linguistics: ACL-IJCNLP 2021. pages 3450-344 3466, Online. Association for Computa- tional I. Linguisties.
Jörg Tiedemann and Santhosh Thottingal. 2020. Opus- mt-bbilding open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation.
Alessandro Raganato. Yves Scherrer, and Jörg Tiede- mann. 2020. An evaluation benchmark for testing the word sense disambiguation capabilities of machine translation systems. In Proceedings of The 12th Lan- guage Resources and Evaluation Conference. Euro- pean Language Resources Association (ELRA).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metries shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Daphne Wang, Mehrnoosh Sadrzadeh, Samson Abram- sky, H Víctor, and VH Cervantes. 2021. Analysing ambiguous nouns and verbs with quantum contextu- ality tools. Journal of Cognitive Science, 22(3):311- 420.
Warren Weaver. 1952. Translation. In Proceedings of the Conference on Mechanical Translation.
Machel Reid and Mikel Artetxe. 2022. PARADISE:
Exploiting parallel data for multilingual sequence- to-sequence pretraining. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 800-810, Seattle, United States. Association for Computational Lin- guistics.
Shufang Xie, Yingce Xia, Lijun Wu, Yiqing Huang, Yang Fan, and Tao Qin. 2022. End-to-end entity- aware neural machine translation. Machine Learning pages 1-23.
Deyi Xiong and Min Zhang. 2014. A sense-based trans- lation model for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Pupers), pages 1459-1469, Baltimore, Mary- land. Association for Computational Linguistics.
Ali Saced, Rao Muhammad Adeel Nawab, and Mark Stevenson. 2021. Investigating the feasibility of deep learning methods for urdu word sense disambigua- tion. Transactions on Asian and Low-Resoure Lan- guage Information Processing, 116
7hen Yang, Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020. CSP:code-switthingg pre- pre-training for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2624-2636, Online. Association for Computational Linguistics.
Benoît Sagot and Darja Fišer. 2008. Building a free french wordnet from multilingual resources. In On- toLex.
Student. 1908. The probable error of a mean. Biometrika, 6(1):1-25.
Yang Zhao, Lu Xiang, Junnan Zhu, Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2020a. Knowledge graph enhanced neural machine translation via multi- task learning on sub-entity granularity. In Proceed- ings of the 28th International Conference on Compu- tational Linguistics, pages 4495 4505.
Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2018. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine trans- lation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 26-35, Brussels, Belgium. Association for Computational Linguistics.
Yang Zhao, Jiajun Zhang. Yu Zhou, and Chengqing Zong 2020b. Knowledge graphs enhanced neural machine translation. In Proceedings of the Twenty- Ninth International Joins Conference on Artificial Intelligence, IJCAI-20, pages 4039-4045. Interna- tional Joint Conferences on Artificial Intelligence Organization. Main track.
Gongbo Tang, Rico Sennrich, and Joakim Nivre. 2019. Encoders help you disambiguate word senses in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
pages 1429- 1435, Hong Kong, China. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Ii, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2021. Multilingual translation from de- noising pre-training. In Findings of the Association

---

Hyperparameter
Romance
Indo-Iranian A.3
Verb Disambiguation Examples
Batch Size
4000
4000
Learning Rate
0.0001
0.0001
company good a year. L'impresa ha trasformato un buon profitoo dopo un anno.
Update Frequency
4
AA;
Adam
Adam
WSP-NMT:
La
copo
Optimizer
w/ eps=5e-07
w/ eps=le--00
(a) "trasformato" "transformed" and incorrect
Weight Decay
0.0001
0.01
translation of "turned". : "fatto" means "made", in "made good profit" and
Dropout
0.05
0.01
translation in context.
Label Smoothing
0.1
0.1
Source:
appropriate money for the increase navy.
Contrastive Lambda
1.0
1.0
AA:
Por adeguare denaro l'aumento della tassa.
Temperature
0.1
0.01
WSP-NMIT
Per stanziare fondi per I'aumento dellImbarcazione.
Clip Norm
10
10
Replacement Ratio
0.1
0.05
(b) "adeguare" is a verb meaning "adapt" or "adjust" and is incorrect sense translation of "appropriate" "stanziare" means "allocate", "allocate funds", and is correct in context.
Table 7: Hyperparameters used for training our models.
Source:
The player had backpedal before catching the bal.
Running Time
Il giocatore aveva dovuto tornare prima di catturare balls.
Model
Pr
Es
Params
(for 100K sents)
WSP.-MMT
grocatore
prmaa prendere
ESCHER 78.72 79.14 83.94 77.52
692 M
mins 213 mins
(c) "aveva dovuto tornare" translates "had return" and incorrect translation of "backpedal". "tornato indietro" means "move (or run) back" and contextually correct.
ConSeC: 79.00 79.30 84.40 77.40 287 M
Table 8: Results of four different ESCHER models trained on English (En), Italian (It), French (Fr) and Spanish (Es) and ConSeC.
Figure 7: Examples of verb disambiguation in DiBiMT Italian hypothesis translations by AA and WSP-NMT.
A Appendix
We show some examples of improved verb dis- ambiguation of WSP-NMT in Figure 7, using ex- amples from the DiBiMT test set. The AA hy- potheses for these sentences contain translations that could have been appropriate certain contexts but are incorrect here. For example, in Figure 7a, turned could translate into transformato when the context is referring to "turning into something". A similar explanation holds for Figure 7b where appropr iare could translate into adeguare when talking about "appropriating style", "appropriating language" "appropriating software". to name few examples. Figure 7c is very interesting since backpedal does mean tornato indietro ("move back") in this context, but "tornare" ("return") is not an appropriate sense translation. These cases reflect how by carefully modelling for cross-lingual convergence of word senses during pretraining, WSP-NMT can avoid subtle translation enrors that may arise when encountering ambiguous words.
A.1
Hyperparameters
We provide our detailed hyperparameter settings in Table 7. We optimize each hyperparameter inde- pendently in separate runs, and use the best settings in our experiments.
A.2 WSD Systems Details
the original AMSSE-WSD work, a multilingual version of the system was made available; however, ESCHER currently offers only an English model. For this reason, we train three distinct ESCHER models specifically tailored for Italian, Spanish, and French. We adopt the methodology outlined in the original work but adapt ing the XL-WSD (Pasini et al., 2021) splits for the respective languages and use mDRB (He et al., 2023) as the underlying multilingual transformer architecture. By doing so, we ensure compatibility and improve performance in the targeted linguistic contexts reported in Barba et al. (2021b) with- out incurring prohibitive computational costs. As a reference, we report in Table 8 the difference in performance between our ESCHER models and the results reported in Barba et al. (2021b) for the multilingual version of their system, ConSeC.
train the models us-


Reviews:
[{"review_id": "dva6eNI0YY", "reviewer": "Reviewer_4nWC", "summary": "", "questions": "1. To validate whether WSP-NMT learns a better representation space, there could be a comparison experiment that sense-appropriate sentence pairs are similar than sense-mismatched sentence pairs. \n\n2. Continuing with Question 1, there could be room for applying a contrastive loss between sense-appropriate sentence pairs and sense-mismatched sentence pairs. \n(It's acknowledged that learning the difference between positive and negative samples will be difficult.) \n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your detailed and comprehensive review. We are pleased you found our work well-motivated and our experiments sound and noteworthy. The concerns you raise are highly interesting and we feel are worth discussing in some depth, so we have put forth our view below:\n\n\n## Reasons to Reject\n\n\nWe do agree that it can be challenging to apply WSP-NMT to rich-resource or very low-resource languages.\n\n1. We see your point, but actually, our data setup follows a similar scale as recent works on KG-enhanced NMT [1][2][3][4] that use data quantities ranging anywhere between 200K-2M parallel sentences. We diverge from the data setup of Pan et al., 2021, since unlike them we do not seek to train a massively multilingual NMT model. We shall clarify these points in the paper.\n\nRegarding rich-resource languages, while we are limited by computational constraints, it is possible that training on huge amounts of parallel data diminishes the gains offered by our approach. Such a result would align with previous works on KG-enhanced NMT – where it has been observed that while knowledge grounding does help with increased reliability, quality, and task-specific performance in quite a few cases, overall results (such as BLEU scores) improve with massive scale regardless.\n\n\n2. Even on the applicability of WSP-NMT for low-resource languages, we agree with you: i) reliable disambiguators are not yet available and ii) knowledge bases for those languages are still not comparable to rich-/medium-resource ones. However this is quickly changing: i) WSD for low-resource languages has been a very emerging area of research recently [5][6][7][8], and, at the same time, ii) BabelNet itself is constantly being updated and scaled up. For example, due to the creation of newer low-resource WordNets, BabelNet expanded from 500 supported languages (v5.1 from Jul 2022) to 520 (v5.2 from Nov 2022) in just 3 months. As the quality of these curated resources improves with community contributions and advances in low-resource NLP/WSD, we anticipate that methods such as WSP-NMT, which we show to be robust in low-data settings (<1M parallel sents), could be increasingly useful.\n\n\nWe conclude by noting that while WSP-NMT is not ideally suited for either of these extreme settings, it does appear to be beneficial for low and medium-data settings of well-resourced (eg. European) languages. This can be quite useful for domain-specific translation in these languages, such as in news, healthcare, e-commerce etc. In such information-centric domains, minimising sense-agnostic errors and improving translation quality with reliable, general-purpose KGs would likely be of value. \n\nThis is what we try to convey in L494-L499, but we will make this clearer in the camera-ready version and shall add a short paragraph of discussion on potential applications of our method.\n\n## Questions to the Author:\n\nVery interesting suggestions! Incidentally, both of these questions are being planned as part of future work.\n\n1. We agree that analyzing the representation space induced by WSP-NMT will make our paper more interesting. We will add the results of a disambiguation probing experiment in the extra page of the camera-ready version, wherein we shall probe encoder representations of ambiguous words, using models trained by AA and WSP-NMT algorithms. To conduct the probing, we shall train a simple linear classifier on these representations using the standard framework for WSD evaluation [9] and compare test accuracy to verify whether our pretraining strategy improves the sense-awareness of the final representation space.\n\n\n2. We were originally planning to try this version of contrastive loss as well. However, we realized a potential issue with considering only other senses of the word as negative examples is that it would minimize entropy and probably affect how general sentence-level representations are learnt. A middle ground would be to take a good mixture of sentences with opposite senses as well as random sentences as negative examples. Another option is to take a weighted sum of these two contrastive losses. We are considering these directions, along with a form of curriculum learning (for learning general representations first and sense-specific ones later), as part of future work. While these should further improve our results, we felt this was outside the scope of our current work, which sought to give a proof of concept of sense-pivoted pretraining.\n\n## Grammar improvements\n\nThank you - as a principle we have not used chatGPT in this work. Sorry about this wording though, we shall correct this.\n\n\n## References\n\n[1] Zhao et al., 2020  Knowledge Graph Enhanced Neural Machine Translation via Multi-task Learning on Sub-entity Granularity\n\n[2] Zhao et al., 2021 Knowledge Graphs Enhanced Neural Machine Translation\n\n[3] Xie et al., 2022 End-to-end entity-aware neural machine translation\n\n[4] Hu et al., 2022 DEEP: DEnoising Entity Pre-training for Neural Machine Translation\n\n[5] Rouhizadeh et al., 2020, Knowledge-based word sense disambiguation with distributional semantic expansion for the Persian language \n\n[6] Saeed et al., 2021, Investigating the Feasibility of Deep Learning Methods for Urdu Word Sense Disambiguation\n\n[7] Bhatia et al., 2022, Role of Genetic Algorithm in Optimization of Hindi Word Sense Disambiguation\n\n[8] Boruah, 2022, A Novel Approach to Word Sense Disambiguation for a Low-Resource Morphologically Rich Language\n\n[9] Raganato et al., 2022, Word sense disambiguation: a unified evaluation framework and empirical comparison", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is clear. \n- The authors target the core weakness of the existing code-switched pre-training strategy, which struggles to address lexical ambiguity by generating noise with sense-agnostic lexicons. \n\n2. The authors show their achievements in resolving lexical ambiguity effectively. \n- It's especially noteworthy for the performance gains on verb inflections, which are difficult to disambiguate due to the plethora of polysemous words. ", "reasons_to_reject": "1. The experimental configuration in terms of high, medium, and low resource settings slightly departs from other studies. \n- The authors set 100\\~200K, 500K\\~1M, 1.5M\\~3M as low, medium, and high resource settings based on parallel corpus size. However, in the PC32 dataset paper [2] used by the previous work [1], which the authors follow the experimental setting, much larger datasets were used as rich-source. Considering Figure 3 and 5 (if x-axis is scaled to a huge size of dataset), it might be unable to beat AA [1] in high-resource setting. \n\n2. The applicability of WSP-NMT is questionable. \n- The authors claim that WSP-NMT outperforms on low-resource language pairs. Since many data-scarce language pairs are under-represented in WSD models and BabelNet, there might be several situations where it is difficult to utilize WSP-NMT. \n\n[2] Lin et al., 2020, Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information\n", "questions_to_authors": "1. To validate whether WSP-NMT learns a better representation space, there could be a comparison experiment that sense-appropriate sentence pairs are similar than sense-mismatched sentence pairs. \n\n2. Continuing with Question 1, there could be room for applying a contrastive loss between sense-appropriate sentence pairs and sense-mismatched sentence pairs. \n(It's acknowledged that learning the difference between positive and negative samples will be difficult.) \n", "paper_topic_and_main_contributions": "This paper proposes a novel pre-training strategy for multilingual NMT models called \"Word Sense Pretraining for Neural Machine Translation (WSP-NMT)\". Recent pre-training paradigms have centered on Aligned Augmentation (AA) [1], which allows pre-training by denoising a few tokens in the source sentences substituted by sense-agnostic multilingual lexicons, thus limiting to fully solve lexical ambiguity. Therefore, the authors shift the code-switched paradigm from sense-agnostic pre-training towards sense-pivoted pre-training. \n\nThe WSP-NMT pipeline consists of three stages: 1) Word sense disambiguation (WSD), 2) Extracting sense translation, 3) Transforming morphological infections. For each stage, they leverage the appropriate SOTA models and knowledge base. \n\nFinally, they effectively demonstrate their improvements in lexical ambiguity resolution on the DiBiMT benchmark dataset along with the BLEU measure. \n\n[1] Pan et al., 2021 Contrastive Learning for Many-to-many Multilingual Neural Machine Translation\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "It would be recommended to revise the expression \"posit\", which is widely used by Chat-GPT but awkward in Point 5 of 4.2 Main Results. ", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "rY9db7Mnsq", "reviewer": "Reviewer_sGfj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your high scores and appreciation of our work! We are pleased you found our approach easy and intuitive and our experimental results effective. \n\nRegarding your concern of our technique’s applicability for low-resource languages, we do agree this is challenging at the moment. But, as we point out to Reviewer 4nWC, low-resource disambiguation is an active and emerging field of research currently [1][2][3][4]. Simultaneously, BabelNet itself is constantly expanding and its quality is constantly being refined. As the quality of both disambiguators and curated resources improves with the research community contributions and advances in low-resource NLP, we anticipate that methods such as ours that are shown to be robust in low-data settings (<1M parallel sents) would be increasingly useful.\n\n### References\n\n[1] Rouhizadeh et al., 2020, Knowledge-based word sense disambiguation with distributional semantic expansion for the Persian language \n\n[2] Saeed et al., 2021, Investigating the Feasibility of Deep Learning Methods for Urdu Word Sense Disambiguation\n\n[3] Bhatia et al., 2022, Role of Genetic Algorithm in Optimization of Hindi Word Sense Disambiguation\n\n[4] Boruah, 2022, A Novel Approach to Word Sense Disambiguation for a Low-Resource Morphologically Rich Language", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "See the main contributions.", "reasons_to_reject": "See the weaknesses in Paper Topic And Main Contributions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "What problem or question this paper addresses\n+ There is a method that learns denoise of sentences that were swapped with other language words using a bilingual dictionary. When different meaning words are used to swap because of polyseme, it leads to harmful sense biases. \n\nMethod\n+ The authors estimate synsets using WSD. Then they obtain words in other languages using a knowledge graph. The obtained words in other languages are used to swap words in the training data for a denoising task. \n\n\nThe main contributions (Strengths) \n+ This method is a straight and reasonable solution to the problem. \n+ The experimental results show the effectiveness of the method. \n\nWeaknesses\n+ This method uses a WSD method, synsets, and a knowledge graph. Therefore, this method cannot be applied to the languages or domains for which these resources are not available. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "drqrL0wzP7", "reviewer": "Reviewer_MJt2", "summary": "", "questions": "1. Can you provide more potential explanations as to why your method might underperform compared to the baseline?\n2. When using larger or smaller models, do you believe the conclusions proposed in this paper remain consistent?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this glowing review! We are pleased that you too found the writing quality of our work high, our idea logical and easy to understand and our experiments comprehensive. We address your concerns below:\n\n## Reasons to Reject\n\n1. Unlike the statistically significant improvements in other language pairs, the performance drop for En-Fr and Fr-En is mostly statistically insignificant (only the AMuSE WSD baseline is significantly worse). As for why significant improvements were not observed for French, this is possibly because of the relatively poorer quality of BabelNet in this language. The WordNet that BabelNet uses for French is WOLF [1], which is an automatically constructed silver-quality resource. \n\nWe promise to clarify both these points in Section 4.2, and also to indicate the statistically significant changes in Figure 3.\n\n2. Actually, we do analyze the contribution of the various subcomponents of our method in Table 4 by verifying the role of WSD quality (with AMuSE and ESCHER) and that of morphological inflection. \n\nMoreover as promised to R1, in order to provide deeper analysis of our results, we will be conducting disambiguation probing of encoder representations of models pretrained using AA and WSP-NMT respectively using a linear WSD classifier. This will help clarify whether our method encodes sense-specific features when compared to sense-agnostic pretraining. We will include these results in the extra page of the camera-ready version.\n\n## Questions to the Author\n\n1. Answered just above, thanks for pointing this out!\n\n2. We had not tried this before but, on your suggestion, we just tried this during the rebuttal phase and observed similar improvements. We trained Transformer-large models for AA and WSP-NMT with Escher WSD (ie. 12 encoders and 12 decoders, instead of 6 and 6 as used in our work), using the same data setup and the same random seed (seed=0). We show our average scores below:\n\nAA: 27.55 spBLEU (En-X) and 28.25 spBLEU (X-En)\n\nWSP-NMT:  28.80 spBLEU (En-X) and 28.90 spBLEU (X-En)\n\nThis translates to an average improvement of +1.25 spBLEU and +0.65 spBLEU for En-X and X-En respectively. En<->Ro still gains the most -- with +2.2 spBLEU and +1.7spBLEU for En-Ro and Ro-En respectively. We shall add these findings in the camera-ready version, along with the results of the WSP-NMT baseline with morphological inflection (which we could not try now due to time constraints, but would likely yield even higher gains).\n\n## References\n\n[1] Sagot and Fišer, 2008. Building a free French wordnet from multilingual resources", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The quality of writing is high and reader-friendly. The paper is well-organized.\n2. The idea presented is logical and easy to understand.\n3. Comprehensive experiments are conducted to verify the effectiveness of the proposed method in different scenarios, such as different language pairs across high, medium, and low resources.", "reasons_to_reject": "1. For some experimental results, there is a lack of reasonable and sufficient explanation. For instance, in figure 3, the authors' method underperforms the baseline in the en-fr and fr-en settings. The reason and analysis for this are missing.\n2. A majority of the experiments focus on the presentation of results. The analyses of the method itself and the experimental outcomes are not comprehensive enough. Given that the authors' method underperforms the baseline in some instances, one might question to what extent the performance improvement brought by this pretraining method can be attributed to the authors' claim of \"moving code-switched pretraining from the word level to the sense level, by leveraging word sense-specific information from Knowledge Bases\".", "questions_to_authors": "1. Can you provide more potential explanations as to why your method might underperform compared to the baseline?\n2. When using larger or smaller models, do you believe the conclusions proposed in this paper remain consistent?", "paper_topic_and_main_contributions": "This paper analyzes the shortcomings of existing methods in solving the polysemy problem, with the main issue being that \"they are unable to handle lexical ambiguity adequately, given their usage of ‘sense-agnostic’ translation lexicons.\" In order to address this issue, the authors propose a sense-pivoted pretraining method, which shifts codes-switched pretraining from the word level to the sense level, leveraging word sense-specific information from Knowledge Bases. Experimental results on WMT'13 and FLORES 200, by training a transformer with 12 layers, demonstrate that the proposed method can enhance translation quality, especially in medium and low resource settings. The improvement in high-resource settings is 1.2 spBLUE and 0.02 COMET22, in medium-resource settings it's 5 spBLUE, and in low-resource settings it's 3-4 spBLUE.\n\nContribution：\n1. Proposal of a novel code-switched pretraining method, integrating Word Sense Disambiguation into NMT pretraining.\n2. Comprehensive experiments are conducted to evaluate the performance of the proposed method.\n3. The authors also evaluate their method on the DiBiMT benchmark, which measures the disambiguation capabilities.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method that consists in using Word Sense Disambiguation along with Knowledge Graphs during pretraining to address the translation of polysemous words.
The problem is highly relevant for the community and the method precisely address weaknesses of current methods.
The paper is clear and well written, the experimentations are well conducted and the results demonstrate the effectiveness of the proposed method, even if some additional analysis would sometimes be beneficial. The authors mentioned in their rebuttal that most of the missing analysis will be added in the camera ready version.
The main concern is the usability of the approach for very low and very high resource languages, but this is not a criteria to reject the paper per se.
Decision: Accept-Findings