SlowGuess/NAACL_2022_OCR huggingface dataset:
These clusters are used to create the following two challenge sets: WSD Bias and Adversarial. The former quantifies the intrinsic bias the model learned during training, while the latter measures how sensitive the model is to the insertion of terms that are usually associated with another sense cluster during training. Both challenge sets evaluate in terms of accuracy of correct disambiguation. A more detailed description of these datasets and their evaluation process is provided in  $\S$  A.1.
DIBIMT We also evaluate on the German and Spanish portions of DIBIMT (Campolungo et al., 2022), a recent fully-manually annotated disambiguation bias challenge set, where models are asked to translate English sentences containing ambiguous words, and their translations are checked for either correct or incorrect translation equivalents, which, in contrast to previous benchmarks, are annotated manually and depend on the context of the sentence instead of relying solely on the sense of the source word.
4.3 Comparison Systems
We compare our sense-enhanced model with the following architectures:
With a fallback strategy to fast-align (Dyer et al., 2013) in case no alignment is produced.
7We filter out all nouns appearing in the stopwords list provided by BabelNet.
1. OPUS (Tiedemann and Thottingal, 2020): a strong bilingual model which uses the same architecture and parameter count as ours, although it was trained on order of magnitudes more data;
2. MBart-50 (Tang et al., 2021): the English-to-many version of the MBart-50 model;
3. Baseline: our base NMT models, trained on the datasets described in § 4.2.
In what follows we refer to our model fine-tuned with SCR as Baseline+SCR.
We note that, due to the way in which the WSD Bias Challenge Sets were constructed (i.e., by using sentences reserved from WMT14, see § 4.2), any fair evaluation against OPUS and MBart-50 is to be considered impossible, as such models have seen the sentences in the challenge sets during training. We therefore evaluate these two models only on standard BLEU, and point out that the resulting scores should only be regarded as references for our models' competence in the translation task.
In what follows, first, we show that our model attains BLEU scores in the same ballpark as state-of-the-art approaches such as OPUS and MBart-50, despite the large gap in terms of parameters or training data. Then, we focus our evaluation on the WSD Bias, and compare our full-fledged model (Baseline+SCR) against its baseline variant.
5.1 General Translation Quality
In Table 1 we observe that the trained baselines are more than competent in the translation task: indeed, when considering average BLEU scores, they place between OPUS, which is trained on much more data but has the same parameter count, and MBart-50 (Tang et al., 2021), which is  $\sim 8$  times larger but is capable of translating English to 50 languages.
In contrast to common debiasing techniques, which often observe a degradation in performance on standard benchmarks (Clark et al., 2019; He et al., 2019), we report consistent BLEU improvements on all language pairs, all of which are statistically significant at different p-values (Table 1), providing empirical proof that the proposed method does not hurt the model's general translation capability, while at the same time it helps models generate less biased translations (as will be discussed in the upcoming sections).
5.2 Disambiguation Bias
Results on the Disambiguation Bias Challenge Sets (§ 4.2) are reported in Table 2, for both of which we show improvements: on the WSD Bias Challenge Set, the bias is reduced, significantly, by more than  $1\%$ ; similarly, on the Adversarial Challenge Set, we see a reduction of homographs mistakenly disambiguated due to the injection of adversarial adjectives of  $0.27\%$ . We attribute this lower impact to the artificial nature of the adversarial sentences, some of which, by manual inspection, display poor grammatical fluency.
5.3 WSD Performance
We conduct an analysis of the performance of EWISER on the English sentences of the WSD Bias Challenge Set, to see how it fares in comparison with our NMT models. Unfortunately, as the sense clusters are not directly associated with BabelNet synsets, we reconstruct this association automatically and manage to retrieve only 1847 of the 3000 sentences in the challenge set.
Having retrieved BabelNet synsets for the target terms, we can apply EWISER and check whether the disambiguated synset matches one of the synsets retrieved for the sense cluster of the challenge sentence. Let us consider our running example, "The energy comes from a distant plant," one last time: if EWISER disambiguates the term plant to its sense of organism, we count it as a mistake, similarly to the case where our NMT model translates it as pianta instead of impianto (i.e., its sense of factory). With this in mind, we evaluate EWISER, Baseline and Baseline+SCR on the aforementioned subset of sentences; we report the results of this evaluation in Table 2 (bottom).
The results indicate that, for this setting, both NMT models actually perform quite a lot better than a pre-trained disambiguation system. One reason for this might be the different distributions the models are trained on: by design, the challenge sentences follow a distribution similar to the corpus used to train the NMT model, whereas EWISER is trained on sentences coming from news corpora from the 1960s and dictionary-like definitions. Moreover, in theory, if we were to apply the refinement process described in § 3.2 to disambiguate the challenge sentences, we would achieve a perfect score, as the target German lemmas are monosemous and thus the disambiguation is im
${}^{8}{10k}$  bootstrap samples of  ${50}\%$  the test set's size each.
EN → DE	EN → FR	EN → ES
WMT14	WMT19	WMT14	WMT13
OPUS†	27.58	39.39	39.93	35.00
MBart-50‡	25.60	35.80	36.12	29.50
Baseline	26.34	36.93	38.05	32.82
Baseline+SCR	27.26	37.74	38.48	33.18
Baseline+SCR_KL	26.13	36.45	37.85	33.15
Baseline+SCR_ARES	25.75	35.93	37.33	32.49
Baseline+SCR_AR	26.11	36.74	37.38	32.93
Baseline+SCR RAND	25.63	34.79	/	/
plicitly solved. The results of using EWISER's raw annotations are discussed in  $\S 5.5$ .
Finally, we choose not to perform a similar comparison on the Adversarial Challenge Set, as its examples are designed to specifically target NMT models via adversarial injections; we leave studying their impact in WSD systems as future work.
Table 1: Standard evaluation results. Numbers represent SacreBLEU scores. Statistical significance is computed according to Paired Bootstrap Resampling (Koehn, 2004) w.r.t. the row above. Underlined numbers represent  $\underline{p} < 0.02$  and  $\underline{p} < 0.001$ . represents systems that accessed more training data than us, but with the same parameter count. represents systems that, beside access to a larger pool of data, also feature bigger underlying models.
MODEL	WSD Bias ↓	Adversarial ↓
Baseline	12.27	4.48
Baseline+SCR	11.23	4.21
Baseline+SCR_KL	12.43	5.14
Baseline+SCR_ARES	12.53	4.75
Baseline+SCR_AR	13.07	4.93
Baseline+SCR RAND	12.56	5.04
EWISER	13.70	/
Baselinecf	11.86	/
Baseline+SCRcf	9.91	/
Table 2: Results on WSD Bias Challenge Sets. Numbers represent error rates (lower is better). Underlined results represent statistical significance at  $p < 0.001$ , compared to the row above, according to McNemar's test (McNemar, 1947).
5.4 System Examples
In Table 3, we report some examples of disambiguation corrected by our model according to the WSD Bias Challenge Set. The baseline is translating the terms to their most frequent sense (column Wrong sense), instead of the correct one (column Target sense). Moreover, the third example shows that this is not only a word matching task, as the improved model is able choose the correct subword
and can capture the nuances of meaning in more uncommon senses.
Ablation on SCR To measure the importance of the KL term in the loss, we fine-tune the model without including it in the SCR objective (§ 3.3) and report the results in Tables 1 and 2 (row Baseline+SCR $_{-KL}$ ). We observe that, without KL, the model struggles to leverage the double inputs efficiently; indeed, its translation performance drops around 1 BLEU point on average, while the error rates increase by roughly  $1\%$  on both bias challenge sets. These results back our intuition that the KL divergence helps to distill sense information from the sense-enhanced inputs, and is indeed a crucial component to our formulation.
Ablation on ARES We also test our system replacing the pre-trained sense embeddings provided by ARES with randomly initialized learnable embeddings and report this result in Tables 1 and 2 (row Baseline+SCR  $-\mathsf{ARES}$ ). As expected, both translation quality and disambiguation bias drop consistently. Indeed, learning sense embeddings from scratch is much harder than learning a mapping between a fixed space and a trainable one.
Ablation on Annotation Refinement We evaluate our sense Annotation Refinement process (§ 3.2) by fine-tuning the model on the unconstrained sense annotations provided by EWISER (Baseline+SCR\_AR), i.e., by considering the synset with the highest confidence on the source word as the correct one, instead of  $S^{*}$ . In the bias
Source sentence / Reference sentence / Baseline output / Enhanced output	Target sense	Wrong sense
S: [...] that both first words start with the same letter.
R: [...] dass beiden Begriffe mit demselben Buchstaben beginnen.
B: [...] dass beiden Wörter mit dem gleichen Brief beginnen.
E: [...] dass beiden Wörter mit dem gleichen Buchstaben beginnen.	alphabet symbol	written message
S: At least since the fall of 2008, leading economies' officials have agreed [...]
R: Spätestens seit Herbst 2008 stimmen die Vertreter führender [...]
B: Zumindest seit dem Fall 2008 haben sich die Beamten [...]
E: Zumindest seit dem Herbst 2008 haben sich die Beamten [...]	season	act of falling
S: The construction of the Deurganck dock lock is [...]
R: Der Bau der Schleuse am Deurganck-Dock ist [...]
B: der Bau der Deurganck-Hafensperre ist [...]
E: der Bau der Deurganck-Hafenschleuse ist [...]	segment of a canal	blockade
Table 3: Examples of sentences that were disambiguated correctly by our enhanced model but not by the baseline. Ambiguous word is in blue, wrong translation is in red, correct translation is in green.
Model	EN → DE	EN → ES
OPUS†	27.99	36.66
MBart-50‡	28.73	33.89
Baseline	24.00	26.44
Baseline+SCR	25.00	25.84
Table 4: Accuracy scores on D1B1MT. † and ‡ have the same meaning as in Table 1. Higher is better.
evaluation (Table 2), the performances on both challenge sets drop significantly  $(p < 0.001)$ , which is in line with EWISER's performance on this challenge set ( $\S$  5.3). Furthermore, the BLEU scores drop too, although not as significantly (Table 1), but still always under-performing with respect to Baseline+SCR.
Ablation on Sense Annotations Finally, we test whether the sense annotations have an impact by replacing them with random senses for the specific word, drawn from the sense vocabulary with uniform probability, during the fine-tuning stage (Baseline  $+\mathrm{SCR}_{\mathrm{RAND}}$ ). As expected, we observe that randomly injecting senses is detrimental, with important performance drops in both the standard and the bias evaluation benchmarks.
5.6 Evaluation on DIBIMT
In Table 4 we report the results obtained on DIBIMT (Campolungo et al., 2022). For the sake of conciseness, we only report accuracy scores as a proxy for the general disambiguation bias dis
played by our models.
While on English  $\rightarrow$  German we observe an improvement of  $1\%$ , the performance on English  $\rightarrow$  Spanish decreases by around  $0.6\%$ . We hypothesize that our English  $\rightarrow$  Spanish model might be undertrained, as its accuracy differs by around  $10\%$  from OPUS, its direct comparison, while on English  $\rightarrow$  German the difference is only of around  $3\%$ . We leave further investigation of this issue, including training larger, more capable models, as future work.
In this paper, we presented a fine-tuning strategy that, by leveraging the explicit sense annotations produced by a novel high-precision technique, effectively reduces the disambiguation bias of a baseline Neural Machine Translation model while at the same time also strengthening translation performances, without introducing any requirement at inference time.
Our analysis on a strong disambiguation system showed that its ability to disambiguate polysemous nouns is worse than that of a baseline NMT model, at least in the studied out-of-domain setting.
We believe that this work paves the way for better bias reduction techniques in MT, while also fostering interest in the issue represented by the disambiguation bias. As future work, we plan to further study the ability of NMT models to perform Word Sense Disambiguation and to strengthen research at the intersection of these two fields, with a view to building stronger and more reliable models.
$^9$ Due to time constraints, we only perform this ablation on the English  $\rightarrow$  German model.
The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 and the PerLIR project (Personal Linguistic resources in Information Retrieval) funded by the MIUR Progetti di ricerca di Rilevante Interesse Nazionale programme (PRIN 2017).
This work was also partially supported by the MIUR under the grant "Dipartimento di eccellenza 2018-2022" of the Department of Computer Science of Sapienza University.
Edoardo Barba, Tommaso Pasini, and Roberto Navigli. 2021. ESC: Redesigning WSD with extractive sense comprehension. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4661-4672, Online. Association for Computational Linguistics.
Edoardo Barba, Luigi Procopio, Niccolò Campolungo, Tommaso Pasini, and Roberto Navigli. 2020. MuLaN: Multilingual label propagation for word sense disambiguation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3837-3844. International Joint Conferences on Artificial Intelligence Organization.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330-4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006-1017, Online. Association for Computational Linguistics.
Giulia Bonansinga and Francis Bond. 2016. Multilingual sense intersection in a parallel corpus with diverse language families. In Proceedings of the 8th
Global WordNet Conference (GWC), pages 44-49, Bucharest, Romania. Global Wordnet Association.
Jose Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. NASARI: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities. Artificial Intelligence, 240:36-64.
Niccolò Campolungo, Federico Martelli, Francesco Saina, and Roberto Navigli. 2022. DiBiMT: A Novel Benchmark for Measuring Word Sense Disambiguation Biases in Machine Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Online and Dublin, Ireland. Association for Computational Linguistics.
Marine Carpuat and Dekai Wu. 2005. Word sense disambiguation vs. statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 387-394, Ann Arbor, Michigan. Association for Computational Linguistics.
Marine Carpuat and Dekai Wu. 2007. Improving statistical machine translation using word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 61-72, Prague, Czech Republic. Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4069-4082, Hong Kong, China. Association for Computational Linguistics.
Simone Conia and Roberto Navigli. 2021. Framing word sense disambiguation as a multi-label problem for model-agnostic knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3269–3275, Online. Association for Computational Linguistics.
Claudio Delli Bovi, Jose Camacho-Collados, Alessandro Raganato, and Roberto Navigli. 2017. EuroSense: Automatic harvesting of multilingual sense annotations from parallel text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 594-600, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Denis Emelin, Ivan Titov, and Rico Sennrich. 2019. Widening the representation bottleneck in neural machine translation with lexical shortcuts. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 102-115, Florence, Italy. Association for Computational Linguistics.
Denis Emelin, Ivan Titov, and Rico Sennrich. 2020. Detecting word sense disambiguation biases in machine translation for model-agnostic adversarial attacks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7635–7653, Online. Association for Computational Linguistics.
Viktor Hangya, Qianchu Liu, Dario Stojanovski, Alexander Fraser, and Anna Korhonen. 2021. Improving machine translation of rare and unseen word senses. In Proceedings of the Sixth Conference on Machine Translation, pages 614-624, Online. Association for Computational Linguistics.
He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132–142, Hong Kong, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-395, Barcelona, Spain. Association for Computational Linguistics.
Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86.
Helen Langone, Benjamin R. Haskell, and George A. Miller. 2004. Annotating WordNet. In Proceedings of the Workshop Frontiers in Corpus Annotation at HLT-NAACL 2004, pages 63–69, Boston, Massachusetts, USA. Association for Computational Linguistics.
Frederick Liu, Han Lu, and Graham Neubig. 2018. Handling homographs in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1336-1345, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
Yixing Luan, Bradley Hauer, Lili Mou, and Grzegorz Kondrak. 2020. Improving word sense disambiguation with translations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4055-4065, Online. Association for Computational Linguistics.
Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2017. Embedding words and senses together via joint knowledge-enhanced training. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 100-111, Vancouver, Canada. Association for Computational Linguistics.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157.
George A. Miller, R.T. Beckwith, Christiane D. Fellbaum, D. Gross, and K. Miller. 1990. WordNet: an online lexical database. Int. J. Lexicogr, 3(4):235-244.
George A. Miller, Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. A semantic concordance. In Proc. of the Workshop on Human Language Technology, pages 303-308, Plainsboro, N.J.
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Ceconi. 2021. Ten years of BabelNet: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4559-4567. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial intelligence, 193:217-250.
Quang-Phuoc Nguyen, Anh-Dung Vo, Joon-Choul Shin, and Cheol-Young Ock. 2018. Effect of word sense disambiguation on neural machine translation: A case study in korean. IEEE Access, 6:38512-38523.