SlowGuess/NAACL_2022_OCR huggingface dataset:
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Luigi Procopio, Edoardo Barba, Federico Martelli, and Roberto Navigli. 2021. MultiMirror: Neural crosslingual word alignment for multilingual word sense disambiguation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3915-3921. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. 2018a. Integrating weakly supervised word sense disambiguation into neural machine translation. Transactions of the Association for Computational Linguistics, 6:635-649.
Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. 2018b. Integrating weakly supervised word sense disambiguation into neural machine translation. Transactions of the Association for Computational Linguistics, 6:635-649.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. Association for Computational Linguistics.
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 470-480, Florence, Italy. Association for Computational Linguistics.
Annette Rios Gonzales, Laura Mascarell, and Rico Senrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11-19, Copenhagen, Denmark. Association for Computational Linguistics.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020a. Sense-annotated corpora for word sense disambiguation in multiple languages and domains. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5905-5911, Marseille, France. European Language Resources Association.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020b. With more contexts comes better performance: Contextualized sense embeddings for all-round word sense disambiguation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3528-3539, Online. Association for Computational Linguistics.
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from denoising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450-3466, Online. Association for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
David Vickrey, Luke Biewald, Marc Teyssier, and Daphne Koller. 2005. Word-sense disambiguation for machine translation. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 771-778, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021. Multi-view subword regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473-482, Online. Association for Computational Linguistics.
Zikang Wang, Linjing Li, and Daniel Zeng. 2020. Knowledge-enhanced natural language inference based on knowledge graphs. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6498-6508, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Warren Weaver. 1949. Translation. Machine Translation of Languages: Fourteen Essays.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing.
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
A.1 Bias Evaluation Challenge Sets
We here provide a more detailed description of the datasets introduced by (Emelin et al., 2020). From § 4.2, recall that these challenge sets are based on sense clusters built on BabelNet, where each sense cluster contains an English polysemous word and a set of German monosesemous terms, which uniquely identify a certain meaning.
We highlight that there is no direct link between the sense clusters and the data produced by our Annotation Refinement process, as the sense clusters are i) heavily manually refined<sup>10</sup> and ii) based on the entire BabelNet4 inventory (16M concepts), while EWISER only covers the subgraph of BabelNet linked to WordNet (117k concepts), as is common in the multilingual WSD setting. As such, we do not consider the evaluation to be in any way more favorable towards our system.
WSD Bias contains sentences whose targeted English term is likely to be translated into a specific different sense due to co-occurrences of words in the sentence itself. For example, in the sentence "a lot of money was spent to renovate the capital" the word capital is likely to be translated into its sense of amount of money due to the presence of the words money and spent. A mistake is detected if the term is translated into any of the German words contained in the most likely sense cluster. The goal of this task is to measure the intrinsic bias the model learned during training.
Adversarial contains two sets of sentences, the original sentence and its adversarial counterpart, built by injecting an adjective that is likely to flip the disambiguation performed by the NMT model towards a specific sense. For example, given the sentence "they met in the spring of 2020", the adversarial example would be "they met in the hot spring of 2020". The injection of hot leads the model to translate spring into its sense of water source as opposed to its correct sense of season. A mistake is detected every time the non-adversarial sentence is translated into the correct
sense, whereas its adversarial counterpart is flipped to the sense cluster the adjective points to. The goal of this task is to measure how sensitive the model is to the insertion of terms that are usually associated with another sense cluster during training.
A.2 Training a Sense-Enhanced NMT Model
Our work is based on the assumption that providing a neural model with sense annotations for ambiguous words helps in disambiguating them. While this is rather intuitive, and has been shown to be the case in previous works (Nguyen et al., 2018; Pu et al., 2018b), we test this hypothesis in our setting by training an NMT model, from scratch, with sense-enhanced sentences only (see § 3.3 for details). We train a model comparable with the Baseline (i.e., same architecture and hyperparameters) on the English  $\rightarrow$  German training set (§ 4.2), and observe that it achieves higher BLEU scores than the Baseline (which is trained on the same data but with plain sentences). For instance, the sense-enhanced model achieves a BLEU score of 27.22 on WMT14 and 36.79 on WMT19, with the first being a statistically significant improvement. This confirms, once again, that sense-enhanced NMT models are on par or better than plain NMT models, although they introduce the heavy requirement of WSD at inference time, which our work aims at dropping.
A.3 Reproducibility Details
Preprocessing Times The preprocessing of the datasets needed to apply Annotation Refinement (lemmatization, Part-of-Speech tagging and then disambiguation through EWISER) required around 4 days in total on an RTX 2080 Ti (roughly 3M sentences per day).
Training infrastructure and duration All our experiments were carried out on either an NVIDIA RTX 2080 Ti or a RTX 3090, depending on availability.
Model training required on average 4 days on a 3090, 7 days on a 2080 Ti. Fine-tuning epochs required around 10 hours each (on a 3090), with most finishing due to early stopping before the end of the second epoch.
Parameter counts We used HelsinkiNLP MarianMT models available on Hugging-Face Transformers (Wolf et al., 2020) (e.g., for EN→DE, the model name is Helsinki-NLP/opus-mt-en-de). For
<sup>10</sup>As discussed in § 5.3, almost  $40\%$  of the challenge instances could not be linked back to BabelNet synsets, further confirming the impact of the manual refinement performed.
MODEL	WSD Bias	Adversarial
Correct ↑	%Error ↓	Correct ↑	%ErrorATTR ↓	%ErrorOTH ↓
Baseline	71.37	12.27	86.10	4.48	0.40
Baseline+SCR	73.27	11.23	87.36	4.21	0.34
Baseline+SCR_KL	70.37	12.43	85.30	5.13	0.40
Baseline+SCR_ARES	70.53	12.53	85.75	4.75	0.45
Baseline+SCR_AR	70.20	13.07	86.40	4.93	0.35
Baseline+SCRRAND	68.83	12.56	84.51	5.04	0.63
EWISER	68.54	13.70	/	/	/
\( Baselinen_{cf} \)	72.77	11.86	/	/	/
Baseline+SCRcf	75.58	9.91	/	/	/
Table 5: Full results on WSD Bias Challenge Sets. Numbers represent percentages.
instance, EN→DE has 74.4M parameters, EN→ES has 77.9M, EN→FR has 75.1M.
For the fine-tuning stage we added ARES (frozen), thus adding a number of parameters equal to ARES's size (1536) times the number of unique synsets in the dataset (refer to Table 6 for approximate numbers). We also added a trainable projection layer of size  $1536 * 512$  (512 is the Transformer's hidden dimension), thus adding  $786k$  trainable parameters (which we drop after the fine-tuning).
Model training hyperparameters Similarly to (Emelin et al., 2020), we trained it on the entire dataset for a max of 100,000 steps with approximately 24k tokens per batch, label smoothing at 0.1 and an inverse square root learning rate scheduler with 4000 warmup steps. As optimizer, we used Adam (Kingma and Ba, 2015) with betas (0.99, 0.98) and learning rate  $7 \cdot 10^{-4}$ , additionally employing an early stopping strategy with patience 5, monitoring the BLEU score on a validation set. We produced translations at inference time using a beam size of 5.
Fine-tuning hyperparameters For the fine-tuning, we resumed training using the weights of the baseline models, changed the learning to  $1 \cdot 10^{-5}$  and reduced the warmup to 1000 steps; additionally, we evaluated the model every  $10\%$  of the fine-tuning steps rather than after each epoch, as we observed fast convergence during fine-tuning and multiple epochs were superfluous.
A.4 Disambiguation Bias Results
Table 5 reports the same results displayed in the paper, but includes the percentage of Correct translations for both challenge sets as well as the percentage of errors made from sentences that, after the injection of the adversarial adjectives, were translated into a sense that was neither the correct one, nor the one targeted by the adversarial injection (i.e., other).
A.5 Data Statistics
CORPUS	EN-DE	EN-ES	EN-FR
# sentences	4.13M	3.54M	5.09M
# tokens (src / tgt)	99.7M / 96.8M	94.7M / 98.7M	133M / 142M
# annotated sentences	2.97M	3.11M	4.25M
# annotations	6.5M	11.2M	13.6M
# EN terms vocab	634k	592k	808k
# EN terms covered	34.0k	40.0k	37.5k
# unique synsets	16.0k	20.5k	15.7k
Table 6: Training and fine-tuning produced data statistics.
A.6 Limitations of this work
Our work focuses on reducing the disambiguation biases picked up by NMT models during training. We acknowledge some limitations in our work:
1. Due to limited computational budget and the large number of resources required to train and fine-tune NMT models from scratch, we had to limit ourselves to one run per experiment, though, despite this, the consistency across languages seems to point to the empirical correctness of the claims.
2. We evaluated the bias reduction explicitly only on the English  $\rightarrow$  German language pair. The reason for this was twofold: first, the datasets introduced by Emelin et al. (2020) only cover said pair, and require the accompanying training data be used in order to fully exploit the co-occurrences (and hence the biases) that the model is evaluated upon; second, upon manual inspection, we found that MuCoW (Raganato et al., 2019) contains many irrelevant candidates in its translation suite, and is in general very strongly affected by the noisy nature of BabelNet.
3. Our pipeline is strictly tied to both the accuracy of the multilingual WSD system employed and by the coverage of the underlying sense inventory. While EWISER and Babel-Net work reasonably well for high-resource languages, the quality of the annotated corpus might decrease for low-resource ones.