SlowGuess/NAACL_2022_OCR huggingface dataset:
Niccolò Campolungo
campolungo@di.uniromal.it
Denis Emelin
University of Edinburgh, Scotland
d.emelin@sms.ed.ac.uk
Tommaso Pasini
pasini@di.uniromal.it
Recent studies have shed some light on a common pitfall of Neural Machine Translation (NMT) models, stemming from their struggle to disambiguate polysemous words without lapsing into their most frequently occurring senses in the training corpus. In this paper, we first provide a novel approach for automatically creating high-precision sense-annotated parallel corpora, and then put forward a specifically tailored fine-tuning strategy for exploiting these sense annotations during training without introducing any additional requirement at inference time. The use of explicit senses proved to be beneficial to reduce the disambiguation bias of a baseline NMT model, while, at the same time, leading our system to attain higher BLEU scores than its vanilla counterpart in 3 language pairs.
Translating a sentence requires the underlying meaning to be captured and then expressed in the target language. Nonetheless, only little attention has been devoted to studying the actual capabilities of Neural Machine Translation (NMT) approaches of modeling different senses of ambiguous words, with recent work showing that systems tend to be biased towards the most frequent meanings found within the training corpus (Emelin et al., 2020). This phenomenon is hard to measure through classical evaluation metrics, such as the BLEU score (Papineni et al., 2002), as they often rely on word-matching heuristics that fail to capture the disambiguation capabilities of the evaluated systems. Therefore, several efforts have been recently devoted to shed some light and create test beds (Rios Gonzales et al., 2017; Raganato et al., 2019; Emelin et al., 2020; Campolungo et al., 2022) to challenge NMT models. Results show that these models still struggle to deal with highly polysemous words, especially when used to express least frequent senses.
For example, given the sentence "The energy comes from a distant plant," both Google Translate and DeepL disambiguate<sup>1</sup> plant to its sense of organism when translating into Italian, and produce the following incorrect sentence "L'energia proviene da una pianta lontana," rather than "L'energia proviene da un impianto lontano," where impianto is the translation for the factory meaning of plant. This suggests that, even when adequate context is provided (energy should be enough to correctly infer the right sense of plant), state-of-the-art models might still be biased towards the most frequent meanings found within training data.
Some recent studies have explored how to leverage explicit sense information within NMT models (Rios Gonzales et al., 2017; Pu et al., 2018a; Nguyen et al., 2018). Nevertheless, including such information is not trivial for three main reasons: i) sense-tagged parallel data is scarce; ii) Word Sense Disambiguation (WSD) systems have not been accurate enough until very recently (Blevins and Zettlemoyer, 2020; Barba et al., 2021); and iii) how explicit senses should be incorporated within neural models is not straightforward.
In this paper, we first introduce a novel approach to make up for the paucity of sense annotations in parallel corpora, leveraging a multilingual WSD system to tag parallel sentences and refine its predictions by means of cross-lingual word alignments and information from a multilingual knowledge base. Then, we fine-tune our baseline models on our sense-tagged corpora via a specifically designed loss function, allowing the injection of word-level semantics into the architecture. We evaluate our approach on standard and challenge test sets, showing that it does indeed improve translation accuracy and mitigates the most frequent sense bias.
To summarize, our contributions are manifold:
At the time of writing: January  $5^{\mathrm{th}}$ , 2022.
Human Language Technologies, pages 4824 - 4838
1. We put forward a novel approach to produce high-precision sense annotations for parallel data, which we apply to three language pairs.
2. We propose a fine-tuning strategy that lets us inject word-level explicit semantics into Neural Machine Translation models, without introducing any additional requirement at inference time.
3. We show that employing explicit sense tags is beneficial in order both to mitigate the sense bias and to improve the translation quality in terms of BLEU score on standard benchmarks.
4. We present a case study on how a state-of-the-art WSD system compares to an NMT model on disambiguating words within a challenging set for detecting sense bias in MT.
We make all the generated datasets, the code of the model and for the experiments available at https://github.com/sapienzanlp/reducing-wsd-bias-in-nmt.
Word Sense Disambiguation was first formulated as a computational task by Weaver (1949) in the context of Machine Translation. The two fields then followed parallel paths, with more or less successful attempts over the years to join them back together (Carpuat and Wu, 2005; Vickrey et al., 2005; Carpuat and Wu, 2007). Indeed, while Carpuat and Wu (2005) reported negative results when trying to integrate the prediction of a supervised WSD approach into a Statistical Machine Translation (SMT) model, the same authors, two years later, successfully improved the performance of a phrase-based SMT approach by leveraging a new phrase-based WSD model (Carpuat and Wu, 2007). More recently, Pu et al. (2018a) and Nguyen et al. (2018) proposed systems that successfully leverage sense information in NMT models, although they introduced a heavy requirement, i.e., that of disambiguating the ambiguous words in the sentence prior to generating a translation, which makes them unfeasible in many real-world settings. Lately, contextualized word embeddings have been employed to produce additional back-translated parallel training data via mining sense-specific target sentences, in order to improve handling of infrequent senses (Hangya et al., 2021).
Nevertheless, the proper treatment of lexical ambiguity is still an open problem, with neural models struggling to translate least frequent senses and often relying on spurious correlations among words (Emelin et al., 2020; Raganato et al., 2019; Rios Gonzales et al., 2017). Thus, the disambiguation bias topic has received renewed interest, and several benchmarks have been introduced in the most recent years with the goal of directly measuring the extent to which neural architectures are able to capture word semantics. One of the first of this kind was ContraWSD (Rios Gonzales et al., 2017). In this first attempt to evaluate WSD capabilities of NMT models, the authors built an adversarial test set where source sentences containing an ambiguous word were associated with a correct translation and several incorrect alternatives. These latter were built by replacing the reference translation for the ambiguous word with the translation of one of its other possible meanings. The task measured whether a model ranked the correct translation higher, i.e., it assigned it a higher probability than the adversarial ones. This study provided evaluation data for two language pairs only, i.e., German  $\rightarrow$  English and German  $\rightarrow$  French, and within a few years it became outdated as modern NMT models could easily attain high performances (Emelin et al., 2019). Thus, MuCoW (Raganato et al., 2019) took things a step further and leveraged BabelNet (Navigli and Ponzetto, 2012; Navigli et al., 2021) – a large multilingual knowledge base – and sense embeddings (Camacho-Collados et al., 2016; Mancini et al., 2017) in order to automatically create adversarial translations for five language pairs while also increasing the difficulty of the task itself; however, the fully automatic nature of these challenge sets made them noisy and prone to containing irrelevant challenge samples.
More recently, Emelin et al. (2020) proposed two challenge sets for the English  $\rightarrow$  German pair, one measuring the model sensitivity to most frequent senses and the other estimating, through adversarial injections, its susceptibility to changing a correct sense to a wrong one. In contrast to previous studies, these challenge sets were based on correlations among words in the training set and relied on manually-refined sense clusters, providing an excellent test bed for measuring semantic bias.
Finally, Campolungo et al. (2022) proposed DIBiMT, the first fully manually annotated test set for measuring the disambiguation bias of neural
machine translation models, covering five language combinations, namely, from English to German, Spanish, Italian, Russian and Chinese. In their work, the authors showed that open neural models still exhibit strong semantic biases towards frequent senses, confirming once again the suspicions about this under-explored issue.
Despite all the effort made in putting forward challenging sets of data to test WSD capabilities of NMT models, to the best of our knowledge, only a few approaches (Rios Gonzales et al., 2017; Liu et al., 2018) have been proposed to mitigate this issue, and none of these is effective with modern Transformer-based architectures. Furthermore, while parallel corpora have been exploited to produce sense annotations in the past (Bonansinga and Bond, 2016; Delli Bovi et al., 2017), they were built by utilizing outdated disambiguation approaches that have recently been surpassed by more advanced neural architectures. Indeed, the Word Sense Disambiguation field has received much attention in the last few years, with several supervised approaches (Conia and Navigli, 2021; Blevins and Zettlemoyer, 2020; Barba et al., 2021) and sense embedding models (Loureiro and Jorge, 2019; Scarlini et al., 2020a,b; Wang et al., 2020) performing close to the upper bound limit of the inter-annotator agreement, which finally makes them feasible for inclusion in other downstream tasks, e.g., Machine Translation.
Thus, differently from previous studies in the literature, we focus on closing the gap between these two fields, i.e., Neural Machine Translation and Word Sense Disambiguation, by putting the recent advances in WSD at the service of NMT models. We propose a novel approach, similar to that introduced in Luan et al. (2020), for creating high-quality sense-annotated parallel corpora, and we use this semantic information to regularize an NMT model, making it less biased and capable of producing higher-quality translations.
3 Reducing the Disambiguation Bias in NMT
Neural Machine Translation models are typically trained end-to-end to produce a target translation given a source sentence and, thus, they can only rely on the input context to resolve the ambiguity of polysemous words therein. Being pattern recognition algorithms at heart, these models fall prey to the inherent bias carried by the frequency of co
occurrence of words within parallel sentences, and thus tend to disambiguate words to the sense they most frequently encountered during training, even when the sentence does provide enough context to identify the correct sense. At the same time, Word Sense Disambiguation models, i.e., models specialized in associating a word in context with one of the meanings within a given sense inventory, have recently displayed remarkable results across different benchmarks and languages (Bevilacqua et al., 2021). The time may now therefore be ripe for them to be successfully included into downstream applications such as Neural Machine Translation. However, data that would allow these two worlds to be brought together, i.e., parallel corpora where words are associated with semantic labels, are currently still produced automatically by leveraging outdated approaches to WSD (Delli Bovi et al., 2017).
In what follows, we first provide some preliminary information about resources and tools that we employ in our method (§ 3.1); then, we introduce a new approach for automatically annotating tokens within parallel sentences with sense annotations, i.e., labels explicitly defining their meanings (§ 3.2); finally, we propose a fine-tuning objective for leveraging such annotations in order to mitigate the sense bias while also improving the translation quality overall (§ 3.3). The intuition behind our work is that fixed sense labels describing word senses would help NMT models better encode the underlying meaning of the input sentence, thus generating less biased and overall better translations.
We draw sense labels from BabelNet (Navigli and Ponzetto, 2012), a multilingual knowledge base created by merging several semantic resources in different languages such as WordNet (Miller et al., 1990), Wikipedia, Wikidata, etc. BabelNet is structured in synsets, i.e., sets of synonymous senses in different languages. For instance, the synset of plant<sup>organism</sup> contains the following lexicalizations: plant<sub>EN</sub>, pianta<sub>IT</sub>, Pflanze<sub>DE</sub>, among others. Additionally, BabelNet provides lemma-to-synsets mappings. For example, the English noun plant belongs to the following nominal synsets: organism, industrial plant, actor in the audience and something placed secretly.<sup>2</sup> Since BabelNet con
$^{2}$ Synsets bn:00035324n, bn:00046568n, bn:00062800n and bn:00062801n respectively, from https://babelnet.org.
tains millions of synsets, which may make the computation too expensive, we restrict the vocabulary to just those containing at least one English sense from WordNet, as is also done in several other works (Barba et al., 2020; Scarlini et al., 2020b; Bevilacqua and Navigli, 2020).
3.2 Building a Sense-Annotated Parallel Corpus
Let us assume that our running example sentence "The energy comes from a distant plant." appears within a parallel corpus paired with the following Italian translation: "L'energia viene da un impianto lontano". As we said, by considering the English sentence alone, the word plant could take several meanings, among which organism and power plant. However, among these, only one is shared with its translation impianto, i.e., the power plant meaning. Therefore, considering the cross-lingual alignment of words may drastically reduce the set of valid meanings, making the disambiguation task much easier. Based on this intuition, given a parallel corpus, we perform the following two steps:
1. Sense Scoring, where we employ a WSD system to assign to each content word a distribution over its possible meanings;
2. Annotation Refinement, where we compute cross-lingual word alignments to reduce lexical ambiguity and finally assign the most suitable sense to each content word.
Sense Scoring In this step, our goal is to assign to every content word within a sentence a distribution over its possible senses in BabelNet. To this end, given as input a sentence  $s^3$  from a parallel corpus  $C$ , we first apply Part-of-Speech tagging and lemmatization to it, then pass it through our WSD system, which returns a distribution over its possible meanings.
Formally, let  $w_{i}$  be a content word in a sentence  $s = [w_{1},\ldots ,w_{n}]$ , and  $\sigma (w_{i})$  the set of synsets associated with  $w_{i}$  in BabelNet. The WSD system assigns a score  $c(S|w_i,s)$  to each synset  $S\in \sigma (w_i)$ ; we denote the synset of  $w_{i}$  with the highest confidence as  $S_{w_i}^*$ . As a result, each content word in a source or target sentence is associated with a sense distribution. However, applying a WSD system alone may not be sufficient to ensure high-quality
annotations, as the application domain may be different from the one of its training set. Therefore, in the next step we take advantage of the translation each sentence is paired with to refine sense annotations.
Annotation Refinement We produce word-level cross-lingual alignments between the source and the target sentences of the parallel corpus: given a pair of parallel sentences  $(s, t)$ , we compute a list of alignments  $\mathcal{A} = \{(w_i^s, w_j^t) | w_i^s \in s, w_j^t \in t\}$ . Thus, given an aligned word pair  $P = (w_i^s, w_j^t) \in \mathcal{A}$ , let  $\sigma(P) = \sigma(w_i^s) \cap \sigma(w_j^t)$ , i.e., the intersection of synsets that the two words may denote according to BabelNet: we discard annotations for any word pair such that  $\sigma(P) = \emptyset \vee |\sigma(w_i^s)| < 2$ . In other words, we retain all the aligned pairs  $(w_i^s, w_j^t)$  such that the source word is polysemous and the intersection of their senses is non-empty, thus ensuring higher annotation precision by leveraging the parallelism of words.
Finally, we assign the same synset  $S^*$  to both words  $(w_i^s, w_j^t)$  in  $P$  as follows:
$$
\begin{array}{l} S ^ {*} = S _ {w _ {i} ^ {s}} ^ {*} = S _ {w _ {j} ^ {t}} ^ {*} \\ = \operatorname * {a r g m a x} _ {S \in \sigma (P)} \left(\frac {c (S | w _ {i} ^ {s} , s)}{Z _ {s}} + \frac {c (S | w _ {j} ^ {t} , t)}{Z _ {t}}\right) \\ \end{array}
$$
$$
Z _ {s} = \sum_ {S \in \sigma (P)} c \left(S \mid w _ {i} ^ {s}, s\right)
$$
$$
Z _ {t} = \sum_ {S \in \sigma (P)} c (S | w _ {j} ^ {t}, t)
$$
that is, the synset with the highest combined confidence score after normalizing over  $\sigma(P)$ , where  $Z_{s}$  and  $Z_{t}$  represent the normalization factors of the probability distributions associated with the synsets of  $w_{i}^{s}$  and  $w_{j}^{t}$ , respectively.
3.3 Semantic Injection
Now that we can generate high-quality sense annotations, we describe our fine-tuning method to inject word-level semantics into a Neural Machine Translation model. Ideally, we want the model to benefit from such annotations during training, while not being dependent on them at inference time. To satisfy both these desiderata, we adapt the model's vocabulary to handle synsets as well as subwords, and propose a specific loss that exploits the injected senses to improve the base model's handling of ambiguous words.
$s$  can be either a source or a target sentence.
Figure 1: Semantic Consistency Regularization (SCR) fine-tuning. KL stands for Kullback-Leibler divergence; CE stands for Cross Entropy.
Semantically Enhancing Sentences In order to work with concepts, we need a way to represent them. Let us consider once more the sentence "The energy comes from a distant plant": we rewrite it in order to also include the exact meaning for plant, which we computed as described in § 3.2: "The energy comes from a distant plant  $plant^{\text{factory}}$ ".
Formally, given a source sentence  $s$  and a word  $w_{i}$  annotated with sense  $S_{w_{i}}^{*}$ , we simply represent  $w_{i}$  as its standard segmentation followed by  $S_{w_{i}}^{*}$ , represented by its sense embedding passed through a linear projection layer (as shown in Figure 2). Additionally, to enforce the connection between the tagged word and its sense annotation, we set the position ids for the word and the sense embedding to the same value, as if they were a single token. This encoding scheme gracefully extends to the whole sentence, yielding the sense-enhanced input representation for a given sentence  $s$ .
Semantic Consistency Regularization We hereby propose the Semantic Consistency Regularization (SCR) objective, inspired by MVR (Wang et al., 2021).
Formally, let  $x'$  and  $x''$  be two encodings (plain and sense-enhanced) of the same input sentence  $x$  and let  $y$  be the target sentence, we define SCR as:
$$
\begin{array}{l} S C R (\theta) = - \log \mathcal {P} _ {\theta} (y | x ^ {\prime}) - \log \mathcal {P} _ {\theta} (y | x ^ {\prime \prime}) \\ + \mathcal {D} _ {\mathrm {K L}} \left(\mathcal {P} _ {\theta} \left(y \mid x ^ {\prime}\right) | | \mathcal {P} _ {\theta} \left(y \mid x ^ {\prime \prime}\right)\right) \\ \end{array}
$$
where  $\theta$  is the set of trainable weights,  $\mathcal{D}_{\mathrm{KL}}$  is the unidirectional Kullback-Leibler divergence (Kullback and Leibler, 1951) and  $\mathcal{P}_{\theta}(y|x)$  represents an output distribution (a visual representation of SCR is reported in Figure 1).
With this formulation, SCR jointly uses the same sentence with and without sense annotations as two separate inputs: while we train the model to be
Figure 2: Sense injection mechanism. Subscripts represent the position id associated with the subword.
able to translate both plain and sense-enhanced sentences, by minimizing the divergence between the output distributions we also force the model to transfer the sense information from the sense-enhanced input to the plain input, much like in a self-distillation process. At the same time, we still maintain the model's capability of translating without sense annotations, thus dropping their requirement at inference time.
4.1 Our Model
We employ as underlying model the standard Transformer architecture (Vaswani et al., 2017), with 6 encoder and 6 decoder layers. Note that, while SCR can be applied to any pre-trained model, we retrain one from scratch because most of the other models available online use part of our test data as
<sup>5</sup>We use randomly-initialized MarianMT models available in HuggingFace's transformers library (Wolf et al., 2020) for easier comparability with their trained versions.
4Details are provided in  $\S 4$
their training data (see § 4.2). Additional details about training configuration and hyperparameters are provided in § A.3.
Fine-tuning with SCR Additionally, to jumpstart the model's capabilities, we encode synsets not as randomly initialized learnable vectors (e.g., by extending the vocabulary), but with frozen pre-trained sense embeddings projected into the model's input space by means of a linear layer, the only additional learnable component of the model (Projection in Figure 2), which is dropped after the fine-tuning stage. As pre-trained sense embeddings we use ARES (Scarlini et al., 2020b), since they provide multilingual representations for each synset in our vocabulary. We study the impact of this choice in  $\S 5.5$ . To perform token-level alignments, we use MultiMirror (Procopio et al., 2021).
We experiment on three distinct language pairs: EN→DE, EN→ES and EN→FR. Following (Emelin et al., 2020), we gather the data from WMT14 for German and French and WMT13 for Spanish, considering only sentences coming from either CommonCrawl, News Commentary or Europarl, to maintain similar order of magnitudes among language pairs (and to contain pre-processing and training times). As validation sets, we employ newstest2014 for EN→DE, newstest2013 for EN→FR and newstest2012 for EN→ES. All datasets employed in this work are freely available for research purposes.
Sense-Enhanced Datasets
We process each parallel sentence of the considered corpora with the procedure described in § 3.2, taking into account only content words whose Part-of-Speech tag is noun, as the challenge sets we evaluate upon only target nominal words.<sup>7</sup>
For POS-tagging and lemmatization we use Stanza (Qi et al., 2020). As disambiguation system, we use EWISER (Bevilacqua and Navigli, 2020), a neural WSD model based on BERT (Devlin et al., 2019), which has attained state-of-the-art performances on English as well as other languages. EWISER has been trained on SemCor (Miller et al., 1993) – the standard training set for WSD – and the WordNet Gloss corpus (Langone et al., 2004)
- a semi-automatically annotated dataset featuring sense definitions. Detailed statistics of the base and parallel corpora produced are provided in § A.5.
Translation Test Set
We evaluate standard translation quality through the newstest datasets available in the specific WMT year (i.e., WMTXX corresponds to newstest20XX). The standard evaluation is carried out by means of SacreBLEU (Post, 2018), with signature BLEU+case.mixed+numrefs.1
+smooth.exp+tok.13a+version.1.5.1.
Disambiguation Bias Challenge Sets
To measure the disambiguation bias of each model we employ the challenge sets introduced by Emelin et al. (2020), composed of sentences reserved from the WMT14 English  $\rightarrow$  German corpus. These challenge sets are based on sense clusters built by automatically merging together BabelNet synsets, which then are manually refined to ensure their correctness. Each sense cluster contains an English polysemous word and a set of German monose-mous terms, which uniquely identify a certain meaning.