SlowGuess/NAACL_2021_OCR huggingface dataset:
Edoardo Barba
Tommaso Pasini $^{2,*}$
<sup>1</sup>Sapienza NLP Group, Department of Computer Science, Sapienza University of Rome
$^{2}$ Department of Computer Science, University of Copenhagen
{barba,navigli}@di.uniroma1.it
tommaso.pasini@di.ku.dk
Word Sense Disambiguation (WSD) is a historical NLP task aimed at linking words in contexts to discrete sense inventories and it is usually cast as a multi-label classification task. Recently, several neural approaches have employed sense definitions to better represent word meanings. Yet, these approaches do not observe the input sentence and the sense definition candidates all at once, thus potentially reducing the model performance and generalization power. We cope with this issue by reframing WSD as a span extraction problem — which we called Extractive Sense Comprehension (ESC) — and propose EsCHER, a transformer-based neural architecture for this new formulation. By means of an extensive array of experiments, we show that ESC unleashes the full potential of our model, leading it to outdo all of its competitors and to set a new state of the art on the English WSD task. In the few-shot scenario, EsCHER proves to exploit training data efficiently, attaining the same performance as its closest competitor while relying on almost three times fewer annotations. Furthermore, EsCHER can nimbly combine data annotated with senses from different lexical resources, achieving performances that were previously out of everyone's reach. The model along with data is available at https://github.com/SapienzaNLP/esc.
Being able to link a piece of raw text to a knowledge base is fundamental in NLP (Navigli, 2009; McCoy et al., 2019; Bender and Koller, 2020), as it can aid neural models to ground their representations on structured resources and enable Natural Language Understanding (Navigli, 2018). A task that is key to achieving this goal is Word Sense Disambiguation (WSD), where, given a sentence
with a target word, a model has to predict its most suitable meaning from a predefined set of labels, i.e., its senses. WSD has not only considerably improved its performance with the advent of deep learning (by around 15 F1 points in 15 years), but it has also shown its benefits in downstream applications such as Neural Machine Translation (Liu et al., 2018; Pu et al., 2018) and Information Extraction (Moro and Navigli, 2013; Delli Bovi et al., 2015), while also being leveraged to enrich the contextual representations of neural models (Peters et al., 2019; Zhang et al., 2019). However, WSD has mostly been framed as a multi-label classification task (Raganato et al., 2017b; Hadiwinoto et al., 2019) over a very large vocabulary of discrete senses. This formulation may limit a model's capabilities to properly represent word meanings, as each sense is only defined by means of its occurrences in a training set, while its inherent meaning remains linguistically unexpressed. Furthermore, rare or unseen senses are either poorly modeled or cannot be modeled at all. These problems have recently been mitigated by integrating sense definitions (glosses) within neural architectures (Kumar et al., 2019; Huang et al., 2019; Blevins and Zettlemoyer, 2020). Yet, despite their large improvements, none of these models attends all the possible definitions of a target word at once, and therefore each lacks the ability to represent both the input context and the candidate definitions together.
Inspired by the Extractive Reading Comprehension framework (Rajpurkar et al., 2016) in the field of Question Answering (QA), we cope with these issues and reframe the WSD problem as a novel text extraction task, which we have called Extractive Sense Comprehension (ESC). In this setting, a model receives as input a sentence with a target word and all its possible sense definitions. Then, we request the model to extract the text span associated with the gloss expressing the target word's most suitable meaning. Within this frame
*Work carried out while at the Sapienza University of Rome.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4661-4672 June 6-11, 2021. ©2021 Association for Computational Linguistics
work, we also propose a transformer-based architecture (ESCHER) that implements the ESC task by attending to the input context and target word definitions jointly. Through an extensive experimental setting, we show that EsCHER surpasses former state-of-the-art approaches by a large margin while, at the same time, requiring almost 3 times less training data points to attain performances comparable to its strongest competitor in a few-shot setting. Furthermore, thanks to our new formulation, the proposed model can effectively carry out predictions across different sense repositories and combine distinct inventories with unmatched nimbleness, attaining even higher results than when limited to a single resource only.
To summarize, this paper brings the following novel contributions:
1. The Extractive Sense Comprehension task (ESC), i.e., a reframing of the Word Sense Disambiguation problem.
2. EsCHER: a transformer-based architecture for ESC, outperforming all the other modern architectures on the WSD task.
3. An extensive study of the proposed model in different training regimes, i.e., in 0-shot, few-shot and fully-supervised settings.
4. A study on combining data annotated with distinct lexicographic resources.
Besides its performance advantages, ESC also comes with other benefits: it does not require a large output vocabulary, and it eases the joint use of corpora annotated with different inventories.
Word Sense Disambiguation (WSD) is one of the long-standing problems in lexical semantics, introduced for the first time in the context of Machine Translation by Weaver (1949). WSD aims at linking a word in context to its most suitable meaning in a predefined sense inventory, which is usually a dictionary where each entry defines a concept via a definition (gloss) and a set of examples. Most approaches to WSD rely on WordNet (Miller et al., 1990) as the underlying inventory of senses for the English language, and SemCor (Miller et al., 1993) as training corpus. WordNet organizes lexical-semantic information by means of a graph where sets of synonyms are grouped into
synsets (concepts) and edges are typed semantic relations.
While early neural models used WordNet as a mere repository of senses (Raganato et al., 2017b; Hadiwinoto et al., 2019), more recent approaches have started to exploit sense definitions (Kumar et al., 2019; Blevins and Zettlemoyer, 2020) and relational information (Bevilacqua and Navigli, 2020; Conia and Navigli, 2021). Sense definitions, in particular, have been shown to be effective for modeling word senses (Luo et al., 2018; Kumar et al., 2019), as they provide information orthogonal to that available in the training data. This has been further investigated under different perspectives by Huang et al. (2019, GlossBERT), Blevins and Zettlemoyer (2020, BEM) and Bevilacqua et al. (2020, Generationary). GlossBERT casts the WSD problem as a binary classification task where, given a word in context and one of its dictionary definitions, it determines whether this definition matches the word meaning expressed in the context. BEM employs a bi-encoder to represent the target word and its sense definitions within the same space. Generationary, instead, has predefined sense inventories at its disposal and directly generates a definition given a word in its context. The strength of these approaches lies in the fact that glosses allow senses that are under-represented within the training corpus to be modeled, hence mitigating the long-standing paucity of sense-annotated data (Pasini, 2020). Nevertheless, none of the above approaches can exploit all definitions at once: indeed, glosses are either provided one at a time (GlossBERT), modeled with one vector only and independently from each other (BEM), or used individually as target text to be generated (Generationary).
Our new formulation (ESC) for the WSD problem stands out from previous approaches inasmuch as it is the first to access the input context and all the target word's definitions together, while, at the same time, dropping the requirement of a predefined sense inventory. Indeed, differently from its competitors, our proposed approach (ESCHER) can scale effectively across different lexical resources even when they were not available at the time of training.
In what follows, we first formalize the Extractive Sense Comprehension task (Section 3.1), then in
Figure 1: Depiction of EsCHER. The model takes as input a sentence concatenated with all the target word's definitions and outputs two indices indicating the start and end token in the input text of the target word definition.
troduce EsCHER, a transformer-based architecture for ESC (Section 3.2), and finally put forward a novel approach for mitigating the bias towards the most frequent meanings (Kilgarriff, 2004) within training data (Section 3.3).
3.1 Extractive Sense Comprehension
To unleash the full potential of attention-based models on the Word Sense Disambiguation task, we reframe WSD as a span-extraction problem. Formally, given a sense inventory  $S$ , we first define the definitional context  $\mathcal{D}_{\hat{w}}$  for the target word  $\hat{w}$  as the concatenation of all the possible definitions  $d_{1}, \ldots, d_{k}$  in  $S$  for  $\hat{w}$ , i.e.,  $\mathcal{D}_{\hat{w}} = w_{1}^{d_{1}} \ldots w_{|d_{1}|}^{d_{1}} \ldots w_{1}^{d_{k}} \ldots w_{|d_{k}|}^{d_{k}}$ , where  $w_{i}^{d_{z}}$  is the  $i$ -th word of the gloss  $d_{z}$  ( $1 \leq z \leq k$ ). Then, we reformulate the task as follows: given a target word  $\hat{w}$ , a context  $c$  in which  $\hat{w}$  occurs and the definitional context  $\mathcal{D}_{\hat{w}}$ , a model has to find the interval  $[i^{*}, j^{*}]$  in  $\mathcal{D}_{\hat{w}}$  which identifies the correct definition  $d^{*} \in \mathcal{D}_{\hat{w}}$  of  $\hat{w}$  in  $c$ . This formulation, on the one hand, aids to better characterize word meanings, thanks to the inclusion of all the target word definitions as additional input. On the other hand, it also relieves the burden of a large output vocabulary – typically in the order of tens of thousands of meanings – which makes the classification cumbersome.
3.2 EsCHER
We now introduce a transformer-based model for the ESC task (Figure 1). It takes as input a context
$c$  with a target word  $\hat{w}^1$  concatenated with  $\mathcal{D}_{\hat{w}}$ . The target word  $\hat{w}$  is surrounded by the tags  $<t>$  and  $</t>$  and each definition in  $\mathcal{D}_{\hat{w}}$  has the first letter capitalized and a period at the end. We separate the context  $c$  and the definitional context  $\mathcal{D}_{\hat{w}}$  with the special symbol  $
$  and surround the whole text with the tags  $
$  and  $
$ .
Formally, given the input:
$$
\begin{array}{l} m = <   s > w _ {1} \dots <   t > \hat {w} <   / t > \dots w _ {n} <   / s > \\ w _ {1} ^ {d _ {1}} \dots w _ {| d _ {1} |} ^ {d _ {1}} \dots w _ {1} ^ {d _ {k}} \dots w _ {| d _ {k} |} ^ {d _ {k}} <   / s > \\ \end{array}
$$
of length  $l$ , the model computes the span  $(i,j)$  containing the predicted gloss for the target word  $\hat{w}$  as follows:
$$
H = \text {t r a n s f o r m e r} (m)
$$
$$
Z = W ^ {T} H + b
$$
$$
Z ^ {s} = \left[ \begin{array}{c c c} Z _ {1 1} & \ldots & Z _ {1 l} \end{array} \right]
$$
$$
Z ^ {e} = \left[ \begin{array}{l l l} Z _ {2 1} & \ldots & Z _ {2 l} \end{array} \right]
$$
where transformer can be any transformer-based architecture,  $H \in \mathbb{R}^{f \times l}$  is the matrix of hidden states, and  $W \in \mathbb{R}^{f \times 2}$  and  $b \in \mathbb{R}^2$  are trainable parameters.  $Z^s$  and  $Z^e$  are two variables containing the logits for each word  $w_u$  indicating, respectively, whether it is the start or the end of the correct definition for target word  $\hat{w}$ .
<sup>1</sup>For the sake of simplicity, in the following we use word to refer to subwords, words and multiwords.
2The  $< s >$  and  $< / s >$  tags can be any special token in a model vocabulary that have been used to divide texts at pretraining time.
${}^{3}f$  indicates the number of dimensions of each hidden state.
Finally, we train the model by averaging two distinct cross-entropy losses that we compute for the start and end indices:
$$
\mathcal {L} _ {s} = - Z _ {i ^ {*}} ^ {s} + l o g \sum_ {v = 1} ^ {l} e x p (Z _ {v} ^ {s})
$$
$$
\mathcal {L} _ {e} = - Z _ {j ^ {*}} ^ {e} + \log \sum_ {v = 1} ^ {l} e x p (Z _ {v} ^ {e})
$$
where  $Z_{i^*}^s$  and  $Z_{j^*}^e$  are the scores associated with the correct start and end indices.
At prediction time, rather than allowing the system to output a span that does not correspond precisely to any definition in  $\mathcal{D}_{\hat{w}}$ , the model outputs a pair  $(i,j)$  such that a definition  $d_{k}\in \mathcal{D}_{\hat{w}}$  starts in  $i$  and ends in  $j$  and its probability is the maximum across all the other gloss spans in  $\mathcal{D}_{\hat{w}}$ . Formally, the model selects its output as follows:
$$
\text {o u t p u t} = \underset {(i, j)} {\arg \max } P (w _ {i}, w _ {j})
$$
$$
P \left(w _ {i}, w _ {j}\right) = P \left(w _ {i} = \text {s t a r t} \mid Z ^ {s}\right) \times
$$
$$
P (w _ {j} = e n d \mid Z ^ {e})
$$
$$
P (w _ {u} = s t a r t \mid Z ^ {s}) = \frac {\exp (Z _ {u} ^ {s})}{\sum_ {v = 1} ^ {l} \exp (Z _ {v} ^ {s})}
$$
$$
P (w _ {u} = e n d \mid Z ^ {e}) = \frac {e x p (Z _ {u} ^ {e})}{\sum_ {v = 1} ^ {l} e x p (Z _ {v} ^ {e})}
$$
where  $P(w_{u} = \text{start} \mid Z^{s})$  and  $P(w_{u} = \text{end} \mid Z^{e})$  indicate the probability that  $w_{u}$  is the start or the end of any of the  $k$  definitions, respectively.
3.3 Rebalancing the Most Frequent Sense Bias
While our approach already allows all the possible definitions of a word to be contextualized by jointly encoding them together with the context sentence, it may still suffer from the high unbalance in sense distribution (Kilgarriff, 2004) and be biased towards the most frequent definition regardless of its contextualization. Our framework allows this issue to be dealt with in an elegant way, which we have called Gloss Noise (GN). GN counterbalances this bias by lowering the prior probability of the most frequent glosses. That is, inspired by the negative sampling technique (Mikolov et al., 2013), GN adds, to each training example,  $k$  frequent definitions that are not related to the target word. We sample the  $k$  glosses from the following multinomial distribution:
$$
p (d _ {i}) = \frac {f _ {d _ {i}}}{\sum_ {j = 1} ^ {| D |} f _ {d _ {j}}}
$$
where  $D$  is the set of all possible definitions in the training set and  $f_{d_i}$  is the frequency of the  $i$ -th definition in a sense-tagged corpus. The value of  $k$ , instead, is sampled from a Poisson distribution with  $\lambda = 1$ , so that the expected number of added definitions is equal to 1. This allows the discrepancy between the training and prediction phases to be kept as small as possible, while also introducing negative signals for frequent senses. Indeed, Gloss Noise ensures that the expected number of times a definition is added as a negative example is equal to the number of times it is seen as a correct one, thereby counterbalancing the high rate at which frequent definitions are seen only as positive examples without overly affecting rare senses.
4 Standard WSD Evaluation
In this Section we introduce the experimental setting we use to evaluate the proposed framework and neural architecture.
Data We use the evaluation suite made available by Raganato et al. (2017a) for the English Word Sense Disambiguation task. It includes SemCor (Miller et al., 1993) for training, i.e., a corpus containing 33,362 sentences and 226,036 instances annotated manually with senses from WordNet 3.0. As common practice, we use SemEval-2007 (SE07; Pradhan et al., 2007) as development set. For testing, we consider all the remaining datasets in the suite, i.e., Senseval-2 (SE2; Edmonds and Cotton, 2001), Senseval-3 (SE3; Snyder and Palmer, 2004), SemEval-2013 (SE13; Navigli et al., 2013), SemEval-2015 (SE15; Moro and Navigli, 2015) and their concatenation (ALL).
In order to measure the extent to which systems generalize to rare and unseen words and definitions (zero-shot settings), we also consider five other test sets that we created from the ALL dataset:
i) MFS, which contains test instances tagged with the most frequent sense for the target word in the training set;
ii) LFS, which contains test instances that are tagged with a sense that is not the most frequent for the target word and that was seen at least once during training;
<sup>4</sup>We note that the evaluation suite includes the dev set, i.e., SemEval-2007, within the ALL dataset, and so do we.
iii) 0-lex, which contains test instances whose lexeme<sup>5</sup> was never seen as a target word during training;
iv) 0-lex-def, which contains test instances with a definition that was never seen associated with the target lexeme during training;
v) 0-def, which contains test instances whose definition has never been seen during training. We note that 0-def differs from 0-lex-def as a definition is tied in WordNet to a synset, i.e., a set of synonymous senses, rather than to a sense; therefore the same definition may be seen associated with different lexemes.
Comparison Systems As baselines, we consider the Most Frequent Sense computed on the training set (MFS SemCor) and two neural models featuring  $\mathrm{BERT}_{large}$  and  $\mathrm{BART}_{large}$  as text encoders, with a linear classifier over the whole sense vocabulary on top. As for the  $\mathrm{BERT}_{large}$  baseline, we follow Blevins and Zettlemoyer (2020) and keep  $\mathrm{BERT}_{large}$  weights fixed, while for  $\mathrm{BART}_{large}$  we finetune the whole model.
As competitors, we consider the following models: GLU (Hadiwinoto et al., 2019), which keeps BERT weights frozen and trains a gated linear unit on top of it;  $\mathrm{SVC}^7$  (Vial et al., 2019), which uses a vocabulary compression technique; EWISE (Kumar et al., 2019); GlossBERT (Huang et al., 2019);  $\mathrm{BEM}^8$  (Blevins and Zettlemoyer, 2020) and EWISER (Bevilacqua and Navigli, 2020), which take advantage of external knowledge such as glosses and semantic relations. We note that EWISER uses a different development set, hence its results are not fully comparable with the others. Finally, we also consider two nearest-neighbour approaches based on synset embedding and vector similarity, i.e., LMMS (Loureiro and Jorge, 2019) and ARES (Scarlini et al., 2020).
ESCHER Setting We use BARTlarge (Lewis et al., 2020; Wolf et al., 2020) as transformer architecture owing to the fact that it is among the strongest models on reading comprehension tasks
such as SQuAD (Rajpurkar et al., 2016) and it allows us to feed sequences up to 1024 subtokens long. $^{10}$  We use the output of its last decoder layer to represent the input tokens and compute the start and end token distributions. We note that EsCHER is directly comparable to the BARTlarge baseline in terms of model complexity as both use the same transformer model with one linear layer on top.
We finetune the whole EsCHER architecture with the Rectified Adam (Liu et al., 2020) optimizer with learning rate set to  $1 \cdot e^{-5}$  for up to 300,000 steps, 20 steps of gradient accumulation and batches made of 700 tokens. In what follows, we report the results for our model with and without Gloss Noise (Section 3.3), denoting them as EsCHER and  $\mathrm{ESCHER_{No - GN}}$ , respectively.
Framework Benchmark In Table 1 we report the F1 scores of EsCHER, ESCHERNo-GN and all the other systems. By comparing BARTlarge and EsCHER, we can measure the effectiveness of our proposed framework, ESC, on the performance of a transformer-based architecture. Indeed, the two architectures are nearly identical except for the last layer, where, for each token, BARTlarge makes a prediction across the whole sense vocabulary, while EsCHER performs a binary classification. Thus, the large difference between the two models (8.5 F1 points) suggests that the Extractive Sense Comprehension formulation of WSD allows the potential of transformer-based architectures to be fully exploited, and, therefore, attain better performance.
When Gloss Noise is enabled (ESCHER row), our model gains 1 F1 point in comparison to when it is disabled  $(\mathrm{ESCHER}_{\mathrm{No - GN}})$ . This highlights that directly mitigating the bias towards the Most Frequent Senses during training is fundamental to making our approach as effective as possible.
Finally, thanks to our new formulation of the WSD problem, a simple model such as EsCHER outperforms all the other approaches by a large margin on the ALL dataset, beating the previous state of the art by 1.7 points (BEM). This corroborates our hunch that the Extractive Sense Comprehension task is an extremely effective formulation of WSD for transformer-based architectures.
10 As for the  $< t >$  and  $< / t >$  symbols, we used the words  $< classify >$  and  $< / classify >$ .
11 Please refer to Appendix B for further details on the training.
${}^{5}\mathrm{\;A}$  (lemma,part of speech) pair.
<sup>6</sup>We identify a sense as a pair (lexeme, definition).
${}^{7}$  Similarly to Blevins and Zettlemoyer (2020),we report the best results of the SVC single model trained on SemCor only.
BEM is the state-of-the-art model in this setting at the time of writing.
9Please see Appendix A for experiments with different transformer pretrained models.
Model	Dev Set	Test Sets	Concatenation of all Datasets
SE07	SE2	SE3	SE13	SE15	Nouns	Verbs	Adj.	Adv.	ALL
Baselines	MFS SemCor	54.5	65.6	66.0	63.8	67.1	67.7	49.8	73.1	80.5	65.5
BERTbase	68.6	75.9	74.4	70.6	75.2	75.7	63.7	78.0	85.8	73.7
BARTlarge	63.5	75.0	72.2	69.3	74.2	74.0	61.6	76.9	86.1	72.2
Prior work	EWISE‡	67.3	73.8	71.1	69.4	74.5	74.0	60.2	78.0	82.1	71.8
GLU	68.1	75.5	73.6	71.1	76.2	—	—	—	—	74.1
LMMS††	68.1	76.3	75.6	75.1	77.0	—	—	—	—	75.4
SVC	—	—	—	—	—	—	—	—	—	75.6
GlossBERT†	72.5	77.7	75.2	76.1	80.4	79.8	67.1	79.6	87.4	77.0
ARES††	71.0	78.0	77.1	78.7	75.0	80.6	68.3	80.5	83.5	77.9
EWISER‡	71.0	78.9	78.4	78.9	79.3	81.7	66.3	81.2	85.8	78.3
BEM†	74.5	79.4	77.4	79.7	81.7	81.4	68.5	83.0	87.9	79.0
Ours	ESCHERNo-GN†	75.0	80.5	76.9	81.1	83.0	83.0	68.5	81.9	86.1	79.7
ESCHER†	76.3	81.7	77.8	82.2	83.2	83.9	69.3	83.8	86.7	80.7
Results on Rare and Unseen Senses In Table 2 we report the results of the three best-performing models, i.e., EsCHER,  $\mathrm{ESCHER_{No - GN}}$  and BEM, on five datasets, measuring how well models perform when dealing with rare words and meanings in different situations (cf. Section 4.1).  $\mathrm{ESCHER_{No - GN}}$  manages to outperform BEM on most datasets, hence already demonstrating that our new framing allows transformers to better generalize on rare words and senses. When enabling Gloss Noise, EsCHER achieves even higher performance on all datasets, falling behind BEM only on the MFS dataset. Interestingly enough, the comparison with BEM on the 0-lex-def and 0-def datasets shows that EsCHER can easily predict definitions that were either seen associated only with lexemes different from the input ones or not seen at all, while, in direct contrast, BEM performs poorly in both scenarios. A similar pattern is observed for the Least Frequent Senses (LFS) dataset, where EsCHER outperforms BEM by 3.6 F1 points at the cost of only 1 point less in predicting the most frequent meanings.
5 Merging Multiple Knowledge Bases
Being able to combine datasets tagged with different inventories is a desirable ability for a model. Indeed, being able to use different datasets grants access to a larger number of examples, while, at the same time, removing the necessity of having