SlowGuess/ACL_2020_OCR huggingface dataset:
Terra Blevins and Luke Zettlemoyer
Paul G. Allen School of Computer Science & Engineering, University of Washington
Facebook AI Research, Seattle
{blvns, lsz}@cs.washington.edu
A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed, causing existing models to generally perform poorly on senses that are either rare or unseen during training. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense. The encoders are jointly optimized in the same representation space, so that sense disambiguation can be performed by finding the nearest sense embedding for each target word embedding. Our system outperforms previous state-of-the-art models on English all-words WSD; these gains predominantly come from improved performance on rare senses, leading to a  $31.1\%$  error reduction on less frequent senses over prior work. This demonstrates that rare senses can be more effectively disambiguated by modeling their definitions.
One of the major challenges of Word Sense Disambiguation (WSD) is overcoming the data sparsity that stems from the Zipfian distribution of senses in natural language (Kilgarriff, 2004). For example, in SemCor (the largest manually annotated dataset for WSD)  $90\%$  of mentions of the word plant correspond to the top two senses of the word, and only half of the ten senses of plant occur in the dataset at all (Miller et al., 1993). Due to this data imbalance, many WSD systems show a strong bias towards predicting the most frequent sense (MFS) of a word regardless of the surrounding context (Postma et al., 2016).
A successful WSD system should be able to overcome this bias and correctly disambiguate cases where a word takes a less frequent sense (LFS), without sacrificing performance on MFS examples.
Previous work has found that incorporating lexical information such as sense definitions, or glosses, into WSD systems improves performance (Luo et al., 2018a,b). Glosses have also been found to improve LFS performance; however, absolute performance on rare senses is still low, with models showing a 62.3 F1 performance drop between the MFS examples and the LFS ones (Kumar et al., 2019).
In this paper, we show that this gap can be significantly reduced by jointly fine-tuning multiple pretrained encoders on WSD. We present a bi-encoder model built on top of BERT (Devlin et al., 2019) that is designed to improve performance on rare and zero-shot senses. Similar to prior work, our system represents the target words and senses in the same embedding space by using a context encoder to represent the target word and surrounding context, and a gloss encoder to represent the sense definitions. However, our two encoders are jointly learned from the WSD objective alone and trained in an end-to-end fashion.
This approach allows our model to outperform prior work on the English all-words WSD task introduced in Raganato et al. (2017b). Analysis of our model shows that these gains come almost entirely from better performance on the less frequent senses, with an 15.6 absolute improvement in F1 performance over the closest performing system; our model also improves on prior work in the zero-shot setting, where we evaluate performance on words and senses not seen during training.
Finally, we train our model in a few-shot setting in order to investigate how well the bi-encoder system learns on a limited set of training examples per sense. The bi-encoder architecture is able to generalize better from the limited number of exam
<sup>1</sup>For example, in the sentence "She planted the tree," the gloss, or meaning, for the sense of plant is "put or set [something] firmly into the ground." (Miller, 1995)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006-1017 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ples than a strong pretrained baseline. This results demonstrates the data efficiency of our system and indicates why it captures LFS well, as less common senses naturally only have a few training examples in the data.
In summary, the overall contributions of this work are as follows:
- We present a jointly optimized bi-encoder model (BEM) for WSD that improves performance on all-words English WSD.
- We show that our model's improvements come from better performance on LFS and zero-shot examples, without sacrificing accuracy on the most common senses.
- We examine why our model performs well on LFS with a number of experiments, including an evaluation of the BEM in a few-shot learning setting demonstrating that the bi-encoder generalizes well from limited data.
The source code and trained models for our WSD bi-encoders can be found at https://github. com/facebookresearch/wsd-biencoders.
Word Sense Disambiguation (WSD) is the task of predicting the particular sense, or meaning, of a word when it occurs in a specific context (Navigli, 2009). Understanding what a word means in context is critical to many NLP tasks, and WSD has been shown to help downstream tasks such as machine translation (MT) (Vickrey et al., 2005; Neale et al., 2016; Rios Gonzales et al., 2017) and information extraction (IE) (Ciaramita and Altun, 2006; Bovi et al., 2015).
The formulation of WSD that we address is all-words WSD, where the model disambiguates every ambiguous word in the data (e.g., Palmer et al. (2001); Moro and Navigli (2015)). Many WSD systems approached this task with manually engineered features that were used to learn an independent classifier, or word expert, for each ambiguous lemma (Zhong and Ng, 2010; Shen et al., 2013). Later work also integrated word embeddings into this independent classifier approach (Rothe and Schütze, 2015; Iacobacci et al., 2016).
Neural models for WSD built on this approach by training encoders for better feature extraction; they then either still learned independent classifiers on top of the encoded features (Kägebäck and Salomonsson, 2016), or labeled each word using a shared output space (Raganato et al., 2017a). Other
neural approaches used semi-supervised learning to augment the learned representations with additional data (Melamud et al., 2016; Yuan et al., 2016).
2.1 Lexical Resources for WSD
Definitions of senses, or glosses, have been shown to be a valuable resource for improving WSD. Lesk (1986) used the overlap between the definitions of senses and the context of the target word to predict the target sense. This approach was later extended to incorporate WordNet graph structure (Banerjee and Pedersen, 2003) and to incorporate word embeddings (Basile et al., 2014). More recently, Luo et al. (2018a,b) added sense glosses as additional inputs into their neural WSD system, significantly improving overall performance.
Most similar to our work, Kumar et al. (2019) represented senses as continuous representations learned from encoded glosses. However, they took a pipelined approach and supervised the gloss encoder with knowledge graph embeddings; they then froze the sense representations to use them as static supervision for training the WSD system. This approach requires an additional form of supervision (for which they used knowledge graph embeddings), making it more difficult to generalize to new data without that source of supervision. In comparison, our model is trained in an end-to-end manner and learns to embed gloss text without additional supervision.
Other work has shown that neural models capture useful semantic information about words from their definitions, and has used them to encode lexical representations (Bahdanau et al., 2017; Bosc and Vincent, 2018). While they focused on representing words, rather than specific senses, their modeling approaches could be extended to sense representations.
2.2 Pretrained NLP Models for WSD
Pretrained models have been shown to capture a surprising amount of word sense information from their pretraining objectives alone (Peters et al., 2018; Stanovsky and Hopkins, 2018; Coenen et al., 2019), allowing the frozen pretrained representations to compete with previous state-of-the-art WSD systems (Hadiwinoto et al., 2019). Building on these findings, Vial et al. (2019) incorporates pretrained BERT representations as inputs into their WSD system, and Loureiro and Jorge
Figure 1: Architecture of our bi-encoder model for WSD. The context sentence and sense gloss text are input into the context and gloss encoders, respectively; each encoder is initialized with BERT. We take the  $i^{th}$  output of the context encoder as the representation for the target word  $w_{i}$ ; the first output of the gloss encoder, which corresponds to the BERT-specific start token [CLS], is used as a representation for each candidate sense  $s$ .  $w_{i}$  is compared to  $s$  with a dot product, and the sen se with the highest similarity to  $w_{i}$  is assigned as the predicted label.
(2019) uses BERT's contextualized outputs to create sense embeddings for each sense in WordNet.
Another approach to using pretrained models for WSD is to formulate the task as a sentence-pair classification problem, in which (context sentence, gloss) pairs are concatenated and cross-encoded with the pretrained model. This reduces the WSD task to a binary classification problem where the model is trained to predict whether the gloss matches the sense of the target word in the context sentence (Huang et al., 2019). Given that transformer compute scales polynomially in the input length, our approach of independently encoding the contexts and sense glosses is more computationally efficient, and we also show that it performs better on the all-words WSD task (Section 5.1).
In this section, we present an approach for WSD that is designed to more accurately model less frequent senses by better leveraging the glosses that define them. The overall model architecture is shown in Figure 1. Our bi-encoder model (BEM) consists of two independent encoders: (1) a context encoder, which represents the target word (and its surrounding context) and (2) a gloss encoder, that embeds the definition text for each word sense. These encoders are trained to embed each token near the representation of its correct word sense. Each encoder is a deep transformer network initialized with BERT, in order to leverage the word sense information it captures from pretraining (Coenen et al., 2019; Hadiwinoto et al., 2019). To describe our approach, we formally define the task of WSD (Section 3.1), and then present the BEM system in
detail (Section 3.2).
3.1 Word Sense Disambiguation
Word Sense Disambiguation (WSD) is the task of assigning a sense to a target word, given its context. More formally, given a word  $w$  and context  $c$ , a WSD system is a function  $f$  such that  $f(w, c) = s$  subject to  $s \in S_w$ , where  $S_w$  is all possible candidate senses of  $w$ .
We focus on the task of all-words WSD, in which every ambiguous word in a given context is disambiguated. In this setting, a WSD model is given as input  $\mathbf{c} = c_0, c_1, \ldots, c_n$  and outputs a sequence of sense predictions  $\mathbf{s} = s_{c_0}^i, s_{c_1}^j, \ldots, s_{c_n}^m$ , where the model predicts the  $i^{th}, j^{th}$ , and  $m^{th}$  senses from the candidate sense sets for  $c_0, c_1$ , and  $c_n$ , respectively. For our approach, we assume for each sense  $s$  that we also have a gloss  $g_s = g_0, g_1, \ldots, g_n$  that defines  $s$ .
3.2 Bi-encoder for WSD
Our bi-encoder architecture independently encodes target words (with their contexts) and sense glosses (Bromley et al., 1994; Humeau et al., 2019). Each of these models are initialized with BERT-base: therefore, the inputs to each encoder are padded with BERT-specific start and end symbols: input  $\mathbf{z} = z_0,z_1,\dots,z_n$  is modified to  $\mathbf{z} = [\mathrm{CLS}]$ ,  $z_0,z_1,\dots,z_n$ , [SEP].
The context encoder, which we define as  $T_{c}$ , takes as input a context sentence  $\mathbf{c}$  containing a set of target words  $\mathbf{w}$  to be disambiguated, s.t.  $\mathbf{c} = c_{0}, c_{1}, \ldots, w_{i}, \ldots, c_{n}$ , where  $w_{i}$  is the  $i^{th}$  target word
2In practice, this means every content word – noun, verb, adjective, and adverb – in the context is disambiguated by the WSD system.
in the context sentence. The encoder then produces a sequence of representations  $\mathbf{r}$ , where
$$
r _ {w _ {i}} = T _ {c} (\mathbf {c}) [ i ]
$$
or the  $i^{th}$  representation output by  $T_{c}$ . For words that are tokenized into multiple subword pieces by the BERT tokenizer, we represent the word by the average representation of its subword pieces. For example, let the  $j^{th}$  through  $k^{th}$  tokens correspond to the subpieces of the  $i^{th}$  word, we have
$$
r _ {w _ {i}} = \frac {1}{k - j} \sum_ {l = j} ^ {k} (T _ {c} (\mathbf {c}) [ l ])
$$
The gloss encoder, defined as  $T_{g}$ , takes in a gloss  $\mathbf{g_s} = g_0, g_1, \ldots, g_m$  that defines the sense  $s$  as input. The gloss encoder represents  $s$  as
$$
r _ {s} = T _ {g} (\mathbf {g _ {s}}) [ 0 ]
$$
where we take the first representation output by the gloss encoder (corresponding to the input [CLS] token) as a global representation for  $s$ .
We then score each candidate sense  $s \in S_w$  for a target word  $w$  by taking the dot product of  $r_w$  against every  $r_s$  for  $s \in S_w$ :
$$
\phi (w, s _ {i}) = r _ {w} \cdot r _ {s _ {i}}
$$
for  $i = 0,\dots,|S_w|$ . During evaluation, we predict the sense  $\hat{s}$  of the target word  $w$  to be the sense  $s_i\in S_w$  whose representation  $r_{s_i}$  has the highest dot product score with  $r_w$ .
We use a cross-entropy loss on the scores for the candidate senses of the target word  $w$  to train our bi-encoder model; the loss function of our system given a (word, sense) pair  $(w, s_i)$  is
$$
\mathcal {L} (w, s _ {i}) = - \phi (w, s _ {i}) + \log \sum_ {j = 0} ^ {| S _ {w} |} \exp (\phi (w, s _ {j}))
$$
4.1 WSD Task and Datasets
We evaluate our BEM system with the WSD framework established in Raganato et al. (2017b). We train our model on SemCor, a large dataset manually annotated with senses from WordNet that contains 226,036 annotated examples covering 33,362 separate senses (Miller et al., 1993). We use the SemEval-2007 (SE07) dataset as our development
set (Pradhan et al., 2007); we hold out Senseval-2 (SE2; Palmer et al. (2001)), Senseval-3 (SE3; Snyder and Palmer (2004)), SemEval-2013 (SE13; Navigli et al. (2013)), and SemEval-2015 (SE15; Moro and Navigli (2015)) as evaluation sets, following standard practice. All sense glosses used in our system are retrieved from WordNet 3.0 (Miller, 1995).
We compare the BEM against a number of baseline systems. We first consider two knowledge-based baselines: WordNet S1, which labels each example with its first (most common) sense as specified in WordNet, and most frequent sense (MFS), which assigns each word the most frequent sense it occurs with in the training data.
We also compare against the pretrained model used to initialize our BEM system, BERT-base (Devlin et al., 2019), by learning a linear classifier for WSD on top of frozen BERT representations output by the final layer. We learn the weights of this output layer by performing a softmax over the possible candidate senses of the target word and masking out any unrelated senses. We find that fine-tuning BERT-base on WSD classification does not improve performance over the frozen model; this finding holds for each of the pretrained encoders we consider. Specific training details for the frozen BERT baseline are given in Section 4.3. Since this baseline uses a standard, discrete classification setup, it backs off to the WordNet S1 predictions for unseen words.
Finally, we compare performance to six recent state-of-the-art systems. The HCAN (Luo et al., 2018a) model incorporates sense glosses as additional inputs into a neural WSD classifier. The EWISE model pretrains a gloss encoder against graph embeddings before freezing the learned sense embeddings and training an LSTM encoder on the WSD task (Kumar et al., 2019). Hadiwinoto et al. (2019) investigates different ways of using the (frozen) pretrained BERT model to perform WSD, with their GLU model performing best; Vial et al. (2019) used various sense vocabulary compression (SVC) approaches to improve WSD learning. The LMMS system performs k-NN on word representations produced BERT against a learned inventory of embeddings for WordNet senses (Loureiro and
3For this work, we report the best result from a comparable setting (i.e., from a single model on the same training data).
Dev SE07	Test Datasets	Concatenation of all Datasets
SE2	SE3	SE13	SE15	Nouns	Verbs	Adj.	Adv.	ALL
Baseline Systems
WordNet S1	55.2	66.8	66.2	63.0	67.8	67.6	50.3	74.3	80.9	65.2
MFS (in training data)	54.5	65.6	66.0	63.8	67.1	67.7	49.8	73.1	80.5	65.5
BERT-base	68.6	75.9	74.4	70.6	75.2	75.7	63.7	78.0	85.8	73.7
Prior Work
HCAN	-	72.8	70.3	68.5	72.8	72.7	58.2	77.4	84.1	71.1
EWISE	67.3	73.8	71.1	69.4	74.5	74.0	60.2	78.0	82.1	71.8
GLU	68.1	75.5	73.6	71.1	76.2	-	-	-	-	74.1
LMMS	68.1	76.3	75.6	75.1	77.0	-	-	-	-	75.4
SVC	-	-	-	-	-	-	-	-	-	75.6
GlossBERT	72.5	77.7	75.2	76.1	80.4	79.8	67.1	79.6	87.4	77.0
BEM	74.5	79.4	77.4	79.7	81.7	81.4	68.5	83.0	87.9	79.0
Table 1: F1-score (\%) on the English all-words WSD task. ALL is the concatenation of all datasets, including the development set SE07. We compare our bi-encoder model (BEM) against the WordNet S1 and most frequent sense (MFS) baselines, as well as a frozen BERT-base classifier and recent prior work on this task.
Jorge, 2019). GlossBERT fine-tunes BERT on WSD by jointly encoding the context sentences and glosses (Huang et al., 2019); this approach relies on a single, cross-encoder model, rather than our more efficient bi-encoder approach to independently encode contexts and glosses.
4.3 Model Architecture and Optimization
Our pretrained baseline is learned using a single linear layer and softmax on the output of the final layer of the frozen BERT-base model. Similarly, each encoder in the bi-encoder model is initialized with BERT-base. We obtain representations from each encoder by taking the outputs from the final layer of each encoder, and we optimize the model with a cross-entropy loss on the dot product score of these representations. Additional hyperparameter and optimization details are given in the supplementary materials.
We present a series of experiments to evaluate our bi-encoder WSD model. We first compare the BEM against several baselines and prior work on English all-words WSD (Section 5.1), and then evaluate performance on the most frequent (MFS), less frequent (LFS), and zero-shot examples (Section 5.2).
5.1 Overall Results
Table 1 shows overall F1 results on the English all-words WSD task (Raganato et al., 2017b). Frozen BERT-base is a strong baseline, outperforming all of the prior work that does not incorporate pretraining into their systems  $(\mathrm{GAS}_{ext}$ , HCAN, and EWISE). The GLU and SVC systems, which use the representations learned by BERT without fin-tuning, both slightly outperform our pretrained baseline. GlossBERT achieves even better WSD performance by fine-tuning BERT with their cross-encoder approach.
However, we also find that our BEM achieves the best F1 score on the aggregated ALL evaluation set, outperforming all baselines and prior work by at least 2 F1 points. This improvement holds across all of the evaluation sets in the WSD evaluation framework as well as for each part-of-speech on which we perform WSD. Therefore, we see that although many of the prior approaches considered build on pretrained models, we empirically observe that our bi-encoder model is a particularly strong method for leveraging BERT.
5.2 Zero-shot and Rare Senses Results
To better understand these overall results, we break down performance across different sense frequencies. We split examples from the aggregated ALL evaluation set into mentions with the most frequent sense (MFS) of the target word and mentions that are labeled with the other, less frequent senses
4We initialize the models with BERT-base due to better baseline performance on WSD than RoBERTa-base, see Section 6.1 for more details
MFS	LFS	Zero-shot
Words	Senses
WordNet S1	100.0	0.0	84.9	53.9
BERT-base	94.9	37.0	84.9	53.6
EWISE	93.5	31.2	91.0	-
BEM	94.1	52.6	91.2	68.9
BEM-bal	89.5	57.0	91.9	71.8
Table 2: F1-score  $(\%)$  on the MFS, LFS, and zero-shot subsets of the ALL evaluation set. Zero-shot examples are the words and senses (respectively) that do not occur in the training data. The balanced BEM system, BEM-bal, is considered in Section 6.2.
Model Ablation	Dev F1	Δ
Full BEM	74.5	-
Frozen Context Encoder	70.1	-4.4
Frozen Gloss Encoder	68.1	-6.4
Tied Encoders	74.1	-0.4
(LFS) of that word. We also consider zero-shot performance for both unseen words and unseen senses by evaluating performance on examples that are not observed during training. We compare our model against the frozen BERT-base baseline and EWISE (Kumar et al., 2019), which also reported performance in these settings (Table 2).
BEM performs best on rare senses. The vast majority of BEM's gains comes from better performance on the LFS examples, leading to a 15.6 F1 improvement over the BERT baseline on the LFS subset. Despite this gain on less frequent senses, BEM remains (approximately) as accurate on the MFS examples as prior work and the BERT baseline. While we still see a large difference of 41.5 F1 points between the MFS and LFS examples with BEM, this is a strong improvement over both the BERT-base baseline and the EWISE system.
BEM shows competitive performance on unseen words. Next we evaluated BEM on zero-shot words that did not occur in the training data. In this setting, WordNet S1 is a very strong baseline that achieves almost 85 F1 points from an untrained knowledge-based approach. Since the BERT-base model backs off to the WordNet S1 baseline for unseen words, it gets the same performance in this
Table 3: Ablations on the bi-encoder model (BEM). We consider the effect of freezing each of the two encoders and of tying the parameters of the encoders on development set performance.
Pretrained Model	Dev F1
BERT-base	68.6
BERT-large	67.5
RoBERTa-base	68.1
RoBERTa-large	69.5
Table 4: Performance of various pretrained encoders on the WSD development set.
setting. The EWISE model from previous work, as well as our BEM, both outperform this baseline, with the BEM achieving a slightly higher F1 score for zero-shot words.
BEM generalizes well to embedding zero-shot senses. The bi-encoder model allows us to predict senses that do not occur in the training set by embedding senses; this is a valuable modeling contribution since many senses do not occur in even the largest manually labeled WSD datasets. We therefore evaluate the BEM and baselines on zero-shot senses. The WordNet most common sense baseline remains strong, and the BERT baseline performs similarly to this WordNet S1 baseline. However, our bi-encoder model outperforms both baselines by at least 15 F1 points. This demonstrates that BEM is able to learn useful sense representations from the gloss text that are able to generalize well to unseen senses.
6 Analysis Experiments
In our model evaluation, we found that BEM outperforms prior work by improving disambiguation of less frequent senses while maintaining high performance on common ones. This section presents a series of analysis experiments in order to determine which aspects of the approach contribute to these improvements. In Section 6.1, we ablate different aspects of our model, and we consider the effect of balancing the training signal across senses with different frequencies in Section 6.2. Finally, we perform a qualitative analysis of the learned sense embedding space in Section 6.3.
6.1 Model Ablations
We ablate aspects of the bi-encoder model in order to see how they contribute to the overall performance; we consider freezing the context encoder, freezing the gloss encoder, and tying the two encoders so that they share the same parameters.
The results are shown in Table 3. A frozen gloss
Figure 2: Representations for the word plant encoded by the frozen BERT-base encoder (left) and the context encoder of our BEM system (right); visualized with t-SNE. Sense glosses are from Wordnet (Miller, 1995).