SlowGuess/ACL_2020_OCR huggingface dataset:
Breaking Through the  $80 \%$ Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information
Michele Bevilacqua and Roberto Navigli
Sapienza NLP Group
{bevilacqua, navigli}@di.uniroma1.it
Neural architectures are the current state of the art in Word Sense Disambiguation (WSD). However, they make limited use of the vast amount of relational information encoded in Lexical Knowledge Bases (LKB). We present Enhanced WSD Integrating Synset Embeddings and Relations (EWISER), a neural supervised architecture that is able to tap into this wealth of knowledge by embedding information from the LKB graph within the neural architecture, and to exploit pretrained synset embeddings, enabling the network to predict synsets that are not in the training set. As a result, we set a new state of the art on almost all the evaluation settings considered, also breaking through, for the first time, the  $80\%$  ceiling on the concatenation of all the standard all-words English WSD evaluation benchmarks. On multilingual all-words WSD, we report state-of-the-art results by training on nothing but English.
There is a growing body of research dealing with the integration of prior knowledge into neural networks for Natural Language Processing (NLP) tasks, be it through pretraining on self-supervised tasks such as language modeling (Peters et al., 2018; Devlin et al., 2019), or through the incorporation of information from knowledge bases (Peters et al., 2019; Logan et al., 2019). In Word Sense Disambiguation (WSD), i.e., the task of associating a word in context with the most appropriate meaning from a finite set of possible choices (Navigli, 2009), the gap between supervision and knowledge (Navigli, 2018) has been overcome by several efforts directed at learning effective vector representations (Loureiro and Jorge, 2019; Scarlini et al., 2020) in the same space as contextualized embeddings, and exploring the usage of definitional knowledge in supervised sequence learning neural architectures
(Luo et al., 2018; Kumar et al., 2019; Huang et al., 2019).
However, the Lexical Knowledge Bases (LKBs) from which such information is retrieved, such as WordNet (Miller, 1995) and BabelNet (Navigli and Ponzetto, 2012), also provide a great wealth of relational knowledge in structured form (i.e., hypernymy, meronymy, similarity, etc.), which is often neglected due to the non-trivial integration of data of this kind into neural architectures. Even though such information can, instead, be exploited by knowledge-based WSD algorithms (Agirre and Soroa, 2009; Moro et al., 2014), rivaling supervised pre-contextualized embedding approaches (Maru et al., 2019), the performances still lag behind (Huang et al., 2019; Vial et al., 2019).
Building on Extended WSD Integrating Sense Embeddings (EWISE) (Kumar et al., 2019), a neural WSD system incorporating prior knowledge through synset embeddings, we present Enhanced WSD Integrating Synset Embeddings and Relations (EWISER), a hybrid knowledge-based and supervised approach to WSD that integrates explicit relational information from the WordNet LKB. Our approach offers the following contributions:
1. We introduce the novel structured logits mechanism, which enables the exploitation of concept relatedness as determined by LKB edges. In our method, pre-softmax scores are a weighted combination of synset-specific scores, and can be computed via dot product with a sparse adjacency matrix.
2. We generalise the sense vector dot product technique from EWISE, showing that off-the-shelf pretrained embeddings can be used.
3. We show that the structured logits mechanism and the use of sense embeddings are orthogonal and can be exploited jointly.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Our approach is simple and extensible, does not require fine tuning of contextualized embeddings, and has a very modest parameter budget apart from synset embeddings. EWISER achieves a new state of the art in all-words English WSD. Moreover, we obtain state-of-the-art performances on the crosslingual all-words WSD evaluation, without using non-English training data.
Supervised WSD Supervised systems have to rely on expensive hand-labeled data to achieve good results (Pasini, 2020). The best approaches currently rely on neural networks. The model presented by Raganato et al. (2017) formulates the task as a token classification problem, with an LSTM with attention classifier producing a probability distribution over both words and senses. Subsequent work has shown that better results can be obtained by only having scores for senses or synsets (Vial et al., 2019). Shallower, simpler networks can achieve even better performances (Uslu et al., 2018).
Contextualized vectors can be exploited in token tagging architectures (Vial et al., 2019; Bevilacqua and Navigli, 2019; Hadiwinoto et al., 2019). However, purely supervised systems are dependent on the data they are trained on, therefore when some sense is underrepresented in the training corpus it is not easy for them to predict it.
LKBs in Supervised WSD More closely related to the core of our contribution, LKB information, such as natural language definitions of word meaning, can be exploited in neural token tagging architectures. For example, in GlossBERT (Huang et al., 2019) a pretrained BERT encoder is fed both the context sentence and the gloss, and is trained to predict whether the gloss correctly describes the use of the target word. Successful results have been obtained by encoding glosses in dense vectors (Luo et al., 2018).
In EWISE (Kumar et al., 2019), WSD is performed in a two-step process: first, gloss embeddings are produced through a training procedure that also takes into account the WordNet's graph structure; then, the gloss embeddings are scored via dot product with a contextual vector computed with an LSTM model, which is trained through regular categorical cross-entropy. Our work builds on top of EWISE in that it generalizes its sense vector dot product approach, but features a novel
mechanism that injects relational knowledge into the architecture through a simple additional sparse dot product operation. Moreover, we show that better performances can be obtained by training the output embedding matrix, and that different sense/synset vectors can be used to initialize the output embeddings.
Note that our approach is different from that of Vial et al. (2019), in that we do not conflate senses together through the use of WordNet hypernymy; rather, we maintain all the original meaning distinctions, and exploit the logit scores over the full vocabulary in a second, distinct step.
3 EWISER: Neural WSD with More Prior Knowledge
3.1 WSD as a classification problem
WSD can be treated as a simple token classification problem, similar to POS tagging or Named Entity Recognition. As such, abstracting away from all the intricacies of any particular supervised model, we need to produce a vector representation  $\mathbf{h} \in \mathbb{R}^d$  of a target word in a given context, and use it to yield a probability distribution over all its possible labels, i.e., its senses or synsets. The simplest way to do this is to learn a weight matrix  $O \in \mathbb{R}^{d \times |\mathcal{V}|}$ , where  $\mathcal{V}$  is the output vocabulary $^1$ , and compute a vector of unnormalized scores  $\mathbf{z}$  as the product of  $\mathbf{h}^T$  and  $O$ . Having multiple instances to classify packed into the matrix  $H$ , we can compute all the scores at the same time by a single dot product followed by a sum over columns with a bias vector:
$$
Z = H O + \mathbf {b} \tag {1}
$$
Finally,  $Z$  is transformed into a probability distribution through a standard softmax activation function. Typically,  $O$  is randomly initialized, and just trained end-to-end with the rest of the architecture (Raganato et al., 2017; Vial et al., 2019; Bevilacqua and Navigli, 2019). During training the categorical cross-entropy loss is computed for each instance  $Z_{i}$ . At inference time, the model predicts the synset  $\hat{s}$  with the highest probability among the set  $S(w_{i}) \subset V$  of possible synsets for word  $w_{i}$ :
$$
\hat {s} _ {i} = \underset {s \in S (w _ {i})} {\operatorname {a r g m a x}} Z _ {i, s} \tag {2}
$$
where, for each  $w_{i}$ ,  $S(w_{i})$  depends on both the lemma and its part-of-speech, and is determined by the WordNet inventory.
<sup>1</sup>We use synsets as output vocabulary.
3.2 Neural WSD Architecture
We now describe a simple neural WSD architecture to be used as the core on top of which we will integrate the EWISER additions. For each word to disambiguate, our network takes as input the sum of the outputs of the last 4 layers of BERT Large (cased) and uses a 2-layer feedforward to compute the logit scores  $Z$ :
$$
B = B _ {- 4} + B _ {- 3} + B _ {- 2} + B _ {- 1}
$$
$$
H _ {0} = \operatorname {B a t c h N o r m} (B) \tag {3}
$$
$$
H _ {1} = \operatorname {s w i s h} \left(H _ {0} W + \mathbf {b}\right)
$$
$$
Z = H _ {1} O
$$
where  $W$ ,  $\mathbf{b}$  are parameters of the models, and  $B_{-4}$  to  $B_{-1}$  are BERT hidden states<sup>2</sup>. We employ the swish activation function (Ramachandran et al., 2018), which has shown very promising results in NLP (Eger et al., 2018).
Note that, while our architecture is very simple, it would be straightforward to incorporate powerful additions such as a sequence encoder – like an LSTM or a Transformer (Vaswani et al., 2017) classifier. While this might indeed produce better performances, improvements of this kind are not directly pertinent to our contribution.
3.3 Structured Logits
The matrix multiplication in Equation 1 is wasteful during both training and inference, as it produces scores over the entire vocabulary  $\mathcal{V}$ , even though the number of possible synsets is much smaller than the cardinality of  $\mathcal{V}$ . Since the model is equally penalized by the cross-entropy loss when it gives a high score to a synset either related or unrelated to the correct one, there is little incentive to learn similar vectors for related synsets. Moreover, computing logits over the whole vocabulary does not bring any benefit in inference, as each score is computed independently, without taking into account connections between output classes.
We address this issue by devising an architecture, i.e., EWISER, that can inject into the network relatedness knowledge as encoded in an arbitrary graph, and use it in training as well as in inference.
3.3.1 Synset Graph in EWISER
As LKBs are structured into graphs, we want to be able to exploit, when computing the probability
distribution vector over  $\mathcal{V}$  for a target word, the explicit information of an arbitrary weighted graph  $G = \langle V,E,w\rangle$ , where  $w:E\to \mathbb{R}$ , and the vertices  $V = \mathcal{V}$  - i.e., the nodes are synsets. Instead of using the vector  $\mathbf{z}$  for prediction, we compute another vector  $\mathbf{q}$  where for each component, i.e.. for each synset  $s$ , the score synset  $\mathbf{q}_s$  is a function of both the "hidden" score  $\mathbf{z}_s$  for  $s$ , and the hidden scores  $\mathbf{z}_{s'}$  for all synsets  $s^\prime$  such that there is an edge  $\langle s^{\prime},s\rangle \in E$ . In order to do this, we calculate  $\mathbf{q}_s$  as  $\mathbf{z}_s$  plus the sum of the products of  $\mathbf{z}_s^\prime$  and the weight of the edge  $\langle s',s\rangle$ .
$$
\mathbf {q} _ {s} = \mathbf {z} _ {s} + \sum_ {s ^ {\prime} \in V | \langle s ^ {\prime}, s \rangle \in E} w \left(\left\langle s ^ {\prime}, s \right\rangle\right) \cdot \mathbf {z} _ {s ^ {\prime}} \tag {4}
$$
As a result,  $\mathbf{q}_s$  is a weighted combination of the scores for all the output vocabulary. In Figure 1 we show this process visually.
3.3.2 Computing  $Q$
The most natural way to encode the graph  $G$  is with the adjacency matrix  $A$ , in which  $A_{s_1s_2} = w(\langle s_1,s_2\rangle)$ . If  $A_{s_1s_2} = 0$  there is no edge between the two synsets. The new logits matrix  $Q$  can be obtained efficiently by simply computing the dot product between the hidden logits  $Z$  and the transposed adjacency matrix  $A^T$ , summing  $Z$  to the results.
$$
Z = H O + \mathbf {b}
$$
$$
Q = Z A ^ {T} + Z \tag {5}
$$
Finally, we apply the softmax function to  $Q$  to get the probabilities.
3.3.3 The matrix  $A$
In our case, we build the graph and adjacency matrix  $A$  from the relations between synsets or senses in WordNet. As WordNet relations are not weighted, for every synset  $s$  we set  $A_{s',s}$  to  $1/N$ , where  $N$  is the number of incoming connections. In this way we avoid imbalanced predictions towards synsets with more incoming connections.
We experiment with including different relations in  $A$ . Our base configuration includes similarity, verb group, and derivationally related<sup>3</sup> edges. As for hypernymy and its inverse, hyponymy, we experiment with different possible ways of including them in  $A$ : (i) including only hypernymy (hyper); (ii) only hyponymy (hypo); (iii) both hypernymy
<sup>3</sup>We connect two synsets with a derivationally related edge if at least one pair of senses therein is connected via a derivationally related edge.
2If a token consists of more than one subword, we average its subword representations.
Figure 1: The structured logits mechanism in EWISER. The example input is the sentence "The root of 4 is 2." Scores for a selection of synsets representing possible senses of root are shown. Going from left to right, the "hidden" logits (z) of related synsets are multiplied by the edge weights, summed together, and then added to the "hidden" logits of the related synsets, resulting in the "final" logits (q).
and hyponymy (hyper+ hypo); (iv) the transitive closure over hypernymy (the set of relations that are obtained by following hypernymy paths) (hyper*); (v) the transitive closure over hypernymy and hyponymy (hyper+ hypo*);
Informally, hypernymy and hyponymy correspond to different kinds of reasoning, which might be characterized as, respectively, inductive ("if it is an electronic device, then it might be a mouse") and deductive ("if it is a mouse, then it is an electronic device"). The closures are a way to flatten the hierarchy, thus enabling multi-hop reasoning by making the  $\mathbf{q}_s$  score dependent on the  $\mathbf{z}$  scores for synsets whose path distance to  $s$  is greater than 1 in the original graph.
Fine-tuning the adjacency matrix If weights in  $A$  are frozen, every connected synset gives an equal contribution to the final score  $\mathbf{q}_s$ . However, it is also reasonable to assume that not all synsets are equally relevant. For example, the score for inanimate object should be less relevant than that for device for predicting the hardware meaning of mouse. Thus, we experiment on fine-tuning  $A$  by only updating non-zero weights.
3.4 Output Layer Weights
While  $O$  can be seen as just the final linear map in the network, it is also reasonable to think about it as a counterpart of an embedding matrix. Whereas in the intermediate layers of the neural network there is no one-to-one mapping between values of the matrix and input or output classes, in  $O$  there is a distinct column for each of the elements in  $\mathcal{V}$ .
As a matter of fact, the logit of synset  $s(\mathbf{z}_s)$  is just the scalar product between  $\mathbf{h}$  and  $O_s^T$ , i.e., the column in  $O$  associated with  $s$ . So, just as with word embeddings,  $O$  can be seen as a collection for vector representations that have one-to-one mappings to output classes. Thus, it is possible to use synset embeddings to provide a better initialization for  $O$  than random. This idea has already been exploited by EWISE (Kumar et al., 2019), in which logit scores over  $V$  are computed by dot product between the hidden vector  $\mathbf{h}$  and the gloss embedding vector  $\mathbf{g}^{(\mathbf{s})}$  as follows:
$$
\mathbf {z} _ {s} = \mathbf {h} ^ {T} \mathbf {g} ^ {(\mathbf {s})} + \mathbf {b} ^ {T} \mathbf {g} ^ {(\mathbf {s})} \tag {6}
$$
where  $\mathbf{b}$  is a learned bias vector. Note that if we pack the synset gloss vector  $\mathbf{g}^{(\mathbf{s})}$  for every  $s\in \mathcal{V}$  into the  $O$  matrix, this looks almost identical to the canonical linear layer in Eq. 1, with the only difference being the fact that the bias is now the result of the dot product between  $\mathbf{b}$  and  $O$ , rather than being directly parametrized as a vector  $\in \mathbb{R}^{|\mathcal{V}|}$ .
3.4.1 Weight Training vs. Freezing vs. Thawing
In EWISE, the sense embeddings are learned independently from the WSD system and kept frozen during training. It is worth exploring whether better results can be achieved by allowing further refining of the weights during training. We expect initialization and freezing (which we refer to as, respectively,  $O$ -init and  $O$ -freeze) to have different effects depending on whether the gold synset is found in the training set. If weights are initialized and then up
dated during training, the columns in  $O$  corresponding to unattested synsets will only receive a "negative" signal from the cross-entropy loss; conversely, attested synsets can be further refined and predicted more accurately. If weights are frozen, the architecture will have to accommodate to the pretrained synset representations, meaning that, especially if there is no learned bias, it will be easier to predict unseen classes. No fine-tuning may, however, result in diminished performance, as the pre-trained synset representations are not tailored to WSD. An additional possibility to achieve better transfer between the information in the embeddings and the WSD system is to use a freeze-then-thaw scheme, similar to the chain-thaw method of Howard and Ruder (2018). The approach entails training an  $O$ -freeze model, restoring the best checkpoint, and then doing further training with  $O$  "thawed", i.e., with trainable weights.
We assess the performance of EWISER in all-words English WSD, against both a simple but competitive baseline, i.e., the simple feedforward network taking BERT hidden states as input described in Section 3.2, and state-of-art approaches. We first experiment separately on the integration of explicit relational information through structured logits (Section 4.1), and the integration of synset embeddings through the initialization of  $O$  (Section 4.2). Then, building on the results of these experiments, we evaluate the full EWISER architecture (Section 4.3). Finally, we assess our approach on cross-lingual WSD (Section 4.4), training on English and evaluating on French, German, Italian and Spanish.
4.1 Structured Logits
As explained in Section 3.3.2, in EWISER, relational knowledge is integrated through a dot product between the logits matrix  $Z$  and the transposed adjacency matrix  $A^T$ . We perform experiments with different configurations that vary according to which edges are included in  $A$ .
We experiment with the edge sets which are listed in Section 3.3.3. For each configuration we evaluate two different training runs, one in which  $A$  is frozen (A-freeze), and the other where edge weights are trained (A-train). We contrast the per
Model Arch.	ALL	No15	No15-
baseline	-	74.2	73.9	52.2
hyper	A-freeze	75.6	75.4	59.8
A-train	75.9	75.5	59.2
hypo	A-freeze	74.6	74.4	57.7
A-train	74.6	74.3	54.5
hyper+hypo	A-freeze	75.7	75.5	59.8
A-train	75.7	75.4	57.7
hyper*	A-freeze	75.2	75.0	58.6
A-train	75.4	75.3	57.7
hyper+hypo*	A-freeze	75.4	75.3	59.9
A-train	74.7	74.4	56.5
Table 1: Evaluation of structured logits on English allwords WSD. F1 is reported.
formance of the models with the above-mentioned baseline.
4.1.2 Data & Hyperparameters
We train the baseline and the configurations under comparison on SemCor (Miller et al., 1994) for 20 epochs, with a batch size of 4000 tokens. We do not employ sentences as context. Rather, we split documents in chunks of at most 100 tokens. The hidden size of the 2-layer feedforward is 512, with a dropout value of 0.2. The optimizer is Adam (Kingma and Ba, 2015), which we employ with a learning rate of  $10^{-4}$ . Following Bevilacqua and Navigli (2019), we select as development set (to select the best epoch) the SemEval-2015 dataset (Moro and Navigli, 2015). As customary, we report the results on the concatenation (ALL) of all the evaluation datasets from Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-2007 (Pradhan et al., 2007), SemEval-2013 (Navigli et al., 2013), and the aforementioned SemEval-2015. In addition, we report performances on ALL with all instances from the development set removed (No15), and on the subset of No15 whose gold synsets do not appear in SemCor  $(\mathbf{No}15^{-})$ .
We report in Table 1 the results of the experiments on the addition of structured logits to the baseline architecture.
As can be seen, the use of hypernyms brings the biggest gain to performances, with the strongest improvement against the baseline reported with simple hypernymy and fine-tuning of  $A$ : 1.7 points on ALL and 1.6 on No15. The closures, i.e., hyper* and hyper+ hypo*, do not seem to be very
beneficial, achieving slightly worse results than the simple counterpart. Much of the improvement seems to come from the increased performance of the unseen split  $\mathrm{No15^{-}}$  where the gold is not in SemCor, with an absolute improvement of 7.6 points with hypernymy edges and no fine-tuning, and of 7 points with hypernymy edges and fine-tuning. Fine-tuning  $A$  makes for better results than keeping the weights of the adjacency matrix fixed on both ALL and No15, but results in slight-to-moderate decreases on  $\mathrm{No15^{-}}$ , as the network is able to adjust the weights in order to bring down the q scores for unseen synsets.
4.2 Output Embeddings
As in EWISE, in EWISER logits are computed by a dot product between a matrix of hidden scores and output synset embeddings. However, we do not train our own synset embeddings: rather, we employ off-the-shelf vectors. In this section we evaluate the performance of different options both in the choice of the embeddings and in how they are integrated into the network. We contrast the performance with our baseline, in which the  $O$  matrix is randomly initialized and the embeddings are trained.
4.2.1 Setting
We experiment with different options for the initialization of  $O$ :
Deconf 300d We use the 300-dimensional vectors released by Pilehvar and Collier (2016), which are built from Word2Vec Google news word embeddings.
LMMS 2048d We use the 2048-dimensional vectors produced by Loureiro and Jorge (2019), built as the concatenation of BERT Large cased states' centroids for instances in SemCor with the synset gloss vector, computed from BERT Large states as well. We normalize the vectors to unit length. Since LMMS vectors are quite big, we reduce the number of dimensions to 512 with truncated SVD.
SensEmBERT+LMMS 2048d SensEmBERT (Scarlini et al., 2020) enhances LMMS by exploiting BabelNet and Wikipedia. SensEmBERT only includes nouns, but its vectors are in the same space as LMMS, so we use the former in combination with verbs, adjectives and adverbs from the latter. We employ the same preprocessing as with LMMS.
Model Arch.	ALL	No15	No15-
baseline	-	74.2	73.9	52.2
Deconf	O-init	75.3	75.2	55.2
O-freeze	66.4	66.0	72.2
O-thaw	75.3	75.2	60.5
O-thaw*	73.8	73.7	62.3
LMMS	O-init	75.5	75.4	55.1
O-freeze	75.9	75.4	59.4
O-thaw	75.4	75.0	57.4
O-thaw*	75.8	75.4	57.3
LMMS + SensEmBERT	O-init	76.1	76.0	59.4
O-freeze	76.3	76.0	64.7
O-thaw	76.4	76.1	62.3
O-thaw*	76.7	76.6	63.4
Table 2: Evaluation of  $O$  initialization and training strategies on English all-words WSD. F1 is reported.
For each sense embedding system, we report results with four different training schemes: plain initialization (O-init); initialization and freezing (O-freeze); restore the best O-freeze, then thaw the weights of  $O$  (O-thaw); the same as for  $O$ -thaw, but reducing the learning rate to  $10^{-5}$  ( $O$ -thaw*). In all cases, synset embeddings are computed as the centroid of the senses contained in the synset.
4.2.2 Data & Hyperparameters