SlowGuess/ACL_2020_OCR huggingface dataset:
Yoav Levine Barak Lenz Or Dagan Ori Ram Dan Padnos Or Sharir Shai Shalev-Shwartz Amnon Shashua Yoav Shoham
AI21 Labs, Tel Aviv, Israel
{yoav1, barak1, ord, orir,...}@ai21.com
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the 'Word in Context' task.
Neural language models have recently undergone a qualitative leap forward, pushing the state of the art on various NLP tasks. Together with advances in network architecture (Vaswani et al., 2017), the use of self-supervision has proven to be central to these achievements, as it allows the network to learn from massive amounts of unannotated text.
The self-supervision strategy employed in BERT (Devlin et al., 2019) involves masking some of the words in an input sentence, and then training the model to predict them given their context. Other proposed approaches for self-supervised objectives, including unidirectional (Radford et al., 2019), permutational (Yang et al., 2019), or word insertion-based (Chan et al., 2019) methods, operate similarly, over words. However, since a given word form can possess multiple meanings (e.g., the word 'bass' can refer to a fish, a guitar, a type of singer, etc.), the word itself is merely a surrogate of its
actual meaning in a given context, referred to as its sense. Indeed, the word-form level is viewed as a surface level which often introduces challenging ambiguity (Navigli, 2009).
In this paper, we bring forth a novel methodology for applying weak-supervision directly on the level of a word's meaning. By infusing word-sense information into BERT's pre-training signal, we explicitly expose the model to lexical semantics when learning from a large unannotated corpus. We call the resultant sense-informed model SenseBERT. Specifically, we add a masked-word sense prediction task as an auxiliary task in BERT's pre-training. Thereby, jointly with the standard word-form level language model, we train a semantic-level language model that predicts the missing word's meaning. Our method does not require sense-annotated data; self-supervised learning from unannotated text is facilitated by using WordNet (Miller, 1998), an expert constructed inventory of word senses, as weak supervision.
We focus on a coarse-grained variant of a word's sense, referred to as its WordNet supersense, in order to mitigate an identified brittleness of fine-grained word-sense systems, caused by arbitrary sense granularity, blurriness, and general subjectiveness (Kilgarriff, 1997; Schneider, 2014). WordNet lexicographers organize all word senses into 45 supersense categories, 26 of which are for nouns, 15 for verbs, 3 for adjectives and 1 for adverbs (see full supersense table in the supplementary materials). Disambiguating a word's supersense has been widely studied as a fundamental lexical categorization task (Ciaramita and Johnson, 2003; Basile, 2012; Schneider and Smith, 2015).
We employ the masked word's allowed supersenses list from WordNet as a set of possible labels for the sense prediction task. The labeling of words with a single supersense (e.g., 'sword' has only the supersense noun.artifact) is straightforward: We
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4656-4667 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
train the network to predict this supersense given the masked word's context. As for words with multiple supersenses (e.g., 'bass' can be: noun.food, noun.animal, noun.artifact, noun.person, etc.), we train the model to predict any of these senses, leading to a simple yet effective soft-labeling scheme.
We show that SenseBERT<sub>BASE</sub> outscores both BERT<sub>BASE</sub> and BERT<sub>LARGE</sub> by a large margin on a supersense variant of the SemEval Word Sense Disambiguation (WSD) data set standardized in Raganato et al. (2017). Notably, SenseBERT receives competitive results on this task without funetuning, i.e., when training a linear classifier over the pretrained embeddings, which serves as a testament for its self-acquisition of lexical semantics. Furthermore, we show that SenseBERT<sub>BASE</sub> surpasses BERT<sub>LARGE</sub> in the Word in Context (WiC) task (Pilehvar and Camacho-Collados, 2019) from the SuperGLUE benchmark (Wang et al., 2019), which directly depends on word-supersense awareness. A single SenseBERT<sub>LARGE</sub> model achieves state of the art performance on WiC with a score of 72.14, improving the score of BERT<sub>LARGE</sub> by 2.5 points.
Neural network based word embeddings first appeared as a static mapping (non-contextualized), where every word is represented by a constant pretrained embedding (Mikolov et al., 2013; Pennington et al., 2014). Such embeddings were shown to contain some amount of word-sense information (Iacobacci et al., 2016; Yuan et al., 2016; Arora et al., 2018; Le et al., 2018). Additionally, sense embeddings computed for each word sense in the word-sense inventory (e.g. WordNet) have been employed, relying on hypernymity relations (Rothe and Schütze, 2015) or the gloss for each sense (Chen et al., 2014). These approaches rely on static word embeddings and require a large amount of annotated data per word sense.
The introduction of contextualized word embeddings (Peters et al., 2018), for which a given word's embedding is context-dependent rather than precomputed, has brought forth a promising prospect for sense-aware word embeddings. Indeed, visualizations in Reif et al. (2019) show that sense sensitive clusters form in BERT's word embedding space. Nevertheless, we identify a clear gap in this ablity. We show that a vanilla BERT model trained with the current word-level self-supervision,
burdened with the implicit task of disambiguating word meanings, often fails to grasp lexical semantics, exhibiting high supersense misclassification rates. Our suggested weakly-supervised word-sense signal allows SenseBERT to significantly bridge this gap.
Moreover, SenseBERT exhibits an improvement in lexical semantics ability (reflected by the Word in Context task score) even when compared to models with WordNet infused linguistic knowledge. Specifically we compare to Peters et al. (2019) who re-contextualize word embeddings via a word-to-entity attention mechanism (where entities are WordNet lemmas and synsets), and to Loureiro and Jorge (2019) which construct sense embeddings from BERT's word embeddings and use the WordNet graph to enhance coverage (see quantitative comparison in table 3).
3 Incorporating Word-Supersense Information in Pre-training
In this section, we present our proposed method for integrating word sense-information within SenseBERT's pre-training. We start by describing the vanilla BERT architecture in subsection 3.1. We conceptually divide it into an internal transformer encoder and an external mapping  $W$  which translates the observed vocabulary space into and out of the transformer encoder space [see illustration in figure 1(a)].
In the subsequent subsections, we frame our contribution to the vanilla BERT architecture as an addition of a parallel external mapping to the words supersenses space, denoted  $S$  [see illustration in figure 1(b)]. Specifically, in section 3.2 we describe the loss function used for learning  $S$  in parallel to  $W$ , effectively implementing word-form and word-sense multi-task learning in the pre-training stage. Then, in section 3.3 we describe our methodology for adding supersense information in  $S$  to the initial Transformer embedding, in parallel to word-level information added by  $W$ . In section 3.4 we address the issue of supersense prediction for out-of-vocabulary words, and in section 3.5 we describe our modification of BERT's masking strategy, prioritizing single-supersensed words which carry a clearer semantic signal.
The input to BERT is a sequence of words  $\{x^{(j)}\in$ $\{0,1\}^{D_W}\}_{j = 1}^N$  where  $15\%$  of the words are re
(b) SenseBERT
Figure 1: SenseBERT includes a masked-word supersense prediction task, pre-trained jointly with BERT's original masked-word prediction task (Devlin et al., 2019) (see section 3.2). As in the original BERT, the mapping from the Transformer dimension to the external dimension is the same both at input and at output ( $W$  for words and  $S$  for supersenses), where  $M$  denotes a fixed mapping between word-forms and their allowed WordNet supersenses (see section 3.3). The vectors  $p^{(j)}$  denote positional embeddings. For clarity, we omit a reference to a sentence-level Next Sentence Prediction task trained jointly with the above.
placed by a [MASK] token (see treatment of subword tokenization in section 3.4). Here  $N$  is the input sentence length,  $D_W$  is the word vocabulary size, and  $x^{(j)}$  is a 1-hot vector corresponding to the  $j^{\mathrm{th}}$  input word. For every masked word, the output of the pretraining task is a word-score vector  $y^{\mathrm{words}} \in \mathbb{R}^{D_W}$  containing the per-word score. BERT's architecture can be decomposed to (1) an internal Transformer encoder architecture (Vaswani et al., 2017) wrapped by (2) an external mapping to the word vocabulary space, denoted by  $W$ .<sup>1</sup>
The Transformer encoder operates over a sequence of word embeddings  $v_{\mathrm{input}}^{(j)} \in \mathbb{R}^d$ , where  $d$  is the Transformer encoder's hidden dimension. These are passed through multiple attention-based Transformer layers, producing a new sequence of contextualized embeddings at each layer. The Transformer encoder output is the final sequence of contextualized word embeddings  $v_{\mathrm{output}}^{(j)} \in \mathbb{R}^d$ .
The external mapping  $W \in \mathbb{R}^{d \times D_W}$  is effectively a translation between the external word vocabulary dimension and the internal Transformer dimension. Original words in the input sentence are translated into the Transformer block by applying this mapping (and adding positional encoding vectors  $p^{(j)} \in \mathbb{R}^d$ ):
$$
v _ {\text {i n p u t}} ^ {(j)} = W x ^ {(j)} + p ^ {(j)} \tag {1}
$$
The word-score vector for a masked word at position  $j$  is extracted from the Transformer encoder output by applying the transpose:  $y^{\text{words}} = W^\top v_{\text{output}}^{(j)}$  [see illustration in figure 1(a)]. The use of the same matrix  $W$  as the mapping in and out of the transformer encoder space is referred to as weight tying (Inan et al., 2017; Press and Wolf, 2017).
Given a masked word in position  $j$ , BERT's original masked-word prediction pre-training task is to have the softmax of the word-score vector  $y^{\mathrm{words}} = W^{\top}v_{\mathrm{output}}^{(j)}$  get as close as possible to a 1-hot vector corresponding to the masked word. This is done by minimizing the cross-entropy loss between the softmax of the word-score vector and a 1-hot vector corresponding to the masked word:
$$
\mathcal {L} _ {\mathrm {L M}} = - \log p (w | \text {c o n t e x t}), \tag {2}
$$
where  $w$  is the masked word, the context is composed of the rest of the input sequence, and the probability is computed by:
$$
p (w | \text {c o n t e x t}) = \frac {\exp \left(y _ {w} ^ {\text {w o r d s}}\right)}{\sum_ {w ^ {\prime}} \exp \left(y _ {w ^ {\prime}} ^ {\text {w o r d s}}\right)}, \tag {3}
$$
where  $y_{w}^{\mathrm{words}}$  denotes the  $w^{\mathrm{th}}$  entry of the word-score vector.
<sup>1</sup>For clarity, we omit a description of the Next Sentence Prediction task which we employ as in Devlin et al. (2019).
3.2 Weakly-Supervised Supersense Prediction Task
Jointly with the above procedure for training the word-level language model of SenseBERT, we train the model to predict the supersense of every masked word, thereby training a semantic-level language model. This is done by adding a parallel external mapping to the words supersenses space, denoted  $S \in \mathbb{R}^{d \times D_S}$  [see illustration in figure 1(b)], where  $D_S = 45$  is the size of supersenses vocabulary. Ideally, the objective is to have the softmax of the sense-score vector  $y^{\text{senses}} \in \mathbb{R}^{D_S} \coloneqq S^\top v_{\text{output}}^{(j)}$  get as close as possible to a 1-hot vector corresponding to the word's supersense in the given context.
For each word  $w$  in our vocabulary, we employ the WordNet word-sense inventory for constructing  $A(w)$ , the set of its "allowed" supersenses. Specifically, we apply a WordNet Lemmatizer on  $w$ , extract the different synsets that are mapped to the lemmatized word in WordNet, and define  $A(w)$  as the union of supersenses coupled to each of these synsets. As exceptions, we set  $A(w) = \emptyset$  for the following: (i) short words (up to 3 characters), since they are often treated as abbreviations, (ii) stop words, as WordNet does not contain their main synset (e.g. 'he' is either the element helium or the hebrew language according to WordNet), and (iii) tokens that represent part-of-word (see section 3.4 for further discussion on these tokens).
Given the above construction, we employ a combination of two loss terms for the supersense-level language model. The following allowed-senses term maximizes the probability that the predicted sense is in the set of allowed supersenses of the masked word  $w$ :
$$
\begin{array}{l} \mathcal {L} _ {\text {S L M}} ^ {\text {a l l o w e d}} = - \log p (s \in A (w) | \text {c o n t e x t}) \\ = - \log \sum_ {s \in A (w)} p (s | \text {c o n t e x t}), \tag {4} \\ \end{array}
$$
where the probability for a supersense  $s$  is given by:
$$
p (s \mid \text {c o n t e x t}) = \frac {\exp \left(y _ {s} ^ {\text {s e n s e s}}\right)}{\sum_ {s ^ {\prime}} \exp \left(y _ {s ^ {\prime}} ^ {\text {s e n s e s}}\right)}. \tag {5}
$$
The soft-labeling scheme given above, which treats all the allowed supersenses of the masked word equally, introduces noise to the supersense labels. We expect that encountering many contexts in a sufficiently large corpus will reinforce the correct labels whereas the signal of incorrect labels will diminish. To illustrate this, consider the following examples for the food context:
1. "This bass is delicious"
(supersenses: noun.food, noun.artifact, etc.)
2. "This chocolate is delicious"
(supersenses: noun.food, noun(attribute, etc.)
3. "This pickle is delicious"
(supersenses: noun.food, noun.state, etc.)
Masking the marked word in each of the examples results in three identical input sequences, each with a different sets of labels. The ground truth label, noun.food, appears in all cases, so that its probability in contexts indicating food is increased whereas the signals supporting other labels cancel out.
While  $\mathcal{L}_{\mathrm{SLM}}^{\mathrm{allowed}}$  pushes the network in the right direction, minimizing this loss could result in the network becoming overconfident in predicting a strict subset of the allowed senses for a given word, i.e., a collapse of the prediction distribution. This is especially acute in the early stages of the training procedure, when the network could converge to the noisy signal of the soft-labeling scheme.
To mitigate this issue, the following regularization term is added to the loss, which encourages a uniform prediction distribution over the allowed supersenses:
$$
\mathcal {L} _ {\mathrm {S L M}} ^ {\mathrm {r e g}} = - \sum_ {s \in A (w)} \frac {1}{| A (w) |} \log p (s | \text {c o n t e x t}), \tag {6}
$$
i.e., a cross-entropy loss with a uniform distribution over the allowed supersenses.
Overall, jointly with the regular word level language model trained with the loss in eq. 2, we train the semantic level language model with a combined loss of the form:
$$
\mathcal {L} _ {\mathrm {S L M}} = \mathcal {L} _ {\mathrm {S L M}} ^ {\text {a l l o w e d}} + \mathcal {L} _ {\mathrm {S L M}} ^ {\mathrm {r e g}}. \tag {7}
$$
3.3 Supersense Aware Input Embeddings
Though in principle two different matrices could have been used for converting in and out of the Tranformer encoder, the BERT architecture employs the same mapping  $W$ . This approach, referred to as weight tying, was shown to yield theoretical and practical benefits (Inan et al., 2017; Press and Wolf, 2017). Intuitively, constructing the Transformer encoder's input embeddings from the same mapping with which the scores are computed improves their quality as it makes the input more sensitive to the training signal.
(a) All Supersenses
(b) Noun Supersenses
Figure 2: UMAP visualization of supersense vectors (rows of the classifier  $S$ ) learned by SenseBERT at pre-training. (a) Clustering by the supersense's part-of-speech. (b) Within noun supersenses, semantically similar supersenses are clustered together (see more details in the supplementary materials).
We follow this approach, and insert our newly proposed semantic-level language model matrix  $S$  in the input in addition to  $W$  [as depicted in figure 1(b)], such that the input vector to the Transformer encoder (eq. 1) is modified to obey:
$$
v _ {\text {i n p u t}} ^ {(j)} = (W + S M) x ^ {(j)} + p ^ {(j)}, \tag {8}
$$
where  $p^{(j)}$  are the regular positional embeddings as used in BERT, and  $M \in \mathbb{R}^{D_S \times D_W}$  is a static 0/1 matrix converting between words and their allowed WordNet supersenses  $A(w)$  (see construction details above).
The above strategy for constructing  $v_{\mathrm{input}}^{(j)}$  allows for the semantic level vectors in  $S$  to come into play and shape the input embeddings even for words which are rarely observed in the training corpus. For such a word, the corresponding row in  $W$  is potentially less informative, since due to the low word frequency the model did not have sufficient chance to adequately learn it. However, since the model learns a representation of its supersense, the corresponding row in  $S$  is informative of the semantic category of the word. Therefore, the input embedding in eq. 8 can potentially help the model to elicit meaningful information even when the masked word is rare, allowing for better exploitation of the training corpus.
3.4 Rare Words Supersense Prediction
At the pre-processing stage, when an out-of-vocabulary (OOV) word is encountered in the corpus, it is divided into several in-vocabulary subword tokens. For the self-supervised word pre
diction task (eq. 2) masked sub-word tokens are straightforwardly predicted as described in section 3.1. In contrast, word-sense supervision is only meaningful at the word level. We compare two alternatives for dealing with tokenized OOV words for the supersense prediction task (eq. 7).
In the first alternative, called  $60K$  vocabulary, we augment BERT's original 30K-token vocabulary (which roughly contained the most frequent words) with additional 30K new words, chosen according to their frequency in Wikipedia. This vocabulary increase allows us to see more of the corpus as whole words for which supersense prediction is a meaningful operation. Additionally, in accordance with the discussion in the previous subsection, our sense-aware input embedding mechanism can help the model extract more information from lower-frequency words. For the cases where a sub-word token is chosen for masking, we only propagate the regular word level loss and do not train the supersense prediction task.
The above addition to the vocabulary results in an increase of approximately 23M parameters over the 110M parameters of  $\mathrm{BERT}_{\mathrm{BASE}}$  and an increase of approximately 30M parameters over the 340M parameters of  $\mathrm{BERT}_{\mathrm{LARGE}}$  (due to different embedding dimensions  $d = 768$  and  $d = 1024$ , respectively). It is worth noting that similar vocabulary sizes in leading models have not resulted in increased sense awareness, as reflected for example in the WiC task results (Liu et al., 2019).
As a second alternative, referred to as average embedding, we employ BERT's regular 30K-token
(b)
Figure 3: (a) A demonstration of supersense probabilities assigned to a masked position within context, as given by SenseBERT's word-supersense level semantic language model (capped at  $5\%$ ). Example words corresponding to each supersense are presented in parentheses. (b) Examples of SenseBERT's prediction on raw text, when the unmasked input sentence is given to the model. This beyond word-form abstraction ability facilitates a more natural elicitation of semantic content at pre-training.
vocabulary and employ a whole-word-masking strategy. Accordingly, all of the tokens of a tokenized OOV word are masked together. In this case, we train the supersense prediction task to predict the WordNet supersenses of this word from the average of the output embeddings at the location of the masked sub-words tokens.
3.5 Single-Supersensed Word Masking
Words that have a single supersense are good anchors for obtaining an unambiguous semantic signal. These words teach the model to accurately map contexts to supersenses, such that it is then able to make correct context-based predictions even when a masked word has several supersenses. We therefore favor such words in the masking strategy, choosing  $50\%$  of the single-supersensed words in each input sequence to be masked. We stop if  $40\%$  of the overall  $15\%$  masking budget is filled with single-supersensed words (this rarely happens), and in any case we randomize the choice of the remaining words to complete this budget. As in the original BERT, 1 out of 10 words chosen for masking is shown to the model as itself rather than replaced with [MASK].
4 Semantic Language Model Visualization
A SenseBERT pretrained as described in section 3 (with training hyperparameters as in Devlin et al. (2019)), has an immediate non-trivial bi-product. The pre-trained mapping to the supersenses space, denoted  $S$ , acts as an additional head predicting a word's supersense given context [see figure 1(b)]. We thereby effectively attain a semantic-level lan
SenseBERTBASE	SemEval-SS Fine-tuned
30K no OOV	81.9
30K average OOV	82.7
60K no OOV	83
Table 1: Testing variants for predicting supersenses of rare words during SenseBERT's pretraining, as described in section 5.1. Results are reported on the SemEval-SS task (see section 5.2).  $30\mathrm{K} / 60\mathrm{K}$  stand for vocabulary size, and no/average OOV stand for not predicting senses for OOV words or predicting senses from the average of the sub-word token embeddings, respectively.
guage model that predicts the missing word's meaning jointly with the standard word-form level language model.
We illustrate the resultant mapping in figure 2, showing a UMAP dimensionality reduction (McInnes et al., 2018) of the rows of  $S$ , which corresponds to the different supersenses. A clear clustering according to the supersense part-of-speech is apparent in figure 2(a). We further identify finer-grained semantic clusters, as shown for example in figure 2(b) and given in more detail in the supplementary materials.
SenseBERT's semantic language model allows predicting a distribution over supersenses rather than over words in a masked position. Figure 3(a) shows the supersense probabilities assigned by SenseBERT in several contexts, demonstrating the model's ability to assign semantically meaningful categories to the masked position.
Finally, we demonstrate that SenseBERT enjoys