SlowGuess/ACL_2020_OCR huggingface dataset:
(a) SemEval-SS	The team used a battery of the newly developed “gene probes”	BERT noun.artifact	SenseBERT noun.group
Ten shirt-sleeved ringers stand in a circle, one foot ahead of the other in a prize-fighter's stance	nounquantity	noun.body
(b) WiC	Sent. A: The kick must be synchronized with the arm movements.	Sent. B: A sidecar is a smooth drink but it has a powerful kick.	Same	Different
Sent. A: Plant bugs in the dissident's apartment.	Sent. B: Plant a spy in Moscow.	Different	Same
Figure 4: Example entries of (a) the SemEval-SS task, where a model is to predict the supersense of the marked word, and (b) the Word in Context (WiC) task where a model must determine whether the underlined word is used in the same/different supersense within sentences A and B. In all displayed examples, taken from the corresponding development sets, SenseBERT predicted the correct label while BERT failed to do so. A quantitative comparison between models is presented in table 2.
an ability to view raw text at a lexical semantic level. Figure 3(b) shows example sentences and their supersense prediction by the pretrained model. Where a vanilla BERT would see only the words of the sentence "Dan cooked a bass on the grill", SenseBERT would also have access to the supersense abstraction: "[Person] [created] [food] on the [artifact]". This sense-level perspective can help the model extract more knowledge from every training example, and to generalize semantically similar notions which do not share the same phrasing.
5 Lexical Semantics Experiments
In this section, we present quantitative evaluations of SenseBERT, pre-trained as described in section 3. We test the model's performance on a supersense-based variant of the SemEval WSD test sets standardized in Raganato et al. (2017), and on the Word in Context (WiC) task (Pilehvar and Camacho-Collados, 2019) (included in the recently introduced SuperGLUE benchmark (Wang et al., 2019)), both directly relying on the network's ability to perform lexical semantic categorization.
5.1 Comparing Rare Words Supersense Prediction Methods
We first report a comparison of the two methods described in section 3.4 for predicting the supersenses of rare words which do not appear in BERT's original vocabulary. The first  $60K$  vocabulary method enriches the vocabulary and the second average embedding method predicts a supersense from the average embeddings of the sub-word tokens com
prising an OOV word. During fine-tuning, when encountering an OOV word we predict the supersenses from the rightmost sub-word token in the 60K vocabulary method and from the average of the sub-word tokens in the average embedding method.
As shown in table 1, both methods perform comparably on the SemEval supersense disambiguation task (see following subsection), yielding an improvement over the baseline of learning supersense information only for whole words in BERT's original 30K-token vocabulary. We continue with the 60K-token vocabulary for the rest of the experiments, but note the average embedding option as a viable competitor for predicting word-level semantics.
5.2 SemEval-SS: Supersense Disambiguation
We test SenseBERT on a Word Supersense Disambiguation task, a coarse grained variant of the common WSD task. We use SemCor (Miller et al., 1993) as our training dataset (226,036 annotated examples), and the SenseEval (Edmonds and Cotton, 2001; Snyder and Palmer, 2004) / SemEval (Pradhan et al., 2007; Navigli et al., 2013; Moro and Navigli, 2015) suite for evaluation (overall 7253 annotated examples), following Raganato et al. (2017). For each word in both training and test sets, we change its fine-grained sense label to its corresponding WordNet supersense, and therefore train the network to predict a given word's supersense. We name this Supersense disambiguation task SemEval-SS. See figure 4(a) for an example
SemEval-SS Frozen	SemEval-SS Fine-tuned	Word in Context
BERTBASE	65.1	79.2	-
BERTLARGE	67.3	81.1	69.6
SenseBERTBASE	75.6	83.0	70.3
SenseBERTLARGE	79.5	83.7	72.1
Table 2: Results on a supersense variant of the SemEval WSD test set standardized in Raganato et al. (2017), which we denote SemEval-SS, and on the Word in Context (WiC) dataset (Pilehvar and Camacho-Collados, 2019) included in the recently introduced SuperGLUE benchmark (Wang et al., 2019). These tasks require a high level of lexical semantic understanding, as can be seen in the examples in figure 4. For both tasks, SenseBERT demonstrates a clear improvement over BERT in the regular fine-tuning setup, where network weights are modified during training on the task. Notably,  $\mathrm{SenseBERT}_{\mathrm{LARGE}}$  achieves state of the art performance on the WiC task. In the SemEval-SS Frozen setting, we train a linear classifier over pretrained embeddings, without changing the network weights. The results show that SenseBERT introduces a dramatic improvement in this setting, implying that its word-sense aware pre-training (section 3) yields embeddings that carries lexical semantic information which is easily extractable for the benefits of downstream tasks. Results for BERT on the SemEval-SS task are attained by employing the published pre-trained BERT models, and the  $\mathrm{BERT}_{\mathrm{LARGE}}$  result on WiC is taken from the baseline scores published on the SuperGLUE benchmark (Wang et al., 2019) (no result has been published for  $\mathrm{BERT}_{\mathrm{BASE}}$ ).
Word in Context
ELMo†	57.7
BERT sense embeddings ††	67.7
BERTLARGE‡	69.6
RoBERTa‡‡	69.9
KnowBERT-W+W◇	70.9
SenseBERT	72.1
Table 3: Test set results for the WiC dataset.
†Pilehvar and Camacho-Collados (2019)
†† Loureiro and Jorge (2019)
$\ddagger$ Wang et al. (2019)
$\ddagger \ddagger$  Liu et al. (2019)
Peters et al. (2019)
from this modified data set.
We show results on the SemEval-SS task for two different training schemes. In the first, we trained a linear classifier over the 'frozen' output embeddings of the examined model - we do not change the the trained SenseBERT's parameters in this scheme. This Frozen setting is a test for the amount of basic lexical semantics readily present in the pre-trained model, easily extricable by further downstream tasks (reminiscent of the semantic probes employed in Hewitt and Manning (2019); Reif et al. (2019).
In the second training scheme we fine-tuned the examined model on the task, allowing its parameters to change during training (see full training details in the supplementary materials). Results attained by employing this training method reflect
the model's potential to acquire word-supersense information given its pre-training.
Table 2 shows a comparison between vanilla BERT and SenseBERT on the supersense disambiguation task. Our semantic level pretraining signal clearly yields embeddings with enhanced word-meaning awareness, relative to embeddings trained with BERT's vanilla word-level signal. SenseBERT<sub>BASE</sub> improves the score of BERT<sub>BASE</sub> in the Frozen setting by over 10 points and SenseBERT<sub>LARGE</sub> improves that of BERT<sub>LARGE</sub> by over 12 points, demonstrating competitive results even without fine-tuning. In the setting of model fine-tuning, we see a clear demonstration of the model's ability to learn word-level semantics, as SenseBERT<sub>BASE</sub> surpasses the score of BERT<sub>LARGE</sub> by 2 points.
5.3 Word in Context (WiC) Task
We test our model on the recently introduced WiC binary classification task. Each instance in WiC has a target word  $w$  for which two contexts are provided, each invoking a specific meaning of  $w$ . The task is to determine whether the occurrences of  $w$  in the two contexts share the same meaning or not, clearly requiring an ability to identify the word's semantic category. The WiC task is defined over supersenses (Pilehvar and Camacho-Collados, 2019) – the negative examples include a word used in two different supersenses and the positive ones include a word used in the same supersense. See figure 4(b) for an example from this data set.
Score	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE
BERTBASE (OURS)	77.5	50.1	92.6	88.7/84.3	85.7/84.6	71.0/88.9	83.6	89.4	67.9
SenseBERTBASE	77.9	54.6	92.2	89.2/85.2	83.5/82.3	70.3/88.8	83.6	90.6	67.5
Table 4: Results on the GLUE benchmark test set.
Results on the WiC task comparing SenseBERT to vanilla BERT are shown in table 2. SenseBERT<sub>BASE</sub> surpasses a larger vanilla model, BERT<sub>LARGE</sub>. As shown in table 3, a single SenseBERT<sub>LARGE</sub> model achieves the state of the art score in this task, demonstrating unprecedented lexical semantic awareness.
5.4 GLUE
The General Language Understanding Evaluation (GLUE; Wang et al. (2018)) benchmark is a popular testbed for language understanding models. It consists of 9 different NLP tasks, covering different linguistic phenomena. We evaluate our model on GLUE, in order to verify that SenseBERT gains its lexical semantic knowledge without compromising performance on other downstream tasks. Due to slight differences in the data used for pretraining BERT and SenseBERT (BookCorpus is not publicly available), we trained a  $\mathrm{BERT}_{\mathrm{BASE}}$  model with the same data used for our models.  $\mathrm{BERT}_{\mathrm{BASE}}$  and SenseBERT were both finetuned using the exact same procedures and hyperparameters. The results are presented in table 4. Indeed, SenseBERT performs on par with BERT, achieving an overall score of 77.9, compared to 77.5 achieved by  $\mathrm{BERT}_{\mathrm{BASE}}$ .
We introduce lexical semantic information into a neural language model's pre-training objective. This results in a boosted word-level semantic awareness of the resultant model, named SenseBERT, which considerably outperforms a vanilla BERT on a SemEval based Supersense Disambiguation task and achieves state of the art results on the Word in Context task. This improvement was obtained without human annotation, but rather by harnessing an external linguistic knowledge source. Our work indicates that semantic signals extending beyond the lexical level can be similarly introduced at the pre-training stage, allowing the network to elicit further insight without human supervision.
We acknowledge useful comments and assistance from our colleagues at AI21 Labs. We would also like to thank the anonymous reviewers for their valuable feedback.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495.
Pierpaolo Basile. 2012. Super-sense tagging using support vector machines and distributional features. In International Workshop on Evaluation of Natural Language and Speech Tool for Italian, pages 176-185. Springer.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019. KERMIT: Generative insertion-based modeling for sequences. arXiv preprint arXiv:1906.01604.
Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1025-1035, Doha, Qatar. Association for Computational Linguistics.
Massimiliano Ciaramita and Mark Johnson. 2003. Supersense tagging of unknown nouns in WordNet. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 168-175.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1-5, Toulouse, France. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 897-907, Berlin, Germany. Association for Computational Linguistics.
Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In ICLR.
Adam Kilgarriff. 1997. I don't believe in word senses. Computers and the Humanities, 31(2):91-113.
Minh Le, Marten Postma, Jacopo Urbani, and Piek Vossen. 2018. A deep dive into word sense disambiguation with LSTM. In Proceedings of the 27th International Conference on Computational Linguistics, pages 354-365, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111-3119. Curran Associates, Inc.
George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993.
Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288-297, Denver, Colorado. Association for Computational Linguistics.
Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41(2).
Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 222-231, Atlanta, Georgia, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87-92, Prague, Czech Republic. Association for Computational Linguistics.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings
of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99-110, Valencia, Spain. Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems 32, pages 8594-8603. Curran Associates, Inc.
Sascha Rothe and Hinrich Schütze. 2015. AutoExtend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1793-1803, Beijing, China. Association for Computational Linguistics.
Nathan Schneider. 2014. Lexical semantic analysis in natural language text. Unpublished Doctoral Dissertation, Carnegie Mellon University.
Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1537-1547, Denver, Colorado. Association for Computational Linguistics.
Benjamin Snyder and Martha Palmer. 2004. The English all-words task. In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41-43, Barcelona, Spain. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information
Processing Systems 32, pages 3266-3280. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1374-1385, Osaka, Japan. The COLING 2016 Organizing Committee.
A Supersenses and Their Representation in SenseBERT
We present in table 5 a comprehensive list of WordNet supersenses, as they appear in the WordNet documentation. In fig. 5 we present a Dendrogram of an Agglomerative hierarchical clustering over the supersense embedding vectors learned by SenseBERT in pre-training. The clustering shows a clear separation between Noun senses and Verb senses. Furthermore, we can observe that semantically related supersenses are clustered together (i.e, noun.animal and noun.plant).
As hyperparameters for the fine-tuning, we used max_seq_length = 128, chose learning rates from  $\{5e - 6, 1e - 5, 2e - 5, 3e - 5, 5e - 5\}$ , batch sizes from  $\{16, 32\}$ , and fine-tuned up to 10 epochs for all the datasets.
Figure 5: Dendrogram visualization of an Agglomerative hierarchical clustering over the supersense vectors (rows of the classifier S) learned by SenseBERT.
Name	Content	Name	Content
adj.all	All adjective clusters	nounquantity	Nouns denoting quantities and units of measure
adj.pert	Relational adjectives (pertainyms)	noun.relation	Nouns denoting relations between people or things or ideas
adv.all	All adverbs	noun.shape	Nouns denoting two and three dimensional shapes
noun.Tops	Unique beginner for nouns	noun.state	Nouns denoting stable states of affairs
noun.act	Nouns denoting acts or actions	noun.substance	Nouns denoting substances
noun animal	Nouns denoting animals	noun.time	Nouns denoting time and temporal relations
noun.artifact	Nouns denoting man-made objects	verb.body	Verbs of grooming, dressing and bodily care
noun(attribute)	Nouns denoting attributes of people and objects	verb/change	Verbs of size, temperature change, intensifying, etc.
noun.body	Nouns denoting body parts	verb.cognition	Verbs of thinking, judging, analyzing, doubting
noun.cognition	Nouns denoting cognitive processes and contents	verbcommunications	Verbs of telling, asking, ordering, singing
noun.communication	Nouns denoting communicative processes and contents	verb.competition	Verbs of fighting, athletic activities
noun.event	Nouns denoting natural events	verbconsumption	Verbs of eating and drinking
noun feelsing	Nouns denoting feelings and emotions	verb(contact	Verbs of touching, hitting, tying, digging
noun.food	Nouns denoting foods and drinks	verb creation	Verbs of sewing, baking, painting, performing
noun.group	Nouns denoting groupings of people or objects	verb.emotion	Verbs of feeling
noun.location	Nouns denoting spatial position	verb.motion	Verbs of walking, flying, swimming
noun.motive	Nouns denoting goals	verb.perception	Verbs of seeing, hearing, feeling
noun.object	Nouns denoting natural objects (not man-made)	verb.possession	Verbs of buying, selling, owning
noun.person	Nouns denoting people	verb/social	Verbs of political and social activities and events
noun.phenomenon	Nouns denoting natural phenomena	verb.stative	Verbs of being, having, spatial relations
noun.plant	Nouns denoting plants	verbweather	Verbs of raining, snowing, thawing, thundering
noun.possession	Nouns denoting possession and transfer of possession	adj.ppl	Participial adjectives
noun-process	Nouns denoting natural processes
Table 5: A list of supersense categories from WordNet lexicographer.