SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1: Comparison of F1 scores for the all-words WSD task. EWISER uses SemEval-2015 (SE15) as development set.  $\dagger$  indicates systems having access to sense definitions.  $\ddagger$  indicates systems having access to sense definitions utilizing synset embeddings.  $\dagger \dagger$  indicates systems using a nearest-neighbor approach based on synset embeddings. The difference in performance attained on the ALL test between Escher and BEM (underlined) is statistically significant with  $p\ll 0.01$  according to the McNemar's test (Dietterich, 1998).
Model	MFS	LFS	0-lex	0-lex-def	0-def
BEM	94.7	52.1	91.2	67.1	68.2
ESCHERNo-GN	93.7	52.8	94.5	74.3	76.4
ESCHER	93.7	55.7	95.1	75.0	76.8
Table 2: Comparison of EsCHER against its competitors on MFS, LFS and zero-shot datasets.
one system for each inventory. However, merging distinct lexicographic resources is not a straightforward task and requires its own complex pipeline. An easier approach could be to concatenate datasets tagged with different vocabularies, which, nonetheless, would expose models to possibly different definitions for nearly identical meanings and to different levels of sense granularity. In this Section we therefore investigate the ability of EsCHER to manage data annotated with distinct sense inventories when simply joining them. To this end, we train EsCHER on the concatenation of SemCor and the Oxford Dictionary dataset (Chang et al., 2018) and compare its performance with the state-of-the-art system at the moment of writing, i.e. BEM, when trained on the same corpus.
5.1 The Oxford Dictionary Dataset
Chang et al. (2018) introduced a dataset containing roughly 785,000 instances for as many sentences and covering 79,004 senses of the Oxford Dictionary of English. The dataset is split
Dataset	Polysemy	Exp. Polysemy	#Senses	#Instances
WordNet	SemCor	6.88	0.76	33,362	226,036
SE07	8.48	0.29	375	455
ALL	5.87	0.54	3,669	7,611
Oxford	Oxfordtrain	3.81	0.98	79,105	555,695
Oxforddev	6.69	0.68	33,197	78,550
Oxfordtest	6.79	0.76	37,714	151,306
into train (Oxfordtrain), dev (Oxforddev) and test $^{12}$  (Oxfordtest). In Table 3 we report its statistics together with those of the training (SemCor), development (SE07) and test (ALL) sets of the standard evaluation suite. Specifically, we show the average polysemy of each dataset (Polysemy), the expressed polysemy (Exp. Polysemy), i.e., for each lexeme we compute the number of senses that appear in the dataset over the number of possible senses it can assume in the reference vocabulary and we average across all lexemes, the number of distinct senses (#Senses) and the number of instances (#Instances). As one can see, Oxfordtrain contains more than two times the instances and senses of SemCor, while having roughly half of SemCor's polysemy but a higher expressed polysemy. As for Oxfordtest, it contains a larger number of instances than ALL, and also a higher polysemy and expressed polysemy.
We analyze three different scenarios: i) Standard, where the system is trained on the same inventory with which it is tested, e.g., trained on Oxfordtrain and tested on Oxfordtest; ii) Zero-shot, in which the system is trained on one sense inventory and tested on the other, e.g., trained on SemCor and tested on Oxfordtest; and iii) Joint, in which the system is jointly trained with the two sense inventories. In order to combine the two different inventories, we train the model by alternating the batches made up of either SemCor or Oxfordtrain instances. Since the number of instances in SemCor is lower than that in Oxfordtrain, we oversample SemCor by repeating its instances. Finally, we select the model with the best macro F1 averaged on the two validation datasets (SE07 and Oxforddev). We add the subscript  $S$ ,  $OT$  and  $S + OT$  to models trained on SemCor, Oxfordtrain and their concatenation,
Table 3: Statistics for training, development and test corpora annotated with two inventories: WordNet (top) and Oxford (bottom).
Model	SE07	ALL	OXdev	OXtest
BEMS	74.5	79.0	61.5	61.7
ESCHERS	76.3	80.7	67.6	67.9
BEMOT	56.9	67.2	84.2	84.3
ESCHEROT	60.7	70.3	86.3	86.3
BEMS+OT	74.9	78.8	85.0	85.2
ESCHERS+OT	77.8	81.5	87.6	87.7
Table 4: Comparison of EsCHER and BEM when using different training sets, i.e., SemCor  $(\mathbf{B}\mathbf{E}\mathbf{M}_S$  and  $\mathrm{ESCHER}_S)$  , Oxfordtrain  $(\mathbf{B}\mathbf{E}\mathbf{M}_{OT}$  and  $\mathrm{ESCHER}_{OT})$  and their concatenation  $(\mathbf{B}\mathbf{E}\mathbf{M}_{s + OT}$  and  $\mathrm{ESCHER}_{S + OT})$
As one can see from Table 4, EsCHER outperforms BEM in all settings. That is, when trained with one inventory and tested on a dataset tagged with the other inventory  $(\mathrm{BEM}_S$  and  $\mathrm{ESCHERS}$  on the Oxford test and  $\mathrm{BEM}_{OT}$  and  $\mathrm{ESCHEROT}$  on ALL), EsCHER attains 6 and 3 points higher performance, respectively, than its competitor. This result is not important per se, but it also suggests that ESC does not bind the model to a single lexical knowledge base. Indeed, by extensively leveraging sense definitions, it allows a transformer-based model to scale on multiple inventories as long as they provide at least one definition for each meaning. BEM, instead, by encoding each gloss independently, falls short in representing definitions that were previously unseen, as also shown in Section 4.2.
When trained on SemCor and Oxfordtrain together, not only can EsCHER handle the two inventories that coexist in the training set effectively, but it also leverages them at its own convenience, achieving 81.5 F1 points on ALL, in contrast to BEM which performs slightly worse than when trained in the Standard scenario.
6 Few-Shot Evaluation
We now move to analyzing the performances of EsCHER in a few-shot scenario, i.e., when the number of samples available for each sense is limited.
Setting We compare EsCHER against BEM, and report the F1 scores on the ALL dataset when varying the number  $k$  of training instances per sense in  $\{1,3,5,10,\text{unlimited}\}$ . We show in Table 5 the
12We refer to the one named test_easy in the original paper.
(a) ALL dataset.
(b) LFS and 0-lex-def datasets.
Figure 2: (a) F1 Performances of BEM and EsCHER on the ALL evaluation dataset, when varying the number of instances per sense seen during training. (b) Performance of EsCHER on the MFS, LFS and 0-lex-def datasets, when varying the number of instances per sense seen during training.
k	Instances
1	33,206
3	64,814
5	83,068
10	109,751
unlimited	226,036
Table 5: Number of instances in the training set at different values of  $k$  .
number of instances drawn from SemCor that are seen at training time for each  $k$ .
We also report the F1 scores of EsCHER on the MFS, LFS and 0-lex-def datasets in the same scenario in order to investigate the extent to which the difference in the number of occurrences for each sense impacts the ability of the model to generalize on rare senses.
Results As one can see from Figure 2a, $^{13}$  EsCHER makes much more efficient use of training data than BEM, needing roughly one third of the instances to attain the same results. In fact, BEM needs more than 5 instances per sense (83,068 instances) to reach the same performance (73.9 F1 points) as that of EsCHER trained with  $k = 1$  (33,206 instances). Furthermore, with roughly half of the instances ( $k = 10$ ) EsCHER attains results that are in the same ballpark as the current state of the art. Interestingly enough, by looking at Fig-
ure 2b, we see that EsCHER's accuracy on the MFS instances rises when adding more examples. This is due to the fact that frequent senses get increasingly represented within the training set, therefore better matching the sense distribution in the test set. Similarly, the performance on the Least Frequent Senses also rises from  $k = 1$  to  $k = 10$ , but slightly drops when considering the whole dataset. By manually inspecting the data we notice that this happens because most of the instances added to the dataset with  $k = 10$  are tagged with the most frequent sense, therefore drastically skewing the sense distribution. Finally, the performance on 0-lex-def remains stable for all  $k$ , hence showing that, despite increasingly skewing the distribution towards the most frequent definitions, our approach can still provide meaningful representations for unseen senses.[14]
In order to get a clear picture of the model's pitfalls and gain insights into possible directions for future work, we perform an analysis of EsCHER misclassifications on the ALL dataset. We find that the mistaken predictions belong to three main categories: most frequent sense bias, insufficient context and WordNet sense granularity. Since we already discussed the first of these in the previous sections, we focus here on the latter two.
$^{13}$ BEM chart from the original paper.
<sup>14</sup>We recall from Section 4.1 that we indicate as a sense a pair (lexeme, definition).
Insufficient Context Annotators often compiled the WSD evaluation datasets by considering each instance in the context of the documents they appear in. In contrast, WSD models typically take into account only the sentence surrounding the target word, discarding a large portion of the available context. This behavior causes a discrepancy where sentences do not provide enough information to disambiguate the target words therein. Indeed, EsCHER mistakes most often appear in sentences with an average length of 27 tokens, i.e., roughly 5 tokens less than the average length in ALL (32).
This suggests that moving the disambiguation context from sentences to documents may improve the performances of models as long as they are capable of handling longer sequences.
WordNet Sense Granularity The granularity of WordNet senses has been considered one of the main reasons behind the complexity of the WSD task (Palmer et al., 2007). To measure the extent to which this affects EsCHER's performance, we utilize the 45 domain-based labels introduced by Lacerra et al. (2020, CSI), which define macro categories for each WordNet sense. For instance, in the CSI inventory, the sense argument%1:10:03:: belongs to the following domains: Culture Anthropology and Society, Language and Linguistics and Communication and Telecommunication.
To better understand the relation between EsCHER predictions and the gold annotations, for each misclassified instance in ALL, we compute the average Jaccard similarity between the CSI labels assigned to the gold annotation of that instance and those assigned to the sense predicted by EsCHER. As an example, EsCHER misclassified an instance annotated with the sense argument%1:10:03::, assigning to it the sense argument%1:10:00::. Examining the domains to which the predicted sense belongs, we can see a considerable overlap (and consequently a high Jaccard similarity) with the domains of the gold sense (i.e. argument%1:10:03::): Culture Anthropology and Society, Politics Government and Nobility, Language and Linguistics and Communication and Telecommunication.
As a term of comparison, we repeat the same procedure when considering a random baseline as WSD model, i.e., one that predicts for each instance a random sense among those of the target word. We
find that EsCHER predictions have an average Jaccard similarity with the gold predictions of 0.49, whereas the random baseline achieves 0.27. This suggests that, even when providing a formally mistaken output, EsCHER still predicts a sense that is correlated, according to CSI labels, to the gold sense. Our analysis calls for further work to improve evaluation in WSD as the F1 score cannot discriminate between predictions that are clearly wrong and predictions that are just slightly different from the gold sense.
In this paper, we introduced a novel framing for the Word Sense Disambiguation problem inspired by the Extractive Reading Comprehension task in QA: given a word in a sentence and a text containing all its possible definitions, a model has to identify the span containing the correct definition for the target word. For this new formulation — which we called Extractive Sense Comprehension (ESC) — we devised a transformer-based architecture (ESCHER), which, differently from previous approaches, can look at all the target word definitions at once, alongside the input sentence. EsCHER surpasses the current state of the art by 1.7 points on the standard English all-words WSD task, thanks to its more efficient use of the training data. Also, when provided with only a few examples for each sense, EsCHER attains remarkable levels of performance, requiring roughly three times less annotated instances than its direct competitor to reach the same performances. Furthermore, our new formulation allows EsCHER to scale across different inventories and to combine them effectively. Indeed, when provided with data annotated with multiple vocabularies, it achieves even better results than when limited to one inventory only, with results in the  $86 - 88\%$  range.
As future work we plan to expand this framework so as to condition the prediction not only on the target word context and definitions, but also on the possible senses of its surrounding words.
The pretrained model, along with code and data, is available at https://github.com/SapienzaNLP/esc.
The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487.
This work was supported in part by the MIUR under grant "Dipartimenti di eccellenza 2018-2022" of the Department of Computer Science of Sapienza University.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or: "how we went beyond word sense inventories and learned to gloss". In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7207-7221, Online. Association for Computational Linguistics.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss informed bi-encoders. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1006–1017, Online. Association for Computational Linguistics.
Ting-Yun Chang, Ta-Chung Chi, Shang-Chi Tsai, and Yun-Nung Chen. 2018. xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks. arXiv preprint arXiv:1809.03348.
Simone Conia and Roberto Navigli. 2021. Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Claudio Delli Bovi, Luca Telesca, and Roberto Navigli. 2015. Large-scale information extraction from textual definitions through deep syntactic and semantic analysis. Transactions of the Association for Computational Linguistics, 3:529-543.
Thomas G Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. *Neural computation*, 10(7):1895-1923.
Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating
Word Sense Disambiguation Systems, pages 1-5, Toulouse, France. Association for Computational Linguistics.
Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan. 2019. Improved word sense disambiguation using pre-trained contextualized word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5297-5306, Hong Kong, China. Association for Computational Linguistics.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3509-3514, Hong Kong, China. Association for Computational Linguistics.
Adam Kilgarriff. 2004. How dominant is the commonest sense of a word? In Text, Speech and Dialogue, pages 103-111, Berlin, Heidelberg. Springer Berlin Heidelberg.
Sawan Kumar, Sharmistha Jat, Karan Saxena, and Partha Talukdar. 2019. Zero-shot word sense disambiguation using sense definition embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5670-5681, Florence, Italy. Association for Computational Linguistics.
Caterina Lacerra, Michele Bevilacqua, Tommaso Pasini, and Roberto Navigli. 2020. CSI: A coarse sense inventory for  $85\%$  word sense disambiguation. In Proceedings of the 34th Conference on Artificial Intelligence, pages 8123-8130. AAAI Press.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Frederick Liu, Han Lu, and Graham Neubig. 2018. Handling homographs in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1336-1345, New Orleans, Louisiana. Association for Computational Linguistics.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate
and beyond. In International Conference on Learning Representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018. Incorporating glosses into neural word sense disambiguation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2473-2482, Melbourne, Australia. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, G.s Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26:3111-3119.
George A. Miller, R.T. Beckwith, Christiane D. Fellbaum, D. Gross, and K. Miller. 1990. Introduction to WordNet: an online lexical database. International Journal of Lexicography, 3(4):235-244.
George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Proceedings of the workshop on Human Language Technology, pages 303-308. Association for Computational Linguistics.
Andrea Moro and Roberto Navigli. 2013. Integrating syntactic and semantic analysis into the open information extraction paradigm. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI), pages 2148-2154. IJCAI/AAAI.
Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288-297, Denver, Colorado. Association for Computational Linguistics.
Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM computing surveys (CSUR), 41(2):1-69.
Roberto Navigli. 2018. Natural Language Understanding: Instructions for (present and future) use. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pages 5697-5702.
Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 222-231, Atlanta, Georgia, USA. Association for Computational Linguistics.
Martha Palmer, Hoa Trang Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Nat. Lang. Eng., 13(2):137-163.
Tommaso Pasini. 2020. The knowledge acquisition bottleneck problem in multilingual word sense disambiguation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4936-4942. International Joint Conferences on Artificial Intelligence Organization. Survey track.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87-92, Prague, Czech Republic. Association for Computational Linguistics.
Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. 2018. Integrating weakly supervised word sense disambiguation into neural machine translation. Transactions of the Association for Computational Linguistics, 6:635-649.
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017a. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99-110, Valencia, Spain. Association for Computational Linguistics.
Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017b. Neural sequence learning models for word sense disambiguation. In Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, pages 1156-1167, Copenhagen, Denmark. Association for Computational Linguistics.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020. With more contexts comes better performance: Contextualized sense embeddings for all-round word sense disambiguation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3528-3539, Online. Association for Computational Linguistics.
Benjamin Snyder and Martha Palmer. 2004. The English all-words task. In Proceedings of SENSEVAL3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41-43, Barcelona, Spain. Association for Computational Linguistics.