SlowGuess/ACL_2020_OCR huggingface dataset:
Evaluation Datasets We consider all the standard English all-words WSD test sets contained in the framework presented by Raganato et al. (2017b), i.e., Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-2007 (Pradhan et al., 2007), SemEval-2013 (Navigli et al., 2013), SemEval-2015 (Moro and Navigli, 2015) and ALL, i.e., the concatenation of all the previous datasets. As regards the domain-specific evaluation we consider the 6 and 3 domains in SemEval-2013 and SemEval-2015, respectively, and test on each of them separately. As for the multilingual evaluation, instead, we test on the Italian, Spanish, French and German datasets of SemEval-2013 and the Italian and Spanish test sets of SemEval-2015.
We note that both datasets make use of old versions of BabelNet (version 1.1.1 and 2.5, respectively). For this reason, previous works used an in-house mapping between BabelNet versions to make them up to date. However, in this process, several gold instances were lost making the datasets smaller than the original ones. To be fair with other approaches, we compare CluBERT against them on the same datasets on which they tested. Moreover, to encourage future comparisons, we also report CluBERT's performance on the newer versions of both gold standards made available by the Sapienza NLP group at https://github.com/SapienzaNLP/mwsd-datasets, which comprise more instances than the older datasets and feature the latest version of BabelNet  $(4.0.1)^{10}$ . As
<sup>10</sup>We used the WordNet split as we can only provide senses within the WordNet part of BabelNet.
https://en.wikipedia.org/wiki/FCFlora 9https://en.wikipedia.org/wiki/ William_Flora
Method	Senseval2	Senseval3	SemEval-2007	SemEval-2013	SemEval-2015	All
CluBERT	68.3	64.6	55.4	69.7	68.0	66.8
UMFS-WE	54.8	52.0	38.2	55.2	54.5	53.1
WCT-VEC	56.4	53.8	40.6	54.9	54.0	54.1
COMP2SENSE	51.5	47.0	37.5	54.2	55.0	50.7
WordNet MFS	67.0	66.0	55.0	63.0	68.0	65.0
Table 4: MFS performance in terms of F1 on all the instances of the test sets in Raganato et al. (2017b). Statistically-significant differences on the ALL dataset between CluBERT and WordNet MFS are underlined.
Method	Precision	Recall	F1
CluBERT	70.9	70.2	70.6
EnDi	66.0	66.0	66.0
DaD	61.0	61.0	61.0
LexSemTM	51.0	48.0	49.0
WordNet MFS	68.0	68.0	68.0
Table 5: MFS performance in terms of Precision, Recall and F1 on the nominal instances of the ALL test set from Raganato et al. (2017b).
a term of comparison, we also report the results of the BabelNet MFS on these datasets. In what follows, we refer to the older versions of the multilingual tasks of SemEval-2013 and SemEval-2015 by juxtaposing the “*” symbol (SemEval-2013* and SemEval-2015*).
On all the aforementioned datasets we report the results in terms of F1, i.e., the harmonic mean of precision and recall.
Most Frequent Sense Strategy We extract the MFS of a target lemma  $l$  from its sense distribution  $d_{l}$  by taking the synset with the highest probability, i.e.,  $MFS(l) = \arg\max(d_{l})$ . Therefore, we use the MFS of a lemma computed according to each system under comparison to tag all of  $l$ 's occurrences within the test sets.
Domain-Specific WSD Setup To assess the ability of CluBERT to scale over different domains and hence to extract a distribution that is skewed towards the topic of the input corpus, we build 8 distinct domain-specific corpora, one for each domain of SemEval-2013 and SemEval-2015's English datasets. For this purpose, we exploit the 34 domain labels (Camacho-Collados and Navigli, 2017) available in BabelNet together with the mapping between synsets and Wikipedia pages to retrieve those pages that are peculiar to a specific domain, hence building a corpus  $\mathcal{C}_{dom}$  specific for
the domain  $dom$ . We then apply CluBERT, EnDi, DaD and LexSemTM on  $\mathcal{C}_{dom}$  and extract their respective MFS specific for each domain<sup>11</sup>.
Downstream Task Setup Finally, we test the benefits brought by CluBERT's distributions by including them in a knowledge-based and a supervised approach, namely:
- UKB $^{12}$  (Agirre et al., 2014): an off-the-shelf state-of-the-art knowledge-based WSD model based on the Personalised PageRank algorithm. When provided, it makes use of the given sense distribution to bias its answers towards the MFS.
- BiLSTM (Raganato et al., 2017a): an end-to-end neural sequence model which employs two bidirectional LSTM layers and an attention mechanism trained on multiple tasks, i.e., fine- and coarse-grained WSD and Part-of-Speech tagging. When provided, it makes use of the MFS backoff strategy whenever it comes to disambiguating a lemma unseen during training.
We compare these two models, firstly, when no prior knowledge is supplied, and then, when WordNet (UKB $_{WN}$ , BiLSTM $_{WN}$ ) and CluBERT (UKB $_{CluBERT}$ , BiLSTM $_{CluBERT}$ ) distributions are provided.
6.1 English WSD Results
As one can see from Table 4, CluBERT attains the highest scores across the board, outperforming all the other automatic approaches by more than 10 F1 points. More interestingly, CluBERT surpasses the hitherto unbeaten manual baseline of WordNet by
<sup>11</sup>We do not compare against UMFS-WE, WCT-VEC and COMP2SENSE inasmuch as code and data are not available.
12Version 3.2 available at http://ixa2.si.ehu.es/ukb/
Method	SemEval-2013	SemEval-2015
Biology	Climate	Finance	Politics	Social Issue	Sport	Math&Computer	Biomedicine	Social Issue
CluBERT	72.9	70.9	69.0	79.2	70.9	61.4	52.3	77.3	75.2
DaD	79.0	63.0	64.0	67.0	68.0	54.0	59.8	63.9	54.3
EnDi	71.0	53.0	60.0	62.0	63.0	57.0	63.0	63.0	55.9
LexSemTM	56.0	47.0	49.0	51.0	52.0	34.0	47.7	63.0	40.7
WordNet MFS	61.0	59.0	52.0	64.0	58.0	56.0	47.2	67.8	62.4
Table 6: MFS performance in terms of F1 on the nominal instances of the different domains in the SemEval-2013 (Navigli et al., 2013) and SemEval-2015 test sets (Moro and Navigli, 2015).
Method	SemEval-2013*	SemEval-2015*
IT	ES	DE	FR	IT	ES
CluBERT	71.7	68.7	69.1	67.1	70.4	68.8
DaD	62.9	58.9	65.5	54.3	61.0	58.0
EnDi	46.2	44.6	49.1	54.3	55.0	52.0
BabelNet MFS	52.3	55.6	49.3	55.1	52.0	53.0
Table 7: MFS F1 scores on the nominal instances of the SemEval-2013* and SemEval-2015* multilingual datasets. Statistically significant differences (at  $\chi^2$  test) with  $p < 0.01$  between CluBERT and the second best performing model are underlined.
a statistically-significant $^{13}$  difference (McNemar, 1947) of almost 2 F1 points on the ALL dataset. In order to set a level playing field with EnDi and DaD, which cover nouns only, we also carried out our evaluation on the ALL dataset focusing on its nominal instances. As shown in Table 5, CluBERT attains an F1 score of 70.6, surpassing the best automatic competitor, i.e., DaD, by more than 4 F1 points. More importantly, our induced distributions also outperform the well-known WordNet MFS strategy by 2.6 F1 points in this setting too.
This demonstrates that CluBERT's distributions are of higher quality than those induced by any of the other automatic and manual competitors.
6.2 Domain-Specific WSD Results
We now focus on testing our distributions on the domain-specific documents available in the SemEval-2013 and SemEval-2015 WSD test sets. As shown in Table 6, CluBERT outperforms all the other competitors on 7 out of the 9 domains by several points, falling behind DaD on the Biology domain and behind EnDi on the Math&Computer one. This is mainly due to the fact that the senses in these two domains are poorly connected in BabelNet, hence making them hard to reach when applying the PPR algorithm (see Section 3.2). DaD, which also exploits the BabelNet graph, seems to
be more robust to this event inasmuch as it relies directly on the connections between domains and synsets and not only on those between words and concepts, as CluBERT does. Nevertheless, when the senses of the target domain are well framed within the semantic network, our approach proves to be able to induce a distribution that accurately reflects the way the meanings of a word are spread within the input corpus. In fact, CluBERT achieves the best results on all the other domains, with the highest improvement of 12.2 F1 points over the current state of the art on the Politics domain of SemEval-2013.
WordNet, instead, shows poor performance in this setting, too. In fact, its MFS information is designed to work on a general domain setting and it cannot be customised easily for other scenarios. All these results further corroborate our findings in the intrinsic evaluation, and they highlight the fact that WordNet distributions no longer reflect the way senses are spread across a corpus.
6.3 Multilingual WSD Results
We now investigate the capabilities of CluBERT to scale over different languages by evaluating it on the multilingual Word Sense Disambiguation tasks of SemEval-2013* and SemEval-2015*. As can be seen from Table 7, the differences in results between CluBERT and the other systems under comparison remain consistent with those reported for English. Our approach, in fact, achieves on average a significant improvement of approximately 9 F1 points over the existing state of the art. This demonstrates that CluBERT makes efficient use of its two complementary resources, i.e., BabelNet and BERT, in this way making up for the paucity of data in non-English languages. Conversely, EnDi and DaD suffer from this shortcoming and perform either poorly (EnDi), or not consistently across languages (DaD). As for the performance on the newer versions of the datasets (Table 8), we note
${}^{13}\chi^2$  test for statistical significance with  $p < 0.05$
Method	SemEval-2013	SemEval-2015
IT	ES	DE	FR	IT	ES
CluBERT	66.6	69.5	72.3	62.3	62.8	61.5
BabelNet MFS	53.2	60.3	76.6	60.0	54.2	50.1
that CluBERT outperforms the BabelNet MFS on all languages but German. The drop in performance on SemEval-2015 when compared to the older version of the dataset, is mainly due to the fact that the datasets now also include all the non-nominal instances which were excluded before to be fair with the other competitors. As for future comparisons, we highly encourage the community to consider the results in Table 8 for CluBERT as they are computed on larger and more updated versions of the datasets.
6.4 Downstream Task Results
Finally, we assess CluBERT MFS effectiveness when used as backoff strategy in two off-the-shelf WSD approaches, i.e., UKB and the BiLSTM with attention model presented by Raganato et al. (2017b) (see Section 6). In Table 9 we report the performance of the two models without MFS, with WordNet MFS and with CluBERT MFS on the ALL WSD dataset. As one can see, not only does our MFS provide a large boost of 4.6 and 5.2 F1 points when compared with the base models without backoff strategy, but it also leads the two systems to attain better performance than when using the WordNet MFS. This strengthens our previous findings and crowns CluBERT as the best backoff strategy compared to all its alternatives.
These results open up to new scenarios where the CluBERT MFS might be preferred as backoff strategy for WSD models to the well-established WordNet MFS. In fact, CluBERT attains higher results than WordNet on several WSD datasets, while at the same time assuring greater flexibility. In fact, whereas WordNet MFS is static, CluBERT can be run on different corpora and can therefore adapt the sense distributions to various circumstances and different languages.
Table 8: MFS F1 scores on all instances of the WordNet split of SemEval-2013 and SemEval-2015 multilingual datasets mapped to the latest BabelNet version (4.0.1). Data available at https://github.com/SapienzaNLP/mwsd-datasets.
Method	Precision	Recall	F1
UKB	63.1	63.1	63.1
UKBWN	67.1	67.1	67.1
UKBClubERT	67.7	67.7	67.7
BiLSTM	68.1	61.6	64.7
BiLSTMWN	69.6	69.6	69.6
BiLSTMCluBERT	69.9	69.9	69.9
Table 9: UKB and BiLSTM Precision, Recall and F1 with and without the MFS backoff strategy on the ALL test set in Raganato et al. (2017b).
In this paper we presented CluBERT, an automatic multilingual approach which induces the distribution of word senses in an arbitrary input corpus by exploiting the contextual information coming from BERT and the lexical-semantic knowledge available in BabelNet. CluBERT attains state-of-the-art results on both intrinsic and extrinsic evaluations, also beating the widely-used and manually-curated WordNet MFS.
When considering input corpora that come from specific domains, CluBERT showed an unmatched nimbleness in shaping the distributions accordingly, hence outperforming its manual and automatic competitors on most domains. Similarly, our approach demonstrated its ability to scale well on different languages, attaining state-of-the-art results on the multilingual WSD tasks. Finally, when injecting CluBERT MFS into off-the-shelf WSD models, we showed that it brings greater benefits than the WordNet MFS. We release the sense distributions in five different languages at https://github.com/SapienzaNLP/clubert.
As future work, we plan to refine our approach by exploiting other strategies for weighting the words in the clusters and to leverage them for automatically building multilingual sense-tagged corpora.
The authors wish to greatly thank Claudio Delli Bovi for his comments, suggestions and late-night discussions on the manuscript.
The authors gratefully acknowledge the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union's Horizon 2020 research and innovation programme.
Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, 40(1):57-84.
Andrew Bennett, Timothy Baldwin, Joy Han Lau, Diana McCarthy, and Francis Bond. 2016. LexSemTm: A Semantic Dataset Based on All-words Unsupervised Sense Distribution Learning. In Proc. of ACL, pages 1513 - 1524.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking the  $80\%$  Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information. In Proc. of ACL.
Sudha Bhingardive, Dhirendra Singh, V Rudramurthy, Hanumant Redkar, and Pushpak Bhattacharyya. 2015. Unsupervised Most Frequent Sense Detection using Word Embeddings. In Proc. of NAACL, pages 1238-1243.
Jose Camacho-Collados and Roberto Navigli. 2017. BabelDomains: Large-Scale Domain Labeling of Lexical Resources. In Proc. of ACL, volume 2, pages 223-228.
Claudio Delli Bovi, Luis Espinosa Anke, and Roberto Navigli. 2015. Knowledge Base Unification via Sense Embeddings and Disambiguation. In Proc. of EMNLP, pages 726-736.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, pages 4171-4186.
Philip Edmonds and Scott Cotton. 2001. Senseval-2: overview. In Proc. of Senseval-2, pages 1-5.
William A. Gale, Kenneth Church, and David Yarowsky. 1992. A method for disambiguating word senses in a corpus. Computers and the Humanities, 26:415-439.
Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan. 2019. Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations. In Proc. of EMNLP-IJCNLP, pages 5296-5305.
Bradley Hauer, Yixing Luan, and Grzegorz Kondrak. 2019. You Shall Know the Most Frequent Sense by the Company it Keeps. In Proc. of ICSC, pages 208-215.
T. Haveliwala, A. Gionis D. Klein, and P. Indyk. 2002. Evaluating Strategies for Similarity Search on the Web. In Proc. of WWW, pages 432-442.
Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for Word Sense Sisambiguation: An Evaluation Etudy. In Proc. of ACL, volume 1, pages 897-907.
S. Lloyd. 2006. Least Squares Quantization in PCM. IEEE Trans. Inf. Theor., 28(2):129-137.
Marco Maru, Federico Scozzafava, Federico Martelli, and Roberto Navigli. 2019. SyntagNet: Challenging Supervised Word Sense Disambiguation with Lexical-Semantic Combinations. In Proc of EMNLP-IJCNLP, pages 3525-3531.
Diana McCarthy, Rob Koeling, and Julie Weeds. 2004a. Ranking WordNet senses automatically. _Recall_ 40:60.
Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004b. Finding predominant senses in untagged text. In Proc. of ACL, pages 280-287.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153-157.
George A. Miller, R.T. Beckwith, Christiane D. Fellbaum, D. Gross, and K. Miller. 1990. Introduction to WordNet: an online lexical database. International Journal of Lexicography, 3(4):235-244.
George A. Miller, Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. A Semantic Concordance. In Proc. of DARPA Workshop on Human Language Technology, pages 303-308, Plainsboro, N.J.
Andrea Moro and Roberto Navigli. 2015. Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proc. of SemEval-2015, pages 288-297.
Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. TACL, 2:231-244.
Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1-69.
Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Proc. of SemEval-2013, volume 2, pages 222-231.
Roberto Navigli and Simone Paolo Ponzetto. 2010. *BabelNet: Building a Very Large Multilingual Semantic Network*. In *Proc. of ACL*, pages 216-225.
Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. Artificial Intelligence, 193:217-250.
Tommaso Pasini. 2020. The Knowledge Acquisition Bottleneck Problem in Multilingual Word Sense Disambiguation. In Proc. of IJCAI. International Joint Conferences on Artificial Intelligence Organization.
Tommaso Pasini and Roberto Navigli. 2018. Two Knowledge-based Methods for High-Performance Sense Distribution Learning. In Proc. of AAAI, pages 5374-5381.
Tommaso Pasini and Roberto Navigli. 2020. Train-o-matic: Supervised word sense disambiguation with no (manual) effort. Artificial Intelligence, 279:103215.
Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. 2013. Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity. In Proc. of ACL, pages 1341-1351.
Sameer S Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task 17: English lexical sample, SRL and all words. In Proc. of SemEval-2007, pages 87-92.
Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. 2018. Integrating Weakly Supervised Word Sense Disambiguation Into Neural Machine Translation. TACL, 6:635-649.
Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017a. Neural Sequence Learning Models for Word Sense Disambiguation. In Proc. of EMNLP, pages 1156-1167.
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017b. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. In Proc. of EACL, pages 99-110.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Proc. of NeurIPS, pages 8592-8600.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2019. Just OneSeC for Producing Multilingual Sense-Annotated Data. In Proc. of ACL, volume 1, page 699709.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020. SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation. In Proc. of AAAI.
Federico Scozzafava, Marco Maru, Fabrizio Brignone, Giovanni Torrisi, and Roberto Navigli. 2020. Personalized PageRank with Syntagmatic Information for Multilingual Word Sense Disambiguation. In Proc. of ACL.
Benjamin Snyder and Martha Palmer. 2004. The English all-Words Task. In Proc. of Senseval-3, pages 41-43.
Rocco Tripodi and Marcello Pelillo. 2015. WSD-games: A Game-Theoretic Algorithm for Unsupervised Word Sense Disambiguation. In Proc. of SemEval, pages 329-334.
Loic Vial, Benjamin Lecouteux, and Didier Schwab. 2019. Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation. In Proc. of Global Wordnet Conference.
Warren Weaver. 1949. Translation. In Machine Translation of Languages: Fourteen Essays, pages 15-23.
Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-Supervised Word Sense Disambiguation with Neural Models. Proceedings of COLING, pages 1374-1385.
Zhi Zhong and Hwee Tou Ng. 2010. It Makes Sense: A Wide-Coverage Word Sense Disambiguation System for Free Text. In Proc. of ACL, pages 78-83.