SlowGuess/ACL_2020_OCR huggingface dataset:
We train our baseline and  $O$ -init models for 20 epochs. The  $O$ -freeze model, which is much slower to converge, is trained for a maximum of 80 epochs.  $O$ - thaw and  $O$ - thaw* are trained for 10 epochs. The data on which we train and report the performances are the same as in Section 4.1.2.
We report in Table 2 the results of the evaluation of the use of synset embeddings for the initialization of the  $O$  output embeddings matrix.
In general, the approach enables much better F1 scores compared to the baseline, but is very dependent on the quality of the embeddings, and on whether they incorporate supervision from SemCor. When using Deconf, which uses the WordNet graph to "deconflate" word-level Word2Vec vectors, with no use of training corpora, the  $O$ -freeze strategy produces the best result on  $\mathrm{No15^{-}}$ , i.e., 72.2, with an absolute increase of 20 points over the baseline. However,  $O$ -freeze with Deconf also achieves the worst result on both ALL and No15, indicating that some form of biasing towards the most frequent synsets, which is an effect of corpus supervision, is required for the global evaluation. Fine-tuning  $O$  enables the model to obtain a decent
S	G	G+	E	System	ALL	No15	No15-	S2	S3	S7	S13	S15	N	V	A	R
✓	✓	-	-	Kumar et al. (2019)	71.8	70.9*	-	73.8	71.1	67.3	69.4	74.5	74.0	60.2	78.0	82.1
✓	✓	-	-	Loureiro and Jorge (2019)	75.4	75.2*	-	76.3	75.6	68.1	75.1	77.0	-	-	-	-
✓	-	-	-	Hadiwinoto et al. (2019)	73.7*	73.2*	-	75.5	73.6	68.1	71.1	76.2	-	-	-	-
✓	✓	-	-	Huang et al. (2019)	77.0*	76.2*	-	77.7	75.2	72.5	76.1	80.4	-	-	-	-
✓	✓	-	-	Scarlini et al. (2020) - Sup.	-	-	-	-	-	-	78.7	-	80.4	-	-	-
✓	-	-	-	Vial et al. (2019)	75.6	-	-	-	-	-	-	-	-	-	-	-
✓	-	-	-	Vial et al. (2019) - ENS	76.7	76.5*	-	77.5	77.4	69.5	76.0	78.3	79.6	65.9	79.5	85.5
✓	†	-	-	EWISERhyper	77.0*	76.9	60.4	77.5	77.9	71.0	76.4	77.8	79.9	66.4	79.0	85.5
✓	✓	-	-	EWISERhyper	77.5	77.3	68.2	78.4	77.4	71.0	77.4	78.7	80.7	65.1	80.9	86.1
✓	†	-	-	EWISERhyper+hypo	76.8	76.8	59.5	77.7	77.9	70.3	76.2	76.3	79.4	65.9	80.0	86.7
✓	✓	-	-	EWISERhyper+hypo	78.3	78.2	69.1	78.9	78.4	71.0	78.9	79.3	81.7	66.3	81.2	85.8
✓	✓	✓	✓	Vial et al. (2019)	77.1	-	-	-	-	-	-	-	-	-	-	-
✓	✓	✓	✓	Vial et al. (2019) - ENS	79.0*	78.4*	-	79.7	77.8	73.4	78.7	82.6	81.4	68.7	83.7	85.5
✓	✓	✓	✓	EWISERhyper	80.1	79.8	75.2	80.8	79.0	75.2	80.7	81.8	82.9	69.4	83.6	87.3
✓	✓	✓	✓	EWISERhyper+hypo	79.8	79.3	75.1	80.2	78.5	73.8	80.6	82.3	82.7	68.5	82.9	87.6
-	-	-	-	Scozzafava et al. (2020)	71.7	71.0*	-	71.6	72.0	59.3	72.2	75.8	-	-	-	-
-	✓	-	-	Scarlini et al. (2020) - KB	-	-	-	-	-	-	74.8	-	75.9	-	-	-
Table 3: Evaluation of the joint use of structured logits and  $O$ -thaw* on English all-words WSD. F1 is reported. The column blocks report (i) the training corpora and system compared; (ii) overall F1; (iii) single dataset F1; (iv) POS-specific F1. †: Incorporates gloss information through synset embeddings. *: Computed from reported scores. ☆: highest F1 that is statistically different from the best one ( $\chi^2$  with  $p = 0.1$ ).
F1 score, with the exception of  $O$ -thaw*, where the training run was underfitting. With LMMS, higher results are obtained, especially when freezing the weights. SensEmBERT with the LMMS backoff achieves the best results on both ALL and No15, with  $O$ -thaw* reaching at least 76.6 on ALL and No15. Probably due to the fact that SensEmBERT relies less on the supervision from SemCor, very strong results are obtained on  $\mathrm{No15^{-}}$  as well, with a margin of over 12 points above the baseline.
As for the training scheme adopted, the best results are obtained from the freeze-then-thaw strategy with learning rate reduction  $(O\text{-thaw}^*)$  and from the simple freezing of  $O$ . Thawing consistently raises the accuracy on ALL and No15, but lowers it on  $\mathrm{No15^{-}}$ , meaning that the fine-tuning of  $O$  shifts the balance of the trade-off between performances on seen and unseen synsets to the benefit of the former.  $O$ -init still improves over the baseline, but is less effective than its alternatives.
4.3 Combining Relational Knowledge and Sense Embeddings
Bringing everything together, we now evaluate the joint exploitation of the  $O$  initialization and structured logits in EWISER.
4.3.1 Setting
Building on the results of the previous experiments, we limit the number of model variants by only including the configurations that separately yielded the best results, namely: (i) the use
of hypernyms (EWISER  ${}_{hyper}$  ) or hypernyms plus hyponyms (EWISER  ${}_{hyper + hypo}$  ) in the graph encoded in  $A$ , training the adjacency matrix, and (ii) the combination of SensEmBERT and LMMS for the output embeddings, trained according to the  $O$ -thaw* scheme, i.e., the freeze-then-thaw approach, with the learning rate set to  ${10}^{-5}$  .
4.3.2 Data & Hyperparameters
In order to make the results of EWISER comparable to those of the state-of-the-art approaches to WSD, we report results when training not only on SemCor (S), but also on the union of SemCor and untagged WordNet glosses  $(\mathbf{G})$ , and on the union of SemCor, tagged WordNet glosses  $(\mathbf{G}^{+})$ , and WordNet examples  $(\mathbf{E})$  as well. When training on glosses, we prepend the lemma of the main sense and a semicolon to the raw gloss, and treat the added word as a tagged instance. We evaluate the model on the datasets mentioned in Section 4.1.2.
4.3.3 Results
In Table 3 we report the results of the unified evaluation. In addition to our systems, we include in the comparison the best systems from the literature, grouping the two sets together in two internally comparable blocks: (i) systems trained on SemCor, possibly making use of LKB information such as untagged glosses or the WordNet graph; (ii) systems that also make use of tagged glosses and examples; (iii) the best performing knowledge-based systems.
In almost every setting compared, EWISER outperforms the previous state of the art. Among systems in the first block (S/G)  $\mathrm{EWISER}_{hyper + hypo}$  trained on S+G obtains the best results on all the datasets except for SemEval-2015, with a margin over the two best performing systems, i.e., GlossBERT and the ensemble of 8 models of Vial et al. (2019), of, respectively, 1.3 and 1.6 points on ALL, and of 2.0 and 1.7 on No15, which does not include our dev set. Even if they do not train on untagged glosses, both  $\mathrm{EWISER}_{hyper}$  and  $\mathrm{EWISER}_{hyper + hypo}$  show comparable performances to GlossBERT on ALL, and better on No15 - without fine-tuning BERT, and with much less compute power required. The results on  $\mathrm{No15}^{-}$ , where  $\mathrm{EWISER}_{hyper + hypo}$  with glosses achieves an F1 of 69.1, almost 10 points more than when not using them, show that definitional knowledge is beneficial for the zero-shot setting.
Adding tagged glosses and WordNet examples further boosts performances, with the best configuration, EWISER $_{hyper}$ , breaking through the 80 points ceiling on ALL, an estimated upper bound on human inter-annotator agreement that is often quoted as the glass ceiling for WSD performance (Navigli, 2009). The only model we can compare with, i.e., the one of Vial et al. (2019), is outperformed on every dataset except for SemEval-2015. On ALL and No15, however, we outscore the competitor by a margin of 1.1 and 1.4 points, establishing a new state of the art in English all-words WSD. The bigger training set improves performances on No15 $^{-}$ , though the gap is not quite closed.
Not surprisingly, even the best knowledge-based systems do not offer competitive performances, since they cannot take advantage of training corpus supervision.
4.4 Cross-lingual WSD
To see whether the strong performances of EWISER carry over to the multilingual setting, we retrain the best global configuration, i.e.,  $\mathrm{EWISER}_{hyper}$  trained on SemCor, WordNet's tagged glosses and usage examples, with BERT multilingual cased. We compare our system against (i) the state of the art in multilingual WSD, i.e. SensEmBERT, which can, however, only disambiguate nouns; (ii) the best performing all-PoS system, i.e. SyntagRank (Scozzafava et al., 2020), a knowledge-based system; (iii) the feedforward baseline. We report results on the French, German,
S13	S15
DE	ES	FR	IT	ES	IT
Scozzafava et al. (2020)	76.4	74.1	70.3	72.1	63.4	69.0
Scarlini et al. (2020)	79.2*	73.4*	77.8*	69.8*	-	-
Ours (baseline)	81.7	76.6	80.8	77.2	67.3	70.6
Ours (EWISER)	80.9	78.8	83.6	77.7	69.5	71.8
Table 4: Evaluation of the joint use of structured logits and  $O$ -thaw* on cross-lingual WSD. F1 is reported.  
*: Recomputed by the authors.
Italian and Spanish all-words evaluation datasets from SemEval-2013, which contain only nouns, and the Italian and Spanish datasets from SemEval-2015, which contain all PoS. We use the revised version of the evaluation datasets<sup>4</sup>, which is updated to be consistent with the 4.0.1 release of the BabelNet graph. As a result, we can test on a larger number of instances than previously possible.
We show the results in Table 4. As can be seen, we outperform SensEmBERT in the four datasets from SemEval-2013, sometimes by a large margin, i.e., by almost 8 points on the Italian dataset. On SemEval-2015 we outperform SyntagRank by 6.1 points on the Spanish dataset and by 2.8 points on Italian one. We also show noticeable improvements over the baseline in 5 out of 6 benchmarks. The evaluation demonstrates that the EWISER approach is robust in the cross-lingual setting as well, outperforming competitors across the board and setting a new state of the art. Moreover, the results provide the empirical grounds for believing that, in addition to the results achieved in the languages featured in the evaluation datasets, comparable figures could also be attained for other languages, at least for several European ones.
In this section we provide a qualitative analysis of our approach. Specifically, we are interested in the capability of the model to predict unseen synsets, thanks to the prior knowledge that is encoded in both the output embeddings  $O$  and the adjacency matrix  $A$ . Consider the following sentences:
(1) a. Corporate debt defaults predicted to increase.
b. Though people are free to change the default, they usually don't.
In Table 5 we report the predictions for the target default in sentences (1a) and (1b) of our best sys
$^{4}$ github.com/SapienzaNLP/mwsd-datasets.
Synset	N	Gloss	w	z (1a)	q (1a)	z (1b)	q (1b)
default.n.01	1	loss due to not showing up	-	8.6	15.9	14.9	24.5
loss.n.03	6	the act of losing someone or something	.50	6.7	-	9.3	-
absence.n.02	8	failure to be present	.48	8.1	-	10.2	-
default.n.02	0	act of failing to meet a financial obligation	-	10.2	17.0	8.9	14.6
default.v.01	1	fail to pay up	.30	14.4	-	11.2	-
failure.n.01	18	an act that fails	.27	9.3	-	8.6	-
nonpayment.n.02	0	loss resulting from failure of a debt to be paid	-	11.0	17.9	9.6	15.5
default.v.01	1	fail to pay up	.30	14.4	-	11.2	-
financial.loss.n.01	0	loss of money or decrease in financial value	.29	8.7	-	8.6	-
default.option.n.01	0	an option that is selected automatically unless an alternative is specified	-	6.7	12.5	14.6	25.5
option.n.02	19	one of a number of things from which only one can be chosen	.76	7.7	-	14.3	-
Table 5: Predictions for sentences (1a) and (1b) of the best model trained on SemCor. In the first row of each block, we report the scores of the four synsets associated in WordNet with the noun default. The following rows contain the scores for synsets that are incident to those in the first row of the block, and contribute to their scores in q. The columns report, from left to right, a sense (therefore synset) identifier, the number of occurrences of that lemma in SemCor, the gloss, the weight of the edge, the hidden logits z and the output logits q.
tem trained on SemCor only, i.e.,  $\mathrm{EWISER}_{\text {hyper }}$ . In both cases, the correct synsets, respectively, default.n.02/nonpayment.n.02 and default_option.n.01, are not in the training set. However, the model is still able to give the correct answer. In the first case, the embedding initialization is enough to predict nonpayment.n.02 (with default.n.02 having the second highest score), as its score in z is already the highest among possible predictions. In the latter, it is the contribution from the synset pointing to default_option.n.01, i.e., option.n.02, that enables the network to make the correct prediction.
However, we must note that the model still overrelies on corpus supervision. Because of this, even though our best overall model, i.e.,  $\mathrm{EWISER}_{hyper}$  trained on SemCor, tagged glosses and examples, is able to distinguish and predict correctly the two well-attested mathematical meanings of root as equation solution and root as the number  $x$  such that  $y = x^2$  in sentences (2a) and (2b) below, it is not able to correctly detect the tooth sense of root (2c), which never occurs in SemCor:
(2) a. The  $n$  roots of a polynomial of degree  $n$  depend continuously on the coefficients.
b. The root of 4 is 2.
c. There's no need to be worried if your dentist prescribes a root canal procedure.
Thus, while the EWISER model is indeed very effective, with the best configuration outdoing the upper bound on inter-annotator agreement, we are still far from having solved the task.
We presented EWISER, a new neural WSD architecture that, by embedding information from the WordNet graph within the neural architecture, can also make use of the relational information that is usually only exploited by knowledge-based systems. Thanks to the joint exploitation of the WordNet graph and to the use of pretrained synset embeddings, EWISER is able to predict meanings which are not found in the training set, thus mitigating the knowledge acquisition bottleneck.
On almost all the evaluation settings, our system beats the previous state of the art. Most notably, our model is the first to break through the 80 F1 ceiling on the overall evaluation, the estimated upper bound on the task. On the multilingual setting, even with no training data besides the English corpora, EWISER sets the new state of the art.
We leave it as future work to explore ways to raise accuracy on unseen synsets without harming performances on frequent synsets. We release the code used in the experiments, as well as pretrained models at github.com/SapienzaNLP/ewiser.
The authors gratefully acknowledge the support of the ERC Consolidator
erc Grant MOUSSE No. 726487 under the European Union's Horizon 2020 research and innovation programme.
This work was supported in part by the MIUR under the grant "Dipartimenti di eccellenza 2018-2022" of the Department of Computer Science of the Sapienza University of Rome.
Eneko Agirre and Aitor Soroa. 2009. Personalizing PageRank for word sense disambiguation. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 33-41, Athens, Greece. Association for Computational Linguistics.
Michele Bevilacqua and Roberto Navigli. 2019. Quasi Bidirectional Encoder Representations from Transformers for word sense disambiguation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 122-131, Varna, Bulgaria. INCOMA Ltd.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1-5, Toulouse, France. Association for Computational Linguistics.
Steffen Eger, Paul Youssef, and Iryna Gurevych. 2018. Is it time to swish? Comparing deep learning activation functions across NLP tasks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4415-4424, Brussels, Belgium. Association for Computational Linguistics.
Christian Hadiwinoto, Hwee Tou Ng, and Wee Chung Gan. 2019. Improved word sense disambiguation using pre-trained contextualized word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5300-5309, Hong Kong, China. Association for Computational Linguistics.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. GlossBERT: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3500-3505, Hong
Kong, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015.
Sawan Kumar, Sharmistha Jat, Karan Saxena, and Partha Talukdar. 2019. Zero-shot word sense disambiguation using sense definition embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5670-5681, Florence, Italy. Association for Computational Linguistics.
Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife Hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962-5971, Florence, Italy. Association for Computational Linguistics.
Daniel Loureiro and Alipio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Zhifang Sui. 2018. Incorporating glosses into neural word sense disambiguation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2473-2482, Melbourne, Australia. Association for Computational Linguistics.
Marco Maru, Federico Scozzafava, Federico Martelli, and Roberto Navigli. 2019. SyntagNet: Challenging supervised word sense disambiguation with lexical-semantic combinations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3525-3531, Hong Kong, China. Association for Computational Linguistics.
George A. Miller. 1995. WordNet: A lexical database for english. Commun. ACM, 38(11):39-41.
George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In Proceedings of HUMAN LANGUAGE TECHNOLOGY: a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation
(SemEval 2015), pages 288-297, Denver, Colorado. Association for Computational Linguistics.
Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231-244.
Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41(2):10:1-10:69.
Roberto Navigli. 2018. Natural Language Understanding: Instructions for (Present and Future) Use. In Proc. of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18), pages 5697-5702, Stockholm, Sweden.
Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 222-231, Atlanta, Georgia, USA. Association for Computational Linguistics.
Roberto Navigli and Simone Paolo Ponzetto. 2012. *BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network*. Artificial Intelligence, 193:217–250.
Tommaso Pasini. 2020. The knowledge acquisition bottleneck problem in multilingual word sense disambiguation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Mohammad Taher Pilehvar and Nigel Collier. 2016. De-convlated semantic representations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1680–1690, Austin, Texas. Association for Computational Linguistics.