Nunc Est Aestimandum: Towards an Evaluation of the Latin WordNet
Authors: Greta Franzini, Andrea Peverelli, Paolo Ruffolo, Marco Carlo Passarotti, Helena Sanna, Edoardo Signoroni, Viviana Ventura, Federica Zampedri (Università Cattolica del Sacro Cuore, Università di Pavia, 2019)
Topic: Latin Resources

Abstract
This paper describes a preliminary expansion and assessment of the Latin WordNet for the purposes of the LiLa: Linking Latin project. The objective of this study is to better understand the implications of expanding and evaluating the sense coverage of the Latin WordNet, with a view to identifying the most effective method for its refinement and inclusion in the LiLa Knowledge Base of Latin resources. Our test empirically demonstrates the inadequacy for Latin of a common semi-automated approach of expansion and informs potential lines of improvement for the resource.

Introduction
WordNets are among the most used lexicosemantic resources in Natural Language Processing (NLP). Indeed, their value is such as to warrant the annual Global WordNet Conference, which is now in its tenth edition. In the words of Fellbaum (1998, p. 52): WordNet [. . . ] is perhaps the most widely used electronic dictionary of English and serves as the lexicon for a varity [sic] of different NLP applications including Information Retrieval (IR), Word Sense Disambiguation (WSD), and Machine Translation (MT). Since the release of the Princeton WordNet (hereafter PWN) in the mid 1980s AUTHOR, interest in providing WordNets for modern languages has far exceeded that for historical languages. With the exception of the Historical Thesaurus of English, whose purpose is not dissimilar to that of a WordNet but whose distinct structure sets it apart from this type of resource, the only two historical language WordNets in existence today are the Latin AUTHOR and the Ancient Greek WordNets AUTHOR: both have limited lexical coverage and the Latin WordNet (hereafter LWN) is particularly noisy (see Section 3). Their incompleteness poses significant challenges to a number of computational analyses, thus restricting the scope for lexico-semantic research. The study described here falls within the scope of the LiLa: Linking Latin project AUTHOR. In its wider effort to connect linguistic resources and NLP tools for Latin in a Linked Data Knowledge Base, LiLa is conducting a first assessment of the LWN. Besides being structurally compatible with LiLa, a refined LWN is essential to the Knowledge Base as a connector between Latin and resources in other languages, thus meeting a growing need in the field of Linguistic Linked Open Data AUTHOR. This paper describes a preliminary assessment of the LWN with a view to better understanding how to approach its expansion and evaluation: Sections 2 and 3 briefly outline existing research in WordNet evaluation and the structure of the LWN, respectively; Section 4 details our evaluation method; Section 5 discusses our preliminary results; finally, Section 6 summarises our contribution and focusses on directions for future research.

Conclusion
This paper describes a preliminary assessment of the implications of evaluating the LWN carried out in the context of the LiLa: Linking Latin project. The objective of LiLa is to connect linguistic resources and NLP tools for Latin with a view to supporting different lines of linguistic and corpus-based research and to connecting Latin to other languages. Owing to its automatic process of creation, the LWN is lexically and semantically limited, as well as noisy, subjecting its inclusion in LiLa to qualitative revision. For a close understanding of the implications of evaluating the LWN, we formulated a first experiment combining a small, automated extension of the sense coverage on the basis of 100 selected LWN lemmas with a follow-up manual revision of their corresponding synset assignments. The purpose of this experiment was to measure the reliability and feasibility of these two approaches in order to identify the most effective evaluation compromise. Our synset recommender system produced many false positives, with only 0.18% (7) machine suggestions approved by all five raters. Even if the precision of the synset-recommendation algorithm were to be improved, recall would likely still be high due to the unavoidable assignment of modern senses to a historical resource. If applied to the entire LWN, the evaluation method described here, coupled with the additional evaluation of the relations between synsets, would turn this process of revision into an unsustainable effort or, at the very least, one that is not achievable within the scope and duration of LiLa. Moving forward, our plan for the improvement of the LWN will develop into various tasks. The first, ongoing effort is the manual removal of the modern senses originally inherited by the LWN. Next, once cleaned, we will extend the sense coverage of the LWN by manually adding the missing senses recorded by the raters for the 100 evaluated lemmas, careful not to introduce too much granularity (i.e., too many senses with only subtle semantic differences); extract hypernyms, synonyms and bags of words from dictionary definitions AUTHOR, as well as lemma groups from three Latin synonym dictionaries: the Latin-English Handbook of Latin Synonymes AUTHOR, the Latin-English The synonymes of the Latin language AUTHOR and the Latin-Czech Latinská synonymika pro školu i dům AUTHOR. These are all freely available online in XML dictionary format (XDXF) and, combined, can supply the LWN with some 1,050 additional lemmas. Thirdly, connect a graph version of the LWN to textual resources in LiLa to acquire lexical knowledge, and explore the possibility of extracting hypernym/hyponym pairs using syntactic patterns AUTHOR. Finally, extend the LWN with Named Entities extracted from the morphological analyser LEMLAT AUTHOR. The data and code repository for this paper are available at: .