Kronos-it: a Dataset for the Italian Semantic Change Detection Task

This paper introduces Kronos-it, a dataset for the evaluation of semantic change point detection algorithms for the Italian language. The dataset is automatically built by using a web scraping strategy. We provide a detailed description about the dataset and its generation, and four state-of-the-art approaches for the semantic change point detection are benchmarked by exploiting the Italian Google ngrams corpus. 1 Background and Motivation Computational approaches to the problem of language change have been gaining momentum over the last decade. The availability of long-term and large-scale digital corpora, and the effectiveness of methods for representing words over time, are the prerequisite behind this interest. However, only few attempts have focused on the evaluation, due to two main issues. First, the amount of data involved limits the possibility to perform a manual evaluation and, secondly, to date no open dataset for the diachronic semantic change has been made available. This last issue has roots in the difficulties of building a gold-standard for detecting the semantic change of terms in a specific corpus or language. The result is a fragmented set of data and evaluation protocols, since each work in this area has used different evaluation datasets or metrics. This phenomenon can be gauged from (Tahmasebi et al., 2019), where it is possible to count at least twenty different datasets used for the evaluation. In this paper, we describe how to build a dataset for the evaluation of semantic change point detection algorithms. In particular, we adopt a Copyright c ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). web scraping strategy for extracting information from an online Italian dictionary. The goal of the extraction is to build a list of lemmas with a set of change points for each lemma. The change points are extracted by analysing information about the year in which the lemma with a specific meaning is observed for the first time. Relying on this information we build a dataset for the Italian language that can be used to evaluate algorithms for the semantic change point detection. We provide a case study in which four different approaches are analysed using a unique corpus. The rest of the article is organised as follows: Section 2 describes how our dataset is built, while Section 3 provides details about the approaches under analysis and the evaluation. Finally, Section 4 closes the paper and provides possible future work. 2 Dataset Construction The main goal of the dataset is to provide for each lemma a set of years which indicate a semantic change for that lemma. Some dictionaries provide historical information about meanings, for example the year in which each meaning is observed for the first time. The main problem is that generally these dictionaries are not digitally available or they are in a format that is not machine readable. Regarding the Italian language, the dictionary “Sabatini Coletti”1 is available on-line. It provides for some lemmas the year in which each meaning was observed for the first time. For example, taking into account the entry for the word “imbarcata” from the dictionary, we capture its original meaning “Group of people who gather to find each other, to leave together”, and other two meanings: 1) “Acrobatic manoeuvre of an air-plane” introduced in 1929; and 2) “fall in love” introduced in 1972. https://dizionari.corriere. it/ dizionario_italiano/ We setup a web scraping algorithm able to extract this information from the dictionary. In particular, the extraction process is composed of several steps: 1. Downloading the list of all lemmas occurring in the online dictionary with the corresponding URL. We obtain a list of 34,504 lemmas; 2. For each lemma, extracting the section of the web page containing the definition with the list of all possible meanings. We obtain a final list of 34,446 definitions; 3. For each definition, extracting the year in which that meaning was introduced. For a given lemma, we are not able to assign the correct year to each of its meaning, but we can only extract a year associated with the lemma. This happens because the dictionary does not follow a clear template for assigning the year to each meaning. Although associating the year of change to the definition of the meaning is not useful for the purpose of our evaluation, it could help to understand the reason behind the semantic change. We plan to fix this limitation in a further release of the dataset. In the rest of the paper we call change point (CP) each pair (lemma, year); 4. Removing those change points that are expressed in the form “III sec. ” (III century) because they refer to a broad period of time rather than to a specific year. Figure 1: The distribution of change points over time. The final dataset2 contains 13,818 lemmas and 13,932 change points. The average change points for lemma is 1.0083 with a standard deviation of 0.0924. The maximum number of change points https://github.com/pippokill/ kronos-it for lemma is 3 and the number of lemmas with more than one change point is 113. The oldest reported change point is 1758, while the most recent one is 2003; this suggests that the dictionary is outdated and it does not contain more recent meanings. The dataset is provided in textual format and reports for each row the lemma followed by a list of years, each one representing a change point. For example: