Kronos-it: a Dataset for the Italian Semantic Change Detection Task
Authors: Pierpaolo Basile, Giovanni Semeraro, Annalina Caputo (Università di Bari Aldo Moro, Dublin City University, 2019)
Topic: Lexical and Semantic Resources and Analysis

Abstract
This paper introduces Kronos-it, a dataset for the evaluation of semantic change point detection algorithms for the Italian language. The dataset is automatically built by using a web scraping strategy. We provide a detailed description about the dataset and its generation, and four state-of-the-art approaches for the semantic change point detection are benchmarked by exploiting the Italian Google n-grams corpus.

Introduction
Computational approaches to the problem of language change have been gaining momentum over the last decade. The availability of long-term and large-scale digital corpora, and the effectiveness of methods for representing words over time, are the prerequisites behind this interest. However, only a few attempts have focused on the evaluation, due to two main issues. First, the amount of data involved limits the possibility to perform a manual evaluation and, secondly, to date no open dataset for diachronic semantic change has been made available. This last issue has roots in the difficulties of building a gold standard for detecting the semantic change of terms in a specific corpus or language. The result is a fragmented set of data and evaluation protocols, since each work in this area has used different evaluation datasets or metrics. This phenomenon can be gauged from (Tahmasebi et al., 2019), where it is possible to count at least twenty different datasets used for the evaluation. In this paper, we describe how to build a dataset for the evaluation of semantic change point detection algorithms. In particular, we adopt a web scraping strategy for extracting information from an online Italian dictionary. The goal of the extraction is to build a list of lemmas with a set of change points for each lemma. The change points are extracted by analysing information about the year in which the lemma with a specific meaning is observed for the first time. Relying on this information, we build a dataset for the Italian language that can be used to evaluate algorithms for semantic change point detection. We provide a case study in which four different approaches are analysed using a unique corpus. The rest of the article is organised as follows: Section 2 describes how our dataset is built, while Section 3 provides details about the approaches under analysis and the evaluation. Finally, Section 4 closes the paper and provides possible future work.

Conclusion
In this paper, we provide details about the construction of a dataset for the evaluation of semantic change point detection algorithms. In particular, our dataset focused on the Italian language and it is built by adopting a web-scraping strategy. We provide a usage example of our dataset by evaluating several approaches for the representation of words over time. The results prove that the task of detecting semantic shift is challenging due to a large number of detected false positive. As future work, we plan to investigate further methods for building time series and detecting semantic shifts in order to improve the overall performance. Moreover, we plan to fix some issues of our extraction process in order to improve the quality of the dataset itself.