Check-IT!: A Corpus of Expert Fact-checked Claims for Italian
Authors: Jacopo Gili, Lucia C. Passaro, Tommaso Caselli (Università di Torino, Università di Pisa, University of Groningen, 2023)
Topic: Fact Checking and Fake News Detection

Abstract
This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims in zero-shot settings as a data-augmentation strategy for the identification of previously fact-checked claims. Our results indicate that models struggles in varying the surface forms of the claims.

Introduction
The pollution of the information ecosystem by misleading or false information has reached unprecedented levels globally. This phenomenon has been fueled by several factors, including the collapse of local and national journalism, growing distrust in science and evidence-based facts, and the rise of computational amplification tools like bots. Manually fact-checking claims is an expensive and time-consuming process, and in many cases, it comes too late. Research has shown that false and inaccurate information spreads online eight times faster than true and reliable information. Allowing this kind of information to circulate freely can have harmful consequences for individuals and groups, and it threatens the fabric of democratic societies. Therefore, it is crucial to implement automatic solutions that can assist content moderators and information professionals in responding quickly to false or misleading information. The full fact-checking verification pipeline involves several steps: (i) assessing whether a claim is worth fact-checking, (ii) checking whether the claim has already been fact-checked, (iii) gathering evidence from reliable sources to evaluate the claim's veracity, and (iv) assessing its truthfulness. Having access to a database of previously fact-checked claims is a valuable resource because claims often get repeated with minor variations, especially by politicians. This resource can save time and mitigate the effects of misinformation. This paper introduces CheckIT!, the first corpus of previously fact-checked claims for Italian. Currently, CheckIT! consists of 3,577 claims from 317 Italian politicians and public figures, each annotated with evidence and veracity labels. The main contributions of the paper include: (i) the introduction of CheckIT!, a fact-checking resource that addresses a gap in Italian-language resources for claim verification and misinformation detection; (ii) a feasibility study on automatic paraphrasing in Italian, exploring the use of advanced NLP techniques for generating high-quality texts that preserve the original meaning of claims while introducing linguistic variations; (iii) the proposal of an initial framework for the automatic extension of fact-checking resources, allowing CheckIT! to continuously grow with additional fact-checked claims and related evidence. The paper is organized as follows: Section 2 describes the data collection process, the harmonization of veracity labels, and an analysis of the dataset. Section 3 discusses the results of paraphrasing experiments using text generation tools for Italian (mT5, Camoscio, ChatGPT), focusing on their quality. Section 4 covers related work, and Section 5 concludes the paper and outlines directions for future development.

Conclusion
This work has introduced CheckIT!, an expert-curated fact-checked repository of claims by politicians and prominent public figures in Italy. CheckIT! spans 10 years of claims and is the first publicly available dataset for fact-checking in Italian. In our analysis of CheckIT!, we observed a decline in the number of fact-checked claims, suggesting that manual fact-checking is becoming increasingly difficult and that automated tools are needed more than ever. We conducted a preliminary investigation of three state-of-the-art automatic text generation tools for paraphrasing claims. By combining multiple automatic measures, it appears that ChatGPT and mT5 are the two best candidates for further exploration, while Camoscio presents non-trivial issues regarding failure to produce output and variations in the generated texts. Future work will focus on three areas: conducting a qualitative (human-based) evaluation of the two best models; evaluating the generated paraphrases for previously fact-checked claim retrieval, following the approach in [5]; and finally, assessing the generated paraphrases against the topics.