Leveraging Large Language Models for Fact Verification in Italian
Authors: Antonio Scaiella, Stefano Costanzo, Elisa Passone, Danilo Croce, Giorgio Gambosi (Università di Roma Tor Vergata, Reveal srl, 2024)
Topic: Fact Checking and Fake News Detection

Abstract
In recent years, Automatic Fact Checking has become a crucial tool for combating fake news by leveraging AI to verify the accuracy of information. Despite significant advancements, most datasets and models are predominantly available in English, posing challenges for other languages. This paper presents an Italian resource based on the dataset made available in the FEVER evaluation campaign, created to train and evaluate fact-checking models in Italian. The dataset comprises approximately 240k examples, with over 2k test examples manually validated. Additionally, we fine-tuned a state-of-the-art LLM, namely LLaMA3, on both the original English and translated Italian datasets, demonstrating that fine-tuning significantly improves model performance. Our results suggest that the fine-tuned models achieve comparable accuracy in both languages, highlighting the value of the proposed resource.

Introduction
In recent years, Automatic Fact Checking (AFC) has assumed a significant role as an instrument to identify fake news. AFC is a process that verifies the truthfulness and accuracy of information, claims, and data contained in a text or speech. The focus is on debunking disinformation and misinformation, intercepting errors, and verifying sources and facts. Automated fact-checking uses AI tools to identify, verify, and respond to misleading claims, using techniques based on natural language processing, machine learning, knowledge representation, and databases to automatically predict the truthfulness of claims AUTHOR. This is a complex process that involves searching, interpreting, and assessing information. As discussed in AUTHOR a NLP framework for automated fact-checking consists of three stages: claim detection to identify claims that require verification; evidence retrieval to find sources supporting or refuting the claim; and claim verification to assess the truthfulness of the claim based on the retrieved evidence. At first, automating the fact-checking process has been discussed in the context of computational journalism in works like AUTHOR, and has received significant attention in the computational linguistics and, in general, the artificial intelligence communities, surveyed in AUTHOR and more recently in AUTHOR and AUTHOR. In particular, in AUTHOR the authors expose a survey on the topic, describing the early developments that were surveyed in AUTHOR, which is an exhaustive overview of the subject. As with most machine learning paradigms AUTHOR, state-of-the-art methods require datasets and benchmarks. One of the most impactful campaigns for collecting a large-scale benchmark is FEVER (Fact Extraction and VERification) AUTHOR. In this context, fact-checking involves verifying whether a claim is supported by one or more pieces of evidence. FEVER is a publicly available dataset designed for claim verification against textual sources. It comprises about 180K claims generated by altering sentences extracted from Wikipedia. The claims are classified into three categories: \textsc{Supported} (a piece of evidence exists and it supports the claim), \textsc{Refutes} (a piece of evidence exists and it contradicts the claim), or \textsc{NotEnoughInfo} (there is insufficient evidence to verify the claim). The challenge, therefore, is to retrieve the relevant evidence and verify the accuracy of the claims, categorizing them with the correct label. Many works like FEVER have recently focused on building data and datasets for the task of Fact Verification, achieving very good results AUTHOR. However, all of these datasets are designed for the English language. Although multilingual models exist (e.g., in AUTHOR), fine-tuning a model on a specific language, pre-training it for a specific task and use case, could lead to a significant decline in quality if applied to another language. Few studies have worked on training models for languages other than English. An example is the work presented in AUTHOR, which focuses on developing automated claim detection for Dutch-language fact-checkers. In this work, we propose a FEVER-IT dataset in which the FEVER dataset has been translated into Italian to train the model for the Italian language. Inspired by SQUAD-IT AUTHOR and MSCOCO-IT AUTHOR, we worked to obtain quality data. Although the training set may be affected by translation errors, the test set will not, as it is composed of manually validated data. Furthermore, while the original FEVER dataset contained evidence only for \textsc{Supports} and \textsc{Refutes}, in this work we have also added and translated examples for the \textsc{NotEnoughInfo} category using the heuristics proposed in AUTHOR. This work extends the experience described in AUTHOR, where translations were done using Google API, by using publicly available models ( AUTHOR) and adding data for the \textsc{NotEnoughInfo} category. The contribution of this work is twofold. Firstly, we release FEVER-IT, a corpus with 228K claims each associated with at least one (possibly useful) piece of evidence, including a test set of 2,000 manually validated claims. In addition, we fine-tuned and validated a state-of-the-art model, LLaMA3 AUTHOR, on both the original English dataset and the Italian dataset. While this provides a high-performance model ready for the task in both languages, the primary goal is to assess whether the quality of the Italian data is comparable to the English one. By training the model separately on each dataset, we can evaluate its stability: if the model performs similarly on the manually validated Italian dataset and the English test set, we can conclude that the quality of the Italian data is on par with the English data. Additionally, we want to assess whether using an Italian train dataset, despite the noise from automatic translation, is truly beneficial. LLMs like LLaMA3 can already perform tasks in other languages through zero-shot or few-shot learning, without requiring fine-tuning on a specific dataset, especially if that dataset is noisy. Therefore, we aim to compare the performance on the test set between a LLaMA3 model that hasn’t been fine-tuned on the noisy Italian data and one that has been fine-tuned, to determine whether fine-tuning actually improves results or if the model performs on par or better without it. The experimental results show that the model without fine-tuning achieves an average accuracy of only about 45%. Fine-tuning on the English dataset yields about 90% mean accuracy, while fine-tuning on the Italian dataset results in a percentage quite similar to the fine-tuned English model and much greater than testing without fine-tuning}. The remainder of the paper is organized as follows: Section sec:relwork discusses related work, Section sec:feverit presents FEVER-IT, Section sec:expeval details the experimental measures, and Section sec:concl provides the conclusions.

Conclusion
In this work, we have introduced FEVER-IT, an Italian version of the FEVER dataset, designed to improve the training and evaluation of models for fact verification in the Italian language. Using a machine translation system, we translated a large-scale dataset of 228,000 claims/pieces of evidence pairs and manually validated 2,000 test instances to ensure meaningful evaluations. This enabled us to fine-tune a state-of-the-art LLM, specifically LLaMA3, and assess its performance in both English and Italian. Our experiments demonstrated that the multilingual model, without fine-tuning, performed similarly on both English and Italian datasets, though the accuracy and stability were limited. Fine-tuning significantly improved the model's performance, achieving over 90% accuracy in both languages. This underscores the importance and effectiveness of the translated dataset, even if it contains some noise. Future work will explore the performance of larger models and further refinement of the dataset to enhance accuracy and generalization capabilities or explore more complex settings such as those described in AUTHOR. The team would like to thank Monika Kakol for her invaluable support in the validation of the translations. This work was supported by Project ECS 0000024 Rome Technopole, - CUP B83C22002820006, NRP Mission 4 Component 2 Investment 1.5, Funded by the European Union - NextGenerationEU. \appendix