When You Doubt, Abstain: A Study of Automated Fact-Checking in Italian Under Domain Shift
Authors: Giovanni Valer, Alan Ramponi, Sara Tonelli (Università di Trento, Fondazione Bruno Kessler, 2023)
Topic: Fact Checking and Fake News Detection

Abstract
Data for building fact-checking models for Italian is scarce, often contains ambiguous claims, and lacks textual diversity. This makes it hard to reliably apply such tools in the real world to support fact-checkers’ work. In this paper, we propose a categorization of claim ambiguity and label the largest Italian test set based on it. Moreover, we create challenge sets across two axes of variation: genres and fact-checking sources. Our experiments using transformer-based semantic search show a large drop in performance under domain shift, and indicate the benefit of models’ abstention in case of lacking evidence.

Introduction
Countering the spread of mis/disinformation is one of the major challenges of our society, but human fact-checkers struggle to cope with the increasing amount of content being published. On these bases, in recent years automated fact-checking has gained increasing attention in the NLP community, resulting in a significant body of works and initiatives, e.g., the Fact Extraction and VERification Workshop (FEVER), at its 7th edition in 2023 [1]. Research efforts in NLP for automated fact-checking span over a plurality of tasks, from claim detection to verdict prediction and justification production [2]. Nevertheless, languages other than English, one of them being Italian, are mostly overlooked in current NLP research on the topic. Specifically, little work has been done to build annotated corpora for the Italian language, which is currently included in just a handful of multilingual datasets, i.e., X-Fact [3] and FakeCovid [4]. To exacerbate the problem, most datasets for automated fact-checking not only include underspecified claims for which verdicts are hard-to-impossible to be determined [5], but also typically lack domain diversity, making it difficult to ascertain the reliability of the resulting fact-checking systems when applied on texts reflecting different genres (e.g., from news headlines to posts on social media). In this paper, we aim to advance automated fact-checking in Italian by examining claim ambiguity in the largest, publicly-available test set to date, and providing means to measure and mitigate the impact of domain shift along genres and sources dimensions of variation. Our study shows that automated fact-checking in Italian is still far from being reliably applied in the real-world, and indicates the benefit of models’ abstention in case of lacking evidence for verification. Contributions i) We propose a categorization of claim ambiguity, ii) annotate the Italian test portions of X-Fact according to it, and iii) create challenge test sets for studying automated fact-checking in Italian under domain shift. We further iv) assess performance shift using transformer-based semantic search, v) highlighting the benefit of abstention in the case of insufficient evidence.

Conclusion
In this work, we show that domains do have a large impact on performance of automated fact-checking for Italian, and the faculty of abstention may be considered to cope with lack of sufficient evidence. Moreover, we contribute to the community by classifying claim ambiguity in the largest Italian test set to date and distributing Italian challenge test sets reflecting diversified domains. Future work includes complementing challenge sets with further versions by multiple annotators as well as automating claim ambiguity assessment. Moreover, the confidence level of the classifier could be investigated and measured with tailored metrics to improve automated fact-checking reliability in handling uncertain cases. In general, as suggested by Schlichtkrull et al. (2023) [16], it would be important to assess the system efficacy with its intended users, in order to evaluate any unforeseen harm possibly caused by actual applications of the technology. In the future, we therefore plan to test our system by including it in the workflow adopted by professional fact-checkers to verify possible cases of mis/disinformation.