Introduction
With a rise in deliberate disinformation, Fact Verification (FV) has become an important NLP application.
FV aims to verify given claims with the evidence retrieved from plain text.
Rapid progress has been made by training large neural models Zhou et al. ( 2019 ); Liu et al. ( 2020 ); Zhong et al. ( 2020 ) on the FEVER dataset Thorne et al. ( 2018 ) , containing more than 100K human-crafted (evidence, claim) pairs based on Wikipedia. Fact verification is also needed in other domains, including news, social media, and scientific documents. This has spurred the creation of a large number of FV datasets, such as COVID-Fact Saakyan et al. ( 2021 ) ,
SciFact Wadden et al. ( 2020 ) , and Climate-FEVER Diggelmann et al. ( 2020 ) .

However, considering that human annotation is time-consuming, costly, and often biased, it is difficult to collect reliable human-labeled data in every domain that demands fact verification. We need to investigate how to build a generalizable fact verification system that adapts to new domains with zero or few samples. Critically, how can we leverage valuable (evidence, claim, label) annotations from rich-resourced domains ( e.g. , Wikipedia) to aid fact verification in the low-resourced ones ( e.g. , scholarly documents, and social media)?
Although FV datasets have been recently created in different domains, little analysis has shown whether FV models generalize across them and to what extent existing datasets can be leveraged to improve performance in these new domains.

In this paper, we bridge this gap by conducting a comprehensive investigation of zero- and few-shot generalization in fact verification . By conducting a holistic study of FV datasets to date, we first carefully select 8 datasets that have artificial or natural claims, human-annotated evidence, and two or three-class labels for our study. We then standardize their data formats as (evidence, claim, label) pairs and create dataset variants with different granularity of evidence, which gives us a total of 11 datasets.
We then conduct a thorough empirical study of generalization and transfer across these 11 datasets. We train models on a source dataset, and then evaluate their performance on a target dataset, either without any additional target training examples ( zero-shot setting ) or with a few additional target examples ( few-shot setting ).

We find that RoBERTa-based FV models tend to overfit the particular training set, generalizing poorly to other datasets. Our in-depth analysis shows generalization is related to several key factors, including dataset size, length of evidence, and the claim type. In particular, we find that Wikipedia-based artificial claims ( e.g. , FEVER) generalize well to natural claims in real-world domains with the growth of dataset size, in contrast to prior work that criticized crowd-sourced claims as having strong annotation bias and being unrepresentative of real-world misinformation Schuster et al. ( 2019 ) . Our few-shot generalization experiment further shows that fine-tuning on a small amount of target training data can substantially improve performance.

Armed with the above insights, we explore two ways to improve the generalization of fact verification models. 1) Domain-specific Pretraining : initializing the FV model with language models pretrained on specialized domains, and 2) Data Augmentation : automatically generating training data for the target domain. Results show that these methods can noticeably improve generalization but still leave unsolved challenges such as inflexibility, high cost, and label consistency.

To the best of our knowledge, this is the first work to perform a thorough investigation of generalization and transfer in fact verification. We open-sourced our dataset collection and codes to support future research towards a universal and robust fact verification system 1 1 1 https://github.com/teacherpeterpan/Fact-Checking-Generalization .