RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Dataset Curation
In this section, we describe the 11 fact verification datasets included in our study.
We first describe the criteria for dataset selection (§ 2.1 ), and then we introduce the dataset processing (§ 2.2 ). We show the key characteristics of the datasets in Table 1 .
Dataset Selection
A large number of datasets have recently been introduced to study various tasks for fact-checking, e.g. , claim detection, evidence retrieval, fact verification, justification production, etc. Our focus, fact verification, in particular, takes a textual claim and a piece of evidence as input to predict the label for the claim. Let’s define these aspects:

\bullet Claim : Claims for fact verification are often textual, sentence-level statements, which are categorized into: 1) real-world natural claims crawled from dedicated websites, textbooks, forums, etc. 2) artificial claims written by crowd-workers.

\bullet Evidence : Evidence is the relevant information source for validating the claim. Textual sources, such as news articles, academic papers, and Wikipedia documents, are one of the most commonly used types of evidence. Based on the granularity, we categorize the evidence in existing datasets into: 1) document-level evidence such as the Wikipedia page Thorne et al. ( 2018 ) , news articles Hanselowski et al. ( 2019 ) , and scientific papers Wadden et al. ( 2020 ) . 2) sentence-level evidence annotated by human experts in the relevant documents to support or refute each claim. 3) no evidence is given for each claim; the model needs to retrieve evidence from a large knowledge source.

\bullet Label : The label definition for the claim also varies across datasets. The most common definition is the binary label with supports and refutes , and the three-class label, i.e. , supports/refutes/not enough info . Some works Wang ( 2017 ); Augenstein et al. ( 2019 ) also employ multi-class labels for more fine-grained degrees of truthfulness ( e.g. true, mostly-true, mixture, etc), where the number of labels vary greatly, ranging from 4 to 27.
Selection Criteria.
We employ the following criteria to select the datasets for our study.

\bullet We consider both natural and artificial claims in various domains.

\bullet We consider the datasets with human-annotated document-level and sentence-level evidence. We exclude datasets without evidence or which provide only non-textual evidence; i.e. , tables, knowledge bases, etc.

\bullet We only consider datasets with the binary or the three-class label annotation due to the difficulty of canonicalizing such multi-class labels.

By conducting a holistic study of fact-checking datasets to date, eight different data sources meet our requirements. The full list of candidate datasets we investigate is given in Appendix A .
Dataset Processing
We then process the selected datasets as follows. 1) We convert each dataset to the unified format of claim-evidence-label triples {(c_{i},e_{i},l_{i})}_{i=1}^{N} . The simplicity of this format allows us to focus on out-of-domain generalization, instead of other orthogonal challenges of fact-checking. 2) We create separate dataset variants by pairing each claim with the evidence in different granularity. This enables us to study the impact of evidence length on generalization. After processing, we obtain the final selection of 11 datasets used. We now briefly introduce the nature of each dataset and its specific processing.
Group I: datasets with artificial claims.
These are based on Wikipedia articles and are often large in size. However, crowd-sourced claims are often written with minimal edits to reference sentences, leading to lexical biases such as the overuse of explicit negation Schuster et al. ( 2019 ) .

\bullet FEVER Thorne et al. ( 2018 ) asks crowd-workers to mutate sentences from Wikipedia articles to create claims.
We use the Wikipedia paragraph associated with each claim as its document-level evidence to construct the FEVER-para dataset. We then use the sentence-level gold evidence for the supports and refutes claims to build the FEVER-sent dataset. However, since sentence-level evidence is not available for NEI claims, we use the system of Malon ( 2018 ) to retrieve the evidence sentences, following Atanasova et al. ( 2020 ) and Pan et al. ( 2021 ) .

\bullet VitaminC Schuster et al. ( 2021 ) creates contrastive evidence pairs for each claim, in which evidence pairs are nearly identical in language and content, with the exception that one supports a claim while the other does not.

\bullet FoolMeTwice Eisenschlos et al. ( 2021 ) designs a multi-player game that leads to diverse strategies for crafting claims ( e.g. , temporal inference) based on Wikipedia, resulting in more complex claims with less lexical overlap with the evidence.
Group II: datasets with natural claims.
These claims are collected from the Internet and then manually verified by professional fact checkers. They represent real-world claims, and originate from diverse domains, such as scholarly documents, news articles, forums, etc. However, due to the difficulty and high cost of manually verifying real-world claims, these datasets are limited in scale.

\bullet Climate-FEVER Diggelmann et al. ( 2020 ) consists of 1,535 real-life claims regarding climate-change collected from the Internet. The top five most relevant sentences from Wikipedia are retrieved as the evidence. Humans then annotate each sentence as supporting, refuting, or not enough information to validate the claim. We use the sentence-level annotation as the evidence for each claim to build the Climate-FEVER-sent . We construct the document-level evidence for each claim by putting together all of its evidence sentences, which gives us the Climate-FEVER-para version.

\bullet Sci-Fact Wadden et al. ( 2020 ) consists of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and sentence-level rationale. We use the annotated rationale as the sentence-level evidence to build the Sci-Fact-sent . We construct the Sci-Fact-para version by using the evidence-containing abstract as the document-level evidence for each claim.

\bullet PubHealth Kotonya and Toni ( 2020 ) contains 11.8K claims accompanied by journalist crafted, gold standard judgments to support/refute the claims. The claims are collected from five fact-checking websites, news headlines, and news reviews. We use the judgment texts as evidence to pair with each claim.

\bullet COVID-Fact Saakyan et al. ( 2021 ) consists of 4,086 claims concerning the COVID19 pandemic crawled from the r/COVID19 subreddit. We use their sentence-level evidence annotated by crowd-workers as the evidence.

\bullet FAVIQ Park et al. ( 2022 ) contains 26k claims converted from natural ambiguous questions posed by real users. The answer-containing Wikipedia paragraph is provided as the document-level evidence for each claim.

Many of the original datasets do not release their test set. Therefore, we use their original split of train/dev sets as our training and evaluation sets. We also standardize the naming of labels as supports , refutes , and NEI . We visualize the global structure of the datasets with tSNE van der Maaten and Hinton ( 2008 ) and analyze the domain divergence in Appendix B .
Zero/Few-shot Generalization
We now explore the generalization ability of fact verification models across the 11 datasets. We first formulate the task of zero/few-shot generalization.
Task Formulation.
Given a claim \mathcal{C} and a piece of evidence \mathcal{P} as inputs, a fact verification model \mathcal{F} predicts a label \mathcal{Y} to verify whether \mathcal{C} is supported, refuted, or can not be verified by the information in \mathcal{P} . In the zero-shot generalization setting, we train models on one source FV dataset, and then evaluate its performance on a target test set, without any additional training data in the target dataset. In the few-shot generalization setting, we assume we have a small amount of target training examples.
Fact Verification Model.
We use the RoBERTa-large Liu et al. ( 2019 ) as the benchmark model for our study since it has achieved state-of-the-art results in many FV datasets. We concatenate the claim and evidence ( [CLS] claim [SEP] evidence ) and use it as input for a classification task to predict the label of the claim. We use the roberta-large (355M parameters) model provided by the HuggingFace library 2 2 2 https://huggingface.co/roberta-large .
Zero-shot generalization results
Table 2 shows the zero-shot generalization results in macro-averaged F1 for the 3-class fact verification task on all the datasets that have supports/refutes/NEI labels, where we partition by dataset group: Group I, top (datasets with artificial claims); Group II, bottom (datasets with natural claims).
In general, the RoBERTa model generalizes poorly in this zero-shot setup. Compared with the in-domain performance (training and testing on the same dataset), the best zero-shot generalization performance shows a large drop of 20.80% on average. This shows that the FV model overfits to the particular dataset and generalizes poorly to unseen datasets. This validates prior work that shows the neural models are brittle when encountering out-of-distribution data. Taking a closer look, we further explore several research questions specific to fact verification behind this general trend.
Do artificial claims and natural claims generalize to each other?
The bottom left of Table 2 shows that the model trained on natural claims generalizes badly to datasets with artificial claims, with an average F1 drop of 72% relative to the in-domain performance on the three artificial datasets. In contrast,
with natural claims 3 3 3 For fair comparison, we don’t count the dataset pairs with the same data source, e.g. , (SciFact-sent, SciFact-para) , the model generalizes better, with an average F1 drop of 56% (bottom right).
This observation supports the argument that artificial claims and natural claims have substantial differences, e.g. , Wikipedia vs. real-world domains, high vs. less lexical overlap, and simple vs. diverse reasoning types, as discussed in § 2.2 and related works Wadden et al. ( 2020 ); Saakyan et al. ( 2021 ) .

However, a surprising and counter-intuitive observation is that the model trained on artificial claims generalizes quite well to natural claims. As shown by the top right section, the average F1 drop narrows to 36.9% when generalizing from artificial to natural claims, markedly better than when generalizing between natural claim datasets (56% average drop). In particular, when trained on FEVER-sent , the model achieves the best generalization results on 3 out of 5 datasets with natural claims.
However, we will show in the following that the large size of artificial claims contributes a lot to its good generalization performance.
Does generalization improve with more data?
To examine whether good generalization on the FEVER and VitaminC datasets comes from their large dataset size, we conducted an experiment controlling for data size. Here, we only take 800 examples for each dataset to train the model. We show the zero-shot generalization results between the five datasets with sentence-level evidence in Table 4 . The results on all datasets are shown in Table 10 in Appendix C .

We find that the model trained on natural claim datasets (Group I) can generalize to other natural claims slightly better than the model trained on artificial claim datasets (Group II) in this controlled setting. This confirms that dataset size contributes a lot to generalization ability. Tables 2 and 4 together show that Wikipedia-based artificial claims still generalize well to natural claims in real-world domains with the growth of dataset size, although crowd-sourced claims have been criticized to have strong annotation bias and cannot represent real-life misinformation Schuster et al. ( 2019 ) .
Which type of label is more difficult to verify?
Table 5 shows the breakdown of the class-wise F1 score. For each dataset, we show the average class-wise F1 when training the model on other datasets (zero-shot) and the class-wise F1 for training on the same dataset (in-domain). The results show that the refutes claim has the worst prediction score (in bold) almost for all datasets, in both the zero-shot and the in-domain setting. The in-domain results are in line with the empirical observation that Jiang et al. ( 2020 ) it is often ambiguous to differentiate between refutes and NEI claims even for trained human annotators. This difficulty still maintains in the zero-shot setting and harms the generalization results.
What is the impact of evidence length?
From Table 2 , we find that fact verification in a dataset with document-level evidence is more difficult than in the same dataset with sentence-level evidence (an average of 13.29% drop of in-domain F1). This is understandable since document-level evidence requires the model to additionally filter out irrelevant information. Climate-FEVER suffers the largest F1 drop of 31.86%, compared with the slight performance drop on FEVER (4.3%) and SciFact (3.72%).
A possible reason is that Climate-FEVER’s document-level evidence consists of different (even contradictory) evidence sentences, which requires the model to reason over multiple sentences instead of just selecting the most relevant one.

In terms of generalization, the datasets with sentence-level evidence in general achieve better generalization results than other datasets compared to their doc-level versions. For example, C-FEVER-sent generalizes better than C-FEVER-para on 5 of the 6 datasets excluding themselves. Models trained on sentence-level datasets generalize well to other document-level datasets, but the converse is not true. These results indicate that training the FV model on more fine-grained evidence yields better generalization. This is consistent with the intuition that providing fine-grained evidence eases models’ learning in FV, showing the importance of accurate evidence retrieval.
Zero-shot generalization for binary FV
Many works Jiang et al. ( 2020 ); Saakyan et al. ( 2021 ) do not consider NEI claims due to their ambiguity. To explore whether our previous observations also hold for the task of binary fact verification , we evaluate the generalization results for all 11 datasets using only the supports and refutes claims for training and evaluation, shown in Table 3 .
In this setting, artificial claims also generalize well to natural claims in other domains. In 6 of the 7 datasets with natural claims, the best generalization score is from a model trained on artificial claims. This also holds for the evidence length: datasets with sentence-level evidence tend to generalize better than document-level datasets. Finally, compared with the three-class result in Table 2 , generalization improves a lot on Climate-FEVER, SciFact, and PubHealth. The reason is that the model struggles in distinguishing between refutes and NEI claims in these datasets, as reflected by Table 5 . Therefore, they benefit a lot from removing the NEI label.
Few-shot generalization results
We now consider the few-shot generalization setting, assuming access to a small number of examples from a target dataset (50 for each class in our experiment). We pre-train a model on a source dataset and then fine-tune it on the target dataset. Our goal is to analyze whether pre-training improves performance compared to training on the target alone.

Table 6 shows the macro-F1 on the evaluation set of all datasets. The rows “SELF-few-shot” and “SELF-full” show the performance of direct training on the 150 samples of the target dataset and the full target training set, respectively (without pre-training on the source dataset). Generally, pre-training on a source FV dataset and fine-tuning to the target outperform “SELF-few-shot” on all 5 datasets and “SELF-full” on 3 out of 5 datasets. This shows that pre-training on a related FV dataset helps to reduce the demand for human-annotated training data in the target domain.

Second, FEVER-sent obtains good generalization performance in all evaluation datasets. This strengthens our finding in Section 3.1 that FEVER generalizes well to datasets with natural claims in real-world domains. Last, after finetuning, we see a dramatic improvement in performance compared to Table 2 . This highlights that current models over-fit the data they are trained on, and small amounts of data from the target distribution can overcome this generalization gap.
Improving Generalization
We then investigate two ways to improve the generalization ability of fact verification: 1) incorporating domain knowledge via pretraining on specialized domains, and 2) automatically generating training data via data augmentation.
Pretraining on Specialized Domains
In-domain knowledge is essential for fact-checking in specialized domains. For example, virology background knowledge is required to verify scientific claims regarding COVID19 Wadden et al. ( 2020 ) . When generalizing an FV model from one domain to another, how to endow the model with such in-domain knowledge is a challenging subject worthy of long-term study. Here we explore one simple solution: initializing the model with language models pretrained on specialized domains.

In Table 7 , we show the zero-shot generalization performance when initializing the FV model with BioBERT Lee et al. ( 2020 ) (pretrained on biology literature) and SciBERT Beltagy et al. ( 2019 ) (pretrained on scholarly documents). Our goal is to explore whether pretraining on specialized domains helps the generalization. To eliminate the impact of other factors such as the model size, we use the BERT model Devlin et al. ( 2019 ) as the baseline, since BioBERT and SciBERT are both based on the BERT model.

We find that BioBERT and SciBERT both outperform the BERT on the generalization scores in Climate-FEVER, SciFact, and PubHealth, with an average improvement of 21.39% and 12.69% in F1, respectively. However, their performance on Wikipedia-based datasets (FEVER and VitaminC) is relatively worse with BERT (-2.6% and -17.7% for BioBERT and SciBERT, respectively). This confirms the generalization of FV in certain domains ( e.g. , science) can be improved with the language models pretrained on relevant domains ( e.g. , scientific papers). We have similar observations for the few-shot generalization setting, shown in Table 11 in Appendix D . Despite the positive results, a suitable pretraining model in certain domains ( e.g. , tweets) is often unavailable. Moreover, this requires re-training the FV model during domain transfer. Therefore, how to develop a more accessible and less expensive way to incorporate in-domain knowledge required for fact-checking still requires further investigation.
Related Work
To overcome the proliferation of misinformation, a great amount of progress has been made in the area of automated fact verification. For modeling, pretraining-based models Nie et al. ( 2019 ); Stammbach and Neumann ( 2019 ); Zhao et al. ( 2020 ); Soleimani et al. ( 2020 ) have been used for better text representation and have achieved promising performance. Graph-based models Zhou et al. ( 2019 ); Liu et al. ( 2020 ); Zhong et al. ( 2020 ) are used to facilitate the reasoning over multiple pieces of evidence. However, most existing models rely on large-scale in-domain training data, which is often unrealistic for every domain that demand fact checking. In this paper, we aim to address this by working towards a generalizable fact verification system that can adapt to different domains with zero or few samples in the target domain.

For datasets, various fact-checking datasets representing different real-world domains are proposed, including both naturally occurring Augenstein et al. ( 2019 ); Gupta and Srikumar ( 2021 ); Saakyan et al. ( 2021 ); Lu et al. ( 2023 ) and human-crafted Thorne et al. ( 2018 ); Sathe et al. ( 2020 ); Schuster et al. ( 2021 ); Atanasova et al. ( 2022 ) fact-checking claims. While these FV datasets focus on different domains, there is still a substantial overlap in the abilities required to verify claims across these datasets. However, little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. Similar studies have been done in other NLP tasks Talmor and Berant ( 2019 ); Hardalov et al. ( 2021 ) , while it is less investigated in fact verification. In this paper, we bridge this gap by conducting a comprehensive study of generalization and transfer across existing FV datasets, revealing several key factors for better generalization.
Conclusion and Future Work
In this work, we perform a thorough empirical investigation of zero- and few-shot generalization over 11 fact verification datasets. Moreover, we conduct an exhaustive analysis and highlight the most important factors influencing the generalization performance. We further empirically explore two ways to improve generalization in fact verification. We highlight several practical takeaways:

\bullet Overall, the FV model generalizes poorly to unseen datasets compared with in-domain evaluation. However, performance is largely improved by fine-tuning on the target data.

\bullet Artificial claims can also generalize well to natural claims with an increase of dataset size.

\bullet Model trained on sentence-level evidence generalize better than document-level evidence.

\bullet The refutes claims are the most difficult to verify among the three labels.

\bullet Domain-specific pretraining and data augmentation consistently improves generalization performance, but they also leave unsolved challenges.

In future work, we plan to experiment with more datasets, including non-English ones. We will also explore the generalization of other sub-tasks in fact-checking, e.g. , claim detection, evidence retrieval.
Limitations
In our study, we primarily focused on assessing the generalization capabilities of Transformer-based models, such as RoBERTa. However, we did not extend our evaluation to include zero- and few-shot learning performance on large language models (LLMs) like InstructGPT Ouyang et al. ( 2022 ) and GPT-4 OpenAI ( 2023 ) , due to the high experimental costs. Recently, these LLMs have demonstrated impressive few-shot learning capacities across a variety of natural language processing tasks, including few-shot fact-checking Pan et al. ( 2023 ) . However, they are API-based and function as black-box models. This restricts our ability to delve deeper into their behavior, given that we cannot access their model weights or fine-tune them directly. On the contrary, Transformer-based models are open-sourced and replicable, providing a wealth of opportunity for more profound insights into our study and paving the way for future research.
Acknowledgements
This research / project is supported by the Ministry of Education, Singapore, under its MOE AcRF TIER 3 Grant (MOE-MOET32022-0001). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.
Figure 1: tSNE plot of [CLS] representations of each dataset; highlighted points denote cluster centroids.; tSNE plot of [CLS] representations of each dataset; highlighted points denote cluster centroids.
(a) Climate-FEVER-sentence; (b) SciFact-sentence; (c) PubHealth; (d) COVID-Fact; Confusion matrices (normalized over columns) of generated claims on four datasets. The desired label is the input label for our claim generation model (BART-gen) and the actual label is the human-annotated label. “Unclassified” means that the generated claim is incomplete or unreadable.
List of the 11 fact verification datasets for our study and their characteristics.
Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).