SlowGuess/NeurIPS_2023_OCR huggingface dataset:
AVERItec: A Dataset for Real-world Claim Verification with Evidence from the Web
Michael Schlichtkrull*, Zhijiang Guo*, Andreas Vlachos
Department of Computer Science and Technology, University of Cambridge
{mss84,zg283,av308}@cam.ac.uk
Existing datasets for automated fact-checking have substantial limitations, such as relying on artificial claims, lacking annotations for evidence and intermediate reasoning, or including evidence published after the claim. In this paper, we introduce AVERITEC, a new dataset of 4,568 real-world claims covering fact-checks by 50 different organizations. Each claim is annotated with question-answer pairs supported by evidence available online, as well as textual justifications explaining how the evidence combines to produce a verdict. Through a multi-round annotation process, we avoid common pitfalls including context dependence, evidence insufficiency, and temporal leakage, and reach a substantial inter-annotator agreement of  $\kappa = 0.619$  on verdicts. We develop a baseline as well as an evaluation scheme for verifying claims through question-answering against the open web.
Fact-checking is considered crucial for limiting the impact of misinformation [Lewandowsky et al., 2020]. Unfortunately, not enough resources are available for manual fact-checking. Automated fact-checking (AFC) has been proposed as an assistive tool for fact-checkers, moderators, and citizen journalists to facilitate it [Cohen et al., 2011, Vlachos and Riedel, 2014], inspiring applications in journalism [Miranda et al., 2019, Dudfield, 2020, Nakov et al., 2021] and other domains, e.g. science [Wadden et al., 2020].
Substantial progress has been made on common benchmarks, such as FEVER [Thorne et al., 2018] and MultiFC [Augenstein et al., 2019]. Nevertheless, existing resources have recently come under criticism. Many datasets (for example, Thorne et al. [2018], Schuster et al. [2021], Aly et al. [2021]) contain purpose-made claims derived from sources such as Wikipedia, and are thus unlike real-world claims checked by journalists. Further, in these datasets, refuted claims are produced by corrupting existing sentences. Datasets that do contain real-world claims either lack evidence annotation [Wang, 2017], or annotate it superficially using automated means, resulting in issues such as including evidence published days or weeks after the investigated claims [Glockner et al., 2022].
To address these limitations we introduce AVERITEC (Automated VERIFICATION of TExtual Claims), which combines real-world claims with realistic evidence retrieved from the web, as well as justifications for veracity labels. We formulate retrieval as question generation and answering, providing a structured representation of the evidence and reasoning supporting or refuting the claim. The free-text justifications detail how the evidence is used to reach the verdict, including cases of conflicting evidence, matching best practises for human fact-checkers [Borel, 2016]. In constructing AVERITEC, we ameliorate three issues afflicting existing datasets with real-world claims:
1. Context Dependence: Ousidhoum et al. [2022] found that claims in some datasets based on fact-checking articles (e.g., Fan et al. [2020]) cannot be verified without additional
Figure 1: Diagram for our annotation process. Claims are first selected and normalized. Then, two rounds of question-answer pair generation and evidence sufficiency check ensure high-quality evidence annotation.
information from the articles they were extracted from. This could for example be due to unresolved coreference or ellipsis, e.g. in "unemployment is rising" it is unclear which geographical/temporal context is being considered.
2. Evidence Insufficiency: Glockner et al. [2022] found that labels in some datasets (e.g., Hanselowski et al. [2019]) often do not match the annotated evidence, because they rely on e.g., assumptions about the speaker of the claim. This is significantly different from not enough evidence verdicts, which is a label for claims where evidence could not be found.
3. Temporal Leaks: Glockner et al. [2022] found that annotations in some datasets (e.g., Augenstein et al. [2019]) contain leaks from the future. For example, a claim from January might be annotated with evidence from March that year. Leaks can also happen between splits, if e.g., evidence for a training claim from 2021 also pertains to a test claim from 2020.
We address (1) through an initial normalisation step, where annotators enrich claims with necessary information from the fact-checking article. We verify the adequacy of this, and address (2), by combining multiple rounds of annotation with a "blind" quality control step, re-annotating any claims with insufficient evidence. We address (3) by restricting annotators to evidence documents published before the claim, and by ordering our training, development, and test splits temporally. With the increasing reliance on large pretrained language models, temporal ordering provides an additional benefit: if the training data for the language model is cut off before the temporal start of the test set, leaks from pretraining cannot occur either.
AVERITEC consists of 4,568 examples, collected from 50 fact-checking organizations using the Google FactCheck Claim Search API<sup>2</sup>; itself based on ClaimReview<sup>3</sup>. Our annotation, which involved up to five annotators per claim, resulted in substantial inter-annotator agreement, with a free-marginal  $\kappa$  of 0.619 [Randolph, 2005]. We further develop a baseline to explore the feasibility of the task, relying on Google Search, BM25 [Robertson and Zaragoza, 2009], retrieved in-context prompts [Liu et al., 2022, Rubin et al., 2022], and a trained stance detection model. AVERITEC is the first AFC dataset to provide both question-answer decomposition and justifications, as well as avoid issues of context dependence, evidence insufficiency, and temporal leaks. Our dataset and baseline are available under a CC-BY-NC-4.0 license at https://github.com/MichSchli/AVeriTeC.
Sourcing real-world claims from fact-checking articles is popular (e.g. Wang [2017]), as extracting claims from fact-checkers guarantees checkworthiness. That is, any claim included in the resulting dataset is deemed interesting enough to be worth the time of a professional journalist (see Hassan et al. [2015]). Previous real-world datasets either lack annotations for evidence, or suffer from context dependence, evidence insufficiency, or temporal leaks. Further, they do not provide annotations for intermediate steps, and only a minority (Alhindi et al. [2018], Kotonya and Toni [2020]) provide justifications. A comparison between AVeriTeC and prior datasets can be seen in Table 1.
$^{2}$ https:// toolbox.google.com/factcheck/apis, available under a CC-BY-4.0 license.
<sup>3</sup>https://www claimreviewproject.com/
Dataset	Source	Claim Type	Independence	Sufficient	Evidence Unleaked	Retrieved
FEVER [Thorne et al., 2018]	Wikipedia	Synthetic	✓	✓	N/A	✓
VitaminC [Schuster et al., 2021]	Wikipedia	Synthetic	✓	✓	N/A	✓
FEVEROUS [Aly et al., 2021]	Wikipedia	Synthetic	✓	✓	N/A	✓
SciFact [Wadden et al., 2020]	Science	Synthetic	✓	✓	N/A	✓
FM2 [Saakyan et al., 2021]	Game	Synthetic	✓	✓	N/A	✓
Covid-Fact [Saakyan et al., 2021]	Reddit	Synthetic	✓	✓	N/A	✓
Liar-Plus [Alhindi et al., 2018]	Factcheck	Real	✗	✓	✗	✗
PolitiHop [Ostrowski et al., 2021]	Factcheck	Real	✗	✓	✗	✗
MultiFC [Augenstein et al., 2019]	Factcheck	Real	✗	✗	✗	✓
XFact [Gupta and Srikumar, 2021]	Factcheck	Real	✗	✗	✗	✓
PubMed [Kotonya and Toni, 2020]	Factcheck	Real	✗	✗	✓	✗
WatClaimCheck [Khan et al., 2022]	Factcheck	Real	✗	✗	✓	✗
ClaimDecomp [Chen et al., 2022]	Factcheck	Real	✗	✗	✓	✗
Snopes [Hanselowski et al., 2019]	Factcheck	Real	✗	✗	✓	✗
QABrief [Fan et al., 2020]	Factcheck	Real	✗	✗	✓	✗
ClimateFEVER [Diggelmann et al., 2020]	Web	Real	✗	✗	✓	✓
HealthVer [Sarrouti et al., 2021]	Web	Real	✗	✗	✓	✓
CHEF [Hu et al., 2022]	Factcheck	Real	✗	✓	✗	✓
AVERITEC	Factcheck	Real	✓	✓	✓	✓
Table 1: Comparison of fact-checking datasets. Source indicates where the claims are collected from, such as Wikipedia, or fact-checking articles (Factcheck). Type indicates whether the claims are synthetic or real-world. Independence indicates whether the claim is context independent. Sufficient indicates whether the evidence can provide sufficient information. Unleaked means whether the evidence contains leaks from the future and retrieved denotes whether the dataset involves evidence retrieval instead of relying on pre-retrieved passages e.g. the fact-checking article.
Beyond evidence insufficiency and temporal leakage, Glockner et al. [2022] also found that many examples require a source guarantee to refute, i.e. a guarantee that the claimant's underlying reason for making the claim is known to the debunker. For example, evidence against the claim "COVID-19 vaccines may kill sharks" can only be found when incorporating the underlying reasoning of the claimant, that the manufacturing of COVID-19 vaccines requires a chemical extracted from sharks. We do not explicitly provide such a guarantee; however, as each claim in AVERITEC is annotated with the original claimant, these underlying reasons can be recovered through question-answer pairs.
Question-answer decomposition is considered a promising strategy; Yang et al. [2022] proposed such a model even without a dataset of annotated question-answer pairs. Two recent datasets cast fact-checking as question-answering: Fan et al. [2020] and Chen et al. [2022]. However, Fan et al.'s [2020] question-answer pairs were only written to add relevant context, not to capture entire the fact-checking process, and thus lack evidence sufficiency. Ousidhoum et al. [2022] furthermore identified context dependence as a significant concern in Fan et al. [2020]: many questions are impossible to generate given only the claim, as they refer to entities and events only mentioned in the original fact-checking article. Chen et al. [2022] did – like us – attempt to ensure evidence sufficiency. However, they take no steps to verify their success. Furthermore, their evidence is taken directly from the fact-checking articles which are written after the claim, thus exhibiting temporal leakage.
3 Annotation Structure
Our dataset consists of 4,568 real-world claims annotated with question-answer pairs representing the evidence, a veracity label, and a textual justification describing how the evidence supports the label. An example can be seen in Figure 2.
Reasoning about evidence is represented through questions and answers. Questions may have multiple answers, a natural way to show potential disagreements in the evidence. Questions can refer to previous questions, allowing for multi-hop reasoning. Answers (other than "No answer could be found.") must be supported by a source url linking to a web document. To avoid sources disappearing from the web, we cache all pages used as evidence in the internet archive<sup>4</sup>.
Claims in AFC datasets are typically supported or refuted by evidence, or there is not enough evidence. We add a fourth class: conflicting evidence/cherry-picking. This covers both conflicting evidence, and
4https://archive.org
Figure 2: Example claim from AVERITEC. As opposed to previous datasets, ours naturally combines question-answer pairs that break down the evidence retrieval with justifications that show how evidence leads to verdicts.
technically true claims that mislead by excluding important context. For real-world claims, sources may interpret events differently, and therefore legitimately disagree. This differs from Schuster et al. [2021], which studies claims for which the evidence has been revised. Adding a fourth class has also recently been discussed in the context of natural language inference [Jiang and Marneffe, 2022], although there it is usually ambiguity in the premise or hypothesis leading to the conflict.
AVERITEC also provides textual justifications that explain how verdicts are reached from the evidence. Where sources disagree, best practices for established fact-checkers is to provide a textual explanation of why the claim misleads [Uscinski and Butler, 2013, Amazeen, 2015]. These justifications can be substantial [Toulmin, 1958], i.e. they may introduce logical leaps supported by commonsense or inductive reasoning beyond the retrieved evidence. For example, if a claim states that  $50\%$  of a population group were vaccinated by February 1st, and evidence shows only  $33\%$  had been vaccinated by January 31st, the justification may reason that a  $50\%$  rate one day later is unlikely.
We include several fields of metadata: the speaker of the claim, the publisher of the claim, the date the claim was published, and the location most relevant to the claim. These can be used to support questions, answers, and justifications. We also annotate the claim type and fact-checking strategy of each claim. Type represents common aspects, e.g., whether claims are about numerical facts; strategy represents the approach of the fact-checkers, e.g., whether they relied on expert testimony. Types and strategies should not be used as input to models (at inference time), but can provide useful data for analysis.
4 Annotation Process
Starting from 8,000 fact-checking articles, we first identified and discarded 537 duplicates and 802 paywalled or dead articles. We passed the remainder through a five-phase pipeline – see Figure 1. First, an annotator extracts claims and relevant metadata from each article, providing context independence. Second, an annotator generates questions and answers them using the web. These annotators also choose a temporary verdict. Third, a different annotator provides a justification and a verdict based solely on the annotated question-answer pairs; this serves as an evidence sufficiency check. Any claim for which the two verdicts do not match is passed through the last two phases again. If the verdicts still disagree, the claim is discarded. Different annotators were used for each claim in each phase – i.e., no annotator saw the same claim twice. Our annotation was performed by the company Appen<sup>5</sup>; details and annotation guidelines can be found in Appendices C and J.
Claim Extraction & Normalisation Annotators extracted the central claims from each fact-checking article, enriching them with the necessary context. This is necessary as many fact-checking articles cover multiple claims (e.g., several rumours circulating about an event). Further, some claims lack adequate contextualization [Ousidhoum et al., 2022]. For example, the claim "we have 21 million unemployed" requires coreference resolution. After claim extraction, we discarded speculative claims,
5https://appen.com/
Split	Train	Dev	Test
Claims	3068	500	1000
Questions / Claim	2.60	2.57	2.57
Reannotated (%)	28.1	24.4	25.1
End date	25-08-2020	31-10-2020	22-12-2021
Labels (S / R / C / N) (%)	27.6 / 56.8 / 6.4 / 9.2	24.4 / 61.0 / 7.6 / 7.0	25.5 / 62.0 / 6.3 / 6.2
Table 2: Descriptive statistics for the dataset. Statistics for labels are split into supported (S), refuted (R), conflicting evidence/cherrypicking (C), and not enough evidence (N). For the dev and test splits, the start date is the end date of the previous split; the train set has no start date.
i.e. unverifiable statements about future events or personal opinions, and multimodal claims, i.e. claims where the type or the strategy inherently involves modalities beyond text.
Question Generation & Answering Annotators generate questions and answer them providing evidence about the claim. The aim is to "deconstruct" the reasoning of the fact-checker into a QA-structure, extracting question-answer pairs that match the content of their enquiries. This makes the process better amenable to annotators who are not trained journalists, and provides structured representations for model development and evaluation. Each question must be accompanied by an answer (or marked unanswerable), and answers must be backed by sources. We advise annotators that extractive answers are preferred, but abstractive answers are also allowed. Annotators in this phase are asked to provide a verdict based on their retrieved evidence, possibly different from the one in the fact-checking article.
As annotators follow the fact-checking articles, the ideal evidence sources for answer are the documents linked from the articles. For answers are not found in the linked documents, we provide access to a custom Google search bar. This search bar is restricted to show documents published only before the claim date, in contrast to prior work [Fan et al., 2020]. Furthermore, unlike Alhindi et al. [2018] and Hanselowski et al. [2019], the fact-checking article itself cannot be used as evidence.
We note that, where annotators could not find the publication date of the claim, their instructions were to use the publication date of the fact-checking article instead. As the process of fact-checking can take journalists several days, there is a window in which news about the claim can be published that we cannot prevent from being used as evidence. Further, the Google API does not always correctly infer the publication dates of articles. As such, our guarantee against temporal leakage is approximate (see Section 8). For the development set, we estimated that around  $6\%$  of answers are sourced from a fact-checking domain. This is primarily because of the earlier publication of an article about the same claim by a different fact-checking organization.
Evidence Sufficiency Check Once question-answer pairs have been generated, we present each claim along with its question-answer pairs to a new annotator. This annotator does not see the fact-checking article. They then produce a verdict and a textual justification for it. We compare this verdict to the one produced by the question and answer annotator, and if they disagree we repeat the question-&-answer generation and sufficiency check steps with new annotators to improve the evidence and the verdict.
5 Dataset Statistics
We split our dataset into training, validation, and test data temporally (see Table 2). Claims have on average 2.60 questions, and questions have on average 1.07 answers. Most answers  $(53\%)$  are extractive, followed by abstractive  $(26\%)$  and boolean  $(17\%)$  answers. A few questions  $(4\%)$  are marked unanswerable as no available evidence could be found by the annotators. Statistics for source document modality, fact-checker strategy, and claim type can be seen in Appendix F. We note that AVERITEC is somewhat unbalanced – the majority of claims are refuted. This is a consequence of our choice to rely on fact-checking articles, as journalists tend to pick false or misleading claims to work on. Our dataset includes all ClaimReview claims labeled supported (or any variation thereof, e.g. true) within our temporal limits (for more detail see Appendix I.1).
To measure the inter-annotator agreement of our annotation scheme, we had a second set of annotators re- annotate 100 claims from the dataset. In this we assumed the claim extraction and normalization step was done, and the annotators repeated the question and answer generation phase. Since we have an unbalanced dataset, following Kazemi et al. [2021], Ousidhoum et al. [2022], we therefore measure agreement using Randolph's [2005] free-marginal multirater  $\kappa$ , an alternative to Fleiss'  $\kappa$  more suitable for unbalanced datasets [Warrens, 2010]. Our observed agreement score of  $\kappa = 0.619$  is substantial, and compares well to those for other "hard" fact-checking annotation tasks, e.g. Kazemi et al. [2021], who got between  $\kappa = .30$  and  $\kappa = .63$  depending on the language. Using Fleiss'  $\kappa$  [Fleiss, 1971], we get an agreement score of 0.503.
6 Evaluation
To evaluate models on AVERItec, we follow Thorne et al. [2018] and score retrieved evidence based on agreement with gold evidence, and give credit to veracity predictions (and justifications) only when correct evidence has been found. However, unlike in FEVER and other datasets using a closed source of evidence such as Wikipedia, AVERItec is intended for use with evidence retrieved from the open web. Since the same evidence may be found in different sources, we cannot rely on exact matching to score retrieved evidence. As such, we instead rely on approximate matching.
To measure how well a set of generated questions and answers match the references, we rely on a pairwise scoring function  $f: S \times S \to \mathbb{R}$ , where  $S$  is the set of sequences of tokens. We then use the Hungarian Algorithm [Kuhn, 1955] to find an optimal matching of generated sequences to reference sequences. Formally, let  $X: \hat{Y} \times Y \to \{0, 1\}$  be a boolean function denoting the assignment between the generated sequences  $\hat{Y}$  and the reference sequences  $Y$ . Then, the total score  $u$  is calculated as:
$$
u _ {f} (\hat {Y}, Y) = \frac {1}{| Y |} \max  \sum_ {\hat {y} \in \hat {Y}} \sum_ {y \in Y} f (\hat {y}, y) X (\hat {y}, y) \tag {1}
$$
If  $f$  is an exact match, we recover the evidence recall score from Thorne et al. [2018]. Our metric is as such a generalization of theirs to the approximate case. In our evaluation, we use the implementation of METEOR [Banerjee and Lavie, 2005] in NLTK [Bird et al., 2009] as the scoring function  $f$  (and refer to our evidence scoring function as Hungarian METEOR hereafter), but any suitable pairwise metric could be used. We chose METEOR over other alternatives (e.g., ROUGE [Lin, 2004]) as it is known to correlate well with human judgments of similarity [Fomicheva and Specia, 2019]. We do not employ a precision metric, as we want to avoid penalizing systems for asking additional relevant information-seeking questions – however, all systems are limited to a maximum of  $k = 10$  question-answer pairs.
We conduct the evaluation with Hungarian METEOR twice: once using only the questions as input sequences, and once using the concatenation of questions and answers. A subtask of AVERITEC is to ask the right questions – as we discuss in Section 7.2, good questions are very useful as search queries even if not accompanied by a good answer. Finding the right angle to criticize a claim is a substantial task by itself; it covers the creativity factor in retrieval discussed by Arnold [2020]. Including the question-only score allows comparison of systems along this axis as well. To evaluate veracity predictions and justifications, we use a cutoff of  $f(\hat{y}, y) \geq \lambda$  to determine whether correct evidence has been retrieved (using concatenated questions and answers); any claim for which the evidence score is lower receives veracity and justification scores of 0.
Many claims can be verified through alternative evidence formulations. Taking an example from the 100 claims annotated twice for Section 5, one annotator might produce the question-answer pair "Where did South Africa rank in alcohol consumption? In 2016, South Africa ranked ninth out of 53 African countries." while another produces "What's the average alcohol consumption per person in South Africa? 7.1 litres". These may both be valid ways of establishing the relative levels of alcohol consumption between South Africa and other countries. We recognize that our evaluation approach can penalize systems for selecting an alternative evidence path; nevertheless, we argue that automatic evaluation on this task is helpful in model development. We note that a similar phenomenon was seen for the original FEVER dataset [Thorne et al., 2018], despite the artificial claims and the exclusive use of Wikipedia as an evidence source. There, the authors suggested crowd-sourced human evaluation as a more reliable alternative – we echo their recommendation. Our annotation process hints at a potential setup for human evaluation: judging if a body of evidence is sufficient for a verdict is exactly what our annotators did during the evidence sufficiency check phase.
Model	Q only	Q + A	Veracity @ (.2/.25/.3)	Justifications @ (.2/.25/.3)
No search	0.19	0.11	0.03	0.02	0.01	0.02	0.01	0.01
Gold evidence	1.00	1.00	0.49	0.49	0.49	0.28	0.28	0.28
AVERITEC -BLOOM-7b	0.26	0.21	0.23	0.15	0.00	0.11	0.07	0.05
gpt-3.5-turbo	0.29	0.16	0.17	0.10	0.06	0.06	0.04	0.02