SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Results for the AVERITEC baseline and ChatGPT (gpt3.5-turbo). Retrieval scores both for questions and for questions + answers are given in terms of Hungarian METEOR score. Veracity and justifications are scored using accuracy and METEOR respectively, in both cases conditioned on correct evidence retrieved at  $\lambda = \{0.2, 0.25, 0.3\}$  (see Section 6). We report results for three versions of the baseline, as discussed in Section 7.2: a version that uses no evidence (no search), a version that uses gold evidence (gold evidence), and the full pipeline described in Section 7.1 (AVERITEC). We also report results for gpt-3.5-turbo (ChatGPT).
We further note that our metric is straightforward to extend to cover evaluation with multiple reference sets. Given a set of sets of question-answer pairs  $R$  representing different questioning strategies, a best-matching score could be computed as  $\max_{Y \in R} u_f(\hat{Y}, Y)$ . As such, if AVERITEC was expanded with annotations for alternative questioning strategies, our metric could score models on these as well.
To understand how our metric should be interpreted, we also computed Hungarian METEOR scores between the question-answer pairs generated during the two rounds of annotation used for inter-annotator agreement in Section 5. At 0.28 for questions and 0.22 for questions and answers, these results are quite low, highlighting the difficulty of automatic evaluation for this task. Investigating claims with low agreement scores, we see that these are actually often a result of different annotators using different evidence sources for the same verdict, or phrasing equivalent question-answer pairs differently. Based on our observations of human annotations, we recommend  $\lambda = 0.25$  as an appropriate cutoff value for our metric. We refer to this metric (for veracity prediction) as AVERITEC score.
7.1 Baseline Model
Our baseline is a pipeline consisting of several components: generation of search questions, search, generation of questions given retrieved evidence, reranking of retrieved evidence, veracity prediction, and generation of justifications. For each step in the pipeline, we carried out experiments with several models. Using our training set, we finetuned, respectively, BERT-large [Devlin et al., 2019] for classification (340M parameters) and BART-large [Lewis et al., 2020] for generation (406M parameters). We furthermore tried a few-shot setup, prompting a large language model (LLM) with retrieved in-context examples [Liu et al., 2022, Rubin et al., 2022] from our training set. Here, we tried the 7b parameter BLOOM model [Scao et al., 2022] and the 13b parameter Vicuna model [Chiang et al., 2023]. We limited ourselves to relatively small models, as we consider runnability crucial for a baseline: all our components can be run on a single Nvidia A100 GPU. As such, our baseline strikes a balance between performance and computational cost.
Search Given a claim, we retrieve evidence documents from the internet using the Google Search API. Following Karadzhov et al. [2017], we use a reduced version of the claim keeping only verbs, nouns, and adjectives as the search term. As we did during annotation, we limit the API to documents published before the estimated date of the claim. We keep all unique documents in the first 30 search results. Initial experiments showed that questions were very useful as additional search terms. As the model does not have access to gold questions during testing, we instead generate questions. We experimented with three models: BART-large, bloom-7b, and Vicuna-13b. Surprisingly, BLOOM performed the best, beating the newer and larger Vicuna; we attribute this primarily to greater topical diversity in the set of questions generated, and thus greater variety in the retrieved evidence pages. We rely on prompting with retrieved in-context examples [Liu et al., 2022, Rubin et al., 2022]. We use BM25 [Robertson and Zaragoza, 2009] to find the 10 most similar claims from the training set,
and use their annotated questions to construct a prompt, with which we generate questions for the claim using BLOOM. We tested  $\{1,3,5,10\}$  in-context examples, finding 10 to perform the best. The full prompt can be seen in Appendix D.1. We add any new unique documents retrieved by searching for these generated questions. This can be seen as a form of query expansion [Mao et al., 2021].
Evidence Selection Once a set of evidence documents has been created for each claim, we pick  $N = 3$  sentences from this set. We first apply a coarse filter to the evidence set, ranking evidence sentences by BM25 score computed against the claim, and discard those outside the top 100. Then, we generate a question for each sentence that is answerable by that sentence, again using BLOOM. We tested {1,3,5,10} in-context examples, finding 10 to perform the best. The full prompt can be seen in Appendix D.2. We then re-rank these question-answer pairs to find the ones most relevant for the claim, using a finetuned BERT-large model [Devlin et al., 2019] (for more details, see Appendix E.1). This somewhat counter-intuitive strategy of retrieving first and then generating questions can be seen as using the generated questions to bridge claims to distantly related evidence sentences. Our approach is similar to the document expansion strategy proposed for question answering in Nogueira et al. [2019], except applied for reranking rather than the initial retrieval step.
Veracity Prediction Once question-answer pairs have been generated, we produce verdicts through a stance detection strategy inspired by past work on filtering evidence [Hanselowski et al., 2019]. We use a finetuned BERT-large model to label each question-answer pair as supporting, refuting, or being irrelevant to the question (for more details, see Appendix E.2). We then deterministically label the claim as follows: 1) if the claim has both supporting and refuting evidence, label it conflicting evidence/cherrypicking. 2) If the claim has only supporting question-answer pairs, label it supported; similar for refuted. 3) Otherwise, label the claim not enough evidence. We tested three different models for veracity prediction: BERT-large, bloom-7b1, and Vicuna-13b. We found BERT to perform better by a slight margin; using gold evidence, we obtained macro-F1 scores of .49, .43, and .48 for the three models respectively.
Justification Generation The final step is to generate a textual justification for the verdict. Here, we rely on BART-large [Lewis et al., 2020] finetuned on our training set (for more details, see Appendix E.3). We use the concatenation of the claim and the retrieved evidence as input; we tried adding the predicted veracity as well, but saw no improvements to performance. We again tested three models: BART-large, bloom-7b1, and Vicuna-13b, respectively obtaining a METEOR score of .28, .23, and .25. Based on our qualitative analysis of 20 claims, the justifications generated by Vicuna are very good, but the model is penalized for being overly verbose – Vicuna generates 36 tokens on average, compared to 21 in the gold data.
7.2 Results
We evaluate as discussed in Section 6. We include results in Table 3 at three thresholds for comparison, although we encourage  $\lambda = 0.25$ . We compare our baseline to two other models: one without access to search, and one using gold question-answer pairs as evidence.
For the no search model, we use prompting to generate questions, following the approach described in Appendix D.1. We leave all answers as "No answer could be found". Generating answers is not an option, as answers must be supported by sources. We use BERT-large finetuned on the training data to predict veracity labels (without any evidence), and the same prompting strategy as discussed in Section 7.1 to generate justifications. For the gold evidence model, we use the gold question-answer pairs provided by our annotators in the place of generated questions and retrieved evidence. That is, we test only the veracity prediction and justification production components.
Analysing retrieval results on the development set, we still find  $\lambda = 0.25$  to be a good cutoff point for the AVERITEC veracity and justification metrics. For borderline question-answer pairs this threshold is high enough that all important information must be produced to meet it, but there is still some room for paraphrasing and partial evidence.
Our baseline has decent performance at  $\lambda = 0.2$  and  $\lambda = 0.25$ , but does not perform well at higher evidence cutoff points. Because of the structure of our pipeline – generate search terms, retrieve and rerank evidence, generate questions to match the reranked evidence – our baseline struggles to match specific evidence sets. If the retrieved evidence paragraph is very short, e.g., a table cell reading
Model	S	R	C	N	Macro
No evidence	.30	.22	.00	.16	.17
Gold evidence	.48	.74	.15	.59	.49
AVERiTec	.41	.69	.10	.16	.23
gpt-3.5-turbo	.62	.71	.02	.20	.39
Table 4: F1-scores for veracity prediction split across labels: supported (S), refuted (R), conflicting evidence/cherrypicking (C), and not enough evidence (N). We also show the macro-average. Again, we report results for three versions of the baseline (see Section 7.2): a version that uses no evidence (no search), a version that uses gold evidence (gold evidence), and the full pipeline ( AVERITEC ). We also report results for gpt-3.5-turbo (ChatGPT).
"January 24th", the question generation model often lacks context to generate the right question. Further, the baseline cannot generate questions with highly abstractive answers, only questions that can be answered directly by sentences in the supporting sources.
We recognize that the retrieval scores of this baseline are quite close to those of the human annotators seen in Section 6. Nevertheless, for evidence retrieved by our baseline, low scores are much more frequently a result of reliance on wrong evidence, rather than equivalent evidence phrased differently and scored incorrectly. Further research is needed to develop an evaluation capable of recognizing this difference, e.g., a trained metric in the style of BLEURT [Sellam et al., 2020].
The gap between gold evidence and retrieved evidence highlights how retrieval remains challenging, also discussed in Arnold [2020]. Manually analysing 20 examples from the development set, we find that Google search results based on the claim and the generated questions contain useful evidence only in 9/20 cases. If the retrieval system had access to the gold questions for use as search queries, correct evidence would be found in 16/20 of these cases; this highlights the need for further development of retrieval and search systems, and especially query/question generation.
We also report individual  $F1$  scores for each veracity class (as well as a macro average) in Table 4. Our veracity prediction model fails to accurately predict Conflicting Evidence/Cherrypicking most of the time, even with gold evidence. Going through the predictions, we see that precision is very low (10% using gold evidence). Labelling claims as Conflicting Evidence/Cherrypicking if any evidence is classified as having different stance leads to many false positives – often, questions that simply add context to supported claims are (incorrectly) labelled as refuting by the stance detection model.
As a way to improve the stance detection component, we tried to generate additional training data using gpt-3.5-turbo (ChatGPT). We paraphrased each claim in the dataset, using the same evidence. We generated one paraphrase per claim. Then, we trained BERT-large on the concatenated original and paraphrased claims. Unfortunately, this failed to yield additional performance, producing a macro- $F1$  score of .46 on gold data; slightly lower than the .49 obtained using only the original claims. The primary cause is a drop in performance for refuted and not enough evidence, which the model trained on paraphrased data conflates more often.
We further include results in Tables 3 and 4 using ChatGPT. As ChatGPT cannot produce sources to back up its answers, this is not directly comparable to our baseline. Nevertheless, it is an interesting point of comparison. We generate evidence and verdicts with ChatGPT, using the prompt described in Appendix G. We find that ChatGPT outperforms our baseline in terms of pure question generation, but nevertheless received a lower AVERITEC score (veracity prediction at  $\lambda = 0.25$ ). This is a consequence of the missing retriever: generated answers often do not match gold answers (i.e., they are either alternative correct answers, or outright hallucinations).
ChatGPT performs well on veracity, especially for supported claims; but those verdicts are often not supported by valid evidence. For example, for the claim “1 cup of dandelion greens = 535% of your daily recommended vitamin K and 112% of vitamin A,” ChatGPT assigned the verdict supported and generated the evidence string “According to the USDA, 1 cup of chopped dandelion greens provides 535% of your daily recommended vitamin K and 338% of vitamin A, which is higher than the claim”. While the verdict is true, there is no such statement from the USDA, and the actual gold evidence relies on several question-answer pairs and a calculation to arrive at the verdict.
8 Limitations
The evaluation metric we have presented alongside AVERITEC contains a significant limitation: no efforts are made to ensure answers and source documents are consistent. As only  $53\%$  of gold answers are fully extractive, it is expected that abstractive models will be employed. Such models though can hallucinate, and can thus make up answers that are not supported by the underlying sources, which our evaluation metric cannot detect. Further research is needed on evaluation to counteract this, along with research on developing an evaluation strategy that better allows verifying claims correctly with different questioning strategies and evidence documents.
While claims geographically concern regions from all around the world, all fact-checking sources and consequently all claims used in our dataset are in English. Further, as we take claims directly from fact-checking articles, our dataset is subject to any biases present within those articles; notably, for internal fact-checking, Barnoy and Reich [2019] documented a selection bias resulting from journalists rating claims by male sources more credible than female sources.
Finally, we note that our reliance on Google Search to avoid temporal leakage is a noisy process. The dates we rely on are the best estimate computed by Google<sup>6</sup>. As such, while in general evidence documents were available when associated claims were published, there may be exceptions.
9 Ethics Statement
Fact-checking is often envisioned as an epistemic tool, limiting the spread and influence of misinformation. The datasets and models described in this paper are not intended for truth-telling, e.g. for the design of automated content moderation systems. The labels and justifications included with this dataset relate only to the evidence recovered by annotators, and as such are subject to the biases of annotators and journalists; furthermore, the machine learning models and search engine used for the baseline contain well-known biases [Noble, 2018, Bender et al., 2021]. Acting on veracity estimates arrived at through biased means, including automatically produced ranking decisions for evidence retrieval, risks causing epistemic harm [Schlichtkrull et al., 2023].
Annotators for our dataset had access to searching the entire web when finding evidence documents. We curated a list of common misinformative sources by combining several public documents<sup>7</sup>, and flagged search results from these sources. Nevertheless, we did not prevent annotators from using them as evidence. Pointing out that a claim originates from an untrustworthy site is an important fact-checking strategy, and, indeed, our list may well contain false positives. A total of 85 answers in AVERITEC rely on a flagged source; moreover, our list is not complete. Our dataset may as such include misleading examples, and can potentially cause harm if relied on as an authoritative source.
We did not take any steps to anonymise the data. The claims discussed in our dataset are based on publicly available data from journalistic publications, and concern public figures and events – references to these are important to fact-check claims. We did not contact these public figures, or the journalists who published the original fact-checking articles. If any person included in our dataset as a speaker of a claim, as the subject of a claim, or as the author of a fact-checking article a claim is based on requests it, we will remove that claim from the dataset.
We have introduced AVERITEC, a new real-world fact-checking dataset consisting of 4,568 claims, each annotated with question-answer pairs decomposing the fact-checking process, as well as justifications. Our multi-step annotation process guarantees high-quality annotations, providing evidence sufficiency and avoiding temporal leakage; it also results in a substantial inter-annotator agreement of  $\kappa = 0.619$ . We have also introduced and analysed a baseline as well as an evaluation scheme, establishing AVERITEC as a new benchmark.
See https://developers.google.com/search/blog/2019/03/help-google-search-know-best-date-for
We combined the following: https://libguides.castleton.edu/evaluating_news/fake_news, https://www+factcheck.org/2017/07/websites-post-fake-satirical-stories/, and https://en.wikipedia.org/wiki/List_of_fake_news_websites
We would like to thank Zhangdie Yuan, Pietro Lesci, Nedjma Ousidhoum, Ieva Staliunaite, and David Corney for their helpful comments, discussions, and feedback. This research was supported by the ERC grant AVeriTeC (GA 865958). Andreas Vlachos is further supported by the EU H2020 grant MONITIO (GA 965576).
Andreas Vlachos and Sebastian Riedel. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18-22, Baltimore, MD, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-2508. URL https://aclanthology.org/W14-2508.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohen, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.609. URL https://aclanthology.org/2020.emnlp-main.609.
William Yang Wang. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-2067. URL https://aclanthology.org/P17-2067.
Matthijs J Warrens. Inequalities between multi-rater kappas. Advances in data analysis and classification, 4(4): 271-286, 2010.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6.
Jing Yang, Didier Vega-Oliveros, Taïs Seibt, and Anderson Rocha. Explainable fact-checking through question answering. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8952-8956, 2022. doi: 10.1109/ICASSP43922.2022.9747214.
A Dataset Access
We release our dataset and baseline at https://github.com/MichSchli/AVeriTeC, and will maintain it there. As we anticipate using the dataset in a future shared task, we are as of submission time only releasing the training and development splits. We will make the test split available privately to reviewers upon request.
B Author Statement
The authors of this paper bear all responsibility in case of violation of copyrights associated with the AVERITECdataset.
C Annotation Details
We carried out our annotations with the help of Appen (https://appen.com/), an Australian private company delivering machine learning products. The annotations took place on a special-purpose platform developed by our team and supplied to Appen. We will make the code for this platform available upon request. Appen provides guarantees that annotators are paid fairly: see https://success.appen.com/hc/en-us/articles/9557008940941-Guide-to-Fair-Pay. We spent a total of €40,835 for crowdworkers in our annotation process.
D Baseline Prompts
D.1 Claim Question Generation
To enrich our search results, we generate additional questions for use as search queries. For each claim, we retrieve the 10 most similar claims from the training dataset (computed using BM25). We combine these into a prompt following the scheme shown in Figure 3. We incorporate both the speaker and the claim itself in a form of preliminary experiments found to be highly effective: "Outrageously, SPEAKER claimed that CLAIM. Criticism includes questions like: ". The adversarial tone encourages the model to generate questions useful for debunking - we found this to be crucial for finding additional useful search results beyond those returned using the claim itself.
D.2 Passage Question Generation
Once search results have been found, we generate questions for each line of each searched document using the process described in Section 7.1. We retrieve the 10 most similar question-answer pairs from the training dataset (computed using BM25 between the answer and the evidence line). We combine these into a prompt following the scheme shown in Figure 3. We experimented also with including the claim when generating the questions, however, we found this to decrease performance by acting as a distractor; BLOOM would generate questions related only to the claim and unrelated to the evidence. Passage question generation was by far the most expensive part of our experiments. While we made sure the model fits in memory of an A100 GPU, we parallelized inference across several. Using eight GPUs, question generation took approximately 24 hours.
D.3 Justification Generation
We use a further prompt to generate justifications given the claim and verdict for the no-evidence baseline. Again, for each claim, we retrieve the 10 most similar claims from the training dataset (computed using BM25). We experimented with the same adversarial form discussed for question generation in Appendix D.1, but did not see any improvements in performance.
E Baseline Models
We finetuned models for several components of our baseline. The following sections list hyperparameter settings for each of those models. All training took place on a single Nvidia A100 GPU.
Outrageously, United We Win claimed that "As mayor, Cory Booker, US Democratic presidential candidate and US New Jersey Senator, passed criminal justice reform in a US Republican Senate." Criticism includes questions like: Did U.S. Sen. Cory Booker pass criminal justice reform? What was the Republicans' majority in the Senate when the bill was signed into law?
Outrageously, Mokwele Kholofelo Banny claimed that "A married couple in Florida, Tito and Amanda Watts, was arrested for selling fake golden tickets to heaven to hundreds of people." Criticism includes questions like: Did married couple Tito and Amanda Watts from Florida get arrested for selling fake golden tickets to heaven?