SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Outrageously, Muhammadu Buhari claimed that "It makes no sense for oil to be cheaper in Nigeria than in Saudi Arabia." Criticism includes questions like: What was the price of petrol in Nigeria in Oct 2020? What was the price of petrol in Saudi Arabia in Oct 2020?
Outrageously, Tea talk and gossip claimed that "Microsoft cofounder Bill Gates said Be nice to nerds. Chances are you'll end up working for one". Criticism includes questions like: Is Bill Gates quoted as saying "Be nice to nerds, chances are you'll end up working for one"?
Outrageously, Sen. Amy Klobuchar claimed that "US President Trump called for reduced funding for the Centre for Disease Control and Prevention". Criticism includes questions like: Did US President Trump propose budget cuts in the funding for the Centre for Disease Control and Prevention?
Outrageously, US Democratic presidential candidate Wayne Messam claimed that "It is illegal for mayors to even bring up gun reform for discussion in Florida, US." Criticism includes questions like:
Figure 3: Example prompt used to generate search questions for the claim "It is illegal for mayors to even bring up gun reform for discussion in Florida, US." with the speaker "US Democratic presidential candidate Wayne Messam".
E.1 Evidence Reranking
We used the BERT-large model [Devlin et al., 2019] with a text classification head, relying on the huggingface implementation [Wolf et al., 2020]. The model has 340 million parameters. We finetuned the model using Adam [Kingma and Ba, 2015] with a learning rate of 0.001 and a batch size of 128. The evidence reranker is trained using negative sampling. For each triple of claim  $c$ , question  $q$ , and answer  $a$ , we construct three negatives by corrupting each of  $c$ ,  $q$ , or  $a$ , for a total of 9 negative samples per positive. Corrupted elements are replaced with randomly selected others from the dataset.
E.2 Stance Detection
The setup for the stance detection model is similar to the evidence reranker. We again used the BERT-large model [Devlin et al., 2019] with a text classification head, relying on the huggingface implementation [Wolf et al., 2020]. The model has 340 million parameters. We finetuned the model using Adam [Kingma and Ba, 2015] with a learning rate of 0.001 and a batch size of 128. To train the stance detection model, we constructed examples from the training set. For claims with supported labels, we created one example per question for a positive stance. For claims with refuted labels, we created one example per question for negative stance. For claims with not enough evidence labels, we created one example per question for a neutral stance. Finally, we discarded all claims with conflicting evidence/cherrypicking as the label.
Evidence: The image of Time magazine cover with Rachel Levine as woman of the year was posted on Facebook by "The United Spot", which is labelled as a satire site. Question answered: Which website said that Rachel Levine was Time's Woman of the Year?
Evidence: Yes, because the wording was actually "complete 57 mega dams". Question answered: In 2017, did the Kenyan Government manifesto say they would construct 57 mega dams?
Evidence: No, because the blog text uses future terminology like "...the bill is being brought in..." and "...this nz food bill will pave the way...". Question answered: Does the blog post imply that this Food Bill is already legislation?
Evidence: China described the reports from Pakistan as "Baseless & fake". Question answered: Did China report any losses relating to this clash?
Evidence: After carrying a few boxes that appeared full of supplies, Pence was informed that the rest of the boxes in the van were empty and that his task was complete. "Well, can I carry the empty ones? Just for the cameras?" Pence joked. "Absolutely," an aide said as the group laughed. Pence then shuts the doors to the van and returns to talk to facility members from the nursing home. Question answered: Were the PPE boxes that Mike Pence delivered empty?
Evidence: Kris tells the magazine Caitlyn was "miserable" and "pissed off" during the last years of their marriage. Question answered:
Figure 4: Example prompt used to generate a question for the evidence line "Kris tells the magazine Caitlyn was "miserable" and "pissed off" during the last years of their marriage."
E.3 Justification Generation
For the justification generation model, we used the BART-large model [Lewis et al., 2020]. As previously we relied on the huggingface implementation [Wolf et al., 2020]. BART-large has 406M parameters. We finetuned the model using Adam [Kingma and Ba, 2015] with a learning rate of 0.001 and a batch size of 128. When generating, we used beam search with 2 beams and a maximum generation length of 100 tokens.
F Dataset statistics
To analyse our dataset, we computed various statistics for each dataset split. An overview of modalities in which evidence was found can be seen in Table 5. Statistics for claim type and fact-checker strategy can be found in Tables and respectively.
Annotators rely on evidence from a wide variety of different sources, taking evidence from a total of 2989 different domains. Interestingly, the most frequent is twitter.com  $(3\%)$ , typically representing announcements from public officials. This is followed by africacheck.org  $(2.5\%)$ , as Africa Check relies to a greater extent on references to its own past articles. After this follow official sources (e.g. cdc.gov  $(1.5\%)$ , who.int  $(1.3\%)$ , gov.uk  $(0.7\%)$ , wikipedia.org  $(1.4\%)$ ) and news media (e.g. nytimes.com  $(1.1\%)$ , washingtonpost.com  $(0.7\%)$ , and reuters.com  $(0.6\%)$ ). An interesting occurrence is a small number of non-textual sources, e.g. youtube.com  $(0.8\%)$ .
G ChatGPT Prompts
For the prompt used for our gpt-3.5-turbo experiments, see Figure 6.
Claim: A married couple in Florida, Tito and Amanda Watts, was arrested for selling fake golden tickets to heaven to hundreds of people.
Our verdict: Refuted.
Our reasoning: The answer and source clearly explain that it was an April Fool's joke so the claim is refuted.
Claim: North Korea blew up the office used for South Korea talks.
Our verdict: Supported.
Our reasoning: The building used was indeed destroyed.
Claim: US President Trump called for reduced funding for the Centre for Disease Control and Prevention.
Our reasoning: From the source, I saw tangible evidence where it stated that there was a proposal by US President Trump to slash more than $1.2 billion of CDC's budget.
Claim: It is illegal for mayors to even bring up gun reform for discussion in Florida, US.
Our verdict: Conflicting Evidence/Cherrypicking.
Our reasoning:
Figure 5: Example prompt used to generate a justification for the claim "It is illegal for mayors to even bring up gun reform for discussion in Florida, US." Evidence and verdict for the claim are produced in previous stages of the pipeline.
Train	Dev	Test
Web text:	68.2	75.5	74.9
PDF:	11.9	7.7	9.7
Metadata:	6.1	5.9	5.0
Web table:	4.9	3.0	2.9
Video:	1.1	1.1	1.9
Image/graphic:	2.0	2.7	1.6
Audio:	0.1	0.0	0.8
Other:	1.3	1.4	0.2
Unanswerable:	4.5	2.8	3.0
Table 5: Evidence modalities (%)
Can you fact-check a claim for me? Classify the given claim into four labels: "true", "false", "not enough evidence" or "conflicting evidence/cherrypicking". Let's think step by step. Provide justification before giving the label. Given claim:
It is illegal for mayors to even bring up gun reform for discussion in Florida, US.
Figure 6: Prompt used to generate evidence and verdicts with ChatGPT for the example claim "It is illegal for mayors to even bring up gun reform for discussion in Florida, US."
Train	Dev	Test
Position Statement	7.8	5.8	7.0
Numerical Claim	33.7	23.8	21.8
Event/Property Claim	57.8	61.4	69.8
Quote Verification	9.6	13.8	7.7
Causal Claim	11.5	10.8	11.9
Table 6: Claim types (%)
Train	Dev	Test
Written Evidence	78.8	88.6	88.0
Numerical Comparison	30.6	19.0	19.2
Fact-checker Reference	6.6	7.4	7.7
Expert Consultation	29.9	27.4	29.6
Satirical Source	3.6	2.0	1.8
Table 7: Fact-checker strategies (%)
H.1 Claim type
We computed baseline performance in terms of veracity at different evidence thresholds for each claim type. Results can be seen in Table 9 below:
I Data Statement
Following Bender and Friedman [2018], we include a data statement describing the characteristics of AVERITEC.
I.1 Curation Rationale
We processed a total of 8,000 texts from the Google FactCheck Claim Search API, which collects English-language articles from fact-checking organizations around the world. We selected claims in the two-year interval between 1/1/2022 and 1/1/2020. Within that span, we selected all claims marked true by fact-checking organizations, as well as a random selection of other claims; this was done to reduce the label imbalance as much as possible.
We discarded claims in several rounds. First, any duplicate claims were discarded using string matching. Then, annotators discarded paywalled claims, as well as claims about or requiring evidence from modalities beyond text. Finally, we discarded any claim for which agreement on a label could not be found after two rounds of annotation.
I.2 Language variety
We include data from 50 different fact-checking organizations around the world. While our data is exclusively English, the editing standards used at different publications differ. As such, several varieties of news domain English should be expected; given the distribution of fact-checkers involved, these will be dominated by en-US, en-IN, en-GB, and en-ZA.
I.3 Speaker demographics
We did not analyse the demographics of the individual speakers for each claim. However, we asked annotators to specify the location most relevant to the claims. The distribution can be seen in Table 10.
Fraction of claims
africacheck.org:	0.154
politifact.com:	0.153
leadstories.com:	0.096
fullfact.org:	0.068
factcheck.afp.com:	0.062
factcheck.org:	0.050
checkyourfact.com:	0.041
misbar.com:	0.032
washingtonpost.com:	0.029
boomlive.in:	0.026
dubawa.org:	0.023
polygraph.info:	0.020
usatoday.com:	0.019
altnews.in:	0.019
indiatoday.in:	0.019
newsmeter.in:	0.018
newsmobile.in:	0.015
factly.in:	0.015
vishvasnews.com:	0.015
aap.com.au:	0.014
theologicalindian.com:	0.013
verafiles.org:	0.011
nytimes.com:	0.011
healthfeedback.org:	0.011
thequint.com:	0.008
newsweek.com:	0.005
icirnigeria.org:	0.005
bbc.co.uk:	0.004
factcheck.thedispatch.com:	0.004
ghanafact.com:	0.003
factcheckni.org:	0.003
theferret.scot:	0.003
rappler.com:	0.003
covid19facts.ca:	0.003
newsmobile.in:80:	0.002
thegazette.com:	0.002
abc.net.au:	0.002
ha-asia.com:	0.002
sciencefeedback.co:	0.001
cbsnews.com:	0.001
fit.thequint.com:	0.001
namibiaFACTCHECK.org.na:	0.001
thejournal.ie:	0.001
poynter.org:	0.001
zimfact.org:	0.001
climatefeedback.org:	0.001
factchecker.in:	0.001
pesacheck.org:	0.001
ghana.dubawa.org:	0.001
scroll.in:	0.001
Table 8: Fact-checking sites used
λ = 0.2	λ = 0.3
Quote Verification	.13	0.7
Numerical Claim	.17	.10
Event/Property Claim	.13	.06
Causal Claim	.11	.04
Position Statement	.10	.04
Table 9: Baseline performance on each claim type, computed with two different evidence standards.
I.4 Annotator demographics
For this dataset, we relied on the company Appen to provide annotators. Although the company itself is headquartered in Australia, demographic details regarding location or nationality of the annotators
Country code	Count
US:	1937
IN:	536
GB:	305
KE:	293
NG:	280
ZA:	191
PH:	73
AU:	56
CN:	55
RU:	38
CA:	31
NZ:	23
GH:	17
IE:	17
LK:	14
TH:	12
FR:	12
PK:	12
IL:	11
IT:	10
DE:	8
ZW:	7
HK:	7
MM:	6
BR:	6
UA:	6
KR:	5
JP:	5
KP:	5
PL:	5
None:	501
Table 10: Count of locations appearing in our dataset. All countries are listed using ISO country codes. Countries with fewer than five occurrences are excluded – we will provide this data upon request.
were unfortunately not shared with us. We employed a total of 25 annotators with an average age of 42, and a gender split of  $64\%$  women and  $36\%$  men.
I.5 Speech situation
The original claims were uttered in a variety of situations. We did not track this statistic for the entire dataset. However, analyzing a randomly selected 20 claims from our dataset, the majority (11) are social media posts. 4 originate from public speeches by politicians, 3 from newspaper articles, 1 from a political candidate's website, and 1 from a viral YouTube video.
The claims were all chosen by fact-checking organizations for analysis, and presented in a journalistic format on their websites.
I.6 Text characteristics
We compute various statistics for the text included in this dataset; see Section 5 and Appendix F. The genre is a mix of political statements, social media posts, and news articles (see the previous subsection).
J Annotation Guidelines
J.1 Introduction
We aim to construct a dataset for automated fact-checking with the following guiding principles. First, we intend to decompose the evidence retrieval process into multiple steps, annotating each individual step as a question-answer pair (see Figure 2). Second, our dataset will be constructed from real-world claims previously checked by journalistic organisations, rather than the artificially created claims used in prior work.
Decomposing claim verification into generations and answering questions allows us to break complex real-world claims down to their components, simplifying the task. For example, in Figure 2, verifying the claim requires knowing the salary of the health commissioner, the governor, the vice president, and Dr. Fauci, so that they can be compared. Four separate questions about salary need to be asked in order to reach a verdict (i.e. that the claim is supported).
By decomposing the evidence retrieval process in this way, we also produce a natural way for systems to justify their verdicts and explain their reasoning to users. In addition to this, we annotate claims with a final justification, providing a textual explanation of how to combine the retrieved answers to reach a verdict. This allows users to follow each step of the retrieval and verification processes, and so understand the reasoning employed by the system.
Claim: Biden lead disappears in NV, AZ, GA, PA on 11 November 2020.
Q1: Which media project Biden will win in Nevada?
A1: ABC News, CBS News, NBC News, CNN, Fox News, Decision Desk HQ, Associated Press, Reuters, and New York Times.
Q2: Which media project Biden will win in Arizona?
A2: Fox News and Associated Pre.
Q3: Which media project Biden will win in Georgia?
A3: None.
Q4: Which media project Biden will win in Pennsylvania?
A4: ABC News, CBS News, NBC News, CNN, Fox News, Decision Desk HQ, Associated Press, Reuters, and New York Times.
Verdict: Refuted
Justification: Many media organizations believe Biden will win in NV, AZ, and PA. As such, his lead has not disappeared.
Figure 7: Example claim and question-answer pairs.
The annotation consists of the following three phases:
1. Claim Normalization.
2. Question Generation.
3. Quality Control.
Each claim should be annotated by different annotators in each phase. An annotator can participate in all three phases, but they will be assigned different claims.
Warning! Components of the AVeriTeC annotation tool may not render correctly in some browsers, specifically Opera Mini. If this is an issue we recommend trying another browser, e.g. Firefox, Chrome, Safari, or regular Opera.
J.2 Sign In
Each annotator will have received an ID and a Password with the access link to the annotation server. The password can be changed after logging into the interface.
- Make sure to log out at the end of the session!
- Do not open multiple tabs/windows of the AVeriTeC annotation tool. Always use only one window during annotation! If you are logged into multiple sessions using the same account, the annotation tool may lose the data you enter.
Figure 8: Interface of the control panel. ① Button for changing the password. ② Button for logout. ③ Start the annotation for this phase. Here is Phase 1 Claim Normalization. ④ The left number shows how many claims have been annotated and the right number shows how many claims are assigned for the current annotator at this phase.
After clicking the START NEXT button, the annotation phase will start. If an annotator is new to the current phase, the interface will provide a guided tour as in Figure 9 for that phase. Please read the hints provided by the tour guide carefully before the annotation.
Figure 9: Interface of the tour guide.
J.3 Phase 1: Claim Normalization
In the first phase, annotators collect metadata about the claims and produce a normalized version of each claim, as shown in Figure 10. The first step is to identify the claim(s) in the fact-checking article. Often, this can be found either in the headline or explicitly in some other place in the fact-checking article. In some cases, there may be a discrepancy between the article and the original claim (e.g. the original claim could be "there are 30 days in March", while the fact-checking article might have the headline "actually, there are 31 days in March"). In those cases, it is important to use the original version of the claim. If there is ambiguity in the article over the exact wording of the claim, annotators should use their own judgment.
Figure 10: Interface of claim normalization.  $①$  The fact-checking article provided.  $②$  Guideline of annotation for this phase. Please read it before annotating. Notice that if the article displays a 404 page or another error, or if it takes more than one minute to load, please click the REPORT & SKIP button.  $③$  Fields for the normalized claim and the corresponding label.  $④$  General information of the claim.  $⑤$  Check-boxes for selecting the type of the claim.  $⑥$  Check-boxes for selecting the fact-checking strategy used.  $⑦$  Button for adding more claims.  $⑧$  Buttons for submitting the current claim, going to the previous claim, and the next claim.
J.3.1 Overview
Here, we give a quick overview of the claim normalization task; an in-detail discussion can be found in subsequent sections. Further documentation can also be found on-the-fly using the tooltips in the annotation interface.
1. First, annotators should read the fact-checking article and identify which claims are being investigated.
2. If the fact-checking article is paywalled or inaccessible due to a 404-page or a similar error message, annotators should report this and skip the claim using the provided button. We warn that some fact-checking articles can take too long to load – as such, while fact-checking articles that do not load at all should be skipped, we ask annotators to wait for at least one minute before skipping an article while it is still trying to load.
3. Most articles focus on one claim. However, some articles investigate multiple claims, or claims with multiple parts – in those cases, annotators should first split these into their parts (see Section J.3.2).
4. Some claims cannot be understood without the context of the fact-checking article, e.g. because they refer to entities not mentioned by name in the claim. In those cases, annotators should add context to the claims (see Section J.3.3).
5. Generally, we prefer claims to be as close as possible to their original form (i.e. the form originally said, not the form used in the fact-checking article). As such, contextualization should be done only when necessary, following the checklist in Section J.3.3.
6. Annotators should extract the verdict assigned to the claim in the article and translate it as closely as possible to one of our four labels - supported, refuted, not enough evidence, or conflicting evidence/ cherry picking (see Section J.3.4). In phase one, annotators should give their own judgments - rather, they should match as closely as possible the judgments given by the fact-checking articles.
7. Claims will have associated metadata, i.e. the date the original claim was made, or the name of the person who made it. Annotators should identify and extract this metadata from the article (see Section J.3.6).
8. Annotators should identify the type of each claim, choosing from the options described in Section J.3.8. These are not mutually exclusive, and more than one claim type can be chosen.
9. Annotators should identify the strategies used in the fact-checking article to verify each claim, choosing from the options described in Section J.3.9. These are not mutually exclusive, and more than one claim type can be chosen.
J.3.2 Claim Splitting
Some claims consist of multiple, easily separable, independent parts (e.g. "The productivity rate in Scotland rose in 2017, and similarly productivity rose in Wales that year"). The first step is to split these compound claims into individual claims. Metadata collection and normalization will then be done independently for each individual claim, and in subsequent phases, they will be treated as separate claims.
When splitting a claim, it is important to ensure that each part is understandable without requiring the others as context. This can be done either by adding metadata in the appropriate field, such as the claimed speaker or claim date, or through rewriting. For example, for the claim "Amazon is doing great damage to tax paying retailers. Towns, cities, and states throughout the U.S. are being hurt - many jobs being lost!", it should be clear what is causing job loss in the second part. A possible split would be "Amazon is doing great damage to tax paying retailers" and "Towns, cities and states throughout the U.S. are being hurt by Amazon - many jobs being lost". That is, it is necessary to rewrite the second part by adding Amazon a second time in order for the second part to be understandable without context.
J.3.3 Claim Contextualization
Some claims are not complete, which means they lack adequate contextualization to be verified. For example, in the claim "We have 21 million unemployed young men and women," there are unresolved pronouns without which the claim cannot be verified (e.g. we refers to Nigeria, as the speaker of the claim is the presidential candidate of Nigeria). Another example is "Israel already had  $50\%$  of its population vaccinated." We need to know when this claim was made to verify its veracity, as time is crucial for this verification. For the latter, metadata is enough to resolve ambiguities; the former needs to be rewritten as "Nigeria has 21 million unemployed young men and women."
Annotators are asked to contextualize claims to the original post by gathering the necessary information. Some information can be included simply as metadata, but this is not always enough – for information not captured by metadata, we ask that the claim itself is rewritten to include said information. Annotators need to follow this checklist:
1. Is the claim referring to entities that can only be identified by reading the associated fact-checking article, even if all metadata is taken into consideration? If so, add the names of the
entities (e.g. "Former first lady said, 'White folks are what's wrong with America'." becomes "Former first lady Michelle Obama said, 'White folks are what's wrong with America'").