SlowGuess/NeurIPS_2023_OCR huggingface dataset:
2. Does the claim have unnecessary quotation marks or references to a speaker (such as the word says in the example here)? If so, remove them (e.g. "Says 'Monica Lewinsky Found Dead' in a burglary." becomes "Monica Lewinsky found dead in a burglary.). Do NOT remove the reference to the speaker if the central problem is to determine if that person actually said the quote, e.g. in the case of quote verification.
3. Is the claim a question? If so, rephrase it as a statement (e.g. "Did a Teamsters strike hinder aid efforts in Puerto Rico after Hurricane Maria?" becomes "A Teamsters strike hindered aid efforts in Puerto Rico after Hurricane Maria in 2017").
4. Does the claim contain pronominal references to entities only mentioned in the fact-checking article? If so, replace the pronoun with the name of that entity. (e.g. "We have 21 million unemployed young men and women." becomes "Nigeria has 21 million unemployed young men and women.")
5. For some fact-checking articles, the title used does not properly match the fact-checked claim. Find the original claim in the article, and use that for producing the normalized version. As shown in Figure 11, the claim should be the first sentence of the article rather than the title.
6. Is the claim too vague to be investigated through the use of evidence, and does the fact-checking article investigate a more specific version of the claim? If so, use the claim investigated in the fact-checking article (e.g. "Towns, cities, and states throughout the U.S. are being hurt by Amazon" might become "Towns, cities, and states throughout the U.S. are losing state tax revenue because of Amazon").
Generally, try to make claims specific enough so that they can be understood and so that appropriate evidence can be found by a person who has not seen the fact-checking article.
Important! We recommend reading through the entire article and understanding the central problem before rewriting the claim. This makes it easier to identify the exact phrasing of the original claim and to make any minimal interventions necessary following our checklist above. When in doubt as to whether a claim should be modified, we recommend leaving it unchanged – we generally prefer claims to be as close as possible to their original form, subject to the constraints listed above.
J.3.4 Labels
We ask annotators to produce a label for the claim relying only on the information on the fact-checking site (and assuming that everything reported it is accurate). For the dataset we are creating, we will be using four labels:
1. The claim is supported. The claim is supported by the arguments and evidence presented.
2. The claim is refuted. The claim is contradicted by the arguments and evidence presented.
3. There is not enough evidence to support or refute the claim. The evidence either directly argues that appropriate evidence cannot be found, or leaves some aspect of the claim neither supported nor refuted. We note that many fact-checking agencies mark claims as refuted (or similar), if supporting evidence does not exist, without giving any refuting evidence. We ask annotators to use not enough evidence for this category, regardless of the original label. In situations where evidence can be found that the claim is unlikely, even if the evidence is not conclusive, annotators may use refuted; here, annotators should use their own judgment. We give a few examples in Section J.3.5.
4. The claim is misleading due to conflicting evidence/ cherry-picking, but not explicitly refuted. This includes cherry-picking (see https://en.wikipedia.org/wiki/Cherry-picking), true-but-misleading claims (e.g. the claim "Alice has never lost an election" with evidence showing Alice has only ever run unopposed), as well as cases where conflicting or internally contradictory evidence can be found.
Conflicting evidence may also be relevant if a situation has recently changed, and the claim fails to mention this (e.g. "Alice is a strong supporter of industrial subsidies" with evidence showing that Alice currently supports industrial subsidies, but in the past opposed industrial
Hoax Alert
Fake News: Donald Trump Did NOT Say "Women, You Have To Treat Them Like Sh*t"
① Sep 16, 2019
by: Maarten Schenk
treated like shit in this interview.
Figure 11: An example of locating the claim.
Did Donald Trump say "Women, you have to treat them like shit" in a 1992 interview in New York Magazine?
that's not true: Trump was not speaking about women general but about a very narrow category of people, to be particular "supermodels... clinging to a rock star's legs". While Trump has definitely insulted or disparaged many individual women in his career (and men... and groups of people...), he did not directly say *all* women have to be
The quote recently reappeared in an image post (archived here) published by Occupy Democrats on Facebook on September 14, 2019:
subsidies). We note that if the claim covers a period of time, and evidence refutes the claim at some timepoints but not others, the whole claim is still refuted – for example, “Alice has always been a strong supporter of industrial subsidies” or “Alice has never been a strong supporter of industrial subsidies”. For a real example from our dataset, consider https://fullfact.org/online/does-polands-migration-policy-explain-its-lack-terror-attacks/ – the claim is that “Poland has had no terror attacks”; evidence shows that Poland had no terror attacks before 2015, but some examples afterward, and should as such be marked refuted.
Despite the claim splitting subtask, some claims may contain multiple parts that are too interconnected to split. This could for example be a claim like "Alice has never lost an election because she always supports cheese subsidies". In such cases, parts of the claim may have different truth values. We discuss a few cases below:
- The claim is implicature, i.e. "X happens because Y" or "X leads to Y". In this case, annotators should find a label for the causal implication, and not for either of the component claims.
- The claim has too components, where one is refuted and the other is not enough information. In this case, the entire claim should be labeled refuted.
- The claim has too components, where one is supported and the other is not enough information. In this case, the entire claim should be labeled not enough information.
Important! The label was given in Phase 1 – and only in Phase 1 – should reflect the decision of the fact checker, not the interpretation of the annotator. In Phase 1, annotators should report the original judgment, as closely as possible, even if they disagree with it.
J.3.5 Deciding Between Refuted and NEE
As mentioned, the line between refuted and not enough evidence requires annotators to rely on their own judgment in cases where refuting evidence cannot be directly found, but the claim is extremely unlikely. As a guiding principle, if annotators would feel doubt regarding the truth value of the claim – given the presented evidence and/or lack of evidence – not enough evidence should be chosen. Below, we give several examples from our dataset:
- "The Covid-19 dusk-to-dawn curfew is Kenya's first-ever nationwide curfew since independence." No evidence can be found that Kenya has implemented a nationwide curfew before the Covid-19 pandemic. However, it is conceivable that evidence of such a curfew would simply not show up in documentation uploaded to the internet. As such, the annotator cannot rule out a prior curfew beyond a reasonable doubt, and such should select not enough evidence as the label.
- "The government in India has announced that it will shut down the internet to avoid panic about the Coronavirus." Evidence can be found that Indian law allows the government to do so as an emergency measure; however, the annotator finds no announcement from the government that the internet actually will be shut down. If other, regular, announcements from the same government body could be found, the claim should be labeled refuted – it would be extremely unlikely that a shutdown on the internet would not be announced via standard channels. However, in this case, standard channels do not make announcements in English, and therefore it is plausible that the announcement has not been found simply because it has not been translated; in this case, the annotator should select not enough evidence (with evidence that no English-language official channel exists).
- "Shakira is Canadian." Evidence can be found that Shakira is usually described as Colombian, was born in Colombia, and holds Colombian citizenship. Furthermore, evidence shows she now resides in Spain. As no evidence of any connection to Canada can be found despite the wealth of information available about her, it is extremely unlikely that she is secretly Canadian; as such, the annotator can select refuted as the label.
A special case of this kind of claim is quote verification, where it can be difficult to establish that someone did not say something. In many cases, evidence can be found that a quote is fictional (e.g. by finding evidence from a service like https://quoteinvestigator.com/), or that it originates from someone else. However, in some cases, there is no readily available evidence. In this case, we advise that annotators document the lack of evidence that the person said the quote itself, or any paraphrase of the quote. Further, annotators should document that some quotes by that person can be found, if possible what the person has said on the same topic, and if possible that the quote has not been said by someone else. This establishes that evidence for the quote should be available, and is not; in that case, annotators can pick refuted as the label. If annotators cannot find any claims by the person or any evidence for the quote (say an entirely fictional person with an entirely fictional quote), they should pick not enough evidence.
For a good example of how to handle these cases, consider the claim "RBI has said that ₹2000 notes are banned and ₹1000 notes have been introduced". As this claim is false, no evidence can be found of RBI making any such announcement; nor that they did not make that particular announcement. Here, the annotator first established where official communication from RBI is published with the question "how do the RBI/central bank make announcements on changes to currency?" Then, after finding that all official communication is posted to the RBI website, they asked a follow-up question testing whether evidence for the claim can be found on the official website.
J.3.6 Metadata Collection
Annotators need to collect metadata through the following three steps.
J.3.7 General Information
- A hyperlink to the original claim, if that is provided by the fact-checking site. Examples of this include Facebook posts, the original article or blog post being fact-checked, and embedded video links. If the original claim has a hyperlink on the fact-checking site, but that hyperlink is dead, annotators should leave the field empty.
Figure 12: An example of an image claim requiring transcription.
- The date of the original claim, regardless of whether it is necessary for verifying the claim. This date is often mentioned by the fact checker, but not in a standardized place where we could automatically retrieve it. Note that the date for the original claim and the fact-checking article (often its publication date) may be different and both are stated in the text. We specifically need the original claim date, as we intend to filter out evidence that appeared after that date. If multiple dates are mentioned, the earliest should be used. If an imprecise date is given (e.g. February 2017), the earliest possible interpretation should be used (i.e. February 1st, 2017).
- The speaker of the original claim, e.g. the person or organization who made the claim.
- The source of the original claim, e.g. the person or organization who published the claim. This is not necessarily the same as the speaker; a person might make a comment in a newspaper, in which case the person is the speaker and the newspaper is the source.
- If the original claim is or refers to an image, video, or audio file, annotators should add a link to that media file (or the page that contains the file, if the media file itself is inaccessible).
- If the original claim is an image that contains text – for example, Figure 12 shows a Facebook meme about Michelle Obama – annotators should transcribe the text that occurs in the image as metadata. In the example, it would be “Michelle Obama said white folks are what’s wrong with America.”
- If the fact-checking article is paywalled or inaccessible due to an error message, annotators should report this and skip the claim using the corresponding button.
J.3.8 Claim Type
The type of the claim itself, independent of the approach taken by the fact checker to verify or refute it, should be chosen from the following list. This is not a mutually exclusive choice – a claim can be speculation about a numerical fact, for example. As such, annotators should choose one or several from the list.
- Speculative Claim: The primary task is to assess whether a prediction is plausible or realistic. For example "the price of crude oil will rise next year."
- Opinion Claim: The claim is a non-factual opinion, e.g. "cannabis should be legalized". This contrasts with factual claims on the same topic, such as "legalization of cannabis has helped reduce opioid deaths."
- Causal Claim: The primary task is to assess whether one thing caused another. For example "the price of crude oil rose because of the Suez blockage".
- Numerical claim. The primary task is to verify whether a numerical fact is true, or to verify whether a comparison between several numerical facts hold, or to determine whether a numerical trend or correlation is supported by evidence.
- Quote Verification. The primary task is to identify whether a quote was actually said by the supposed speaker. Claims only fall under this category if the quote to be verified directly figures in the claim, e.g. "Boris Johnson told journalists 'my favourite colour is red, because I love tomatoes'".
- Position Statement. The primary task is to identify whether a public figure has taken a certain position, e.g. supporting a particular policy or idea. For example, "Edward Heath opposed privatisation". This also includes statements that opinions have changed, e.g. "Edward Heath opposed privatisation before the election, but changed his mind after coming into office". Factual claims about the actions of people (e.g. "Edward Heath nationalised Rolls-Royce") are not position statements (they are event or property claims); claims about the attitudes of people (e.g. "Edward Heath supported the nationalisation of Rolls-Royce") are.
- Event/Property Claim. The primary task is to determine the veracity of a narrative about a particular event or series of events, or to identify whether a certain non-numerical property is true, e.g. a person attending a particular university. Some properties represent causal relationships, e.g. "The prime minister never flies, because he has a fear of airplanes". In those cases, the claim should be interpreted as both a property claim and a causal claim.
- Media Publishing Claim. The primary task is to identify the original source for a (potentially doctored) image, video, or audio file. This covers both doctored media, and media that has been taken out of context (e.g. a politician is claimed to have shared a certain photo, and the task is to determine if they actually did). This also includes HTML-doctoring of social media posts. We will discard all claims in this category.
- Media Analysis Claim. The primary task is to perform complex reasoning about pieces of media, distinct from doctoring. This could for example be checking whether a geographical location is really where a video was taken, or determining whether a specific person is actually the speaker in an audio clip. The claim itself must directly involve media analysis e.g. "the speaker of these two clips is the same". Claims where the original source is video but which can be understood and verified without viewing the original source, do not fall under this category. An original video or audio file can feature as metadata in fact-checking articles, but claims are only complex media claims if analysis of the video or audio beyond just extracting a quote is necessary for verification.
Several claim types – speculative claims, opinion claims, media publishing claims, and media analysis claims – will not be included in later phases.
J.3.9 Fact-checking Strategy
After identifying the claim type, we ask annotators to classify the approach taken by the fact checker according to the article. This is independent of the claim type, as a fact-checker might take any number of approaches to a given claim. Again, one or several options should be chosen from the following list:
- Written Evidence. The fact-checking process involved finding contradicting or supporting written evidence, e.g., a news article directly refuting or supporting the claim.
- Numerical Comparison. The fact-checking process involved numerical comparisons, such as verifying that one number is greater than another.
- Consultation. The fact checkers directly reached out to relevant experts or people involved with the story, reporting new information from such sources as part of the fact-checking article.
- Satirical Source Identification. The fact-checking process involved identifying the source of the claim as satire, e.g. The Onion.
- Media Source Discovery. The fact-checking process involved finding the original source of a (potentially doctored) image, video, or soundbite.
- Image analysis. The fact-checking process involved image analysis, such as comparing two images.
- Video Analysis. The fact-checking process involved analysing video, such as identifying the people in a video clip.
- Audio Analysis The fact-checking process involved analysing audio, such as determining which song was played in the background of an audio recording.
- Geolocation. The fact-checking process involved determining the geographical location of an image or a video clip, through the comparison of landmarks to pictures from Google Streetview or similar.
- Fact-checker Reference. The fact-checking process involved a reference to a previous fact-check of the same claim, either by the same or a different organisation. Reasoning or evidence from the referenced article was necessary to verify the claim.
Claims only labelled as solved through Fact-checker Reference will not be included in later phases.
J.4 Phase 2: Question Generation and Answering
The next round of annotation aims to produce pairs of questions and answers providing evidence to verify the claim. The primary sources of evidence are the URLs linked in the fact-checking article. We also provide access to a custom search bar to retrieve evidence.
Figure 13: Interface of question generation.  $①$  Guideline of annotation for this phase. Please read it before annotating. Notice that if the article displays a 404 page or another error, or if it takes more than one minute to load, please click the REPORT & SKIP button.  $②$  The claim and the associated metadata.  $③$  Fields for the first question and its answers. Annotators can add up to 3 answers for each question if necessary. The text fields of metadata of question answer pairs are also provided.  $④$  Annotators can use the plus button to add as many questions as they want. Please select the label of this claim after finishing the question and answer generation.  $⑤$  Buttons for submitting the current claim, going to the previous claim, and next claim.  $⑥$  The custom search engine.
J.4.1 Overview
Here, we give a quick overview of the question generation task; in-detail discussion can be found in subsequent sections. Further documentation can also be found on-the-fly using the tooltips in the annotation interface.
16. Sometimes, annotators may be in doubt as to whether an additional question should be added to further support the verdict. Generally speaking, we always prefer to have as many question-answer pairs as possible, so if in doubt annotators should veer on the side of adding that additional question.
Important! Annotators should not choose a label if the retrieved evidence does not support it; for example, if the label conflicting evidence is chosen, there should be evidence documenting the conflict. Labels in phase two can contradict the label of the fact-checker, if the annotator believes it is appropriate.
J.4.2 Claim Correction
In addition to gathering question-answer pairs, Phase Two also acts as quality control for the claim contextualization in Phase One. This means if Phase Two annotators encounter a claim that is malformed or not properly contextualized, they can correct it. The guidelines for claim contextualization can be seen in Section J.3.3; the same criteria hold. Based on our initial review of the data entered in Phase One, Claim Correction is rarely necessary. Below are some examples from the data of claims that should be corrected in Phase Two:
1. The claim "Nigerian vice presidential candidate Peter Obi claimed that Capital expenditure in 2016 was N1.2 trillion and 2017 was N1.5 trillion," given the article https://afri cachecheck.org/fact-checks/reports/battle-titans-fact-checking-arch-riv als-race-nigermias-presidency. The article verifies the numerical value of capital expenditure in Nigeria, not whether Peter Obi has claimed anything about it. The original article is not quote verification, but the annotator has changed the claim to that. Here, the Phase Two annotator should correct the claim to simply "Nigerian capital expenditure in 2016 was N1.2 trillion and 2017 was N1.5 trillion."
2. The claim "Abolish all charter schools", given the article https://www.factcheck.org/2020/07/trump-twists-bidens-position-on-school-choice-charter-schools/. This is a position statement about Joe Biden's stance on charter schools; however, the annotator has removed all reference to Joe Biden. The Phase Two annotator should correct the claim to "Joe Biden wants to abolish all charter schools".
3. The claim "Is Florida doing five times better than New Jersey?", given the article https://leadstories.com/hoax-alert/2020/07/fact-check-florida-is-not-doing-five-times-better-in-deaths-than-new-york-and-new-jersey.html. The claim has mistakenly been phrased as a question. It is also too vague. The Phase Two annotator should correct this, following the article: "Florida is doing five times better than New Jersey in COVID-19 deaths per 1 million population".
J.4.3 Question Generation
To ensure the quality of the generated questions, we ask the annotators to create their questions as follows:
- Questions should be well-formed, rather than search engine queries (e.g. "where is Cambridge?" rather than "Cambridge location").
- Questions should be standalone and understandable without any previous questions.
- Questions should be based on the version of the claim shown in the interface (i.e. the version extracted by phase one annotators), and not on the version in the fact-checking article. If an annotator believes a phase one claim has been extracted wrongly, they can correct it using the appropriate box.
- The annotators should avoid any question that directly asks whether or not the claim holds, e.g. "is it true that [claim]".
- The annotators should ask all questions necessary to gather the evidence needed for the verdict, including world knowledge that might seem obvious, but could depend for example on where one is from. For example, Europeans might have better knowledge of European geography/history than Americans, and vice-versa.
- As a guiding principle, at least 2 questions should be asked. This is not a hard limit, however, and the annotators can proceed with only one question asked if they do not feel more are needed.
The following are examples used to illustrate how questions should be asked. These are based on the real claim "the US in 2017 has the largest percentage of immigrants, almost tied now with the historical high as a percentage of immigrants living in this country":