SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Good: What was the population of the US in 2017?
- Good: How many immigrants live in the US in 2017?
- Bad: What was the population of the US? [No time specified to find a statistic]
- Bad: What was the population there in 2017? [What does there refer to?]
- Bad: Is it true that the US in 2017 has the largest percentage of immigrants, almost tied now with the historical high as a percentage of immigrants living in this country? [Directly paraphrases the claim]
J.4.4 Metadata
Questions about metadata can be used to draw attention to aspects of the claim, in order to reason about publication date or publication source. If, for example, the claim "aliens made contact with earth March 3rd, 2021" was published on September 1st, 2020, the publication date can be used to refute the claim. In such cases, we ask annotators to first generate a question/answer pair - "when was this claim made?" "September 1st, 2020" - which can then be used to refute the claim. Similarly, questions about publication source can be used to refute satirical claims - "where was this claim published?", "www.theonion.com", "what is The Onion?", "The Onion is an American digital media company and newspaper organization that publishes satirical articles on international, national, and local news".
J.4.5 Common sense assumptions and world knowledge
As a part of the question generation process, annotators may have to make assumptions and/or use world knowledge to interpret the claim. For example, for the claim "Shakira is Canadian", it may be necessary to choose what it means to be Canadian. This is expressed in how questions are formulated, e.g. "does Shakira have Canadian citizenship?" or "where does Shakira live?". This may also involve politically charged judgments. For example, some First Nations people are classed as Canadian by the Canadian government, but do not use that label for themselves.
In such cases, we ask annotators to follow – as closely as possible – the judgments made by the fact-checking websites. If the annotators feel that these are incomplete or misleading, they can add additional questions.
For example, for the claim "Edward Heath opposed privatisation", a fact checker might provide his party manifesto as evidence. A corresponding question could then be "what did the 1970 Conservative Party manifesto say about privatisation?" An annotator could encounter evidence for the nationalisation of Rolls Royce during Heath's government, which the fact-checking article did not take into account. In that case, the annotator might want to add an additional question, such as "did Heath's government nationalise any companies?". The annotators should ask both questions.
Important! As opposed to Phase 1, annotators in Phase 2 should use their own judgment to assign labels (although they should not ignore evidence used by the fact-checker). As such, if they disagree with the fact-checker about the label, they can select a different label.
J.4.6 Answer Generation
To find answers to questions, the annotators can rely on metadata, or on any sources linked from the factchecking site. Where these fail to produce appropriate information – either because they are not relevant to an asked question or because they refer to sources which have been taken down – we provide search functionalities as an alternative. Note that the annotators are not allowed to use the fact-checking article itself as a source, only the pages hyper-linked in the fact-checking article (and
only when they are not from fact-checking websites). Similarly, other fact-checking articles found through search should be avoided.
Figure 14: Interface of the search bar. (1) Search bar and the location option. Annotators can change the localization of the search engine by selecting the country code here. (2) Search results returned by the search engine.
Once an answer has been found, annotators can choose between the following four options to enter it:
- Extractive: The answer can be copied directly from the source. We ask the annotators to use their browser's copy-paste mechanism to enter it.
- Abstractive: A freeform answer can be constructed based on the source, but not directly copy-pasted.
- Boolean: This is a special case of abstractive answers, where a yes/no is sufficient to answer the question. A second box must be used to give an explanation for the verdict grounded in the source (e.g. "yes, because...").
- Unanswerable: No source can be found to answer the question.
For extractive, abstractive, and boolean answers, the annotators are also asked to copy-paste a link to the source URL they used to answer the question. Extractive answers are preferred to abstractive and boolean answers.
In some cases, annotators might find different answers from different sources. Our annotation tools allow adding additional answers, up to three. While we provide this functionality, we ask that annotators try to rephrase the question to yield a single answer before adding additional answers.
We note that if the annotators can only find a partial answer to a question, they can still use that. In such cases, please give the partial answer rather than marking the question as unanswerable.
Our search engine marks pages originating from known sources of misinformation and/or satire. We do not prevent annotators from using such sources, but we ask that annotators avoid them if at all possible. In the event that an annotator wishes to use information from such a source, we strongly prefer that the finds similar, corroborating information from an additional source in order to further substantiate the evidence.
While answering a question, we furthermore ask annotators to adhere to the following:
- DO NOT use any other browser window/search bar to find an answer. You MUST use the provided search bar only.
- DO NOT give a verdict for the claim until you have finished questions and answers.
- DO NOT use the fact-checking article itself, or any other version of it you find on the internet, as evidence to support an answer.
- DO NOT submit answers using other articles from fact-checking websites, such as politifact.com or factcheck.org, as evidence.
- DO NOT simply reference the source as an authority in abstractive answers (and boolean explanations), e.g. do not use answers like "yes, because the Guardian says so". Rather, write out what the source says, e.g. "yes, because £18.1 bn is 41% of the budget". If you consider it important to mention the source, write that the source says - e.g., "yes, because according to the Guardian £18.1 bn was spent, which is 41% of the budget".
J.4.7 Reasoning Chains of Claims
Annotators can build up reasoning chains across multiple questions, meaning that answers of one question can be used in the next question. For example, for the claim "the fastest train in Japan drives at a top speed of  $400\mathrm{km / h}$ ", the first question is "What is the fastest Japanese train?" The answer is "The fastest Japanese train is Shinkansen ALFA-X". Based on the answer, we can further ask the second question to get more details, "What is the maximum operating speed of the Shinkansen ALFA-X". Note that while the generation of the second question assumes knowledge of the answer to the first, it is understandable without it.
J.4.8 Confirmation
After submitting the question/answer pairs for a claim, annotators will be presented with a confirmation screen (see Figure 15). Annotators will be shown the question/answer pairs they have entered, along with the verdict, and asked to confirm a second time that the verdict is supported by the evidence.
Figure 15: Before moving on to the next claim, phase two annotators will be shown a confirmation screen to make sure that their chosen verdict is correct.
J.5 Phase 3: Quality Control
Once we have collected evidence in the form of generated questions and retrieved answers, we want to provide a measure of quality. Given a claim with associated evidence, we ask a third round of annotators to give a verdict for the claim. Crucially, the annotators at this round do not have access to the original fact-checking article, or to the claim label.
J.5.1 Overview
Here, we give a quick overview of the quality control task; in-detail discussion can be found in the following sections. Further documentation can also be found on-the-fly using the tooltips in the annotation interface.
1. Annotators should first read the claim, the metadata, and the question-answer pairs. This is the only information which should be used during this phase
2. It is important that annotators in the quality control phase do not use web search to find additional information, or rely on background knowledge which an average English speaker might not have. Commonsense facts that are known to (almost) everyone can be used - see Section J.5.2.
3. If the claim, or any of the question-answer pairs lack context, they can be flagged. This helps us diagnose what is wrong with a set of question-answer pairs in the case annotators disagree over the label.
4. After reviewing the claim and the QA pairs, annotators should assign a label to the claim (see the four labels introduced in Section J.3.4).
5. Finally, annotators should write a short statement justifying the verdict. If any commonsense information (e.g. background knowledge which an average English speaker is likely to have) is used to give the verdict, but that information is not mentioned in any question-answer pair, it should be mentioned in the justification. For advice regarding justification production, see Section J.5.3.
J.5.2 Commonsense Knowledge
When giving a verdict, annotators sometimes need to rely on commonsense knowledge. Here, we consider only basic facts which an average English speaker is likely to know - e.g. "Earth is a planet" or "raindrops consist of water". No other information beyond the question-answer pairs can be used in this phase.
We ask annotators to be relatively strict with what they consider commonsense, but use their own judgment. For example, we would consider "Canada is a country" commonsense, but not "Canada is the third-largest country in terms of land mass". If an annotator is in doubt as to whether something is considered commonsense, they should not consider it commonsense.
J.5.3 Justification Production
In addition to the verdict, we as mentioned also ask annotators in Phase Three to write a short statement justifying their verdict. This justification should explain the reasoning process used to reach the verdict, along with any commonsense knowledge. If calculations or comparisons were used, e.g. "6.3% is greater than 6.1%" or "10-4=6", they should be explicitly stated in the justification. Similarly, any rounding logic – e.g. "4.3 million is approximately 4 million" – should be explicitly stated here.
Other than commonsense knowledge, there should not be any new information presented in this statement. The justification should only describe how the annotators used the information present in the claim, the metadata, and the QA-pairs to reach their verdict. If a verdict cannot be reached, e.g. if the not enough information-label is chosen, annotators should instead describe what information is missing - e.g. "I cannot determine if Canada is the third-largest country, because the questions do not specify how large any countries are."
Similarly, in cases of conflicting evidence, annotators should describe which questions and answers lead to the conflict, and how they contradict – e.g. "This claim is cherry-picked as it looks only at the
Figure 16: Interface of quality control. ① Text field for entering the justification. ② Label of the claim and the checkbox of unreadable. Notice that once the unreadable option is selected, annotators do not need to select the label for the claim. ③ The question corresponds to the current claim. Here we have two question-answer pairs. If the annotator think the there exist potential problems with this question, check any options applied. ④ The answers correspond to the question on the left. If the annotator think the there exist potential problems with the answer, check any options applied. ⑤ Buttons for submitting the current claim, going to the previous claim, and next claim.
price of vanilla icecream, for which an increase did take place, but leaves out other flavours, where no increase happened."
J.6 Dispute Resolution
For some claims, there may be a disagreement between the labels produced by annotators in the question generation and quality control phases. In those cases, the claim will go through a second round of question generation and quality control. While the instructions given in Sections J.4.3 and J.5 still apply, we give a few extra recommendations specific to dispute resolution here.
J.6.1 Vague Claims
Some claims may pass to the dispute resolution phase because they are too vague for annotators in phases two and three to agree on the meaning. In order to catch these cases, the final step of dispute resolution – that is, the extra quality control step at the end – includes an additional label, Claim Too Vague. This should be select when and only when an annotator can understand the claim (e.g. it is readable), but there is too much doubt over how it is supposed to be interpreted. For example, the claim “Ohio is the best state” is too vague as it is not clear what “best” refers to.
J.6.2 Adding and Modifying Questions
The aim of dispute resolution is to resolve the conflict so that a potential new reader would come to a conclusive verdict. As such, the annotator should not necessarily agree with either the Phase Two or the Phase Three-verdict; they should attempt to make the fact-checking unambiguous. There may be cases where new questions must be added, and cases where existing questions should be changed but no new questions are necessary. There may also be cases where no change to the evidence is necessary at all, but where either the Phase Two or Phase Three-annotator has simply entered a wrong verdict. For this final category adding additional evidence to provide clarity can still be helpful, but it is not necessary; annotators should use their own judgment here.
J.6.3 NEI-verdicts
A common case for dispute resolution is the situation where the Phase Two annotator has selected Supported, Refuted, or Conflicting Evidence/Cherrypicking as the verdict, but the Phase Three annotator has selected Not Enough Evidence. This can happen for example if Phase Two annotators forget to gather some of the evidence they use to reach the verdict, rely on aspects only stated in the fact-checking article without making it explicit through a question-answer pair, or overestimate the strength of the evidence they have gathered. In these cases, the aim of dispute resolution is to gather additional evidence and resolve the conflict that way; i.e. it is not sufficient to give a Not Enough Information-verdict without attempting to add evidence (although the same time limit as in P2 applies).