1. The annotator should first read the claim and metadata provided by the previous annotator, and the associated fact-checking article (including the verdict). We note that because phase-one annotators sometimes split decompose claims into parts, in some cases not all sections of the fact-checking article will be relevant.
2. The task is then to generation questions and answers about the claim such that a verdict can be given without knowledge of the fact-checking article. The sources and strategies used in the fact-checking article can serve as inspiration for questions and evidence for answers, but the fact-checking article should not be directly referenced as a source.
3. If an annotator believes a phase one claim has been extracted wrongly, they can correct it using the appropriate box. This is not necessary for most claims, but adds an extra layer of quality control. Guidance on correcting claims along with examples can be found in Section J.4.2.
4. We recommend constructing question-answer pairs iteratively, one at a time. That is, annotators should ask a question and attempt to answer it, and only then proceed to the next question.
5. Guidance on generating questions can be found in Section J.4.3.
6. Answers should be sought from the metadata, any of the sources listed on the fact-checking article (e.g. any hyperlinks to other sites), and when that is not possible (e.g. due to the hyperlinks being dead) from the internet using the search bar we provide.
7. Questions about metadata can be used to draw attention to aspects of the claim, in order to reason about publication date or publication source (see Section J.4.4).
8. WARNING: For persistence, we have stored all fact-checking articles on archive.org. Fact-checking articles may feature double-archivedlinks using both archive.org and archive.is, e.g.
https://web.archive.org/web/20201229212702/https://archive.md/28fMd Archive.org returns a 404 page for these. To view such a link, please just copy-paste the archive.is part (e.g.
https://archive.md/28fMd) into your browser.
9. Answers should be accompanied by a hyperlink to the source, and the type of the source - e.g. web text, a pdf - should be specified. We note that if the source type is set as metadata, the source link will automatically be set to the word metadata.
10. Answers can be either extraction, e.g. copy-pasted directly from the source, abstractive, e.g. written in free-form based on the source, or boolean, e.g. written as yes/no with an explanation taken either extractively or abstractively from the source. Where possible, we strongly prefer extractive answers.
11. If an answer cannot be found, we also allow annotators to mark the question as unanswerable. We ask annotators to use this instead of deleting unanswerable questions.
12. Guidance on generating answers can be found in Section J.4.6.
13. If enough questions have been asked to support a verdict, or if at least ten minutes have passed without the annotator finding enough evidence, a verdict should be given from our for labels described in Section J.3.4.
14. Annotators in phase two should base their verdict on the question-answer pairs they have generated, and not on the fact-checking article. Depending on what information has been retrieved, they may therefore disagree with the article.
15. Before proceeding to the next hit, the annotator will be shown a warning with the QA-pairs they have generated. They will also be shown their assigned label. They will be asked to confirm that the collected evidence is sufficient to assign the label they have chosen to the claim.