SlowGuess/NAACL_2022_OCR huggingface dataset:
The annotator first needs to read the fact-checking article carefully, and then use one or two sentences to summarize the event described in this article as a claim. The annotator is encouraged to directly extract the sentences in the article as the claim, such as the title of the article, or the first few sentences in the first paragraph. If the annotator cannot find the sentence that can serve as a claim, you can use your own language to write the claim. When extracting the claim, there are the following considerations:
Each claim must be complete.
- For each claim, explicit bias should be removed.
Each claim should not have information leakage.
Please read the following detailed guidelines and corresponding normalized examples carefully:
- If the event described in the claim lacks important details, such as time and location, annotator need to add these necessary metadata to make the claim complete before it can be verified. For example, a total of 120,800 people took the entrance examination this year, but the enrollment plan is only 43,000. The claim needs to be rewritten as follows: In 2019, a total of 120,800 people participated in the Chengdu high school entrance examination, but the enrollment plan was only 43,000.
- If there exist special symbols in the claim, such symbols that may lead to bias for claim verification should be removed. For example: "Natural spray" helps you "sleep instantly".
Quotation marks should be removed in the sentence, as these special symbols implicitly indicates such a claim is non-factual. The model can predict the veracity simply based on the special symbols in the claim. This claim needs to be rewritten as: Natural spray helps you sleep instantly.
- Claims contains words that will lead to information leakage should be removed. For example: Rumors! Natural spray helps you sleep instantly. The word "Rumor!" in the claim directly pointed out that the claim is nonfactual, causing information leakage. The word "Rumor!" should be removed. Do not include information leaking words such as "rumors", "errors", "scams", etc. in the claim.
- Claims used rhetorical questions need to be rewritten into declarative sentences. Since most of the claims in the form of rhetorical question are nonfactual, the form of the rhetorical question exhibits a bias in the dataset. For example: if someone else gets the COVID-19 vaccine, can we not get the vaccine? It needs to be rewritten as: if someone else gets a COVID-19 vaccine, we do not need to get the vaccine..
- Claims that do not related to factuality should be discarded. There are two major types of claims that cannot be verified. The first category is speculative claims, such as: Shenzhen housing prices will rise next year. The second category is claims expressing personal opinions, such as: I think Donald Trump should be the president.
- A claim contains multiple statements should be split into multiple claims to be verified one by one. For example, a claim stated that: First, if someone else gets the COVID-19 vaccine, you do not need to get one. Also, the COVID-19 vaccine only needs one shot to protect against the virus. This claim includes two sub-claims. It needs to be split into two claims.
A.1.2 声明标注的指引 Guidelines for claim labeling
标注者在抽取出和规范化声明之后，需要根据事实验证的文章给出的结论，给每个声明打上标签。我们提供了以下三种标签，请选择其中
的一种。注意的是，对于大部分为真，部分为真，大部分为假，部分为假和半真半假的情况，我们统一归类为信息不足：
- 支持，有充分证据表明这个声明是被证据所支持的。
- 反对，有充分证据表明这个声明是被证据所反对的。
- 信息不足，没有足够的证据表明这个声明是被支持还是反对。
After extracting and normalizing the claim, annotators needs to label each claim based on the conclusions of the fact-checking article. We provide the following three labels, please choose one of them. Note that for conclusions such as mostly true, partially true, mostly false, partial false and mixture, we consider them as not enough information:
- Supported, there is sufficient evidence to show that this claim is supported by the evidence.
- Refuted, there is sufficient evidence to show that this claim is refuted by the evidence.
- Not enough information, there is not enough evidence to show whether this claim is supported or refuted.
A.1.3 证据标注的指引 Guidelines for evidence labelling
标注者需要阅读规范化过后的声明，事实验证的文章还有搜集到的源文档。标注者首先需要理解文章的验证思路，再从源文档当中直接选择能够作为证据的句子。针对每个声明，标注者最少选择1个，最多选择5个相关的句子作为证据。在选择句子作为证据的时候有以下注意事项：
- 请标注者选择完整的句子，以句号为结束标志。
- 选择句子作为证据的条件是，在仅仅基于当前选中的句子作为证据的前提下，能够验证给定的声明。也就是说，选中的句子必须要提供给足够的信息来帮助判断声明的事实性。
- 如果出现多于5个句子能够作为证据的情况，选择你认为最相关的5个句子；或者能够形成推理逻辑链的句子；或者和事实验证文章推理过程最相似的句子。
- 如果出现源文档互相矛盾的情况，优先选择支持事实验证文章结论的文档，从中选择相关的句子作为证据。
- 如果提供的源文档并没有提供足够的证据来验证声明，请报告这条声明。
- 如果提供的源文档只包含和事实验证文章结论矛盾的证据，请报告这条声明。
The annotator needs to read the normalized claim, the fact-checking article and the collected documents. Annotators first need to understand the verification process in the fact-checking article, and then directly select sentences from the sources documents. These selected sentence are used as evidence to verify the claim. For each claim, annotators should select at least 1 and at most 5 relevant sentences as evidence. There are the following considerations when choosing sentences as evidence:
- Please select a complete sentence which ends with a period.
- When selecting sentences as evidence, the annotator should consider given the selected sentences if the given claim can be verified. In other words, the selected sentences must provide sufficient information to predict the factuality of the given claim.
- If there are more than 5 sentences that can be used as evidence, choose the 5 sentences that you think are the most relevant; or the sentences that can form a reasoning chain for verification; or the sentence that is most similar to the reasoning process of the fact-checking article.
- If there are conflicting source documents, the documents that support the conclusion of the fact-checking article should be considered, and the most relevant sentences in these documents are selected as evidence.
- If the source documents do not provide sufficient evidence to verify the statement, please report the claim.
- If the source documents only contains evidence that contradicts the conclusion of the fact-checking article, please report this claim.
A.1.4 数据验证的指引 Guidelines for data validation
给定一个声明和搜集到的证据句子，标注者需要根据证据去判断这个声明的真实性。如果标注者认为提供的声明缺失重要信息，或者是不可读的，请报告该条声明。我们提供了以下三种标签，请选择其中的一种：
Given a claim and the evidence sentences, the annotator needs to label the factuality of the claim based on the evidence. If the annotator believes that the given claim lacks important information or is unreadable, please report the claim. We provide three kinds of labels, please choose one of them:
A.1.5 判断声明领域的指引 Guidelines for determining claim domain
标注者需要阅读声明，根据给出的五个领域判断声明属于哪个领域：
• 政治：主要是关于国际与国内政治等方面的声明。
- 公卫：主要是关于公共卫生方面的声明，比如有关新冠病毒，人体健康，食品安全等方面。
- 科学：主要是关于自然科学和工程技术等方面的声明。
文化：主要是关于历史，人文，娱乐，体育等方面的声明。
- 社会：主要是除了上述四类，社会生活方面的声明。
The annotator needs to read the claim and determine which domain the claim belongs to based on the five domains given:
- Politics: Claims mainly focus on international and domestic politics.
- Health: Claims mainly focus on public health, including topic related to COVID-19, health care, food safety, etc.
- Science: Claims mainly focus on natural science and technology.
Culture: Claims mainly focus on history, humanities, entertainment, sports, etc.
- Society: Claims not related on the above four categories, and related to daily social life.
A.1.6 验证声明的挑战（多选）Claim verification challenges (Multiple choice)
对声明进行事实验证往往会遇到许多挑战，挑战可以分为以下四类，标注者需要阅读事实验证的文章，判断验证声明时会遇到哪些挑战：
- 证据搜集：通过搜集证据，比如找相关的新闻，论文，法律法规等来验证声明。
• 专家咨询：通过咨询专家或者相关人士，比如外交部发言人陈述，部委回复，记者采访等来验证声明。
- 数值推理：通过数值的比较，趋势的分析来验证声明。
- 多模态：通过除了文本外的其他证据，比如图片、视频、音频来验证声明。
Factual verification of a claim often encounters many challenges. The challenges are summarized into the following four categories. The annotator needs to read the fact-checking article to determine which challenges will be encountered in verifying the claim:
- Evidence Collection: Verify the claim by collecting evidence, such as finding relevant news, papers, laws and regulations, etc.
- Expert Consultation: Verify the claim by consulting experts or related people, such as statements by the spokesperson of the Ministry of Foreign Affairs, replies from ministries and commissions, interviews with reporters, etc.
- Numerical Reasoning: Verify the claim by numerical comparison, trend analysis, etc.
- Multi-Modality: Verify the claim with other evidence besides articles, such as pictures, videos, and audio.