SlowGuess/NAACL_2022_OCR huggingface dataset:
Xuming Hu $^{1*}$ , Zhijiang Guo $^{2*}$ , Guanyu Wu $^{1}$ , Aiwei Liu $^{1}$ , Lijie Wen $^{1\dagger}$ , Philip S. Yu $^{1,3}$
$^{1}$ Tsinghua University
<sup>3</sup>University of Illinois at Chicago
$^{1}\{hxml9, wugy18, liuaw20\} @mails.tsinghua.edu.cn$
$^{2}$ zg283@cam.ac.uk  $^{1}$ wenlj@tsinghua.edu.cn  $^{3}$ psyu@uic.edu
The explosion of misinformation spreading in the media ecosystem urges for automated fact-checking. While misinformation spans both geographic and linguistic boundaries, most work in the field has focused on English. Datasets and tools available in other languages, such as Chinese, are limited. In order to bridge this gap, we construct CHEF, the first CHinese Evidence-based Fact-checking dataset of 10K real-world claims. The dataset covers multiple domains, ranging from politics to public health, and provides annotated evidence retrieved from the Internet. Further, we develop established baselines and a novel approach that is able to model the evidence retrieval as a latent variable, allowing jointly training with the veracity prediction model in an end-to-end fashion. Extensive experiments show that CHEF will provide a challenging testbed for the development of fact-checking systems designed to retrieve and reason over non-English claims. Source code and data are available<sup>1</sup>.
Misinformation is being spread online at increasing rates, posing a challenge to media platforms from newswire to social media. In order to combat the proliferation of misinformation, fact-checking is an essential task that assesses the veracity of a given claim based on evidence (Vlachos and Riedel, 2014). Fact-checking is commonly conducted by journalists. However, fact-checking is a time-consuming task, which can take journalists several hours or days (Adair et al., 2017). Thus, there is a need for automating the process.
Although misinformation spans both geographic and linguistic boundaries, most existing works focused on English (Wang, 2017; Thorne et al., 2018; Augenstein et al., 2019; Hanselowski et al., 2019;
Claim: 2019年,共有12.08万人参加成都中考,但招生计划只有4.3万。In 2019, a total of 120,800 students participated in the high school entrance examination in Chengdu, but schools only enrolled 43,000 students.
Document: 今年共有12.08万人参加中考，这个是成都全市，包括了20个区，高新区和天府新区的总参考人数。月前，教育局公布了2019年的普高招生计划。招生计划数进一步增加，上普高的机会更大了...中心城区（13个区）招生计划为43015人。This year, 120,800 people participated in the high school entrance examination. This number is for the entire city of Chengdu, including 20 districts, high-tech zone and Tianfu new district. A month ago, the Education Bureau announced the 2019 high school enrollment plan. The number of enrollment will be increased, indicating that there is a greater chance of going to high school... The plan of the central area (including 13 districts) is 43,015.
Verdict: Refuted; Domain: Society
Challenges: Evidence Collection; Numerical Reasoning
Table 1: An example from CHEF (Chinese is translated into English). The claim is refuted by the evidence, which are sentences retrieved (highlighted) from the document. For brevity, only the relevant snippet of the document is shown.
Chen et al., 2020). There only exists a handful of non-English datasets for verifying real-world claims. However, these datasets are either small in size (Baly et al., 2018), or designed for multilingual systems (Gupta and Srikumar, 2021). On the other hand, Khouja (2020) and Nørregaard and Derczynski (2021) created claims by paraphrasing sentences from non-English articles, but synthetic claims cannot replace real-world claims for training generally applicable fact-checking systems.
To bridge this gap, we introduce a Chinese dataset for Evidence-based Fact-checking (CHEF). CHEF includes claims that are not only relevant to the Chinese world, but also originally made in Chinese. It consists of 10,000 real-world claims, collected from 6 Chinese fact-checking websites covered multiple domains and paired with annotated evidence. To ensure annotation consistency,
<https://github.com/THU-BPM/CHEF>
*Equally Contributed.
Human Language Technologies, pages 3362 - 3376
Dataset	Natural	Domain	#Claims	Language	Evidence
Type	Source	Retrieved	Annotated
FEVER (Thorne et al., 2018)	X	Multiple	185,445	English	Text	Wiki	✓	✓
HOVER (Jiang et al., 2020)	X	Multiple	26,171	English	Text	Wiki	✓	✓
TabFact (Chen et al., 2020)	X	Multiple	92,283	English	Table	Wiki	X	✓
InfoTabs (Gupta et al., 2020)	X	Multiple	23,738	English	Table	Wiki	X	✓
ANT (Khouja, 2020)	X	Multiple	4,547	Arabic	X	X	X	X
VitaminC (Schuster et al., 2021)	X	Multiple	488,904	English	Text	Wiki	X	✓
DanFEVER (Nørregaard and Derczynski, 2021)	X	Multiple	6,407	Danish	Text	Wiki	✓	✓
FEVEROUS (Aly et al., 2021)	X	Multiple	87,026	English	Text/Table	Wiki	✓	✓
PolitiFact (Vlachos and Riedel, 2014)	✓	Politics	106	English	Meta/Text	FC	X	X
PunditFact (Rashkin et al., 2017)	✓	Multiple	4,361	English	X	X	X	X
Liar (Wang, 2017)	✓	Multiple	12,836	English	Meta	FC	X	X
Verify (Baly et al., 2018)	✓	Politics	422	Mul(2)	Text	Internet	✓	X
MultiFC (Augenstein et al., 2019)	✓	Multiple	36,534	English	Meta/Text	Internet	✓	X
Snopes (Hanselowski et al., 2019)	✓	Multiple	6,422	English	Text	FC	X	X
SciFact (Wadden et al., 2020)	✓	Science	1,409	English	Text	Paper	X	X
PUBHEALTH (Kotonya and Toni, 2020b)	✓	Health	11,832	English	Text	FC	X	X
AnswerFact (Zhang et al., 2020)	✓	Product	60,864	English	Meta/Text	Amazon	✓	X
FakeCovid (Shahi and Nandini, 2020)	✓	Health	5,182	Mul(3)	X	X	X	X
XFact (Gupta and Srikumar, 2021)	✓	Multiple	31,189	Mul(25)	Meta/Text	Internet	✓	X
CHEF	✓	Multiple	10,000	Chinese	Meta/Text	Internet	✓	✓
Table 2: Comparisons of fact-checking datasets. Type in the header means the type of evidence used, which can be text, metadata or both. Source means where the evidence are collected from, such as Wikipedia (Wiki), fact-checking websites (FC). Retrieved denotes if the evidence is given or retrieved from the source. Annotated means whether the evidence is manually annotated. Verify, FakeCovid, XFact contain claims in multiple languages, but Chinese claims are not included.
we developed suitable guidelines and performed data validation<sup>2</sup>. We shared some of the insights obtained during the annotation process that we hope will be beneficial to other non-English annotation efforts. Table 1 shows an instance from CHEF. In order to verify the claim, one needs to first retrieve the evidence sentences from related documents (e.g. government reports), then predict the veracity based on the evidence. After comparing the statistics of the entire city and central area, we can reach the verdict that the claim is refuted by evidence. To characterize the challenge of the dataset presented, we perform a thorough analysis and demonstrate the utility of the dataset by developing two types of baselines, including pipeline and joint systems. Our key contributions are summarized as follows:
1. We provide the first sizable multi-domain Chinese dataset for automated fact-checking. It consists of 10K real-world claims with manually annotated evidence sentences.
2. We further propose an approach that is able to model the evidence selection as a latent variable, which can be jointly trained with the veracity prediction module.
3. We develop several established baselines and
conduct a detailed analysis of the systems evaluated on the dataset, identifying challenges that need to be addressed in future research.
2 Background: Dataset Comparisons
In this section, we reviewed the existing fact-checking dataset as summarized in Table 2. Following Guo et al. (2022), we grouped the datasets into two categories: natural and synthetic. Natural datasets consist of real-world claims, while synthetic datasets contain claims created artificially by mutating sentences from Wikipedia articles.
2.1 Non-English Dataset
Existing efforts in the construction of non-English datasets are limited, both in scope and in size. Verify (Baly et al., 2018) contains 422 claims in Arabic and FakeCovid (Shahi and Nandini, 2020) has 3,306 non-English claims about COVID-19. Though XFact (Gupta et al., 2020) includes 31,189 claims in 25 languages, it mainly focuses on the multilingual setting, where the average number of instances per language is 1,248. More importantly, these datasets do not include annotated evidence. For example, XFact used search summaries returned by Google as evidence. On the other hand, Khouja (2020) and Nørregaard and Derczynski (2021) constructed synthetic datasets by mutating sentences from Arabic news and Danish
2The annotation guideline is provided in the appendix.
Wikipedia articles, respectively. Unlike previous efforts, CHEF consists of 10K real-world claims paired with annotated evidence.
There exist Chinese datasets focused on rumor detection (Ma et al., 2016; Zhang et al., 2021), which is classified into claim detection (Kotonya and Toni, 2020a; Guo et al., 2022), as it is based on language subjectivity and growth of readership (Qazvinian et al., 2011). A claim can be factual regardless of whether it is a rumour (Zubiaga et al., 2018). Unlike existing rumor detection datasets, CHEF focuses on factuality of the claim.
2.2 Evidence-Based Fact-Checking
Early efforts predicted the veracity solely based on the claims or with metadata (Rashkin et al., 2017; Wang, 2017), but relying on surface patterns of claims without considering the state of the world fails to identify well-presented misinformation (Schuster et al., 2020). Therefore, synthetic datasets (Thorne et al., 2018; Jiang et al., 2020; Aly et al., 2021) considered Wikipedia as the source of evidence and annotated the sentences supporting or refuting each claim. However, these efforts restricted world knowledge to a single source (i.e. Wikipedia), ignoring the challenge of retrieving evidence from heterogeneous sources on the Internet.
To address this, recent natural datasets (Augenstein et al., 2019; Gupta and Srikumar, 2021) used the summary snippets returned by Google as evidence. One key limitation of this approach is that summary snippets do not provide sufficient information to verify the claim. Gupta and Srikumar (2021) showed that only  $45\%$  of snippets provide sufficient information, while  $83\%$  of the full text from web pages provides sufficient evidence to determine veracity of the claim. To construct a better evidence-based dataset, we retrieve documents from web pages and manually select relevant evidence sentences from documents as evidence. Such a design makes CHEF suitable to train fact-checking systems that can extract evidence from web-sources and validate real-world claims based on evidence found on the Internet.
CHEF is constructed in four stages: data collection, claim labeling, evidence retrieval and data validation. Data collection selects sources, crawls claims and associated metadata. Claim labeling identifies claims from fact-checking articles and
Website	Domain	URL	Total
Piyao	Multiple	www.piyao.org.cn	3,741
TFC	Multiple	tfc-taiwan.org.tw	1,759
Mygopen	Multiple	www.mygopen.com	1,654
Jiaozhen	Multiple	vp的事实.qq.com	157
Cnews	Multiple	m.chinanews.com	2,689
Total	Multiple	-	10,000
Table 3: Statistics of data sources. Piyao, TFC, My-GoPen and Jiaozhen are fact-checking websites. Cnews is a news website.
assigns the veracity labels of claims based on the article. Evidence retrieval collects documents from the Internet and selects the most relevant sentences as evidence. Data validation controls the annotation quality. The annotation team has 25 members, 5 of them are only involved in data validation. All annotators are native Chinese speakers. To ensure the annotation quality, they were trained by the authors and went through several pilot annotations.
We crawled all active Chinese fact-checking websites listed by Duke Reporters<sup>3</sup>. However, most claims fact-checked by the fact-checkers are non-factual, solely relying on such claims will lead to an imbalance dataset. Therefore, we followed Kotonya and Toni (2020b) by crawling articles from the news review site. As shown in Table 3, this resulted in 5 websites in total. From each website, we crawled the full text of the article and corresponding metadata (e.g. author, domain, URL publication date). Totally, we crawled 14,770 fact-checking and news articles. There exists a number of crawling issues, such as the article could not be retrieved, or the content is not textual. We removed such instances. Next, we checked the dataset for duplications. Upon manual inspections, this was mainly due to them appearing on different websites. All duplications would be in the training split of the dataset, so that the model would not have an unfair advantage. As shown in Figure 5, claims cover multiple domains, including politics, public health, science, society and culture. More than  $36\%$  of claims belong to public health domain, as many fact-checking articles focused on countering misinformation related to COVID-19. The society domain has second most claims, which involves social events that are closely related to people's daily lives.
3www.reporterslab.org/fact-checking/
Figure 1: Distributions of domains. Each instance is categorized into five different domains.
3.2 Claim Labeling
The major challenge of constructing a non-English dataset is extracting a claim and its veracity from a fact-checking article usually requires human efforts. Unlike fact-checking articles in English, many non-English articles (e.g. Chinese, Hindi, Filipino) do not explicitly give the claim and assign the veracity. Therefore, extracting the claim, which can appear in the title, or anywhere in the article, requires manual efforts. Before labeling the claim, we need to extract them from the fact-checking articles. When performing claim extraction, annotators need to read the fact-checking article first, then identify the claim. They are encouraged to select sentences directly from the article. However, resulted sentences may not be complete, which means they do not provide enough context for fact-checking. One common case is that the sentence describing a fact often lacks the time stamp, or the location. For example, the claim "Price of pork increases dramatically due to the African swine fever" is factual in 2020 but non-factual in 2021. Therefore, annotators are asked to complete the claim by adding missing content to ensure the claim to be standalone for later verification<sup>4</sup>. Another issue is that Chinese fact-checkers tend to use rhetorical questions to express non-factual claims. To alleviate the bias that the factuality of a claim can be decided by its surface form, annotators are required to paraphrase the questions into declarative statements.
Next, annotators are required to label the extracted claims. English fact-checking articles often provide different truth-rating scales, such as false, mostly false and mixture, while many non-English counterparts do not have such taxonomies. Therefore, annotators need to label the extracted claim based on the understanding of the fact-checking
Figure 2: Distributions of challenges. Each instance can have multiple challenges. Evidence collection means finding relevant textual information from web-sources. Expert consultation collects information directly from relevant people. Numerical reasoning requires inference over numbers and multi-modality requires collect and infer over multi-modal evidence.
article. Journalism researchers showed that fine-grained labels are often assigned inconsistently due to subjectivity (Uscinski and Butler, 2013; Lim, 2018). Therefore, we chose to follow previous efforts (Thorne et al., 2018; Hanselowski et al., 2019) by adopting three types of labels: supported (SUP), refuted (REF) and not enough information (NEI), given the evidence. The distribution of labels in CHEF is shown in Table 4. CHEF consists of a majority of refuted claims, as the majority of fact-checking articles aim to debunk non-factual claims.
3.3 Evidence Retrieval
When verifying a claim, journalists first find information relating to the fact and evaluate it given the collected evidence. As shown in Figure 1, the biggest challenge of verifying a claim is to collect relevant evidence. In order to validate real-world claims, we chose to manually extract evidence from web-sources. We have two measures to ensure the reliability of the evidence. Firstly, we maintained a list of misinformation and disinformation websites, all search results from these websites will be filtered out. Secondly, we required the annotators to manually select evidence sentences from the search results. In order to collect evidence from the web-sources, we first submitted each claim as a query to the Google Search API by following Augenstein et al. (2019) and Gupta and Srikumar (2021). The ten most highly ranked search results are retrieved. For each result, we saved the search rank, URL, time stamp and document. Then we filtered out results from fact-checking websites to prevent the
4More annotation details are provided in appendix.
Split	SUP	REF	NEI	Total
Train	2,877	4,399	776	8,002
Dev	333	333	333	999
Test	333	333	333	999
Avg #Words in the Claim	28
Avg #Words in the Google Snippets	68
Avg #Words in the Evidence Sentences	126
Avg #Words in the Source Documents	3,691
Table 4: Dataset split sizes and statistics for CHEF.
answer from being trivially found. Next, annotators were asked to select sentences from the resulted documents. To maintain a balance between keeping relevant and removing irrelevant information, we followed Thorne et al. (2018) and Hanselowski et al. (2019) that up to five sentences were selected as evidence. Before deciding which sentences should be selected, annotators were required to answer auxiliary questions, such as "Whether selected sentences provide sufficient information for factual verification?" They were encouraged to select the five most relevant sentences, but they were allowed to pick less when relevant sentences are not available. A small fraction  $(5.6\%)$  of instances do not have any relevant evidence, and we chose to discard them.
3.4 Data Validation
To ensure the annotation consistency, we conducted an additional 5-way inter-annotator agreement and manual validation. For inter-annotator agreement, we randomly selected  $3\%$ $(n = 310)$  of claims to be annotated by 5 annotators. We calculated the Fleiss  $K$  score (Fleiss, 1971) to be 0.74, which is comparable with 0.68 reported in Thorne et al. (2018) and 0.70 in Hanselowski et al. (2019). In order to verify if evidence sentences provide sufficient information, we chose another 310 instances. The second group of annotators were required to assign the labels based on the evidence sentences. We found that  $88.7\%$  of the instances were labeled correctly and  $83.6\%$  of them provided sufficient information to determine the veracity. Finally, as shown in Table 4, we partitioned the dataset CHEF into training, development and test sets. Our development and test sets have balanced class distributions. Each claim is paired with Google snippets, evidence sentences and source documents.
4 Baseline Systems
Unlike previous natural datasets, CHEF requires the system to first retrieve the evidence sentences from the documents, then predict the veracity based on the evidence. Therefore, we design two types of baselines: pipeline and joint systems.
4.1 Pipeline System
The pipeline system treats evidence retrieval and veracity prediction as two independent steps.
4.1.1 Evidence Retrieval
Given the claim and documents, this step aims to select the most relevant sentences from documents as evidence, which can be viewed as a ranking problem. Thus, we adopt the following models:
Surface Ranker Following retrieval models designed for synthetic datasets (Thorne et al., 2018; Jiang et al., 2020; Aly et al., 2021), We use TF-IDF to sort the most similar sentences first and tune a cut-off using validation accuracy on the dev set.
Semantic Ranker Inspired by Nie et al. (2019) and Liu et al. (2020), we choose semantic matching based on BERT (Devlin et al., 2019) pre-trained on Chinese corpus (Wolf et al., 2020). The cosine similarity scores between the embedding of the claim and the embeddings of other sentences in the document are used for ranking.
Hybrid Ranker Since semantic encoding is complementary to surface form matching, they can be combined for better ranking. Following Shaar et al. (2020), we use the rankSVM, based on the feature sets of rankings returned by TF-IDF and the similarity scores computed with BERT.
Google Snippets As discussed in Section 2, existing natural datasets (Augenstein et al., 2019; Gupta and Srikumar, 2021) do not require the system to retrieve the evidence sentences from the documents. Instead, they used summary snippets returned by the Google Search Engine as evidence. We also include this type of evidence for comparisons.
4.1.2 Veracity Prediction
After retrieving the evidence sentences, veracity prediction aims to predict the label of the given claim. We implement the following classifiers:
BERT-Based Model Following Jiang et al. (2020) and Schuster et al. (2021), we use a multi-layer perceptron with embeddings from BERT as
the classifier. The embeddings of claim and retrieved evidence are concatenated as the input. The model performs the classification based on the output representation of the CLS token.
Attention-Based Model Following Gupta and Srikumar (2021), we first extract the output embedding of the CLS token of each selected evidence and calculate relevance weights with the claim through dot product attention. Then we feed the concatenated claim and weighted evidence into the BERT-based classifier.
Graph-Based Model Recent efforts (Zhou et al., 2019; Zhong et al., 2020; Liu et al., 2020) showed that graphs help to capture richer interactions among multiple evidence for fact-checking. We adopt the Kernel Graph Attention Network (Liu et al., 2020) for veracity prediction. The evidence graph is constructed based on the claim and evidence sentences, then node and edge kernels are used to conduct fine-grained evidence propagation. The updated node representations are used to calculate the claim label probability.
4.2 Joint System
Evidence retrieval in the pipeline system could not solicit optimization feedback obtained from veracity prediction. In order to optimize two steps jointly, we proposed to model the evidence retrieval as a latent variable. The joint system contains two modules: a latent retriever and a classifier. For the classifier, we used the same models described in Section 4.1. Latent retriever labels each sentence in the documents with a binary mask. Sentences labeled with 1 are selected as the evidence, while sentences labeled with 0 will be neglected.
4.2.1 Latent Retriever
We built the latent retriever based on the Hard Kumaraswamy distribution (Bastings et al., 2019), which gives support to binary outcomes and allows for reparameterized gradient estimates. We first stretch the Kumaraswamy distribution (Kumaraswamy, 1980) to include 0 and 1 by the support of open interval  $(l,r)$  where  $l < 0$  and  $r > 1$ , defined as  $K^{\prime} \sim \mathrm{Kuma}(a,b,l,r)$  with CDF:
$$
F _ {K ^ {\prime}} \left(k ^ {\prime}; a, b, l, r\right) = F _ {K ^ {\prime}} \left(\left(k ^ {\prime} - l\right) / (r - l); a, b\right) \tag {1}
$$
A sigmoid function  $k^{\prime \prime} = \min (1, \max (0, k^{\prime}))$  is used to rectify random variables into the closed interval [0, 1], denoted by  $K^{\prime \prime} \sim \mathrm{HardKuma}(a, b, l, r)$  and  $k^{\prime \prime} = s(u; a, b, l, r)$  for short. Note that we map all negative values  $k^{\prime} \in (l, 0]$  into  $k^{\prime \prime} = 0$  and  $k^{\prime} \in [1, r)$  into  $k^{\prime \prime} = 1$  deterministically, so the sets whose masses under  $\mathrm{Kuma}(k^{\prime}|a, b, l, r)$  are available in the closed form:
$$
\mathbb {P} \left(K ^ {\prime \prime} = 0\right) = F _ {K} \left(\frac {- l}{r - l}; a, b\right) \tag {2}
$$
$$
\mathbb {P} (K ^ {\prime \prime} = 1) = 1 - F _ {K} (\frac {1 - l}{r - l}; a, b)
$$