SlowGuess/ACL_2020_OCR huggingface dataset:
Shaden Shaar<sup>1</sup>, Nikolay Babulkov<sup>2</sup>, Giovanni Da San Martino<sup>1</sup>, Preslav Nakov<sup>1</sup>
$^{1}$ Qatar Computing Research Institute, HBKU, Doha, Qatar
$^{2}$ Sofia University, Sofia, Bulgaria
{sshar, gmartino, pnakov}@hbku.edu.qa
nbabulkov@gmail.com
The recent proliferation of "fake news" has triggered a number of responses, most notably the emergence of several manual fact-checking initiatives. As a result and over time, a large number of fact-checked claims have been accumulated, which increases the likelihood that a new claim in social media or a new statement by a politician might have already been fact-checked by some trusted fact-checking organization, as viral claims often come back after a while in social media, and politicians like to repeat their favorite statements, true or false, over and over again. As manual fact-checking is very time-consuming (and fully automatic fact-checking has credibility issues), it is important to try to save this effort and to avoid wasting time on claims that have already been fact-checked. Interestingly, despite the importance of the task, it has been largely ignored by the research community so far. Here, we aim to bridge this gap. In particular, we formulate the task and we discuss how it relates to, but also differs from, previous work. We further create a specialized dataset, which we release to the research community. Finally, we present learning-to-rank experiments that demonstrate sizable improvements over state-of-the-art retrieval and textual similarity approaches.
The year 2016 was marked by massive disinformation campaigns related to Brexit and the US Presidential Elections. While false statements are not a new phenomenon, e.g., yellow press and tabloids have been around for decades, this time things were notably different in terms of scale and effectiveness thanks to social media platforms, which provided both a medium to reach millions of users and an easy way to micro-target specific narrow groups of voters based on precise geographical, demographic, psychological, and/or political profiling.
Governments, international organizations, tech companies, media, journalists, and regular users launched a number of initiatives to limit the impact of the newly emerging large-scale weaponization of disinformation $^{1}$  online. Notably, this included manual fact-checking initiatives, which aimed at debunking various false claims, with the hope to limit its impact, but also to educate the public that not all claims online are true.
Over time, the number of such initiatives grew substantially, e.g., at the time of writing, the Duke Reporters' Lab lists 237 active fact-checking organizations plus another 92 inactive.2 While some organizations debunked just a couple of hundred claims, others such as Politifact, FactCheck.org, Snopes, and Full Fact have fact-checked thousands or even tens of thousands of claims.
The value of these collections of resources has been recognized in the research community, and they have been used to train systems to perform automatic fact-checking (Popat et al., 2017; Wang, 2017; Zlatkova et al., 2019) or to detect checkworthy claims in political debates (Hassan et al., 2015; Gencheva et al., 2017; Patwari et al., 2017; Vasileva et al., 2019). There have also been datasets that combine claims from multiple fact-checking organizations (Augenstein et al., 2019), again with the aim of performing automatic fact-checking.
<sup>1</sup>In the public discourse, the problem is generally known as "fake news", a term that was declared Word of the Year 2017 by Collins dictionary. Despite its popularity, it remains a confusing term, with no generally agreed upon definition. It is also misleading as it puts emphasis on (a) the claim being false, while generally ignoring (b) its intention to do harm. In contrast, the term disinformation covers both aspects (a) and (b), and it is generally preferred at the EU level.
$^{2}$ http://reporterslab.org/
fact-checking/
<sup>3</sup>http://www.politifact.com/
<sup>4</sup>http://www+factcheck.org/
5http://www.snopes.com/
<http://fullfact.org/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3607-3618
Figure 1: A general information verification pipeline.
It has been argued that checking against a database of previously fact-checked claims should be an integral step of an end-to-end automated fact-checking pipeline (Hassan et al., 2017). This is illustrated in Figure 1, which shows the general steps of such a pipeline (Elsayed et al., 2019):  $(i)$  assess the check-worthiness of the claim (which could come from social media, from a political debate, etc.),  $(ii)$  check whether a similar claim has been previously fact-checked (the task we focus on here),  $(iii)$  retrieve evidence (from the Web, from social media, from Wikipedia, from a knowledge base, etc.), and  $(iv)$  assess the factuality of the claim.
From a fact-checkers' point of view, the abundance of previously fact-checked claims increases the likelihood that the next claim that needs to be checked would have been fact-checked already by some trusted organization. Indeed, viral claims often come back after a while in social media, and politicians are known to repeat the same claims over and over again.[7] Thus, before spending hours fact-checking a claim manually, it is worth first making sure that nobody has done it already.
On another point, manual fact-checking often comes too late. A study has shown that "fake news" spreads six times faster than real news (Vosoughi et al., 2018). Another study has indicated that over  $50\%$  of the spread of some viral claims happens within the first ten minutes of their posting on social media (Zaman et al., 2014). At the same time, detecting that a new viral claim has already been fact-checked can be done automatically and very quickly, thus allowing for a timely action that can limit the spread and the potential malicious impact.
From a journalistic perspective, the ability to check quickly whether a claim has been previously fact-checked could be revolutionizing as it would allow putting politicians on the spot in real time, e.g., during a live interview. In such a scenario, automatic fact-checking would be of limited utility as, given the current state of technology, it does not offer enough credibility in the eyes of a journalist.
Interestingly, despite the importance of the task of detecting whether a claim has been fact-checked in the past, it has been largely ignored by the research community. Here, we aim to bridge this gap. Our contributions can be summarized as follows:
- We formulate the task and we discuss how it relates to, but differs from, previous work.
- We create a specialized dataset, which we release to the research community.8 Unlike previous work in fact-checking, which used normalized claims from fact-checking datasets, we work with naturally occurring claims, e.g., in debates or in social media.
- We propose a learning-to-rank model that achieves sizable improvements over state-of-the-art retrieval and textual similarity models.
The remainder of this paper is organized as follows: Section 2 discusses related work, Section 3 introduces the task, Section 4 presents the dataset, Section 5 discusses the evaluation measures, Section 6 presents the models we experiment with, Section 7 described our experiments, and Section 8 concludes and discusses future work.
To the best of our knowledge, the task of detecting whether a claim has been previously fact-checked was not addressed before. Hassan et al. (2017) mentioned it as an integral step of their end-to-end automated fact-checking pipeline, but there was very little detail provided about this component and it was not evaluated.
In an industrial setting, Google has developed Fact Check Explorer, which is an exploration tool that allows users to search a number of fact-checking websites (those that use ClaimReview from schema.org) for the mentions of a topic, a person, etc. However, the tool cannot handle a complex claim, as it runs Google search, which is not optimized for semantic matching of long claims. While this might change in the future, as there have been reports that Google has started using BERT in its search, at the time of writing, the tool could not handle a long claim as an input.
$^{8}$ Data and code are available at the following URL: https://github.com/sshaar/  
That-is-a-Known-Lie  
 $^{9}$ http://toolbox.google.com/factcheck/explorer  
 $^{10}$ http://schema.org/ClaimReview
President Trump has repeated one claim over 80 times: http://tinyurl.com/yblcb5q5.
A very similar work is the ClaimsKG dataset and system (Tchechmedjiev et al., 2019), which includes  $28\mathrm{K}$  claims from multiple sources, organized into a knowledge graph (KG). The system can perform data exploration, e.g., it can find all claims that contain a certain named entity or keyphrase. In contrast, we are interested in detecting whether a claim was previously fact-checked.
Other work has focused on creating datasets of textual fact-checked claims, without building KGs. Some of the larger ones include the Liar, Liar dataset of 12.8K claims from PolitiFact (Wang, 2017), and the MultiFC dataset of 38K claims from 26 fact-checking organizations (Augenstein et al., 2019), the 10K claims Truth of Various Shades (Rashkin et al., 2017) dataset, among several other datasets, which were used for automatic fact-checking of individual claims, not for checking whether an input claim was fact-checked previously. Note that while the above work used manually normalized claims as input, we work with naturally occurring claims as they were made in political debates and speeches or in social media.
There has also been a lot of research on automatic fact-checking of claims and rumors, going in several different directions. One research direction focuses on the social aspects of the claim and how users in social media react to it (Canini et al., 2011; Castillo et al., 2011; Ma et al., 2016; Gorrell et al., 2019; Ma et al., 2019). Another direction mines the Web for information that proves or disproves the claim (Mukherjee and Weikum, 2015; Karadzhov et al., 2017; Popat et al., 2017; Baly et al., 2018b; Mihaylova et al., 2018; Nadeem et al., 2019). In either case, it is important to model the reliability of the source as well as the stance of the claim with respect to other claims; in fact, it has been proposed that a claim can be fact-checked based on its source alone (Baly et al., 2018a) or based on its stance alone (Dungs et al., 2018). A third direction performs fact-checking against Wikipedia (Thorne et al., 2018; Nie et al., 2019), or against a general collection of documents (Miranda et al., 2019). A fourth direction uses a knowledge base or a knowledge graph (Ciampaglia et al., 2015; Shiadralkar et al., 2017; Gad-Elrab et al., 2019a,b; Huynh and Papotti, 2019). Yet another direction performs fact-checking based on tables (Chen et al., 2019). There is also recent work on using language models as knowledge bases (Petroni et al., 2019). Ours is yet another research direction.
While our main contribution here is the new task and the new dataset, we should also mentioned some work on retrieving documents. In our experiments, we perform retrieval using BM25 (Robertson and Zaragoza, 2009) and re-ranking using BERT-based similarity, which is a common strategy in recent state-of-the-art retrieval models (Akkalyoncu Yilmaz et al., 2019a; Nogueira and Cho, 2019; Akkalyoncu Yilmaz et al., 2019b).
Our approach is most similar to that of (Akkalonyncu Yilmaz et al., 2019a), but we differ, as we perform matching, both with BM25 and with BERT, against the normalized claim, against the title, and against the full text of the articles in the fact-checking dataset; we also use both scores and reciprocal ranks when combining different scores and rankings. Moreover, we use sentence-BERT instead of BERT. Previous work has argued that BERT by itself does not yield good sentence representation. Thus, approaches such as sentence-BERT (Reimers and Gurevych, 2019) have been proposed, which are specifically trained to produce good sentence-level representations. This is achieved using Siamese BERT networks that are fine-tuned on NLI and STS-B data. Indeed, in our experiments, we found sentence-BERT to perform much better than BERT. The Universal Sentence Encoder (Cer et al., 2018) is another alternative, but sentence-BERT worked better in our experiments.
Finally, our task is related to semantic relatedness tasks, e.g., from the GLUE benchmark (Wang et al., 2018), such as natural language inference, or NLI task (Williams et al., 2018), recognizing textual entailment, or RTE (Bentivogli et al., 2009), paraphrase detection (Dolan and Brockett, 2005), and semantic textual similarity, or STS-B (Cer et al., 2017). However, it also differs from them, as we will see in the following section.
We define the task as follows: Given a checkworthy input claim and a set of verified claims, rank those verified claims, so that the claims that can help verify the input claim, or a sub-claim in it, are ranked above any claim that is not helpful to verify the input claim.
Table 1 shows some examples of input-verified claim pairs, where the input claims are sentences from the 2016 US Presidential debates, and the verified claims are the corresponding fact-checked counter-parts in PolitiFact.
No.	Input claim	Manually annotated claim in PolitiFact
1	Richard Nixon released tax returns when he was under audit.	Richard Nixon released tax returns when he was under audit.
2	Hillary wants to give amnesty.	Says Hillary Clinton “wants to have open borders.”
3	People with tremendous medical difficulty and medical problems are pouring in, and in many, in many cases, it’s contagious.	Says “2,267 caravan invaders have tuberculosis, HIV, chickenpox and other health issues”
4	He actually advocated for the actions we took in Libya and urged that Gadhafi be taken out, after actually doing some business with him one time.	Says Donald Trump is “on record extensively supporting the intervention in Libya.”
5	He actually advocated for the actions we took in Libya and urged that Gadhafi be taken out, after actually doing some business with him one time.	When Moammar Gadhafi was set to visit the United Nations, and no one would let him stay in New York, Trump allowed Gadhafi to set up an elaborate tent at his Westchester County (New York) estate.
Table 1: PolitiFact: Input-verified claim pairs. The input claims are sentences from the 2016 US Presidential debates, and the verified claims are their corresponding fact-checked counter-parts in PolitiFact.
We can see on line 1 of Table 1 a trivial case, where the verified claim is identical to the input claim; however, such cases are not very frequent, as the experiments with the BM25 baseline in Section 7 below will show.
Lines 2 and 3 show harder cases, where the input claim and its manually annotated counter-part are quite different in their lexical choice, and yet the latter can serve to verify the former.
Lines 4 and 5, show a complex input claim, which contains two sub-claims, each of which is verified by two corresponding claims in PolitiFact.
From the above examples, it is clear that ours is not a paraphrasing task, as illustrated by examples 2-5. It is also not a natural language inference (NLI) or a recognizing textual entailment (RTE) task, as a claim can have sub-claims, which complicates entailment reasoning (as illustrated by examples 4-5). Finally, the task goes beyond simple textual similarity, and thus it is not just an instance of semantic textual similarity (STS-B).
Note that we do not try to define formally what makes a verified claim a good match for an input claim. Instead, we trust the manual annotations for this by fact-checking experts, which they perform when they comment on the claims made in political debates and speeches. In many cases, the fact-checkers have explicitly indicated which previously fact-checked claim corresponds to a given original claim in a debate/speech. A similar approach was adopted for a related task, e.g., it was used to obtain annotated training and testing data for the Check-Worthiness task of the CLEF Check-That! Lab (Atanasova et al., 2018, 2019; Barrón-Cedeno et al., 2020).
We created two datasets by collecting, for each of them, a set of verified claims and matching input-verified claims pairs (below, we will also refer to these pairs as Input-VerClaim pairs): the first dataset, PolitiFact, is about political debates and speeches and it is described in Section 4.1; the second dataset, Snopes, includes tweets, and it is described in Section 4.2.
4.1 PolitiFact Dataset
PolitiFact is a fact-checking website that focuses on claims made by politicians, elected officials, and influential people in general. PolitiFact fact-checks claims by assigning a truth value to them and publishing an article that gives background information and explains the assigned label. This is similar to how other fact-checking websites operate.
We retrieved a total of 16,636 verified claims from PolitiFact, populating for each of them the following fields:
- VerClaim: the text of the claim, which is a normalized version of the original claim, as the human fact-checkers typically reformulate it, e.g., to make it clearer, context-independent, and self-contained;
- TruthValue: the label assigned to the claim;[11]
- Title: the title of the article on PolitiFact that discusses the claim;
- Body: the body of the article.
<sup>11</sup>We do not use the claim veracity labels in our experiments, but we collect them for possible future use.
No.	Tweet	Manually annotated claim in Snopes
1	Welp...its official...Kim Kardashian finally decided to divorce Kanye West https://t.co/C2p25mxWJO — Ashlee Marie Preston (@AshleeMPreston) October 12, 2018	Kanye West and Kim Kardashian announced that they were divorcing in October 2018.
2	Kim Kardashian and Kanye West are splitting up https://t.co/epwKG7aSBg pic.twitter.com/u7qqojWVIR — ELLE Magazine (US) (@ELLEmagazine) October 18, 2018	Kanye West and Kim Kardashian announced that they were divorcing in October 2018.
3	Everyone should be able to access high-quality, affordable, gender-affirming health care. But the Trump administration is trying to roll back important protections for trans Americans. Help fight back by leaving a comment for HHS in protest: https://t.co/pKDcOqbsc7 — Elizabeth Warren (@ewarren) August 13, 2019	U.S. Sen. Elizabeth Warren said or argued to the effect that taxpayers must fund sex reassignment surgery.
Table 2: Snopes: Input-VerClaim claim pairs. The input claims are tweets and the verified claims are their corresponding fact-checked counter-parts in Snopes.
Often, after a major political event, such as a political speech or a debate, PolitiFact publishes reports<sup>12</sup> that discuss the factuality of some of the claims made during that event. Importantly for us, in these reports, some of the claims are linked to previously verified claims in PolitiFact. Such pairs of an original claim and a previously verified claim form our Claim-VerClaim pairs.
We collected such overview reports for 78 public events in the period 2012-2019, from which we collected a total of 768 Input-VerClaim pairs. Given an Input claim, we refer to the corresponding verified claim in the pair as its matching VerClaim claim. In general, there is a 1:1 correspondence, but in some cases an Input claim is mapped to multiple VerClaim claims in the database, and in other cases, multiple Input claims are matched to the same VerClaim claim.
Thus, the task in Section 3 reads as follows when instantiated to the PolitiFact dataset: given an Input claim, rank all 16,636 VerClaim claims, so that its matching VerClaim claims are ranked at the top.
4.2 Snopes Dataset
Snopes is a website specialized in fact-checking myths, rumors, and urban legends. We used information from it to create a second dataset, this time focusing on tweets. We started with a typical article about a claim, and we looked inside the article for links to tweets that are possibly making that claim. Note that some tweets mentioned in the article are not making the corresponding verified claim, and some are not making any claims; we manually checked and filtered out such tweets.
We collected 1,000 suitable tweets as Input claims, and we paired them with the corresponding claim that the page is about as the VerClaim claim. We further extracted from the article its Title, and the TruthValue of the Input claim (a rating of the claims assigned from Snodes $^{13}$ ).
Examples of input-VerClaim pairs are shown in Table 2. Comparing them to the ones from Table 1, we can observe that the Snopes tweets are generally more self-contained and context-independent.
Finally, we created a set of VerClaim claims to match against using the Snopes claims in the ClaimsKG dataset (Tchechmedjiev et al., 2019). Ultimately, our Snopes dataset consists of 1,000 input-VerClaim pairs and 10,396 verified claims.
Statistics about the datasets are shown in Table 3; the datasets are available online.
In section 3, we discussed that matching some of the input claims with the corresponding verified claims can be a non-trivial task, and we gave examples of easy and hard cases. To capture this distinction, we classify Input-VerClaim pairs into two types. Type-1 pairs are such for which the Input claim can be matched to the VerClaim using simple approximate string matching techniques., e.g., as in line 1 of Table 1 and lines 1-2 of Table 2. Conversely, Type-2 pairs are such for which the Input claim cannot be easily mapped to the VerClaim, e.g., as in lines 2-5 of Table 1 and line 3 of Table 2. We manually annotated a sample of 100 pairs from PolitiFact input-VerClaimpairs and we found  $48\%$  of them to be of Type-2.
<sup>12</sup>Note that these reports discuss multiple claims, unlike the typical PolitiFact article about a specific claim.
13http://www.snopes.com/ fact-check-ratings/
PolitiFact	Snopes
Input-VerClaim pairs	768	1,000
- training	614	800
- testing	154	200
Total # of verified claims	16,636	10,396
Table 3: Statistics about the datasets: shown are the number of Input-VerClaim pairs and the total number of VerClaim claims to match an Input claim against. Note that each VerClaim comes with an associated fact-checking analysis document in PolitiFact/Snopes.
PolitiFact	Snopes
Threshold	#	%	#	%
0.75	55	8%	11	1%
0.50	128	17%	75	8%
0.25	201	27%	504	50%
0.00	768	100%	1,000	100%
Table 4: Analysis of the task complexity: number of Input-VerClaim pairs in PolitiFact and Snopes with TF-IDF-weighted cosine similarity above a threshold.
We further analyzed the complexity of matching an Input claim to the VerClaim from the same Input-VerClaim pair using word-level TF-IDF-weighted cosine similarity. Table 4 shows the number of pairs for which this similarity is above a threshold. We can see that, for PolitiFact, only  $27\%$  of the pairs have a similarity score that is above 0.25, while for Snopes, this percentage is at  $50\%$ , which suggests Snopes should be easier than PolitiFact.
5 Evaluation Measures
We treat the task as a ranking problem. Thus, we use ranking evaluation measures, namely mean reciprocal rank (MRR), Mean Average Precision (MAP), and MAP truncated to rank  $k$  (MAP@  $k$ ). We also report HasPositive@  $k$ , i.e., whether there is a true positive among the top-  $k$  results.
Measures such as  $\mathrm{MAP}@\mathbf{k}$  and HasPositive@  $k$  for  $k\in \{1,3,5\}$  would be relevant in a scenario, where a journalist needs to verify claims in real time, in which case the system would return a short list of 3-5 claims that the journalist can quickly skim and make sure they are indeed a true match.
We further report  $\mathrm{MAP} @ k$  and HasPositive@  $k$  for  $k \in \{10, 20\}$  as well as MAP (untruncated), which would be more suitable in a non-real-time scenario, where recall would be more important.
Here, we describe the models we experiment with.
6.1 BM25