SlowGuess/ACL_2020_OCR huggingface dataset:
Christopher Hidey, $^{1*}$  Tuhin Chakrabarty, $^{1}$  Tariq Alhindi, $^{1}$  Siddharth Varia, $^{1}$  Kriste Krstovski, $^{1,3}$  Mona Diab, $^{4,5*}$  and Smaranda Muresan $^{1,2}$
$^{1}$ Department of Computer Science, Columbia University
$^{2}$ Data Science Institute, Columbia University
<sup>3</sup>Columbia Business School, Columbia University <sup>4</sup>Facebook AI
$^{5}$ Department of Computer Science, George Washington University
{chidey, tariq, smara}@cs.columbia.edu, mtdiab@email.gwu.edu
{tuhin.chakrabarty, sv2504, kriste.krstovski}@columbia.edu
The increased focus on misinformation has spurred development of data and systems for detecting the veracity of a claim as well as retrieving authoritative evidence. The Fact Extraction and VERIFICATION (FEVER) dataset provides such a resource for evaluating end-to-end fact-checking, requiring retrieval of evidence from Wikipedia to validate a veracity prediction. We show that current systems for FEVER are vulnerable to three categories of realistic challenges for fact-checking – multiple propositions, temporal reasoning, and ambiguity and lexical variation – and introduce a resource with these types of claims. Then we present a system designed to be resilient to these “attacks” using multiple pointer networks for document selection and jointly modeling a sequence of evidence sentences and veracity relation predictions. We find that in handling these attacks we obtain state-of-the-art results on FEVER, largely due to improved evidence retrieval.
The growing presence of biased, one-sided, and often altered discourse, is posing a challenge to our media platforms from newswire to social media (Vosoughi et al., 2018). To overcome this challenge, fact-checking has emerged as a necessary part of journalism, where experts examine "check-worthy" claims (Hassan et al., 2017) published by others for their "shades" of truth (e.g., FactCheck.org or PolitiFact). However, this process is time-consuming, and thus building computational models for automatic fact-checking has become an active area of research (Graves, 2018). Advances were made possible by new open source datasets and shared tasks: the Fact Extraction and Verification Shared Task (FEVER) 1.0 and 2.0 (Thorne et al., 2018; Thorne
Claim: Murda Beatz's real name is Marshall Mathers. Evidence: [Murda Beatz] Shane Lee Lindstrom (born February 11, 1994), known professionally as Murda Beatz, is a Canadian hip hop record producer and songwriter from Fort Erie, Ontario.
Label: REFUTES
Figure 1: Example from FEVER 1.0 Dataset
and Vlachos, 2019), SemEval 2019 Shared Task 8: Fact-Checking in Community Forums (Mihaylova et al., 2019), and LIAR  $(+)$  datasets with claims from PolitiFact (Wang, 2017; Alhindi et al., 2018).
The FEVER 1.0 shared task dataset (Thorne et al., 2018) has enabled the development of end-to-end fact-checking systems, requiring document retrieval and evidence sentence extraction to corroborate a veracity relation prediction (supports, refutes, not enough info). An example is given in Figure 1. Since the claims in FEVER 1.0 were manually written using information from Wikipedia, the dataset may lack linguistic challenges that occur in verifying naturally occurring check-worthy claims, such as temporal reasoning or lexical generalization/specification. Thorne and Vlachos (2019) designed a second shared task (FEVER 2.0) for participants to create adversarial claims ("attacks") to break state-of-the-art systems and then develop systems to resolve those attacks.
We present a novel dataset of adversarial examples for fact extraction and verification in three challenging categories: 1) multiple propositions (claims that require multi-hop document or sentence retrieval); 2) temporal reasoning (date comparisons, ordering of events); and 3) named entity ambiguity and lexical variation (Section 4). We show that state-of-the-art systems are vulnerable to adversarial attacks from this dataset (Section 6). In addition, we take steps toward addressing these vulnerabilities, presenting a system for end-to-end fact-checking that brings two novel contri
*Work completed in part at Amazon
8593
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8593-8606 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
butions using pointer networks: 1) a document ranking model; and 2) a joint model for evidence sentence selection and veracity relation prediction framed as a sequence labeling task (Section 5). Our new system achieves state-of-the-art results for FEVER and we present an evaluation of our models including ablation studies (Section 6). Data and code will be released to the community.
Approaches for predicting the veracity of naturally-occurring claims have focused on statements fact-checked by journalists or organizations such as PolitiFact.org (Vlachos and Riedel, 2014; Alhindi et al., 2018), news articles (Pomerleau and Rao, 2017), or answers in community forums (Mihaylova et al., 2018, 2019). However, those datasets are not suited for end-to-end fact-checking as they provide sources and evidence while FEVER (Thorne et al., 2018) requires retrieval.
Initial work on FEVER focused on a pipeline approach of retrieving documents, selecting sentences, and then using an entailment module (Malon, 2018; Hanselowski et al., 2018; Tokala et al., 2019); the winning entry for the FEVER 1.0 shared task (Nie et al., 2019a) used three homogeneous neural models. Other work has jointly learned either evidence extraction and question answering (Nishida et al., 2019) or sentence selection and relation prediction (Yin and Roth, 2018; Hidey and Diab, 2018); unlike these approaches, we use the same sequential evidence prediction architecture for both document and sentence selection, jointly predicting a sequence of labels in the latter step. More recently, Zhou et al. (2019) proposed a graph-based framework for multi-hop retrieval, whereas we model evidence sequentially.
Language-based adversarial attacks have often involved transformations of the input such as phrase insertion to distract question answering systems (Jia and Liang, 2017) or to force a model to always make the same prediction (Wallace et al., 2019). Other research has resulted in adversarial methods for paraphrasing with universal replacement rules (Ribeiro et al., 2018) or lexical substitution (Alzantot et al., 2018; Ren et al., 2019). While our strategies include insertion and replacement, we focus specifically on challenges in fact-checking. The task of natural language inference
(Bowman et al., 2015; Williams et al., 2018) provides similar challenges: examples for numerical reasoning and lexical inference have been shown to be difficult (Glockner et al., 2018; Nie et al., 2019b) and improved models on these types are likely to be useful for fact-checking. Finally, (Thorne and Vlachos, 2019) provided a baseline for the FEVER 2.0 shared task with entailment-based perturbations. Other participants generated adversarial claims using implicative phrases such as "not clear" (Kim and Allan, 2019) or GPT-2 (Niewinski et al., 2019). In comparison, we present a diverse set of attacks motivated by realistic, challenging categories and further develop models to address those attacks.
3 Problem Formulation and Datasets
We address the end-to-end fact-checking problem in the context of FEVER (Thorne et al., 2018), a task where a system is required to verify a claim by providing evidence from Wikipedia. To be successful, a system needs to predict both the correct veracity relation- supported (S), refuted (R), or not enough information (NEI)- and the correct set of evidence sentences (not applicable for NEI). The FEVER 1.0 dataset (Thorne et al., 2018) was created by extracting sentences from popular Wikipedia pages and mutating them with paraphrases or other edit operations to create a claim. Then, each claim was labeled and paired with evidence or the empty set for NEI. Overall, there are 185,445 claims, of which 90,367 are s, 40,107 are R, and 45,971 are NEI. Thorne and Vlachos (2019) introduced an adversarial set up for the FEVER 2.0 shared task - participants submitted claims to break existing systems and a system designed to withstand such attacks. The organizers provided a baseline of 1000 adversarial examples with negation and entailment-preserving/-altering transformations and this set was combined with examples from participants to form the FEVER 2.0 dataset. Table 1 shows the partition of FEVER 1.0 and 2.0 data (hereafter FV1/FV2-train/dev/test).
Dataset	Train	Dev.	Blind Test
FEVER 1.0	145,449	19,998	19,998
FEVER 2.0	-	1,174	1,180
Table 1: FEVER Dataset Statistics
4 Adversarial Dataset for Fact-checking
While the FEVER dataset is a valuable resource, our goal is to evaluate complex adversarial claims
https://github.com/chridey/ fever2-columbia
which resemble check-worthy claims found in news articles, speeches, debates, and online discussions. We thus propose three types of attacks based on analysis of FV1 or prior literature: those using multiple propositions, requiring temporal and numerical reasoning, and involving lexical variation.
For the multi-propositional type, Graves (2018) notes that professional fact-checking organizations need to synthesise evidence from multiple sources; automated systems struggle with claims such as "Lesotho is the smallest country in Africa." In FV1-dev,  $83.18\%$  of S and R claims require only a single piece of evidence and  $89\%$  require only a single Wikipedia page. Furthermore, our previous work on FEVER 1.0 found that our model can fully retrieve  $86\%$  of evidence sentences from Wikipedia when only a single sentence is required, but the number drops to  $17\%$  when 2 sentences are required and  $3\%$  when 3 or more sentences are required (Hidey and Diab, 2018).
For the second type, check-worthy claims are often numerical (Francis, 2016) and temporal reasoning is especially challenging (Mirza and Tonelli, 2016). Rashkin et al. (2017) and Jiang and Wilson (2018) showed that numbers and comparatives are indicative of truthful statements in news, but the presence of a date alone does not indicate its veracity. In FV1-dev, only  $17.81\%$  of the claims contain dates and  $0.22\%$  contain time information. To understand how current systems perform on these types of claims, we evaluated three state-of-the-art systems from FEVER 1.0 (Hanselowski et al., 2018; Yoneda et al., 2018; Nie et al., 2019a), and examined the predictions where the systems disagreed. We found that in characterizing these predictions according to the named entities present in the claims, the most frequent types were numerical and temporal (such as percent, money, quantity, and date).
Finally, adversarial attacks for lexical variation, where words may be inserted or replaced or changed with some other edit operation, have been shown to be effective for similar tasks such as natural language inference (Nie et al., 2019b) and question answering (Jia and Liang, 2017), so we include these types of attacks as well. For the fact-checking task, models must match words and entities across claim and evidence to make a veracity prediction. As claims often contain ambiguous entities (Thorne and Vlachos, 2018) or lexical features indicative
of credibility (Nakashole and Mitchell, 2014), we desire models resilient to minor changes in entities (Hanselowski et al., 2018) and words (Alzantot et al., 2018).
We thus create an adversarial dataset of 1000 examples, with 417 multi-propositional, 313 temporal and 270 lexically variational. Representative examples are provided in Appendix A.
Multiple Propositions Check-worthy claims often consist of multiple propositions (Graves, 2018). In the FEVER task, checking these claims may require retrieving evidence sequentially after resolving entities and events, understanding discourse connectives, and evaluating each proposition.
Consider the claim "Janet Leigh was from New York and was an author." The Wikipedia page [Janet Leigh] contains evidence that she was an author, but makes no mention of New York. We generate new claims of the CONJUNCTION type automatically by mining claims from FV1-dev and extracting entities from the subject position. We then combine two claims by replacing the subject in one sentence with a discourse connective such as "and." The new label is S if both original claims are S, R if at least one claim is R, and NEI otherwise.
While CONJUNCTION claims provide a way to evaluate multiple propositions about a single entity, these claims only require evidence from a single page; hence we create new examples requiring reasoning over multiple pages. To create MULTI-HOP examples, we select claims from FV1-dev whose evidence obtained from a single page  $P$  contains at least one other entity having a valid page  $Q$ . We then modify the claim by appending information about the entity which can be verified from  $Q$ . For example, given the claim "The Nice Guys is a 2016 action comedy film." we make a multi-hop claim by obtaining the page [Shane Black] (the director) and appending the phrase "directed by a Danish screenwriter known for the film Lethal Weapon."
While multi-hop retrieval provides a way to evaluate the S and R cases, composition of multiple propositions may also be necessary for NEI, as the relation of the claim and evidence may be changed by more general/specific phrases. We thus add ADDITIONAL UNVERIFIABLE PROPOSITIONS that change the gold label to NEI. We selected claims from FV1-dev and added propositions which have no evidence in Wikipedia (e.g. for the claim "Duff McKagan is an American citizen," we can add the reduced relative clause "born in Seattle").
2As determined by NER using Spacy: https://spacy.io
8595
Temporal Reasoning Many check-worthy claims contain dates or time periods and to verify them requires models that handle temporal reasoning (Thorne and Vlachos, 2017).
In order to evaluate the ability of current systems to handle temporal reasoning we modify claims from FV1-dev. More specifically, using claims with the phrase "in  $<$  date>" we automatically generate seven modified claims using simple DATE MANIPULATION heuristics: arithmetic (e.g., "in 2001"  $\rightarrow$  "4 years before 2005"), range ("in 2001"  $\rightarrow$  "before 2008"), and verbalization ("in 2001"  $\rightarrow$  "in the first decade of the 21st century").
We also create examples requiring MULTI-HOP TEMPORAL REASONING, where the system must evaluate an event in relation to another. Consider the S claim "The first governor of the Indiana Territory lived long enough to see it become a state." A system must resolve entity references (Indiana Territory and its first governor, William Henry Harrison) and compare dates of events (the admittance of Indiana in 1816 and death of Harrison in 1841). While multi-hop retrieval may resolve references, the model must understand the meaning of "lived long enough to see" and evaluate the comparative statement. To create claims of this type, we mine Wikipedia by selecting a page  $X$  and extracting sentences with the pattern "is/was/named the  $A$  of  $Y$ " (e.g.  $A$  is "first governor") where  $Y$  links to another page. Then we manually create temporal claims by examining dates on  $X$  and  $Y$  and describing the relation between the entities and events.
Named Entity Ambiguity and Lexical Variation As fact-checking systems are sensitive to lexical choice (Nakashole and Mitchell, 2014; Rashkin et al., 2017), we consider how variations in entities and words may affect veracity relation prediction.
ENTITY DISAMBIGUATION has been shown to be important for retrieving the correct page for an entity among multiple candidates (Hanselowski et al., 2018). To create examples that contain ambiguous entities we selected claims from FV1-dev where at least one Wikipedia disambiguation page was returned by the Wikipedia python API. We then created a new claim using one of the documents returned from the disambiguation list. For example the claim "Patrick Stewart is someone who does acting for a living." returns a disambiguation page, which in turn gives a list of pages
such as [Patrick Stewart] and [Patrick Maxwell Stewart].
Finally, as previous work has shown that neural models are vulnerable to LEXICAL SUBSTITUTION (Alzantot et al., 2018), we apply their genetic algorithm approach to replace words via counter-fitted embeddings. We make a claim adversarial to a model fine-tuned on claims and gold evidence by replacing synonyms, hybernyms, or hyponyms, e.g. created  $\rightarrow$  established, leader  $\rightarrow$  chief. We manually remove ungrammatical claims or incorrect relations.
Verifying check-worthy claims such as those in Section 4 requires a system to 1) make sequential decisions to handle multiple propositions, 2) support temporal reasoning, and 3) handle ambiguity and complex lexical relations. To address the first requirement we make use of a pointer network (Vinyals et al., 2015) in two novel ways: i) to rerank candidate documents and ii) to jointly predict a sequence of evidence sentences and veracity relations in order to compose evidence (Figure 3). To address the second we add a post-processing step for simple temporal reasoning. To address the third we use rich, contextualized representations. Specifically, we fine-tune BERT (Devlin et al., 2019) as this model has shown excellent performance on related tasks and was pre-trained on Wikipedia.
Figure 2: Our FEVER pipeline: 1) Retrieving Wikipedia pages by selecting an initial candidate set (1a) and ranking the top  $D$  (1b); 2) Identifying the top  $N$  sentences; 3) Predicting supports, refutes, or not enough info. Dashed arrows indicate fine-tuning steps.
Our full pipeline is presented in Figure 2. We first identify an initial candidate set of documents
<sup>3</sup>https://pypi.org/project/wikipedia/
8596
Figure 3: Pointer network architecture. Claim and evidence (page title or sentence) are embedded with BERT and evidence is sequentially predicted (for sentence selection the relation sequence is jointly predicted).
(1a) by combining the top  $M$  pages from a TF-IDF search using DrQA (Chen et al., 2017) with pages from the approach of Chakrabarty et al. (2018), which provides results from Google search and predicted named entities and noun phrases. Then, we perform document ranking by selecting the top  $D < M$  pages with a pointer network (1b). Next, an  $N$ -long sequence of evidence sentences (2) and veracity relation labels (3) are predicted jointly by another pointer network.
Prior to training, we fine-tune BERT for document and sentence ranking on claim/title and claim/sentence pairs, respectively. Each claim and evidence pair in the FEVER 1.0 dataset has both the title of the Wikipedia article and at least one sentence associated with the evidence, so we can train on each of these pairs directly. For the claim "Michelle Obama's husband was born in Kenya", shown in Figure 3, we obtain representations by pairing this claim with evidence sentences such as "Obama was born in Hawaii" and article titles such as [Barack Obama].
The core component of our approach is the pointer network, as seen in Figure 3. Unlike our previous work (Hidey and Diab, 2018), we use the pointer network to re-rank candidate documents and jointly predict a sequence of evidence sentences and relations. Given a candidate set of evidence (as either document titles or sentences) and a respective fine-tuned BERT model, we extract
features for every claim  $c$  and evidence  $e_p$  pair by summing the [CLS] embedding for the top 4 layers (as recommended by Devlin et al. (2019)):
$$
m _ {p} = B E R T (c, e _ {p}) \tag {1}
$$
Next, to select the top  $k$  evidence, we use a pointer network over the evidence for claim  $c$  to extract evidence recurrently by computing the extraction probability  $P(p_{t}|p_{0}\dots p_{t - 1})$  for evidence  $e_p$  at time  $t < k$ . At time  $t$ , we update the hidden state  $z_{t}$  of the pointer network decoder. Then we compute the weighted average  $h_t^q$  of the entire evidence set using  $q$  hops over the evidence (Vinyals et al., 2016; Sukhbaatar et al., 2015):4
$$
\begin{array}{l} \alpha_ {t} ^ {o} = \mathrm {s o f t m a x} (v _ {h} ^ {T} \tanh (W _ {g} m _ {p} + W _ {a} h _ {t} ^ {o - 1})) \\ h _ {t} ^ {o} = \sum_ {j} \alpha_ {t} ^ {o} W _ {g} m _ {j} \tag {2} \\ \end{array}
$$
We concatenate  $m_p$  and  $h_t^q$  and use a multi-layer perceptron (MLP) to predict  $p_t$ . The loss is then:
$$
\mathcal {L} \left(\theta_ {p t r}\right) = - 1 / k \sum_ {t = 0} ^ {k - 1} \log P _ {\theta_ {p t r}} \left(p _ {t} \mid p _ {0: t - 1}\right) \tag {3}
$$
We train on gold evidence and perform inference with beam search for both document ranking (Section 5.1) and joint sentence selection and relation prediction (Section 5.2).
5.1 Document Ranking
In order to obtain representations as input to the pointer network for document ranking, we leverage the fact that Wikipedia articles all have a title (e.g. [Barack Obama]), and fine-tune BERT on title and claim pairs, in lieu of examining the entire document text (which due to its length is not suitable for BERT). Because the title often overlaps lexically with the claim (e.g. [Michelle Obama]), we can train the model to locate the title in the claim. Furthermore, the words in the title co-occur with words in the article (e.g. Barack and Michelle), which the pre-trained BERT language model may be attuned to. We thus fine-tune a classifier on a dataset created from title and claim pairs (where positive examples are titles of gold evidence pages and negative are randomly sampled from our candidate set), obtaining  $90.0\%$  accuracy. Given the fine-tuned model, we extract features using Equation 1 where  $e_p$  is a title, and use Equation 3 to learn to predict a sequence of titles as in Figure 3.
4 Initially,  $h^{t,0}$  is set to  $z_{t}, v_{h}, W_{g}$ , and  $W_{a}$  are learned.
8597
5.2 Joint Sentence Selection and Relation Prediction
The sentence selection and relation prediction tasks are closely linked, as predicting the correct evidence is necessary for predicting S or R and the representation should reflect the interaction between a claim and an evidence set. Conversely, if a claim and an evidence set are unrelated, the model should predict NEI. We thus jointly model this interaction by sharing the parameters of the pointer network - the hidden state of the decoder is used for both tasks and the models differ only by a final MLP.
Sentence Selection Similar to our document selection fine-tuning approach, we fine-tune a classifier on claim and evidence sentence pairs to obtain BERT embeddings. However, instead of training a binary classifier for the presence of valid evidence we train directly on veracity relation prediction, which is better suited for the end task. We create a dataset by pairing each claim with its set of gold evidence sentences. As gold evidence is not available for NEI relations, we sample sentences from our candidate documents to maintain a balanced dataset. We then fine-tune a BERT classifier on relation prediction, obtaining  $93\%$  accuracy. Given the fine-tuned model, we extract features using Equation 1 where  $e_p$  is a sentence, and use Equation 3 to learn to predict a sequence of sentences.