SlowGuess/ACL_2020_OCR huggingface dataset:
Zachary G. Ives
{yizhang5, zives, danroth}@cis.upenn.edu
In an era where generating content and publishing it is so easy, we are bombarded with information and are exposed to all kinds of claims, some of which do not always rank high on the truth scale. This paper suggests that the key to a longer-term, holistic, and systematic approach to navigating this information pollution is capturing the provenance of claims. To do that, we develop a formal definition of provenance graph for a given natural language claim, aiming to understand where the claim may come from and how it has evolved. To construct the graph, we model provenance inference, formulated mainly as an information extraction task and addressed via a textual entailment model. We evaluate our approach using two benchmark datasets, showing initial success in capturing the notion of provenance and its effectiveness on the application of claim verification.
Never before have humans been able to generate and disseminate content so easily, leading to a contamination of information supply with irrelevant, redundant, unsolicited, and often low-value information (Orman, 1984). While significant attention has been devoted recently to identifying false claims, the age of "information pollution" we live in calls for the development of additional important insights. At the heart of these insights is the need to determine the provenance of claims — who first made a given claim, and how an original claim developed and changed over time (and potentially across contributors).
Consider the following claim: "Facebook soon plans to charge fees to users of the social network." As shown in Figure 1, a typical modern
Figure 1: A typical claim verification pipeline of our running example. It starts with searching for evidence, and feed it to a textual entailment model  $f$  to decide the claim's veracity.
claim verification pipeline starts with searching for existing evidence for the given query claim, and then leverages textual entailment models to determine the veracity of the claim relative to the evidence (Thorne et al., 2018). However, sites such as snopes.com and other fact-checking websites will not only provide their conclusion about the veracity of the claim relative to the evidence, but would also seek additional information that explains why people may think it fake. For example, Snopes details how the claim originated from nationalreport.net. The original version of the claim is related to the query claim, as well as other relevant claims, but carries a different meaning. It says: "Facebook could cease to exist, if they don't do something about their rising costs". Subsequently, the inaccurate claim, triggered by the original one, has been repeated by other websites and retweeted on social media, as shown in Figure 2, possibly increasing the level of credibility some readers assign to it.
The origins and causal derivations of data, as described above, are explicitly modeled in the context of databases (Cheney et al., 2009) and scientific workflow systems (Davidson and Freire, 2008), where they are termed "data provenance." We argue
'https://www.snopes.com/fact-check/ facebook-implementing-user-fees/
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4416-4426 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
that modeling and understanding the provenance of a claim made in natural language is also very important since, beyond attribution, it helps people understand the background and the context in which a claim was generated, how different aspects of the claim are combined, and how a claim has been changed over time by different agents. At the same time, provenance provides us with an explanation for why people think a claim is real or fake, by looking at its history. Even if all one wants is to determine a stance relative to a claim, this may involve considering more than just its current incarnation, but rather its evolution over time and all of the sources that contributed to this evolution. Similarly, one may want to consider who influenced a claim, or who influences a specific author of multiple claims, and this can be accomplished by considering the origin and evolution of these claims. Figure 2 shows that our notion of provenance can not only provide us with evidence but also with the structure of- and relationships among supporting evidence and claims.
Figure 2: A provenance subgraph of our running example. The nodes represent information sources, each with a statement that influenced the target claim. The edges are in the direction of influence, and labels indicate the relations between the corresponding statements.
In this paper, we propose and develop a computational framework for claim provenance graphs, which provide information and supporting evidence about where a claim is believed to have originated and how it has been disseminated. Our challenge is to infer and reconstruct this graph using available evidence. A claim provenance graph consists of two components:
1. As nodes: the sources that may have made the query claim and earlier versions of it, or those influencing the eventual query claim;
2. As labeled edges: the relationships between the claims made by sources.
Like provenance graphs in other fields (including the W3C PROV specification (Belhajjame et al., 2013)), a claim provenance graph tracks the data, operations, and parties responsible for a claim. Unlike most prior provenance graphs, claim provenance is often inferred, uncertain, and comprised of approximate relationships (e.g., "textually entailed"), as indicated in Figure 2.
However, inferring the provenance graph of a claim is a difficult task. In our current implementation of this notion, given a natural language claim in a document, we search for the claim on the web, restricting our focus to content published prior to the document (eliminating many sources that could not have influenced the document). A match to the claim search may itself make a statement about the claim, or it may in turn report a statement relevant to the claim made by other sources. If a source mentioned in the article is describing the claim, one of the sub-tasks is to identify the correct source(s). Therefore, we view obtaining the nodes of the provenance graph as an information extraction (IE) problem. However, in contrast to a typical IE approach that uses annotated data (Hendrickx et al., 2009), Wikipedia or other large scale knowledge bases (Auer et al., 2007), identifying sources of a statement in an article is an IE task which is very hard to annotate. The reason is that both the statement and its sources can be described implicitly in the given text, and this may require additional reasoning or coreference resolution. In this work, we tackle this IE problem as a textual entailment (TE) problem, and propose a solution that leverages off-the-shelf semantic role labeling tools to generate candidates for source identification. Following that, we wikify extracted source mentions, which further allows us to link nodes in the provenance graph and label them. As an application, we propose models that can use the provenance graph to improve the estimation of claims' veracity.
The key contributions of this paper are (1) it is the first work to study and formally define the notion of a provenance graph for a natural language claim; (2) it proposes a TE model to automatically extract provenance information, regardless of whether the relevant statement and the source are described explicitly or implicitly in the text; this is then used to construct a graph and label its edges; (3) it develops techniques that exploit the provenance graph to improve claim verifica
tion. We provide initial experimental support for our novel formulation by studying the effectiveness of extracting sources and the benefit of leveraging provenance graph when doing claim verification.
It is important to note that we have not solved the claim provenance problem. We introduce it and explain its importance, provide an initial formulation and an implementation. We argue that, already at this point, our initial formulation and the results it supports provide a significant contribution. We point to a range of future work directions that we discuss at the end of this paper.
2 Provenance of Claims
Given a target claim and a large corpus, we want to infer the provenance graph of the claim from the given corpus. This graph will represent previously-made statements with their sources, which, with high probability, ultimately led to the target claim. In this section, we first define the claim provenance graph, and present the problems one must solve to infer it. Note that to distinguish between the query claim and the claims in its previous versions, we use statements to refer those previously-made claims by other sources.
Let  $S_{D}(q)$  be the set of sources making statements about claim  $q$  in corpus  $D$ , and  $t_{s}(q)$ , an individual statement made by  $s \in S_{D}(q)$ .
Definition 1 (Claim Provenance Graph) Let
$G_{D}(q) = (V,E,L)$  denote the provenance graph of  $q$  given  $D$ . Here  $G_{D}(q)$  is a labeled directed acyclic graph;  $V = \{\langle s,t_s(q)\rangle \cup q|s\in S_D(q)\}$  is a set of nodes.  $\forall \langle s,t_s(q)\rangle \in V$ ,  $s$  is the source making statement  $t_s(q)$  that is related to the derivation of  $q$ .  $E$  represents a set of labeled directed edges, and denote  $v_{i} = \langle s_{i},t_{s_{i}}(q)\rangle$ ,  $v_{j} = \langle s_{j},t_{s_{j}}(q)\rangle$ , such that  $\forall (v_i,v_j,l)\in E$ ,  $v_{i},v_{j}\in V$ , the presence of an edge  $(v_{i},v_{j})$  indicates that  $t_{s_i}(q)$  influences the creation of  $t_{s_j}(q)$  via relation  $l\in L$ . Note that  $q$  is the sink node of  $G_{D}(q)$ , whose outdegree equals to 0.
Edge Label Set We use  $L$  to categorize how a current statement may be derived by a previous one. Typically, it includes (1) identical, when a source quotes a statement from another source; (2) paraphrased, when a source describes the same statement with different words; (3) textually entailed, when the previous statement can support
the current one; (4) motivated, when the previous statement potentially influences the appearance of the current one. Practically, we further consider there are two sub-types of 'motivated'. One is triggered, in our running example, the appearance of the claim is very likely due to other related claims, such as "Facebook should charge users.", the other one is contradicted, when the derived statement has an opposite opinion.
Therefore, the problem we are to solve is given the query claim  $q$  and the corpus  $D$ , we want to automatically construct its provenance graph  $G_{D}(q)$ .
2.2 Problem Overview
To construct the provenance graph, it is obvious that we need to (1) obtain the sources that describe the statements about the claim, i.e.,  $S_{D}(q)$ ; (2) infer the relationship between the sources and the statements, i.e., determine the labeled edges of the provenance graph. To accomplish those two goals, we divide our problem into three subproblems.
Problem 1: Claim Search Detecting the sources requires locating the statements about the claim in the corpus. Therefore, searching for related (and contradictory) sentences to the given claim is a critical aspect. However, it is difficult to locate all statements accurately, since a claim can be spread in many different ways. Moreover, we do not know, when one source proposes a statement, if the statement was a hypothesis supported by the claim, was another claim associated, or it was just simply consistent with the claim. In our running example, the claim of interest can be paraphrased as "Using Facebook will cost money", or can be described as "Facebook would be implementing a tiered membership system.", which entails the claim.
Problem 2: Source Extraction Claim Search returns a list of articles with sentences related to the given claim, and the next step is to identify who authored those sentences. We assume there are two cases. One is that the writer of the article makes a statement about the claim; the other is that some other source mentioned in the article describes the claim<sup>2</sup>. For example, one of the articles returned by the "New York Post" has a paragraph:
"...First, Facebook should charge users a nominal $5-a-month fee. You can give seniors a discount
2We leave for future work a richer model that might also allow for a source to make a claim after being indirectly influenced by another uncited source.
so you do not lose them." In this example, it is clear to the reader that the author of the paragraph is making a statement about the given claim.
Consider another example: "... In September 2014 the fake news site National Report published a fictitious article positing that Facebook would begin charging users \(2.99 per month starting 1 November 2014..." In this paragraph, the writer is making a statement about how the "National Report" asserts the given claim.
In this work, we consider source extraction as an information extraction task. Given a statement  $c$  and the context around  $c$ , denoted as  $T(c)$ , from the article returned by claim search — we are to determine if there exist sources mentioned in the context which are describing the statement, and if so, to identify the correct sources.
Problem 3: Provenance Graph Construction
Source Extraction provides us with a multitude of sources mentioned in the articles that are describing the claims. In the previous examples, the sources are "National Report" and "nypost.com" respectively. However, source extraction only provides a two-layer directed graph, i.e., the writer/url of the article is directed from the sources mentioned in the text. To further complete the provenance graph, we then need to identify the same sources from the sources extracted. For example, the same statement made by "New York Post" and "NY Post" obtained from the text should link to the same statement made by "nypost.com". After connecting the subgraphs, we then need to determine the relationship between the statements about the claim on the edge, which we view as a classification problem.
3 Inferring the Provenance Graph
To infer the provenance graph for the given claim, we need to solve the three problems outlined in Section 2. Here, we propose a pipelined solution, and elaborate them one by one.
3.1 Searching for the Context
As we described in Section 2, accurately locating the previous statements about the claim is a very challenging problem. Therefore, instead of directly searching for a possible previous statement, we search for related context, where the source are describing a statement related to the claim.
Specifically, we rank sentences in the given corpus, by computing the cosine similarity to the given
claim with their ELMo (Peters et al., 2018) representations. Then, we choose sentences that are most similar and fetch their context in a window size  $w$ , which means we consider  $w$  sentences before and after the returned sentence together as the context, from which we will extract the sources. Note that a returned sentence is denoted as  $c$ , and its context is denoted as  $T(c)$ .
3.2 Extraction as Textual Entailment
Given a sentence  $c$  within its context  $T(c)$  returned by claim search for  $q$ , we need to identify the sources in  $T(c)$  that are talking about a statement related to  $q$ . This is actually an IE task. Typically, IE is a sequential tagging problem: it needs to learn linguistic patterns from annotated data using syntactic and semantic features, which can express the targeted semantic relations. Most of the solutions in the literature (Surdeanu et al., 2012; Schmitz et al., 2012; Chan and Roth, 2011; Li and Ji, 2014) focus on extracting relationships between two named entities or two nominals. However, in our problem, the relationship of interest is between a nominal/an entity and a statement. The statement can be written either explicitly or implicitly in the given context, and what we only know is that the statement is about  $q$ . Therefore, annotation is hard, and existing IE solutions can not be used in this case. Furthermore, the source and the statement may appear across sentences rather than within a single sentence, therefore, coreference resolution may be necessary. For example, "The website Hoax Slayer said the message dates back to 2012 and has recently resurfaced ... it also noted Facebook has no plans to start charging users for normal access..." requires a cross-sentence relation extraction (Peng et al., 2017).
Rather than tackling the problem as a sequential tagging problem, we model it as a textual entailment (TE) problem (Dagan et al., 2013). Similar to QA-SRL (He et al., 2015), TE-IE task formulation has the advantages of (1) easier annotation (2) being able to capture implicit statements and implicit sources which requires coreference resolution.
TE Modeling We use the dataset (Choi et al., 2005) that contains a set of annotated articles. For each article, it annotates "who" has an opinion on "what". Formally, given a corpus  $D$ , for each article  $d \in D$ , our training data comes in the form of pairs  $\{(q_i^d, S_i^d)\}_{i=1}^N$ , where we view  $q_i^d$  as a claim, and  $\forall s \in S_i^d$  is the source of  $q_i^d$  mentioned in  $d$ .
Figure 3: Our TE extraction model: we transform the source extraction problem to be a textual entailment task. In our fine-tuned model based on BERT, our objective function considers (1) binary prediction correctness for sources by cross-entropy loss; (2) difference between positive and negative examples by margin ranking loss.
We search for related sentences and their context for each  $q_i^d$ , and denote the returned set of context as  $\{T(c_i^d)\}$ . Therefore, given  $q_i^d$ , a related sentence  $c_i^d$  with its  $T(c_i^d)$ , our problem is to identify  $s$  from  $T(c_i^d)$ , if  $s \in S_i^d$ .
As we have described, it is hard to directly use existing sequential tagging techniques to solve this problem. Instead, we model it as a TE task. Assume we are given a candidate list of sources, which is a list of spans in text  $T(c_i^d)$ , denoted as  $sc(c_i^d)$  (we will describe how to generate the candidate list later). Then, if we view the context  $T(c_i^d)$  as a premise, and generate a sentence following the pattern that the source  $s$  "claims"/"says" the claim  $q_i^d$ , where  $s \in sc(c_i^d)$  as the hypothesis, we transform the tagging problem to a TE problem. If the premise denoted as  $a_i^d$  can entail the hypothesis denoted as  $b_i^d[s]$ , it means that  $s \in S_i^d$ , otherwise  $s \notin S_i^d$  which means  $s$  does not say anything about  $c_i^d$ . For each candidate  $s \in sc(c_i^d)$ , we have a binary classification problem: learn a function  $F$  that can decide if  $a_i^d$  can entail  $b_i^d[s]$ .
However, given the query claim  $q_{i}^{d}$ , a related sentence  $c_{i}^{d}$ , with its context  $T(c_{i}^{d})$  and the candidate list  $sc(c_{i}^{d})$ , the binary decisions mentioned above are not made independently over the candidate sources. Besides fitting a label that is either entailment or not, the representation of the correct claims should be different from incorrect ones, so that we can have a better chance to learn the discriminative features. We reflect this idea by including a margin ranking loss within our model.
Specifically, we design our model on top of a pre-trained language model for general purpose (BERT) (Devlin et al., 2018), so that we can have a representation of sentences that can capture both semantic and syntactic information. We concatenate
$a_{i}^{d}, b_{i}^{d}[s]$  with separation tokens of BERT to the pretrained model as shown in Figure 3, and represent the output as  $E_{i}^{d}[s]$ . Then, we add another hidden layer, and feed its result through a final classifier  $F$  to do binary prediction, where  $F$  is a feed forward network followed by a linear layer
$$
\hat {y} = F (h (s)) \tag {1}
$$
where  $h(s) = \tanh (W_1E_i^d [s] + b_1)$ , and  $\hat{y} \in \mathbb{R}^{C}$  represents the predicted scores for each class, and consequently the predicted class is given by  $\hat{y} = \arg\max_{i}\hat{y}_{i}$ . Here  $C = \{0,1\}$  and  $W_{1},b_{1}$  are learned parameters.
Then, we use cross-entropy loss as a part of our optimization goals.
$$
L _ {c r o s s} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {c} y _ {c} ^ {i} \log \frac {\exp \left(\hat {y _ {c}} ^ {i}\right)}{\sum_ {c ^ {\prime}} \exp \left(\hat {y _ {c ^ {\prime}}} ^ {i}\right)} \tag {2}
$$
where  $y_c^i$  is an indicator that if  $y^i$ 's label is  $c$ .
At the same time, if  $s_j$  is a positive example, which means  $s_j \in S_i^d$ , we randomly sample for  $s_j$  a negative example denoted as  $s_j^- \in sc(c_i^d)$  and  $s_j^- \notin S_i^d$ . In this case, we are to maximize the difference between  $h(s_j)$  and  $h(s_j^-)$ , and we reflect it by adding a margin ranking loss as follows:
$$
L _ {p a i r} ^ {+} = \frac {1}{N ^ {+}} \sum_ {j = 1} ^ {N ^ {+}} \max  \left(0, 1 - \left(h \left(s _ {j}\right) - h \left(s _ {j} ^ {-}\right)\right)\right) \tag {3}
$$
Similarly, we can also sample a positive example  $s_j^+$  for a negative source  $s_j$  to get:
$$
L _ {p a i r} ^ {-} = \frac {1}{N ^ {-}} \sum_ {j = 1} ^ {N ^ {-}} \max  \left(0, 1 + \left(h \left(s _ {j}\right) - h \left(s _ {j} ^ {+}\right)\right)\right) \tag {4}
$$
where  $N^{+}, N^{-}$  are the numbers of positive and negative examples in the annotated data.
For training, we use a loss function  $L$  combining both cross-entropy loss for binary prediction and the margin ranking loss to maximize the difference between positive and negative examples to finetune the language model. That is:
$$
L = \lambda L _ {c r o s s} + (1 - \lambda) L _ {p a i r} \tag {5}
$$
where  $L_{pair} = L_{pair}^{+} + L_{pair}^{-}$ , and  $\lambda$  is the parameter to trade off different objectives.
Candidate Generation. The next question is how to generate source candidate list  $sc(c_i^d)$  for  $c_i^d$  given  $T(c_i^d)$ . Here, we leverage an off-the-shelf semantic role labeling (SRL) tool (He et al., 2018) that can parse the sentences  $T(c_i^d)$  to tell us "who did what to whom" in the appropriate sentences. We then take all "who", i.e., the span of the text with tag ARG0 detected as a candidate source of  $c_i^d$ . Even though only the "who" followed by a verb such as "say" or "claim" can be the source theoretically, we included all of them as candidates, and leave the identification made by our TE model.
Note that here we only use SRL to generate candidate sources. Considering (1) the noisy relationship produced by SRL parser, (2) the cross-sentence relationship between the source and the claim, and (3) the fact that a claim can be paraphrased with multiple sentences, we do not determine the sources based on the matching between the claim and the span of text with tag ARG1. We will also show the comparison in our evaluation.
Augmenting Training Data Besides the sources and claims provided by the annotated data, there are still many sentences in the document with a pattern that "who" says or claims "what", which is useful to train the model. To get those examples, we use the off-the-shelf SRL tool to parse all of the sentences in the document, and then compute the similarity between the verb in the parsed sentence and the verb attached with the sources annotated in the text. If the average similarity is higher than a threshold, we include the ARG0 and ARG1 in the parsed sentence as a positive example of the source and the claim. In terms of creating the corresponding negative examples, we randomly replace either ARG0 or ARG1 with other sources or claims. Then we use those created examples to incrementally fine-tune our TE extraction model, which can lead to a better performance.
3.3 Constructing the Graph
After extracting the provenance information, the last process is to construct the provenance graph.