SlowGuess/ACL_2020_OCR huggingface dataset:
Wanjun Zhong $^{1*}$ , Jingjing Xu $^{3*}$ , Duyu Tang $^{2}$ , Zenan Xu $^{1}$ , Nan Duan $^{2}$ , Ming Zhou $^{2}$ , Jiahai Wang $^{1}$  and Jian Yin $^{1}$
<sup>1</sup> The School of Data and Computer Science, Sun Yat-sen University. Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R.China  
<sup>2</sup> Microsoft Research <sup>3</sup> MOE Key Lab of Computational Linguistics, Peking University {zhongwj25@mail2,xuzn@mail2}.sysu.edu.cn {wangjiah@mail,issjyin@mail}.sysu.edu.cn {dutang,nanduan,mingzhou}@microsoft.com; jingjingxu@pku.edu.cn
Fact checking is a challenging task because verifying the truthfulness of a claim requires reasoning about multiple retrievable evidence. In this work, we present a method suitable for reasoning about the semantic-level structure of evidence. Unlike most previous works, which typically represent evidence sentences with either string concatenation or fusing the features of isolated evidence sentences, our approach operates on rich semantic structures of evidence obtained by semantic role labeling. We propose two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet. Specifically, using XLNet as the backbone, we first utilize the graph structure to re-define the relative distances of words, with the intuition that semantically related words should have short distances. Then, we adopt graph convolutional network and graph attention network to propagate and aggregate information from neighboring nodes on the graph. We evaluate our system on FEVER, a benchmark dataset for fact checking, and find that rich structural information is helpful and both our graph-based mechanisms improve the accuracy. Our model is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score.
Internet provides an efficient way for individuals and organizations to quickly spread information to massive audiences. However, malicious people spread false news, which may have significant influence on public opinions, stock prices, even presidential elections (Faris et al., 2017). Vosoughi et al. (2018) show that false news reaches more people
Figure 1: A motivating example for fact checking and the FEVER task. Verifying the claim requires understanding the semantic structure of multiple evidence sentences and the reasoning process over the structure.
than the truth. The situation is more urgent as advanced pre-trained language models (Radford et al., 2019) can produce remarkably coherent and fluent texts, which lowers the barrier for the abuse of creating deceptive content. In this paper, we study fact checking with the goal of automatically assessing the truthfulness of a textual claim by looking for textual evidence.
Previous works are dominated by natural language inference models (Dagan et al., 2013; Angeli and Manning, 2014) because the task requires reasoning of the claim and retrieved evidence sentences. They typically either concatenate evidence sentences into a single string, which is used in top systems in the FEVER challenge (Thorne et al., 2018b), or use feature fusion to aggregate the features of isolated evidence sentences (Zhou et al., 2019). However, both methods fail to capture rich semantic-level structures among multiple evidence, which also prevents the use of deeper reasoning model for fact checking. In Figure 1, we give a motivating example. Making the correct prediction requires a model to reason based on the understanding that "Rodney King riots" is occurred in "Los Angeles County" from the first evidence, and that "Los Angeles County" is "the most populous county
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6170-6180 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
in the USA" from the second evidence. It is therefore desirable to mine the semantic structure of evidence and leverage it to verify the truthfulness of the claim.
Under the aforementioned consideration, we present a graph-based reasoning approach for fact checking. With a given claim, we represent the retrieved evidence sentences as a graph, and then use the graph structure to guide the reasoning process. Specifically, we apply semantic role labeling (SRL) to parse each evidence sentence, and establish links between arguments to construct the graph. When developing the reasoning approach, we intend to simultaneously leverage rich semantic structures of evidence embodied in the graph and powerful contextual semantics learnt in pre-trained models like BERT (Devlin et al., 2018), GPT (Radford et al., 2019) and XLNet (Yang et al., 2019). To achieve this, we first re-define the distance between words based on the graph structure when producing contextual representations of words. Furthermore, we adopt graph convolutional network and graph attention network to propagate and aggregate information over the graph structure. In this way, the reasoning process employs semantic representations at both word/sub-word level and graph level.
We conduct experiments on FEVER (Thorne et al., 2018a), which is one of the most influential benchmark datasets for fact checking. FEVER consists of 185,445 verified claims, and evidence sentences for each claim are natural language sentences from Wikipedia. We follow the official evaluation protocol of FEVER, and demonstrate that our approach achieves state-of-the-art performance in terms of both claim classification accuracy and FEVER score. Ablation study shows that the integration of graph-driven representation learning mechanisms improves the performance. We briefly summarize our contributions as follows.
- We propose a graph-based reasoning approach for fact checking. Our system apply Semantic Role Labeling (SRL) to construct graphs and present two graph-driven representation learning mechanisms.
- Results verify that both graph-based mechanisms improve the accuracy, and our final system achieves state-of-the-art performance on the FEVER dataset.
2 Task Definition and Pipeline
With a textual claim given as the input, the problem of fact checking is to find supporting evidence sentences to verify the truthfulness of the claim.
We conduct our research on FEVER (Thorne et al., 2018a), short for Fact Extraction and Verification, a benchmark dataset for fact checking. Systems are required to retrieve evidence sentences from Wikipedia, and predict the claim as "SUPPORTED", "REFUTED" or "NOT ENOUGH INFO (NEI) ", standing for that the claim is supported by the evidence, refuted by the evidence, and is not verifiable, respectively. There are two official evaluation metrics in FEVER. The first is the accuracy for three-way classification. The second is FEVER score, which further measures the percentage of correct retrieved evidence for "SUPPORTED" and "REFUTED" categories. Both the statistic of FEVER dataset and the equation for calculating FEVER score are given in Appendix B.
Figure 2: Our pipeline for fact checking on FEVER. The main contribution of this work is a graph-based reasoning model for claim verification.
Here, we present an overview of our pipeline for FEVER, which follows the majority of previous studies. Our pipeline consists of three main components: a document retrieval model, a sentence-level evidence selection model, and a claim verification model. Figure 2 gives an overview of the pipeline. With a given claim, the document retrieval model retrieves the most related documents from a given collection of Wikipedia documents. With retrieved documents, the evidence selection model selects top- $k$  related sentences as the evidence. Finally, the claim verification model takes the claim and evidence sentences as the input and outputs the veracity of the claim.
The main contribution of this work is the graph-based reasoning approach for claim verification, which is explained dually in Section 3. Our
6171
Figure 3: The constructed graph for the motivating example with two evidence sentences. Each box describes a "tuple" which is extracted by SRL triggered by a verb. Blue solid lines indicate edges that connect arguments within a tuple and red dotted lines indicate edges that connect argument across different tuples.
strategies for document selection and evidence selection are described in Section 4.
3 Graph-Based Reasoning Approach
In this section, we introduce our graph-based reasoning approach for claim verification, which is the main contribution of this paper. Taking a claim and retrieved evidence sentences<sup>1</sup> as the input, our approach predicts the truthfulness of the claim. For FEVER, it is a three-way classification problem, which predicts the claim as “SUPPORTED”, “REFUTED” or “NOT ENOUGH INFO (NEI)”.
The basic idea of our approach is to employ the intrinsic structure of evidence to assess the truthfulness of the claim. As shown in the motivating example in Figure 1, making the correct prediction needs good understanding of the semantic-level structure of evidence and the reasoning process based on that structure. In this section, we first describe our graph construction module (§3.1). Then, we present how to apply graph structure for fact checking, including a contextual representation learning mechanism with graph-based distance calculation (§3.2), and graph convolutional network and graph attention network to propagate and aggregate information over the graph (§3.3 and §3.4).
3.1 Graph Construction
Taking evidence sentences as the input, we would like to build a graph to reveal the intrinsic structure of these evidence. There might be many different
ways to construct the graph, such as open information extraction (Banko et al., 2007), named entity recognition plus relation classification, sequence-to-sequence generation which is trained to produce structured tuples (Goodrich et al., 2019), etc. In this work, we adopt a practical and flexible way based on semantic role labeling (Carreras and Marquez, 2004). Specifically, with the given evidence sentences, our graph construction operates in the following steps.
- For each sentence, we parse it to tuples<sup>2</sup> with an off-the-shelf SRL toolkit developed by AllenLNP<sup>3</sup>, which is a re-implementation of a BERT-based model (Shi and Lin, 2019).
- For each tuple, we regard its elements with certain types as the nodes of the graph. We heuristically set those types as verb, argument, location and temporal, which can also be easily extended to include more types. We create edges for every two nodes within a tuple.
- We create edges for nodes across different tuples to capture the structure information among multiple evidence sentences. Our idea is to create edges for nodes that are literally similar with each other. Assuming entity  $A$  and entity  $B$  come from different tuples, we add one edge if one of the following conditions is satisfied: (1)  $A$  equals  $B$ ; (2)  $A$  contains  $B$ ; (3) the number of overlapped words
${}^{2}$  A sentence could be parsed as multiple tuples.
3https://demo.allennlp.org/
semantic-role-labeling
<sup>1</sup>Details about how to retrieve evidence for a claim are described in Section 4.
6172
Figure 4: An overview of our graph-based reasoning approach for claim verification. Taking a claim and evidence sentences as the input, we first calculate contextual word representations with graph-based distance (§3.2). After that, we use graph convolutional network to propagate information over the graph (§3.3), and use graph attention network to aggregate information (§3.4) before making the final prediction.
between  $A$  and  $B$  is larger than the half of the minimum number of words in  $A$  and  $B$ .
Figure 3 shows the constructed graph of the evidence in the motivating example. In order to obtain the structure information of the claim, we use the same pipeline to represent a claim as a graph.
Our graph construction module offers an approach on modeling structure of multiple evidence, which could be further developed in the future.
3.2 Contextual Word Representations with Graph Distance
We describe the use of graph for learning graph-enhanced contextual representations of words<sup>4</sup>.
Our basic idea is to shorten the distance between two semantically related words on the graph, which helps to enhance their relationship when we calculate contextual word representations with a Transformer-based (Vaswani et al., 2017) pretrained model like BERT and XLNet. Supposing we have five evidence sentences  $\{s_1, s_2, \ldots, s_5\}$  and the word  $w_{1i}$  from  $s_1$  and the word  $w_{5j}$  from  $s_5$  are connected on the graph, simply concatenating evidence sentences as a single string fails to capture their semantic-level structure, and would give a large distance to  $w_{1i}$  and  $w_{5j}$ , which is the number of words between them across other three sentences (i.e.,  $s_2, s_3, \text{and} s_4$ ). An intuitive way to achieve our goal is to define an  $N \times N$  matrix of distances of words along the graph, where  $N$  is the total number of words in the evidence. However, this is unacceptable in practice because the
representation learning procedure will take huge memory space, which is also observed by Shaw et al. (2018).
In this work, we adopt pre-trained model XLNet (Yang et al., 2019) as the backbone of our approach because it naturally involves the concept of relative position<sup>5</sup>. Pre-trained models capture rich contextual representations of words, which is helpful for our task which requires sentence-level reasoning. Considering the aforementioned issues, we implement an approximate solution to trade off between the efficiency of implementation and the informativeness of the graph. Specifically, we reorder evidence sentences with a topology sort algorithm with the intuition that closely linked nodes should exist in neighboring sentences. This would prefer that neighboring sentences contain either parent nodes or sibling nodes, so as to better capture the semantic relatedness between different evidence sentences. We present our implementation in Appendix A. The algorithm begins from nodes without incident relations. For each node without incident relations, we recursively visit its child nodes in a depth-first searching way.
After obtaining graph-based relative position of words, we feed the sorted sequence into XLNet to obtain the contextual representations. Meanwhile, we obtain the representation  $h([CLS])$  for a special token [CLS], which stands for the joint representation of the claim and the evidence in Transformer-based architecture.
<sup>4</sup>In Transformer-based representation learning pipeline, the basic computational unit can also be word-piece. For simplicity, we use the term "word" in this paper.
Our approach can also be easily adapted to BERT by adding relative position like Shaw et al. (2018).
6173
3.3 Graph Convolutional Network
We have injected the graph information in Transformer and obtained  $h([CLS])$ , which captures the semantic interaction between the claim and the evidence at word level 6. As shown in our motivating example in Figure 1 and the constructed graph in Figure 3, the reasoning process needs to operate on span/argument-level, where the basic computational unit typically consists of multiple words like "Rodney King riots" and "the most popular county in the USA".
To further exploit graph information beyond word level, we first calculate the representation of a node, which is a word span in the graph, by averaging the contextual representations of words contained in the node. After that, we employ multilayer graph convolutional network (GCNs) (Kipf and Welling, 2016) to update the node representation by aggregating representations from their neighbors on the graph. Formally, we denote  $G$  as the graph constructed by the previous graph construction method and make  $H \in \mathbb{R}^{N^v \times d}$  a matrix containing representation of all nodes, where  $N^v$  and  $d$  denote the number of nodes and the dimension of node representations, respectively. Each row  $H_i \in \mathbb{R}^d$  is the representation of node  $i$ . We introduce an adjacency matrix  $A$  of graph  $G$  and its degree matrix  $D$ , where we add self-loops to matrix  $A$  and  $D_{ii} = \sum_j A_{ij}$ . One-layer GCNs will aggregate information through one-hop edges, which is calculated as follows:
$$
H _ {i} ^ {(1)} = \rho (\widetilde {A} H _ {i} W _ {0}), \tag {1}
$$
where  $H_{i}^{(1)}\in \mathbf{R}^{d}$  is the new  $d$  -dimension representation of node  $i$ $\widetilde{A} = D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$  is the normalized symmetric adjacency matrix,  $W_{0}$  is a weight matrix, and  $\rho$  is an activation function. To exploit information from the multi-hop neighboring nodes, we stack multiple GCNs layers:
$$
H _ {i} ^ {(j + 1)} = \rho \left(\widetilde {A} H _ {i} ^ {(j)} W _ {j}\right), \tag {2}
$$
where  $j$  denotes the layer number and  $H_{i}^{0}$  is the initial representation of node  $i$  initialized from the contextual representation. We simplify  $H^{(k)}$  as  $H$  for later use, where  $H$  indicates the representation of all nodes updated by  $k$ -layer GCNs.
The graph learning mechanism will be performed separately for claim-based and evidence-based graph. Therefore, we denote  $H_{c}$  and  $H_{e}$  as the representations of all nodes in claim-based graph and evidence-based graphs, respectively. Afterwards, we utilize the graph attention network to align the graph-level node representation learned for two graphs before making the final prediction.
3.4 Graph Attention Network
We explore the related information between two graphs and make semantic alignment for final prediction. Let  $H_{e} \in \mathbf{R}^{N_{e}^{v} \times d}$  and  $H_{c} \in \mathbf{R}^{N_{c}^{v} \times d}$  denote matrices containing representations of all nodes in evidence-based and claim-based graph respectively, where  $N_{e}^{v}$  and  $N_{c}^{v}$  denote number of nodes in the corresponding graph.
We first employ a graph attention mechanism (Veličković et al., 2017) to generate a claim-specific evidence representation for each node in claim-based graph. Specifically, we first take each  $h_c^i \in H_c$  as query, and take all node representations  $h_e^j \in H_e$  as keys. We then perform graph attention on the nodes, an attention mechanism  $a: \mathbf{R}^d \times \mathbf{R}^d \to \mathbf{R}$  to compute attention coefficient as follows:
$$
e _ {i j} = a (\boldsymbol {W _ {c}} h _ {c} ^ {i}, \boldsymbol {W _ {e}} h _ {e} ^ {j}) \qquad (3)
$$
which means the importance of evidence node  $j$  to the claim node  $i$ .  $W_{c} \in \mathbf{R}^{F \times d}$  and  $W_{e} \in \mathbf{R}^{F \times d}$  is the weight matrix and  $F$  is the dimension of attention feature. We use the dot-product function as  $a$  here. We then normalize  $e_{ij}$  using the softmax function:
$$
\alpha_ {i j} = \operatorname {s o f t m a x} _ {j} \left(e _ {i j}\right) = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k \in N _ {e} ^ {v}} \exp \left(e _ {i k}\right)} \tag {4}
$$
After that, we calculate a claim-centric evidence representation  $\mathbf{X} = [x_{1},\ldots ,x_{N_{c}^{v}}]$  using the weighted sum over  $\mathbf{H}_{e}$ :
$$
x _ {i} = \sum_ {j \in N _ {e} ^ {v}} \alpha_ {i j} h _ {e} ^ {j} \tag {5}
$$
We then perform node-to-node alignment and calculate aligned vectors  $A = [a_{1}, \ldots, a_{N_{c}^{v}}]$  by the claim node representation  $H^{c}$  and the claim-centric evidence representation  $X$ ,
$$
a _ {i} = f _ {\text {a l i g n}} \left(h _ {c} ^ {i}, x ^ {i}\right), \tag {6}
$$
where  $f_{\text{align}}(\cdot)$  denotes the alignment function. Inspired by Shen et al. (2018), we design our alignment function as:
$$
f _ {a l i g n} (x, y) = W _ {a} [ x, y, x - y, x \odot y ], \tag {7}
$$
<sup>6</sup>By "word" in "word-level", we mean the basic computational unit in XLNet, and thus  $h([CLS])$  capture the sophisticated interaction between words via multi-layer multi-head attention operations.
where  $W_{a} \in \mathbb{R}^{d \times 4 \times d}$  is a weight matrix and  $\odot$  is element-wise Hadamard product. The final output  $g$  is obtained by the mean pooling over  $A$ . We then feed the concatenated vector of  $g$  and the final hidden vector  $h([CLS])$  from XLNet through a MLP layer for the final prediction.
4 Document Retrieval and Evidence Selection
In this section, we briefly describe our document retrieval and evidence selection components to make the paper self contained.
4.1 Document Retrieval
The document retrieval model takes a claim and a collection of Wikipedia documents as the input, and returns  $m$  most relevant documents.
We mainly follow Nie et al. (2019), the top-performing system on the FEVER shared task (Thorne et al., 2018b). The document retrieval model first uses keyword matching to filter candidate documents from the massive Wikipedia documents. Then, NSMN (Nie et al., 2019) is applied to handle the documents with disambiguation titles, which are  $10\%$  of the whole documents. Documents without disambiguation title are assigned with higher scores in the resulting list. The input to the NSMN model includes the claim and candidate documents with disambiguation title. At a high level, NSMN model has encoding, alignment, matching and output layers. Readers who are interested are recommended to refer to the original paper for more details.
Finally, we select top-10 documents from the resulting list.
4.2 Sentence-Level Evidence Selection
Taking a claim and all the sentences from retrieved documents as the input, evidence selection model returns the top- $k$  most relevant sentences.
We regard evidence selection as a semantic matching problem, and leverage rich contextual representations embodied in pre-trained models like XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019a) to measure the relevance of a claim to every evidence candidate. Let's take XLNet as an example. The input of the sentence selector is
$$
c e _ {i} = [ C l a i m, S E P, E v i d e n c e _ {i}, S E P, C L S ]
$$
where Claim and  $Evidence_{i}$  indicate tokenized word-pieces of original claim and  $i^{th}$  evidence candidate,  $d$  denotes the dimension of hidden vector,
and  $SEP$  and  $CLS$  are symbols indicating ending of a sentence and ending of a whole input, respectively. The final representation  $h_{ce_i} \in \mathbb{R}^d$  is obtained via extracting the hidden vector of the  $CLS$  token.
After that, we employ an MLP layer and a softmax layer to compute score  $s_{ce_i}^+$  for each evidence candidate. Then, we rank all the evidence sentences by score  $s_{ce_i}^+$ . The model is trained on the training data with a standard cross-entropy loss. Following the official setting in FEVER, we select top-5 evidence sentences. The performance of our evidence selection model is shown in Appendix C.
We evaluate on FEVER (Thorne et al., 2018a), a benchmark dataset for fact extraction and verification. Each instance in FEVER dataset consists of a claim, groups of ground-truth evidence from Wikipedia and a label (i.e., "SUPPORTED", "REFUTED" or "NOT ENOUGH INFO (NEI)"), indicating its veracity. FEVER includes a dump of Wikipedia, which contains 5,416,537 pre-processed documents. The two official evaluation metrics of FEVER are label accuracy and FEVER score, as described in Section 2. Label accuracy is the primary evaluation metric we apply for our experiments because it directly measures the performance of the claim verification model. We also report FEVER score for comparison, which measures whether both the predicted label and the retrieved evidence are correct. No evidence is required if the predicted label is NEI.
We compare our system to the following baselines, including three top-performing systems on FEVER shared task, a recent work GEAR (Zhou et al., 2019), and a concurrent work by Liu et al. (2019b).
- Nie et al. (2019) employ a semantic matching neural network for both evidence selection and claim verification.
- Yoneda et al. (2018) infer the veracity of each claim-evidence pair and make final prediction by aggregating multiple predicted labels.
- Hanselowski et al. (2018) encode each claim-evidence pair separately, and use a pooling function to aggregate features for prediction.