SlowGuess/ACL_2020_OCR huggingface dataset:
Zhenghao Liu<sup>1</sup>
Chenyan Xiong
Maosong Sun
Zhiyuan Liu
$^{1}$ Department of Computer Science and Technology, Tsinghua University, Beijing, China  
 $^{2}$ Institute for Artificial Intelligence, Tsinghua University, Beijing, China
State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China
$^{2}$ Microsoft Research AI, Redmond, USA
Fact Verification requires fine-grained natural language inference capability that finds subtle clues to identify the syntactical and semantically correct but not well-supported claims. This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions. Given a claim and a set of potential evidence sentences that form an evidence graph, KGAT introduces node kernels, which better measure the importance of the evidence node, and edge kernels, which conduct fine-grained evidence propagation in the graph, into Graph Attention Networks for more accurate fact verification. KGAT achieves a  $70.38\%$  FEVER score and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification. Our analyses illustrate that, compared to dot-product attentions, the kernel-based attention concentrates more on relevant evidence sentences and meaningful clues in the evidence graph, which is the main source of KGAT's effectiveness. All source codes of this work are available at https://github.com/thunlp/KernelGAT.
Online contents with false information, such as fake news, political deception, and online rumors, have been growing significantly and spread widely over the past several years. How to automatically "fact check" the integrity of textual contents, to prevent the spread of fake news, and to avoid the undesired social influences of maliciously fabricated statements, is urgently needed for our society.
Recent research formulates this problem as the fact verification task, which targets to automatically verify the integrity of statements using trustworthy corpora, e.g., Wikipedia (Thorne et al., 2018a). For example, as shown in Figure 1, a system could first
Figure 1: An Example of Fact Verification System.
retrieve related evidence sentences from the background corpus, conduct joint reasoning over these sentences, and aggregate the signals to verify the claim integrity (Nie et al., 2019a; Zhou et al., 2019; Yoneda et al., 2018; Hanselowski et al., 2018).
There are two challenges for evidence reasoning and aggregation in fact verification. One is that no ground truth evidence is given; the evidence sentences are retrieved from background corpora, which inevitably contain noise. The other is that the false claims are often deliberately fabricated; they may be semantically correct but are not supported. This makes fact verification a rather challenging task, as it requires the fine-grained reasoning ability to distinguish the subtle differences between truth and false statements (Zhou et al., 2019).
This paper presents a new neural structural reasoning model, Kernel Graph Attention Network (KGAT), that provides more fine-grained evidence selection and reasoning capability for fact verification using neural matching kernels (Xiong et al., 2017; Dai et al., 2018). Given retrieved evidence
7342
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-7351 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
pieces, KGAT first constructs an evidence graph, using claim and evidence as graph nodes and fully-connected edges. It then utilizes two sets of kernels, one on the edges, which selectively summarize clues for a more fine-grained node representation and propagate clues among neighbor nodes through a multi-layer graph attention; and the other on the nodes, which performs more accurate evidence selection by better matching evidence with the claim. These signals are combined by KGAT, to jointly learn and reason on the evidence graph for more accurate fact verification.
In our experiments on FEVER (Thorne et al., 2018a), a large-scale fact verification benchmark, KGAT achieves a  $70.38\%$  FEVER score, significantly outperforming previous BERT and Graph Neural Network (GNN) based approaches (Zhou et al., 2019). Our experiments demonstrate KGAT's strong effectiveness especially on facts that require multiple evidence reasoning: our kernel-based attentions provide more sparse and focused attention patterns, which are the main source of KGAT's effectiveness.
The FEVER shared task (Thorne et al., 2018a) aims to develop automatic fact verification systems to check the veracity of human-generated claims by extracting evidence from Wikipedia. The recently launched FEVER shared task 1.0 is hosted as a competition on Codelab<sup>1</sup> with a blind test set and has drawn lots of attention from NLP community.
Existing fact verification models usually employ FEVER's official baseline (Thorne et al., 2018a) with a three-step pipeline system (Chen et al., 2017a): document retrieval, sentence retrieval and claim verification. Many of them mainly focus on the claim verification step. Nie et al. (2019a) concatenates all evidence together to verify the claim. One can also conduct reasoning for each claim evidence pair and aggregate them to the claim label (Luken et al., 2018; Yoneda et al., 2018; Hanselowski et al., 2018). TwoWingOS (Yin and Roth, 2018) further incorporates evidence identification to improve claim verification.
GEAR (Zhou et al., 2019) formulates claim verification as a graph reasoning task and provides two kinds of attentions. It conducts reasoning and aggregation over claim evidence pairs with
a graph model (Velicković et al., 2017; Scarselli et al., 2008; Kipf and Welling, 2017). Zhong et al. (2019) further employs XLNet (Yang et al., 2019) and establishes a semantic-level graph for reasoning for a better performance. These graph based models establish node interactions for joint reasoning over several evidence pieces.
Many fact verification systems leverage Natural Language Inference (NLI) techniques (Chen et al., 2017b; Ghaeini et al., 2018; Parikh et al., 2016; Radford et al., 2018; Peters et al., 2018; Li et al., 2019) to verify the claim. The NLI task aims to classify the relationship between a pair of premise and hypothesis as either entailment, contradiction or neutral, similar to the FEVER task, though the later requires systems to find the evidence pieces themselves and there are often multiple evidence pieces. One of the most widely used NLI models in FEVER is Enhanced Sequential Inference Model (ESIM) (Chen et al., 2017b), which employs some forms of hard or soft alignment to associate the relevant sub-components between premise and hypothesis. BERT, the pre-trained deep bidirectional Transformer, has also been used for better text representation in FEVER and achieved better performance (Devlin et al., 2019; Li et al., 2019; Zhou et al., 2019; Soleimani et al., 2019).
The recent development of neural information retrieval models, especially the interaction based ones, have shown promising effectiveness in extracting soft match patterns from query-document interactions (Hu et al., 2014; Pang et al., 2016; Guo et al., 2016; Xiong et al., 2017; Dai et al., 2018). One of the effective ways to model text matches is to leverage matching kernels (Xiong et al., 2017; Dai et al., 2018), which summarize word or phrase interactions in the learned embedding space between query and documents. The kernel extracts matching patterns which provide a variety of relevance match signals and shows strong performance in various ad-hoc retrieval dataset (Dai and Callan, 2019). Recent research also has shown kernels can be integrated with contextualized representations, i.e., BERT, to better model the relevance between query and documents (MacAvaney et al., 2019).
3 Kernel Graph Attention Network
This section describes our Kernel Graph Attention Network (KGAT) and its application in Fact Verification. Following previous research, KGAT first constructs an evidence graph using retrieved
1https://competitions.cdalab.org/ competitions/18814
7343
evidence sentences  $D = \{e^{1},\dots ,e^{p},\dots ,e^{l}\}$  for claim  $c$ , and then uses the evidence graph to predict the claim label  $y$  (Sec. 3.1 and 3.2). As shown in Figure 2, the reasoning model includes two main components: Evidence Propagation with Edge Kernels (Sec. 3.3) and Evidence Selection with Node Kernels (Sec. 3.4).
3.1 Reasoning with Evidence Graph
Similar to previous research (Zhou et al., 2019), KGAT constructs the evidence graph  $G$  by using each claim-evidence pair as a node and connects all node pairs with edges, making it a fully-connected evidence graph with  $l$  nodes:  $N = \{n^1, \dots, n^p, \dots, n^l\}$ .
KGAT unifies both multiple and single evidence reasoning scenarios and produces a probability  $P(y|c, D)$  to predict claim label  $y$ . Different from previous work (Zhou et al., 2019), we follow the standard graph label prediction setting in graph neural network (Veličković et al., 2017) and split the prediction into two components: 1) the label prediction in each node conditioned on the whole graph  $P(y|n^p, G)$ ; 2) the evidence selection probability  $P(n^p|G)$ :
$$
P (y | c, D) = \sum_ {p = 1} ^ {l} P (y | c, e ^ {p}, D) P \left(e ^ {p} | c, D\right), \tag {1}
$$
or in the graph notation:
$$
P (y | G) = \sum_ {p = 1} ^ {l} P (y | n ^ {p}, G) P (n ^ {p} | G). \tag {2}
$$
The joint reasoning probability  $P(y|n^p, G)$  calculates node label prediction with multiple evidence. The readout module (Knyazev et al., 2019) calculates the probability  $P(n^p | G)$  and attentively combines per-node signals for prediction.
The rest of this section describes the initialization of node representations  $(n^p)$  in Sec. 3.2, the calculation of per-node predictions  $P(y|n^p,G)$  with Edge Kernels (Sec. 3.3), and the readout module  $P(n^{p}|G)$  with Node Kernels (Sec. 3.4).
3.2 Initial Node Representations
The node representations are initialized by feeding the concatenated sequence of claim, document (Wiki) title, and evidence sentence, to pre-trained BERT model (Devlin et al., 2019). Specifically, in the node  $n_p$ , the claim and evidence correspond to  $m$  tokens (with "[SEP]") and  $n$  tokens (with
Figure 2: KGAT Architecture.
Wikipedia title and "[SEP]". Using the BERT encoder, we get the token hidden states  $H^{p}$  with the given node  $n^{p}$ :
$$
H ^ {p} = \operatorname {B E R T} \left(n ^ {p}\right). \tag {3}
$$
The representation of the first token ("[CLS]") is denoted as the initial representation of node  $n^p$ :
$$
z ^ {p} = H _ {0} ^ {p}. \tag {4}
$$
The rest of the sequences  $H_{1:m+n}^p$  are also used to represent the claim and evidence tokens:  $H_{1:m}^p$  for the claim tokens and  $H_{m+1:m+n}^p$  for the evidence tokens.
3.3 Edge Kernel for Evidence Propagation
The evidence propagation and per-node label prediction in KGAT are conducted by Edge Kernels, which attentively propagate information among nodes in the graph  $G$  along the edges with the kernel attention mechanism.
Specifically, KGAT calculates the node  $n^p$ 's representation  $v^p$  with the kernel attention mechanism, and uses it to produce the per-node claim prediction  $y$ :
$$
v ^ {p} = \operatorname {E d g e - K e r n e l} (n ^ {p}, G),
$$
$$
P (y \mid n ^ {p}, G) = \operatorname {s o f t m a x} _ {y} (\operatorname {L i n e a r} (v ^ {p})). \tag {5}
$$
The edge kernel of KGAT conducts a hierarchical attention mechanism to propagate information between nodes. It uses token level attentions to
7344
produce node representations and sentence level attentions to propagate information along edges.
Token Level Attention. The token level attention uses kernels to get the fine-grained representation  $\hat{z}^{q\rightarrow p}$  of neighbor node  $n^q$  , according to node  $n^p$  . The content propagation and the attention are controlled by kernels.
To get the attention weight  $\alpha_{i}^{q\to p}$  for  $i$  -th token in  $n^q$  , we first conduct a translation matrix  $M^{q\rightarrow p}$  between  $q$  -th node and  $p$  -th node. Each element of the translation matrix  $M_{ij}^{q\rightarrow p}$  in  $M^{q\rightarrow p}$  is the cosine similarity of their corresponding tokens' BERT representations:
$$
M _ {i j} ^ {q \rightarrow p} = \cos \left(H _ {i} ^ {q}, H _ {j} ^ {p}\right). \tag {6}
$$
Then we use  $K$  kernels to extract the matching feature  $\vec{K}(M_i^{q\rightarrow p})$  from the translation matrix  $M^{q\rightarrow p}$  (Xiong et al., 2017; Dai et al., 2018; Qiao et al., 2019; MacAvaney et al., 2019):
$$
\vec {K} (M _ {i} ^ {q \rightarrow p}) = \{K _ {1} (M _ {i} ^ {q \rightarrow p}), \dots , K _ {K} (M _ {i} ^ {q \rightarrow p}) \}. \quad (7)
$$
Each kernel  $K_{k}$  utilizes a Gaussian kernel to extract features and summarizes the translation score to support multi-level interactions:
$$
K _ {k} \left(M _ {i} ^ {q \rightarrow p}\right) = \log \sum_ {j} \exp \left(- \frac {\left(M _ {i j} ^ {q \rightarrow p} - \mu_ {k}\right) ^ {2}}{2 \delta_ {k} ^ {2}}\right), \tag {8}
$$
where  $\mu_{k}$  and  $\delta_{k}$  are the mean and width for the  $k$ -th kernel, which captures a certain level of interactions between the tokens (Xiong et al., 2017).
Then each token's attention weight  $\alpha_{i}^{q\rightarrow p}$  is calculated using a linear layer:
$$
\alpha_ {i} ^ {q \rightarrow p} = \operatorname {s o f t m a x} _ {i} \left(\operatorname {L i n e a r} \left(\vec {K} \left(M _ {i} ^ {q \rightarrow p}\right)\right)\right). \tag {9}
$$
The attention weights are used to combine the token representations  $(\hat{z}^{q\rightarrow p})$  ..
$$
\hat {z} ^ {q \rightarrow p} = \sum_ {i = 1} ^ {m + n} \alpha_ {i} ^ {q \rightarrow p} \cdot H _ {i} ^ {q}, \tag {10}
$$
which encodes the content signals to propagate from node  $n^q$  to node  $n^p$ .
Sentence Level Attention. The sentence level attention combines neighbor node information to node representation  $v^{p}$ . The aggregation is done by a graph attention mechanism, the same with previous work (Zhou et al., 2019).
It first calculate the attention weight  $\beta^{q\to p}$  of  $n^q$  node according to the  $p$ -th node  $n^p$ :
$$
\beta^ {q \rightarrow p} = \operatorname {s o f t m a x} _ {q} \left(\mathrm {M L P} \left(z ^ {p} \circ \hat {z} ^ {q \rightarrow p}\right)\right), \tag {11}
$$
where  $\circ$  denotes the concatenate operator and  $z^p$  is the initial representation of  $n^p$ .
Then the  $p$ -th node's representation is updated by combining the neighbor node representations  $\hat{z}^{q\rightarrow p}$  with the attention:
$$
v ^ {p} = \left(\sum_ {q = 1} ^ {l} \beta^ {q \rightarrow p} \cdot \hat {z} ^ {q \rightarrow p}\right) \circ z ^ {p}. \tag {12}
$$
It updates the node representation with its neighbors, and the updated information are selected first by the token level attention (Eq. 9) and then the sentence level attention (Eq. 11).
Sentence Level Claim Label Prediction. The updated  $p$ -th node representation  $v^{p}$  is used to calculate the claim label probability  $P(y|n^{p})$ :
$$
P (y \mid n ^ {p}, G) = \operatorname {s o f t m a x} _ {y} (\operatorname {L i n e a r} (v ^ {p})). \tag {13}
$$
The prediction of the label probability for each node is also conditioned on the entire graph  $G$ , as the node representation is updated by gather information from its graph neighbors.
3.4 Node Kernel for Evidence Aggregation
The per-node predictions are combined by the "readout" function in graph neural networks (Zhou et al., 2019), where KGAT uses node kernels to learn the importance of each evidence.
It first uses node kernels to calculate the readout representation  $\phi (n^p)$  for each node  $n^p$ :
$$
\phi \left(n ^ {p}\right) = \operatorname {N o d e - K e r n e l} \left(n ^ {p}\right). \tag {14}
$$
Similar to the edge kernels, we first conduct a translation matrix  $M^{c \to e^p}$  between the  $p$ -th claim and evidence, using their hidden state set  $H_{1:m}^p$  and  $H_{m+1:m+n}^p$ . The kernel match features  $\vec{K}(M_i^{c \to e^p})$  on the translation matrix are combined to produce the node selection representation  $\phi(n^p)$ :
$$
\phi (n ^ {p}) = \frac {1}{m} \cdot \sum_ {i = 1} ^ {m} \vec {K} \left(M _ {i} ^ {c \rightarrow e ^ {p}}\right). \tag {15}
$$
This representation is used in the readout to calculate  $p$ -th evidence selection probability  $P(n^{p}|G)$ :
$$
P \left(n ^ {p} | G\right) = \operatorname {s o f t m a x} _ {p} \left(\operatorname {L i n e a r} \left(\phi \left(n ^ {p}\right)\right)\right). \tag {16}
$$
KGAT leverages the kernels multi-level soft matching capability (Xiong et al., 2017) to weight the node-level predictions in the evidence graph based on their relevance with the claim:
$$
P (y | G) = \sum_ {p = 1} ^ {l} P (y | n ^ {p}, G) P \left(n ^ {p} | G\right). \tag {17}
$$
7345
The whole model is trained end-to-end by minimizing the cross entropy loss:
$$
L = \text {C r o s s E n t r o p y} \left(y ^ {*}, P (y | G)\right), \tag {18}
$$
using the ground truth verification label  $y^{*}$ .
4 Experimental Methodology
This section describes the dataset, evaluation metrics, baselines, and implementation details in our experiments.
Dataset. A large scale public fact verification dataset FEVER (Thorne et al., 2018a) is used in our experiments. The FEVER consists of 185,455 annotated claims with 5,416,537 Wikipedia documents from the June 2017 Wikipedia dump. All claims are classified as SUPPORTS, REFUTES or NOT ENOUGH INFO by annotators. The dataset partition is kept the same with the FEVER Shared Task (Thorne et al., 2018b) as shown in Table 1.
Evaluation Metrics. The official evaluation metrics² for claim verification include Label Accuracy (LA) and FEVER score. LA is a general evaluation metric, which calculates claim classification accuracy rate without considering retrieved evidence. The FEVER score considers whether one complete set of golden evidence is provided and better reflects the inference ability.
We also evaluate Golden FEVER (GFEVER) scores, which is the FEVER score but with golden evidence provided to the system, an easier setting. Precision, Recall and F1 are used to evaluate evidence sentence retrieval accuracy using the provided sentence level labels (whether the sentence is evidence or not to verify the claim).
Baselines. The baselines include top models during FEVER 1.0 task and BERT based models.
Three top models in FEVER 1.0 shared task are compared. Athene (Hanselowski et al., 2018) and UNC NLP (Nie et al., 2019a) utilize ESIM to encode claim evidence pairs. UCL MRG (Yoneda et al., 2018) leverages Convolutional Neural Network (CNN) to encode claim and evidence. These three models aggregate evidence by attention mechanism or label aggregation component.
The BERT based models are our main baselines, they significantly outperform previous methods without pre-training. BERT-pair, BERT-concat and GEAR are three baselines from the previous
Split	SUPPORTED	REFUTED	NOT ENOUGH INFO
Train	80,035	29,775	35,639
Dev	6,666	6,666	6,666
Test	6,666	6,666	6,666
Table 1: Statistics of FEVER Dataset.
work (Zhou et al., 2019). BERT-pair and BERT-concat regard claim-evidence pair individually or concatenate all evidence together to predict claim label. GEAR utilizes a graph attention network to extract supplement information from other evidence and aggregate all evidence through an attention layer. Soleimani et al. (2019); Nie et al. (2019b) are also compared in our experiments. They implement BERT sentence retrieval for a better performance. In addition, we replace kernel with dot product to implement our GAT version, which is similar to GEAR, to evaluate kernel's effectiveness.
Implementation Details. The rest of this section describes our implementation details.
Document retrieval. The document retrieval step retrieves related Wikipedia pages and is kept the same with previous work (Hanselowski et al., 2018; Zhou et al., 2019; Soleimani et al., 2019). For a given claim, it first utilizes the constituency parser in AllenNLP (Gardner et al., 2018) to extract all phrases which potentially indicate entities. Then it uses these phrases as queries to find relevant Wikipedia pages through the online MediaWiki API<sup>3</sup>. Then the convinced article are reserved (Hanselowski et al., 2018).
Sentence retrieval. The sentence retrieval part focuses on selecting related sentences from retrieved pages. There are two sentence retrieval models in our experiments: ESIM based sentence retrieval and BERT based sentence retrieval. The ESIM based sentence retrieval keeps the same as the previous work (Hanselowski et al., 2018; Zhou et al., 2019). The base version of BERT is used to implement our BERT based sentence retrieval model. We use the "[CLS]" hidden state to represent claim and evidence sentence pair. Then a learning to rank layer is leveraged to project "[CLS]" hidden state to ranking score. Pairwise loss is used to optimize the ranking model. Some work (Zhao et al., 2020; Ye et al., 2020) also employs our BERT based sentence retrieval in their experiments.
Claim verification. During training, we set the
$^{2}$ https://github.com/sheffieldnlp/ fever-scorer
3https://www.mediawiki.org/wiki/API: Main_page
7346
Model	Dev	Test
LA	FEVER	LA	FEVER
Athene (Hanselowski et al., 2018)	68.49	64.74	65.46	61.58
UCL MRG (Yoneda et al., 2018)	69.66	65.41	67.62	62.52
UNC NLP (Nie et al., 2019a)	69.72	66.49	68.21	64.21
BERT Concat (Zhou et al., 2019)	73.67	68.89	71.01	65.64
BERT Pair (Zhou et al., 2019)	73.30	68.90	69.75	65.18
GEAR (Zhou et al., 2019)	74.84	70.69	71.60	67.10
GAT (BERT Base) w. ESIM Retrieval	75.13	71.04	72.03	67.56
KGAT (BERT Base) w. ESIM Retrieval	75.51	71.61	72.48	68.16
SR-MRS (Nie et al., 2019b)	75.12	70.18	72.56	67.26
BERT (Base) (Soleimani et al., 2019)	73.51	71.38	70.67	68.50
KGAT (BERT Base)	78.02	75.88	72.81	69.40
BERT (Large) (Soleimani et al., 2019)	74.59	72.42	71.86	69.66
KGAT (BERT Large)	77.91	75.86	73.61	70.24
KGAT (RoBERTa Large)	78.29	76.11	74.07	70.38
batch size to 4 and accumulate step to 8. All models are evaluated with LA on the development set and trained for two epochs. The training and development sets are built with golden evidence and higher ranked evidence with sentence retrieval. All claims are assigned with five pieces of evidence. The BERT (Base), BERT (Large) and RoBERTa (Liu et al., 2019) are evaluated in claim verification.
In our experiments, the max length is set to 130. All models are implemented with PyTorch. BERT inherits huggingface's implementation<sup>4</sup>. Adam optimizer is used with learning rate  $= 5\mathrm{e} - 5$  and warm up proportion  $= 0.1$ . The kernel size is set to 21, the same as previous work (Qiao et al., 2019).
5 Evaluation Result
The experiments are conducted to study the performance of KGAT, its advantages on different reasoning scenarios, and the effectiveness of kernels.
The fact verification performances are shown in Table 2. Several testing scenarios are conducted to compare KGAT effectiveness to BERT based baselines: BERT (Base) Encoder with ESIM retrieved sentences, with BERT retrieved sentences, and BERT (Large) Encoder with BERT retrieved sentences.
Compared with baseline models, KGAT is the best on all testing scenarios. With ESIM sentence retrieval, same as the previous work (Zhou et al., 2019; Hanselowski et al., 2018), KGAT outperforms the graph attention models GEAR and our GAT on both development and testing sets.
Table 2: Fact Verification Accuracy. The performances of top models during FEVER 1.0 shared task and BERT based models with different scenarios are presented.
Model	Prec@5	Rec@5	F1@5	FEVER
Dev	ESIM	24.08	86.72	37.69	71.70
BERT	27.29	94.37	42.34	75.88
Test	ESIM	23.51	84.66	36.80	68.16
BERT	25.21	87.47	39.14	69.40
Table 3: Evidence Sentence Retrieval Accuracy. Sentence level Precision, Recall and F1 are evaluated by official evaluation (Thorne et al., 2018a).