SlowGuess/ACL_2020_OCR huggingface dataset:
Method	Label Acc (%)	FEVER Score (%)
Hanselowski et al. (2018)	65.46	61.58
Yoneda et al. (2018)	67.62	62.52
Nie et al. (2019)	68.21	64.21
GEAR (Zhou et al., 2019)	71.60	67.10
KGAT (Liu et al., 2019b)	72.81	69.40
DREAM (our approach)	76.85	70.60
- GEAR (Zhou et al., 2019) uses BERT to obtain claim-specific representation for each evidence sentence, and applies graph network by regarding each evidence sentence as a node in the graph.
- KGAT (Liu et al., 2019b) is concurrent with our work, which regards sentences as the nodes of a graph and uses Kernel Graph Attention Network to aggregate information.
5.2 Model Comparison
Table 1 reports the performance of our model and baselines on the blind test set with the score showed on the public leaderboard<sup>7</sup>. As shown in Table 1, in terms of label accuracy, our model significantly outperforms previous systems with  $76.85\%$  on the test set. It is worth noting that, our approach, which exploits explicit graph-level semantic structure of evidence obtained by SRL, outperforms GEAR and KGAT, both of which regard sentences as the nodes and use model to learn the implicit structure of evidence<sup>8</sup>. By the time our paper is submitted, our system achieves state-of-the-art performance in terms of both evaluation metrics on the leaderboard.
Table 2 presents the label accuracy on the development set after eliminating different components (including the graph-based relative distance ( $\S$ 3.2) and graph convolutional network and graph attention network ( $\S$ 3.3 and  $\S$ 3.4) separately in our model.
Table 1: Performance on the blind test set on FEVER. Our approach is abbreviated as DREAM.
Model	Label Accuracy
DREAM	79.16
-w/o Relative Distance	78.35
-w/o GCN&GAN	77.12
-w/o both above modules	75.40
Table 2: Ablation study on develop set.
The last row in Table 2 corresponds to the baseline where all the evidence sentences are simply concatenated as a single string, where no explicit graph structure is used at all for fact verification.
As shown in Table 2, compared to the XLNet baseline, incorporating both graph-based modules brings  $3.76\%$  improvement on label accuracy. Removing the graph-based distance drops  $0.81\%$  in terms of label accuracy. The graph-based distance mechanism can shorten the distance of two closely-linked nodes and help the model to learn their dependency. Removing the graph-based reasoning module drops  $2.04\%$  because graph reasoning module captures the structural information and performs deep reasoning about that. Figure 5 gives a case study of our approach.
We randomly select 200 incorrectly predicted instances and summarize the primary types of errors.
The first type of errors is caused by failing to match the semantic meaning between phrases that describe the same event. For example, the claim states "Winter's Tale is a book", while the evidence states "Winter's Tale is a 1983 novel by Mark Helprin". The model fails to realize that "novel" belongs to "book" and states that the claim is refuted. Solving this type of errors needs to involve external knowledge (e.g. ConceptNet (Speer et al., 2017)) that can indicate logical relationships between different events.
The misleading information in the retrieved evidence causes the second type of errors. For example, the claim states "The Gifted is a movie", and the ground-truth evidence states "The Gifted is an upcoming American television series". However, the retrieved evidence also contains "The Gifted is a 2014 Filipino dark comedy-drama movie", which misleads the model to make the wrong judgment.
In general, fact checking involves assessing the truthfulness of a claim. In literature, a claim can be
The public leaderboard for perpetual evaluation of FEVER is https://competitions.codalab.org/competitions/18814#results. DREAM is our user name on the leaderboard.
<sup>8</sup>We don't overclaim that the superiority of our system to GEAR and KGAT only comes from the explicit graph structure, because we have differences in other components like sentence selection and the pre-trained model.
6176
Claim	Text: Congressional Space Medal of Honor is the highest award given only to astronauts by NASA.
Tuples: ('Congressional Space Medal of Honor', 'is', 'the highest award given only to astronauts by NASA')
('the highest award', 'given', 'only', 'to astronauts', 'by NASA')
Evidence #1	Text: The highest award given by NASA, Congressional Space Medal of Honor is awarded by the President of the United States in Congress's name on recommendations from the Administrator of the National Aeronautics and Space Administration.
Tuples: ('The highest award', 'given', 'by NASA')
('Congressional Space Medal of Honor', 'awarded', 'by the President of the United States')
Evidence #2	Text: To be awarded the Congressional Space Medal of Honor, an astronaut must perform feats of extraordinary accomplishment while participating in space flight under the authority of NASA.
Tuples: ('awarded', 'the Congressional Space Medal of Honor')
('To be awarded the Congressional Space Medal of Honor', 'an astronaut', 'perform', 'feats of extraordinary accomplishment')
('an astronaut', 'participating', 'in space flight', 'under the authority of NASA')
Figure 5: A case study of our approach. Facts shared across the claim and the evidence are highlighted with different colors.
a text or a subject-predicate-object triple (Nakashole and Mitchell, 2014). In this work, we only consider textual claims. Existing datasets differ from data source and the type of supporting evidence for verifying the claim. An early work by Vlachos and Riedel (2014) constructs 221 labeled claims in the political domain from POLITIFACT.COM and CHANNEL4.COM, giving metadata of the speaker as the evidence. POLIFACT is further investigated by following works, including Ferreira and Vlachos (2016) who build Emergent with 300 labeled rumors and about 2.6K news articles, Wang (2017) who builds LIAR with 12.8K annotated short statements and six fine-grained labels, and Rashkin et al. (2017) who collect claims without meta-data while providing 74K news articles. We study FEVER (Thorne et al., 2018a), which requires aggregating information from multiple pieces of evidence from Wikipedia for making the conclusion. FEVER contains 185,445 annotated instances, which to the best of our knowledge is the largest benchmark dataset in this area.
The majority of participating teams in the FEVER challenge (Thorne et al., 2018b) use the same pipeline consisting of three components, namely document selection, evidence sentence selection, and claim verification. In document selec
tion phase, participants typically extract named entities from a claim as the query and use Wikipedia search API. In the evidence selection phase, participants measure the similarity between the claim and an evidence sentence candidate by training a classification model like Enhanced LSTM (Chen et al., 2016) in a supervised setting or using string similarity function like TFIDF without trainable parameters. Padia et al. (2018) utilizes semantic frames for evidence selection. In this work, our focus is the claim classification phase. Top-ranked three systems aggregate pieces of evidence through concatenating evidence sentences into a single string (Nie et al., 2019), classifying each evidence-claim pair separately, merging the results (Yoneda et al., 2018), and encoding each evidence-claim pair followed by pooling operation (Hanselowski et al., 2018). Zhou et al. (2019) are the first to use BERT to calculate claim-specific evidence sentence representations, and then develop a graph network to aggregate the information on top of BERT, regarding each evidence as a node in the graph. Our work differs from Zhou et al. (2019) in that (1) the construction of our graph requires understanding the syntax of each sentence, which could be viewed as a more fine-grained graph, and (2) both the contextual representation learning module and the reasoning module have model innovations of taking the graph information into consideration. Instead of training each component separately, Yin and Roth (2018) show that joint learning could improve both claim verification and evidence selection.
In this work, we present a graph-based approach for fact checking. When assessing the veracity of a claim giving multiple evidence sentences, our approach is built upon an automatically constructed graph, which is derived based on semantic role labeling. To better exploit the graph information, we propose two graph-based modules, one for calculating contextual word embeddings using graph-based distance in XLNet, and the other for learning representations of graph components and reasoning over the graph. Experiments show that both graph-based modules bring improvements and our final system is the state-of-the-art on the public leaderboard by the time our paper is submitted.
Evidence selection is an important component of fact checking as finding irrelevant evidence may lead to different predictions. A potential solution
6177
is to jointly learn evidence selection and claim verification model, which we leave as a future work.
Wanjun Zhong, Zenan Xu, Jiahai Wang and Jian Yin are supported by the National Natural Science Foundation of China (U1711262, U1611264, U1711261, U1811261, U1811264, U1911203), National Key R&D Program of China (2018YFB1004404), Guangdong Basic and Applied Basic Research Foundation (2019B1515130001), Key R&D Program of Guangdong Province (2018B010107005). The corresponding author is Jian Yin.
Gabor Angeli and Christopher D Manning. 2014. Naturalli: Natural logic inference for common sense reasoning. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 534-545.
Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In *Ijcai*, volume 7, pages 2670-2676.
Xavier Carreras and Lluis Márquez. 2004. Introduction to the conll-2004 shared task: Semantic role labeling. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89-97.
Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2016. Enhanced LSTM for natural language inference. arXiv preprint arXiv:1609.06038.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1-220.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler. 2017. Partisanship, propaganda, and disinformation: Online media and the 2016 us presidential election.
William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163-1168.
Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166-175. ACM.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. Ukp-athene: Multi-sentence textual entailment for claim verification. arXiv preprint arXiv:1809.01479.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Zhenghao Liu, Chenyan Xiong, and Maosong Sun. 2019b. Kernel graph attention network for fact verification. arXiv preprint arXiv:1910.09796.
Ndapandula Nakashole and Tom M Mitchell. 2014. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1009-1019.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859-6866.
Ankur Padia, Francis Ferraro, and Tim Finin. 2018. Team UMBC-FEVER: Claim verification using semantic lexical resources. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 161-165, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931-2937.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155.
Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018. Improved semantic-aware network embedding with fine-grained word alignment. arXiv preprint arXiv:1808.09633.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (fever) shared task. arXiv preprint arXiv:1811.10971.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18-22.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359(6380):1146-1151.
William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Wenpeng Yin and Dan Roth. 2018. Twowingos: A twowing optimization strategy for evidential claim verification. arXiv preprint arXiv:1808.03465.
Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Ucl machine reading group: Four factor framework for fact finding (hexaf). In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 97-102.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 892-901, Florence, Italy. Association for Computational Linguistics.
A Typology Sort Algorithm
Algorithm 1 Graph-based Distance Calculation Algorithm.
Require: A sequence of nodes  $S = \{s_i, s_2, \dots, s_n\}$ ; A set of relations  $R = \{r_1, r_2, \dots, r_m\}$
1: function DFS(node, visited, sorted_sequence)
2: for each child  $s_c$  in node's children do
3: if  $s_c$  has no incident edges and visited  $[s_c] == 0$  then
4: visited  $[s_c] = 1$
5: DFS  $(s_c,$  visited)
6: end if
7: end for
8: sorted_sequence.append(0, node)
9: end function
10: sorted_sequence  $= []$
11: visited  $= [0$  for i in range(n)]
12: S,R = changed_to_acyclic_graph(S,R)
13: for each node  $s_i$  in  $S$  do
14: if  $s_i$  has no incident edges and visited[i] == 0 then
15: visited[i] = 1
16: for each child  $s_c$  in  $s_i$ 's children do
17:  $\mathrm{DFS}(s_c,$  visited, sorted_sequence)
18: end for
19: sorted_sequence.append(0,  $s_i$ )
20: end if
21: end for
22: return sorted_sequence
B FEVER
The statistic of FEVER is shown in Table 3.
Split	SUPPORTED	REFUTED	NEI
Training	80,035	29,775	35,659
Dev	6,666	6,666	6,666
Test	6,666	6,666	6,666
Table 3: Split size of SUPPORTED, REFUTED and NOT ENOUGH INFO (NEI) classes in FEVER.
FEVER score is calculated with equation 8, where  $y$  is the ground truth label,  $\hat{y}$  is the predicted label,  $\pmb {E} = [E_1,\dots ,E_k]$  is a set of ground-truth evidence, and  $\hat{\pmb{E}} = [\hat{E}_1,\dots ,\hat{E}_5]$  is a set of predicted evidence.
$$
I n s t a n c e \_ C o r r e c t (y, \hat {y}, \boldsymbol {E}, \hat {\boldsymbol {E}}) \stackrel {d e f} {=}
$$
$$
y = \hat {y} \wedge (y = N E I \vee \text {E v i d e n c e C o r r e c t} (\boldsymbol {E}, \hat {\boldsymbol {E}})) \tag {8}
$$
C Evidence Selection Results
In this part, we present the performance of the sentence-level evidence selection module that we develop with different backbone. We take the concatenation of claim and each evidence as input, and take the last hidden vector to calculate the score for evidence ranking. In our experiments, we try both
6179
RoBERTa and XLNet. From Table 4, we can see that RoBERTa performs slightly better than XLNet here. When we submit our system on the leaderboard, we use RoBERTa as the evidence selection model.
Model	Dev. Set	Test Set
Acc.	Rec.	F1	Acc.	Rec.	F1
XLNet	26.60	87.33	40.79	25.55	85.34	39.33
RoBERTa	26.67	87.64	40.90	25.63	85.57	39.45
Table 4: Results of evidence selection models.
In this part, we describe the training details of our experiments. We employ cross-entropy loss as the loss function. We apply AdamW as the optimizer for model training. For evidence selection model, we set learning rate as 1e-5, batch size as 8 and maximum sequence length as 128.