SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 8: Performance comparison of six fact checking system configurations with evidence oracle, WikiAPI, Dense Passage Retrieval (Karpukhin et al., 2020), BERT (Devlin et al., 2019), CorefBERT (Ye et al., 2020) with Kernel Graph Attention Network (KGAT) (Liu et al., 2020) on FEVER (Thorne et al., 2018) and our Colloquial Claims.
Evidence Selection. WikiAPI and DPR both use BERT (Devlin et al., 2019) to encode sentences and sort them out from the documents.
Claim Verification. We test two approaches: (1) BERT and (2) CorefBERT (Ye et al., 2020), which is one of the best performing methods on FEVER. The CorefBERT pretrains BERT to better capture the coreference information in text. We also apply kernel graph attention network (KGAT) (Liu et al., 2020) on BERT and CorefBERT for fine-grained attention using evidence graphs. More details can be found in Appendix.
Table 8 compares the performance of fact checking systems on FEVER and our Colloquial Claims. Both label accuracy and FEVER-score significantly decrease for all systems on our Colloquial Claims, compared to FEVER. The WikiAPI+BERT+KGAT(CorefBERT) system performs on par with best performing models for FEVER by label accuracy of  $73.8\%$ . However, it degenerates on the colloquial dataset with the label accuracy of  $60.9\%$ . We remind that our Colloquial Claims shares the same document pool, annotated evidence sentences, and similar semantics with claims from FEVER. Thus, it is the difference in the claim's style that makes the fact checking systems fatally degenerate.
The WikiAPI, used in many fact checking systems (Hanselowski et al., 2018; Chernyavskiy and Ilvovsky, 2019; Stammbach and Neumann, 2019; Zhou et al., 2019; Liu et al., 2020), shows superior performance than DPR on the FEVER dataset, with document recall of  $90.0\%$ . On Colloquial Claims, however, it crashes down to  $72.2\%$ . Meanwhile, the DPR shows more robust document retrieval on
Colloquial Claims than WikiAPI.
Apart from document retrieval and evidence selection, we can also observe performance decrease in the systems with evidence oracles. This indicates that claim verification is also more difficult on Colloquial Claims.
5.4 Challenges in Colloquial Claims
We analyze the causes of degeneration in document retrieval and claim verification in relation to the colloquial traits. We compare three document retrieval methods along with the oracle: WikiAPI, DrQA (Chen et al., 2017), and Dense Passage Retrieval (DPR). DrQA is another variation of term-matching method based on TF-IDF. Table 9 shows the titles of ten most documents by each retriever.
Filler Words Unnecessary of Fact Checking. In colloquial language, claims are not always composed of factual remarks requiring verification. Filler words (e.g. "I see", "yeah, like") are also frequently mixed in the utterances, as shown in Table 5. Hence, our Colloquial Claims requires systems to partition the parts that affect veracity from the ones that do not. However, Table 9 shows that word-matching retrieval systems, such as WikiAPI and DrQA, are vulnerable to those insignificant parts. They naively retrieve filler word related documents very frequently.
Minding the Context. Considering the context inside the sentence is essential for verifying colloquial claims. Lexical variation and polysemy is common in colloquial language. Such variations and ambiguity are tolerable because common context flows in the utterance. For example, in the colloquial claim of "Niko Coster-Waldau is also the host of the show. He was with Fox at one point," it is easy to see the word "Fox" stands for "Fox Broad-
Oracle	WikiAPI
Pakistan	It
Pocahontas	I
SpongeBob	You
Far from the Madding Crowd	Yes (band)
Samsung	Yes (album)
Two and a Half Men	He
Elizabeth of York	That
Ice-T	They
Spiderman	There There (novel)
Sausage Party	HES
DrQA	DPR
Heroes of Russia	Minor League
Yeah Yeah	Beverly Hillbillies
Yea	Ed and Lorraine Warren
H*** Yeah	Benjamin Franklin
Yea (football club)	Yin and Yang
Stefanie Drootin	Hunger Games (film)
Minor League	Sausage Party
Video Games	Ice-T
Google Search	Mormons
Google Apps	Burj Khalifa
Table 9: Comparison of the titles of the top-10 retrieved documents between oracle, WikiAPI, DrQA and DPR.
casting Company" based on the context. However, it is well known that simple term-matching methods cannot capture such context (Karpukhin et al., 2020). Thus, we observe that systems instead simply retrieve the document of "fox". Also, Table 9 shows another example of contextless retrieval. The document "Yes (band)", "There There (novel)", and "Yea (football club)" are naively retrieved by the systems, due to simple filler words in colloquial claims.
Overcoming the Colloquial Traits. Methods based on TF-IDF or word-matching are good at recognizing core keywords, but suffer at capturing the rich semantics of context. On the other hand, the DPR, a similarity search method based on dense embeddings, shows promising results. Results in Table 9 illustrate that DPR is able to ignore the context-irrelevant entities and focus more on fact-related entities. Compared to other retrieval methods, the ten most retrieved documents from DPR does not contain any filler words. Since filler words are irrelevant to the veracity of colloquial claims, the DPR learns their insignificance. Therefore, dense representation can be important for making fact-checking systems to be robust on claims in dialogues.
Fact Checking and Verification. The need for claim verification has led to annotated fact check-
ing datasets (Thorne et al., 2018; Baly et al., 2018; Augenstein et al., 2019; Jiang et al., 2020; Wadden et al., 2020; Chen et al., 2020). Recent works deploy adversarial attacks against fact checking systems (Thorne et al., 2019a,b; Niewinski et al., 2019; Atanasova et al., 2020b) and attempt to improve the system through generation (Atanasova et al., 2020a; Goyal and Durrett, 2020; Fan et al., 2020). Existing works tend to focus on verifying news or Wikipedia. However, verifying facts is not limited to such formal texts. Compared to previous works, we focus on verifying claims in the dialogue domain, which resembles more daily life situations.
A special case of fact verification is rumour detection. Its goal is to determine the veracity of rumours from social media (Li et al., 2019). The rumour is classified based on the reactions of chained messages (Gorrell et al., 2019). The procedure and characteristics of rumour detection is quite different from the fact checking pipeline (Gorrell et al., 2019). In our task, we verify the claims based on factuality from the related documents, rather than stances of the comments.
Safety in Open-domain Dialogue. Recently, much work has studied safety issues of machine dialogue agents in several aspects. Wulczyn et al. (2017) attempt to detect personal attacks in Wikipedia talk pages. Henderson et al. (2018) note the axes of bias, adversarial examples, privacy and safety, and propose that the community should aim to provide conditional safety guarantees. Khatri et al. (2018) train a sensitive language detector to evaluate the utterances in a chatbot dataset. Dinan et al. (2019a) propose a framework for dialogue agents to be robust to malicious human attacks. Other works have attempted to mitigate biases, such as gender bias (Dinan et al., 2020a) and racial bias (Sap et al., 2019). Recently, Tran et al. (2020) modify BERT (Devlin et al., 2019) to detect hatespeech. Xu et al. (2020) introduce a method to distill safety standards into the generative dialogue agent.
Previous works cover a wide range of dialogue safety, yet the risk of disinformation and misinformation remain understudied. In this work, we extend dialogue safety to cover verification of responses with false information.
This work aimed to open up new discussions in the intersection of fact checking and dialogue safety. In order to study how existing fact checking systems behave on claims in dialogues, we curate colloquial claims by transferring the styles of claims in FEVER (Thorne et al., 2018) to colloquialism. We leverage BART (Lewis et al., 2020) and Wizard of Wikipedia (WoW) (Dinan et al., 2019b). We finetune BART to generate the wizard's responses with knowledge sentences from WoW. Then, we input FEVER claims to generate claim-grounded utterances. We oversample candidate claims and apply filters to compensate quality. We showed that existing fact checking systems well-performing on FEVER degenerate on colloquial claims. We found that the document retriever is the weakest spot in the system which is even vulnerable to filler words. We compared the characteristic differences between claims in formal style and ones in colloquialism. An important future direction will be building a dialogue dataset for fact checking.
We would like to thank Jinseo Jeong and Myeong-jang Pyeon for their valuable comments. We also thank the anonymous reviewers for their thoughtful suggestions on this work. This research was supported by Samsung Research Funding Center of Samsung Electronics under project number SRFC-IT2101-01. Gunhee Kim is the corresponding author.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020a. Generating Fact Checking Explanations. In ACL.
Pepa Atanasova, Dustin Wright, and Isabelle Augenstein. 2020b. Generating Label Cohesive and Well-Formed Adversarial Claims. In EMNLP.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In EMNLP-IJCNLP.
R. Baly, Mitra Mohtarami, James R. Glass, Lluis Marquez i Villodre, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating Stance Detection and Fact Checking in a Unified Corpus. In NAACL-HLT.
Timothy W Bickmore, Ha Trinh, Stefan Olafsson, Teresa K O'Leary, Reza Asadi, Nathaniel M Rickles, and Ricardo Cruz. 2018. Patient and Consumer Safety Risks when using Conversational Assistants for Medical Information: an Observational Study of Siri, Alexa, and Google Assistant. Journal of Medical Internet Research, 20(9):e11510.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial Filters of Dataset Biases. In ICML.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In ACL.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. TabFact: A Large-scale Dataset for Table-based Fact Verification. In ICLR.
Anton Chernyavskiy and Dmitry Ilvovsky. 2019. Extract and Aggregate: A Novel Domain-Independent Approach to Factual Data Verification. In FEVER.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020a. Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation. In EMNLP.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. In EMNLP-IJCNLP.
Emily Dinan, Verena Rieser, Alborz Geramifard, Zhou Yu, and Dilek Hakkani-Tür. 2020b. Safety for Conversational AI Workshop. https:// safetyforconvai.splashthat.com/.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In ICLR.
Mihail Eric and Christopher D Manning. 2017. Key-Value Retrieval Networks for Task-Oriented Dialogue. In SIGDIAL.
Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. Generating Fact Checking Briefs. In EMNLP.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A Knowledge-Grounded Neural Conversation Model. In AAAI.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinglang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI. 2019. Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In Interspeech.
Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2019. RumourEval 2019: Determining Rumour Veracity and Support for Rumours. In *NAACL-HLT SemEval workshop*.
Tanya Goyal and Greg Durrett. 2020. Evaluating Factuality in Generation with Dependency-level Entailment. In Findings of EMNLP.
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018. UKP-Athene: Multisentence Textual Entailment for Claim Verification. In FEVER.
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical Challenges in Data-Driven Dialogue Systems. In AIES.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In ICLR.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification. In EMNLP.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In EMNLP.
Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018. Detecting offensive content in open-domain conversations using two stage semi-supervision. In NeurIPS ConvAI Workshop.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension. In ACL.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A Persona-Based Neural Conversation Model. In ACL.
Quanzhi Li, Q. Zhang, L. Si, and Yingchi Liu. 2019. Rumor Detection on Social Media: Datasets, Methods and Opportunities. In EMNLP NLP4IF Workshop.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained Fact Verification with Kernel Graph Attention Network. In ACL, pages 7342-7351.
A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. ParlAI: A Dialog Research Software Platform. arXiv:1705.06476.
Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M Khapra. 2018. Towards Exploiting Background Knowledge for Building Conversation Systems. In EMNLP.
Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining Fact Extraction and Verification with Neural Semantic Matching Networks. In AAAI.
Piotr Niewinski, Maria Pszona, and Maria Janicka. 2019. GEM: Generative Enhanced Model for Adversarial Attacks. In EMNLP FEVER Workshop.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In ACL System Demonstration.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. 2019. Conversing by Reading: Contentful Neural Conversation with On-Demand Machine Reading. In ACL.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for Building an Open-Domain Chatbot. arXiv:2004.13637.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In AAAI.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In ACL.
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter, 19(1):22-36.
Dominik Stammbach and Guenter Neumann. 2019. Team DOMLIN: Exploiting evidence enhancement for the fever shared task. In FEVER.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERIFICATION. In NAACL-HLT.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019a. Evaluating Adversarial Attacks Against Multiple Fact Verification Systems. In EMNLP.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019b. The FEVER2.0 Shared Task. In EMNLP FEVER Workshop.
Thanh Tran, Yifan Hu, Changwei Hu, Kevin Yen, Fei Tan, Kyumin Lee, and Serim Park. 2020. HABERTOR: An Efficient and Effective Deep Hatespeech Detector. In EMNLP.
David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hananeh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In EMNLP.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0. LDC, 23.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A Broad-coverage Challenge Corpus for Sentence Understanding through Inference. In NAACL-HLT.
M. Wolf, K. Miller, and F. Grodzinsky. 2017. Why We should have Seen that Coming: Comments on Microsoft's Tay "Experiment," and Wider Implications. SIGCAS Computers and Society, 47:54-64.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funowicz, et al. 2019. Transformers: State-of-the-Art Natural Language Processing. arXiv:1910.03771.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020. Diverse and Informative Dialogue Generation with Context-Specific CommonSense Knowledge Awareness. In ACL.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In TheWebConf.
Jing Xu, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for Safety in Open-domain Chatbots. arXiv:2010.07079.
Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang, and Zhoujun Li. 2020. StyleDGPT: Stylized Response Generation with Pre-trained Language Models. In Findings of EMNLP.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In EMNLP.
Tom Young, Erik Cambria, Iti Chaturvedi, Minlie Huang, Hao Zhou, and Subham Biswas. 2018. Aug-mentation End-to-end Dialog Systems with Common- sense Knowledge. In AAAI.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-Scale Generative Pre-Training for Conversational Response Generation. arXiv:1911.00536.
Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Reasoning over semantic-level graph for fact checking. In ACL.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018a. Common-sense Knowledge Aware Conversation Generation with Graph Attention. In *IJCAI*.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification. In ACL.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018b. A Dataset for Document Grounded Conversations. In EMNLP.
A Implementation Details of AFLITE
We use adversarial filtering method AFLITE (Sakaguchi et al., 2020; Bras et al., 2020) to select top- $k$  candidate claims which are most difficult to discriminate from responses in Wizard of Wikipedia (WoW) (Dinan et al., 2019b). The algorithm takes as input the original WoW and Colloquial Claims, then returns each filtered dataset. AFLITE comprised with two steps: (i) precomputing phase and (ii) filtering phase.
In precomputing phase, we randomly sample  $10\%$  of instances from WoW and Colloquial Claims to fine-tune RoBERTa-large. We then use finetuned RoBERTa to pre-compute embeddings for the rest of the instances as the input for the filtering phase. We discard samples used for fine-tuning from the final dataset.
In filtering phase, we use an ensemble of linear classifiers to iteratively discard easily distinguishable instances. At each iteration, we train 32 linear classifiers on different random partitions of the data and collect their predictions on their rest of the instances. For each instance, we compute its score as the ratio of correct predictions over the total number of predictions, and remove top- $n$  instances whose score is above threshold 0.75. We remove top-1000 instances among the entire WoW, and top-2 instances for each candidate sets in Colloquial Claims. We repeat this process until we have less than 3 instances for each candidate set or scores in candidate set are below the threshold.
B Other Implementation Details
For finetuning BART-large (Lewis et al., 2020) on Wizard of Wikipedia (Dinan et al., 2019b) dataset, we use the ParlAI framework $^{4}$  (Miller et al., 2017) with default hyperparameters. We use RoBERTLarge (Liu et al., 2019) from HuggingFace's Transformers $^{5}$  (Wolf et al., 2019) to implement bidirectional NLI, and named entity recognition module from Stanza $^{6}$  (Qi et al., 2020) to extract named entities from generated claims and claims in FEVER. We use official code from the authors to implement KGAT and BERT evidence selector $^{7}$  (Liu et al., 2020), CorefBERT $^{8}$  (Ye et al., 2020), DPR $^{9}$
4https://parl.ai/ 5https://huggingface.co/transformers/ 6https://github.com/stanfordlp/stanza 7https://github.com/thunlp/KernelGAT 8https://github.com/thunlp/CorefBERT 9https://github.com/facebookresearch/ DPR
(Karpukhin et al., 2020), and WikiAPI document retriever $^{10}$  (Hanselowski et al., 2018). We finetune CorefBERT-base for CorefBERT and BERT-base for BERT evidence selector, BERT claim verifier and DPR. We use default hyperparameters for all the experiments.
For DPR, we use preprocessed English Wikipedia dump from FEVER  $1.0^{11}$  as the source documents for retrieval, which contains 25,248,398 evidence sentences from 5,396,106 documents. We use documents from top-10 retrieved evidences as a document retrieval result, which contains 7.2 documents in average.
All the experiments are run on up to 8 NVIDIA Quadro RTX 6000 GPUs.
C Claim Examples
FEVER (REFUTED):
Dave Gibbons has always been unable to write.
For some reason Dave Gibbons has always been unable to write.
FEVER (SUPPORTED):
Phillip Glass has written eleven concertos.
I'd like to suggest Phillip Glass. He has written a total of eleven concertos!
Planet Hollywood Las Vegas is owned by Leonardo DiCaprio.
Oh okay well if you ever come to LV go to the Planet Hollywood building, its owned by Leonardo DiCaprio.
FEVER (NOTENOUGHINFO):
General Motors had only one automotive-component.
That company used to be called General Motors, General Motors had only one automotive-component.
Steve Ditko studied art at the Cartoonist and Illustrators School.
10https://github.com/UKPLab/ fever-2018-team-athene 11https://fever.ai/resources.html
That's cool. I read that Steve Ditko studied at the Cartoonist and Illustrators School.
Arjit Singh goes unmentioned in the Indian media.
I heard Arjit Singh doesn't get much attention in the Indian media.
The Cry of the Owl is based on Patricia Highsmith's eighth novel "Push".
Yep! In fact, the movie Cry of the Owl is based on a Patricia Highsmith book called Push!
Justin Chatwin is an actor.
In case you didn't already know Justin Chatwin is an actor.
Dreamer (2005 film) was directed by Michael Bay only.
It is true! There was even a Michael Bay film called Dreamer released in 2005.
Harvard University is a commuter school.
I hear that Harvard is a commuter school.
In 2015, among Americans,  $44\%$  of adults had consumed alcoholic drink in the last month.