SlowGuess/NAACL_2021_OCR huggingface dataset:
given the concatenated gold evidence and claim, a BERT-base model (Devlin et al., 2019) outputs a binary entailment / contradiction label.
For end-to-end label accuracy, we use the same models but test only retrieved (rather than gold) passages. During training we include both the gold and the top two retrieved passages.
5 Experiment Results: Machines Spotting False Claims
This section studies the performance of existing automatic methods on FM2 for both the retrieval of evidence (Section 5.1) and for entailment once the results are retrieved (Section 5.2).
5.1 Retrieval Results
Retrieving evidence for FOOLMETwICE is considerably harder (Table 5); we also include comparable results on FEVER. The documents retrieved by DPR are consistently better than the ones by a TF-IDF system for both of the datasets we tested, which is consistent with other work on dense text retrieval (Guu et al., 2020).
Table 4: Claims and evidence shown to players in the voting phase: the voter must detect which claim is incorrect. Initially, the player only sees the claim—if the player can answer with only that, they get the most points. Automatically found evidence is shown one by one upon the voter's request. Waiting and asking for evidence both decrease the time—and the points—available. Eventually, if time does not run out, the gold evidence selected by the author of the claim is shown.
Dataset	R-Precision	Recall@5	Recall@10
TF-IDF	FEVER	25.3	44.1	53.2
FOOLMETWICE	10.4	21.2	28.3
DPR	FEVER	32.0	50.4	58.7
FOOLMETWICE	25.3	42.6	51.0
Table 5: Results of evidence retrieval baselines on FOOLMETWICE and FEVER.  $R$ -Precision is defined as the precision@k, where k is the number of gold evidence snippets for the claim. FOOLMETWICE is harder for both the retriever systems.
5.2 Entailment Results
This section presents the results of training a BERT (Devlin et al., 2019) model for the entailment task of FOOLMETwICE. Given a claim and the gold evidence, does the evidence support or refute the claim? To compare with FEVER, we discard all not enough evidence examples, because the lack of evidence for this class makes it trivial to classify correctly.
Following Gururangan et al. (2018), we first train a claim-only classifier, which ignores the evidence text. FOOLMETwICE examples are harder to classify without looking at the evidence (Table 6), indicating that the claims contain fewer
Dataset	Claim-Only	EASY	HARD	ALL
FOOLMETWICE	61.9	86.1	66.4	78.1
FEVER	79.1	97.1	79.3	93.3
Table 6: Comparison of dev accuracy between FEVER and FOOLMETwice for different partitions of the data and when using only claims. The partition into EASY and HARD splits is based on the claim-only classifier: the claim-only classifier can solve EASY examples. FOOLMETwice examples thus are comparable to HARD FEVER's difficulty.
Retriever	Top-1	Top-3	Top-5
Oracle	69.3	-	-
TF-IDF	62.3	62.0	61.2
DPR	64.2	63.6	63.9
Table 7: End-to-end label accuracy results of retrieval followed by entailment on FOOLMETwICE. We vary the number of retrieved examples at prediction time. We compare against using the gold evidence as an oracle, which differs from Table 5 in using a single 100 word passage as evidence.
"give away" artifacts compared to FEVER as already suggested by Table 3. We provide additional discussion in Appendix C.
Like the techniques proposed by Clark et al. (2019), the claim-only classifier can also be used on both FOOLMETWICE and FEVER to split the dev sets into "easy" and "hard" partitions: The EASY partition contains all examples correctly classified by a claim-only classifier, and the HARD partition has everything else. The similar accuracy of the FOOLMETWICE dev and HARD FEVER dev partitions further suggests that FOOLMETWICE is comparable to the harder and higher-quality subset of FEVER (Table 6).
We also train an end-to-end verification model that, rather than taking evidence as given, must use noisy passages from a retrieval system (Section 4.1). At train time, we generate multiple training instances for each claim using either the gold evidence or the top two retrieved examples. At prediction time, we average the logit scores of each of the top- $k$  retrieved passages (Table 7). We include a so-called oracle setting for a fair comparison of the improvement margin. This number differs from Table 6 in that it uses a single gold 100 word passage as evidence instead of short sentences, which might introduce noise.
6 Dataset Analysis: Humans Spotting and Writing False Claims
While the previous section focuses on how well automatic methods can detect false claims, this section focuses on human ability. Voters are usually right and were fooled  $20.40\%$  of the time. This section addresses how players are fooled and how this compares to computers.
To provide a better picture of the strategies players use to craft challenging claims, we manually sample fifty instances from the development set that both models and humans answer incorrectly. We focus on these examples because they are the most difficult and are the emphasis of our adversarial technique. Two claims were mislabeled and two more lacked a necessary evidence span. Table 8 shows examples of each of the strategies, which we discuss in more detail in this section.
Temporal Many of the most challenging claims require an inference about time: whether one event happened before another, how long an event happened, or whether an event happened during a period. While many of these are based on years, centuries, or other explicit markers of time, some authors use narrative time. For example, the page for the novel *As I Lay Dying* describes the plot in order, so it's difficult for either a system or a human given sentences (without knowing where they appear in the original page) to know when Addie Bundren dies. This shows some of the limitations of the setup: not only must voters reason across multiple pieces of evidence, this reasoning is only possible if they know the *order* in the underlying evidence. Other markers of time include "the pilot" for the first episode of *The Office*; readers must realize that if Kelly Kapoor was introduced in the episode *Diversity Day*, that implies Mindy Kaling's character did not appear in the pilot.
Reasoning A related, but more general, strategy requires the reader to reason: mathematically, applying definitions, or understanding hyponomy. For example, knowing that the child of your cousin is your second cousin or recognizing that "This mirrors the Disney Parks East regional division consisting of Shanghai Disney Resort, Hong Kong Disneyland and Walt Disney Attractions Japan..." implies that there are more than two Walt Disney resorts outside of the United States.
Name	Ratio	Label	Claim & Gold Evidence
Temporal	26%	R	Claim: The Flavian Amphitheatre, which was mainly used for gladiatorial contests, could hold over 50,000 people, and animal hunts continued until the 10th century.
Evidence: Animal hunts continued until at least 523, when Anicius Maximus celebrated his consulship with some venationes, criticised by King Theodoric for their high cost.
Reasoning	26%	S	Claim: Darius Milhaud was a French composer that had a child that was his second Cousin.
Evidence: In 1925, Milhaud married his cousin, Madeleine (1902–2008), an actress and reciter. In 1930 she gave birth to a son, the painter and sculptor Daniel Milhaud, who was the couple's only child.
Paraphrase	22%	R	Claim: Sister Carrie sold poorly, and was criticized for taking the Lord's title in vain.
Evidence: The book was also criticized for never mentioning the name of God.
Diversion	16%	S	Claim: Following his retirement from the MLB, Prince Hal became a top executive of a real estate company.
Evidence: After his retirement from baseball, Newhouser was away from the sport for 20 years, serving as a bank vice president.
Controversy	8%	S	Claim: Francis Marion fought in the Revolutionary War and was an influence for the protagonist in the movie, The Patriot, where his character highly altered to show him as good natured.
Evidence: Sean Busick ... says that based on the facts, “Marion deserves to be remembered as one of the heroes of the War for Independence.” ... the film’s depiction of Martin “as a family man and hero who single-handedly defeats countless hostile Brits” ... was one of the “egregious oversights” that TIME magazine cited when listing The Patriot as number one ... historically misleading [film]”
Table 8: An ontology of human strategies for creating challenging claims in our dataset, sampled from claims that challenged both humans and computers.
Paraphrase A well-known strategy to confuse entailment systems is to change words so that there are fewer exact matches. Some of these are straightforward: "Titration is used when doctors test how much sugar is in a patient's liquid waste" is almost a direct paraphrase of "glucose in urine may indicate diabetes in a patient". Other paraphrases are more poetic: "Charles Evans Hughes shuffled off this mortal coil in Massachusetts, and then was taken to New York to be submerged in soil" paraphrasing "Hughes died in what is now the Tiffany Cottage of the Wianno Club in Osterville, Massachusetts. He is interred at Woodlawn Cemetery in the Bronx, New York City". These paraphrases are realistic, similar to how humans might restate facts to make them more accessible or more interesting to a reader.
Diversion An interesting strategy to fool the retrieval phase of FEVER systems is to create claims that point to specific text but not the text that refutes or supports the claim. For example, "Following his retirement from the MLB, Prince Hal became a top executive of a company" retrieves information about how Hal Newhouser earned the nickname "Prince Hal" and his later business investments but not his post-baseball career in banking.
Controversy A more fundamental issue with entailment systems is that even trusted sources such as Wikipedia contain contradictory evidence. This is most prominent with interpretations of works of fiction, where there are multiple theories about the same work. A skillfully written claim can retrieve one viewpoint while using an opposing viewpoint as the gold evidence.
For example, one claim strongly took the position that the end of the film Inception was a dream. Voters saw evidence to the contrary and thought the claim was refuted. Because systems focus on the highest scoring retrieved passages (as do the human voters), this lead both humans and computers to overlook the disputed interpretations.
6.1 What was difficult for humans?
The amount of evidence a human needs is a unique metric of how difficult a claim is for humans (although incremental evidence is recommended for question answering systems in Boyd-Graber and Börschinger (2020), to the best of our knowledge it has not been applied to entailment or validation). The claims that most challenge humans typically use diversion (e.g., "The Quiet Man was a song by Bing Crosby about a soldier who lost his voice from a bomb in World War 2"), which is particularly challenging for retrieval systems. Other com
mon strategies for the claims most challenging for humans were paraphrase, which can "hide" the relevant evidence and prevent retrieval, and reasoning, which often requires multiple pieces of evidence to reach a conclusion.
7 Limitations and Conclusion
While this paper seeks to advance the ability of humans and computers to support or refute statements entailed from a static, reliable source, the goal of examining arbitrary statements remains elusive. By construction, we have focused on statements that are incorrect because of factual errors. Other datasets that use human-sourced obfuscations or deception are more nuanced and use framing or shading (Pan and Kosicki, 1993), which models trained on this dataset cannot detect. Our goal is to focus on clear facts that can be recognized by computers, which is already challenging enough.
Further improving verification likely requires creating targeted datasets that focus on specific strategies for creating statements that are refuted by evidence, perhaps selecting different explanations for particular users (Feng and Boyd-Graber, 2019). Likewise, a more complicated task likely requires more nuanced incentives and instructions for authors. However, this dataset provides a foundation to build these richer, more challenging datasets for entailment.
As our work involves human participants, all players provided informed consent and no personally identifiable information (PII) was collected or will be released. The collected data have been vetted for presence of PII as well as offensive language through heuristics and random sampling.
Some participants received fair compensation in the United States in exchange for playing the game, but that compensation was not tied to speed or accuracy to prevent distorting the motivation of players. Intrinsic motivation, such as curiosity, competitiveness, creative drive and fun, rather than extrinsic motivation has been shown to produce higher quality results (Gneezy and Rustichini, 2000).
The released data and the experiments we conducted are in English, therefore we do not claim generalization of our findings across languages. However, we believe that the proposed methods could be applied in other languages using other available corpora as a source of evidence.
First and foremost, we would like to specially thank Connie Tao for her guidance and assistance in managing the project. The project would also have been impossible without the FM2 players. We also would want to thank Thomas Müller, William Cohen, Dipanjan Das, Slav Petrov, Pedro Rodriguez, Massimiliano Ciaramita, and Christian Buck for comments on the drafts and testing the game. We also thank the anonymous reviewers for their time, constructive feedback, useful comments and suggestions about this work. Boyd-Graber is supported by NSF Grant IIS-1822494.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of Empirical Methods in Natural Language Processing.
Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares, and Emily Pitler. 2020. New protocols and negative results for textual entailment data collection. In Proceedings of Empirical Methods in Natural Language Processing.
Jordan Boyd-Graber and Benjamin Borschinger. 2020. What question answering can learn from trivia nerds. In Proceedings of the Association for Computational Linguistics, pages 7422-7435.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the Association for Computational Linguistics.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of Empirical Methods in Natural Language Processing.
Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and Foldit players. 2010. Predicting protein structures with a multiplayer online game. Nature, 466(7307):756-760.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rationale, evaluation and approaches. Journal of Natural Language Engineering, 4.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020. Understanding tables with intermediate pre-training. In Findings of the Association for Computational Linguistics: EMNLP.
Shi Feng and Jordan Boyd-Graber. 2019. What AI can do for me: Evaluating machine learning interpretations in cooperative play. In International Conference on Intelligent User Interfaces.
Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In Proceedings of the Association for Computational Linguistics.
Uri Gneezy and Aldo Rustichini. 2000. Pay enough or don't pay at all. The Quarterly Journal of Economics, 115(3):791-810.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Conference of the North American Chapter of the Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the International Conference of Machine Learning.
L. Hays and M. Hayse. 2017. Chapter 8—Game On! Experiential learning with tabletop games. In Pete McDonnell, editor, The Experiential Library, pages 103-115. Chandos Publishing.
Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. 2014. Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the World Wide Web Conference. Association for Computing Machinery.
Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP, pages 3441-3460, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of Empirical Methods in Natural Language Processing.
Firas Khatib, Frank DiMaio, Seth Cooper, Maciej Kazmierczyk, Miroslaw Gilski, Szymon Krzywda, Helena Zabranska, Iva Pichova, James Thompson, Zoran Popovic, Mariusz Jaskolski, David Baker, Foldit Contenders Group, and Foldit Void Crusurers Group. 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18(10):1175-1177.
Stacey Kuznetsov. 2006. Motivations of contributors to Wikipedia. SIGCAS Computers and Society, 36(2):1-8.
Qiang Liu, Jian Peng, and Alexander T Ihler. 2012. Variational inference for crowdsourcing. In Proceedings of Advances in Neural Information Processing Systems.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the Association for Computational Linguistics.
Jakob Nielsen. 1989. Usability engineering at a discount. In International Conference on Human Factors in Computing Systems.
Zhongdang Pan and Gerald M. Kosicki. 1993. Framing analysis: An approach to news discourse. Political Communication, 10(1):55-75.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the Association for Computational Linguistics.
Fabio Petroni, Aleksandra Piktus, Aleksandra Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Roktaschel, and Sebastian Riedel. 2021. KILT: A benchmark for knowledge intensive language tasks. In Conference of the North American Chapter of the Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
T. Rid. 2020. Active Measures: The Secret History of Disinformation and Political Warfare. Profile.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of Empirical Methods in Natural Language Processing.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Evaluating adversarial attacks against multiple fact verification systems. In Proceedings of Empirical Methods in Natural Language Processing.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal, editors. 2018. Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER). Brussels, Belgium.
Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Language Resources and Evaluation Conference.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Towards debiasing NLU models from unknown biases. In Proceedings of Empirical Methods in Natural Language Processing.
Clara Vania, Ruijie Chen, and Samuel R. Bowman. 2020. Asking Crowdworkers to Write Entailment Examples: The Best of Bad options. In Proceedings of the Asia-Pacific Chapter of the Association for Computational Linguistics.
Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In International Conference on Human Factors in Computing Systems.
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering. Transactions of the Association for Computational Linguistics, 7:387-401.
Susan C. Wilkinson, Will Reader, and Stephen J. Payne. 2012. Adaptive browsing: Sensitivity to time pressure and task difficulty. International Journal of Human-Computer Studies, 70(1):14-25.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In *Conference of the North American Chapter of the Association for Computational Linguistics*.
Heng-Li Yang and Cheng-Yu Lai. 2010. Motivations of Wikipedia content contributors. Computers in Human Behavior, 26(6):1377 - 1383. Online Interactivity: Role of Technology in Behavior Change.
Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In Proceedings of the Association for Computational Linguistics.
In this section we provide details on the hyperparameters used and the experimental setup. All BERT models described are of base size (12 layers, 16 attention heads, 768 hidden dimension), and contain 110 million parameters.
The training is done for 10 epochs, a learning rate of  $10^{-5}$ . We use a batch size of 32 and a learning rate of 512. On a single Cloud TPU v2 the model can process one batch 180ms, and a full epoch in around one minute. For all the reported results we take the median over 3 random seeds.
B Game Interface
In this section we include screenshots of the three main screens of the game. Figure 2 shows menu interface that allows players to choose topics according to their interests, we include many Wikipedia categories to ensure a diverse set of options. Figure 3 has example of the voting game, the simplest and fastest way to engage with the game and understand how to be a good author as well. Finally, figure 4 shows the authoring user interface, that displays the retrieved and selected gold evidence as the user types. Matching tokens in the text and the retrieved evidence are highlighted.
C Local Mutual Information
Tables 9, 10 list the top-10 predictive bigrams for the REFUTES label using Schuster et al. (2019)'s method of computing Local Mutual Information (LMI), defined for a bigram  $b$  and label  $l$  as:
$$
\operatorname {L M I} (b, l) = p (b, l) \cdot \log \left(\frac {p (l \mid b)}{p (l)}\right)
$$
where the probabilities use the empirical counts.