SlowGuess/NAACL_2021_OCR huggingface dataset:
Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Borschinger Google Research
{eisenjulian, bdhingra, jbulian, bboerschinger} @google.com
Jordan Boyd-Graber*
CS, iSchool, UMIACS, LSC University of Maryland
We release FOOLMETWICE (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using "shortcuts" compared to other entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players "pay" to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and game code.
1 Introducing a Game of Challenging Claims
Given a statement—and a large collection of textual knowledge—how do you find evidence that shows a reader that the statement is true or false? This problem takes on multiple forms in the natural language processing (NLP) community. Given only a single statement and a single sentence, this decision process is called recognizing textual entailment (Dagan et al., 2010, RTE) or natural language inference (Bowman et al., 2015; Williams et al., 2018, NLI). Given a single statement and a vast pool of possible evidence (e.g., all of Wikipedia), this problem is called verification (Thorne et al., 2018; Jiang et al., 2020).
Stage 1: Players write claims to fool others
Stage 5: Automatically build dataset
Keep highest quality claims with their selected evidence
Figure 1: Overview of the data generation pipeline. In stages 1 to 4, players write challenging claims either entailed or refuted by evidence from Wikipedia (Section 3.1). They are then tasked to spot the refuted claim among a group (Section 3.2). The claims and evidence are available for download.
We review existing resources for the latter task in Section 2 and how they have spawned a vibrant subcommunity around related tasks. However, these datasets fail to challenge modern NLP models such as BERT (Devlin et al., 2019) or T5 (Raffel et al., 2020) that achieve "super-human performance" despite also exhibiting "annotation artifacts" that hurt their generalization potential (Gururangan et al., 2018; Tsuchiya, 2018). Our goal is twofold: (1) to build a new, challenging dataset (statistics for FOOLMETwICE in Table 1) that tests models' abl
*Work completed while a Visiting Research Scientist at Google.
  <https://github.com/google-research/ fool-me-twice>
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 352-365 June 6-11, 2021. ©2021 Association for Computational Linguistics
Claims	Entailed
Proportion	Pages	Avg. #
Tokens
Claim	Evidence
Train	10,419	49.2%	1,811	15	30
Dev	1,169	51.0%	209	15	31
Test	1,380	49.4%	234	15	31
Total	12,968	49.4%	2,254	15	30
Table 1: Statistics of the FOOLMETwICE dataset. The train/dev/test split is based on disjoint Wikipedia pages. The number of tokens is an average value computed with a white-space tokenizer. Our dataset is balanced between entailed (true) and not entailed (false) claims.
ity to retrieve evidence and verify claims and (2) to show that engineering the incentive structure of data collection experiments can produce more accurate and realistic outcomes.
This dataset lends itself to automatic training and it characterizes what factual errors humans can most easily detect and which are most likely to fool them (Section 3.2). This is analogous to the creation of unsupported or refuted claims in the wild, which are not random, but evolve as part of an information arms race (Rid, 2020). Unlike previous datasets that rely on crowd-sourcing, we develop an online game to create a platform where motivated authors can create plausible sounding "facts" that other users must debunk.
Not only does this create more realistic claims—the best must withstand human scrutiny—it also creates a way to better evaluate the evidence that support or refute claims. As we surface the evidence, humans use that evidence to decide which claims are true or false; these signals can further improve our systems (Figure 1). We apply baseline models for retrieval and classification to our dataset (Section 4) and examine how their ability to detect wrong statements differs from humans' (Section 5).
Entailment is a key task in natural language understanding. Dagan et al. (2010) describe it as an AI-complete task: solve it, and you can solve all of artificial intelligence. Typically, entailment is presented as: given a premise ("Brooklyn is the most populous of New York City's boroughs"), decide whether a hypothesis ("Manhattan has more residents than Brooklyn") is entailed—supported—by the premise. Even simple examples show the promise (and complexity) of this task. To recognize that this hypothesis is contradicted, a model
must: know that Manhattan is a borough of New York, recognize that “X is the most populous borough” entails “X has more residents than any other borough”, and correctly combine this knowledge to recognize the contradiction.
2.1 Entailment and Retrieval Datasets
Despite the promise of entailment, it has not been a silver bullet for the NLP community to solve artificial intelligence. One possible explanation is highlighted by a line of work that shows existing entailment datasets have artifacts. Poliak et al. (2018) show entailment can often be solved by looking only at the hypothesis, while Feng et al. (2019) show that artifacts can infect the premise as well. This is especially common in the biggest datasets for NLI such as SNLI and MNLI (Gururangan et al., 2018). While there are algorithmic solutions to addressing these issues (Utama et al., 2020), many have turned to building better datasets.
Both Bowman et al. (2020) and Vania et al. (2020) propose alternative methods for collecting entailment pairs from crowdworkers and measure success via improvements in other general tasks via transfer learning. While the proposed methods prove to be ineffective for that goal, we view NLI is as an important end task in itself (e.g., for misinformation, QA, dialogue, generation evaluation). Hence, we argue that constructing challenging entailment datasets is useful beyond just transfer learning.
Like this paper, Nie et al. (2020) focus on adversarial entailment, but their authors only see a single piece of evidence. We expand this human-in-the-loop adversarial setting to include the essential retrieval component of fact verification. Thus, authors have more strategies on hand; in addition to creating challenging examples through paraphrasing, they can make it difficult to find relevant information in the first place or distract with related—but distinct—information.
This is exactly the setting of a recent shared task, FEVER (Thorne et al., 2018, Fact Extraction and VERIFICATION), which creates a more general entailment setting: given a claim, find relevant evidence from Wikipedia, and determine whether the evidence has enough information to either support or refute the claim. This generalizes the entailment problem to a large, broadly accepted set of premises (all sentences in Wikipedia) and adds an additional retrieval step to find relevant evidence.
Woody Allen is a person.
The Shining was directed.
François de Belleforest wrote.
Not Enough Info
Lisa Kudrow was in a car.
Tipper Gore was curated to Al Gore.
International Relations includes animals.
Tipper Gore was created in 1048.
Alpha House is inspired by nobody.
Toy Story is incapable of being a film.
Table 2: Examples from FEVER, which separates entailment examples into three categories. The crowdworkers who authored the examples often edit the first line of the Wikipedia article but not in ways that sound like a plausible hypothesis. We develop a game to build more complex, challenging examples.
FEVER has obvious connections to problems in education, journalism, and information science. Thus, it has caught the attention of a subcommunity focused on building systems for FEVER shared tasks. Despite this excitement, Schuster et al. (2019) show that FEVER has many of the same issues as entailment datasets. FEVER has broad or nonsensical claims (Table 2) and many of the claims are generated from the very first line of source Wikipedia documents. This is not just an artifact of crowd-sourcing; a more fundamental problem is that there is no clear definition of what makes a good FEVER example. To date, adversarial FEVER example generation uses automatic rules to increase their difficulty (Thorne et al., 2019). To address these identified weaknesses, Sections 3.1 and 3.2 define a game where the claim writers have a clear objective of "fooling" other human players.
2.2 Gamification for Data Collection
Creating datasets through a fun interactive design is often called gamification. Ipeirotis and Gabrilovich (2014) focus on multiple choice question answering in technical domains such as medicine and rely on redundancy and calibration questions to generate new knowledge. The ESP game (von Ahn and Dabbish, 2004) asks users to write labels for an image that agree as much as possible with other players' labels.
Another well-known example is protein folding (Cooper et al., 2010), an online game² that
Dataset split	Top Bigrams by LMI (highest predictive power first)
FEVER Train	is only, did not, not a, was not, incapable of, only a
FEVER Dev	is only, only a, incapable of, is incapable, was only, did not
FM2 Train	the second, is a, was a, was the, is the, of his
FM2 Dev	by a, on the, innocent iii, statue of, for his, pope innocen
Table 3: Top 6 bigrams with the highest LMI (Schuster et al., 2019) for REFUTES in each dataset and each split. Overlapping bigrams are bolded. Compared to FEVER, FOOLMETWICE contains fewer bigrams that "give away" the label on both the train and dev set.
tasks players to twist and bend protein structures, often besting computer algorithms and driving biological innovations (Khatib et al., 2011).
Crucially, these games are either individual or cooperative; in contrast, FOOLMETWICE exploits the adversarial nature of players fooling each other. FOOLMETWICE most closely resembles Balder-dash, a board game where players guess which definition of a word is legitimate that is used in information literacy courses (Hays and Hayse, 2017).
In all cases, the intrinsic motivation driven by these games can lead to better outcomes and fewer attempts to "game" the system (Kuznetsov, 2006; Yang and Lai, 2010). Thus our approach constitutes a viable alternative to traditional isolated labelling tasks in crowd-sourcing platforms, where tying payment to completing tasks sometimes hurts final results (Gneezy and Rustichini, 2000).
3 FOOLMETWICE Game Mechanics
This section outlines the two phases of the game: authoring claims (Section 3.1) and voting on those claims (Section 3.2). While these sections present the game in its final form, this is the reflection of an iterative process.
We first began with a paper version (Nielsen, 1989) of the game, which showed that a time constraint made the game feel more fun and encouraged people to not read individual pieces of evidence too intently. Without the timer, people tried to look for tiny clues in text that probably were not there (Wilkinson et al., 2012). We then moved onto a version of the game presented via slides where we experimented with design choices such as the number of claims players distinguish between, and the number of evidence sentences they see while doing that. Examples of the final web interface are shown in Appendix B.
$^{2}$ https://fold.it
3.1 Crafting Challenging Claims
Our goal is to create a computer game that produces human-authored, interesting, challenging claims paired with evidence that either supports or refutes each claim. One prerequisite for this is that claims avoid high lexical overlap with the knowledge corpus. We thus need to encourage authors to craft claims that cannot be trivially matched to evidence. While this approach has been used for question answering (Wallace et al., 2019; Bartolo et al., 2020), which has a similar retrieval step, to our knowledge it has not been applied to entailment or FEVER.
We recruit users employed at Google, all proficient in English, to play-test the game. At the beginning of each round, we ask each user to generate a true or false statement. We randomly choose a Wikipedia page as a knowledge source and ask them to highlight one or two evidence spans that support (or refute) their claim. They are instructed to write statements that would likely fool other players trying to determine the claim's veracity quickly and/or without looking at the evidence that support the claim. The reward system defined in the next section is built to be aligned with this objective.
To help authors write hard claims, not entirely similar to the evidence, we show the user what evidence a TF-IDF retrieval system would select from the source and highlight the words that help IR systems select evidence. This implicitly encourages them to craft the claims in a manner such that overlap with the evidence is low (Section 3.2). We include screenshots of the user interface and more details about our design choices in the appendix. Because the players see evidence selected by our retrieval systems, difficult claims for players are also challenging for computers. See Table 3 for a comparison on highly predictive bigrams between FEVER and FOOLMETwice (details about how these are computed are in the appendix).
3.2 Spotting the Incorrect Statement
In the game's second phase, players select the incorrect statement from claims written by other players (Table 4). To separate these two phases of the game, we refer to players in this phase of the game as voters. If a voter can correctly answer quickly (e.g., through their own world knowledge or artifacts), they get up to 120 points, the maximum possible. The author and voter split the points: any points the
voter leaves "on the table" go to the author. Challenging claims reward the author with more points but easy ones let the voter increase their total.
We do not want to keep claims that are easy to identify as true or false. If the average player can tell through artifacts or common sense that a claim will not be supported, it is uninteresting as an entailment example. For example, if someone sees the claim "Tipper Gore was born in 1048" and remembers that Al Gore was the vice president of the United States in the twentieth century, they can identify that this claim is false. We also want claims that require the voters to carefully read evidence from Wikipedia (Table 4). Voters can ask for hints provided by our evidence selection system (Section 4.1). For each piece of evidence shown, the number of points available to the voters decreases, and points decrease as time progresses as well.
All possible outcomes provide useful information: correct and incorrect choices, with and without evidence. As mentioned before, if voters spot the wrong statement unaided, the claim has underlying issues. When a voter can spot the wrong claim with the help of a particular piece of evidence, then this is a clue that the evidence (and the mechanism that selected it) is useful.
This allows us to specifically optimize for evidence that helps players better answer questions. When voters go from confused to confident about the correct answer, that is a signal that the evidence was effective. When voters select an incorrect answer, that is a signal that the evidence was not effective (or, indeed, misleading).
When voters need more time and evidence and are almost fooled (i.e., nearly think a true statement is incorrect), this is a sign that the statement is challenging for the human-computer team seeking to verify entailment. The statement must be convincingly written, consistent with voter's world knowledge, and also consistent with the evidence players see. Our game setting helps create conditions where these "tricky" examples can be crafted.
We use two heuristics to ensure quality claims. First, we search for "easy" examples that were consistently solved without inspecting the evidence - however, we were not able to find any. Next, we search for examples which are "too difficult" by computing a maximum a posteriori estimate of the Bernoulli distribution of correct and incorrect votes for each claim. The prior distribution matches the overall accuracy of the dataset (80% of votes are
3Each voting task should take at maximum two minutes, and each point corresponds to a second.
correct) and is equivalent to adding five pseudocounts (one wrong, four correct) for each question. We use this smoothed estimate rather than the maximum likelihood estimate to account for claims lacking votes. The expected value of that posterior given a  $\mathrm{Beta}(4,1)$  prior is (Liu et al., 2012):
$$
\alpha \sim \operatorname {B e t a} (4, 1)
$$
$$
\alpha \mid C _ {i} \sim \operatorname {B e t a} \left(4 + \sum_ {i} C _ {i}, 1 + \sum_ {i} \left(1 - C _ {i}\right)\right),
$$
where  $i$  sums over the votes, and  $C_i$  is one if the vote was correct and zero otherwise. We analyze all twenty-five claims below a 0.5 threshold and identified three incorrect examples which we subsequently removed.
3.3 Incentive Structure
Players earn points in two ways: either spotting incorrect claims by voting as early as possible or authoring challenging claims. They alternate between the two roles in every game session. These two rewards are in opposition to each other.
Because the goal of the voters is to find the claim that is incorrect, claim authors (of either entailed or refuted claims) only get points when voters are not fooled and when the voters need evidence. The total points are split between the voter and authors when the voter correctly guesses, making this a zero-sum game. As a voter requests evidence or takes more time, a larger fraction of the total points will go to authors. Thus, authors are encouraged to write difficult claims; voters are encouraged to select claims correctly.
When a voter guesses incorrectly, they get no points, to ensure the examples are valid. While incorrect guesses can happen for impossible claims, writing claims that are merely difficult is a better strategy since easy claims that may be spotted quickly are awarded no points.<sup>4</sup>
In addition to humans voting on claims, we also ask users which of the two claims they "like" more, independent of voters' accuracy. People like true claims (0.39) more than false claims (0.35,  $t = 2.53$ ,  $p = 0.01$ ), except for claims about science and technology, where people prefer false claims (0.46) more than true claims (0.32,  $t = -2.50$ ,  $p = 0.02$ ). Authors get points when voters like their claims; this additional incentive encourages authors to create interesting and surprising examples.
4 Methods: Subtasks and Models
Each of the instances in FoolMETwice is a tuple  $(c, e, l)$ : a natural language claim  $c$ , evidence  $e$  from a knowledge corpus  $\mathcal{K}$  (in our case Wikipedia), and a binary label  $l$  (entailment / contradiction).<sup>5</sup> From this we define two sub-tasks, following Thorne et al. (2018). The first sub-task, retrieval, requires systems to select candidate evidence from  $\mathcal{K}$  (including, perhaps, the gold evidence  $e$ ). The second sub-task is entailment, where systems given claim  $c$  and the gold evidence  $e$  need to make a final prediction for the label  $l$ . We also consider an end-to-end setting. Instead of the gold evidence, systems only have access to the retrieved evidence  $\hat{e}$  at test time. In the rest of this section we define baseline models for each of the sub-tasks.
Our setting resembles the retrieval setting in the KILT benchmark (Petroni et al., 2021), but the results are evaluated at the evidence level as opposed to the page level, to represent a more realistic use case. The evidence corpus can be found online and consists of twenty-two million text passages, each having a length of a hundred words, from five million pages of the English Wikipedia image from August 2019. We align gold FOOLMETwICE evidence to this knowledge source by selecting the passage with highest overlap with each evidence sentence, according to the modified  $n$ -gram precision component of the BLEU (Papineni et al., 2002). We remove 1598 examples<sup>7</sup> where the precision was less than 0.5.
We evaluate two baselines. The first one follows Chen et al. (2017) and uses a TF-IDF retrieval model with unigrams and bigrams and  $2^{20}$  hash buckets. The title of page is added to the passage content for additional context. The second baseline uses Dense Passage Retrieval (Karpukhin et al., 2020, DPR), using the same fixed pre-trained passage embeddings and query encoder as the ones used in Petroni et al. (2021).
4.2 Entailment
For the second component of the task, we follow state-of-the-art entailment models (Zhou et al., 2019; Liu et al., 2020; Eisenschlos et al., 2020):
${}^{5}$  Unlike FEVER,we do not allow authors to write claims that lack "enough information".
<sup>6</sup>http://github.com/facebookresearch/KILT/
This happens because FOOLMETWICE was constructed from a more recent version of Wikipedia than KILT.
4We also allow players to flag obscene, incorrect, or otherwise problematic claims.
TopicClaim	IcelandAfter ending its personal union with Denmark, Iceland was last invaded by the United Kingdom in Operation Fork.	Elizabeth IIAfter Elizabeth II's accession, she changed her house's name from "House of Saxe-Coburg and Gotha" to the "House of Windsor", rejecting the "House of Edinburgh".	Time Left 3:00
Retrieved Evidence(Revealed Incrementally)	The Danish-Icelandic Act of Union, an agreement with Denmark signed on 1 December 1918 and valid for 25 years, recognised Iceland as a fully sovereign and independent state in a personal union with Denmark.	Philip suggested House of Edinburgh, after his ducal title.	-0:30
Possession of Iceland passed from the Kingdom of Norway (872–1397) to the Kalmar Union in 1415, when the kingdoms of Norway, Denmark and Sweden were united.	The Duke's uncle, Lord Mountbatten, advocated the name House of Mountbatten.	-0:30
A month later, British armed forces conducted Operation Fork, the invasion and occupation of the country, violating Icelandic neutrality.	The Duke complained, "I am the only man in the country not allowed to give his name to his own children."	-0:30
Beginning on 20 May 1944, Icelanders voted in a four-day plebisite on whether to terminate the personal union with Denmark, abolish the monarchy, and establish a republic.	With Elizabeth's accession, it seemed probable the royal house would bear the Duke of Edinburgh's name, in line with the custom of a wife taking her husband's surname on marriage.	-0:30
· ·	·
Gold Evidence	After the German occupation of Denmark on 9 April 1940, the Althing replaced the King with a regent and declared that the Icelandic government would take control of its own defence and foreign affairs.A month later, British armed forces conducted Operation Fork, the invasion and occupation of the country, violating Icelandic neutrality.	With Elizabeth's accession, it seemed probable the royal house would bear the Duke of Edinburgh's name, in line with the custom of a wife taking her husband's surname on marriage.The British Prime Minister, Winston Churchill, and Eliza-beth's grandmother, Queen Mary, favoured the retention of the House of Windsor, and so on 9 April 1952 Elizabeth issued a declaration that Windsor would continue to be the name of the royal house.	-0:30 -0:30
aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn aepn a