SlowGuess/NAACL_2021_OCR huggingface dataset:
Consistent with the much lower claim-only classifier (see Table 6), FOOLMETWICE contains no "give away" bigrams that are highly predictive of the label on both the training and development data whereas, as previously reported by Schuster et al. (2019), FEVER has many. Moreover, the "quality" of predictive bigrams for FEVER suggests that annotators (subconsciously) used specific strategies when writing REFUTES examples ("is only", "did not", "is incapable"), but no such patterns can be seen for FOOLMETWICE.
Bigram	Train LMI ×10-5	Dev LMI ×10-5
is only	622	938
did not	859	528
not a	775	481
was not	729	-
incapable of	721	710
only a	455	717
is incapable	474	551
was only	-	536
has only	447	-
yet to	420	384
of being	-	385
Table 9: Top-10 highest LMI bigrams for REFUTES label in FEVER for both Train and Dev. Note the large overlap of label-predictive bigram artefacts.
Bigram	Train
LMI ×10-5	Dev
LMI ×10-5
by a	-	562
mad,	-	502
, mad	-	502
on the	-	473
innocent iii	-	467
statue of	-	426
for his	-	407
pope innocent	-	407
mary,	-	365
queen of	-	365
the second	338	-
is a	312	-
was a	307	-
was the	306	-
is the	233	-
of his	200	-
has never	189	-
was born	177	-
written by	165	-
about a	162	-
Table 10: Top-10 LMI bigrams for REFUTES label in FOOLMETWICE for both Train and Dev. Note both the absence of "give-away" bigram overlap between Train and Dev; and the more "random" quality of predictive bigrams compared to those for FEVER in Table 9
Choose your next challenge
LEARN HOW TO PLAY
Watch this quick video tutorial to learn how to fool and not be fooled.
FIND THE FIB
Learn about and find which statement is false.
PLAY
WRITE A FALSE CLAIM
GO
Read about Gladiator and come up with a sentence.
GO
FIND THE FIB
GO
GO
WRITE A FALSE CLAIM
GO
Read about Michael Faraday and come up with a sentence.
Figure 2: Menu where players select between authoring or voting on claims. A diverse set of categories is presented to engage people according to their interests.
Figure 3: The voting interface shows one entailed and refuted claim. The player has two decide which one is the refuted one before time runs out. Getting clues consumes 30 seconds in the timer.
Fool Me Twice Play - Leaderboard
LEVEL 3  $\cdot$  DEBUT AUTHOR  $\cdot$  9965 POINTS
Figure 4: In the writing screen, players are asked to write either an entailed or refuted evidence given the evidence on the right hand side. As they write, a retrieval system picks the most relevant evidence. They can mark the gold evidence that supports or contradicts the claim, and are instructed to write in such a way that the gold evidence is not at the top of the retrieved list.