SlowGuess/ACL_2020_OCR huggingface dataset:
Jordan Boyd-Graber\*,†  
iSchool, CS, UMIACS, LSC  
University of Maryland  
jbg@umiacs.umd.edu
Benjamin Borschinger†
$\dagger$  Google Research Zurich {jbg, bboerschinger}@google.com
Question answering (QA) is not just building systems; this NLP subfield also creates and curates challenging question datasets that reveal the best systems. We argue that QA datasets—and QA leaderboards—closely resemble trivia tournaments: the questions agents—humans or machines—answer reveals a "winner". However, the research community has ignored the lessons from decades of the trivia community creating vibrant, fair, and effective QA competitions. After detailing problems with existing QA datasets, we outline several lessons that transfer to QA research: removing ambiguity, identifying better QA agents, and adjudicating disputes.
This paper takes an unconventional analysis to answer "where we've been and where we're going" in question answering (QA). Instead of approaching the question only as computer scientists, we apply the best practices of trivia tournaments to QA datasets.
The QA community is obsessed with evaluation. Schools, companies, and newspapers hail new SOTAs and topping leaderboards, giving rise to troubling claims (Lipton and Steinhardt, 2019) that an "AI model tops humans" (Najberg, 2018) because it 'won' some leaderboard, putting "millions of jobs at risk" (Cuthbertson, 2018). But what is a leaderboard? A leaderboard is a statistic about QA accuracy that induces a ranking over participants.
Newsflash: this is the same as a trivia tournament. The trivia community has been doing this for decades (Jennings, 2006); Section 2 details this overlap between the qualities of a first-class QA dataset (and its requisite leaderboard). The experts running these tournaments are imperfect, but they've learned from their past mistakes (see Appendix A for a brief historical perspective) and cre
ated a community that reliably identifies those best at question answering. Beyond the format of the competition, trivia norms ensure individual questions are clear, unambiguous, and reward knowledge (Section 3).
We are not saying that academic QA should surrender to trivia questions or the community—far from it! The trivia community does not understand the real world information seeking needs of users or what questions challenge computers. However, they have well-tested protocols to declare that someone is better at answering questions than another. This collection of tradecraft and principles can nonetheless help the QA community.
Beyond these general concepts that QA can learn from, Section 4 reviews how the "gold standard" of trivia formats, Quizbowl can improve traditional QA. We then briefly discuss how research that uses fun, fair, and good trivia questions can benefit from the expertise, pedantry, and passion of the trivia community (Section 5).
2 Surprise, this is a Trivia Tournament!
"My research isn't a silly trivia tournament," you say. That may be, but let us first tell you a little about what running a tournament is like, and perhaps you might see similarities.
First, the questions. Either you write them yourself or you pay someone to write questions by a particular date (sometimes people on the Internet).
Then, you advertise. You talk about your questions: who is writing them, what subjects are covered, and why people should try to answer them.
Next, you have the tournament. You keep your questions secure until test time, collect answers from all participants, and declare a winner. Afterward, people use the questions to train for future tournaments.
These have natural analogs to crowd sourcing
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7422-7435 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
questions, writing the paper, advertising, and running a leaderboard. Trivia nerds cannot help you form hypotheses or write your paper, but they can tell you how to run a fun, well-calibrated, and discriminative tournament.
Such tournaments are designed to effectively find a winner, which matches the scientific goal of knowing which model best answers questions. Our goal is not to encourage the QA community to adopt the quirks and gimmicks of trivia games. Instead, it's to encourage experiments and datasets that consistently and efficiently find the systems that best answer questions.
2.1 Are we having fun?
Many authors use crowdworkers to establish human accuracy (Rajpurkar et al., 2016; Choi et al., 2018). However, they are not the only humans who should answer a dataset's questions. So should the dataset's creators.
In the trivia world, this is called a play test: get in the shoes of someone answering the questions. If you find them boring, repetitive, or uninteresting, so will crowdworkers. If you can find shortcuts to answer questions (Rondeau and Hazen, 2018; Kaushik and Lipton, 2018), so will a computer.
Concretely, Weissenborn et al. (2017) catalog artifacts in SQuAD (Rajpurkar et al., 2018), the most popular QA leaderboard. If you see a list like "Along with Canada and the United Kingdom, what country...", you can ignore the rest of the question and just type Ctrl+F (Yuan et al., 2019; Russell, 2020) to find the third country—Australia in this case—that appears with "Canada and the UK". Other times, a SQuAD playtest would reveal frustrating questions that are i) answerable given the information but not with a direct span, ii) answerable only given facts beyond the given paragraph, iii) unintentionally embedded in a discourse, resulting in arbitrary correct answers, iv) or non-questions.
SearchQA (Dunn et al., 2017), derived from Jeopardy!, asks "An article that he wrote about his riverboat days was eventually expanded into *Life on the Mississippi.* The apprentice and newspaper writer who wrote the article is named Samuel Langhorne Clemens; however, the reference answer is his later pen name, Mark Twain. Most QA evaluation metrics would count Samuel Clemens as incorrect. In a real game of Jeopardy!, this would not be an issue (Section 3.1).
Of course, fun is relative, and any dataset is bound to contain errors. However, playtesting is an easy way to find systematic problems: unfair, unfun playtests make for ineffective leaderboards. Eating your own dog food can help diagnose artifacts, scoring issues, or other shortcomings early in the process.
The deeper issues when creating a QA task are: i) have you designed a task that is internally consistent, ii) supported by a scoring metric that matches your goals, iii) using gold annotations that reward those who do the task well? Imagine someone who loves answering the questions your task poses: would they have fun on your task? This is the foundation of Gamification (von Ahn, 2006), which can create quality data from users motivated by fun rather than pay. Even if you pay crowdworkers, unfun questions may undermine your dataset goals.
2.2 Am I measuring what I care about?
Answering questions requires multiple skills: identifying answer mentions (Hermann et al., 2015), naming the answer (Yih et al., 2015), abstaining when necessary (Rajpurkar et al., 2018), and justifying an answer (Thorne et al., 2018). In QA, the emphasis on SOTA and leaderboards has focused attention on single automatically computable metrics—systems tend to be compared by their 'SQuAD score' or their 'NQ score', as if this were all there is to say about their relative capabilities. Like QA leaderboards, trivia tournaments need to decide on a single winner, but they explicitly recognize that there are more interesting comparisons.
A tournament may recognize different background/resources—high school, small school, undergraduates (Hentzel, 2018). Similarly, more practical leaderboards would reflect training time or resource requirements (see Dodge et al., 2019) including ‘constrained’ or ‘unconstrained’ training (Bojar et al., 2014). Tournaments also give specific awards (e.g., highest score without
<sup>1</sup>A source paragraph says "In [Commonwealth countries]... the term is generally restricted to... Private education in North America covers the whole gamut..."; thus, "What is the term private school restricted to in the US?" has the information needed but not as a span.
${}^{2}$  A source paragraph says "Sculptors [in the collection include] Nicholas Stone, Caius Gabriel Cibber, [...] Thomas Brock, Alfred Gilbert, [...] and Eric Gill", i.e., a list of names; thus, the question "Which British sculptor whose work includes the Queen Victoria memorial in front of Buckingham Palace is included in the V&A collection?" should be unanswerable in SQUAD.
${}^{3}$  A question "Who else did Luther use violent rhetoric towards?" has the gold answer "writings condemning the Jews and in diatribes against Turks".
incorrect answers). Again, there are obvious leaderboard analogs that would go beyond a single number. In SQuAD 2.0 (Rajpurkar et al., 2018), abstaining contributes the same to the overall  $F_{1}$  as a fully correct answer, obscuring whether a system is more precise or an effective abstainer. If the task recognizes both abilities as important, reporting a single score risks implicitly prioritizing one balance of the two.
2.3 Do my questions separate the best?
Assume that you have picked a metric (or a set of metrics) that captures what you care about. A leaderboard based on this metric can rack up citations as people chase the top spot. But your leaderboard is only useful if it is discriminative: the best system reliably wins.
There are many ways questions might not be discriminative. If every system gets a question right (e.g., abstain on non-questions like "asdf" or correctly answer "What is the capital of Poland?"), the dataset does not separate participants. Similarly, if every system flubs "what is the oldest north-facing kosher restaurant", it is not discriminative. Sugawara et al. (2018) call these questions "easy" and "hard"; we instead argue for a three-way distinction.
In between easy questions (system answers correctly with probability 1.0) and hard (probability 0.0), questions with probabilities nearer to 0.5 are more interesting. Taking a cue from Vygotsky's proximal development theory of human learning (Chaiklin, 2003), these discriminative questions—rather than the easy or the hard ones—should most improve QA systems. These Goldilocks<sup>4</sup> questions (not random noise) decide who tops the leaderboard. Unfortunately, existing datasets have many easy questions. Sugawara et al. (2020) find that ablations like shuffling word order (Feng et al., 2018), shuffling sentences, or only offering the most similar sentence do not impair systems. Newer datasets such as DROP (Dua et al., 2019) and HellaSwag (Zellers et al., 2019) are harder for today's systems; because Goldilocks is a moving target, we propose annual evaluations in Section 5.
2.4 Why so few Goldilocks questions?
This is a common problem in trivia tournaments, particularly pub quizzes (Diamond, 2009), where
Figure 1: Two datasets with 0.16 annotation error: the top, however, better discriminates QA ability. In the good dataset (top), most questions are challenging but not impossible. In the bad dataset (bottom), there are more trivial or impossible questions and annotation error is concentrated on the challenging, discriminative questions. Thus, a smaller fraction of questions decide who sits atop the leaderboard, requiring a larger test set.
challenging questions can scare off patrons. Many quiz masters prefer popularity with players and thus write easier questions.
Sometimes there are fewer Goldilocks questions not by choice, but by chance: a dataset becomes less discriminative through annotation error. All datasets have some annotation error; if this annotation error is concentrated on the Goldilocks questions, the dataset will be less useful. As we write this in 2020, humans and computers sometimes struggle on the same questions.
Figure 1 shows two datasets of the same size with the same annotation error. However, they have different difficulty distributions and correlation of annotation error and difficulty. The dataset that has more discriminative questions and consistent annotator error has fewer questions that do not discriminate the winner of the leaderboard. We call this the effective dataset proportion  $\rho$  (higher is better). Figure 2 shows the test set size required to reliably discriminate systems for different  $\rho$ , based on a simulation (Appendix B).
At this point, you may despair about how big a dataset you need. The same terror besets trivia tournament organizers. Instead of writing more questions, they use pyramidality (Section 4) to make every question count.
3 The Craft of Question Writing
Trivia enthusiasts agree that questions need to be well written (despite other disagreements). Asking "good questions" requires sophisticated pragmatic
<sup>4</sup>In a British folktale first recorded by Robert Southey, the character Goldilocks finds three beds: one too hard, one not hard enough, and one "just right".
Using a more sophisticated simulation approach, the TREC 2002 QA test set (Voorhees, 2003) could not discriminate systems with less than a seven absolute score point difference.
Figure 2: How much test data do you need to discriminate two systems with  $95\%$  confidence? This depends on both the difference in accuracy between the systems ( $x$  axis) and the average accuracy of the systems (closer to  $50\%$  is harder). Test set creators do not have much control over those. They do have control, however, over how many questions are discriminative. If all questions are discriminative (right), you only need 2500 questions, but if three quarters of your questions are too easy, too hard, or have annotation errors (left), you'll need 15000.
reasoning (Hawkins et al., 2015), and pedagogy explicitly acknowledges the complexity of writing effective questions for assessing student performance (Haladyna, 2004, focusing on multiple choice questions).
QA datasets, however, are often collected from the wild or written by untrained crowdworkers. Crowdworkers lack experience in crafting questions and may introduce idiosyncrasies that shortcut machine learning (Geva et al., 2019). Similarly, data collected from the wild such as Natural Questions (Kwiatkowski et al., 2019) or AmazonQA (Gupta et al., 2019) by design have vast variations in quality. In the previous section, we focused on how datasets as a whole should be structured. Now, we focus on how specific questions should be structured to make the dataset as valuable as possible.
3.1 Avoiding ambiguity and assumptions
Ambiguity in questions not only frustrates answerers who resolve the ambiguity 'incorrectly'. Ambiguity also frustrates the goal of using questions to assess knowledge. Thus, the US Department of Transportation explicitly bans ambiguous questions from exams for flight instructors (Flight Standards Service, 2008); and the trivia community has likewise developed rules and norms that prevent ambiguity. While this is true in many contexts, examples are rife in format called Quizbowl (Boyd-Graber et al., 2012), whose very long questions<sup>6</sup> showcase trivia writers' tactics. For example, Quizbowl author Zhu Ying (writing for the 2005 PARFAIT tournament) asks participants to identify a fictional
character while warning against possible confusion [emphasis added]:
He's not Sherlock Holmes, but his address is 221B. He's not the Janitor on Scrubs, but his father is played by R. Lee Ermy. [...] For ten points, name this misanthropic, crippled, Vicodindependent central character of a FOX medical drama.
ANSWER: Gregory House, MD
In contrast, QA datasets often contain ambiguous and under-specified questions. While this sometimes reflects real world complexities such as actual under-specified or ill-formed search queries (Faruqui and Das, 2018; Kwiatkowski et al., 2019), ignoring this ambiguity is problematic. As a concrete example, Natural Questions (Kwiatkowski et al., 2019) answers "what year did the us hockey team win the Olympics" with 1960 and 1980, ignoring the US women's team, which won in 1998 and 2018, and further assuming the query is about ice rather than field hockey (also an Olympic event). Natural Questions associate a page about the United States men's national ice hockey team, arbitrarily removing the ambiguity post hoc. However, this does not resolve the ambiguity, which persists in the original question: information retrieval arbitrarily provides one of many interpretations. True to their name, Natural Questions are often under-specified when users ask a question online.
The problem is neither that such questions exist nor that machine reading QA considers questions given an associated context. The problem is that tasks do not explicitly acknowledge the original ambiguity and gloss over the implicit assumptions in the data. This introduces potential noise and bias (i.e., giving a bonus to systems that make the same assumptions as the dataset) in leaderboard rankings.
<sup>6</sup>Like Jeopardy!, they are not syntactically questions but still are designed to elicit knowledge-based responses; for consistency, we still call them questions.
At best, these will become part of the measurement error of datasets (no dataset is perfect). At worst, they will recapitulate the biases that went into the creation of the datasets. Then, the community will implicitly equate the biases with correctness: you get high scores if you adopt this set of assumptions. These enter into real-world systems, further perpetuating the bias. Playtesting can reveal these issues (Section 2.1), as implicit assumptions can rob a player of correctly answered questions. If you wanted to answer 2014 to "when did Michigan last win the championship"—when the Michigan State Spartans won the Women's Cross Country championship—and you cannot because you chose the wrong school, the wrong sport, and the wrong gender, you would complain as a player; researchers instead discover latent assumptions that creep into the data.<sup>7</sup>
It is worth emphasizing that this is not a purely hypothetical problem. For example, Open Domain Retrieval Question Answering (Lee et al., 2019) deliberately avoids providing a reference context for the question in its framing but, in re-purposing data such as Natural Questions, opaquely relies on it for the gold answers.
3.2 Avoiding superficial evaluations
A related issue is that, in the words of Voorhees and Tice (2000), "there is no such thing as a question with an obvious answer". As a consequence, trivia question authors delineate acceptable and unacceptable answers.
For example, in writing for the trivia tournament Harvard Fall XI, Robert Chu uses a mental model of an answerer to explicitly delineate the range of acceptable correct answers:
In Newtonian gravity, this quantity satisfies Poisson's equation. [...] For a dipole, this quantity is given by negative the dipole moment dotted with the electric field. [...] For 10 points, name this form of energy contrasted with kinetic.
ANSWER: potential energy (prompt on energy; accept specific types like electrical potential energy or gravitational potential energy; do not accept or prompt on just "potential")
Likewise, the style guides for writing questions stipulate that you must give the answer type clearly and early on. These mentions specify whether you want a book, a collection, a movement, etc. It also
signals the level of specificity requested. For example, a question about a date must state "day and month required" (September 11, "month and year required" (April 1968), or "day, month, and year required" (September 1, 1939). This is true for other answers as well: city and team, party and country, or more generally "two answers required". Despite these conventions, no pre-defined set of answers is perfect, and every worthwhile trivia competition has a process for adjudicating answers.
In high school and college national competitions and game shows, if low-level staff cannot resolve the issue by throwing out a single question or accepting minor variations (America instead of USA), the low-level staff contacts the tournament director. The tournament director, who has a deeper knowledge of rules and questions, often decide the issue. If not, the protest goes through an adjudication process designed to minimize bias: write the summary of the dispute, get all parties to agree to the summary, and then hand the decision off to mutually agreed experts from the tournament's phone tree. The substance of the disagreement is communicated (without identities), and the experts apply the rules and decide.
Consider what happened when a particularly inept Jeopardy! contestant<sup>9</sup> did not answer laproscope to "Your surgeon could choose to take a look inside you with this type of fiber-optic instrument". Since the van Doren scandal (Freedman, 1997), every television trivia contestant has an advocate assigned from an auditing company. In this case, the advocate initiated a process that went to a panel of judges who then ruled that endoscope (a more general term) was also correct.
The need for a similar process seems to have been well-recognized in the earliest days of QA system bake-offs such as TREC-QA, and Voorhees (2008) notes that
[d]ifferent QA runs very seldom return exactly the same [answer], and it is quite difficult to determine automatically whether the difference [...] is significant.
In stark contrast to this, QA datasets typically only provide a single string or, if one is lucky, several strings. A correct answer means exactly matching these strings or at least having a high token overlap  $F_{1}$ , and failure to agree with the pre-recorded admissible answers will put you at an uncontestable disadvantage on the leaderboard (Section 2.2).
<sup>7</sup>Where to draw the line is a matter of judgment; computers—which lack common sense—might find questions ambiguous where humans would not.
8 https://www.nagt.com/rules/#protest
9 http://www.j-archive.com/showgame.php?game_id=6112
To illustrate how current evaluations fall short of meaningful discrimination, we qualitatively analyze two near-SOTA systems on SQuAD V1.1: the original XLNet (Yang et al., 2019) and a subsequent iteration called XLNet-123. $^{10}$
Despite XLNet-123's margin of almost four absolute  $F_{1}$  (94 vs 98) on development data, a manual inspection of a sample of 100 of XLNet-123's wins indicate that around two-thirds are 'spurious':  $56\%$  are likely to be considered not only equally good but essentially identical;  $7\%$  are cases where the answer set omits a correct alternative; and  $5\%$  of cases are 'bad' questions.[11]
Our goal is not to dwell on the exact proportions, to minimize the achievements of these strong systems, or to minimize the usefulness of quantitative evaluations. We merely want to raise the limitation of blind automation for distinguishing between systems on a leaderboard.
Taking our cue from the trivia community, we present an alternative for MRQA. Blind test sets are created for a specific time; all systems are submitted simultaneously. Then, all questions and answers are revealed. System authors can protest correctness rulings on questions, directly addressing the issues above. After agreement is reached, quantitative metrics are computed for comparison purposes—despite their inherent limitations they at least can be trusted. Adopting this for MRQA would require creating a new, smaller test set every year. However, this would gradually refine the annotations and process.
This suggestion is not novel: Voorhees and Tice (2000) accept automatic evaluations "for experiments internal to an organization where the benefits of a reusable test collection are most significant (and the limitations are likely to be understood)" (our emphasis) but that "satisfactory techniques for [automatically] evaluating new runs" have not been found yet. We are not aware of any change on this front—if anything, we seem to have become more insensitive as a community to just how limited our current evaluations are.
3.3 Focus on the bubble
While every question should be perfect, time and resources are limited. Thus, authors and editors of tournaments "focus on the bubble", where the