SlowGuess/ACL_2020_OCR huggingface dataset:
"bubble" are the questions most likely to discriminate between top teams at the tournament. These questions are thoroughly playtested, vetted, and edited. Only after these questions have been perfected will the other questions undergo the same level of polish.
For computers, the same logic applies. Authors should ensure that these discriminative questions are correct, free of ambiguity, and unimpeachable. However, as far as we can tell, the authors of QA datasets do not give any special attention to these questions.
Unlike a human trivia tournament, however—with finite patience of the participants—this does not mean that you should necessarily remove all of the easy or hard questions from your dataset. This could inadvertently lead to systems unable to answer simple questions like “who is buried in Grant’s tomb?” (Dwan, 2000, Chapter 7). Instead, focus more resources on the bubble.
4 Why Quizbowl is the Gold Standard
We now focus our thus far wide-ranging QA discussion to a specific format: Quizbowl, which has many of the desirable properties outlined above. We have no delusion that mainstream QA will universally adopt this format (indeed, a monoculture would be bad). However, given the community's emphasis on fair evaluation, computer QA can borrow aspects from the gold standard of human QA.
We have shown example of Quizbowl questions, but we have not explained how the format works; see Rodriguez et al. (2019) for more. You might be scared off by how long the questions are. However, in real Quizbowl trivia tournaments, they are not finished because the questions are interruptible.
Interruptible A moderator reads a question. Once someone knows the answer, they use a signaling device to "buzz in". If the player who buzzed is right, they get points. Otherwise, they lose points and the question continues for the other team.
Not all trivia games with buzzers have this property, however. For example, take Jeopardy!, the subject of Watson's tour de force (Ferrucci et al., 2010). While Jeopardy! also uses signaling devices, these only work once the question has been read in its entirety; Ken Jennings, one of the top Jeopardy! players (and also a Quizbowler) explains it on a Planet Money interview (Malone, 2019):
<sup>10</sup>We could not find a paper describing XLNet-123, the submission is by http://tia today.
Examples in Appendix C.
Jennings: The buzzer is not live until Alex finishes reading the question. And if you buzz in before your buzzer goes live, you actually lock yourself out for a fraction of a second. So the big mistake on the show is people who are all adrenalized and are buzzing too quickly, too eagerly.
Malone: OK. To some degree, Jeopardy! is kind of a video game, and a crappy video game where it's, like, light goes on, press button—that's it.
Jennings: (Laughter) Yeah.
Jeopardy!’s buzzers are a gimmick to ensure good television; however, Quizbowl buzzers discriminate knowledge (Section 2.3). Similarly, while TriviaQA (Joshi et al., 2017) is written by knowledgeable writers, the questions are not pyramidal.
Pyramidal Recall that effective datasets discriminate the best from the rest—the higher the proportion of effective questions  $\rho$ , the better. Quizbowl's  $\rho$  is nearly 1.0 because discrimination happens within a question: after every word, an answerer must decide if they know enough to answer. Quizbowl questions are arranged so that questions are maximally pyramidal: questions begin with hard clues—ones that require deep understanding—to more accessible clues that are well known.
Well-Edited Quizbowl questions are created in phases. First, the author selects the answer and assembles (pyramidal) clues. A subject editor then removes ambiguity, adjusts acceptable answers, and tweaks clues to optimize discrimination. Finally, a packetizer ensures the overall set is diverse, has uniform difficulty, and is without repeats.
Unnatural Trivia questions are fake: the asker already knows the answer. But they're no more fake than a course's final exam, which—like leaderboards—are designed to test knowledge.
Experts know when questions are ambiguous (Section 3.1); while "what play has a character whose father is dead" could be Hamlet, Antigone, or Proof, a good writer's knowledge avoids the ambiguity. When authors omit these cues, the question is derided as a "hose" (Eltinge, 2013), which robs the tournament of fun (Section 2.1).
One of the benefits of contrived formats is a focus on specific phenomena. Dua et al. (2019) exclude questions an existing MRQA system could answer to focus on challenging quantitative reasoning. One of the trivia experts consulted in Wallace et al. (2019) crafted a question that tripped up neural QA by embedding the phrase "this author opens
Crime and Punishment" into a question; the top system confidently answers Fyodor Dostoyevski. However, that phrase was in a longer question "The narrator in Cogwheels by this author opens Crime and Punishment to find it has become The Brothers Karamazov". Again, this shows the inventiveness and linguistic dexterity of the trivia community.
A counterargument is that real-life questions—e.g., on Yahoo! Questions (Szpektor and Dror, 2013), Quora (Iyer et al., 2017) or web search (Kwiatkowski et al., 2019)—ignore the craft of question writing. Real humans react to unclear questions with confusion or divergent answers, explicitly answering with how they interpreted the original question (“I assume you meant...”).
Given real world applications will have to deal with the inherent noise and ambiguity of unclear questions, our systems must cope with it. However, addressing the real world cannot happen by glossing over its complexity.
Complicated Quizbowl is more complex than other datasets. Unlike other datasets where you just need to decide what to answer, in Quizbowl you also need to choose when to answer the question.[12] While this improves the dataset's discrimination, it can hurt popularity because you cannot copy/paste code from other QA tasks. The cumbersome pyramidal structure complicates some questions (e.g., what is log base four of sixty-four).
5 A Call to Action
You may disagree with the superiority of Quizbowl as a QA framework (de gustibus non est disputandum). In this final section, we hope to distill our advice into a call to action regardless of your question format or source. Here are our recommendations if you want to have an effective leaderboard.
Talk to Trivia Nerds You should talk toTrivia nerds because they have useful information (not just about the election of 1876). Trivia is not just the accumulation of information but also connecting disparate facts (Jennings, 2006). These skills are exactly those we want computers to develop.
Trivia nerds are writing questions anyway; we
12This complex methodology can be an advantage. The underlying mechanisms of systems that can play Quizbowl (e.g., reinforcement learning) share properties with other tasks, such as simultaneous translation (Grissom II et al., 2014; Ma et al., 2019), human incremental processing (Levy et al., 2008; Levy, 2011), and opponent modeling (He et al., 2016).
can save money and time if we pool resources. $^{13}$  Computer scientists benefit if the trivia community writes questions that aren't trivial for computers to solve (e.g., avoiding quotes and named entities). The trivia community benefits from tools that make their job easier: show related questions, link to Wikipedia, or predict where humans will answer.
Likewise, the broader public has unique knowledge and skills. In contrast to low-paid crowdworkers, public platforms for question answering and citizen science (Bowser et al., 2013) are brimming with free expertise if you can engage the relevant communities. For example, the Quora query "Is there a nuclear control room on nuclear aircraft carriers?" is purportedly answered by someone who worked in such a room (Humphries, 2017). As machine learning algorithms improve, the "good enough" crowdsourcing that got us this far may not be enough for continued progress.
Eat Your Own Dog Food As you develop new question answering tasks, you should feel comfortable playing the task as a human. Importantly, this is not just to replicate what crowdworkers are doing (also important) but to remove hidden assumptions, institute fair metrics, and define the task well. For this to feel real, you will need to keep score; have all of your coauthors participate and compare scores.
Again, we emphasize that human and computer skills are not identical, but this is a benefit: humans' natural aversion to unfairness will help you create a better task, while computers will blindly optimize an objective function (Bostrom, 2003). As you go through the process of playing on your question-answer dataset, you can see where you might have fallen short on the goals we outline in Section 3.
Won't Somebody Look at the Data? After QA datasets are released, there should also be deeper, more frequent discussion of actual questions within the NLP community. Part of every post-mortem of trivia tournaments is a detailed discussion of the questions, where good questions are praised and bad questions are excoriated. This is not meant to shame the writers but rather to help build and reinforce cultural norms: questions should be well-
written, precise, and fulfill the creator's goals. Just like trivia tournaments, QA datasets resemble a product for sale. Creators want people to invest time and sometimes money (e.g., GPU hours) in using their data and submitting to their leaderboards. It is "good business" to build a reputation for quality questions and discussing individual questions.
Similarly, discussing and comparing the actual predictions made by the competing systems should be part of any competition culture—without it, it is hard to tell what a couple of points on some leaderboard mean. To make this possible, we recommend that leaderboards include an easy way for anyone to download a system's development predictions for qualitative analyses.
Make Questions Discriminative We argue that questions should be discriminative (Section 2.3), and while Quizbowl is one solution (Section 4), not everyone is crazy enough to adopt this (beautiful) format. For more traditional QA tasks, you can maximize the usefulness of your dataset by ensuring as many questions as possible are challenging (but not impossible) for today's QA systems.
But you can use some Quizbowl intuitions to improve discrimination. In visual QA, you can offer increasing resolutions of the image. For other settings, create pyramidality by adding metadata: coreference, disambiguation, or alignment to a knowledge base. In short, consider multiple versions/view of your data that progress from difficult to easy. This not only makes more of your dataset discriminative but also reveals what makes a question answerable.
Embrace Multiple Answers or Specify Specificity As QA moves to more complicated formats and answer candidates, what constitutes a correct answer becomes more complicated. Fully automatic evaluations are valuable for both training and quick-turnaround evaluation. In the case annotators disagree, the question should explicitly state what level of specificity is required (e.g., September 1, 1939 vs. 1939 or Leninism vs. socialism). Or, if not all questions have a single answer, link answers to a knowledge base with multiple surface forms or explicitly enumerate which answers are acceptable.
Appreciate Ambiguity If your intended QA application has to handle ambiguous questions, do justice to the ambiguity by making it part of your task—for example, recognize the original ambigu
13 Many question answering datasets benefit from the efforts of the trivia community. Ethically using the data, however, requires acknowledging their contributions and using their input to create datasets (Jo and Gebru, 2020, Consent and Inclusivity criterion).
ity and resolve it (“did you mean...”) instead of giving credit for happening to ‘fit the data’.
To ensure that our datasets properly "isolate the property that motivated [the dataset] in the first place" (Zaenen, 2006), we need to explicitly appreciate the unavoidable ambiguity instead of silently glossing over it.[14]
This is already an active area of research, with conversational QA being a new setting actively explored by several datasets (Reddy et al., 2018; Choi et al., 2018); and other work explicitly focusing on identifying useful clarification questions (Rao and Daumé III), thematically linked questions (Elgohary et al., 2018) or resolving ambiguities that arise from coreference or pragmatic constraints by rewriting underspecified question strings (Elgohary et al., 2019; Min et al., 2020).
Revel in Spectacle However, with more complicated systems and evaluations, a return to the yearly evaluations of TRECQA may be the best option. This improves not only the quality of evaluation (we can have real-time human judging) but also lets the test set reflect the build it/break it cycle (Ruef et al., 2016), as attempted by the 2019 iteration of FEVER (Thorne et al., 2019). Moreover, another lesson the QA community could learn from trivia games is to turn it into a spectacle: exciting games with a telegenic host. This has a benefit to the public, who see how QA systems fail on difficult questions and to QA researchers, who have a spoonful of fun sugar to inspect their systems' output and their competitors'.
In between full automation and expensive humans in the loop are automatic metrics that mimic the flexibility of human raters, inspired by machine translation evaluations (Papineni et al., 2002; Specia and Farzindar, 2010) or summarization (Lin, 2004). However, we should not forget that these metrics were introduced as 'understudies'—good enough when quick evaluations are needed for system building but no substitute for a proper evaluation. In machine translation, Laubli et al. (2020) reveal that crowdworkers cannot spot the errors that neural MT systems make—fortunately, trivia nerds are cheaper than professional translators.
Be Honest in Crowning QA Champions Leaderboards are a ranking over entrants based on
a ranking over numbers. This can be problematic for several reasons. The first is that single numbers have some variance; it's better to communicate estimates with error bars.
While—particularly for leaderboards—it is tempting to turn everything into a single number, there are often different sub-tasks and systems who deserve recognition. A simple model that requires less training data or runs in under ten milliseconds may be objectively more useful than a bloated, brittle monster of a system that has a slightly higher  $F_{1}$  (Dodge et al., 2019). While you may only rank by a single metric (this is what trivia tournaments do too), you may want to recognize the highest-scoring model that was built by undergrads, took no more than one second per example, was trained only on Wikipedia, etc.
Finally, if you want to make human-computer comparisons, pick the right humans. Paraphrasing a participant of the 2019 MRQA workshop (Fisch et al., 2019), a system better than the average human at brain surgery does not imply superhuman performance in brain surgery. Likewise, beating a distracted crowdworker on QA is not QA's endgame. If your task is realistic, fun, and challenging, you will find experts to play against your computer. Not only will this give you human baselines worth reporting—they can also tell you how to fix your QA dataset... after all, they've been at it longer than you have.
Acknowledgements This work was supported by Google's Visiting Researcher program. Boyd-Graber is also supported by NSF Grant IIS-1822494. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor. Thanks to Christian Buck for creating the NQ playtesting environment that spurred the initial idea for this paper. Thanks to Jon Clark and Michael Collins for the exciting e-mail thread that forced the authors to articulate their positions for the first time. Thanks to Kevin Kwok for permission to use Protobowl screenshot and information. Hearty thanks to all those who read and provided feedback on drafts: Matt Gardner, Roger Craig, Massimiliano Ciaramita, Jon May, Zachary Lipton, and Divyansh Kaushik. And finally, thanks to the trivia community for providing a convivial home for pedants and know-it-alls; may more people listen to you.
14 Not surprisingly, 'inherent' ambiguity is not limited to QA; Pavlick and Kwiatkowski (2019) show natural language inference has 'inherent disagreements' between humans and advocate for recovering the full range of accepted inferences.
Luis von Ahn. 2006. Games with a purpose. Computer, 39:92-94.
David Baber. 2015. Television Game Show Hosts: Biographies of 32 Stars. McFarland.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation.
Nick Bostrom. 2003. Ethical issues in advanced artificial intelligence. Institute of Advanced Studies in Systems Research and Cybernetics, 2:12-17.
Anne Bowser, Derek Hansen, Yurong He, Carol Boston, Matthew Reid, Logan Gunnell, and Jennifer Preece. 2013. Using gamification to inspire new citizen science volunteers. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification '13, pages 18-25, New York, NY, USA. ACM.
Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daume III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of Empirical Methods in Natural Language Processing.
Seth Chaiklin. 2003. The Zone of Proximal Development in Vygotsky's Analysis of Learning and Instruction, Learning in Doing: Social, Cognitive and Computational Perspectives, page 39-64. Cambridge University Press.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of Empirical Methods in Natural Language Processing.
Anthony Cuthbertson. 2018. Robots can now read better than humans, putting millions of jobs at risk. Newsweek.
Paul Diamond. 2009. How To Make 100 Pounds A Night (Or More) As A Pub Quizmaster. DP Quiz.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Conference of the North American Chapter of the Association for Computational Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. CoRR, abs/1704.05179.
R. Dwan. 2000. As Long as They're Laughing: Groucho Marx and You Bet Your Life. Midnight Marquee Press.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In *Empirical Methods in Natural Language Processing*.
Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. 2018. Dataset and baselines for sequential open-domain question answering. In Empirical Methods in Natural Language Processing.
Stephen Eltinge. 2013. Quizbowl lexicon.
Manaal Faruqui and Dipanjan Das. 2018. Identifying well-formed natural language questions. In Proceedings of Empirical Methods in Natural Language Processing.
Shi Feng, Eric Wallace, Alvin Grissom II, Pedro Rodriguez, Mohit Iyyer, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretation difficult. In Proceedings of Empirical Methods in Natural Language Processing.
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building Watson: An Overview of the DeepQA Project. AI Magazine, 31(3).
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen, editors. 2019. Proceedings of the 2nd Workshop on Machine Reading for Question Answering. Association for Computational Linguistics, Hong Kong, China.
Flight Standards Service. 2008. Aviation Instructor's Handbook, volume FAA-H-8083-9A. Federal Aviation Administration, Department of Transportation.
Morris Freedman. 1997. The fall of Charlie Van Doren. The Virginia Quarterly Review, 73(1):157-165.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of Empirical Methods in Natural Language Processing.
Alvin Grissom II, He He, Jordan Boyd-Graber, and John Morgan. 2014. Don't until the final verb wait: Reinforcement learning for simultaneous machine translation. In Proceedings of Empirical Methods in Natural Language Processing.
Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary C. Lipton. 2019. AmazonQA: A review-based question answering task. In International Joint Conference on Artificial Intelligence.
Thomas M. Haladyna. 2004. Developing and Validating Multiple-choice Test Items. Lawrence Erlbaum Associates.
Robert X. D. Hawkins, Andreas Stuhlmüller, Judith Degen, and Noah D. Goodman. 2015. Why do you ask? Good questions provoke informative answers. In Proceedings of the 37th Annual Meeting of the Cognitive Science Society.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daume III. 2016. Opponent modeling in deep reinforcement learning. In Proceedings of the International Conference of Machine Learning.
R. Robert Hentzel. 2018. NAFTA eligibility overview.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of Advances in Neural Information Processing Systems.
Bryan Humphries. 2017. Is there a nuclear control room on nuclear aircraft carriers?
Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017. Quora question pairs.
Ken Jennings. 2006. *Brainiac: adventures in the curious, competitive, compulsive world of trivia buffs*. Villard.
Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the Association for Computational Linguistics.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of Empirical Methods in Natural Language Processing.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics.