SlowGuess/ACL_2020_OCR huggingface dataset:
Samuel Laubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human-machine parity in language translation. Journal of Artificial Intelligence Research, 67.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Association for Computational Linguistics.
Roger Levy. 2011. Integrating surprisal and uncertain-input models in online sentence comprehension: Formal techniques and empirical results. In Proceedings of the Association for Computational Linguistics.
Roger P. Levy, Florencia Reali, and Thomas L. Griffiths. 2008. Modeling the effects of memory on human online sentence processing with particle filters. In Proceedings of Advances in Neural Information Processing Systems.
Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out.
Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling trends in machine learning scholarship. Queue, 17(1).
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proceedings of the Association for Computational Linguistics.
Kenny Malone. 2019. How uncle Jamie broke Jeopardy. Planet Money, (912).
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions.
Adam Najberg. 2018. Alibaba AI model tops humans in reading comprehension.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the Association for Computational Linguistics.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677-694.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing.
Sudha Rao and Hal Daume III. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. 2019. Quizbowl: The case for incremental question answering. CoRR, abs/1904.04792.
Marc-Antoine Rondeau and T. J. Hazen. 2018. Systematic error analysis of the Stanford question answering dataset. In Proceedings of the Workshop on Machine Reading for Question Answering.
Andrew Ruef, Michael Hicks, James Parker, Dave Levin, Michelle L. Mazurek, and Piotr Mardziel. 2016. Build it, break it, fix it: Contesting secure development. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security.
Dan Russell. 2020. Why control-F is the single most important thing you can teach someone about search. SearchReSearch.
Lucia Specia and Atefeh Farzindar. 2010. A.: Estimating machine translation post-editing effort with HTER. In In: AMTA 2010 Workshop, Bringing MT to the User: MT Research and the Translation Industry. The 9th Conference of the Association for Machine Translation in the Americas.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In Proceedings of Empirical Methods in Natural Language Processing.
Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the benchmarking capacity of machine reading comprehension datasets. In Association for the Advancement of Artificial Intelligence.
Idan Szpektor and Gideon Dror. 2013. From query to question in one click: Suggesting synthetic questions to searchers. In Proceedings of the World Wide Web Conference.
David Taylor, Colin McNulty, and Jo Meek. 2012. Your starter for ten: 50 years of University Challenge.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal, editors. 2018. Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER). Association for Computational Linguistics.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER).
Ellen M. Voorhees. 2003. Evaluating the evaluation: A case study using the TREC 2002 question answering track. In *Conference of the North American Chapter of the Association for Computational Linguistics*.
Ellen M. Voorhees. 2008. Evaluating Question Answering System Performance, pages 409-430. Springer Netherlands, Dordrecht.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval.
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. Transactions of the Association of Computational Linguistics, 10.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Conference on Computational Natural Language Learning.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the Association for Computational Linguistics.
Xingdi Yuan, Jie Fu, Marc-Alexandre Cote, Yi Tay, Christopher Pal, and Adam Trischler. 2019. Interactive machine comprehension with information seeking agents.
Annie Zaenen. 2006. Mark-up barking up the wrong tree. Computational Linguistics, 32(4):577-580.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the Association for Computational Linguistics.
Footnote numbers continue from main article.
A An Abridged History of Modern Trivia
In the United States, modern trivia exploded immediately after World War II via countless game shows including College Bowl (Baber, 2015), the precursor to Quizbowl. The craze spread to the United Kingdom in a bootlegged version of Quizbowl called University Challenge (now licensed by ITV) and pub quizzes (Taylor et al., 2012).
The initial explosion, however, was not without controversy. A string of cheating scandals, most notably the Van Doren (Freedman, 1997) scandal (the subject of the film Quiz Show), and the 1977 entry of Quizbowl into intercollegiate competition forced trivia to "grow up". Professional organizations and more "grownup" game shows like Jeopardy! (the "all responses in the form of a question" gimmick grew out of how some game shows gave contestants the answers) helped created formalized structures for trivia.
As the generation that grew up with formalized trivia reached adulthood, they sought to make the outcomes of trivia competitions more rigorous, eschewing the randomness that makes for good television. Organizations like National Academic Quiz Tournaments and the Academic Competition Federation created routes for the best players to help direct how trivia competitions would be run. In 2019, these organizations have institutionalized the best practices of "good trivia" described here.
B Simulating the Test Set Needed
We simulate a head-to-head trivia competition where System A and System B have an accuracy  $a$  (probability of getting a question right) separated by some difference:  $a_{A} - a_{B}\equiv \Delta$ . We then simulate this on a test set of size  $N$  scaled by the effective dataset proportion  $\rho$  via draws from two Binomial distributions with success probabilities of  $a_{A}$  and  $a_{B}$ :
$$
R _ {a} \sim \operatorname {B i n o m i a l} (\rho N, a _ {A}
$$
$$
R _ {b} \sim \operatorname {B i n o m i a l} (\rho N, a _ {B}) \tag {1}
$$
and see the minimum test set questions (using an experiment size of 5000) needed to detect the better system  $95\%$  of the time (i.e., the minimum  $N$  such
that  $R_{a} > R_{b}$  from Equation 1 in 0.95 of the experiments). Our emphasis, however is  $\rho$ : the smaller the percentage of discriminative questions (either because of difficulty or because of annotation error), the larger your test set must be.[15]
C Qualitative Analysis Examples
We provide some concrete examples for the classes into which we classified the XLNet-123 wins over XLNet. We indicate gold answer spans (provided by the human annotators) by underlining (there may be, the XLNet answer span by bold face, and the XLNet-123 answer span by italics, combining for tokens shared between spans as is appropriate.
C.1 Insignificant and significant span differences
QUESTION: What type of vote must the Parliament have to either block or suggest changes to the Commission's proposals?
CONTEXT: The essence is there are three readings, starting with a Commission proposal, where the Parliament must vote by a majority of all MEPs (not just those present) to block or suggest changes
a majority of all MEPs is as good an answer as majority, yet its Exact Match score is 0. The problem is not merely one of picking a soft metric; even its Token-F1 score is merely 0.4, effectively penalizing a system for giving a more complete answer. The limitations of Token-F1 become even clearer in light of the following significant span difference:
QUESTION: What measure of a computational problem broadly defines the inherent difficulty of the solution?
CONTEXT: A problem is regarded as inherently difficult if its solution requires significant resources, whatever the algorithm used.
We agree with the automatic evaluation that a system answering significant resources to this question should not be given full (and possibly no) credit as it fails to mention relevant context. Nevertheless, the Token-F1 of this answer is 0.57, i.e., larger than for the insignificant span difference just discussed.
C.2 Missing Gold Answers
We also observed 7 (out of 100) cases of missing gold answers. As an example, consider
15Disclaimer: This should be only one of many considerations in deciding on the size of your test set. Other factors may include balancing for demographic properties, covering linguistic variation, or capturing task-specific phenomena.
QUESTION: What would someone who is civilly disobedient do in court?
CONTEXT: Steven Barkan writes that if defendants plead not guilty, "they must decide whether their primary goal will be to win an acquittal and avoid imprisonment or a fine, or to use the proceedings as a forum to inform the jury and the public of the political circumstances surrounding the case and their reasons for breaking the law via civil disobedience." [...]
In countries such as the United States whose laws guarantee the right to a jury trial but do not excuse lawbreaking for political purposes, some civil disobedients seek jury nullification.
While annotators did mark two distinct spans as gold answers, they ignored jury nullification which is a fine answer to the question and should be rewarded. Reasonable people can disagree whether this is a missing answer or if it is excluded by a subtlety in the question's phrasing. This is precisely the point—relying on a pre-collected answer strings without a process for adjudicating disagreements in official comparisons does not do justice to the complexity of question answering.
C.3 Bad Questions
We also observed 5 cases of genuinely bad questions. Consider
QUESTION: What library contains the Selmur Productions catalogue?
CONTEXT: Also part of the library is the aforementioned Selznick library, the Cinerama Productions/Palomar theatrical library and the Selmur Productions catalog that the network acquired some years back
This is an annotation error—the correct answer to the question is not available from the paragraph and would have to be (the American Broadcast Company's) Programming Library. While we have to live with annotation errors as part of reality, it is not clear that we ought to accept them for official evaluations—any human taking a closer look at the paragraph, as part of an adjudication process, would concede that the question is problematic.
Other cases of 'annotation' error are more subtle, involving meaning-changing typos, for example:
QUESTION: Which French kind [sic] issued this declaration?
CONTEXT: They retained the religious provisions of the Edict of Nantes until the rule of Louis XIV, who progressively increased persecution of them until he issued the Edict of Fontainebleau (1685), which abolished all legal recognition of Protestantism in France
While one could debate whether or not systems ought to be able to do ‘charitable’ reinterpretations
of the question text, this is part of the point—cases like these warrant discussion and should not be silently glossed over when 'computing the score'.