SlowGuess/NAACL_2022_OCR huggingface dataset:
answer types. Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022.
Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950-3959.
Jekaterina Novikova, Ondrej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
P David Pearson and Margaret C Gallagher. 1983. The instruction of reading comprehension. Contemporary educational psychology, 8(3):317-344.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2401-2410.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog 1.8 (2019): 9.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737-2746.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & Analgesia, 126(5):1763-1768.
Tim Steuer, Anna Filighera, Tobias Meuser, and Christoph Rensing. 2021. I do not understand what i cannot define: Automatic question generation with pedagogically-driven content selection. arXiv preprint arXiv:2110.04123.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930-3939.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
We trained five of the QGen models used in the Quiz Design study. They were all trained for ten epochs on the training portion of the SQuAD dataset (Rajpurkar et al., 2016), using the ADAM optimizer (Kingma and Ba, 2015), with hyperparameter tuning based on model loss on the validation set. The model checkpoint that achieves the lowest validation loss is selected as the final model. Selected hyper-parameters were:
Distil-GPT2: batch-size 32, learning rate  $2 * 10^{-5}$ .
GPT2-base: batch-size 32, learning rate  $2 * 10^{-5}$ .
GPT2-medium: batch-size 16, learning rate  $2 * 10^{-5}$ .
BART-base: batch-size 32, learning rate 1 *  $10^{-4}$ .
BART-large: batch-size 32, learning rate  $2 * 10^{-5}$ .
Finally, the last two QGen we used are publicly available on the HuggingFace model hub (Wolf et al., 2020), and we use them as is:
ProphetNet: mrcosoft/ prophetnet-large- uncased-squad-qg
MixQG: Salesforce/mixqg-large
With all models, we used beam search to generate candidate questions, using a beam-size of 2, and a sequence length maximum of 30.
B Guidelines to Annotators
We provide the exact guidelines that were given to study participants before they started the annotation procedure:
1. Your objective is to design a quiz about a particular topic for a class of students. The procedure is the following:
2. Select a quiz topic from the list (for example "Sustainable Energy")
3. The system will load a text about the topic.
4. Select a concept that you want to quiz your students on (for example a phrase, a figure, or a keyword) and confirm your selection.
5. Important: It is recommended to select shorter concepts, and not full sentences to
obtain more precise question. Selecting concepts of up to about 8 words is ideal.
6. The system will load a list of questions that attempt to quiz students about the selected concept.
7. Go over each question, and remove ones you would not include in your quiz. We will next go over types of questions that should be removed.
8. Important: you can keep one, multiple or none of the questions (if none of the questions are satisfactory). For each question you remove, you have to choose the reason that the question is unsatisfactory (more on this later).
9. Once you've finalized the question for a concept, select another concept and repeat the question selection process. Try to select 8-12 concepts per topic to generate long enough quizzes.
10. Once you've finished a full quiz set, you can move on to another quiz topic. We have found that in one hour, you should be able to complete the quizzes for 5 topics.
Following these guidelines, the annotators were provided definitions for each error category, as well as examples similar to the ones shown in Table A1.
C Error Categorization Question Examples
The examples listed in Table A1 were collected during a formative study to establish an error categorization for the task of Question Generation.
D Interface Screenshot
Figure A1 displays a screenshot of the interface used for the Quiz Design Task.
Category	Finer Category	Example Question	Rationale
Disfluent	Wrong Tense	What were historically used to disen-franchise racial minorities?	Should be "What was histori-cally..."
Awkward Phrasing	When did the woolly mammoth die?	Should be "go instinct" rather than "die"
Not a Question	In January 2020, scientists reported that climate-modeling of the extinction event favors the asteroid impact and not volcanism?	Sentence in declarative format
Repetition	Who led the team that led the K-Pg boundary clay?	"led" is repeated twice
Off Target	Unanswerable	Why are DNA studies so important?	Not answered in the DNA Wikipedia page.
Other Answer Span	Who designed the Statue of Liberty?	True answer is Bartholdi, even though target answer was Eiffel (the metalwork builder)
Wrong Ctxt	Too Specific	Where was the 181 km (114 mi) crater discovered?	Not standard to have unit translations in questions
Reveals Answer	What was the name of the Federal Reserve System? (leading to the creation of the Federal Reserve System)	Question's target answer is Fed-eral Reserve System
Inconsistent	What are the only two animals that survived the Cretaceous-Paleogene extinc-tion?	The Wikipedia article mentions species and not animals
Not Specific Enough	What are some ectothermic species?	Too many ectothermic species are mentioned in the article.
Table A1: Example generated questions collected during formative study. These examples form the basis for the error categorization we propose for the QGen task.
How was californium first synthesized?
How was the element first synthesized?
How was Californium first synthesized?
What was the first atomic number?
Figure A1: Screenshot of annotation interface used for the Quiz Design Task. The teacher has selected the concept highlighted in blue in the reading material in the left column. In the right column, the system gives proposes candidate questions, which can be added to the quiz, or refused with a reason.
Off Target Wrong Context