SlowGuess/NAACL_2022_OCR huggingface dataset:
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. GooAQ: Open question answering with diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421-433, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Payal Khullar, Konigari Rachna, Mukul Hase, and Manish Shrivastava. 2018. Automatic question generation using relative pronouns and adverbs. In Proceedings of ACL 2018, Student Research Workshop, pages 153-158, Melbourne, Australia. Association for Computational Linguistics.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Philippe Laban, John Canny, and Marti A Hearst. 2020. What's the latest? a question-driven news chatbot. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 380-387.
Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding.
In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 889-898, Beijing, China. Association for Computational Linguistics.
Adam D. Lelkes, Vinh Q. Tran, and Cong Yu. 2021. Quiz-Style Question Generation for News Stories. In Proc. of the the Web Conf. 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. CoRR, abs/1910.13461.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101.
Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950-3959.
Liangming Pan, Wenhu Chen, Wenhan Xiong, MinYen Kan, and William Yang Wang. 2021. Zero-shot fact verification by claim generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 476-483, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401-2410, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring
the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Clément Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-questeval: A referenceless metric for data to text semantic evaluation. arXiv preprint arXiv:2104.07555.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computational Linguistics.
Katherine Stasaski, Manav Rathod, Tony Tu, Yunfang Xiao, and Marti A. Hearst. 2021. Automatically generating cause-and-effect questions from passages. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 158-170, Online. Association for Computational Linguistics.
Dan Su, Yan Xu, Wenliang Dai, Ziwei Ji, Tiezheng Yu, and Pascale Fung. 2020. Multi-hop question generation with graph convolutional network. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4636-4647, Online. Association for Computational Linguistics.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930-3939, Brussels, Belgium. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018. Learning to ask questions in open-domain conversational systems with typed decoders. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2193-2203, Melbourne, Australia. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Wenhao Liu, Pascale Fung, and Caiming Xiong. 2021. Qaconv: Question answering on informative conversations. arXiv preprint arXiv:2105.06912.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. TWEEQA: A social media focused question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5020-5031, Florence, Italy. Association for Computational Linguistics.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020a. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020b. Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. CoRR, abs/2001.04063.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. CoRR, abs/1912.08777.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with BERT*. In 8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Wenjie Zhou, Minghua Zhang, and Yunfang Wu. 2019. Question-type driven question generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6032-6037, Hong Kong, China. Association for Computational Linguistics.
Training datasets are listed in Table 1. For training MixQG, we use several pre-trained text-to-text model checkpoints from the HuggingFace library (Wolf et al., 2020). We finetune them for question generation using our combined dataset described in Section 2.1. For most experiments done in this paper, we finetune on a T5-base model (Raffel et al., 2020). We also scale up the model and report results for T5-large, T5-3B, and BART-large settings (Appendix D). We train for 100,000 steps (or 22 epochs) with a learning rate of  $3 \times 10^{-5}$  using the AdamW (Loshchilov and Hutter, 2017) optimizer and a batch size of 32. All training was done on eight A100 NVIDIA GPUs and took approximately 35 hours.
B Quiz Design Task Details
We recruit teachers or ex-teachers from an online group forum. In total, 20 participants filled out the interest form, 14 were selected, and 10 completed the study. The participants had been teachers for at least a year and 3.6 years on average, and had taught diverse subjects such as sciences, history, literature, and IT topics, at various levels from primary school to college-level. The study was meant to last a maximum of two hours, and participants were gifted a $50 gift card upon completion.
Participants were tasked with creating between 5-7 quizzes, each with a minimum of 8 concepts, and could pick from a set list of 7 quiz topics, which we pre-selected from the list of featured Wikipedia articles<sup>3</sup>. We purposefully selected articles within different domains to benchmark the QGen models in diverse topical settings: two in physics (Sustainable Energy, Californium Atom), two in biology (DNA, Enzymes), two in history (Statue of Liberty, Palazzo Pitti), and one in geology (the K-T extinction). Participants were given the first 500 words
$^{3}$ https://en.wikipedia.org/wiki/Wikipedia:Featured_articles
of the Wikipedia page of each topic as reading material to select Quiz concepts from. User interface is shown in Figure 4. Hierarchical categorization of errors for question generation is shown in Figure 3.
C Qualitative Study Details
To understand MixQG's performance beyond automated metrics, we analyze its generated questions in Table 5. It shows several examples of questions generated by MixQG-3B on the validation sets of different datasates along with the ground-truth questions. We also generate question-answer pairs on Wikipedia articles using a pipeline approach as shown in Figure 5. First, we use a summarization model to obtain the summary of the context. Then we feed each sentence of the summary as the target answer to MixQG and obtain the questions. We observe that the generated questions are grammatically fluent, relevant to the input, and answerable by the target answer paragraph. We find that feeding in longer answers to the model generates more general, higher-level questions about the source article, while short answers prompt more factoid-style questions. As a result, we are able to generate questions of varied cognitive levels from the same source document by restricting the answer part of the input.
D Scaling
Table 7 shows the performance of differently sized MixQG models on SQuAD dataset. We additionally train MixQG model based on BART-large checkpoint, referred to as  $\mathrm{MixQG}_{\text{large}}^{BART}$ . As expected, the largest MixQG model (3 billion parameters) performs best among the different model size variants.
Figure 3: Hierarchical categorization of errors for question generation. Three error categories (Disfluent, Off Target, Wrong Context) each with several subtypes.
4https://huggingface.co/facebook/ bart-large-cnn
Quiz Design
Re-Open Tutorial
Californium is a radioactive chemical element with the symbol Cf and atomic number 98. The element was first synthesized in 1950 at the Lawrence Berkeley National Laboratory (then the University of California Radiation Laboratory), by bombarding curium with alpha particles (helium-4 ions). It is an actinide element, the sixth transuranium element to be synthesized, and has the second-highest atomic mass of all the elements that have been produced in amounts large enough to see with the unaided eye (after einsteinium). The element was named after the university and the U.S. state of California.
Two crystalline forms exist for californium under normal pressure: one above and one below  $900^{\circ}\mathrm{C}$  (1,650 °F). A third form exists at high
How was californium first synthesized?
How was the element first synthesized?
How was Californium first synthesized?
What was the first atomic number?
Off Target Wrong Context Disfluent
Figure 4: Screenshot of annotation interface used for the Quiz Design Task. The teacher has selected the concept highlighted in blue in the reading material in the left column. In the right column, the system gives proposes candidate questions, which can be added to the quiz, or refused with a reason.
Figure 5: Example of generating QA pairs using summarization and MixQG.
Dataset	Source	Questions
SQuAD	Gold	What happened to NASA's yearly budget after the first landing?
MixQG	What happened to NASA's budget after the first successful moon landing?
Gold	How many of Warsaw's inhabitants spoke Polish in 1933?
MixQG	In 1933, how many of Warsaw's inhabitants were of Polish mother tongue?
Gold	How long does it take for new areas to have significant oil production?
MixQG	How long did it take to develop new oil fields?
NarQA	Gold	What are Mulder and Scully doing at the beginning of the story?
MixQG	What are Mulder and Scully doing in Dallas?
Gold	How does Chris make extra money?
MixQG	What does Chris Hughes do for a living?
Gold	Who died in this story?
MixQG	What is the name of Mortimer Tregennis's sister?
MCTest	Gold	How many of Mikes friends came to the party
MixQG	How many of Mike's friends came to the party?
Gold	Where did Jenny want to go on a trip to?
MixQG	Where did Jenny want to go on a trip?
Gold	What was Stewart doing at the beach?
MixQG	What was Stewart doing when he saw the bug?
BoolQ	Gold	is there an amtrak station in pensacola florida
MixQG	is there an amtrak station in pensacola florida
Gold	can a minor sit at a bar in nj
MixQG	can a minor drink at a bar in new jersey
Gold	can a polar bear and a grizzly mate
MixQG	is there such a thing as a grizzly polar bear
QAConv	Gold	What Xbox 360 game is the best dance game for a wife on the Wii?
MixQG	What is a good dance game for the Wii?
Gold	Which state is taking its proper place and represents the country's diversity much better?
MixQG	What state is taking its proper place as early in the primary?
Gold	In what month will the BLM say go ahead and start the stimulation experiment ?
MixQG	When will the experiment start?
Table 5: Example of MixQG-3B generated questions and their corresponding ground-truth labels.
Context	Model	Question	Annotation
For example, californium can be used [...] when studying materials using neutron diffraction and neutron spectroscopy.	MixQG-L	What methods are used to study ma-terials using californium as a source of neutrons?	No Error
BART-L	What is Californium used to study materials?	Disfluent
With the exception of some ectothermic species [...] no tetrapods weighing more than 25 kilograms (55 pounds) survived.	MixQG-L	What size tetrapods did not survive the extinction?	No Error
ProphetNet	How much did tetrapods weigh at the time of the Cretaceous-Paleogene extinction?	Off Tgt.
The two DNA strands are known as polynucleotides as they are composed of simpler monomeric units called nucleotides.	MixQG-L	What are polynucleotides composed of?	No Error
BART-L	What are polynucleotides?	Off Tgt.
The Statue of Liberty (Liberty Enlightening the World) is a colossal neoclassical sculpture on [...]	ProphetNet	What is another name for the Statue of Liberty?	No Error
MixQG-L	What is the English translation of the Statue of Liberty?	Off Tgt.
Californium. The element was named after the university and the U.S. state of California.	ProphetNet	What is Californium named after?	No Error
MixQG-L	Where did Californium get its name?	Wrong Ctxt
Fossil fuels provide 85% of the world's energy consumption and the energy system [...]	BART-L	How much of the world's energy consumption does fossil fuels pro-vide?	No Error
MixQG-L	What percentage of the world's energy consumption is fossil fuels?	Disfluent
Table 6: Success and failure cases of the MixQG model from the Quiz Design evaluation. Comparisons to the ProphetNet and BART-Large models are included, with each model receiving the context with a target answer (in bold), and being annotated with an error label by a teacher.
Model	BLEU	R1	R2	RL	RLsum	METEOR	BERTScore
ProphetNetlarge	22.88	51.37	29.48	47.11	47.09	41.46	0.4931
MixQGBARTlarge	23.30	54.44	31.92	50.18	50.18	43.47	0.5622
MixQGbase	23.53	54.39	32.06	50.05	50.02	43.83	0.5566
MixQGlarge	24.42	55.52	33.13	50.99	50.97	45.07	0.5699
MixQG3b	25.42	56.11	33.91	51.85	51.86	45.75	0.5789
Table 7: Evaluation of differently-sized MixQG models on SQuAD. Base, Large and 3B refer to model configurations with 220 million, 770 million and 3 billion parameters, respectively.