SlowGuess/NAACL_2022_OCR huggingface dataset:
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Charles Henderson and Melissa H Dancy. 2007. Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics. Physical Review Special Topics-Physics Education Research, 3(2):020102.
Kenneth Holstein and Vincent Aleven. 2021. Designing for human-ai complementarity in k-12 education. arXiv preprint arXiv:2104.01266.
Kenneth Holstein, Erik Harpstead, Rebecca Gulotta, and Jodi Forlizzi. 2020. Replay enactments: Exploring possible futures through historical data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, pages 1607-1618.
Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. 2018. Student learning benefits of a mixed-reality teacher awareness tool in ai-enhanced classrooms. In International conference on artificial intelligence in education, pages 154-168. Springer.
Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159-166.
Gwo-Jen Hwang, Mei-Rong Alice Chen, Han-Yu Sung, and Mengwei Lin. 2019. Effects of integrating a concept mapping-based summarization strategy into flipped learning on students' reading performances and perceptions in chinese courses. Br. J. Educ. Technol., 50:2703-2719.
Kentaro Inui, Atsushi Fujita, Tetsuro Takahashi, Ryu Iida, and Tomoya Iwakura. 2003. Text simplification for reading assistance: a project note. In Proceedings of the second international workshop on Paraphrasing, pages 9-16.
Sungku Kang, Lalit Patil, Arvind Rangarajan, Abha Moitra, Tao Jia, Dean M. Robinson, and Debasish Dutta. 2019. Automated feedback generation for formal manufacturing rule extraction. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 33:289 - 301.
Anisia Katinskaia and Roman Yangarber. 2021. Assessing grammatical correctness in language learning. In BEA.
Kenneth R Koedinger, John R Anderson, William H Hadley, Mary A Mark, et al. 1997. Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1):30-43.
Rohit Kumar, Carolyn Penstein Rose, Yi-Chia Wang, Mahesh Joshi, and Allen Robinson. 2007. Tutorial dialogue as adaptive collaborative learning support. Frontiers in artificial intelligence and applications, 158:383.
Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1):121-204.
Minwoo Lee, Seungpil Won, Juae Kim, Hwanhee Lee, Cheoneum Park, and Kyomin Jung. 2021. Crossaug. Proceedings of the 30th ACM International Conference on Information Knowledge Management.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Jeri L Little and Elizabeth Ligon Bjork. 2015. Optimizing multiple-choice tests as tools for learning. Memory & Cognition, 43(1):14-26.
Jeri L Little, Elizabeth Ligon Bjork, Robert A Bjork, and Genna Angelo. 2012. Multiple-choice tests ex- onerated, at least of some charges: Fostering testinduced learning and avoiding test-induced forgetting. Psychological science, 23(11):1337-1344.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. ArXiv, abs/1908.08345.
Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization-informed question generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4134-4148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mukta Majumder and Sujan Kumar Saha. 2015. A system for generating multiple choice questions: With a novel approach for sentence selection. In Proceedings of the 2nd workshop on natural language processing techniques for educational applications, pages 64-72.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2021. Muss: Multilingual unsupervised sentence simplification by mining paraphrases.
Louis Martin, Benoit Sagot, Eric Villemonte de la Clergerie, and Antoine Bordes. 2020. Controllable sentence simplification. In LREC.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919.
Bill Moggridge and Bill Atkinson. 2007. Designing interactions, volume 17. MIT press Cambridge.
William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K Dey, and Min Kyung Lee. 2012. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference, pages 338-347.
Andrew M Olney, Arthur C Graesser, and Natalie K Person. 2012. Question generation from concept maps. Dialogue & Discourse, 3(2):75-99.
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan. 2020. Semantic graphs for generating deep questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1463-1475, Online. Association for Computational Linguistics.
Andreas Papasalouros, Konstantinos Kanaris, and Konstantinos Kotis. 2008. Automatic generation of multiple choice questions from domain ontologies. In e-Learning, pages 427-434. CiteSeer.
Rakesh Patra and Sujan Kumar Saha. 2019. A hybrid approach for automatic generation of named entity distractors for multiple choice questions. Education and Information Technologies, 24(2):973-993.
Perusall. 2021. Perusall.
Sarah E. Petersen and Mari Ostendorf. 2007. Text simplification for language learners: a corpus analysis. In SLaTE.
Irina Rets and Jekaterina Rogaten. 2021. To simplify or not? facilitating english l2 users' comprehension and processing of open educational resources in english using text simplification. J. Comput. Assist. Learn., 37:705-717.
Steven Ross, Michael H. Long, and Yasukata Yano. 1991. Simplification or elaboration? the effects of two types of text modifications on foreign language reading comprehension.
Daniel L Schwartz, Catherine C Chase, Marily A Oppezzo, and Doris B Chin. 2011. Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. Journal of educational psychology, 103(4):759.
Advaith Siddharthan and Napoleon Katsos. 2010. Reformulating discourse connectives for non-expert readers. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1002-1010, Los Angeles, California. Association for Computational Linguistics.
Dee U Silverthorn, Patti M Thorn, and Marilla D Svinicki. 2006. It's difficult to change the way we teach: lessons from the integrative themes in physiology curriculum module project. Advances in physiology Education, 30(4):204-214.
Megan A Smith and Jeffrey D Karpicke. 2014. Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory, 22(7):784-802.
Linfeng Song and Lin Zhao. 2016. Question generation from a knowledge base with web exploration. arXiv preprint arXiv:1610.03807.
Marilyne Stains, Jordan Harshman, Megan K Barker, Stephanie V Chasteen, Renee Cole, Sue Ellen DeChenne-Peters, MK Eagan, Joan M Esson, Jennifer K Knight, Frank A Laski, et al. 2018. Anatomy of stem teaching in north american universities. Science, 359(6383):1468-1470.
Katherine Stasaski and Marti A Hearst. 2017. Multiple choice question generation utilizing an ontology. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 303-312.
Elizabeth A Stevens, Sunyoung Park, and Sharon Vaughn. 2019. A review of summarizing and main idea interventions for struggling readers in grades 3 through 12: 1978-2016. *Remedial and Special Education*, 40(3):131-149.
Dan Su, Yan Xu, Wenliang Dai, Ziwei Ji, Tiezheng Yu, and Pascale Fung. 2020. Multi-hop question generation with graph convolutional network. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4636-4647, Online. Association for Computational Linguistics.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930-3939, Brussels, Belgium. Association for Computational Linguistics.
Adel I. Tweisii. 1998. The effects of the amount and type of simplification on foreign language reading comprehension. Reading in a foreign language, 11:191-204.
Tiiu Üksik, Jelena Kallas, Kristina Koppel, Katrin Tsepelina, and Raili Pool. 2021. Estonian as a second language teacher's tools. In *BEA*.
Sowmya Vajjala and Ivana Lucic. 2019. On understanding the relation between expert annotations of text readability and target reader comprehension. In *BEA@ACL*.
Sharon Vaughn, Janette K Klingner, Elizabeth A Swanson, Alison G Boardman, Greg Roberts, Sarojani S Mohammed, and Stephanie J Stillman-Spisak. 2011.
Efficacy of collaborative strategic reading with middle school students. American educational research journal, 48(4):938-964.
Tatiana Vodolazova and Elena Lloret. 2019. Towards adaptive text summarization: How does compression rate affect summary readability of 12 texts? In RANLP.
Ke Wang, Rishabh Singh, and Zhendong Su. 2018. Search, align, and repair: data-driven feedback generation for introductory programming exercises. Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation.
Xu Wang, Carolyn Rose, and Ken Koedinger. 2021. Seeing beyond expert blind spots: Online learning design for scale and quality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-14.
Xu Wang, Srinivasa Teja Talluri, Carolyn Rose, and Kenneth Koedinger. 2019. Upgrade: Sourcing student open-ended solutions to create scalable learning opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning@ Scale, pages 1-10.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Tongshuang Wu, Michael Terry, and Carrie J Cai. 2021. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. arXiv preprint arXiv:2110.01691.
Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-ai interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems, pages 1-13.
Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.
Wenjie Zhou, Minghua Zhang, and Yunfang Wu. 2019. Question-type driven question generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6032-6037, Hong Kong, China. Association for Computational Linguistics.
A Details for NLP Models on Selected Tasks
Here we describe implementation details for the NLP models used in §5.
A.1 Sentence Extraction
We extract salient sentences from the given context using extractive summarization model. The automatic extractive summarization task is approached as sentence classification, where each sentence  $i$  is assigned a label  $y_{i} \in \{0,1\}$ , with 1 indicating the sentence is predicted for inclusion in the summary, otherwise 0.
We use BERTSUMEXT (Liu and Lapata, 2019) that adds a sentence positional encoding schema to the large pre-trained BERT model. In our experiment, we take the released checkpoint trained on CNN/DailyMail dataset and only consider paragraph-level context as the input.
A.2 Abstractive Summarization
A.2.1 Pre-trained Summarization Model
We first use BERTSUMEXTABS (Liu and Lapata, 2019), which is first fine-tuned on the extractive summarization task and then on abstractive summarization. We again use a checkpoint of BERTSUMEXTABS fine-tuned on CNN-DailyMail released by the authors.
A.2.2 Summarization Model for Scientific Domain
To build an abstractive summarization model that is suitable for scientific papers, we build BARTSUM-HCI by fine-tuning BART (Lewis et al., 2020) on an automatically aligned section-summary pairs collected from HCI papers on arXiv.
Dataset Collection (arXiv-HCI). We first retrieve papers from arXiv.org with search query cat:cs.HC to identify all HCI relevant papers, resulting in 8658 academic papers. Since abstract is expected to condense the most salient information of a paper, we consider each sentence in abstract as a sentence-level summary of relevant sections in the paper. To align each abstract sentence with the corresponding section context, we apply BM25 to rank all sections by their lexicon similarity to the sentence and picked the top 2 sections to create two section-sentence pairs. This creates the arXiv-HCI dataset, which contains 72755 section-sentence pairs. The average numbers of tokens in
context and summary are 468.62 and 23.01, respectively.
Continued Training We then fine-tune BART on arXiv-HCI with train/validation/test splits of 71301/727/727. We use the checkpoint bart-base-finetuned-arxiv released by HuggingFace<sup>2</sup>, which has been trained on scientific papers.
The model is trained for 3 epochs, using Adam optimizer with default parameters  $(\beta_{1},\beta_{2}) = (0.9,$  0.999) and  $\epsilon = 1\mathrm{e} - 08$  . The learning rate is initialized as  $3\times 10^{-5}$  with 2000 warm-up steps and the weight decay is set to 0.01. After training for 25000 steps with batch size 8, the model is evaluated with ROUGE-1 of 20.51, ROUGE-2 of 5.30, and ROUGE-L of 16.73 on the test set.
A.3 Simplification
We use the AudienCe-CEntric Sentence Simplification (ACCESS) model (Martin et al., 2020) for sentence simplification. In our experiments, we take the released checkpoint<sup>3</sup> of ACCESS, which is pre-trained and evaluated on WikiLarge dataset. During our inference process, the default control parameters by ACCESS are used.
We also experiment with the Multilingual Unsupervised Sentence Simplification (MUSS) model (Martin et al., 2021). The dataset used for MUSS training is a combination of WikiLarge dataset and mined paraphrases from CCNet.
A.4 Paraphrasing
We create a paraphrase generation model BART-Para-SCI by fine-tuning the bart-paraphrase<sup>4</sup> checkpoint on the ParaSCI-ACL (Dong et al., 2021) dataset, which contains 33,981 paraphrase pairs from articles published in ACL conferences and workshops. The model is trained for 10 epochs, using Adam optimizer with default parameters  $(\beta_{1},\beta_{2}) = (0.9,0.999)$  and  $\epsilon = 1\mathrm{e} - 08$ . The learning rate is initialized as  $3\times 10^{-5}$  with 2500 warm-up steps. The weight decay is set to 0.01.
A.5 Negation
To generate distractors, we use CROSSAUG, which is proposed as a data augmentation method by training BART to generate negative claims (Lee
et al., 2021). It is fine-tuned on the WikiFactCheck-English dataset, with positive claims as the inputs and their corresponding negative claims as the outputs. We use the checkpoint<sup>5</sup> released by its author.
B MCQ Samples with Human-generated and Machine-generated MCQ Options
B.1 Context-option Generation Comparison Samples
Table 1 lists sample machine-generated options along with human-constructed transformed texts, based on the same context from our  $P2MCQ$  dataset (§3). For Simplification, Paraphrasing, Negation and Abstractive Summarization tasks, the sentence-level contexts are used as inputs; for the Extraction task, the paragraph-level contexts are used.
B.2 Selected Samples from  $P2MCQ$  Dataset
Figure 2 shows sample multiple choice questions in our  $P2MCQ$  dataset discussed in  $\S 3$ .
C NLP Text Transformation Operation Strategies
In Tables 2 and 3, we define our coding scheme for different text transformation operations, along with selected samples from our  $P2MCQ$  dataset.
D Distribution of Operations in  $P2MCQ$  Dataset
Figure 3 illustrates the distribution of different tasks used as single operation and along with other operations based on the  $P2MCQ$  dataset.
$^{2}$ https://huggingface.co/mse30/  
bart-base-finetuned-arxiv  
 $^{3}$ http://dl.fbaipublicfiles.com/access/  
best_model.tar.gz  
 $^{4}$ https://huggingface.co/eugenesiow/  
bart-paraphrase
$^{5}$ https://huggingface.co/minwhoo/bart-base-negative-claim-generation
Question #1: See below a figure on how users interact with the system presented in the paper. Which of the following is NOT correct about what the paper does?
Emerging technologies such as Augmented Reality (AR), have the potential to radically transform education by making challenging concepts visible and accessible to novices. In this project, we have designed a Hololens-based system in which collaborators are exposed to an unstructured learning activity in which they learned about the invisible physics involved in audio speakers. They learned topics ranging from spatial knowledge, such as shape of magnetic fields, to abstract conceptual knowledge, such as relationships between electricity and magnetism. We compared participants' learning, attitudes and collaboration with a tangible interface multiple experimental conditions containing layers of AR information. We found that educational AR representations were beneficial for learning specific knowledge and increasing participants' self-efficacy (i.e., their ability to learn concepts in physics). However, we also found that participants in conditions that did not contain AR educational content, learned some concepts better than other groups and became more curious about physics. We discuss learning and collaboration differences, as well as benefits and detriments of implementing augmented reality for unstructured learning activities.
$\rightarrow \mathsf{A}[\mathsf{Target}]$  The paper found that students learned all physics concepts better in the AR condition than those in the control conditions.
$\rightarrow$  B.[Distractor] The authors developed a Hololens-based system to provide AR learning experience for students.
$\rightarrow \mathbb{C}$  [Distractor] Students learned physics in the AR system collaboratively.
$\rightarrow$  D.[Distractor] In this paper, we present the results of a systematic study of the learning environment of students with the average age 23 in their first semester of Educational science at the university of Munich.
$\rightarrow$  A.[Negation] However, we also found that participants in conditions that did contain AR educational content, learned some concepts better than other groups and became more curious about physics.
$\rightarrow$  B.[Paraphrasing] To this end, we have designed an audio speaker-based system in which collaborators learned about the invisible physics involved in audio speakers through an unstructured learning activity.
$\rightarrow$  C.[Simplification (MUSS)+Paraphrasing] The authors compared the learning, attitudes and collaboration of participants with a compromise interface under different experimental conditions.
Question #2: The authors did an experiment to examine the relative benefits of guided discovery, explore-construct, and a combination with the AI mixed-reality system. Which of the following is NOT correct on why this is an important research question?
The Guided-Discovery and Combined conditions yield better explanation of predictions and constructed towers. Such better explanation indicates better scientific understanding of the underlying physics principles. Our results indicate the power of guided discovery and potential weakness of a sole focus on less-guided exploration and construction. We also find evidence of benefits of combining some exploration and construction along with guided discovery. In particular, the results of the tower building and prediction tasks suggest that children may better learn to use scientific explanations to facilitate engineering when some exploratory construction is added to guided discovery. ... While the Guided-Discovery condition implements deliberate practice recommendations to guide children in scientific inquiry toward learning science content, the Explore-Construction condition implements constructivist recommendations through a more authentic construction/building activity. ... It is worth noting substantial common ground in the learning support recommendations of deliberate practice and constructivism, particularly, focusing on engaging students in learning-by-doing and on more task-oriented or reactive guidance rather than extended up-front telling. Our goal is not to dispel the general merits of constructivism or deliberate practice, but to refine understanding of the effectiveness of particular variations within.
$\rightarrow$  A[Target] To evaluate the effectiveness of a system, we need to fully evaluate the different modalities and use scenarios of the system.
$\rightarrow$  B.[Distractor] Guided discovery provides specific cases for kids to reason about, however, kids do not experience openly construct towers. If we want kids to be able to build stable towers, we need to give them explore-construct activities.
$\rightarrow \mathbf{C}$  [Distractor] Explore-construct provides an open experience for kids to build towers, but guided discovery provides deliberate practice opportunities for kids to apply physics principles.
$\rightarrow$  D.[Distractor] They are both genres of active learning. There are theories and arguments behind either Guided-discovery and Explore-construct for one to be better than the other. Knowing the answer can help the design of active learning mechanisms.
$\rightarrow$  A.[Summarization (Bart-HCI)] In particular, the results of tower building and prediction tasks suggest that children may better learn to use scientific explanations to facilitate engineering when some exploratory construction is added to guided discovery.
$\rightarrow$  B.[Simplification (MUSS)] The Guided-Discovery condition is designed to guide children in scientific inquiry toward learning science content through deliberate practice. The Explore-Construction condition is designed to help children learn through more authentic construction/building activity.
$\rightarrow$  C.[Summarization (Bart-HCI)+Paraphrasing] Our goal is not to object to the general merits of constructivism or deliberate practice, but to refine understanding of the effectiveness of particular variations within the learning support recommendations.
Question #3: Which of the following is NOT correct about the results and implications of the evaluation study?