SlowGuess/NAACL_2022_OCR huggingface dataset:
Xu Wang Simin Fan Jessica Houghton Lu Wang
{xwanghci, oliviaaaa, houghj, wangluxy}@umich.edu
NLP-powered automatic question generation (QG) techniques carry great pedagogical potential of saving educators' time and benefiting student learning. Yet, QG systems have not been widely adopted in classrooms to date. In this work, we aim to pinpoint key impediments and investigate how to improve the usability of automatic QG techniques for educational purposes by understanding how instructors construct questions and identifying touch points to enhance the underlying NLP models. We perform an in-depth need finding study with 11 instructors across 7 different universities, and summarize their thought processes and needs when creating questions. While instructors show great interests in using NLP systems to support question design, none of them has used such tools in practice. They resort to multiple sources of information, ranging from domain knowledge to students' misconceptions, all of which missing from today's QG systems. We argue that building effective human-NLP collaborative QG systems that emphasize instructor control and explainability is imperative for real-world adoption. We call for QG systems to provide process-oriented support, use modular design, and handle diverse sources of input.
Decades of educational research has shown the benefit of active learning in improving students' learning outcomes where students actively engage with the materials, e.g., through question answering and problem solving (Crouch and Mazur, 2001; Deslauriers et al., 2019; Koedinger et al., 1997), compared with passively receiving lectures or reading texts (Chi and Wylie, 2014; Freeman et al., 2014). However, most instructors still use traditional teaching methods, e.g., lecturing, in large-enrolment college courses (Henderson and Dancy, 2007; Handelsman et al., 2004; Stains et al., 2018), due to their limited time and resources in creating opportunities for active learning (Dancy and Henderson, 2007;
Silverthorn et al., 2006; Fagen et al., 2002). Meanwhile, there is a growing interest in leveraging NLP methods to serve educational needs. Specifically, automatic question generation (QG) is promising in enhancing students' active learning experience by creating problem-solving activities at scale (Alsubait et al., 2016).
Still, the adoption of automatic QG systems in classrooms is low (Kurdi et al., 2020; Alsubait et al., 2016), mainly because those models are only suitable for specific domains, e.g., language learning, and the generated questions are often of low quality and limited in types and difficulty levels (Kurdi et al., 2020; Alsubait et al., 2016). For QG systems to provide more meaningful support to instructors' work, we need to address at least two fundamental challenges. First, designing questions to assist learning is a highly complex, creative, and knowledge intensive process. Typical college instructors have received years of training on the subject domain, and their question construction process is hard to be fully automated by one single model. Second, classroom is a high-stake environment, and instructors often have pre-defined goals. As a result, they may not prefer imperfect and fallible AI models (Holstein and Aleven, 2021).
To fill the gap between the design of NLP-powered QG systems and the reality in classrooms, we conducted a novel need finding study, with two major goals: (i) To understand how instructors create questions of high educational value, including decision making processes and information sources used throughout the procedure. (ii) To identify key touch points to improve, design, and re-create NLP systems that instructors would find useful in their practice.
Concretely, we interviewed and observed 11 instructors from 7 different universities as they created questions based on readings to support student learning of the content. We piloted the interview study protocol for several rounds, before finalizing
Human Language Technologies, pages 291 - 309
a two-phase protocol. In the first phase, we conducted an in-depth case study of one instructor's quiz design experience for a semester-long course. We extracted the instructor's output and manually labeled the context of each action. We then propose a novel method: text replay enactment, where we replayed the instructor's process of generating the output and annotated what NLP tasks could be used to accomplish the instructor's text transformation operations. This method successfully addresses the challenge that users are unable to directly articulate what they want from AI (Yang et al., 2020). Next, we summarized a list of NLP tasks that this instructor found useful, and used them as design probes in a subsequent interview study with 10 instructors. We then analyzed our data using affinity diagrams. On a subset of these tasks, we reported a qualitative analysis of current NLP models' performance and pointed out major issues that should be addressed to allow classroom adoption.
Our major findings include: 1) None of the instructors interviewed are currently using any type of automatic question generation tools, though they expressed a strong desire to find better ways to design quality questions to support active learning and effective reading. 2) Instructors' natural behaviors and thought processes reveal multiple reasons why existing automatic QG techniques fall short: They leverage multiple sources of information that cover domain knowledge, educational goals, and student misconceptions; They view question creation as an iterative process; They often apply a suite of techniques and strategies. All of these traits are largely ignored by existing end-to-end QG models. 3) Finally, instructors also reported challenges when constructing questions, and showed strong interests in receiving suggestions and support from NLP systems if the outputs are of high quality and interpretable, and that they have sufficient control when interacting with the system.
Based on these findings, we argue that developing effective human-NLP collaborative QG systems is a promising direction, and propose three design implications. First, it is critical to provide process-oriented support to instructors, rather than automating the process end to end. Second, we advocate for modular system design with different NLP components that give instructors strong control over which NLP components to use to meet their goals. Third, we argue that future QG systems should accept diverse data input, including
expert domain knowledge, students' background, educational goals, and pedagogical context.
Finally, we stress the value of understanding stakeholders' needs to better design and deploy NLP systems. As anticipated, users have concerns about how well NLP can do, and are hesitant to use AI-based tools in high-stake environments. Therefore, an important first step in developing human-NLP collaborative systems is to understand user needs and increase user buy-in. As we learned from the interviews, using simple models with reliable performance in the beginning while requesting more user input may be a good approach to bootstrap. We also encourage our community to collaborate with HCI researchers to develop user-friendly interfaces and investigate users' adoption and preferences of NLP tools in context of use.<sup>1</sup>
2.1 Automatic Question Generation (QG)
Here we focus on describing QG models for educational purposes, and refer readers to a detailed literature review in Kurdi et al. (2020). Existing QG systems primarily produce questions that target low-level cognitive skills, such as factual wh-questions and cloze questions that can be answered with short phrases (Heilman and Smith, 2010; Chali and Hasan, 2015; Olney et al., 2012). These simple questions only contain limited concepts (Song and Zhao, 2016) and, as a consequence, offer limited opportunities to control question difficulty level (Alsubait et al., 2016) and are limited for assessment only, leaving many learning opportunities unfulfilled. Additionally, prior work largely develops domain-specific techniques for learning language, math, and medicine knowledge, relying on existing domain ontology (Alsubait et al., 2016; Stasaski and Hearst, 2017).
Our study investigates multiple-choice question (MCQ) generation (Ch and Saha, 2018). MCQs can be graded automatically and offers immediate feedback to students, which has profound benefits to scale active learning. Prior research has shown compelling results on the educational value of MCQs in comparison to open-ended question items. For example, Smith and Karpicke found that students performed equally well on English reading tasks no matter whether they practiced with multiple-choice, short-answer, or hybrid questions
<sup>1</sup>Data, code, and models used in this paper are released at: https://github.com/Olivia-fsm/P2MCQ.
(Smith and Karpicke, 2014). Similarly, multiple-choice questions are shown to provide a win-win situation compared to open-ended cued-recall tests on English reading tasks (Little and Bjork, 2015; Little et al., 2012). The authors find that both open-ended and cued-recall tests foster retention of previously tested information, but multiple-choice tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall tests did not. Recent work (Wang et al., 2021) demonstrates that even for less well-defined domains, MCQs could exercise critical thinking elements and require students to evaluate the quality of MCQ options, especially when common student misconceptions were used as distractors (Wang et al., 2019).
This poses unique challenges in automatic generation of MCQs. First, creating rich-content and meaningful correct options that help students retrieve correct and relevant information. Second, generating plausible distractors that help students internalize incorrect information. Well-designed and meaningful distractors require students to evaluate and contrast options (Wang et al., 2019), that involves going beyond surface traits and considering deeper connecting principles (Schwartz et al., 2011). However, only phrase-level replacement has been commonly studied (Papasalouros et al., 2008) in automatic generation of MCQ options and distractors. Recent automatic QG systems are built on end-to-end trained neural generation models, where a single question is generated from a given context (Sun et al., 2018; Zhou et al., 2019; Zhang and Bansal, 2019). Although questions that require multi-hop reasoning (Pan et al., 2020; Su et al., 2020) or reading long text (Bi et al., 2021; Cheng et al., 2021; Cao and Wang, 2021) have been studied, these systems do not offer control over question difficulty nor consider different sources of knowledge. In this work, we propose to modularize the automatic QG process with different components, e.g., summarization, simplification, or contradiction generation, to offer flexible interface for instructors to control various aspects of the produced questions.
2.2 NLP Tasks for Education
NLP systems have been developed to support a variety of educational applications, including writing assistance (Bellino and Bascuñán, 2020; Frankenberg-Garcia et al., 2018), reading comprehension support (Ross et al., 1991; Vodolazova and
Lloret, 2019; Vajjala and Lucic, 2019; Siddharthan and Katsos, 2010; Chatzipanagiotidis et al., 2021), language learning systems (Tweissi, 1998; Petersen and Ostendorf, 2007; Katinskaia and Yangarber, 2021; Üksik et al., 2021), and generating feedback for programming and design (Wang et al., 2018; Kang et al., 2019). Here we provide a brief overview of the state-of-the-art NLP models that are relevant to the question generation process.
Summarization, the ability of condensing a reading material into a concise passage, has been used for evaluating and improving students' reading comprehension ability (Edmonds et al., 2009; Vaughn et al., 2011; Stevens et al., 2019; Hwang et al., 2019). Similarly, paraphrasing also demonstrates students' content comprehension skills (Haynes and Fillmer, 1984) since it requires the ability of conveying the same semantic meaning with different languages. However, both tasks are rarely studied for question generation (Lyu et al., 2021). Simplification, on the other hand, has demonstrated its values in language learning and reading comprehension (Tweissi, 1998; Inui et al., 2003; Petersen and Ostendorf, 2007; Rets and Rogaten, 2021), and is often treated as a preprocessing step to convert complex sentences into simpler versions before creating questions (Majumder and Saha, 2015; Patra and Saha, 2019). State-of-the-art NLP models for these three tasks are all based on neural generation systems, which are known to suffer from errors and lack of controllability (Maynez et al., 2020). Therefore, their usefulness for QG will largely depend on output quality and new designs with enhanced explainability and easy-to-use interface.
2.3 Human-AI Systems for Education
The concept of human-AI systems is recently introduced into the education domain, where human inputs are continuously solicited and there exists a collaborative relationship between humans (often teachers) and AI algorithms to provide effective instruction to students. They differ from prior AI for education systems where human (teacher and student) input is often elicited before the development of the system (Koedinger et al., 1997; Kumar et al., 2007). Human-AI systems have been developed to help human teachers more easily identify the students that are struggling (Holstein et al., 2018), design higher quality assignment questions (Wang et al., 2019), and offer aggregated feedback to stu
dents' programs (Glassman et al., 2015). They all demonstrate the advantages of combining both humans (robust and flexible) and AI (low-cost and quick) in addressing real-world challenges in education, and it inspires us to develop human-AI systems for QG.
Also relevant to this work is the abundant literature on human-AI interaction (Horvitz, 1999; Amershi et al., 2019; Yang et al., 2020), which highlights the need for considering user's goals and behaviors and points to general design guidelines when prototyping and designing algorithmic experiences. We consider this work to be an instantiation of the idea in a specific context, examining human-NLP interaction when instructors design questions for educational purposes. We expect our findings to contribute to the future development of human-AI systems that address this specific educational problem, and to the greater body of work on human-AI interaction and design principles.
3 Need Finding Study Methodology
We wanted to understand how instructors construct questions that align with their educational needs. We also wanted to probe on when instructors think an intelligent system might offer support or their work. To address these needs, we chose to conduct a qualitative study consisting of observations and semi-structured interviews. It has become a standard HCI approach when designing new software systems that address human needs.
In this work, we conducted an IRB-approved two-phase need finding study. We went through rounds of piloting before reaching at the final protocol, which addressed two challenges that emerged during the pilots. 1) An interview-alone approach is insufficient since it is hard for instructors to recollect all details of question design. In addition, it requires concrete textual input for us to understand how humans use texts when designing questions. 2) Instructors are not able to directly articulate their NLP needs (Yang et al., 2020). To address the first challenge, we propose a specific scenario where instructors design multiple-choice style quiz questions to support active reading. On one hand, it targets an authentic problem since text reading is a passive learning experience that occurs everyday in college classrooms. Active reading opportunities are much needed to support students' comprehension and learning. On the other hand, it is a question design scenario where the text input is ex
plicit. To address the second challenge, we propose a text replay enactment method where we, as NLP researchers, review users' operation on the text and annotate possible NLP tasks that can be applied to make the text transformation.
3.1 Phase 1: Case Study and Replay Enactment of Text
The case study contains an in-depth analysis of one instructor's quiz design experience for a semester-long course on human-computer interaction at the University of Michigan. We will refer to the course as HCI101 for the rest of this paper. The course has 1-2 required readings per lecture, including academic papers and book chapters. The instructor (Instructor A) designed quiz questions for each reading text to facilitate students' reading and understanding of the material.
3.1.1 P2MCQ Dataset
We obtained all questions that Instructor A created during HCI101, including 160 multiple-choice questions with 629 question options in total (197 correct answers and 432 incorrect answers or distractors). These quizzes were manually written based on 30 reading materials (24 conference papers and 6 book chapters) and were assigned throughout HCI101. Example questions can be seen in Figure 2.
Context Annotation One annotator aligned each question option with its supporting context in the original reading material. Both sentence-level and paragraph-level contexts are extracted. For each single option, the sentence-level context include sentences with its supportive evidence, and the paragraph-level context refers to the whole paragraph containing those sentences. During context annotation, questions and options without supportive evidence in the given material (i.e., they were designed based on the instructor's prior knowledge, not from the text) were removed. Instructor A also checked our annotations for accuracy. This resulted in a dataset of 128 multiple-choice questions, including 439 question options (110 target options and 329 distractors). We release the P2MCQ dataset with this paper.
3.1.2 Text Replay Enactment
We are inspired by user-centered design methods, including user enactment (Odom et al., 2012) and replay enactment (Holstein et al., 2020), through which designers imagine possible futures and use
Paper Content
...Our final system consists of an interactive hardware system that replicates an audio speaker (Fig. 1). The system is composed of multiple Hololens devices networked together. Physical electronic modules allow groups of learners to collaborate while observing 3D visualizations of invisible phenomena occurring in the physical space. Sound is produced by a diaphragm membrane with an attached magnet.
Replay Enactment
Entity Replacement
The system allows groups of learners to collaborate while observing 3D visualizations of invisible phenomena occurring in the physical space. Sound is produced by a diaphragm membrane with an attached magnet.
The system provides 3D visualizations of invisible phenomena occurring in the physical space, and sound is produced by a diaphragm membrane with an attached magnet.
The system provides 3D visualizations of invisible phenomena occurring in the physical space and provides audio information.
Contradiction/Negation
The system provides 3D visualizations of invisible phenomena occurring in the physical space but does not provide audio information
Question Options
The system provides 3D visualizations of invisible phenomena occurring in the physical space but does not provide audio information.
Figure 1: An example Text Replay Enactment process. For every context-option pair, we as NLP researchers annotate what NLP modules are needed to generate the final question option from the original context. In this case, 5 NLP modules: Extraction, Entity Replacement, Simplification, Inference, and Negation are needed.
them as conversation starters to elicit users' feedback on technologies that do not exist in users' previous experience. We propose a text replay enactment method where researchers analyze user's written text and the context, to annotate what possible NLP tasks could help the user achieve this outcome. A sample enactment process is shown in Fig 1. To ensure that the annotation process is consistent with the instructor's intention, we started with a reflective interview with Instructor A on a sample of the P2MCQ dataset (10 question stems and 40 options). Instructor A described how they arrived at the question stem and each of the options. We then applied the text replay enactment process on the entire P2MCQ dataset. All four researchers first constructed the annotation scheme collaboratively with  $5\%$  of the dataset. Two researchers then completed another  $13\%$  of the dataset and verified inter-rater reliability. One researcher further completed the annotation. The resulted annotation scheme with NLP tasks and examples can be found in Table 3. These NLP tasks are used in phase 2 to probe into users' needs for NLP support.
3.2 Phase 2: Interview and Observation Study
We recruited participants through social media (including mailing lists and social groups of professors) and offline correspondences. 10 instructors from 7 different universities participated in the study. The instructors have teaching experience ranging from 2 to 40 years and are from disciplines including computer science, information science, data science, education, developmental psychology, and political science.
The interviews were done through Zoom and each lasted between 50 and 75 minutes. Participants were given a $50 Gift Card. We first asked participants to share how they approached reading assignments. The majority of the session was
spent having the participant design questions based on a reading text of their choice. Specifically, we asked them to design questions (MCQ preferred) that could help their students understand and learn from the content. We asked the participants to think aloud throughout the process. We provided question templates sampled from the P2MCQ dataset to help them jump start. Participants were able to design 3 to 10 multiple-choice questions (with one question stem and four options) during the session. We then asked the participant to reflect on how they arrived at each question stem and option, and shared the challenges they had encountered throughout the process. At last, the researcher asked the participant to imagine there being an intelligent system to provide support alongside the quiz design process. For each of the NLP tasks in Table 3, we presented the task in the context of the user's own quiz design experience, and asked the participant to share to what extent did they find the support useful. We transcribed the interview recordings and analyzed our data using affinity diagrams (Moggridge and Atkinson, 2007), which is a commonly used method in HCI to identify emerging themes from qualitative data and discover opportunities for technologies to enhance future.
Through affinity diagramming, we observe emerging themes from the interview data. We present our findings and use participants' quotes to illustrate these themes. We refer to participants as P1-P10.
4.1 Supporting active reading is desirable yet expensive
All participants mentioned that there should be better ways to support students in reading. For example, P5 said "It is definitely a problem that instructors face." Participants shared the techniques
they have used to support reading and the limitations of such approaches. For example, reading summaries is not scalable as grading can be challenging; collaborative annotation platforms such as Perusall (Perusall, 2021) encourage participation but do not necessarily make sure students get the key messages. Most participants expressed interest in using this approach if question design becomes less expensive and more accessible, as P2 said "I wouldn't have the time to do it this way, but if you were to give me the questions, I think it would be amazing."
4.2 Why Automatic Question Generation Techniques Are Insufficient