SlowGuess/NAACL_2022_OCR huggingface dataset:
Instructors often use the questions for specific educational goals, leverage diverse data sources when designing questions, and iteratively improve the content. We point out five categories of reasons on why existing QG techniques are insufficient, as they overlook these critical factors behind making questions of high educational value.
4.2.1 I want the questions to be aligned with my teaching goals
Instructors are more interested in targeting higher-level concepts instead of specific details, when supporting active reading. P2 mentioned "students complain when asking specific questions about the reading, because they feel like we're not testing them for knowledge, we're testing them for obscure pieces of information in a paper." Other instructors also worry that using detail-oriented quiz questions may make the students perceive it as a reading comprehension task and mindlessly answer the questions without thinking about the broader implications of the reading. Instructors require the quiz questions to meet their objectives of assigning the reading and help students "get a gist" and "think deeply" of the material. For example, P10 mentioned "I'm never trying to just see how much they can memorize. We are really trying to get at the conceptual stuff." P1 wanted the students to think critically about the assigned text: "This paper itself has all of these pretty deep flaws, and I want students to see the flaws."
4.2.2 This is not based on the text. It's from my own experience.
An emerging theme from the interviews is that instructors rely on their prior knowledge when designing questions that they think are of educational
value. This is a frequent quote we get from participants "So this [question option] I'm not getting from the article. This is from my own experience." There are three categories of information that instructors leverage: 1) key messages in the text; 2) common student misconceptions; 3) course syllabus and prerequisite relationships.
First, instructors rely on their memory of what are the key takeaways from the text. They would often skim the reading text and say "I want people to get right away [a concept].
Second, instructors use students' common misconceptions when designing questions and distractors. For example, P5 said "Most students got it wrong, because they have a shallow understanding of the content." P10 also made similar comments "Students have trouble distinguishing what's a milestone and what's a mechanism, so I could imagine a question...."
Third, instructors consider syllabus information and think about how the questions are connected to previous or subsequent activities in the course. For example, P2 said "[I'm interested in] quiz questions that are heavily integrating a broad set of concepts throughout the course. I want to test them on the whole understanding of the material...."
4.2.3 The distribution and sequence of questions are also important
In addition to the questions themselves, instructors also keep thinking about the meta-level properties of the quiz, such as the distribution and sequence of questions. E.g., P4 said "I want to make sure my questions cover all three sections in that chapter." P6 paid attention to the sequence of questions as they were in particular interested in using the questions as a reading guide.
4.2.4 It is an iterative process
Another frequent quote from the quiz design processes is that "I'll write it down first and see whether I like it." Multiple participants mentioned that "this may not be the final question" or "I'd like to make this better." Question design is an iterative process where instructors keep revising the text and may jump between questions.
4.2.5 Instructors combine a suite of techniques and strategies
One emerging theme from both phases of our study is that instructors combine a variety of strategies when designing questions and apply different strate
gies across contexts. From text replay enactment, we found that multiple text operations are often needed (in  $57\%$  of the dataset) in order to produce the target text used in the question (Figures 3 and 4).
4.3 Challenges in Human QG
The two mostly mentioned challenges are 1) figuring out the key messages that instructors want students to take away from the text; 2) coming up with distractors (incorrect options). Multiple participants described how identifying the key messages is critical and hard. P4 expressed "I think the challenging thing is to find out the outcome I want students to have after reading the article." P5 also mentioned "I think one challenge is like, I realized there's a lot of information in the reading, and seeing what are the key things I want students to get across." Additionally, all participants found coming up with plausible distractors to be difficult. P5 commented "coming up with the distractors, it is challenging. Because I don't know what would distract the student, but also not be too confusing."
For novice instructors, producing the question stem is difficult. We saw participants often referred to the question templates for ideas. Participants also requested meta-level monitoring, e.g., making sure the difficulty of the questions is reasonable, the questions have a good coverage of the content, etc. Some instructors mentioned it was challenging to ensure that questions are "good". P7 expressed their frustration that "What I always want, and never had in a multiple-choice quizzing tool is a way to say if they've chosen this answer, they have demonstrated these learning objectives and failed to demonstrate these learning objectives." P5 said "maybe I write a question, and then there could be support like 'it could have been framed better.' Bringing in learning sciences principles and suggest how to write good questions would be useful." Finally, some encounter challenges in phrasing and wording. P6 mentioned, "the hardest part for me is to decide how to rephrase it."
4.4 Desires for NLP Support
In general, participants showed positivity towards receiving support in the process, and would want to spend the minimal time possible on question design. Participants had mixed opinions about the proposed NLP support concerning the level of control they could have and the performance of the NLP models. Instructors considered some models
to be beneficial in some cases but not all. We also observed individual differences on which models they found useful. For example, for summarization, P7 did not think it would be useful since the reading text was already very condensed, but P6 and P9 felt that it would help shorten their own rereading time and extract useful information. Participants consider many tasks to be really hard to be automated, and questioned how the models would be able to produce the desirable outcomes. For example, P7 felt that in order to use simplification, the model performance would have to be perfect, because they are concerned with losing important nuances or misrepresenting ideas in the actual text. Ultimately, the participants were interested in trying the system, and thought that it could help them perform QG faster.
5 Evaluation of Existing NLP Tools
Given the observations from §4, here we select popular NLP models for the most frequent text transformation operations used by the instructors and investigate existing issues. We show both instructor-constructed transformations and system outputs in Table 1 in Appendix B. Model implementation and dataset collection details are given in Appendix A.
Here we use the context consisting of one or multiple sentences annotated on the quiz dataset as described in §3. We first conduct sentence selection and abstractive summarization based on BERTSUMEXT and BERTSUMEXTABS models (Liu and Lapata, 2019), which are extractive and abstractive summarization models fine-tuned on news articles.
To build an abstractive summarization model that is suitable for scientific domains, we further fine-tune the sequence-to-sequence BART model (Lewis et al., 2020) on our newly collected HCI article summarization dataset that consists of section-summary sentence pairs.
Given the same context, we then build models for text simplification using ACCESS (Martin et al., 2020) and MUSS (Martin et al., 2021), which are built on top of BERT and BART, respectively; a paraphrasing model that fine-tunes BART using ParaSCI (Dong et al., 2021) which contains paraphrase pairs from scientific papers; a negation generation model based on CROSSAUG (Lee et al., 2021) which fine-tunes BART to produce text that contradicts the given context.
Three major findings are made. For sentence se
lection and summarization models, it is often hard to interpret why the output is produced as is. For example, it is unclear why a sentence is treated as more important and thus selected from a given context. Second, for the generation-driven models, such as abstractive summarization, simplification, and paraphrasing, their output is sometimes only tangentially relevant to the content. The output frequently contains content that cannot be inferred from the input as well as errors, which can hurt instructor experience and even raise ethical concerns. Moreover, the diversity of the generations is rather limited. For example, simplification and negation models tend to focus on changing specific words. Overall, all these NLP models are trained without user needs being specified, thus do not offer control over which topic to summarize, what phrase or content to simplify or paraphrase, and which knowledge to be used as a pivot to create distractors. These issues point to the future directions for building summarization models with explainability, guaranteed faithfulness and correctness, as well as that allow users to exert control over where the transformation should be applied.
6 Implications for Research
Our study reveals instructors' natural processes of constructing questions, the challenges they have, and when they may benefit from NLP support. Specifically, we see a strong desire for user control, where humans provide input to NLP systems and can decide when to use NLP outcomes.
6.1 Implications for Developing Human-NLP Collaborative QG Systems
Recommendation 1: Instead of generating outcomes, providing process-oriented support is more desirable. First, instructors considered it to be highly critical for the questions to serve their teaching goals. As P10 put it "Whatever the goals are, they are shaping those questions, because mostly, the texts, chapters or articles, they were not written with the course in mind. I have to take a reading that was meant for one purpose and pull it to my purpose." When QG systems do not align with instructors' goals, it's hard for them to meet the educational needs and support student learning. Second, all participants viewed question design as an iterative process and preferred to have the opportunity to keep revising and improving content. Third, we observed that instructors had challenges
in doing QG themselves, e.g., when the text was dense, refreshing their memories of the key messages was time-consuming; when they wanted to include a distractor on one concept, figuring out the exact language for an option was hard; instructors may also jump around the whole article and make inferences based on text spread in the article.
We argue that it is more productive for NLP systems to provide process-oriented support to instructors instead of trying to generate complete outcomes immediately. As an example, most instructors liked the idea of highlighting key phrases in the text for the system to identify relevant content. Instructors also liked having the system summarize content for them to decide what questions to design. When asked about using Contradiction/Negation to create distractors, they applauded the idea of telling the system which parts to negate. We see a future direction of building human-NLP collaborative systems where expert input is continuously solicited to tell the system where to pay attention to.
Recommendation 2: Develop QG systems with NLP modules that provide the flexibility of applying modules depending on the context. Instructors had different preferences on which NLP modules to use depending on the goal of the questions, students' prior knowledge, etc. For example, instructors want to use easy questions at times as quick comprehension checks: "This one [question] is actually quite easy. If they read the paper they'll know." Other times, they want to make challenging questions that provoke deep thinking, as we discussed in depth in §4.2.1. Even for a single question, instructors prefer using different strategies to design options. E.g., P5 suggested using Contradiction/Negation to generate one distractor and using Extracting Parallel Concepts to generate another. This ensures that a question contains rich information that benefits student learning. Additionally, we also observed substantive cases where instructors chained multiple NLP modules (as shown in Figure 1) to generate an option.
We argue that framing QG as a single NLP task does not align with educators' process of question creation. Instead, we encourage QG systems to provide modular NLP tools and give instructors the flexibility of choosing which NLP tools to use as they see fit. This approach will also greatly increase instructors' trust to the system and the interpretability of QG systems. We provide a list of NLP tasks that instructors find useful (Table 3), the frequency
of these tasks in the P2MCQ dataset (Figure 3), and the preliminary results on the performance of existing NLP tools on these tasks (Table 2).
Recommendation 3: QG systems need diverse data sources as input. Traditional QG systems mostly use only one text source as input. We observed instructors relying on diverse data sources when designing questions. Some instructors wanted to use student-created examples in questions to reflect common misconceptions. P10 said "Since I have lots of examples of what students have said, I could use student examples and say, which ones of these are good examples?" P7 mentioned an interest in using the whole course content as a data source: "I could imagine a system taking not only the paper in but also taking in the rest of my course content, just as secondary, like what is foreshadowing, what is he going to teach in about two weeks." Even within a single document, we observed that users may aggregate texts from non-adjacent locations as input. We encourage QG systems to take diverse data sources as input, including previous student answers, course syllabus, lecture notes, relevant reading materials, and give instructors the flexibility to select the input sources.
6.2 Implications for Research on Human-Centered NLP
6.2.1 Towards More Robust NLP Outcomes
As detailed in §5, we observed a considerable quality gap between human-generated options and machine-generated ones. First, end-to-end neural models often produce extraneous information. To address this, we propose to solicit user input and train models to focus on user-specified requirements. As an example, for sentence simplification, user may choose to keep several phrases and ask the system to simplify the rest of the content. We encourage researchers to work on models that support and comply with different forms of user control. Second, we argue for modular NLP systems. With the large language models becoming the de facto tool for NLP tasks of varying difficulty levels, dividing the tasks into steps and chaining the outputs could improve the quality of task outcomes (Wu et al., 2021). Our study revealed that instructors applied complex transformations on text when creating meaningful questions. We further proposed a list of pre-defined NLP tasks that instructors frequently employed, and evaluated fine-tuned language models for these tasks. In future work, we
consider the inclusion of user input as part of the modular system design, e.g., allowing users to construct prompts or instructions (Floridi and Chiriatti, 2020; Wei et al., 2021) in-situ to interact with the large language models to better satisfy their needs.
6.2.2 Prototyping Human-Centered NLP Systems
In this study, we also surface challenges in designing and developing human-centered NLP systems. First, users are not able to articulate their needs around NLP. For example, users are not able to say they want the system to make an inference or complete an entity replacement at a certain point. We consider the text replay enactment approach to be effective in helping researchers understand user intentions and desires for NLP and encourage others to further explore and extend this method. Second, we found that having users directly evaluate NLP outcomes is hard, as it often requires them to read the context and outputs by several models. When we tried this in a pilot interview, a user found this to be too cognitively demanding. A more effective way is to visualize the NLP outcomes in context to reduce cognitive load, and investigate user adoption and preferences in real usage. We also encourage collaboration between NLP and HCI researchers in making progress on novel methods to prototype NLP experiences and build usable NLP systems.
QG has been an area of interest in the NLP community. However, the adoption of QG systems in classrooms is low. The goal of this work is to investigate this gap and explore directions for future QG systems that can meet stakeholders' needs. Our work reveals that existing QG systems do not take information that instructors deem critical when designing questions to support learning, including educational goals, and student misconceptions. We surface instructors' desires for receiving support during question design. We make recommendations on how to develop process-oriented, modular, human-NLP collaborative QG systems.
This work is supported in part by National Science Foundation through grant IIS-2046016. We thank all our participants for their time. We thank the anonymous reviewers for their valuable suggestions on various aspects of this work.
Tahani Alsubait, Bijan Parsia, and Ulrike Sattler. 2016. Ontology-based multiple choice question generation. KI-Künstliche Intelligenz, 30(2):183-188.
Saleema Amershi, Daniel S. Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi T. Iqbal, Paul N. Bennett, Kori Inkpen Quinn, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for human-ai interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
Alessio Bellino and Daniela Bascuñán. 2020. Design and evaluation of writebetter: A corpus-based writing assistant. IEEE Access, 8:70216-70233.
Sheng Bi, Xiya Cheng, Yuan-Fang Li, Lizhen Qu, Shirong Shen, Guilin Qi, Lu Pan, and Yinlin Jiang. 2021. Simple or complex? complexity-controllable question generation with soft templates and deep mixture of experts model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4645-4654, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. Controllable open-ended question generation with a new question type ontology. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6424-6439, Online. Association for Computational Linguistics.
Dhawaleswar Rao Ch and Sujan Kumar Saha. 2018. Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies, 13(1):14-25.
Yllias Chali and Sadid A Hasan. 2015. Towards topic-to-question generation. Computational Linguistics, 41(1):1-20.
Savvas Chatzipanagiotidis, Maria Giagkou, and Walt Detmar Meurers. 2021. Broad linguistic complexity analysis for greek readability classification. In BEA.
Yi Cheng, Siyao Li, Bang Liu, Ruihui Zhao, Sujian Li, Chenghua Lin, and Yefeng Zheng. 2021. Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5968-5978, Online. Association for Computational Linguistics.
Michelene TH Chi and Ruth Wylie. 2014. The icap framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4):219-243.
Catherine H Crouch and Eric Mazur. 2001. Peer instruction: Ten years of experience and results. American journal of physics, 69(9):970-977.
Melissa Dancy and Charles Henderson. 2007. Framework for articulating instructional practices and conceptions. Physical Review Special Topics-Physics Education Research, 3(1):010103.
Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019. Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39):19251-19257.
Qingxiu Dong, Xiaojun Wan, and Yue Cao. 2021. Parasci: A large scientific paraphrase dataset for longer paraphrase generation. In EACL.
Meaghan S Edmonds, Sharon Vaughn, Jade Wexler, Colleen Reutebuch, Amory Cable, Kathryn Klingler Tackett, and Jennifer Wick Schnakenberg. 2009. A synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers. Review of educational research, 79(1):262-300.
Adam P Fagen, Catherine H Crouch, and Eric Mazur. 2002. Peer instruction: Results from a range of classrooms. The physics teacher, 40(4):206-209.
Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. *Minds and Machines*, 30(4):681–694.
A. Frankenberg-Garcia, Robert Lew, Jonathan C. Roberts, Geraint Paul Rees, and Nirwan Sharma. 2018. Developing a writing assistant to help eap writers with collocations in real time. *ReCALL*, 31:23 - 39.
Scott Freeman, Sarah L Eddy, Miles McDonough, Michelle K Smith, Nnadozie Okoroafor, Hannah Jordt, and Mary Pat Wenderoth. 2014. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the national academy of sciences, 111(23):8410-8415.
Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller. 2015. Overcode: Visualizing variation in student solutions to programming problems at scale. ACM Transactions on Computer-Human Interaction (TOCHI), 22(2):1-35.
Jo Handelsman, Diane Ebert-May, Robert Beichner, Peter Bruns, Amy Chang, Robert DeHaan, Jim Gentile, Sarah Lauffer, James Stewart, Shirley M Tilghman, et al. 2004. Scientific teaching.
John Earl Haynes and H. Thompson Fillmer. 1984. Paraphrasing and reading comprehension. Literacy Research and Instruction, 24:76-79.