Few-shot Question Generation for Reading Comprehension (2024)

Abstract: According to the internationally recognized PIRLS (Progress in International Reading Literacy Study) assessment standards, reading comprehension questions should require not only information retrieval, but also higher-order processes such as inferencing, interpreting and evaluation. However, these kinds of questions are often not available in large quantities for training question generation models. This paper investigates whether pre-trained Large Language Models (LLMs) can produce higherorder questions. Human assessment on a Chinese dataset shows that few-shot LLM prompting generates more usable and higher-order questions than two competitive neural baselines.

heading: 1 Introduction
text: Given the importance of asking questions for effective learning (Dillon, 2006; Etemadzadeh et al., 2013; Kurdi et al., 2020), there has been extensive effort in developing automatic Question Generation (QG) models to produce high-quality questions for reading materials in educational systems (Heilman and Smith, 2010; Lindberg et al., 2013). Through automatic creation of pedagogical and assessment material, QG benefits teachers by reducing their workload. It also levels the playing field for students, providing them with instant and free access to questions for review and practice.
According to PIRLS (Progress in International Reading Literacy Study), reading comprehension questions should require not only information retrieval, but also higher-order processes such as inferencing, interpreting and evaluation (Mullis and Martin, 2019). However, existing QG benchmarks such as SQuAD (Rajpurkar et al., 2016) mostly focus on factoid short-answer questions. There is therefore a dearth of publicly available training data for the more challenging types of questions (Mulla and Gharpure, 2023) — those requiring inference,
synthesis and critique — especially for languages other than English.
This paper investigates the generation of these higher-order questions with few or no training samples. Our contribution is two-fold. First, we report the first QG evaluation based on PIRLS, an internationally recognized standard for reading comprehension assessment, and demonstrate a high level of human agreement on PIRLS question type classification (Table 1). Second, in experiments on a Chinese dataset, we show that existing QG neural models generate predominantly informationretrieval questions, while few-shot prompting of a Large Language Model (LLM) can generate higher proportions of higher-order questions. The LLMbased approach can therefore produce a balanced set of questions that is desirable in the education setting with minimal supervision.

heading: 2 Previous work
text: Early QG approaches mostly relied on heuristics, linguistic templates and rules (Labutov et al., 2015; Mostow et al., 2016). With the availability of large-scale datasets, QG began to be formulated as a sequence-to-sequence generation task. An encoder-decoder architecture with a global attention mechanism was found to be ef-
21
fective (Du et al., 2017; Kim et al., 2019), but can be further improved with transformer-based approaches (Scialom et al., 2019), and fully finetuned language models (LM) (Xiao et al., 2021). Answer-agnostic QG can be performed via joint Question and Answer Generation (QAG) (Lewis et al., 2021). A QAG model based on fine-tuning encoder-decoder LMs produces high-quality questions (Ushio et al., 2022), but has not been evaluated in terms of question type.
There have been a few QG studies on LLMs in the education setting. On a textbook dataset, few-shot prompting with GPT-3 was able to generate human-like questions ready for classroom use (Wang et al., 2022). A similar approach with InstructGPT achieved an adherence rate between 67% and 69% for generating 9 question types (Elkins et al., 2023). A fine-tuned version of ChatGPT was able to generate questions that are competitive with human ones in terms of readability, correctness, coherence and engagement (Xiao et al., 2023). It remains unknown how these approaches compare to off-the-shelf neural QG models in terms of generating higher-order questions.

heading: 3 Evaluation metric
text: To accurately evaluate the utility and nature of the generated questions, manual assessment is neces-
sary since automatic methods cannot yet reliably determine usability and PIRLS question types.

heading: 3.1 Usability
text: The human assessor assesses the quality of the question on the following three-point scale:
Usable without revision The question can be used as is: it is grammatical, fluent, and relevant for the input passage.
Usable with minor revision The question is relevant for the input passage, but requires improvement in its linguistic quality, e.g., correction of grammatical errors, better vocabulary choice or phrasing.
Unusable The question is irrelevant for the passage, or cannot be understood.
A question classified as one of the first two categories is said to be “usable”. Only usable questions are further analyzed on their question type.

heading: 3.2 PIRLS question type
text: According to the International Association for the Evaluation of Educational Achievement, a reading comprehension question should address one of four comprehension processes, as defined in the PIRLS standards (Table 1):
Retrieval The answer is explicitly given in a text span in the passage.
Inferencing Answering the question requires inferences about ideas or information that is not explicitly stated.
Integrating Answering the question “requires comprehension of the entire text, or at least significant portions of it.” (Mullis and Martin, 2019)
Evaluation The answer “involves a judgement about some aspect of the text”, and is not necessarily found in the passage.
Example questions can be found in Table 2.1 A question classified as Inferencing, Integrating or Evaluation is considered as “higher order”. For pedagogical purposes, a well-balanced set of questions should include not only Retrieval questions but also higher-order ones (Mullis and Martin, 2019).

heading: 4 Approach
text: We adopted the answer-agnostic setting for QG, since the target answer is not always found within the input text. The input is a Chinese text without any specified answer span.

heading: 4.1 Baseline: pipeline model
text: We used the DuReader pipeline QG model (Li et al., 2021), a publicly available QG system for Chinese. It performs two subtasks in sequence: answer generation2 using an extractor trained in the Universal IE framework (Lu et al., 2022)3; followed by question generation4 with a base model fine-tuned with UNIMO (Li et al., 2021).5

heading: 4.2 Baseline: Seq2seq model
text: A seq2seq model, trained directly to generate a question-answer pair from a passage, serves as a second baseline. It has been found to be robust in comparison with the pipeline and multitask approach, and computationally less intensive (Ushio et al., 2023).6 We used the Chinese version of their
1The Chinese passage is taken from a Chinese-language public examinations at https://www.hkeaa.edu.hk/en/sa_tsa/
2https://github.com/PaddlePaddle/PaddleNLP/tree /develop/applications/question_answering/unsupervised_qa
3We used the extractor uie-base-answer-extractor and the filter uie-base-qa-filter
4https://github.com/PaddlePaddle/Research/tree/master/ NLP/UNIMO
5unimo-text-1.0-question-generation 6https://github.com/asahi417/lm-question-generation/
publicly available end-to-end QAG model.7

heading: 4.3 LLM: Zero-shot
text: We used the Chinese version of Stanford Alpaca (Cui et al., 2023)8, a LLaMA Model that can comprehend and execute instructions (Touvron et al., 2023).9 We are not aware of any published research on prompt engineering for Chinese QG. Six candidate prompts, with varied keywords on inference, reasoning, and word usage were informally evaluated on a small set of passages randomly taken from Chinese-language public examinations.10 As shown in Table 7 (Appendix B), the following prompt produced the largest number of usable and non-word-matching questions:
基于给定的文章，生成一个需要推断 的简答题。你的输出应该包含一个简 答问题和这个问题的对应的答案。 文章:<input>
[Translation: “Based on the given passage, generate a short-answer question that requires deduction. Your output should include a question and its answer. Passage: <input>]

heading: 4.4 LLM: Few-shot
text: In the few-shot approach, the prompt above is accompanied with N sample pairs of input passage and question, according to the template in Table 8 (Appendix B). We set N = 5, with all five sample passage-question pairs taken from the public examination papers mentioned above.

heading: 5 Dataset
text: Our evaluation data was drawn from the dev set of DuReader_robust (Tang et al., 2021), a widely used Chinese Q&A dataset11. Due to its filtering step, the pipeline model in Section 4.1 may not
7mt5-small-zhquad-qag 8Chinese-Alpaca-2-7B 9https://github.com/tatsu-lab/stanford_alpaca
10https://www.hkeaa.edu.hk/en/sa_tsa/ 11https://github.com/baidu/DuReader
generate any question for some passages. Our test set consists of the first 200 passages for which the pipeline model successfully produced an output.
Two human assessors, both native speakers of Chinese with a Bachelor’s degree, independently evaluated the questions generated for each of these 200 passages in terms of their usability (Section 3.1) and question type (Section 3.2). A third assessor, a native speaker of Chinese with a Master’s degree, adjudicated in case of disagreement.

heading: 6 Agreement
text: The two assessors agreed 85.0% of the time in the 3-way classification on usability (Section 3.1), leading to a Kappa of 0.739, a “substantial” level of agreement (Landis and Koch, 1977).
In terms of question types, the two human assessors agreed in 93.5% of the cases, yielding a Kappa of 0.861, at the “Almost perfect” level of agreement (Landis and Koch, 1977) The most common disagreement (19 cases) is between Retrieval and Inferencing, typically in judging whether a paraphrase deviates sufficiently from the original expression to require inference. The two assessors also disagreed in 9 cases on whether the answer must be derived from different parts of the passage (Integrating) or from just a single sentence (Inferencing).

heading: 7 Results

heading: 7.1 Usability
text: The LLM-based approaches attained higher usability rates (Table 3). Among questions generated by zero-shot prompting, 62.5% can be used without revision. Few-shot prompting, with only five example passage-question pairs, produced a significant boost, with 71% ready for use without revision. The pipeline and Seq2seq neural models yielded substantially more unusable questions and fewer questions that are immediately ready (35.0%). The amount of unlabeled language data used in training — an order of magnitude larger in LLMs than the
neural models — likely contributed to the grammaticality and fluency of the generated questions.

heading: 7.2 PIRLS question types
text: Both neural QG models produced very limited number of higher-order questions, likely because there were few such questions in the training samples. Despite the lack of such samples, zero-shot LLM produces substantially more higher-order questions (29.5%), and few-shot prompting further increases the proportion (31.5%) (Table 4). It appears that Alpaca was able to learn the characteristics of higherorder questions even with only five samples.

heading: 8 Conclusion
text: Higher-order questions are important for assessment in reading comprehension. However, there is a lack of publicly available datasets of these challenging questions in languages other than English. This paper has presented the first study on automatic question generation (QG) for reading comprehension based on PIRLS, assuming no or minimal supervision. Experiments on Chinese passages show that zero-shot LLM produces more usable and more higher-order questions than two competitive off-the-shelf neural QG models, and few-shot prompting further improves the performance.
In future work, we plan to investigate tailored prompts for producing the different PIRLS question types, and to construct a Chinese dataset of higherorder questions for fine-tuning an LLM.
Limitations
The evaluation has focused on the quality of the questions, but cannot show their pedagogical impact on the students. At the time of system deployment, users should be clearly informed that the automatically generated questions should be viewed only as a first draft, to minimize the risk that the teacher may fail to edit an unusable question and pass it to students.

heading: Acknowledgements
text: We gratefully acknowledge support from the eLearning Ancillary Facilities Programme of the Quality Education Fund (Project “Knowledge Overlord - A Self-sustaining AI Game-based Online Platform to Enhance Student’s Literacy Ability and 21st Century Skills”); and from a Teaching Development Grant at City University of Hong Kong (project 6000834).

heading: A Appendix: Instruction to Human Assessors
text: The human assessors gave consent to the data collection and were informed that the results would
remain anonymous. They were shown the following instructions: <passage> <question>
1. Is the question understandable and relevant for the passage?
2. Does the language quality of the question need to be improved?
3. If the answer to #1 is “Yes”, choose one of the categories for the question:
• Retrieval (Focus on and Retrieve Explicitly Stated Information)
• Inferencing (Make Straightforward Inferences)
• Integrating (Interpret and Integrate Ideas and Information)
• Evaluation (Evaluate and Critique Content Textual Elements)

heading: B Appendix: Prompt selection and implementation
text: Table 6 lists the six prompts that were evaluated. The top of Table 7 shows zero-shot evaluation results on a set of 42 passages randomly chosen from public examinations on the Chinese-language subject in Hong Kong.12 “Creative” refers to the parameter values {temperature=0.8, top_p=1}. Prompt #3 was found to produce the highest proportion of usable questions and questions that are not word-matching in nature.
The bottom of Table 7 shows the tuning of the temperature and top_p values. “Conservative” refers to the values {temperature=0.5, top_p=0.5}; “Less Creative” refers to the values {temperature=0.6, top_p=0.9}. We empirically set the temperature and top_p values at 0.6 and 0.9 in the rest of the experiments since they produced more usable and non-word-matching questions than the other values.
The few-shot template is shown in Table 8.
12https://www.hkeaa.edu.hk/en/sa_tsa/

heading: Future Work
text: In future work, we plan to investigate tailored prompts for producing the different PIRLS question types, and to construct a Chinese dataset of higherorder questions for fine-tuning an LLM. Limitations
The evaluation has focused on the quality of the questions, but cannot show their pedagogical impact on the students. At the time of system deployment, users should be clearly informed that the automatically generated questions should be viewed only as a first draft, to minimize the risk that the teacher may fail to edit an unusable question and pass it to students