Planning First, Question Second: An LLM-Guided Method for Controllable Question Generation (2024)

Abstract: In the field of education, for better assessment of students’ abilities, generated questions often need to meet experts’ requirements, indicating the need for controllable question generation (CQG). However, current CQG methods mainly focus on difficulty control, neglecting the control of question content and assessed abilities, which are also crucial in educational QG. In this paper, we propose an LLM-guided method PFQS (for Planning First, Question Second), which utilizes Llama 2 to generate an answer plan and then generates questions based on it. The plan not only includes candidate answers but also integrates LLM’s understanding and multiple requirements, which make question generation simple and controllable. We evaluate our approach on the FairytaleQA dataset, a well-structured QA dataset derived from child-friendly storybooks. In the dataset, the attribute label represents content control, while the local_or_sum and ex_or_im labels denote difficulty control. Experimental results demonstrate that our approach outperforms previous state-of-the-art results and achieves better consistency with requirements compared to prompt-based method. Further application of our method to Llama 2 and Mistral also leads to improved requirement consistency in a zero-shot setting.

heading: 1 Introduction
text: Educational studies over the years have demonstrated that asking questions foster reading comprehension skills, critical thinking skills and writing skills (Godfrey, 2001; Etemadzadeh et al., 2013; Joseph et al., 2016). In order to maximize their educational value, questions posed in the teaching and learning activity should be reasonably designed in terms of content and diversified by the levels of questioning (Shanmugavelu et al., 2020). Recent advancements in natural language processing have
∗Corresponding author.
spurred active exploration of question generation (QG) systems with a focus on educational applications (Xu et al., 2022; Yao et al., 2022; Zhao et al., 2022). Besides, several studies have recognized the importance of diversity (Eo et al., 2023; Yoon and Bak, 2023) and difficulty (Cheng et al., 2021; Bi et al., 2021; Uto et al., 2023) of generated QA pairs, which are key factors in educational QG.
In certain educational settings, such as exams, it is crucial that automatically generated questions derived from a given context are not only pertinent to the context, but also capable of assessing students’ diverse abilities (Francis et al., 2005) and can be controlled in terms of difficulty (Bachman, 1990; Benedetto et al., 2023). However, meeting these practical requirements presents challenges for educational QG. For instance, given the sample in Table 1, existing QG methods usually overlook labels of questions which are annotated by experts,
4715
typically focusing on either content or difficulty control of questions, but not both simultaneously. Furthermore, current research lacks a unified standard for defining and measuring difficulty, and a single criterion for difficulty may not be sufficiently reasonable.
To alleviate the above limitations, we propose an LLM-guided method PFQS (Planning First, Question Second) that enhances the controllability of generated questions in terms of content and difficulty simultaneously. Our approach consists of two steps. Firstly, we utilize Llama 2 (Touvron et al., 2023) to generate an answer plan, a concept proposed in this paper, containing various information on candidate answers. In addition to candidate answers and answer-containing sentences extracted by Llama 2, the plan in our method contains various control information to ensure controllable question generation. The inclusion of answer-containing sentences is inspired by Back et al. (2021). In terms of control information, we prioritize content control followed by difficulty control and express control information in various forms, as content control is typically linked to candidate answer selection, while difficulty control is based on these answers. Secondly, we combine designed prompt, generated plan and given context together and feed them into QG models to generate QA pairs. Furthermore, we design several prompts to enable large language models (LLMs) to assist in evaluating whether the content and difficulty of the generated questions meet predefined requirements.
As the example shown in Table 1, questions generated by PFQS are diverse and exhibit higher consistency with gold questions and labels in the dataset. Extensive Experimental results demonstrate that our PFQS method significantly outperforms the existing state-of-the-art method, with an improvement of up to 0.254→0.413 on MAP@1 with RougeL (Lin, 2004) F1 and 0.8783→0.8965 on MAP@1 with BERTScore (Zhang et al., 2019) F1. Evaluations also indicate that PFQS ensures better requirement consistency for both small and large language models compared with common prompt-based method, implying the value of our approach in the era of LLM. The main contributions of this paper are summarized as follows.
• We propose a novel QG method (PFQS), which generates questions based on LLMgenerated answer plan. The plan contains more information than usual prompt and is
more beneficial for QG.
• We address both content and difficulty control in QG by incorporating various control information in a specific order and form during the planning process in PFQS to effectively utilize the power of control information.
• We conduct experiments on FairytaleQA. The results show that PFQS remarkably outperforms previous state-of-the-art results, improving requirement consistency for both small and large language models.

heading: 2 Related Works

heading: 2.1 Candidate Answer Selection
text: Candidate answer selection in question generation (QG) involves selecting potential answers for generating high-quality and relevant questions. Various QG methods (Duan et al., 2017; Subramanian et al., 2018; Yao et al., 2022) extract named entities, noun phrases or key phrases as candidate answers. However, these methods often limit the diversity of candidate answers by focusing only on explicit extractions. To tackle this issue, a recent QG method (Eo et al., 2023) trains an answer generator to produce candidate answers. Additionally, several QG methods (Tang et al., 2017; Dugan et al., 2022) simultaneously train QA and QG models to enhance the relevance between questions and answers, implicitly improving answer selection. In this paper, instead of introducing another model or employing multi-task learning, we leverage the capabilities of LLM to generate an answer plan containing diverse and relevant candidate answers.

heading: 2.2 Educational Question Generation
text: The task of educational question generation aims to automatically generate natural language questions for educational purposes. FairytaleQA (Xu et al., 2022) stands out as a representative dataset in educational QG. The questions in this dataset, annotated by experts, are suitable for evaluating children’s reading comprehension skills. Based on this dataset, Yao et al. (2022) present a threestep pipeline that first extracts candidate answers, then generates appropriate questions, and finally ranks the generated question-answer pairs. Dugan et al. (2022) train a QG model based on T5 (Raffel et al., 2020) by multi-task learning, resulting in questions with improved relevance, interpretability, and acceptability. Zhao et al. (2022) propose
a QG framework that involves learning question type distribution and event-centric summarization to address the high-cognitive demand in question generation. Additionally, recent studies (Eo et al., 2023; Yoon and Bak, 2023) shed light on the significance of diversity in question generation. In this paper, we explore a wider array of factors that influence the educational value of generated questions, including their content and difficulty.

heading: 2.3 Controllable Question Generation
text: Controllable question generation (CQG) has two main categories: type controllable question generation (TCQG) and difficulty controllable question generation (DCQG). TCQG is a less-explored task (Cao and Wang, 2021). Cao and Wang (2021) and Gao et al. (2022) define different question type ontologies to generate type-aware questions. DCQG is a relatively new task (Kurdi et al., 2020). The definition of question difficulty has not been consistent over the past few years. Gao et al. (2018) use R-Net (Wang et al., 2017) and BiDAF (Seo et al., 2016) to assess question difficulty and use an LSTM-based model to generate questions. Cheng et al. (2021) defines question difficulty as the number of inference steps needed to answer a question and propose a graph-based framework for question generation. Bi et al. (2021) design five domain-independent features to measure complexity and incorporate soft templates and deep mixture of experts (Shen et al., 2019) to generate difficulty-controllable and high diversity questions. Srivastava and Goodman (2021) and Uto et al. (2023) take learner’s ability into consideration and generate questions with appropriate difficulty for each learner. In this paper, we consider both content (similar to type) and difficulty control of generated questions, which are key factors in educational QG.

heading: 3 Method
text: Our question generation method comprises two main steps: plan generation and question generation. In the first step, we use an instructiontuned LLM, specifically Llama 2 (Touvron et al., 2023), to generate an answer plan based on the given context and predefined requirements (i.e. expert-annotated labels in the dataset, where the attribute label represents content control, and the local_or_sum and ex_or_im labels denote difficulty control). The plan typically includes several points that serve as candidate answers or answer-
containing sentences extracted by the LLM, incorporating content and difficulty control information. In the second step, the generated plan, along with the context and the designed prompt, is fed into QG models to generate question-answer pairs. The overall architecture of our method is illustrated in Figure 1.

heading: 3.1 Plan Generation
text: Initial Plan Generation During the initial plan generation process, we use prompts to guide the LLM in generating answer plans based on attribute labels of questions in the dataset. The detailed design of prompts can be found in Appendix A.1. Mathematically, given the context c, the attribute label att, the prompt template T init, we firstly obtain LLM’s input T init(c, att). Subsequently, the initial plan P init = (p1, p2, . . . , pT initp ) (T init p denotes the number of points in P init) can be generated by the following expression:
P init = LLM(T init(c, att)) (1)
Plan Fusion After discussing the initial plan generation, the next paragraphs will focus on obtaining the key point and fusing the two types of plans.
The process of generating the key point is similar to the initial plan generation. We use the LLM to identify answer-containing sentences by designing prompts. The detailed design of the prompts is available in Appendix A.2. With the context c, the question q, the answer a, and the prompt template T key, we can get LLM’s input T key(c, q, a). Then, the key point P key can be expressed as follows:
P key = LLM(T key(c, q, a)) (2)
After the initial plan and key points are generated, they need to be fused together. This step is necessary because some points in the initial plan are not suitable for generating questions, and these points are expected to be filtered in this step. Given the initial plan P init = (p1, p2, . . . , pT initp ) and the key point P key, the fused plan is calculated as shown in Equation (3).
P fus = {pi|sim(pi, P key) ≥ t, pi ∈ P init} (3)
where sim(·, ·) is a semantic similarity function (for which we use the cosine similarity) and t denotes the similarity threshold. In this expression,
t is not a fixed value. We set t = t̂ which makes |P fus|t=t̂ = min(T initp , 5).
Note that plan fusion only occurs during the training phase since gold question-answer pairs are unavailable in the testing phase. Moreover, not all initial plans undergo fusion in a training epoch. About 30 percent of initial plans are fused with key points in each epoch.
Further Revision So far, either the initial plan or the fused plan only considers the attribute label (content control), ignoring the difficulty control of questions, which is also crucial in educational QG. In this part, we mainly focus on local_or_sum (answer involves one or multiple sentences) and ex_or_im (answer is explicit or implicit) labels in the dataset, which represent the difficulty of questions from two perspectives. The following paragraphs will introduce how to incorporate these two labels into the existing plan.
Given a plan P = (p1, p2, . . . , pTp) and a source context c = (s1, s2, . . . , sTc), where Tp denotes the number of points in P and Tc denotes the number of sentences in c. Let ls and ei denote local_or_sum and ex_or_im label respectively. In dataset, ls ∈ {local, summary}, ei ∈ {explicit, implicit}. For the four value combinations of ls and ei, the plan will be adjusted in different ways.
Let sim(·, ·) be a semantic similarity func-
tion. For each pi ∈ P , we find the sentence in context which is most semantically similar to pi. This sentence’s index is denoted as mi = argmaxj sim(pi, sj). In addition, sentences next to sj are defined as S̃j = {sj−1, sj , sj+1}. We will use these representations for the following instructions.
Case 1: ls = local, ei = explicit. We aim for each point of the plan to match a sentence in the context. For each pi ∈ P , if sim(pi, smi) > t1 (default value of t1 is 0.6), then replace pi by smi .
Case 2: ls = local, ei = implicit. Unlike Case 1, it is preferable for points in the plan not to be overly similar to sentences in the context. For each pi ∈ P , if sim(pi, smi) > t2 (default value of t2 is 0.5), then paraphrase pi using LLM. The Details of paraphrasing are presented in Appendix A.3.
Case 3: ls = summary, ei = explicit. Different from Case 1 and 2, for each pi ∈ P , we consider both smi and the sentences next to it. Let Si = {s|sim(pi, s) > t3, s ∈ S̃mi} (default value of t3 is 0.5). If sim(pi, smi) > t1 and |Si| > 1, then replace pi with sentences in Si in their original order in c.
Case 4: ls = summary, ei = implicit. For each pi ∈ P , similar to the above, let S′i = {s|sim(pi, s) > t4, s ∈ S̃mi} (default value of t4 is 0.45). If sim(pi, smi) > t2 and |S′i| > 1, then paraphrase pi using LLM. The Details of paraphrasing are presented in Appendix A.3.

heading: 3.2 Question Generation
text: Before question generation, we design a basic prompt which includes all three labels in FairytaleQA. The detailed design of prompt is shown in Appendix A.4. We combine the prompt, the answer plan and the given context together as input for QG model.
Let att, ls and ei denote the attribute, local_or_sum and ex_or_im labels respectively. Given the prompt template T qg, we firstly get the prompt a = T qg(att, ls, ei). Additionally, given the context c and the plan p, the QG task in this paper can be defined as generating a question-answer pair ŷ, such that:
ŷ = argmax y
P (y|a, p, c)
= argmax y
P (y|x) (4)
where P (y|x) is the conditional log-likelihood of the predicted sequence y, given the input x = (a, p, c).
Giving a training corpus: D = {(x(i), y(i))}|D|i=1 where x = (a, p, c), the QG model’s training objective is to minimize the negative log-likelihood of the training data with respect to all the parameters, denoted by θ.
L = − |D|∑
i=1
logP (y(i)|x(i); θ)
= − |D|∑
i=1
|y(i)|∑
j=1
logP (y (i) j |x(i), y (i) <j ; θ)
(5)
Once the model is trained, we do inference using beam search. If the QG model is expected to generate only one QA pair, the model’s input remains the same as that during the training process. However, if the QG model is required to generate several QA pairs, the answer plan needs some modification before being fed into the QG model. The final plan consists of several points randomly selected from the original plan, and we believe that such a plan is helpful for enhancing the diversity of the generated questions.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Dataset We utilize the FairytaleQA (Xu et al., 2022) dataset in our experiments. FairytaleQA is
a high-quality dataset created specifically for children’s reading comprehension. It comprises 10,580 questions of varying difficulty sourced from 278 children-friendly stories, covering seven types of narrative elements. The training, validation and test sets contain 8,548, 1,025 and 1,007 QA pairs respectively. To generate questions with controlled content and difficulty, we utilize narrative element labels (i.e. attribute) and difficulty labels (i.e. local_or_sum and ex_or_im) in the dataset, which are annotated by experts.
Metrics In line with previous studies, we adopt the MAP@N score as the primary metric in our main experiment. We use MAP@N with Rouge-L (Lin, 2004) and MAP@N with BertScore (Zhang et al., 2019) to evaluate token-level similarity and semantic similarity between generated and groundtruth QA pairs. Additionally, to evaluate the effectiveness of our control over question generation, we assess the consistency between the generated questions and the annotated labels (the attribute label for content control, and the local_or_sum and ex_or_im labels for difficulty control) by Llama 2 (Touvron et al., 2023) and designed rules. Related details are provided in Appendix B.
Baselines In our experiments, we compare our PFQS method with four baselines. 1) FQAG (Yao et al., 2022), which generate questions through a three-step pipeline; 2) SQG (Dugan et al., 2022), which utilizes summaries of given context to generate questions; 3) DQAG (Eo et al., 2023), which enhances the diversity of generated questions by producing different interrogative sentences and implicit/explicit answers; 4) BART-large (Lewis et al., 2020), which is trained on FairytaleQA without any expert-annotated labels. It is important to note that all these baselines are designed for answeragnostic question generation and are expected to generate QA pairs instead of solely generating questions based on answers. Consequently, several QG methods that generate questions alone or generate questions based on given answers are not included in baselines.

heading: 4.2 Main Results
text: The performance of baselines and our method are presented in Table 2 with the following main insights.
Result on MAP@N with Rouge-L As shown in Table 2, in terms of MAP@N with Rouge-L,
our PFQS method demonstrates a significant performance improvement over four baseline models across all splits and top-N results. Especially in the test set, PFQS outperforms DQAG by +0.044 in the MAP@10, +0.081 in the MAP@5, +0.115 in the MAP@3, and +0.159 in the MAP@1, representing a noteworthy advancement.
Result on MAP@N with BERTScore In terms of MAP@N with BERTScore, our method also achieves higher performance in all settings except for the MAP@10 test result. In the best case of MAP@10, DQAG outperforms FQAG and SQG by +0.0079/+0.0101 and +0.0100/+0.0116 respectively in valid/test split, indicating generated QA pairs of DQAG are semantically better than those of FQAG and SQG. By comparison, our method records 0.9198/0.9173 in valid/test split (MAP@10), achieving results comparable to DQAG. This means that questions generated by our method are also high-quality semantically. Additionally, our method outperforms DQAG by +0.0053 in the MAP@5, +0.0105 in the MAP@3, and +0.0182 in the MAP@1 in the test result of BERTScore, the tendency of which is consistent with the MAP@N with Rouge-L F1 result.

heading: 4.3 Ablation Study
text: In Table 3, we investigate the effects of prompt and plan on the model’s performance. Specifi-
cally, by comparing the results of BART-large + Prompt and BART-large + Plan, we observe that prompt has a greater impact on improving MAP@1 scores than plan for BART-large, indicating that the prompt enhances QG model’s accuracy in question generation. By comparison, after adding the plan, BART-large’s MAP@3, MAP@5 and MAP@10 scores significantly increase, suggesting that the plan enables QG model to generate a wider variety of questions. Furthermore, BART-large performs better when both prompt and plan are added simultaneously than when either of them is added separately. Therefore, in our PFQS method, we input both prompt and plan into QG model for better performance. Finally, by comparing the results of BART-large + Plan with and without fusion, it is evident that fusion operation positively impacts the improvement of MAP@N scores, regardless of whether prompt is added or not.
The purpose of conducting experiment of SLMQ is to visually demonstrate the distinction between prompt and plan. In Table 3, only BART-large + Prompt performs worse on SLMQ than MAP@1 in both Rouge-L and BERTScore (0.394→0.385, 0.8942→0.8908 respectively), implying the following insights: 1) BART-large + Prompt struggles to generate diverse questions from a fixed input, resulting in lower SLMQ and MAP@3, MAP@5 and MAP@10 scores; 2) BART-large + Prompt achieves a high MAP@1 score on non-SLMQ
datasets, suggesting it excels at question generation without the need for diversity; 3) BART-large + Plan exhibits a completely different tendency in all top-N outcomes compared with BART-large + Prompt and is better at question generation requiring diversity.

heading: 4.4 Label Consistency Evaluation
text: Previous studies have overlooked the importance of expert-annotated labels of questions in FairytaleQA. These labels, however, are vital for a comprehensive assessment of students’ abilities and for controlling question content and difficulty. Specifically, attribute label covers seven types of narrative elements and questions about different narrative elements typically reflect different question content and assess students’ various abilities. For instance, questions about settings typically assess students’ ability to extract information, while those on causal relationships often require a deeper understanding of context. Besides, local_or_sum and ex_or_im labels indicate question difficulty from two dimensions. A method which generates questions more consistent with these two labels is considered to be better at controlling question difficulty.
BART-large In Table 4, we mainly evaluate the impact of our PFQS method on label consistency after it is applied to BART-large. It is observable that BART-large + Plan generates questions which are better consistent with attribute label while BART-large + Plan (adjusted) generates questions with better consistency regarding local_or_sum and ex_or_im labels. This result is easily understood, as the initial plan only contains information on narrative elements (corresponding to attribute label) and the adjusted plan takes all three labels into consideration. Additionally, by comparing results of BART-large with and without prompt, it is indicated that prompt performs differently when the plan is adjusted or not. Because the initial plan only contains information on narrative elements and the prompt contains all three labels, BART-large with prompt and plan achieves obviously better label consistency on local_or_sum and ex_or_im than BART-large with plan only. By comparison, adjusted plan contains information on all three labels itself, as a result, prompt cannot bring much help in this case.
Llama 2 and Mistral In Table 5, we apply our PFQS method to Llama 2 and Mistral in a zero-shot manner, evaluating its impact on label consistency.
Overall, the addition of prompt and plan, especially plan, notably enhances LLM’s performance on label consistency, such as 0.8669→0.9017 on local_or_sum label for Llama 2 and 0.5938→0.7259 on ex_or_im label for Mistral. For both Llama 2 and Mistral, models with adjusted plan exhibit the best performance on control difficulty and are comparable to other models in terms of consistency with attribute. Furthermore, by comparing results of two LLMs with and without prompt and referring to results in Table 4, we observe that the impact of prompt on LLMs resembles that of prompt on BART-large. On the one hand, the prompt indeed enhances label consistency when added to Llama 2/Mistral + Plan, due to the fact that the prompt can compensate for local_or_sum and ex_or_im information not contained in initial plan. On the other hand, the addition of prompt cannot improve the performance of Llama 2/Mistral + Plan (adjusted), and may even lead to a decrease in performance. This is because adjusted plan contains information about all three labels and the addition of prompt does not provide any new information.

heading: 4.5 Other Analysis
text: Performance of Multiple QG Models We investigate our PFQS method’s impact on multiple QG models. We choose BART-large (Lewis et al., 2020), T5-base (Raffel et al., 2020) and flan-T5base (Chung et al., 2022) respectively as QG models for experiments and explore the performance of these QG models in four different situations: 1) trained directly on FairytaleQA; 2) with prompt added; 3) with plan added; 4) with both prompt and plan added.
According to the experimental results in Table 6, both prompt and plan contribute to the enhancement of QG model performance, but through different ways. It is indicated that prompt has a stronger effect on improving MAP@1 scores, while plan facilitates the generation of a wider variety of questions and leads to improved performance in MAP@3, MAP@5 and MAP@10. Furthermore, all these three QG models perform better when both prompt and plan are added. Finally, note that above conclusions are applicable to all three models, implying the robustness of our PFQS method.
Performance of Plan Generation Methods We investigate various methods to generate planning. These plans are fed into the BART-based QG model along with given context, and question-answer pairs will be generated. Descriptions of plans in the table are as follows. 1) Initial: corresponding to initial plan P init in the method section and generated by Equation (1); 2) Fused: corresponding to fused plan P fus in the method section and generated by Equation (3) (utilized only in training phase); 3) Filtered: same as fused plan, but used in both train-
ing and testing phase; 4) Random: consisting of several points randomly selected from initial plan; 5) Gold: corresponding to key point P key in the method section and generated by Equation (2); 6) Initial+Gold: derived from the connection of initial plan and key point.
In Table 7, as plans have fewer points and become more precise (Initial→Filtered→Gold), the MAP@1 score of the models increases, but MAP@3, MAP@5 and MAP@10 scores decrease. Due to this trend, we ultimately opt for a fused plan for question generation, which retains most of the content of the initial plan and incorporates some information from the filtered and gold plans. In this way, the fused plan leads to improvements in all MAP@N scores compared to the initial plan. Additionally, it is observable that incorporating both initial and gold plans yields significant improvements in all outcomes. However, realistically, gold plan cannot be added. Finding a way to incorporate this type of information would be a meaningful area for future work.

heading: 5 Conclusion
text: In this paper, we propose PFQS, a novel QG method with LLM-guided answer planning at its core. Unlike existing QG methods, PFQS utilizes LLM to generate an answer plan for candidate answer selection, rather than training another model. The generated plan extends the standard prompt, typically including candidate answers, LLM’s understanding, and control information, all of which contribute to question generation. Experimental results indicate that PFQS achieves outstanding performance and integrating the plan enhances the diversity and label consistency of the QG model, in comparison to using a standard prompt. Our approach opens up new possibilities for incorporating
more information into prompts with the assistance of LLM, instead of training another model when faced with detailed or complex requirements in controllable question generation tasks.
In the future, with the emergence of more and more educational question generation datasets with expert-annotated labels, the planning first, question second idea in our QG method may gradually show its application value.
Limitations
We only evaluate our method and other compared methods on FairytaleQA, as our method requires suitable annotated labels in the dataset for generating answer plans. If there are more datasets with expert-annotated labels, we believe that our method will be more comprehensively evaluated. In addition, we attempted several methods to build a ranker model for helping QG model achieve higher MAP@1 score. However, these ranker models did not bring any stable improvement in either RougeL or BERTScore. We believe that a robust ranker model or a dataset including positive and negative samples will be good future works in educational question generation.
Ethics Statement
In this paper, we propose an LLM-guided method to improve the ability of QG model to generate diverse and label-consistent questions. The dataset (FairytaleQA) and models (BART-large, T5base, flan-T5-base, Llama-2-7B-chat, Mistral-7Binstruct-v0.1, all-MiniLM-L6-v2 and RoBERTalarge) we use are all public and all the references drawn from the work of others are marked with citations. During experiments, random seeds are chosen completely randomly and remain fixed across different configurations of model, so that few biases or discriminations are introduced in experiments. Finally, the plan, prompt and questions are all generated based on the text or labels in FairytaleQA and do not contain harmful information, so there are no ethical issues in our work.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their valuable feedback and helpful comments. This work was supported by the National Natural Science Foundation of China (No.62277002).

heading: A Design of Main Prompts
text: A.1 Initial Plan Generation
In this section, we show prompts which are used to guide Llama 2 to generate initial plan. In FairytaleQA, there are totally seven attribute labels of questions annotated by experts. We first design a prompt template as shown in Table 8.
To fill the template, we replace [CONTEXT] token by given context and replace [ATTRIBUTE] token using mapping rules shown in Table 9.
A.2 Key Point Generation
In this section, we present prompts used to let Llama 2 generate answer-containing sentences which is so-called key point. The prompt template is shown in Table 10.
To fill the template, we replace [CONTEXT], [QUESTION] and [ANSWER] token by given context, gold question and gold answer respectively.
A.3 Paraphrasing
In this section, we present prompts used to let Llama 2 paraphrase given sentences. The prompt template is shown in Table 11.
When use the template, we replace [SENTENCE] token by given sentence and feed the whole prompt into Llama 2.
A.4 Question Generation
In this section, we present a prompt template used for QG models. The prompt template includes three labels in dataset and is shown in Table 12.
To use the prompt template, we replace [ATTRIBUTE] and [EX_OR_IM] token by attribute and ex_or_im label respectively and modify [LOCAL_OR_SUM] token using mapping rules shown in Table 13.

heading: B Label Consistency Evaluation Details
text: In this section, we introduce how to evaluate the consistency between generated questions and annotated labels (i.e. attribute, local_or_sum and ex_or_im). For attribute and local_or_sum label, prompt templates are shown in Table 14.
When use the template, we replace [CONTEXT],
[Q&A] and [ATTRIBUTE] token by given context, pair of question and answer and attribute respectively, then feed the whole prompt into Llama 2.
For ex_or_im label, we find that Llama 2 cannot distinguish explicit and implicit answers very well. Moreover, whether an answer is explicit or implicit mainly involves the comparison and judgment on token level. As a result, we propose the following rules to judge given answer is explicit or implicit:
A = #word appearing in both answer and context
B = #word in answer
ifA ≤ B/2, then the answer is implicit; ifA > B/2, then the answer is explicit.
where #word denote the number of words without stopwords.
C Implementation Details
In our PFQS method, we use Llama-2-7B-chat (Touvron et al., 2023) model to generate answer plans. In the main experiment, following previous studies, we initialize the QG model with pretrained BART-large (Lewis et al., 2020). Hyperparameters are follow: learning rate = 2e-5; batch size = 24; epoch = 10. Besides, we select the commonly used sentence-transformer model, all-MiniLM-L6v2 (Reimers and Gurevych, 2019) to encode texts and compute cosine similarity between texts. Our code is implemented on Huggingface (Wolf et al., 2020), whereas Adam (Kingma and Ba, 2014) is used for optimization. All models are trained on 1 Nvidia-A100-40G GPU, and most models can be trained within an hour. In some experiments, to evaluate impacts of our PFQS method on LLMs, we use Llama-2-7B-chat (Touvron et al., 2023) and Mistral-7B-instruct-v0.1 (Jiang et al., 2023) model for question generation without training. Additionally, We use RoBERTa-large (Liu et al., 2019) model for BERTScore. For each configuration of our method and all compared methods, we conduct 5 independent runs and report the average score.

heading: D Case Study
text: Table 15 and Table 16 present two complete examples generated by applying our method to the BART-large model. These two examples show a typical situation of plan generation, where some points are helpful for generating questions, while others are less suitable. Specifically, multiple questions are generated based on some points (point 3 in both case 1 and 2), but there is no question generated based on certain points. This phenomenon indicates that the QG model has the ability to filter plan instead of generating one question for each point of plan. Besides, it is observable that generated QA pairs in two cases have high quality and diversity.

heading: E List of Software and Data Licences Used in this Work
text: Main dependencies for our method are as follows. They are all public and free for research use.
• FairytaleQA: https://github.com/ uci-soe/FairytaleQAData, under an Apache License 2.0.
• Huggingface Transformers: https:// github.com/huggingface/transformers/ blob/master/LICENSE, under an Apache License 2.0.
• Huggingface Datasets: https: //github.com/huggingface/datasets/ blob/master/LICENSE, under an Apache License 2.0.
• Huggingface Evaluate: https: //github.com/huggingface/evaluate/ blob/main/LICENSE, under an Apache License 2.0.
• Pytorch: https://github.com/pytorch/ pytorch/blob/main/LICENSE, Misc.
• NLTK: https://github.com/nltk/nltk/ blob/develop/LICENSE.txt, under an Apache License 2.0.
• Llama 2: https://github.com/ facebookresearch/llama/blob/main/ LICENSE, under the LLAMA 2 Community License.
• Mistral: https://github.com/mistralai/ mistral-src/blob/main/LICENSE, under an Apache License 2.0.

heading: Future Work
text: In the future, with the emergence of more and more educational question generation datasets with expert-annotated labels, the planning first, question second idea in our QG method may gradually show its application value. Limitations
We only evaluate our method and other compared methods on FairytaleQA, as our method requires suitable annotated labels in the dataset for generating answer plans. If there are more datasets with expert-annotated labels, we believe that our method will be more comprehensively evaluated. In addition, we attempted several methods to build a ranker model for helping QG model achieve higher MAP@1 score. However, these ranker models did not bring any stable improvement in either RougeL or BERTScore. We believe that a robust ranker model or a dataset including positive and negative samples will be good future works in educational question generation