Diversity Enhanced Narrative Question Generation for Storybooks
Authors: Hokeun Yoon
JinYeong Bak
Venue: EMNLP 2023 Main (2023)
Keywords: Question Generation
Natural Language Generation
NLP applications

Abstract: Question generation (QG) from a given context can enhance comprehension, engagement, assessment, and overall efficacy in learning or conversational environments. Despite recent advancements in QG, the challenge of enhancing or measuring the diversity of generated questions often remains unaddressed. In this paper, we introduce a multi-question generation model (mQG), which is capable of generating multiple, diverse, and answerable questions by focusing on context and questions. To validate the answerability of the generated questions, we employ a SQuAD 2.0 fine-tuned question answering model, classifying the questions as answerable or not. We train and evaluate mQG on the FairytaleQA dataset, a well-structured QA dataset based on storybooks, with narrative questions. We further apply a zero-shot adaptation on the TellMeWhy and SQuAD1.1 datasets. mQG shows promising results across various evaluation metrics, among strong baselines.

Diversity Enhanced Narrative Question Generation for StoryBooks
Hokeun Yoon Sungkyunkwan University Suwon, South Korea hkyoon950g. skku. edu
Jin Yeong Bak Sungkyunkwan University Suwon, South Korea jy bak@skku. edu
Abstract
instructing annotators create pre-determined number of questions for a given context. Recent QG research (Wang et al., 2020a; Yao et al., 2022), however, tends to rely on automatic evaluation semantic similarity with golden questions, often overlooking the potential for diverse aspects of questions. When generating multiple questions, diversity is erucial aspect to consider. The di- versity of questions can span several dimensions, including varied aspects of the context, different an- swer types, and different phrasings for essentially the same question (Karttunen, 1977). This diver- sity allows for a more comprehensive exploration of the context. The diversity of questions can be broadly measured based on the type of answers they require; explicit questions with answers that can be explicitly found in the reading materials, and implicit questions with answers that require deduc- tive reasoning. The crafting of multiple questions, bearing in mind both diversity and alignment with reading materials, poses a cognitively demanding and time-consuming task for humans.
Question generation (QG) from a given con- enhance comprehension, engagement, assessment, and overall efficacy in learning or conversational environments. Despite recent advancements in QG, the challenge of enhanc- ing or measuring the diversity of generated questions often remains unaddressed. this paper, we introduce multi-question generation model (mQG), which is capable of generating multiple, diverse, and answerable questions by focusing on context and questions. To validate the answerability of the generated questions. we employ SQuAD2.0 fine-tuned question answering model, classifying the questions as answerable or not. We train and evaluate mQG the FairytaleQA dataset, a serl-structurrd QA dataset based on storybooks, with narra- tive questions. We further apply a zero-shot adaptation the ThllMeWhy and SQuAD1.1 datasets. mQG shows promising results across various evaluation metrics, among strong base- lines.'
1 Introduction
One significant application of generating diverse and multiple questions is education. It has been observed that children can develop better reading comprehension skills at carly age by creating narrative questions themselves and being asked comprehension-relan questions about storybooks (Francis et al., 2005; Janssen et al., 2009). Reading comprehension an essential skill that requires learners to combine knowledge and reason about relations, entities, and events across a given context (Kim, 2017; Mohseni Takaloo and Ahmadi, 2017). Consequently, system that can generate diverse and multiple narrative questions able enhancement educational resources, aiding in student engagement and promoting a deeper un- derstanding of study materials.
Question generation (QG), focusing on the ques- tions derived from specific text passages or doc- uments, plays an integral role in a wide array of domains. It improves question answering (QA) systems (Sultan et al., 2020), enriches educational experiences (Yao et al., 2022), and enhances the engagement factor in chatbots (Laban et al., 2020). The effectiveness of QG tasks can be significantly improved by generating multiple questions, ensur- ing a broader, more comprehensive exploration of the content.
valu-
The importance of generating and evaluating multiple questions becomes evident when we exam- ine the creation process of QA datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Xu et al., 2022). Traditional QA dataset creation typically involves
Recently, some researchers have attempted generate multiple narrative questions. For educa- tional applications, Yao et al. (2022) proposed to
!ooe: https: wtt. wtw.:/uwb...ooo/loooo//////gg

---

generate question-answer pairs with three-step pipeline. As they use heuristic-generated answers to generate narrative questions most of their out- come is restricted to explicit questions. Also, Zhao et al. (2022) proposed to generate certain types of narrative questions and they tried to restrict the number of generated questions to number of ground-truth questions, insisting that knowing question type distribution for each context is sub- skill in education (Paris and Paris, 2003). We set these two approaches as our main baselines.
tion model that is trained using the maximum question similarity loss LMQs and employs a recursive referencing process for generating a wide array of questions while preserving semantic correctness.
We introduce answerability evaluation model capable of classifying questions im- plicit, explicit, unanswerable.
2 Related Work
2.1 Question Generation
To address the above challenges, we introduce a multi-question generation model (mQG) that gen- erates diverse and contextually relevant questions by referencing questions from the same context. mQG is trained with maximum question similarity loss LusQs which is designed to make the repre- sentation of reference questions and the representa- tion of larget question similar. Moreover, mQG employs a recursive generation framework, where previously generated questions are recursively fed back into the model as mQG is trained to output different questions from reference questions. Same our two baselines, mQG is trained and evalu- ated on the FairytaleQA dataset, which focuses narrative comprehension of storybooks. This dataset designed to provide high-quality narra- tive QA pairs for students from kindergarten to eighth grade (ages 4 to 14), and labeled questions as explicit or implicit. We adopt Self-BLen BLEU (Zhu et al., 2018) to evaluate the diversity of generated questions. Beyond diversity, to consider gener- ated questions relevant to the context, we demon- strate the answerability evaluation model to assess whether the generated questions are answerable. We also evaluate on TellMeWhy (Lal et al., 2021) and SQuAD1.1 (Rajpurkar et al., 2016) datasets with zero-shot adaptation to further analyze the performance of mQG in different settings. Differ- ing from previous approaches, mQG successfully generates substantial number of diverse and an- swerable narrative questions.
Based on given contents, question generation aims to generate natural language questions, where the generated questions are able to be addressed with the given contents. After neural approaches took large proportion in QG (Yuan et al., 2017; Zhou et al., 2017), QG can largely be separated by target answer aspect into answer-aware QG and unaware QG. Answer-aware QG, as its name implies, provides an answer to a model and prompts it to generate questions based on those answers. On the other hand, answer-unaware QG mainly focuses on the context to formulate ques- tions. The introduction of pre- pre-trained Language Models (LMs) further accelerated advancements in QG, and many works have demonstrated significant improvement in the answer-aware QG task and pre- sented promising possibilities for QG (Zhang and Bansal, 2019; Dong et al., 2019; Yan et al., 2020). This approach inherently favors explicit questions, which can be directly answered with the provided context. In answer-unaware QG, only handful of studies have been conducted, primarily focus- ing strategies such sentence selection from paragraph (Du and Cardie, 2017). employing trans- former architectures with out-of-vocabulayy meth- ods (Scialom et al., 2019), and generating questions based on silver summaries (Zhao et al., 2022). this paper, we utilize answer-unaware question gen- eration, giving consideration to both the diversity and quality of explicit and implicit questions.
The main contributions of this paper are summa- rized as follows.
2.2 Diversity
In natural language generation (NLG), generating outputs that are not only correct but also diverse is essential. In the decoding aspect, diversity has been researched in areas such as top-k sampling (Fan et al., 2018), and nucleus sampling (Holtz- man et al., 2020). These decoding methods tried to sample tokens from less likely vocabularies. Cer- tain studies have focused on training models to
We expand the scope of the question genera- tion task by generating comprehensive set of questions, regardless of our knowledge of the answers, and subsequently categorize them into answerable and non-answerable ques- tions.
We introduce mQG, a novel question genera-

---

FairytaleQA Dataset
Training Process
Story (Cuntext)
the wild people who dwell in the south-west are masters many
L MaQ + Lee
they of the maddle kingsomm kom to their country by promising them doughters moringg.. but their promises are not to he trusted. once there poor family, who agreed labor for three years for one of the wild men in order to become his son-in-law:
[O,, Qm]
[TQ]
the
Mean Pooling Layer
Mean Pooling Layer
Pooling
Layer
[O
0nd [O
Qunl
(TQ
10.1
Ground-truth Question
Who agreed labor three years for wild men onder become his son-in-loow?
[b]
BART Encoder
BART Decoder
Ground-truth Question
Who dwelied the of many black aris?
QT [SEP] Context (SEP) Question(1) Question(m)
Target Question
Ground-trum truth Question
What did the wild people lure of the Middle Kingdom to their country?
Overview the training process of mQG. Question(1) to Question(m) refer to ground-truth questions from the same context (orange), without ground-truth question (purple) input BART Decoder. QT and [h] denote the wh-word corresponding to the target question and overall encoder representation.
Figure I:
yield more diverse outputs (Welleck al., 2020; Yao al., 2022), and on leveraging the combi- nation of contrastive training and generation (Su et al., 2022). Recently, Sultan al. (2020) evalu- ated the importance of diversity in QG, insisting that diverse and accurate questions yield better QA results. Additionally, some researchers explored diversity in QG based on relevant topic (Hu et al., 2018), content selectors with question type model- ing (Wang et al., 2020b), control of question type (Cao and Wang, 2021), and difficulty level (Cheng et al., 2021). While these studies have addressed various aspects of diversity in QG, there still con- siderable room for further research in this area. In this paper, we consider diversity a significant chal- lenge in the question generation task and propose model that can generate wide range of answerable questions.
wh-words (what, when, where, which, who, why, how) question types. Mathematically, given the context C, question type QT, and history of gener- ated questions H. = (GQ1, GQ2. ..., GQ;-1), this task can be defined as generating a question, GQ, where:
- argmax (Pro H.)) (1)
For the training process, we extract wh-words from each question by applying part-of-speech tagging with the Spacy² English Model. Due to the absence of a history of generated questions and an insuffi- cient number of questions context per question type in the FairytaleQA dataset, we utilize ground- truth questions that only share the context the history of questions within the training process.
3.2 Diversity Enhanced Training
3 Method
mQG is built upon BART (Lewis et al., 2020), which has demonstrated remarkable performance in various natural language processing tasks. The primary pre-training objective of BART is to construct the masked input based on unmasked input. To further leverage the capabilities of the pre-trained BART, we introduce maximum ques- tion similarity loss LMQS- This loss is designed to promote similar representations for different ques- tions from the encoder and decoder.
In this section, formalize the multi-question gen- eration task and introduce our mQG. We first for- mulate our task and then explain how our model's training process incorporates a maximum question similarity loss LMQs- Finally, we provide a de- tailed outline of our recursive generation frame- work.
3.1 Task Formulation
As shown in Figure the encoder takes in three inputs: the question type, which signifies the type of question to be generated; the context, which pro-
The QG task in this paper aims generate each question using a given context, question type, and the history of questions generated from the same context with the same question type. We use seven
Phttps://spacy..t

---

vides the necessary information for question gen- eration; and ground-truth questions from the same context, serving as reference questions. These three inputs are concatenated, with a [SEP] token in- serted between them. The encoder processes the imput sequence and produces its corresponding rep- resentations. Subsequently, the decoder generates the representation for the target question. To calcu- late the maximum question similarity loss LMQ. we use mean pooling layers convert question representations into sentence-level representations. The maximum question similarity loss LMQ is calculated between the sentence-level representa- tion of the reference questions and the sentence- level representation of generated question. By encouraging the representation of different ques- tions to be similar, promote the generation of diverse questions that differ from reference ques- tions.
Generating Process
Step1
Generated Quration(Q1))
QT [SEP] Caty]CCnnnee Contest [SEP)
Step2
Generated Quedion((Qg)
BARRT
BART Decoder
Encodder
[SEP] Centext (SEP) GOI
The Recursive Generation Framework of mQG. This framework involves iterative process, using previously generated questions as input for sub- sequent steps, thereby creating recursive cycle. Fach iteration maintains the use of the same question type.
Figure 2:
For the subsequent generation steps, the previously generated questions are recursively fed back into the model. Specifically, the previous questions are concatenated with the same question type and context, separated by [SEP] token. This concate- nated sequence is then used as input for the next generation step. This recursive generation process continues until the desired number of questions per context per question type is achieved.
Given a set of reference questions sentence- level representation as Q - {Q1,..., Qm, and a sentence-level representation of the target ques- tion as TQ, the maximum question similarity loss LMQ is computed as follows:
LMQS 1 m
m
max(0, sQ;,
(2)
The use of this recursive generation process al- lows mQG to generate multiple questions while considering the previously generated questions. Following the training process of mQG, this gener- ation process enables mQG build upon its own previous outputs and generate different questions from previous outputs. We use beam search for the decoding method and return multiple sequences to exclude pre-generated questions. By leveraging a recursive framework, mQG demonstrates its capa- bility generate variety of diverse questions that contextually relevant and coherent.
where (QQ,TQ) is cosine similarity calculation between representations. By optimizing the model parameters maximize the sentence-level simi- larity between these different representations, we guide mQG generate diverse questions within the range of semantic correctness. This is achieved by ensuring that all the representations, which are the ground truth questions, are semantically cor- rect. In doing so, we maintain balance between diversity and accuracy in the generated questions. The overall training objective L is defined as
4 Experiments
L = LCE + LMQS
(3)
4.1 Dataset
LCE refers to the cross-entropy loss from a target question. As cross- entropy loss is calculated at the token level, the use of cross-entropy loss enhances mQG to generate syntactically correct questions.
FairytaleQA (Xu et al., 2022). We train mQG with the FairytaleQA dataset, which is constructed for educational purposes. Each book is split into sections and annotators were instructed to create average 2-3 narrative question-answer pairs per section. All question-answer pairs are annotated based on seven question types that capture nar- rative elements/relations. Questions are labeled as explicit or implicit questions based on whether or not the answer source can directly found
3.3 Recursive Generation Framework
Figure 2 illustrates the generation process of mQG, First, the encoder takes question type, and context as input. The decoder then generates a question based on the information provided by the encoder.

---

in the context. The original FairytaleQA dataset is constructed in train/validation/tess set with 232/23/23 books and /5548/1,022/1, QA pairs. From the entire dataset, a small portion of questions (985 out of 10,580) spans multiple paragraphs. As mQG and baselines are fit for one paragraph we remove those questions. To cross-validate, we ran- domly shuffled the dataset and split it by books in train//aliddation/esst set with roughly matching 80/10/10 (%).
denote architecture without ranking module QAG and the top 10 questions per context chosen by the ranking module as QAG (top 10).
EQG. EQG model (Zhao et al., 2022) gen- crates questions based on silver summaries. Silver summary is method proposed by Demszky et al. (2018), which inserts answers into the semantic parsed questions with rule-based method. EQG consists of three steps: 1) generate question type distribution for each context with BERT; 2) generate silver summary with BART, using question type, question type ordering from a question type distribution module, and context; 3) generate question based on silver summary, question type, and question ordering with BART, Without question type distribution module, EQG is able to generate multiple questions. Since our approach is generate multiple questions we set the EQG baseline without question type distribution module.
4.2 Baselines
In the experiments, compare mQG with four baselines; an end-to-end model initialized with BART-large, and methods proposed in Su et al. (2022), Yao et al. (2022), Zhao et al. (2022) denoted as CB, QAG, and EQG. The last two baselines are designed for multiple question generation purposes.
E2E. As the FairytaleQA dataset consists of multiple questions in one context, we concat all questions and train the BART-large model to generate questions based on each context. To match the number of generated questions, we set the maximal target length to 280 tokens which roughly matches the number of generated questions setting of mQG.
4.3
Automatic Evaluation
4.3.1 Evaluation Metrics
In evaluating question generation, both the quality and diversity of the generated questions are crit- ical components. Thus, we evaluate each aspect with separate automatic evaluation metrics. We use Rouge-L score (Lin, 2004), BERTScore (Zhang et al., 2020), and BLEURT (Sellam et al., 2020) measure the quality of generated questions. Similar to Yao et al. (2022), for each ground-truth question, we find the highest semantic similarity score on generated questions from the same context than average overall semantic similarity scores. And, with multiple questions generated from the same context, we recognize the necessity to measure di- versity automatically. For diversity mcasurement, Self-BLEU score (Zhu et al., 2018) which was introduced to evaluate just variety of sen- tences. The Self-BLEU score, which uses each generated sentence as a hypothesis and others as references, is employed to evaluate the diversity of questions generated from the same context. A lower Self-BLEU score represents greater diver- sity, All metrics ranges are between to except Rouge-L score (0 to 100).
CB (Contrastive Baseline). We construct this baseline following the framework in Su al. (2022), which tackles the problem of diversity in open-ended text generation. This framework first trains the language model using contrastive loss and decodes it with a contrastive search method. Since the contrastive baseline is proven for diverse apply it to GPT2 (denoted as
text generation CB (GPT2)), and BART (denoted CB (BART)) and set as our baseline. During generation, the maximal target length is set to 280 tokens.
QAG. This baseline follows a question-answer generation architecture by Yao et al. (2022). This architecture first generates answers based on heuristic-based based answer generation module, which generates multiple answers per context. With the generated answers, BART generates corresponding questions. And, verify the quality of the generated questions, DistilBERT ranking module ranks each QA pair and chooses the top questions. As our task to generate multiple questions, we
4.3.2 Answerability Evaluation Model
In order to evaluate whether the generated ques- tions correspond to the context, we leverage SQuAD2.0 dataset (Rajpurkar et al., 2018) to build

---

FairytaleQA
Generated Questions Per Section
Answerable Questions Per Section
Rouge-L, F1
BERTScore
BLEURT
Self-BLEU
Architecture
F1
SE
M
SE
M
SE
M
SE
M
SE
SE
E2E
1.58
0.07
144
0.12
36.05
0.35
0.8960
0.0062
0.4064
0.0104
CB (BART)
1.60
0.03
1.49
0.04
36.89
0.68
0.9074
0.0017
0.4045
0.0072
CB (GPT2)
3.28
0.43
0.96
0.56
26.47
1.27
0.8937
0.0020
0.3328
0.0077
0.8906
0.0120
BQG
28.00
0.00
3.80
0.76
41.05
1.61
0,9136
0.0034
0.4293
0.0118
0.9864
0.0043
QAG (top10)
9.95
0.14
6.57
0.39
45.44
0.81
0,9208
0.0006
0.4444
0.0076
0.7608
0.0078
QAG
26.97
0.50
15.95
1.24
53.77
1.03
0.9323
0.0009
0,5140
0.01
0.8874
0.0030
mQG
28.00
0.00
23.08
0.36
58.90
0.37
0.9394
0.0005
0.5698
0.0033
0.6389
0.0079
Table I: Three cross-validation results on the FairytaleQA dataset. # Answerable Questions Per Section is based on
the answerability evaluation model, as described in section 4.3.1. 1 means higher is better, 4 means lower is better. Due to low number of questions, Self-BLEU which cannot be measured is marked with hyphen. M, SE denotes mean and standard error. mQG generates the highest number of answerable questions with greater diversity.
an evaluation model. SQuAD2.0 is question- answering dataset with 100K answerable questions and 50K unanswerable questions. This dataset is used to enhance the evaluation model by classify- ing whether the questions are answerable or not, We use DeBERTa-base (He et al., 2021) as the backbone model.
for each section. In '# Answerable Questions Per Section', as duplicate questions within the same context are not needed, we leave only one question from duplicate questions. Even though mQG is able to generate multiple questions within the max- imum token length of BART, we roughly match the number of questions to QAG for fair comparison in Rouge-L FI, setting mQG generate 4 questions per section per question type, totaling 28 questions per section. The same setting is applied to EQG, as EQG does not have limitations in generating multiple questions.
To achieve our goal, we train the evaluation model on the QA task following implementation in Devlin et al. (2019). We construct two dense layers above the encoder; one for the answer start position and the other for the answer end position. And, as unanswerable questions and implicit questions do not have an answer span, for these questions [CLS] token is assigned the answer start position and the answer end position. For implicit questions in the FairytaleQA dataset, we add special token [IMP] and assign it an answer start span and answer end span. First, we train the evaluation model with the SQuAD2.0 dataset on the QA task. For the second step, we train the evaluation model again with the FairytaleQA dataset. By utilizing two-step training, the evaluation model is able to classify generated questions as explicit, implicit, unanswerable. The number of answerable ques- tions per section in Table 1 are based classified results by the evaluation model. the evaluation model classifies generated questions as implicit or explicit, then we count them as answerable. (An- swerability evaluation model details are given in Appendix A.)
General baselines (E2E and CB) that generate multiple questions one iteration show significant underperformance in the Rouge-L Fl score and the number of generated questions, compared to strong baselines (QAG and EQG), and the mQG. This indicates that to generate multiple questions, specific model is needed. Across all evaluation metrics, mQG consistently outperforms the base- lines.
4.4 Human Evaluation
We evaluate the diversity and quality of generated questions on the FairytaleQA dataset with human judges. We hire five anmotators, proficient English as their first foreign language, to further evaluate the diversity and quality of the generated questions. We follow the human evaluation procedure described by Cao and Wang (2021) and compare mQG, with two robust baselines, EQG and QAG.
4.3.3 Results
Table presents evaluation results on the Fairy- taleQA test set. '# Generated Questions Per Sec- tion' refers to the number of questions generated
Question Diversity. In the question diver- sity study, we randomly sample 5 books from the

---

Architecture
Type (%)
Syntax (%)
Content (%)
Architecture
Appro.
Ans.
EQG
22.0
EQG
4.85
4.46
QAG
33.0
22.0
34.5
QAG
4.60
4.43
mQG
77.0
70.5
60.0
mQQ
4.79
4.47
Ground-truth
4.71
4.76
Table 2: Human evaluation on diversity. The percent- age samples ranked first among other models. Krip- pendorf's s alphas are 0.69, 0.51, and 0.38 for the three dimensions. Ties are allowed. mQG demonstrates the most diversity in all dimensions.
Table 3: Human evaluation on appropriateness (Appro.) and answerability (Ans.). The Krippendorf's alphas are 0.14 and 0.27 for the two dimensions. Ties are allowed. all models, not much difference observed compared to ground truth questions.
original test set; and for each book, we randomly sample 8 sections, totaling 40 sections. For each section, we randomly sample three questions question set from each model, and provide only the question sets for annotation. For each question set, the annotators rank the three models on scale of (highest) to (lowest) based on three dimensions of diversity: type-whether the three selected questions have different question types; syntax-whether the three selected questions use different syntax; and content-whether the three selected questions need to be addressed with diverse answers.
assessing model performance as it illuminates the model's ability to generalize beyond the specific examples it was trained on.
4.5.1 Dataset
TellMeWhy (Lal et al., 2021). TellMeWhy dataset comprises free-form why-questions related to events in short sections. The dataset was created using template-based transformations
to generate questions, with crowdsourcing gather answers. Sections were sourced from ROCStories (Mostafazadeh et al., 2016), a similar domain to the training dataset (FairytaleQAA). TellMeWhy contains a mixture of explicit and implicit questions. Approximately 28.82% of questions in the dataset are implicit. We evaluate with 1,134 sections and 10,689 questions from the test split.
As shown in Table 2, on all dimensions, human annotators rate mQG as generating the most diverse questions compared to the other mod- els, with each question requiring different answer.
Question Quality. In the question quality study, we again randomly sample 5 books from the original test For each book, select random sample of 8 sections. Each section contains four questions, each randomly sampled from three models and ground-truth, totaling 160 questions. Two dimensions are rated from 1 (worst) to (best): appropriateness--wheteerr the question is semantically correct; unswerabilty-wwhthee the question can be addressed by given section.
SQuAD1.1 (Rajpurkar et al., 2016). Squad1.1 dataset a comprehensive benchmark that focuses on machine comprehension, question generation, and question answering tasks. It consists of a large collection of articles from Wikipedia, covering a wide range of topics, which is different source from the training dataset (FairytaleQA). Each article accompanied by of only explicit questions. We evaluate with 2,429 sections, and 12,010 questions from the SQuAD1.1 test split created by Du et al. (2017).
As shown in Table 3, all models, when compared the ground-truth, generate semantically correct questions. Given that mQG can generate a broad diversity of questions, these results confirm that mQG fulfills our goal of generating multiple ques- tions while maintaining semantic correctness and relevance to the context.
4.5.2 Zero-shot Results
In zero-shot evaluation, we compare mQG with two strong baselines, EQG and QAG. Initially, we examine the performance on the Tellmewhy dataset in Table 4. Given that the TellMeWhy dataset only contains why-questions, we select why-questions from the generated questions for evaluation. mQG achieved the highest semantic similarity scores and outperformed bascline models in terms of the num- ber of answerable questions and exhibited better
4.5 Zero-shot Performance Evaluation
We conduct zero-shot evaluation on two distinct datasets, to test mQG more in various real-world scenarios, where contexts and desired questions can differ. Zero-shot evaluation is essential for

---

TellMeWhy
Generated Questions Per Section
Answerable Questions Per Section
Rouge-L Fl
BERTScore FL
Architecture
BLEURT
Self-BLEU
EQG
4.00
0.63
35,91
0.9129
0,4126
0.9425
QAG
1.53
0.45
30.35
0.9231
0.4360
mQG
4.00
2.10
56.17
0.9361
0.5475
0.3191
Table 4: Zero-shot evaluation result on Tel dataset. Due to a low number of questions. Beff-BLUU which
cannot be measured is marked with a hyphen. mQG shows the highest semantic similarity scores with more diversity and generates the largest number of answerable questions.
SQuAD1.1
Generated
Answerable Questions Per Section
Rouge-L F
BERTScore F
Architecture
Questions Per Section
BLEURT
Self-BLEU
EQG
28.00
3.74
30.31
0.8977
0,4219 0.5265
0.9695
QAG
29.77
14.40
46.75
0.9203
0.7172
mQG
28.00
20.15
45.38
0,9211
0.5508
0.6157
Table 5: Zero-shot evaluation result on SQuAD1.1 dataset. mQG generates the most answerable questions with more diversity.
diversity. Zero-shot evaluation on the Tellmewhy dataset, which contains mix of explicit and im- plicit questions, demonstrates the ability of mQG to generate different question styles based answers effectively.
Evaluation on Different Question Number Settings
0.2
0.4
60
Table 5 shows evaluation results the SQuAD1.1 dataset. Even with an out-of-domain dataset, mQG still demonstrates notable perfor- mance. mQG outperforms in generating diverse questions and producing a greater number of an- swerable questions compared to other baselines. However, in the Rouge-L. F1 score, mQG is slightly lower than QAG. This can be attributed to the ex- clusive focus of the SQuAD dataset explicit questions, and the answer-aware question genera- tion method used by QAG, which is renowned for its effectiveness in generating explicit questions. Yet, when employing embedding- based evaluation methods such as BERTScore and BLEURT, mQG outperforms the baseline models, particularly in the case of BLEURT. The fact that mQG still demon- strates decent performance on the SQuAD dataset, despite the limitation of the dataset explicit ques- tions and its status as an out-of-domain dataset, further emphasizes the effectiveness of mQG.
Rouge-L
0.6
Self
0.8
RougeL
45
1.0
5
20
Number of Questions Per Sec Per
Figure 3: Results of different question number settings on the original FairytaleQA test set. SeIf-BLFUU pre- sented here in reversed format allow for more intuitive visual comparison. Intersections of the curves represent the optimal trade-off between two metrics.
5 Ablation Study
5.1 Setting of Question Number
Given that mQG can be set with the number of questions to generate, we conduct an experiment various settings of question number per section per question type to generate. In Figure 3, the eval- uation result is based on the original FairytaleQA test set. As the quantity of generated questions creases, the Rouge-L Fl score provides satisfactory results, though diversity decreases. This indicates
Through these two different settings, we see promising results mQG. It shows the adaptability mQG to diverse question styles and domains, fur- ther validating the robustness and utility of mQG.

---

VairytaleQA
# Generated
Answerable Questions Section
Rouge-L F1
BERTScore F1
Questions
BLEURT
Selr-BLEU
Architecture
Section
M
SE
M
SE
SE
M
mQG
28.00
0.00
23.08
0.36
58.90
0.37
0.9394
0.0005
0.5698
0.0033
0.6389
0.0079
-MMq
28.00
0.00
22.67
0.28
58.66
0.08
0.9394
0.0003
0.5703
0.0019
0,7006
0.0045
-Mrgs & reference questions
28.00
0.00
22.65
0.41
54.76
0.22
0.9353
0.0005
0.5428
0.0011
0.7529
0.0032
Table 6: questions.
The comparison results of mQG with and without maximum question similarity loss and reference
that a significant increase in the number of gener- ated questions tends to produce similar questions with different phrasings. Setting the number of gen- erated questions 4 shows the optimal trade-off between the Rouge-L Fl and the Self-BLEU.
of datasets, highlighting its potential utility in envi- ronments that demand diverse narrative questions.
Limitations
mQG framework utilizes recursive feedback mechanism for generating questions during the in- ference stage. However, the quality of these gen- erated questions remains uncertain. If the quality of previously generated questions is poor, this may adversely impact the quality of subsequent ques- tions produced by mQG. Moreover, the quantity of questions that be generated is limited by max- imum token threshold. Another limitation the potential risk of misclassification by the evaluation model, which could lead to the categorization unanswerable questions as answerable. Despite our efforts to mitigate this risk, the evaluation model is still at level of uncertainty in accurately classi- fying the generated questions. Even with the fact that reliability scores can be low in NLP tasks, in the quality human evaluation, the reliability scores are relatively low. This can lead to uncertainty in the results.
5.2 Analysis of Maximum Question Similarity Loss and Recursive Framework
As discussed in section 5.2, mQG aims to increase diversity within questions while maintaining se- mantic correctness. mQG w/o LMQs refers to the mQG model only trained with LCg. For mQG w/o L LQQ and reference questions, give only ques- tion type and context input while training, and no recursive framework is used in inference. Table 6 shows that the mQG model with maximum ques- tion similarity loss Lasqs and reference questions hugely increase diversity. Additionally, the number of answerable questions has also improved. This could be attributed to the fact that all ground-truth questions are answerable, and mQG maximizes the similarity between these questions and continually references the most probable question during infer- ence. These results indicate that each framework of mQG effectively enhances the probability of generating a diverse set possible questions.
Ethics Statement
The results are appropriately placed in the context of prior and existing research. All generation mod- els trained the FairytaleQA dataset which is publicty available and has no ethical issues annotated by educational experts. In the human evaluation process, we pay annotators more than the minimum wage.
6 Conclusion
this work, we extend the scope of answer- unaware question generation to generate multiple diverse questions. We propose a novel framework that applies a maximum question similarity loss during training to promote question diversity, fol- lowed by recursive generation process for further refinement. Additionally, evaluation model is introduced to verify the answerability of the gen- erated questions. Recognizing the essential role of narrative questions in education, we train and evaluate mQG accordingly. Comprehensive experi- ments validate the efficacy mQG across a variety
Acknowledgements
We would like thank the anonymous review- for their helpful questions and comments. JinYeong Bak is the corresponding author. This work was partly supported by Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea

---

government (MSIT) (No.2022-0-00680, Abduc- tive inference framework using omni-data for un- derstanding complex causal relations & No.2019- 0-00421, AI Graduate School Support Program (Sungkyunkwan University)), and a grant from the National Research Foundation of Korea (NRF) INRF-2021R1A44003311
Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1342-1352. Vancouver, Canada. Association for Computational Linguisties.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
References
David Francis, Jack M Fletcher, Hugh W Catts, and Bruce Tomblin. 2005. Dimensions affecting the assessment of reading comprehension. In Children 's Reading Comprehension and Assessment.
Shuyang Cao and Lu Wang. 2021. Controllable open- ended question generation with A new question type ontology. CoRR, abs/2107.00152.
Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence- level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362-367, Baltimore, Maryland, USA. Association for Compu- tational Linguistics.
Pengeheng Hc, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference Learning Representations.
Yi Cheng, Siyao Li, Bang Liu, Ruihui Zhao, Sujian Li, Chenghua Lin, and Yefeng Zheng. 2021. Guiding question genera-
the growth: affiiuult-controlllae tion through step-by-step rewriting. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing Online.
Wenpeng Hu, Bing Liu, Rui Yan, Dongyan Zhao, and Jinwen Ma. 2018. Topic-based question generation.
Tanja Janssen, Martine Braaksma, and Michel Couzijn. 2009. Self-questioning in the literature classroom: Effects on students' interpretation and appreciation of short stories. LJ-Educational Studies in Language and Literature, 91):91-116.
(Volume I: Long Papers). pages Association for Computational Linguistics.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, abs/1809 809.0292..
Lauri Karttunen. 1977. Syntax and semantics of ques- tions. Linguistics and Philosophy, 1(1):3-44.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-44 4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Young-Suk Grace Kim. 2017. Why the simple view of reading is not simplistic: Unpacking component skills of reading using a direct and indirect effect model of reading (dier). Scientific Studies of Reading. 21 (14):310-333.
Philippe Laban, John Canny, and Marti A. Hearst. 2020. What's the latest? question-driven news chatbot. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 380-387, Online. Association for Computational Linguistics.
Dong. Nan Yang. Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang. Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. CoRR, abs/ 1905.03197.
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. TellMeWhy: A dataset for answering why-questions in narratives. In Findings the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages 596-610, Online. Association for Computational Lin- guistics.
Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing. pages 2067-2073. Copenhagen. Denmark. Asso- ciation for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising equeence-t--sqqenne pre-training
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual

---

for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
answer-agnostic neural question generation. In Pro- ceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 6027- 6032, Florence, Italy, Association for Computational Linguistics.
Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 18922, Online. Association for Computational Linguisties.
Nahid Mohseni Takaloo and Mohammad Reza and Ah- madi. 2017. The effect of learners' motivation on their reading comprehension skill: A literature re- view. International Journal of Research in English Education, 2(3).
Yixuan Su, Tian Lan, Yan Wang. Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 20/6 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839 849, San Diego, California. Association for Computational Linguis- tics.
Md Arafat Sultan, Shubham Chandel. Ramón Fernan- dez Astudillo, and Vittorio Castelli. 2020. On the importance of diversity in question generation for QA. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 5651 Online. Association for Computational Linguistics.
Siyuan Wang. Zhongyu Wei, Zhihao Fan, Zengfeng Huang, Weijian Sun, Qi Zhang. and Xuanjing Huang. 2020a. PathQG: Neural question generation from facts. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (FMNLP), pages 9066-9075, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-1-1, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Zhen Wang, Siwei Rao, Jie Zhang. Zhen Qin, Guangjian Tian, and Jun Wang. 2020b. Diversify question gen- eration with continuous content selectors and ques- tion type modeling. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2134 2143, Online. Association for Computational Linguistics.
Alison Paris and Scott Paris. 2003. Assessing narrative comprehension young children. Reading Research Quarterly- READ RES QUART, 38:36-76.
Pranav Rajpurkar. Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020. Neu- ral text generation with unlikelihood training. In International Conference on Learning Representa- tions.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Ying Xu, Dakuo Wang. Mo Yu, Daniel Ritchie, Bing- sheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022. Fan- tastic questions and where to find them: FairytaleQA an authentic dataset for narrative comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 447-460, Dublin, Ireland. Association for Computational Linguisties.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domaie machine comprehension of text. In Pro- ceedings of the 2013 Conference Empirical Meth- ods Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computa- tional Linguistics.
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future n- gram for sequence- sequence-to-seqeeee to-seunence pre-training. CoRR. 0e5/2001.04065
Thomas Seialom, Benjamin Piwowarski, and Jacopo Staiano. 2019. Self-attention architectures for

---

Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Li, Mo Yu, and Ying Xu. 2022. is Al's urrn to ask humans question: Question- answer pair generation for children's story books. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 731 731-744, Dublin, Ireland. Association for Computational Linguistics.
Start
DeBERTa
[CLS] [IMPLICIT] question [SEP] context
Xingdi Yuan. Tong Wang. Çaglar Gülçehre, Alessan- dro Sordoni. Philip Bachman, Sandeep Subramanian, Saizheng Zhang, and Adam Trischler. 2017. Ma- chine comprehension by text-to-text neural question generation. CoRR,
Figure 4: Overview of Answerability Evaluation Model.
Explicit
Implicit
Total
Shiyue Zhang and Mohit Bansal. 2019. Address- ing semantic drift in question generation for semi- supervised question answering. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.
questions
5,376
1,963
7,309
Table 7: The number of questions of the FairytaleQA dataset after annotation mistakes were removed.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q
Weinherger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with bert.
90
Relio
85-
Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, and Xiaojuan Ma. 2022. Educa- tional question generation children storybooks via question type distribution learning and event-centric summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 5073-5085, Dublin, Ireland. Association for Computational Lin- guistics.
80-
75
70-
Expertment2
Experiment3
13
Threshold (1)
Figure 5: The answerable ratio of val+test set by differ- ent threshold settings.
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. CoRR, abs/1704.0177
with cross-and cross-annotated answers had different aspects of answers (explicit, implicit). We removed those questions and a number of total questions after pre- processing is described in Table 7.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research amp; Development in Information Re- trieval, SIGIR 18, page 1097-1100, New York, NY, USA. Association for Computing Machinery.
A.2 Evaluation Model Postprocessing
In terms of post-processing, we take similar ap- proach by Devlin al. (2019). Classified results Ye of each question are formulated as:
Appendix
A Further Analysis Evaluation Model
if CLSse ase + and CLS > IMPse + else if IMPse age otherwise.
No Answer,
A.1 Preprocessing Dataset
Ue
Implicit,
To evaluate each cross- validatii set with an an- swerability evaluation model, train the evalu- ation model with different FairytaleQA trainsets. One is an originally constructed trainset and the others are randomly split by books. From the Fairy- taleQA A dataset, some explicit questions were not able to be found the section and some questions
Explicit.
(4)
CLS,e denotes score of [CLS] token as answer start span and answer end span. 7M Pso denotes score of [IMP] token as answer start span and an- swer end span. age denotes the best score of answer

---

F1
Accuracy
Seif-BLEU
Example Questions
Why did the Dragon King want to cupture monkey?
M
SE
M
SE
Why couldn' Dragon King's servants capture monkey?
0,3150
Why did the Dragon King consult his chief steward?
Explicit
78.72
0.52
88.26
0.17
Why was the Dragon King greally puzzled??
Implicit
64.76
2.05
64.76
2.05
Why did Dragon King want cupture monkeyy?
Total
75.28
1.01
82,49
0.81
Why couldn't 's the Dragon King's 'in'ss servants capture monkey?
Why did the Dragon King comsult chiif steward?
How did the Dragon King consult chief steward?
Table 8: Ground-truth val+test results on the eval- uation model. Each model is trained with each cross- validation trainset.
Why did the Dmgon King consult his chief?
Why did the Dragon King consult steward?
0.7830
Why did the Dragon King consult his chief steward?
How did the King consult his chief steward!?
Why did the Dragon King consalt chief steward?
Why did the Dragon King
chief?
FairytaleQA
0.9014
Why did the Dragon King consult his chief steward?
Ground-truth
QAG (top10)
QAG
EQG
mQG
How did the Dragon King consult chief steward?
Explicit
74. 10%
79.05%
71.69%
54,42%
60.65%
Implicit
21.22%
2.50%
5.08%
33.95%
22.38%
Table 10: Examples on Self-BLEU scores with 4 ques-
Ans.
4.68%
18.45%
23.23%
11.63%
16.97%
tions each.
Total
919
2,835
7,534
1,402
8,820
Table 9: The FairyTaleQA test set analysis of questions by answer types, classified by evaluation model. Total denotes the number of questions after duplicates from the same context are removed. Each answer type is denoted with a proportion each model.
A.4 Classified Questions Analysis
We analyze the ratio of questions classified into dif- ferent answer types by the answerability evaluation model. Even though the ground-truth questions do not contain unanswerable questions, the evalu- ation model classifies approximately 4.5% of the questions unanswerable, shown in Table 5. The problem of answer-aware question generation is
start span and answer end span without [CLS] and [IMP]. Additionally, if an answer end span indice is lower than an answer start span indice we clas- sify it as no answer. Threshold is selected on the ground-truth set maximize the performance. This threshold is set differently for each evaluation model. Figure 5 shows the answerable ratio per- centage by different threshold settings, We also train three evaluation models with each train set for cross validation in the main results. We select each threshold before significant drop in the an- swerable ratio observed. -12, -10, and -11 are each threshold for experimentl, experiment2, and experiment3.
QAG uses the answer as an input in the question generation process, and our results show that QAG is not for generating implicit questions, as only about 5. % of questions are clas- sified as implicit. The EQG baseline generates both explicit and implicit questions but only has small number of total questions after removing duplicates. On the other hand, the mQG still has large number of questions even after removing duplicates, totaling 8,820, with explicit and im- plicit questions roughly in a 3-to-1 ratio. These results show that the mQG generates both types of multiple questions better than other baselines.
A.3 Evaluation Model Results
B Diversity Exploration
We perform cross-validation measure the perfor- mance of the main results in Table 1, and as result, we train each evaluation model with each trainset. Since our goal is to classify questions as explicit, implicit, or unanswerable, we count explicit ques- tions as accurate if at least one of the predicted answer tokens is found in the ground-trui truth answer. This is denoted as "Accuracy" in Table 8. The F1 measurement follows the implementation by De- vlin et al. (2019). The evaluation model classifies explicit questions more accurately than implicit questions.
For diversity evaluation, we calculate the Self- BLEU score among generated questions from the same context. Self-BLEU score is based on BLEU evaluation method (Papineni et al., 2002). The BLEU evaluation method has many criticisms for eval evaluating sentence-level corpus. If a higher-order n-gram precision goes to O, the total BLEU score goes O. As an outcome, many variations ap- plying the smoothing method for the BLEU score have shown (Chen and Cherry, 2014). We apply 'smoothing I' described Chen and Cherry (2014) since all the generated questions are sentence-level.

---

FairstaleQA
FairytaleQA
Answerable Questions Per Section
Architecture
Decdonnng Method
Rouge-L F
Self-BLEU)
Answerable Questions Per Section
Rouge-L. F1
Self-BLEU
mQQG-T DQG-BART
1m5
17.89
30.50
0.5476
23.35
98.24
0.6243
p-0.1
16.89
53.45
0.7826
0.0
22.89
58.49
0.4747
p=0.5
18.01
53.54
0.7622
0.2
23.16
59,40
0.4117
p=0.75
19.12
54.45
0.7321
0.4
23.23
59.54
0.4052
pm0.95
20.06
54,90
0.7135
23.26
58.44
0.4261
0.8
23.34
59.29
0,4288
Table 11: Performance of mQG with different back- bone models and decoding methods on the original test set. b=5 denotes beam search with beam size set to p denotes nucleus sampling (NS@p: p 0.1, 0.5, 0.75, 0.95). All models are to generate 28 questions per section.
1.0
23.35
58.24
0,4210
2.0
23.28
58.28
0.4297
3.0
23.34
58.42
0,4478
5.0
23.50
58.15
0.4527
Table 12: mQG results on different / settings on the original set. 0.0 equals mQG w/o maximum question similarity loss LMQs. All models are set to generate 28 questions per section.
Examples of Self-BLEU scores are shown in table 10. When the Self-BLEU score goes up to 0.7830, almost all questions can be addressed by the same answers.
Architecture
Rouge-L (ori) Rouge-L (alt)
Diff
FairytaleQA
C Decoding Method and Model Selection
EQG
41.05
39.35
1.70
QAG
53.77
53.13
0.64
Moreover, in addition to the main results, com- pare the performance of mQG between different backbone models and decoding methods. In Table 11, T5-based mQG exhibits the best Self-BLEU score but significantly lags behind BART-based mQG in terms of # Answerable Questions Per Sec- tion and Rouge-L score. This suggests that T5- based mQG struggles generate semantically cor- rect questions. When comparing decoding meth- ods, beam search outperforms nucleus sampling in all dimensions. This is due to the decoding pro- cess of mQG, which returns multiple sequences to exclude pre-generated questions. Beam search utilizes tree search algorithm, whereas nucleus sampling does not. As a result, nucleus sampling tends to generate duplicate questions.
mQG
58.90
58.36
0.54
TellMeWhy
EQG
35,91
15.08
20.83
QAG
30.35
23.93
6.42
mQG
56.17
51.57
4.60
SQuAD1.1
EQG
30.31
25.84
4.47
QAG
46.75
44.85
1.90
mQG
45.38
43.20
2.18
Table 13: Comparison results on Rouge-L calculation. FairytaleQA results are the mean value cross- validation results. Rouge-I. (alt) denotes one-to-one match calculation. Diff denotes the difference between Rouge-L. (ori) and Rouge-I. (alt).
of implementing MQS loss to enhance diversity within the bounds of semantic correctness.
D Weighting Factor Impact on Performance
E Another Rouge-L Calculation
To determine how MQS loss affects training, we conduct experiments with the mQG model using different settings for the weighting factor 3. The overall training objective L. is defined as
As mentioned in Section 4.3, we calculate the Rouge-L score only to find the highest score for each ground-truth question. This calculation method may lead to the one- one-to-many matching problem. To determine if the problem has occurred, compare the results with another Rouge-L cal- culation Rouge-L (alt). This calculation excludes previously matched generated questions, allowing for only one-to-one matches. In Table 13, most Rouge-L (alt) results exhibit slightly lower scores in comparison to Rouge-L (ori), suggesting that
LCE + LMQS
(5)
is calculated between ques- tions that share context and question type. The optimal point of diversity is achieved when 3 is set to 0.4. As 3 increases, the Self-BLEU score decreases, while the number of answerable ques- tions increases. This outcome aligns with our goal
In Table 12,

---

one-to-many problems have occurred, although the impact is relatively minor as the ground- ques- tions are a unique set of questions. The signifi- cant difference the Thy dataset can be attributed to the limited number 'why" questions generated.
F Implementation Details
For the mQG model, we use the MQS loss of the validation set as the selecting criteria. For the mQG models without MQS loss, we use MLE loss as the selecting criteria. Total training time was about hours with RTX A6000 GPU. We initialize the mQG model with pretrained BART-large, which has 406M parameters. Hyperparameters are follow: learning rate 5e-6; batch size 8; epoch 15
We use RoBERTa-large model for BERTScore and BLEURT-20 model for BLEURT. For the evaluation model, we load SQuAD 2.0 finetuned DeBERTa-base model which has 86M param- eters, to further finetune. Total training time about hour with RTX A6000 GPU. Hyperparameters are follow: learning rate 5e-6; batch size 16; epoch=
G Examples of Generated Questions
Tables 14 and 15 show the generated examples of the mQG, EQG, QAG, and ground truth questions with the according section and classified results with the answerability evaluation model. Even with different settings for generating multiple questions, EQG still generated duplicate questions because it guided the model only with special tokens to generate multiple questions. QAG has generated different questions but with less diversity. In all questions, the evaluation model accurately classi- fied the questions. Given the sufficient number of questions generated by each model, selected four questions representative examples. Given the sufficient number of questions generated by each model, we selected questions as representa- tive examples.
'https: //hggg//g/iiigaaee ace, ca/ co/deepset/ deberta-va v3-base-squade

---

Section
But his brother complained of heing weary, and at length they decided to remain there for the night. When Andrew awoke he found himself alone; and he saw neither brother nor boat, until he came to the highest point of the island. Then he discovered him far out, darting for land like a sea-gull. Andrew did not understand the whole affair. There were still provisions there, well dish of curd, his gun and various other things. So Andrew wasted but little time in thought. "He will come back this evening," " said he. "Only fool loses heart long as he can eat." " But in the evening there was no brother to be seen, and Andrew waited day by day, and week by week; until at last, he realized that his brother had marooned him on this barren island in order to able to keep their inheritance for himself, and not have to divide it. And such was the case, for when John Nicholas came in sight of land on his homeward trip, he had capsized the boat, and declared that Lucky Andrew had been drowned.
Ground-truth Questions
What was John Nicholas doing when Andrew saw him? (Explicit)
Why did John Nicholas capsize the boat when he reached land? (Implicit)
Why did Andrew want the inheritance himself? (Implicit) How did Andrew feel when he saw his brother and boat far out? (Implicit)
mQG
What did John Nicholas declare when he came in sight of land? (Explicit)
Why did John Nicholas marooned his brother on a barren island? (Explicit)
Why did the brother want to keep their inheritance for himself? (Implicit)
Why did John Nicholas declare that Lucky Andrew had been drowned? (Implicit)
QAG
what did andrew find on the island? (Explicit)
what did andrew find on the island? (Explicit)
who did andrew see after he woke up? (No Answer)
when andrew awoke he found himself alone? (No Answer)
EQG
What happened after John Nicholas came in sight of land his homeward trip? (Explicit) What happened after John Nicholas came in sight of land on his homeward trip? (Explicit) Why did the brother complain of being weary? (Implicit) Why did the brother complain of being weary? (Implicit)
Table 14: Example of generated questions of mQG, EQG, and QAG for a given context. the right of each question, classified results by the answerability evaluation model are shown in parentheses. Generated questions of mQG have the most diverse syntax questions considering question types and still, semantically correspond to the given section.

---

Section
At any rate, his poor young wife was sorely to be pitied, for she was left with a little baby-boy to bring up, and, as times were bad, she had not much to live on. But she loved her baby dearly, and worked all day amongst her cows, and pigs, and hens, in order to earn enough money to buy food and clothes for both herself and him. Now, on the morning of which I am speaking, she rose very early and went out to feed her pigs, for rent-day creature, the market that very day, as she thought that the price that it would fetch would go long way towards paying her rent. And because she thought so, her heart was light, and she hummed little song to herself as she crossed the yard with her bucket on one arm and her baby-boy on the other.
coming on, and she intended to take one of them, great, big, fat
Ground-truth Questions
What did the widowed Mistress do to earn enough money? (Explicit)
Why did the widowed Mistress want to sell her pig? (Explicit) How did the widowed Mistress feel while she thought of selling the pig? (Implicit)
mQG
What did the young woman do because rent-day was coming on? (Explicit)
Why did the young wife want to buy a great, big, creature? (Explicit)
Where did the young woman intend to take one of the pigs? (Explicit)
How did the young wife feel to bring up her baby? (Implicit)
EQG
Why did the wife go out to feed her pigs? (Explicit)
Why did the wife go out feed her pigs? (Explicit)
How did the wife feel when she saw the pig? (Implicit) How did the wife feel when she saw the pig? (Implicit) QAG
how did the wife carn money? (Explicit)
what day was it? (Implicit)
what did the wife feed her pigs? (No Answer) how many pigs did the wife plan buy? (No Answer)
Table 15: Another example of generated questions of mQG, EQG, and QAG for a given context. On the right of each question, classified results by the answerability evaluation model are shown in parentheses. mQG model has generated questions most answerable questions with diversity.

---

Please set cuestions. hard to determine the superioriy anc interiority. it's okay to give the same ranktno. much sossible. bast,
ovalucte
Where did the prisoner go after he left ng's presence?
Below persgraph from book Four question releted the book are givan and please rato eash quoct on. It's dkay give the came scora. (t.
Whet the king tall the young man do oft ha leftr
How did the yourng man leel after he lelt the ling't presence??
grew
grne warring
hix
neigi bor along fewer twanty ands, they mowed ane
who was the old woman?
swaih snedher poer neighter take trouble begin som now the others took hold and that no Nimazif would not be able to do anything alonc.
what did the young man ralae? how long did
bring back lis acer?
Cuestions
C
A: Mi haspen wit the neight bor comes clong with fower than twarry hander
Where did
appree
neightor wlong? the poor noighbor saw how others book
Which quaetione has divenno
The more gremmeticall correct questions, the Higher the score.
O
O
0
0
O
O
0
0
0
0
0
0
0
0
B
0
2. Which set queelione consiets of more diverse phresee??
Delommia much quehqquiiiiiii ammowadle from i paoegapl and yive at lig
O
score as you think is possibre.
cuestion whot's
trink It's
O
O
possibie Infer from the paragraph, plansa say it's possible answer II.
the regardless the difficulyy
a
O
O
0
Which quoedieme roin paovidl mom riveme
betlie manking for and aneiione that cumplex reesoring to areweer)
O
0
0
0
0
0
0
0
0
O
O
O
Figure 7: The question sheet for quality human evalua- tion.
Figure 6: The question sheet for diversity human evalu- ation.


Reviews:
[{"review_id": "uilPwSezoH", "reviewer": "Reviewer_h6Ka", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews and constructive suggestion!\n\n### Experiment reliance on BART\n\nWhile we acknowledge that BART may be perceived as dated and relatively smaller, our current paper intentionally centers around a specific task to provide a focused analysis. In addition, we only used BART for a fair comparison, given that the baseline papers, QAG and EQG, also employed BART. Still, we fully agree that recent advancements in LLMs are crucial. We are eager to take this to future work with various tasks (summarization, conversation, etc.) requiring diversity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is innovative, particularly with the introduction of L_mqs to enhance the diversity of question generation. The authors have carried out thorough research, encompassing both human and automated evaluations, to validate its efficacy.\n\n\n\n\n", "reasons_to_reject": "The experiment relies on BART, which is somewhat dated, and the size of BART is small. It would be beneficial if the authors could incorporate more experiments using recent advancements in LLM.\n\n\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a multi-question generation model designed to produce diverse and answerable questions grounded in context. Notably, the L_mqs proposed by the authors emphasizes generating a variety of questions. The authors further undertake extensive research, employing both human and automated evaluations, to showcase the model's efficacy. For the automated diversity assessment, the authors also adopt the self-bleu metric.\n\n\n\n\n", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vIExY2CRL3", "reviewer": "Reviewer_V2An", "summary": "", "questions": "A. Are EQG and QAG based on BART-large as well?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and constructive suggestions!\n\n### 1. Decoding methods\nThank you for your insightful feedback regarding decoding methods. Here are some reasons we only used beam search:\n+ Sultan et al. (2020) claim that sampling is more beneficial for improving QA model performance through QG. In fact, Sultan et al. (2020) argue that beam search performed better in terms of QG performance itself.\n+ Sampling-based methods include randomness in each inference, leading to inconsistent outputs. Considering that mQG undergoes multiple rounds of inference for a single context, we believe producing consistent outputs is crucial. Hence, we opted to utilize beam search, which enables us to achieve this desired consistency.\n+ Our main baselines, EQG and QAG papers, also exclusively employed beam search. Due to the fair comparison, we decided to follow their approach.\n\nWe apologize for omitting these details in the paper and will incorporate them in the camera-ready version if accepted.\n\n[1] Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. 2020. On the Importance of Diversity in Question Generation for QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5651–5656, Online. Association for Computational Linguistics.\n\n\n### 2. Only used one token-based automatic evaluation metric, Rouge-L\n|     | FairyTaleQA | TellMeWhy  | SQuAD      |\n|-----|-------------|------------|------------|\n| EQG | 0.9167      | 0.9129     | 0.8977     |\n| QAG | 0.9340      | 0.9342     | 0.9203     |\n| mQG | **0.9391**  | **0.9361** | **0.9210** |\n#### BERTScore (RoBERTa-large) Results\n\nThank you for bringing up this important point. We took your feedback and evaluated mQG on BERTScore [2]. This addition will allow us to consider semantic aspects that might not be adequately captured by Rouge-L.\n\nWe have constructed the table with our two main baselines. We are pleased to share that even with BERTScore [2], mQG consistently outperformed the baseline methods in all datasets.  Your feedback has been helpful in refining our evaluation strategy to ensure a more robust and meaningful assessment of our approach. We will add this result in the camera-ready version if accepted.\n\n[2] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi:\nBERTScore: Evaluating Text Generation with BERT. CoRR abs/1904.09675 (2019)\n\n### 3. Calculation of Rouge-L score\nAppreciating the reviewer's insights, regarding the Rouge-L F1 score computed on the FairytaleQA test dataset using a combination set of ground-truth questions sharing context, the score is 29.06. This particular score highlights a scenario where minimal overlap exists among the ground-truth questions. Considering this result, if multiple ground-truth questions had indeed matched some generated questions, it's reasonable to assume that the resulting score would have been significantly lower across the performance of all models. We will add this information in the camera-ready version if accepted.\n\n### 4. Reliability of human evaluation\n| Human Eval | Fleiss's kappa |\n|------------|---------------|\n| Type       |          0.53 |\n| Syntax     |          0.37 |\n| Content    |          0.26 |\n| Appro.     |          0.11 |\n| Ans.       |          0.15 |\n\nWe appreciate the reviewer's observation regarding Krippendorff's alpha values in the human evaluation section.\nIt's important to acknowledge that the reliability of Krippendorff's alpha can vary depending on the specific task being evaluated, the number of annotators, and the number of classes. Furthermore, it's also worth noting that even in the human evaluation by Cao and Wang (2021), which we followed, krippendorff's alpha did not exceed two-thirds. We have additionally calculated Fleiss’s kappa which is a similar inter-rater reliability test. Richard et al (1977) describe that Fleiss’s kappa score range between 0.20~0.40 indicates fair agreement on a condition of “two annotators, on two classes”. In the table, you can see all of our diversity evaluation results (Type, Syntax, Content) are above fair agreement even though we had five annotators to annotate three classes. For Appro. and Ans., as we had five annotators to rate on a five-point scale we believe it is an acceptable score.\n\n[3] Shuyang Cao and Lu Wang. 2021. Controllable Open-ended Question Generation with A New Question Type Ontology. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6424–6439, Online. Association for Computational Linguistics.\n\n[4] Landis, J. Richard, and Gary G. Koch. “The Measurement of Observer Agreement for Categorical Data.” Biometrics, vol. 33, no. 1, 1977, pp. 159–74. JSTOR, https://doi.org/10.2307/2529310.\n\n### 5. Weighting factor for MQS loss \nWe expect that adjusting weights might not lead to a significant increase in diversity for a couple of reasons:\n\n+ The MQS loss seeks the diversity of questions at the sentence level. This setup might inherently limit the diversity to the diversity range of the reference questions.\n\n+ The MQS loss already contributes one-third of the total initial loss. Given that BART's architecture generates text token by token, token-level loss (Cross-Entropy loss) is essential for ensuring the syntactic correctness of the generated questions. We think the amount of MQS loss is high enough to get to the optimal points.\n\n### Q. Are EQG and QAG based on BART-large as well?\nFollowing the implementation details described in their papers, we have used BART-large as their base model.\n\n### Missing reference\nThank you for pointing out the missing reference, which also explicitly modeled question types to generate questions.  Although Wang et al (2020)'s work is similar to mQG in utilizing question types there were some difficulties in using it as our baseline. Wang et al (2020) tried to get a question type distribution from context and answer pairs and use it to generate questions with different question types. Their approach of using answers as input additionally creates the need to annotate the answers for each context. This was not suitable for the large number of possible questions we were looking for. We will add this information in the related work of the camera-ready version if accepted.\n\n[5] Zhen Wang, Siwei Rao, Jie Zhang, Zhen Qin, Guangjian Tian, and Jun Wang. 2020. Diversify Question Generation with Continuous Content Selectors and Question Type Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2134–2143, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method increased the diversity of the generated questions for both in-domain and cross-domain settings, as measured by Self-BLEU.\n- As suggested by human evaluation, the proposed method can indeed diversify the types of generated questions by specifying QT in the inputs.", "reasons_to_reject": "- The authors claim that \"These decoding methods tried to sample tokens from less likely vocabularies. This solution is less relevant in more constrained tasks such as question generation and machine translation, where beam search is a preferred method.\" and exclude all sampling-based methods from the comparison. However, previous works, including one of the authors cited work by Sultan et al, have demonstrated the effectiveness of sampling-based methods for diversifying question generation.\n- The authors only used one token-based automatic evaluation metric, Rouge-L, for the quality evaluation, which may neglect semantic similarity and is not conclusive enough especially when the task is to increase the diversity.\n- The Rouge-L score is calculated as \"for each ground-truth question, we find the highest Rouge-L F1 score on generated questions from the same context than average overall scores.\", which may result in multiple groud-truth questions getting matched to few generated questions and leads to an incomprehensive evaluation of the quality of all the generated questions.\n- For the human evaluation part, most aspects have Krippendorf's alphas less than two-thirds, which indicates poor reliability. Only the question type aspect has a moderate level of reliability. \n- The effectiveness of explicitly modeling question types for diverse question generation has been studied in previous work[1].\n- Although the paper includes an ablation study that evaluated the architecture w/o the MQS loss, it remains unclear how the MQS loss affects the training and generations. It could be better if the authors can add a weighting factor and show the performance of the models in terms of question quality and diversity when the MQS loss is incorporated into the overall loss at different weights (particularly higher weights).", "questions_to_authors": "A. Are EQG and QAG based on BART-large as well?", "paper_topic_and_main_contributions": "This paper introduces a multi-question generation method (mQG) that focuses on generating diverse and answerable questions from a given context. The proposed method uses BART to recursively generate questions based on the input of question type, context, and previously generated questions. It enforces a maximum question similarity loss, which is intended to promote similar hidden representations for different questions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Diversify Question Generation with Continuous Content Selectors and Question Type Modeling https://aclanthology.org/2020.findings-emnlp.194/"}, "extra_text": {}}, {"review_id": "K0uoWsqZV4", "reviewer": "Reviewer_t52J", "summary": "", "questions": "I would like to put the following questions to the authors:\n\n- (A) What are your thoughts on question diversity if adjusting the relative weights of LCE and LMQS in the overall loss function?\n- (B) Still, related to the first question, is it feasible to modify the proposed training objective to incorporate the desired degree of diversity? \n- (C) Have other diversity metrics been considered for evaluating question diversity? For example, see work by Li et al. (2016), which has been applied in Wang et al. (2022) and in Bulathwela et al. (2023).\n- (D) Is there an idea on how to make the evaluation model improve the classification of implicit questions?\n- (E) Following this research, what is the future work?\n\nReferences:\n\nJiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.\n\nWang, Z., Valdez, J., Basu Mallick, D., Baraniuk, R.G. (2022). Towards Human-Like Educational Question Generation with Large Language Models. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_13\n\nBulathwela, S., Muse, H., Yilmaz, E. (2023). Scalable Educational Question Generation with Pre-trained Language Models. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_27", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews and constructive suggestions!\n\n### 1. Justification for choosing BART\n|     mQG    | # generated Questions Per Section | # Answerable Questions Per Section&uarr; | Rouge-L F1&uarr; | Self-BLEU&darr; |\n|:----------:|:---------------------------------------:|:----------------------------------------------:|:-------------------:|:---------------:|\n| T5-large  |                                   28.00 |                                          17.89 |               30.59 |      **0.5476** |\n| BART-large |                                   28.00 |                                      **23.35** |           **58.24** |          0.6243 |\n\nWe appreciate your thoughtful consideration of choosing a base model for building the final model. To address this concern, we conducted an in-depth performance comparison between BART and T5 on the FairyTaleQA dataset. For T5, we have adopted the prefix (‘generate question: ’) by Ushio et al. (2022). As shown in the table, T5 generated a significantly low number of answerable questions and Rouge-L F1 score. We think that as T5 is already trained with multiple supervised tasks in a text-to-text manner, it is not fit for our specific way of training mQG. This substantiates our decision to utilize BART as the base model for building our final model, mQG.\nWe will include these comparative results and the rationale for our choice in the camera-ready version if accepted.\n\n[1] Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2022. Generative Language Models for Paragraph-Level Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 670–688, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\n### 2. Explanation on CE loss\nThank you for the advice. In our observation, CE loss did play a vital role in making questions syntactically correct as it is calculated within the token level and mQS loss is calculated at the sentence level. Sorry for missing this information. We will add the essential role of CE loss in the camera-ready version if accepted.\n\n### 3. Limitations of Rouge-L for Quality Evaluation\nThank you for the suggestion to point out the limitation of the Rouge-L score. We looked through the reference papers you recommended. We totally agree that the Rouge-L score over ground-truth questions may not correlate to question quality. That is also the reason why we conducted a human evaluation on Table 3 to assess the quality of questions more comprehensively. In our human evaluation results regarding quality, there is a difference from the Rouge-L scores. All models have generated semantically correct questions. We will elaborate on this aspect and its connection to the limitations of Rouge-L score in the camera-ready version if accepted.\n\n|     | FairyTaleQA | TellMeWhy  | SQuAD      |\n|-----|-------------|------------|------------|\n| EQG | 0.9167      | 0.9129     | 0.8977     |\n| QAG | 0.9340      | 0.9342     | 0.9203     |\n| mQG | **0.9391**  | **0.9361** | **0.9210** |\n\n#### BERTScore (RoBERTa-large) Results\n\nAdditionally, we have evaluated mQG on BERTScore [2] with Reviewers (Reviewer V2An and t52J) concern over Rouge-L score. On all datasets, mQG outperformed our main baselines. We will add these evaluation results on BERTScore in the camera-ready version if accepted.\n\n[2] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi:\nBERTScore: Evaluating Text Generation with BERT. CoRR abs/1904.09675 (2019)\n\n### Questions for the Authors\nWe appreciate your insightful questions and suggestions for our future work. Here are our responses:\n### A, B) Relative Weights Outcome\nAdjusting the relative weights to achieve higher diversity seems difficult at this time. The MQS loss operates at the sentence level, comparing reference questions. This setup might inherently limit the diversity to the diversity range of the reference questions.\n\n### C) Diversity Metrics\nWe sincerely appreciate your thoughtful suggestion to explore the distinct-n score as an additional diversity metric. While we recognize the merits of this metric and its potential to enhance our evaluation, we have chosen to focus solely on the self-BLEU metric for assessing question diversity. As we know, a distinct-n score counts distinct n-grams in a single sentence. Our task is to check diversity between questions so we think Self-BLEU is more favorable for our paper. We understand the potential benefits of considering additional diversity metrics and will certainly keep this suggestion in mind.\n\n### D) Improving Implicit Question Classification\n+ Shown in Table 7 (page 12), only about 27% of questions were implicit questions. We think the low accuracy of implicit questions compared to explicit questions is due to this low number of implicit questions. \n+ Some conditions of unanswerable questions in SQuAD2.0 can be directed to the definition of implicit questions. This may cause model confusion.\nDue to the reasons mentioned above, we believe that the lower classification results observed for implicit questions can be attributed to issues within the datasets. However, problems stemming from dataset quality have been widely acknowledged within the realm of Natural Language Generation (NLG). We consider an approach that utilizes the Answer Generation (AG) model to address this challenge.\n\nThe proposed strategy involves training the AG model to generate responses of \"No Answer\" for questions that are inherently unanswerable and to produce actual answers for questions that can be answered, regardless of whether they are explicit or implicit in nature. During the inference phase, a specific threshold probability for generating the \"No Answer\" response could be established. This threshold could serve as a criterion for identifying questions that are likely unanswerable. By adopting this methodology, we anticipate a potential enhancement in the categorization of questions, thus streamlining the process of obtaining accurate answers for such inquiries.\n\n### E) Future Work\nBuilding on this research, we have identified potential directions for future work:\n+ In this paper, we used explicit modeling of question types. Recognizing the complexity of language and the multitude of ways questions can be formulated, we acknowledge that our current modeling might not encompass all subtleties. To illustrate this point, consider the work by Olney et al. (2012), which delved into categorizing questions with a psychological aspect. We plan to apply this nuanced approach to question categorization, which we believe will further enhance the performance of mQG.\n+ With the rapid and continuous advancements in LLMs, we are excited about the potential applications of our method. By extending our approach to exploit the power of LLMs, we can address a wide range of NLG tasks (summarization, dialogue, etc.) that demand outputs with varying degrees of complexity and diversity.\n+ With our current answerability evaluation model, we encountered limitations in its ability to reliably classify implicit questions, which are inherently more challenging due to their nuanced nature. As implicit questions often lack explicit indicators, they can lead to misclassifications, affecting the overall performance. We will work on the AG model to make better classifications of questions into answerable or not.\n\n[3] Olney, Andrew M., Arthur C. Graesser and Natalie K. Person. “Question Generation from Concept Maps.” Dialogue Discourse 3 (2012): 75-99.\n\n### Missing reference\nSorry for the confusing reference. The few studies indicate three reference papers in lines 173~175. We will revise this missing part and make changes in the camera-ready version if accepted.\n\n### Typos, Grammar, Style, and Presentation\nThank you for pointing out areas where improvements are needed in our writing, figure design, and table formatting. We will make the following changes in the revised version.\n+ Add an outline of the paper and objectively state contribution and findings in the introduction\n+ Change the purple line into the blue line in Figure 1\n+ Range explanation on each evaluation metrics\n+ Check for additional typos", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of the paper, as perceived by me, are as follows:\n\n- The generation method. In particular, introducing a maximum question similarity loss (LMQS) is grounded and relevant for generating diverse questions, which is a challenge in question generation. The use of a recursive generation framework should also be highlighted positively;\n- Robustness of evaluation. The evaluation is comprehensive, varied, and in-depth, with multiple studies attesting to the quality and diversity of the questions generated. In particular, the answerability model, and human evaluation should be highlighted positively. Zero-shot evaluation shows the applicability of the model in other contexts;\n- The comparison and discussion of the results with recent baselines highlight the value of the work;\n- The ablation studies and further analysis of the evaluation model are helpful in order to better understand the proposed model's limitations and applications.", "reasons_to_reject": "The weaknesses of the paper, as perceived by me, are as follows:\n\n- Lack of justification in the choice of the base model (BART) for building the final model (mQG). While BART has demonstrated remarkable performance in various NLP tasks, it has also been shown that, for example, T5 may be better for the specific task of question generation (Ushio et al., 2022). If BART is indeed the best choice, I suggest that the authors better substantiate its use;\n- Lack of justification for combining LCE (cross-entropy loss) and LMQS (maximum question similarity loss). I wonder if the overall training objective may lead to a trade-off between promoting closeness to reference questions and promoting diversity in question generation. I suggest that the authors elaborate on this topic beyond just referring that cross-entropy loss is also used (at the end of Section 3.2);\n- Measuring the quality of generated questions with Rouge-L F1 score. Specifically, Callison-Burch et al. (2006) and Liu et al. (2016) have shown that automatic metrics may not correlate well with quality since they essentially compute the n-gram similarity between the ground-truth and generated text. I suggest that the authors address the metric's limitations, particularly in this paper's context, where the aim is to generate diverse questions.\n\nReferences:\n\nAsahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados. 2022. Generative Language Models for Paragraph-Level Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 670–688, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\n\nChris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 249–256, Trento, Italy, April 2006.\n\nChia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas, November 2016.", "questions_to_authors": "I would like to put the following questions to the authors:\n\n- (A) What are your thoughts on question diversity if adjusting the relative weights of LCE and LMQS in the overall loss function?\n- (B) Still, related to the first question, is it feasible to modify the proposed training objective to incorporate the desired degree of diversity? \n- (C) Have other diversity metrics been considered for evaluating question diversity? For example, see work by Li et al. (2016), which has been applied in Wang et al. (2022) and in Bulathwela et al. (2023).\n- (D) Is there an idea on how to make the evaluation model improve the classification of implicit questions?\n- (E) Following this research, what is the future work?\n\nReferences:\n\nJiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.\n\nWang, Z., Valdez, J., Basu Mallick, D., Baraniuk, R.G. (2022). Towards Human-Like Educational Question Generation with Large Language Models. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_13\n\nBulathwela, S., Muse, H., Yilmaz, E. (2023). Scalable Educational Question Generation with Pre-trained Language Models. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_27", "paper_topic_and_main_contributions": "The main contributions of the paper, as perceived by me, are as follows:\n\n- The introduction of a multi-question generation model that aims to generate diverse questions;\n- An in-depth experimental study on the feasibility of the diversity question generator through an evaluation that includes several baselines, an answerability model, human feedback and zero-shot scenario;\n- Ablation studies for further evaluating the generator robustness, allowing the reader to analyze the importance and interplay of different factors within the model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "In section 2.2 (Diversity), what exactly are the few studies that addressed diversity as a challenge in QG?"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest that the authors objectively state the contributions and findings in the Introduction.\nThe outline of the article can also be explained there.\n\nIn Figure 1, the arrows have three different colors, which come from boxes with other colors. I suggest that the authors use fewer colors for ease of following.\n\nIn Table 1 and others, the ranges for each metric are missing.\n\nThere are sentences where the word's first letter appears in the upper case, after a comma. For example, see page 1, line 78, line 8, line 571."}}]

Author rebuttal:

Meta-review: Main contributions:
* This paper introduces a multi-question generation model aimed at producing a wide array of contextually grounded, answerable questions with enhanced diversity. Notably, the LMQS (Maximum Question Similarity Loss) proposed by the authors underscores the generation of a variety of questions. The authors conduct a comprehensive research study, incorporating both human and automated evaluations, to demonstrate the model's effectiveness. In the context of automated diversity assessment, the authors also employ the Self-BLEU metric.

Reasons to accept:
* The approach demonstrates innovation, particularly through the introduction of LMQS to amplify the diversity in question generation. The authors have conducted extensive research, encompassing both human and automated evaluations, to validate its effectiveness. The ablation studies and further analysis of the evaluation model are helpful in order to better understand the proposed model's limitations and applications.

Reasons to reject:
* The use of BART was criticized as outdated and T5 was suggested as a more modern choice. In the rebuttal, the authors noted that T5 did not perform as well and will add details to the paper.
* Combination of cross entropy and the new LMQS. The authors addressed this concern by explaining that the mix with cross entropy was essential for better performance and will explain this in the final version.
* The use of ROUGE-L was criticized. In the rebuttal, the authors have addressed this concern with the use of BertScore.
* Krippendorff's alpha was criticized for being below two-thirds, but this has been shown to be typical for NLP tasks.
Decision: Accept-Main