Improving Question Generation with Multi-level Content Planning
Authors: Zehua Xia
Qi Gou
Bowen Yu
Haiyang Yu
Fei Huang
Yongbin Li
Nguyen Cam-Tu
Venue: EMNLP 2023 Findings (2023)
Keywords: questin generation
multi-level content planning

Abstract: This paper addresses the problem of generating questions from a given context and an answer, specifically focusing on questions that require multi-hop reasoning across an extended context. Previous studies have suggested that key phrase selection is essential for question generation (QG), yet it is still challenging to connect such disjointed phrases into meaningful questions, particularly for long context. To mitigate this issue, we propose  MultiFactor, a novel QG framework based on multi-level content planning. Specifically, MultiFactor includes two components: FA-Model, which simultaneously selects key phrases and generates full answers, and Q-Model which takes the generated full answer as an additional input to generate questions. Here, full answer generation is introduced to connect the short answer with the selected key phrases, thus forming an answer-aware summary to facilitate QG. Both FA-Model and Q-Model are formalized as simple-yet-effective Phrase-Enhanced Transformers, our joint model for phrase selection and text generation. Experimental results show that our method outperforms strong baselines on two popular QG datasets. Our code is available at https://github.com/zeaver/MultiFactor.

Improving Question Generation with Multi-level Content Planning
Zehua Xia', Qi Gou', Bowen Yu², Haiyang Yu², Fei Huang?
Yongbin Li and Cam-Tu Nguyen
State Key Laboratory for Novel Software Technology, Nanjing University, China Allibaba Group
{zehuaxia, ead.gal@@smail.... htjoi/emmmannnu......... www..nu..cccooom.mmmm edu. cn
(yubowen. yifei yhy, f. (bbanaa@llibabaiingg shuide. lyylllyllbbbaaaaa com ncamtuênju. edu. cn
Abstract
Source Paragraph 1: Richard Homssy & Sons Richard Harnsby
engine machinery manufacturer in Lincoloshire, England from 1828 until The company was pioneer manufacture of the oil engine develcped by Herbert Alroyd Stuart, which was marieted under the "Hormsy-skneyd track systom vehides, solling patent Holt co. lprodocessor to Caterpillar Inc.] in America, in 1918, Richard Harmiiy & Sons became subsidiary of the reighbouring engineering film Rustons of Lincoin, create "Buston Hornsby".
This paper addresses the problem of generat- ing questions from a given context and an an- swer, specifically focusing on questions that require multi-hop reasoning across extended context. Previous studies have suggested that key phrase selection is essential for question generation (QG), yet it is still challenging to connect such disjointed phrases into meaning- ful questions, particularly for long context. To mitigate this issue, we propose MultiFactor, a novel QG framework based on multi-level content plamning, Specifically, MultiFactor in- cludes two components: FA-model, which si- multaneously selects key phrases and gener- ates full answers, and Q-model which takes the generated full answer as an additional in- put generate questions. Here, full answer generation is introduced to connect the short answer with the selected key phrases, thus forming an answer-aware summary to facilitate QG. Both FA-model and Q-model are formal- ized as Transformers, our joint model for phrase selec- tion and text generation. Experimental results show that our method outperforms strong base- lines on two popular QG datasets. Our code is available at https://gi thub, com/zeaver/ Mult Multor
The
developed carly
Source Paragraph 2: Akroyd Stuart
Herbert Alroyd Stuart January 1864, February 1927, Hallas) was an English insentor who noted for invention of the hot bulb engine, or heavy oil engine. Atroyd-Stuart was born in Halifax, Yorkshire, but lived Australia for period in his early at Newbury Grammar School Inow st. Bartholomew's Schooll and Finsbury Technizal College London.
Yorkshire, England 19
educated
What the date of birth of the Engish inventor that Bevelloped the Ricard Hornsb Sons engine?
Gold Question:
Arswer: -on-ee--- 1864
Vanilla QG: When was the Englis inwentor Herbert Alroyd Stuart born?
Phrase-leval QG: When was the English inventor who developed the Engine bom?
MultiFactor: When was English Inventor Who developed the engine
plonsered by Richard Homaay & Sons born?
Generated Full Answer: The English inventor who devatoped the of engine
pioneered Richard Homsky Sons born January
Figure 1: An example from HotpotQA in which the question generated by MultiFactor requires reasoning over disjointed facts across documents.
Thrase-Enhanee
tems to enhance the quality of the output ques- tions. This task is particularly important for gen- erating complex questions, that require reasoning over long context. Based on the content granu- larity, prior research (Zhang et al., 2021) can be broadly categorized into two groups: phrase-level and sentence-level content planning. On one hand, the majority of prior work (Sun et al., 2018; Liu et al., 2019; Pan et al., 2020; Cao and Wang. 2021; Fei et al., 2022: Subramanian et al., 2018) has fo- cused on phrase-level planning, where the system identifies key phrases in the context and generates questions based on them. For instance, given the answer "28 January 1864" and a two-paragraphs context in Figure 1, can recognize "English Inventor," "the oil engine." "Herbert Akroyd Stu- art" important text for generating questions. For long context, however, it still challenging for machines to connect such disjointed facts to form meaningful questions. On the other hand, sentence-
1 Introduction
Question Generation (QG) is a crucial task in the field of Natural Language Processing (NLP) which focuses on creating human-like questions based on given source context and specific answer. In recent years, QG has gained considerable atten- tion from both academic and industrial commu- nitics due to its potential applications in question answering (Duan et al., 2017), machine reading comprehension (Du et al., 2017), and automatic conversation (Pan et al., 2019; Ling et al., 2020).
Effective content planning is essential for QG sys-
*Corresponding authors.

---

level content planning, as demonstrated by Du and Cardie (2017), aims at automatic sentence selec- tion to reduce the context length. For instance, given the sample in Figure I, one can choose the underscored sentences to facilitate QG. Unfortu- nately, it is observable that the selected sentences still contain redundant information that may nega- tively impact question generation. Therefore, we believe that an effective automatic content planning both the phrase and the sentence levels crucial for generating questions.
the concept of full answers in an attempt of multi-level content planning for QG. As such, our study helps shed light on the influence of the answer-aware summary on QG.
We design MultiFactor framework follow- ing a simple yet effective pipeline of Phrase- enhanced Transformers (PET), which jointly model the phrase selection task and the text generation task. Leveraging the power of pretrained language models, PET achieves high effectiveness while keeping the addi- tional number of parameters fairly low in com- parison to the base model.
this paper, we investigate a novel framework. MultiFactor, based on multi-level content planning for QG. At the fine-grained level, answer-aware phrases are selected as the focus for downstream QG. At the coarse-grained level, full answer generation is trained to connect such (disjointed) phrases and form complete sentence. Intuitively, full answer can be regarded as an answer-aware summary of the context, from which complex ques- tions are more conveniently generated. As shown in Figure I, MultiFactor is able to connect the short answer with the selected phrases, and thus create a question that requires more hops of reasoning compared to Vanilla QG. It is also notable that we follow generative approach instead of selection approach (Du and Cardie, 2017) sentence-level content planning. Figure demonstrates that our generated full answer contains more focused infor- mation than the selected (underscored) sentences.
Experimental results validate the effectiveness of MultiFactor on two settings of HotpotQA, a popular benchmark on multi-hop QG, and SQuAD 1.1, dataset with shorter context.
Related Work
Early Question Generation (QG) systems (Mostow and Chen, 2009; Chali and Hasan, 2012: Heilman, 201 1) followed rule-based approach. This ap- proach, however, suffers from a number of issues, such as poor generalization and high-maintenance costs. With the introduction of large QA datasets such as SQuAD (Rajpurkar et al., 2016) and Hot- potQA (Yang al., 2018), the neural-based ap- proach has become the mainstream in recent years. In general, these methods formalize QG a problem (Du et al., 2017), on which a number of innovations have been made from the following perspectives.
Specifically, MultiFactor includes two components: I) A FA-model that simultaneously selects key phrases and generate full answers; and 2) A Q- model that takes the generated full answer additional input for QG. To realize these compo- nents, we propose Phrase-Enhanced Transformer (PET), where the phrase selection is regarded as a joint task with the generation task both in FA- FA-model and Q-model. Here, the phrase selection model and the generation model share the Transformer en- coder, enabling better representation learning for both tasks. The selected phrase probabilities are then used to bias to the Transformer Decoder to fo- more the answer-aware phrases. In general, PET is simple yet effective as we can leverage the power of pretrained language models for both the phrase selection and the generation tasks.
Enhanced Input Representation Recent ques- tion generation (QG) systems have used auxiliary information to improve the representation of the input sequence. For example, Du et al. (2017) used paragraph embeddings to enhance the input sen- tence embedding. Du and Cardie (2018) further improved input sentence encoding by incorporating co-reference chain information within preceding sentences. Other studies (Su al., 2020; Pan al., 2020; Fei et al., 2021; Sachan et al., 2020a) en- hanced input encoding by incorporating semantic relationships, which are obtained by extracting a se- mantic or entity graph from the corresponding pas- sage, and then applying graph attention networks (GATs) (Veličković et al., 2018).
Our main contributions are summarized as follows:
One of the challenges in QG is that the model might generate answer- inreee-tr questions, such as pro-
To our knowledge, we are the first to introduce

---

ducing inappropriate question words for given answer. To overcome this issue, different strategies have been proposed to effectively exploit answer information for imput representation. For example, Zhou et al. (2017); Zhao et al. (2018); Liu et al. (2019) marked the answer location in the imput pas- sage. Meanwhile, Song et al. (2018); Chen et al. (2020) exploited complex passage-answar interac- tion strategies. Kim et al. (2019); Sun al. (2018), on the other hand, sought to avoid answer-included questions by using separating encoders for answers and passages. Compared to these works, we also aim to make better use of answer information but we do so from the new perspective of full answers.
construction. In addition, PET is able leverage the power of pre-trained language models for its ef- fectiveness. Thirdly, we perform content planning at the sentence level by following the generative approach instead of the extraction approach as pre- sented in the study by Du and Cardie (2017). The example in Figure shows that our generated full answer contains less redundant information than selecting entire sentences of supported facts.
Diversity While the majority of previous studies focus on generating context-relevant questions, re- cent studies (Cho et al., 2019; Wang et al., 2020b; Fan et al., 2018; Narayan et al., 2022) have sought to improve diversity of QG. Although not yet consider the diversity issue, vides a convenient way improve diversity while maintaining consistency. For example, one can per- form diverse phrase selection or look for diverse ways to turn full answers into questions. At the same time, different strategies can be used to make sure that the full answer is faithful to the given context, thus improving the consistency.
framework pro-
Content Planning The purpose of content plan- ning is to identify essential information from con- text. Content planning is widely used in in text generation tasks such QA/QG, dialogue system (Fu et al., 2022: Zhang et al., 2023; Gou et al., 2023), and summarization (Chen et al., 2022). Pre- vious studies (Sun et al., 2018; Liu et al., 2019) predicted "clue" words based on their proximity to the answer. This approach works well for sim- ple QG from short contexts. For more complex questions that require reasoning from multiple sen- tences, researchers selected entire sentences from the input (documents, paragraphs) the focus for QG, as in the study conducted by Du and Cardic (2017). Nevertheless, coarse-graine guasse-rained content plan- ning at the sentence level may include irrelevant information. Therefore, recent studies (Pan et al., 2020; Fei et al., 2021, 2022) have focused on ob- taining finer-grained information the phrase level for question generation. In these studies, semantic graphs are first constructed through dependency parsing or information extraction tools. Then, a node classification module is leveraged to choose essential nodes (phrases) for question generation.
3 Methodology
3.1 MultiFactor Question Generation
Given a source context [W1, W/2 ... Wy, and answer [01, 22, , .r.. the objective is to generate relevant question q (g), 42. ..... 9r,l; where Te, Ta. and Tg denote the number of tokens in c, a and g respectively. we can generate full answers [s1, 82, . .r,] of Ta tokens, thus obtaining answer-relevan sum- maries of the context. The full answers are subse- quently used for generating questions as follows:
is presumed that
p(qlc, a) Eg ppppp,ss
(1)
Q model
FA model
where Q model and FA model refer to the question generation and the full answer generation mod- els, respectively. Each Q-model and FA-model is formalized as a Phrase-enhanced Transformer (PET), our proposal for text generation with phrase planning. In the following, we denote a PET as Φ x > where x is the input sequence and y is the output sequence. For the FA-model, the input sequence is e @ a and the output is the full answer s, where indicates string concatenation. As for the Q-model, the input is x = c ( a ! s with s being the best full answer from FA FA-model, and the output is the question q. The PET model Ø
Our study focuses on content plamning for Question Generation (QG) but differs from previous stud- ies in several ways. Firstly, we target automatic content-planning at both the fine-grained level of phrases and the coarse- -raiined level of sentences. As far as we know, we are the first that consider multiple levels of granularity for automatic content planning. Secondly, we propose novel phrase- enhanced transformer (PET) which a simple yet effective for phrase-level content planning. Com- pared to Graph-base based methods, PET is relatively simpler as it eliminates the need for semantic graph

---

p()
91
iquestiono
(6:)
FFN
Q Model
{s}
Softnas() Lisart))
Multi-Head Cross Attention
-onswer>
<full answers
[b2)
K
V
full anowers
H
FA Model
Encoder
Masked Multi-Head Self Attention
scontexT> OOREer
X
X legars
Phraa not
Mox Mazn Poaking
C, Low High Probubiitty
Figure Overview of our MultiFactor is shown the left. Here, FA-model and Q-model share the same architecture of Enhanced Transformer demonstrated on the right.
firsts select phrases that can consistently be used generate the output, then integrates the phrase probabilities as soft constraints for the decoder to do generation. The overview of MultiFactor is
encoding obtained from the transformer encoder:
H = Encoder(z))
h MeauMMxxPooling (11 fi
demonstrated in Figure 2. The Phrase-Enhanced Transformer is detailed in the following section.
Softmax [Linear [h]]}
where II E RTxx with T. and d being the length of input sequence and dimensions of hidden states, respectively. Here, Encoder indicates the Trans- former encoder, of which the details can be found in (Devlin al., 2019). The phrase representation h is obtained by concatenating MaxPoolingg() and Mean Pooling()) of the hidden states { corre- sponding to i-th phrase. We then employ a linear network with Softmax(() as the phrase selection probability estimator (Galke and Scherp, 2022).
3.2 Phrase Enhanced Transformer
We propose Phrase-enhanced Transformer (PET), simple yet effective Transformer-based model to infuse phrase selection probability from encoder into decoder improve question generation.
Formally, given the input sequence x, and I. phrase candidates, the i-th phrase wi (i E {1.... L}) is a sequence of L; tokens { {unq "i:. "......,
ex-
tracted from the context x, where 15 indicates the index of token j of the i-th phrase in x. The phrase- level content plamning is formalized as assigning a label E [0, 1] to each phrase in the candidate pool, where is if the phrase should be selected and 0 otherwise. The phrase information is then integrated generate auto-regressively:
Probabilistic Fusion in Decoder Decoder sumes previously generated tokens Y1... then generates the next one as follows:
= SoftmmaxLLinerr [DecLayers(yn..t -SStttax(Liieerreelllaeess..... H)]]
where H is the Encoder output, and DecLayers indicates a stack of N decoder layers. Like Trans- former, each PET decoder layer contains three sub- layers: 1) the masked multi-head attention layer; 2) the multi-head cross aoteenttonnion layer; 3) the fully connected -ordwfd network. Considering the multi-head cross-attention sublayer is the interac- tion module between the encoder and decoder, we modify it to take into account the phrase selection probability 2; as shown Figure
p(y|0,2 2) II p(u xr No:t-1)
Encoder and Phrase Selection Recall that the input x contains the context e and the answer a in both Q-model and FA-model, and thus we select the candidate phrases only from the context by extracting entities, verbs and noun phrases using SpaCy'. The phrase selection is formalized as bi- nary classification task, where the input is phrase
Here, we detail the underlying mechanism of each cross-atted cross-attention head and how we modify to en- code phrase information. Let us recall that the
'htpss//spacy.io/

---

input for a cross-attention layer includes a query state, a key state, and value state. The query state Q" is the (linear) projection of the output of the first sublayer (the masked multi-head attention layer). Intuitively, Q" encapsulates the information about the previously generated tokens. The key state k = HW* and the value state V HW are two linear projections of the Encoder output H. W Rdxxx and Wo Rdddd are the layer parameters, where dy and d, are the dimensions of the key and value states. The output of the cross- attention layer is then calculated as follows:
Ho
Datasct
SQuAD 1.1
Sup.
Full
Context Len.
49.3
210.7
26.8
Question Len.
18.0
18.0
10.9
Train/Dew/Test
89947/500/7405
86635/8965/8964
Table The statistics of HotpotQA and SQuAD I.I, where Supp. and Full indicate the supporting facts set- ting and the full setting of HotpotQA.
simpler task on SQuAD 1.1. There are two set- tings for HotpoiQA (see Table I): 1) HotpotQA (sup. facts) where the sentences that contain sup- porting facts for answers are known in advance; 2) HotpotQA (full) where the context is longer and contains several paragraphs from different docu- ments. For SQuAD 1.1, we use the split proposed in Zhou et al. (2017). Although our MultiFactor is expected to work best on HotpotQA (full), we consider HotpotQA (sup. facts) and SQuAD 1.1 to investigate the benefits of multi-level content planning for short contexts.
QK
CrossAtten K,
KQ, V) = Softmax
V Vok
Here, we drop the superscripts for simplicity, but the notations should be clear from the context. The- oretically, one can inject the phrase information either V or K*. In practice, however, updat- ing the value state introduces noises that counter the effect of pretraining Transformer- based models, which are commonly used for generation. As a result, we integrate phrase probabilities to the key state, thus replacing k by new key state Kth
Metries Following previous studies, we exploit commonly-used metries for evaluation, including BLEU (Papineni et al., 2002), METEOR (Baner- jee and Lavie, 2005) and ROUGE-L (Lin, 2004). We also report the recently proposed BERTScore (Zhang et al., 2020) in the ablation study.
khh = S + Hw*
5, 2,, =]; j in th phrase
[1,0], j not in any phrase
where W6 E R2xdx is the probabilistic fusion layer. Here, 2; the groundtruth phrase label for phrase during training (2: E {0, 1J),, and the pre- dicted probabilities to select the i-th phrase during inference (z; E [0, 1)). In @-model, we choose all tokens w; in the full answer important tokens.
Implementation Details We exploit two base models for MultiFactor: T5-base and MixQG- base 3, To train FA-model, we apply QA2D (Dem- szky et al., 2018) to convert question and answer pairs to obtain pseudo (gold) full answers. Both Q-model and FA-model are trained with À of 1. Our code is implemented on Huggingface (Wolf et al., 2020), whereas AdamW (Loshchilov and Hutter, 2019) is used for optimization. More training de- tails and data format are provided in Appendix B.
Training Given the training data set of triples (z, 2, y), where æ is the input, y is the groundtruth output sequence and indicates the labels for phrases that can be found in y. we can simulta- neously train the phrase selection and the text gen- eration model by optimizing the following loss:
Baselines The basclines (in Table 2) can be grouped into several categories: 1) Early seq2seq methods that use GRU/LSTM and attention for the input representation, such as SemQG, NGQ++, and E2sa-at-mep-ggg 2) Graph-based methods for content planning like ADDQG, DP-Graph. IGND, CQG, MulQG, GATENLL+CT, Graph2seq+RL; 3) Pretrained-languaaee models based methods, includ- ing Ta-base, CQG, MixQG, and QA4QG. Among these baselines, MixQG and QA4QG are strong
L = CrossEntroppy (ů. sl + X CrossEntropy (2, x]
where & is the predicted labels for phrase selection, ÿ is the predicted output, X is hyper-parameter.
4 Experiments
4.1 Experimental Setup
Datasets We evaluate our method on two differ- ent QG tasks: a complex task on HotpotQA and
https://huuginnffic....t-ts-ssa coltS--- http://mwnginnifeecccouuustttttr

---

Model
B-1
B-2
B-3
B-4
MTR
R-L
Encoder Inpul: Supporting Facts Sentences
SemQG (Zhang and Bansal, 2019)
39.92
26.73
18.73
14.71
19.29
35.63
ADDQG (Wang et al., 2020a)
44.34
31.32
22.68
17.54
20.56
38.09
F+R+A (Xie al., 2020)
37.97
15.41
19.61
35.12
DP-Graph (Pan ct al., 2020)
40.55
27.21
20.13
15.53
20.15
36.94
IGND (Fei et al., 2021)
41.22
24.71
18.99
16.36
24.19
38.34
T5-hase (Raffel et al., 2020)
47.78
36.39
29.44
24.48
25.59
43.17
CQG (Fei et al., 2022)
49.71
37.04
29.93
25.09
27.45
41.83
HotporQA
MixQG-base (Murakhovs') et al., 2022)f
49.60
37.78
30.58
25.45
26.36
43.21
QA4QG-large (Su al., 2022)
49.55
37.91
30.79
25.70
27.44
46.48
MultiFactor (T5-hase)
53.46
40.95
33.29
27.80
28.25
43.80
MultiTactor (MixQG-base)
54.17
41.50
33.74
28.22
28.60
44.17
Encoder Input: Full Document Context
MulQG (Su al., 2020)
40.15
26.71
19.73
15.20
20.51
35.30
GATENLL+CT (Sachan et al., 2020b)
20.02
22.40
39.49
T5-hase (Raffel et al., 2020)
42.68
31.67
25.21
20.70
22.57
40.25
MixQG-base (Murakhovs') ka et al., 2022)
45.28
33.72
26.90
22.13
23.78
41.21
QA4QG-large (Su et al., 2022)
46.45
33.83
26.35
21.21
25.53
42.44
MultiFactor (TS-base)
51.41
39.31
31.90
26.66
29.66
43.37
MultiFactor (MixQG-bass
54.84
42.41
34.69
29.12
30.01
45.20
NQG++ (Zhou et al., 2017)
42.46
26.33
18.46
13.51
s2sa-at-mep-ge gsn (Zhao al., 2018)
44.51
29.07
21.06
15.82
19.67
44.24
APM (Sun et al., 2018)
43.02
28.14
20.51
15.64
SQuAD 1.1
Graph2seq+RI. (Chen et al., 2020)
18.30
21.70
45.98
T5-hase (Raffel et al., 2020)
47.96
33.58
25.54
20.15
24.21
40.33
IGND (Fei et al., 2021)
50.82
34.73
25.64
20.33
48.94
MixQG-base (Murakhovs' ka et al., 2022)f
49.69
35.19
26.70
21.44
25.48
41.22
MultiFactor (T5-base)
49.56
35.00
26.78
21.24
25.63
41.22
MultiFactor (MixQG-base)
50.51
35.78
27.42
21.75
25.55
41.62
Table 2: Automatic evaluation results on HotpotQA (Yang et al., 2018) and SQuAD 1.1 (Rajpurkar et al., 2016).
The Bold and underline mark the best and second-best results. The B-x, MTR, and R-L mean BLEU-x, METEOR, and ROUGE-L, respectively. We mark the results reproduced by ourselves with y other results are from Fei et al. (2022), Su et al. (2022) and Fei et al. (2021).
ones with QA4QG being the state-of-the- art model HotpotQA. Here, MixQG is pretrained model tailored for the QG task whereas QA4QG exploits Question Answering (QA) model to enhance QG.
large models (for QA and QG) with total number of parameters of 800M, whereas MultiFactor has total number of parameters of around 440M corre- sponding two T5/MixQG-base models. Here, the extra parameters associated with phrase selection in PET (T5/MixQG- is only 0.02M, which is relatively small compared to the number of param- eters in T5/MixQG-bas.
4.2 Main Results
The performance MultiFactor and baselines are shown in Table 2 with the following main insights.
On HotpotQA, it is observable that our method obtains superior results on nearly all evaluation metrics. Specifically, MultiFactor outperforms the current state-of-the-an model QA4QG by about 8 and 2.5 BLEU-4 points in the full and the support- ing facts setting, respectively. Note that we achieve such results with a smaller number of model param- eters compared to QA4QG-large. Specifically, the current state-off of ooftthe-rr art model exploits two BART-
By cross-referencing the performance of common baselines (MixQG or QA4QG) on HotpotQA (full) and HotpotQA (supp. facts), it is evident that these basclines are more effective on HotpotQA (supp. facts). This is intuitive since the provided support- ing sentences can be regarded as sentence-level content planning that benefits those on HotpotQA (supp. facts). However, even without this advan- tage, MultiFactor on HotpotQA (full.) outperforms

---

Model
B-4
MTR
R-L
BSc
Model
B-4
MTR
R-L
BSc
HotporQA SSupporting Facts)
PET-Q
27.45
28.28
43,46
52.41
MultiFactor
27.80
28.26
43.80
52.86
Fine-tuned
25.45
26.36
43.21
51.49
Q-model
Cls+Gen
25.90
26.73
43.55
52.04
w/o Context
27.63
28.13
43.66
52.69
One-hot PET-Q
27.48
28.28
43.46
52.63
Oracle-FA
31.61
29.66
48.84
56.15
PET-Q
27.79
28.46
43.94
53.05
w Gold-FA
91.08
64.10
93.00
93.77
MultiFactor
28.22
28.60
44.17
53.44
HotpotQA (Full Document)
Table 4: Results MultiFactor (T5-base) and its vari- ants on HotpotQA (supp. facts).
Fine-tuned
22.13
23.78
41.21
48.76
Cls+Gen
22.39
23.95
41.40
48.95
One-hot PET-Q
26.61
28.94
43.11
52.21
PET-Q denotes MultiFactor without the full an- swer information.
PET-Q
26.82
29.04
43.53
52.58
MultiFactor
29.12
30.01
45.20
54.49
SQuAD
Phrase-level Content Planning By comparing PET-Q, one-hot PET-Q and Cls+Gen to the fine- tuned MixQG-base Table 3, we can draw several observations. First, adding the phrase selection task helps improve QG performance. Second, inte- grating phrase selection to the decoder (in One-hot PET-Q and PET-Q) is more effective than just ex- ploiting phrase classification (as in Cls+Gen). Finally, it is recommended to uti- lize soft labels (as in PET-Q) instead of hard labels (as One-hot PET-Q) to bias the decoder.
Fine-tuned
19.96
24.39
39.77
55.31
Cls+Gen
20.14
24.45
39.83
55.34
One-hot PET-Q
21.10
25.35
41.10
56.52
PET-Q
21.33
25.38
41.58
56.89
MultiFactor
21.75
25.55
41.62
56.93
Table 3: The ablation study for MultiFactor (MixQG- base), MultiFactor (T5-base) is shown in Appendix C.
additional task
these baselines on HotpotQA (supp. facts), show- ing the advantages of MultiFactor for long context.
Sentence-level Content Planning By compar- ing MultiFactor to other variants in Table 3, be- comes apparent that using the full answer predic- tion helps improve the performance QG most cases. The contribution of the FA-model is particu- larly evident in HotpotQA (full), where the context is longer. In this instance, the FA-model provides answer-aware summary of the context, which benefits downstream QG. In contrast, for SQuaD where the context is shorter, the FAAmmoeel still helps but its impact appears be less notable.
On SQuAD, MultiFactor is better than most base- lines multiple evaluation metrics, demonstrating the benefits of multi-level content planning even for short-contexts. However, the margin of improve- ment not as significant as that seen on HotpotQA. MultiFactor falls behind some baselines, such as IGND, terms ROUGE-L. This could be due to the fact that generating questions on SQuAD requires information mainly from a single sentence. Therefore, simple copy mechanism like that used in IGND may lead to higher ROUGE-L.
4.4 The Roles Q-model and FA-model
4.3 Ablation Study
We investigate two possible causes that may impact the effectiveness of MultiFactor, including poten- tial errors in converting full answers questions in Q-model, and error propagation from the FA-model to the Q-model. For the first cause, we evaluate Q- model (w/ Gold-FA), which takes as input the gold full answers, rather than FA-model outputs. For the second cause, we assess Q-model (w/o Con- text) and Q-model (w/ Oracle-FA). Here, Q-model (w/ Oracle-FA) is provided with the oracle answer, which is the output with the highest BLEU among the top five outputs of FA-model.
We study the impact of different components in MultiFactor and show the results with MixQG- base in Table and more details in Appendix C. Here, "Fine-tuned" indicates the MixQG-base model, which is finetuned for our QG tasks. For Cls+Gen, the phrase selection task and the genera- tion task share the encoder and jointly trained like in PET. The phrase information, however, is not integrated into the decoder for generation, just to enhance the encoder. One-hot PET-Q indicates that instead of using the soft labels (probabilities of a phrase to be selected), we use the predicted hard labels (0 or 1) to inject into PET. And finally,
Table reveals several observations on HotpotQA

---

(supp. facts) with MultilFator (T5-base). Firstly, the high effectiivnness of Q-model (with Gold-FA) indicates that the difficulty of QG largely lies in the full answer generation. Nevertheless, we can still improve Q-model further, by, e.g., predicting the question type based on the grammatical role of the short answer in FA-model outputs. Secondly, Q-model (w/o Context) outperforms PET-Q but not MultiFactor. This might be because context pro- vides useful information to mitigate the error prop- agation from FA-model. Finally, the superior of Q-model (with Oracle-FA) over MultiFactor shows that the greedy output of FA-model is suboptimal, and thus being able to evaluate the top FA-model outputs can help improve overall effectiveness.
Model
T5
PET-Q
Multi
Ocl-FA
Correct
83.5
86.0
87.5
89.5
Hop Error
11.5
9.5
9.0
7.5
Semantic Error
5.0
4.5
3.5
3.0
Error (Bridge)
13.5
11.0
9.0
7.0
Error (Comparison)
3.0
3.5
3.5
Table 5: Human evaluation results on HotpoiQA (supp. facts), where Multi and OcI-FA indicates MultiFactor (T5-base) and its variant where Q-model is given the oracle full answer (w/ Oracle-FA), The last two lines show the error rates where questions are of bridge or comparison types.
one can see significant reduction of semantic error in MultiFactor compared to PET-Q. This is because the model better understands how a short answer is positioned in a full answer context, as such we can reduce the disclosure of (short) answers or the wrong choice question types. However, there is still room for improvement as MultiFactor (w/ Oracle-FA) is still much better than the one with the greedy full answer from FA FA-model (referred to as Multi in Table 5). Particularly, there should be significant reduction in hop error if one can choose better outputs from FA-model.
4.5 Human Evaluation
Automatic evaluation with respect to one gold ques- tion cannot account for multiple valid variations that can be generated from the same input contex- Vanswer. As a result, three people were recruited to evaluate four models (T5-base, PET-Q, MultiFac- tor and its variant with Oracle-FA) on 200 random test samples from HotpotQA (supp. facts). Note that the evaluators independently judged whether each generated question is correct or erroneous. In addition, they were not aware of the identity of the models in advance. In the case of an error, evalua- tors are requested to choose between two types of errors: hop errors and semantic errors. Hop errors refer to questions that miss key information needed to reason the answer, while semantic errors indicate questions that disclose answers or is nonsensical. Additionally, we analyse the ratio of errors in two types of questions on HotpotQA: bridge, which requires multiple hops of information across doc- uments, and comparison, which often starts with "which one" or the answer is of yes/no type. Hu- man evaluation results are shown in Table 5, and we also present some examples in Appendix E.
Error Analysis on Question Types It observ- able that multi-level content planning plays im- portant roles in reducing errors associated with "bridge" type questions, which is intuitive given the nature of this type. However, we do not observe any significant improvement with comparison type. Further examination reveals two possible reasons: 1) the number of this type of questions is compara- bly limit; 2) QA2D performs poorly in reconstruct- ing the full answers for this type. Further studies are expected to mitigate these issues.
4.6 Comparison with LLM-based QG
As Large Language Model (LLM) performs out- standingly in various text generation tasks, we evaluate the performance of GPT-3.5 zero-shot (Brown et al., 2020) and LoRA fine-tuned Llama2- 7B (Hu al., 2022; Touvron al., 2023) on Hot- potQA (full document). Implementation details regarding instructions and LoRA paraeeearsmeters are provided in Appendix B.
MultiFactor vs Others Comparing MultiFactor to other models (T5, PET-Q) in Table 5, we ob- serve an increase in the number of correct ques- tions, showing that -ulti-level content planning is effective. The improvement of MultiFactor over PET-Q is more noticeable in contrast with that in Table 4 with automatic metrics. This partially vali- dates the role of full answers even with short con- texts. In such instances, full answers can be seen as an answer-an anseer-aware paraphrase of the context that is more convenient for downstream QG. In addition,
Automatic Evaluation The performance of Llama2-7B 7B and GPT-3.5-Turbo (zero-shot) in com- parison with MultFactor, T5-base (finetuned) and
"Via the Azure OpenAl Service.

---

Model
B-4
MTR
R-L
BSc
Model
Win
Tie
Lose
MultiFactor
Llama2- 7----BB-----A
W: T5-base
26.66
29.66
43.37
52.76
u.s. T5-base
20
60
20
w: MixQG-base
29.12
30.01
45.20
54.49
R.S. MultiFactor (T5-base)
13
65
22
T5-base
20.70
22.57
40.25
44.06
GPT-3.5-Turbo w. zero-shot
MixQG-base
22.13
23.78
41.21
48.76
v.s. MultiFactor (TS-base)
20
29
51
Hlama--B--LRRM,
16.53
21.35
33.03
37.44
GPT-3.5-Turoo
Table 7: Human evaluation on GPT-3.5 zero-shot and
W. zero-shot
8.78
14.84
22.48
28.38
LoRA fine-tuned Llama2-7B in comparison with Multi- Factor (T5-base).
Table 6: The automatic scores of GPT-3.5 zero-shot, LoRA fine-tuned Llama2-7B HotpotQA full docu- ment setting.
in-depth analysis also reveals a common issue with GPT-3.5-Turbo (zero-shot): its output questions of- ten reveal the given answers. Therefore, multi-level content planning in instruction or demonstration for GPT-3.5-Turbo could be used address this issue LLM-based QG, potentially resulting better performance.
MixQG-base (finetuned) are given in Table 6, where several observations can be made. Firstly, MultiFactor outperforms other methods auto- matic scores by large margin. Secondly, finetun- ing results in better automatic scores comparing to zero-shot in-context lcarning with GPT-3.5-Tuie Finally, Llama2-7B-LoRA is inferior to methods that are based on finetuning moderate models (TS- base/MixQG-baae) across all of these metrics.
Conclusion and Future Works
This paper presents MultiFaator, novel QG method with multi-level content planning. Specif- ically, MultiFactor consists of F'Amoddel, which simultaneously select important phrases and gen- erate an answer-aware summary (a full answer). and Q-model, which takes the generated full an- swer into account for question generation. Both FA-model and Q-model are formalized as our sim- ple yet effective PET. Experiments on HotpotQA and SQuAD 1.1 demonstrate the effectiveness of method.
Human Evaluation As LLM tend to use wider variety of words, automatic scores based on one gold question do not precisely reflect the quality of these models. As a result, we conducted human evaluation and showed the results Table 7. Since OpenAl service may regard some prompts in- valid (i.e. non-safe for work), the evaluation was conducted on 100 valid samples from the sample pool that we considered in Section 4.5. The human annotators were asked to compare a pair of meth- ods on two dimensions, the factual consistency and complexity. The first dimension is to ensure that the generated questions are correct, and the second dimension is to prioritize complicated questions as it is the objective of multi-hop QG.
Our in-depth analysis shows that there is a lot of room for improvement following this line of work. On one hand, can improve the full answer gen- eration model. On the other hand, we enhance the Q-model in MultiFactor either by exploiting multiple generated full answers or reducing the error propagation.
Human evaluation results from Table show that human annotators prefer MultiFactor (T5-base) to Llama2-7B-LoRA and GPT-3.5-Turbo (zero-shot). Additionally, Llama2-7b-LoRA, outperforms GPT- 3.5-Turbo (zero-shot), which is consistent with the automatic evaluation results in Table 6. Interest- ingly, although T5-base (finetuning) outperforms Llama2-7B-LORA in Table 6, human evaluation shows that these methods perform comparably, The low automatic scores for Alama2-7B-Loof are due to its tendency to rephrase outputs instead copying the original context. Last but not least,
Limitations
Our work may have some limitations. First, the experiments are only English corpus. The ef- fectiveness of MultiFactor is verified on the datasets of other languages. Second, the context length in sentence-level QG task is not very long as shown Table 8. For particularly long contexts 500 or 1000), it needs more explorations.

---

7 Ethics Statement
stractive summarization. In Advances in Neural Infor- mation Processing Systems, volume 35, pages 24516- 24528. Curran Associates, Inc.
MultilFactor aims to improve the performance of the answer-aware QG task, especially the complex QG. During our research, we did not collect any other datasets, instead conduct our experiments and construct the corresponding full answer on these previously works. Our generation is completely within the scope of the datasets. Even the result is incorrect, it is still controllable and harmless, no potential risk. The model is currently English lan- guage only, whose practical applications is limited in the real world.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020. Reinforcement learning based graph-to-sequence model for natural question generation. In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia.
Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019, Mixture content selection for diverse sequence generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3121-3131. Hong Kong, China. Association for Computational Lin- guistics.
8 Acknowledgements
We would like to thank the anonymous review- ers for their insightful comments. We also like to thank Dr. Jingyang Li for their helpful suggestions. This work was supported by Alibaba Innovative Research project "Document Grounded Dialogue System".
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. CoRR, (bs//809.00922
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171 1476, Minneapolis, Minnesota. Association for Computational Linguistics.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2067-2073. Association for Computational Lin- guistics.
An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summariza- tion. pages 65-72. Ann Arbor, Michigan. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2018 Harvesting paragraph-level question-answer pairs from Wikipedia. In Proceedings the 56th Annual Meeting the Associ- ation for Computational Linguistics, pages 1907-1917, Melbourne, Australia. Association for Computational Linguistics.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing ask: Neural question generation for reading com- prehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1352. Association for Computational Linguistics.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answering. In Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pages 866 874, Copenhagen, Denmark. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. Controllable open- ended question generation with new question type ontology. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natu- ral Language Processing, pages 6424-6439, Online. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- erarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics, pages 889-898, Melbourne, Aus- tralia. Association for Computational Linguistics.
Yllias Chali and Sadid A Hasan. 2012. Towards auto- matic topical question generation. In Proceedings of COLING 2012. pages 475-492.
Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang. Wei Wu, and Xuanjing Huang. 2022. CQG: A sim- ple and effective controlled generation framework for
Xiuying Chen, Mingzhe Li, Xin Gao, and Xiangliang Zhang. 2022. Towards improving faithfulness in ab-

---

multi-hop question generation. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics, pages 6896-6906, Dublin, Ireland. Association for Computational Linguisties.
Francisco, CA, USA, May 13-17, 2019, pages 1106- 1118. ACM.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
7ichu Fei, Qi Zhang, and Yaqian Zhou. 2021. Iterative GNN-based decoder for question generation. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2573-2582, On- line and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jack Mostow and Wei Chen. 2009, Generating in- struction automatically for the reading strategy of self- questioning. In International Conference on Artificial Intelligence in Education.
Haomin Fu. Yeqin Zhang, Haiyang Yu, Jian Sun, Fei Huang. Luo Si. Yongbin Li, and Cam Tu Nguyen. 2022. Doc2Bot: Accessing heterogeneous documents via con- versational bots. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1820- 1836, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Lidiya Murakhovs' ka, Chien-Sheng Wu, Philippe La- ban, Tong Niu, Wenhao Liu, and Caiming Xiong. 2022. MixQG: Neural question generation with mixed answer types. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 1486-1497, Seattle, United States. Association for Computational Linguis- tics.
Lukas Galke and Ansgar Scherp. 2022. Bag-of-words vs. graph vs. sequence in text classification: Questioning the necessity of text- graptg and the surprising strength of wide MLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- ties, pages 4038 4051, Dublin, Ireland. Association for Computational Linguistics.
Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez. Dipanjan Das, Michael Collins, and Mirella Lapata, 2022. A well-composed text is half done! com- position sampling for diverse conditional generation. In Proceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguisties, pages 1319-1339, Dublin, Ireland. Association for Computational Linguis- tics.
Qi Gou, Zehua Xia, and Wenzhe Du 2023. Cross- lingual data augmentation for document-groundee dia- log systems in low resource languages. In Proceed- ings of the Third DialDoe Workshop on Document- grounded Dialogue and Conversational Question An- swering, pages 1-7, Toronto, Canada. Association for Computational Linguistics.
Boyuan Pan, Hao Li, Ziyu Yao. Deng Cai, and Huan Sun. 2019. Reinforced dynamic reasoning for conversa- tional question generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 21 4-2124, Florence, Italy. Associa- tion for Computational Linguistics.
Michael Heilman. 201 I. Automatic Factual Question Generation from Text. Ph.D. thesis, Carnegie Mellon University, USA.
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan. 2020. Semantic graphs for generating deep questions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1463-1475, Online, Association for Computational Linguistics.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalua- tion of machine translation. In Proceedings of the 40th Annual Meeting of the Asvociation for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Ky- omin Jung. 2019. Improving neural question generation using answer separation. In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 6602-6609.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Associa- tion for Computational Linguisties.
Yanxiang Ling. Fei Cai, Honghui Chen, and Maarten de Rijke. 2020. Leveraging context for neural ques- tion generation in open-domain dialogue systems. In Proceedings of The Web Conference 2020, WWW '20, page 2486-2492, New York, NY, USA. Association for Computing Machinery.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2383-2392, Austin, Texas. As- sociation for Computational Linguistics.
Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. 2019. Learning to generate questions by learning what not to generate. In The World Wide Web Conference, WWW 2079, San
Devendra Singh Sachan, Lingfei Wu, Mrinmaya Sachan, and William L. Hamilton. 2020a. Stronger transform- ers for neural multi-hop question generation. CoRR, abs/2010. 1374.

---

Devendra Singh Sachan, Lingfei Wu, Mrinmaya Sachan, and William L. Hamilton. 2020b. Stronger transform- ers for neural multi-hop question generation. CoRR, abs/2010.1 11374.
Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Liuyin Wang, Zihan Xu, Zibo Lin, Haitao Zheng, and Ying Shen. 2020a. Answer-driven deep question gener- ation based on reinforcement learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5159-5170, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Linfeng Song. Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. 2018. Leveraging context informa- tion for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 869-574, New Orleans, Louisiana. Association for Computational Linguistics.
Zhen Wang, Siwei Rao, Jie Zhang, Zhen Qin, Guangjian Tian, and Jun Wang. 2020b. Diversify question genera- tion with continuous content selectors and question type modeling, In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 2134-21444 Online. Association for Computational Linguistics.
Dan Su. Peng Xu, and Pascale Fung. 2022. QA4QG: us- ing question answering to constrain multi-hop question generation. In IEEE International Conference Acous- tics, Speech and Signal Processing, pages 8232-8236. IEEE.
Dan Su, Yan Xu, Wenliang Dai, Ziwei Ji, Tiezheng Yu, and Pascale Fung. 2020. Multi-hop question genera- tion with graph convolutional network. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4636-464) Online. Association for Com- putational Linguisties.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pages 38- 45, Online, Association for Computational Linguistics.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 78-88, 88, Melbourne, Australia. Association for Computational Linguistics.
Yuxi Xie, Liangming Pan, Dongzhe Wang, Min-Yen Kan, and Yansong Feng. 2020. Exploring question- specific rewards for generating deep questions. In Pro- ceedings of the 28th International Conference on Com- putational Linguistics, pages 2534-2546, Barcelona, Spain (Online). International Committee on Computa- tional Linguistics.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei Hc, Yanjun Ma, and Shi Wang. 2018. Ansuer-focused and position- aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages for Computational Linguistics.
Association
Zhilin Yang. Peng Qi, Saizheng Zhang. Yoshua Ben- gio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018 HotporQA: A dataset for di- verse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Meth- ods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Lin- guistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- alc, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyn- thia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Is- abel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril. Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xi- aoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kam- badur, Sharan Narang, Aurélien Rodriguez, Robert Sto- jnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288.
Ruqing Zhang, Jiafeng Guo, Lu Chen, Yixing Fan, and Xueqi Cheng. 2021. A review on question generation from natural language text. ACM Trans. Inf. Syst., 40(1).
Shiyue Zhang and Mohit Bansal. 2019. Addressing se- mantic drift in question generation for semi-superrs supervise question answering. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing pages 2495-2509, Hong Kong, China. Association for Computational Linguisties.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi 2020. Bertscore: Eval- uating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020,
Petar Veličković, Guillem Cucurull, Arantxa Casanova,

---

Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
B Implementation Details
Model Details MixQG pre-trained series models are fine-tuned from T5, having the same architec- ture and number of parameters. In addition to basic modules, MultiFactor adds a classifier (2d x 2) and La probability infusion layers (2 x d), where d, La donate the model dimensions and the number of decoder layers. Specifically, when initializing with T5-base (220M, d = 768, La = 12). Multi- Factor only increases the number of parameters by 1536 x 2 12 x 2 768 z 0.02M (-0.01%).
Yeqin Zhang, Haomin Fu, Cheng Fu. Haiyang Yu, Yong- bin Li. and Cam- Tu Nguyen. 2023. Coarse-to- fine knowledge selection for document grounded dialogs. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Yao 7hao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3901-3910, Brussels, Belgium. Association for Computational Lin- guistics.
Training Details Because we train the model with fixed epochs on HotpoiQA and the dev size too small (500), we select the best result on test dataset directly following the previous work (Pan et al., 2020; Su et al., 2022) on HotpotQA. On SQuAD 1.1, we select the result based on the dev set. Max length of HotpotQA full is 512, two oth- ers is 256. Moreover, the learning rate for MixQG- base is lower than that of the normal T5-base, stated in (Murakhovs' ka al., 2022). As result, we have opted to employ learning rates of 5e-5 and 2e-5 for MixQG-base on HotpoiQA and SQuAD 1.1, respectively. while ba-e are lc-4 and 5c-5, All the batchsize is 32, except that HotpotQA-full is 16, where the training epoch is 5 instead of 10. We turn off the sampling, and beam size are and 5 on HotpotQA and SQuAD 1.1, respectively. Others parameters default value in Huggingface trainer and generator configuration files. More parameters and time cost of training and inference are in Table 9.
Qingyu Zhou, Nan Yang. Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou, 2017. Neural question generation from text: A preliminary study. In Natural Language Processing and Chinese Computing, volume 10619 of Lecture Notes Computer Science, pages 662-671. Springer.
A Statistic of Datasets
Here, we list the length of context, question and answer of the HotpotQA and SQuAD 1.1 datasets in Table 8. HotpotQA supporting facts and full document settings share the same output and semi- gold full answers.
Train
Valid
Test
HotporQA (Supporting Facts Sentence)
Context
2/1/12/49.31
107/16/48.67
170/13/50.24
Question
89/4/18.07
80/6/17.58
6./7/16.30
Answer
69/1/2.35
15/1/2.39
30/1/2.62
Phrase
1.86/8.66 66
1.73/8.61
1.57/9.14 14
FA
82579/89947
459/500
6763/7405
Data Format We list the input formats of these experiments mentioned before in Table 10. And we use special tokens: <ans>, <passage>, <fa> to present the answer, context, and full answer start tokens.
HotporQA (Full Document)
Context
2381/29/210.22
690/41/205.15
1371/35/216.68
Question
89/4/18.07
80/6/17.58
43/7/16.30
Answer
69/1/2.35
15/1/2.39
30/1/2.62
Phrase
7.46/36.49
7.35/35.43
6.63/37.27
FA
82579/89947
459/500
6763/7405
Instructions and LoRA hyper-parameters The instruction of
SQuAD 1.I
Context
285/2/26.83
150/2/27.6 .61
150/4/27.47
is shown Figure As for LoRA fine-tuned hyper- parameters, we follow the leama-recipes de-
Question
38/1/10.94
31/1/11.01
28/3/1
Answer
34/1/3.24
27/1/3.43
30/1/3,44 44
fault settings, where r = 28, a = 32.
Phrase
2.34/6.80
3.75/7.06
3.75/7.05
FA
84976/86635
8864/8965
8840/8964
C Ablation Study on T5
Table 8: The statistic of max/min/meen token length from NLTK tokenizer. the number of positive/negative phrases and the number of valid/total full answer(FFA) examples in HotpotQA and SQuAD 1.1 datasets.
Considering T5 is a more general Text2Text Pre- trained Lanuague Model, we also conduct ablation
httpps: //gi thub. ccmm//aachhookreeearr///llaaa cipes

---

f"Given the context and
answer, please help generate multi-hop question. : [[answer)) (nnext))Nnnuesttion.
Type
Input
B-model
(answer) <passage> (context)
Qundell
<ans> (answer) <as> [fa] <passage [context]
Figure 3: The zero-shot instruction shown in python code.
we Context
[answer] <fa> [full answer)
MixQG
[answer] In <fi> [fa] <passage> (context)
TS
<ans> [answer] <passage> [context]
HotporQA
MixQG
(answer) in <passage> (context)
Dataset
SQuAD 1.1
Sup.
Full
Table 10: Input formats in our experiments.
Learning rate
5e-5/1c-4
5e-5 /le-4
2e-5/5c-5
Batch size
16
32
32
Warm-up ratio
0.1
0.1
0.1
Epochs
5
10
15
Training time
20
20
18
Inference time
16
15
15
Beam size
5
Table 9: Details of training and inference. Data in learning rate is MMixQG-baseT3--baae The unit of training inference time min/ (epoch-6PU__um))
Model
B-4
MTR
R-L
BSc
HotporQA (Supporting Facts)
Fine-tuned
24.48
25.59
43.17
50.93
Cls+Gen
25.36
26.33
43.38
51.49
studies on T5-base, and the results are shown in Table 11.
One-hot PET-Q
27.01
28.11
42.91
52.31
PET-Q
27.45
28.28
43.46
52.41
MultiFactor
27.80
28.26
43.80
52.86
D Ablation Study Flan-T5
HotpotQA (Full Document)
We conducted experiments initialized with Flan-T5 base to evaluate the performance instruaction- finetuning model on HotpotQA full document set- ting. Results are shown in Table 12. Instruction is shown in Figure 3. Corss compared with these re- sults in Table and 11, Flan-T5 -5---ee outperforms T5-base significantly but still worse than MixQG- base. MixQG QG-specific pre-trained model and fine-tuned on nine various answer-type QA datasets from the T5-base. These results are line with our expectations.
Fine-tuned
20.70
22.57
40.25
44.06
Cls+Gen
20.81
22.61
40.58
44.24
One-hot PET-Q
25.94
28.75
43.10
51.63
PET-Q
26.35
29.54
43.08
52.33
MultiFactor
26.66
29.66
43.37
52.76
SQuAD 1.1
Fine-tuned Cls+Gen
20.15
24.21
40.33
55.18
20.29
24.27
40.34
55.22
Onc-hot PET-Q
20.31
25.49
40.43
56.06
PET-Q
21.13
25.34
41.03
56.21
MultiFactor
21.24
25.63
41.22
56.55
Table 11: The ablation study for MultiFactor, where the B-4, MTR, R-L and BSc means BLEU-4, METEOR, ROUGE-L and BERTScore, respectively.,.
E Error Examples
We list some error examples shown in Figure 4. In hop error, we show three types of hop errors: wrong hop, missing hop, and fabricating information, re- spectively. In semantic error, we list a declarative generation instead of a question and a nonsensi- cal case in which the output is longer than the in- put. Lastly, we present a comparison type where both the pseudo gold and generated full answer wrong, although almost comparison-type QA has no pseudo gold full answer.
Model
B-4
MTR
R-L
BSc
Fine-tuned
21.69
23.31
40.82
47.68
MultiFactor
28.82
29.14
44.87
53.67
Table 12: The ablation study on Flan-T5-base on Hot- potQA full document setting.

---

Error Examples
Facts:
20205 Accra floods. Mayor of Accra Metropolitan Assembly, Oio Vanderpuje described the flaccing cittccl.
floods. At 25 people have died
flooding directly, petrol station explesion caused by the flooding killd least 200 more people.
ii 2015 Aczra explosion. June 2015, an explosion and occumred at a petroi station in Ghana's capital city Accra, killing over people.
Answer: an explosion and fire occurred at a petrol station
Gold Question: what caused the death over 250 people Accre, Ghana?
Generated Question: What heppened at petrol statlon Ghana's is capital city Accra on June 4, 2015, that killed over 250 people and caused flood the Accra Meteopolian Apsembly, Alfred Oko Venderpeije devenbed the floeding ee cntical?
Error Analysis: hop ertor. A wrong hop, the explosion and fire did not cause a flood.
Facts:
Jacksonville station. It serves the "Silver Meteor" and "Silver Star" trains as well as the Thruwey Motorcoach to Lakeland. il Silver Star (Amirak train). The Silver Star is a 1522 mi parsenger train route in the "Siver Service" brand operated by Amtrak, running from New York City south to Mismi, Floride via the Northeast Corridor to Washington, D.C., then via Richmond, Virginis, Raleigh, North Caroline, Colum bia, South Caroline; Savannsh, Georgia Jacksanville, Florida: Orfando, Florida and Tempu, F Florida.
Answer: 1522
Gold Question: How many miles does the train, which pesses through the Amtrak Jacksonvill station and shares the track with the Silver Meteor, nun? Generated Question: Jacksonville station serves the train that how many miles long?
Error Analysis: Hop error. Missing information or key hop. The Jacksonville station servers leest trains: "Silver Metsor" and "Silver Star", The generated queetion does not directly ask the "Silver Star", which runs 1522 miles.
Facts:
West Caks,, Houston, West Oaks small subdivision Houston, Texas
June 12, 1924) American politician 1993 and the 43rd Vice President the United States from 1981 to 1989.
George H. Rush. George Harbert Walker
Presidiant of United States from 1989
Answer:
Subdivision Generated Question: number preeident of United Statee men who West Houston?
Question: George H.
a resident of
Texas, becamee number President in 1989?
Error Analysis: Hop error. Fabricating the Information Bush awned West Oaks, Houston. Nota, emror caused by the insufficient Input information. The source text does not indicate that was resident West Oaks Subdivision Houston Texas.
Facts:
Value premium, Other experts, such as John C, Bogle, have argued that no value premium exists, claiming that F Famn and French's research period dependent.
il John C, Dogle, He is the founder and refied chief executive of The Vanguard Group.
Answer: The Vanguard Group
Gold Question: In regards to Value pramium an expert argued that no value promium exists, what did this exgert found?
Generated Question: Value premium is a term used to describe the value of a product over other terms, other experts, such es John C, Bogle, have argued that no value pramium exists, claiming that Fama and French's rasearch is period dopandint, Bogia is an American business exacutive, philantrropiat, and fomar
CEO, Bogle is the founder and refired chief execulive of whichhh grgerization?
Error Analysis: Semantic error. The question is even longer than the input, which is considered as nonsensical generation. Because it utllizes the knowledge stored in parameters and igmore the inpul source information. which poesibly leads hellusinatio.
Facts:
The Story of Them Festuring Van Morrison. The Story of Them Feeturing Van Morrison compilation sibum that includes almost every song recorded the Northern frish bend Them during the two-year history of the band when it featured Van Morrison the vocalist for the group. il Them (bend). Them were Northern Irish band formed Beffest in April 1984, most prominently known for the rock standard !'Glorie!" and launching
singer Van Morrioon's musical career.
Answer: Them
Gold Question: The Story of Them Featuring Van Merrisan compliation album recorded by what Northem lrish band formed Bellast in April 1984?
Generated Question: The Story of Them Featuring Van Morrison compiaation album that includes almest every song recorded them Norther Irish band,
prominently known for the garage rock standard "Glaria" and launching singer Van Montisn's musical career?
Error Analysis: Samantic emor. fact, generation declarativa sentance. And the The Story Tham Featuring Van Monrisn not launch singer
Morrison's career, which an enror hop. Facts:
These Times. In These Times is an American poliiiclly progresavvidemmcaet socialist monthly magazine news and opini on published in Chicago. Illnois. Mutiintional Montor. The Mutinational Monitor was bimonthly magazine founded by Ralph in 1980.
Answer: Multinetional Moritor
Gold Question: Which magazina has more issues each month. These Times or Muttintionna Moniter?
Peeudo-gold Full Answer: Mullinetional Moritor has more issues esch month, These Times or Multinational Monitor.
Generated Question: Which magazine was founded first, These Times or Mutiiational Monitor?
Generated Full Answer: Mullinational Moniler wes foundeed firet, In These Times or Muttineionnl Monitor.
Error Analysis: Hop error. Both input facts indicate the magazine publication frequency attrbutt but no established time. Only a few comparison type QA pair was consiructed a full unswer successfully. This example is an exception, providing wrong serrencce level plamning and generating a lousy question.
We show six representative error examples, which includes three hop error, two semantic error and one typical comparison error cases.
Figure 4:


Reviews:
[{"review_id": "WUCpwtzp0M", "reviewer": "Reviewer_MLhS", "summary": "", "questions": "1. How to determine which phrases from context are important to the answer or target question in both cases: while generating supporting sentence as well as target question?\n2. From the experimental results, using the generate full answer sentence seems marginal impact on performance across all metrics and datasets. If context is too enough, generating another sentence might be overkilling. It would be better to see the performance of model (Du and Cardie (2017) which use the important sentences extraction instead generating sentence in case of long context. The missing result of this baseline make the impact of answer sentence generation questionable. \n3. Instruction-tuned model like FlanT5-x model shows better performance in QA or QG tasks compared to T5 models. They shows more capability over reasoning. It would be  better to see the performance of instruction-tuned model as baseline in question generation.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We provide our response to your questions and concerns as follows.\n\n**Q1: How to determine which phrases from context are important to the answer or target question in both cases: while generating supporting sentence as well as target question?**\n\nAs for the phrase-level planning, we follow the previous works’ setting, such as  DP-Graph [1], IGND [3], JointGen [4], and CQG [5], to first select a pool of candidate phrases from context using off-the-self NER and phrase selection tools. We then train PET models to simultaneously select phrases and generate full answers or questions. During training, if a phrase appears in the gold question by the crowdsource, we consider it as a positive phrase. Please refer to line 338 - 346 for more information, we will make it clearer in the later version.\n\n[1] Pan, Liangming, et al. Semantic Graphs for Generating Deep Questions. ACL, 2020.\n\n[2] Fei, Zichu, et al. Iterative GNN-Based Decoder for Question Generation. EMNLP, 2021.\n\n[3] Cao, Shuyang, et al. Controllable Open-Ended Question Generation with A New Question Type Ontology. ACL, 2021.\n\n[4] Fei, Zichu, et al. CQG: A Simple and Effective Controlled Generation Framework for Multi-Hop Question Generation. ACL, 2022.\n\n\n**Concern1: The significance of generating full sentence doesn't much reflect in experimental results as Table 9 as well as Table 3. It is not clear that how the way of preparing ground truth answer sentence effectively serve the main purpose of summarizing the answer-relevant information.**\n\nAnswer relevant summary helps remove unrelevant information from context and facilitates question generation, particularly from long context. In our model, full answer generation attempts to generate an abstractive summary relevant to the given (short) answer. The role of full answer is supported by the significant improvement of MultiFactor over PET-Q on HotpotQA (full document setting) in Table 9. We admit the marginal improvement of automatic metrics in Table 3 and on SQuAD dataset in Table 9. However, it does not mean generating full answers does not help. It could just mean that the automatic evaluation metrics are limited. From human evaluation in Table 5, we can see that MultiFactor is significantly better than Pet-Q, particularly for bridge questions that require multi-hop information. Additionally, better performance is expected for better full answers, as shown in the gap between MultiFactor and Oracle-FA in Table 5. We will make our discussion clear.\n\n**Concern2: The absence of QG model with content selection in extractive way in comparison doesn't make paper's position strong. It would be better to see the performance of model (Du and Cardie, 2017) which use the important sentences extraction instead generating sentence in case of long context. The missing result of this baseline make the impact of answer sentence generation questionable.**\n\nIt would be unfair to compare with Du and Carie [1] since this work relies on LSTM as base models. However, in HotpotQA, the supporting facts are the answer-relevant sentences marked by crowdsourcing. As a result, we could treat systems on HotpotQA [2] (support facts) as those that make use of (gold) extractive summary (like Du and Carie) instead of full answer generation like ours. We can cross-reference the performance in Table 2 to see the effectiveness of full answer generation. Specifically, state-of-the-art systems on HotpotQA (support facts) such as CQG[3], MixQG-base[4], QA4QG[5], and PET-Q do not work better than MultiFactor on Hotpot QA (full documents). For example, QA4QG-large on HotpotQA (support facts) obtains Bleu-1 of 49.55, much lower than Multifactor (MixQG-base) with 54.84 of Blue-1. Note that we can conduct such a comparison because the test set is the same, and only the input context is different in the two settings of HotpotQA. We have briefly discussed this from Line 414 to Line 424, but we will clarify it in the final version. \n\n[1] Du, Xinya, et al. Learning to Ask: Neural Question Generation for Reading Comprehension. ACL, 2017.\n\n[2] Yang, Zhilin, et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.\n\n[3] Fei, Zichu, et al. Iterative GNN-Based Decoder for Question Generation. EMNLP, 2021.\n\n[4] Murakhovs'ka, Lidiya, et al. MixQG: Neural Question Generation with Mixed Answer Types. NAACL, 2022.\n\n[5] Su, Dan, et al. QA4QG: Using Question Answering to Constrain Multi-Hop Question Generation. ICASSP, 2022.\n\n\n**Q3: Instruction-tuned model like FlanT5-x model shows better performance in QA or QG tasks compared to T5 models. They shows more capability over reasoning. It would be better to see the performance of instruction-tuned model as baseline in question generation.**\n\nThank you for your suggestions. We have conducted our experiments with two base LM, T5, and MixQG which is finetuned from T5-x for better QG. The experiments aim to verify the effectiveness of our multi-level content planning. As such, adding another base model like Flan-T5 is nice-to-have but may not bring more insights from the perspective of our study. In addition, previous baselines such as QA4QG also do not rely on instruction finetuning models. Hence, doing so may not be fair.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using the generative property of LLM to summarize the target-relevant information then use it to generate target question is one of the strongest points of this paper. \n2. Secondly, using the phrase-probabilities to augment key values in transformer attention layers is a notable way to help decoder make use of factual information.", "reasons_to_reject": "1. The significance of generating full sentence doesn't much reflect in experimental results as Table 9 as well as Table 3. It is not clear that how the way of preparing ground truth answer sentence effectively serve the main purpose of summarizing the answer-relevant information. This may question one of major contributions in this paper. In addition, the absence of QG model with content selection in extractive way in comparison doesn't make paper's position strong. \n2. Few concerns are listed in the questions for the authors section.", "questions_to_authors": "1. How to determine which phrases from context are important to the answer or target question in both cases: while generating supporting sentence as well as target question?\n2. From the experimental results, using the generate full answer sentence seems marginal impact on performance across all metrics and datasets. If context is too enough, generating another sentence might be overkilling. It would be better to see the performance of model (Du and Cardie (2017) which use the important sentences extraction instead generating sentence in case of long context. The missing result of this baseline make the impact of answer sentence generation questionable. \n3. Instruction-tuned model like FlanT5-x model shows better performance in QA or QG tasks compared to T5 models. They shows more capability over reasoning. It would be  better to see the performance of instruction-tuned model as baseline in question generation.", "paper_topic_and_main_contributions": "This paper focuses on question generation task. More specifically, it generate questions from long context in 2-stage model. In first stage, it  learns jointly to select keyphrases and generate a supporting sentence from given the context and answer, followed by the second stage of generating target question by using  generated supporting sentence along with the given context as well as answer. The idea behind generating supporting sentence instead of extracting relevant sentence is that generating sentence can summarize relevant context information well.  In both generation steps, a new phrase-enhanced attention mechanism has been introduced here.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BRGmZZo18T", "reviewer": "Reviewer_kFf7", "summary": "", "questions": "1. For your HotpotQA (full) setting, you used the distractor setting or fullwiki setting, if fullwiki setting, how did you retrieve the full document context?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive suggestions. We provide our response to your questions and concerns as follows.\n\n**Concern1: The performance of NER will influence the final results a lot.**\n\nThe performance of named entity recognition or phrase recognition has been improved significantly in recent years. In fact, previous methods such as DP-Graph [1], MulQG [2], IGND [3], JointGen [4], and CQG [5], also used named entity from spacy or other toolkits as the preprocessing stage for their entity graph construction. The current challenge still lies in handling disjointed facts in a long context. Our solution of an answer-aware summary (full answers) could help mitigate this challenge.\n\n\n**Concern2: This method, MultiFactor, seems only focus on generating questions whose answer is short phrase or one entity so that it can not generate some complex questions, such non-factoid questions.**\n\nThe focus on the task of factoid question generation allows for deeper analysis. Note that our research is in line with many previous research on QG, such as DP-Graph [1], MulQG [2], IGND [3], JointGen [4], and CQG [5], etc. Notably, CQG only focuses on HotpotQA supporting facts setting, whereas we conduct experiments on two settings of HotpotQA and SQuAD for deeper analysis. However, we do agree that the summary-to-QG is an interesting approach to non-factoid question generation, which we intend to explore in the near future.\n\n\n**Concern3: This paper lack of comparing their MultiFactor with some strong baselines, such as zero-shot LLMs.**\n\nOur main idea is to conduct multi-level content planning for QG. In our paper, this idea is realized with medium-sized language models (T5 and MixQG) as base models for full-answer generation and question generation. However, our idea could also be realized with LLM, where both the F-model and Q-model are implemented with LLM. Alternatively, the F-model can be a medium size model where the Q-model is implemented with LLM. Such considerations are nice but too big for a conference paper, given that our analysis has already exceeded the page limit.\n\nNevertheless, we take your advice and compare MultiFactor with zero-shot GPT-3.5 Turbo. Specifically, we randomly select 1000 samples from the split of Fei [5, 6] test set, in the full-document setting to evaluate LLM’s zero-shot performance on automatic metrics. The results are shown in the following:\n\n|            **Model**           |  **BLEU4**  | **ROUGE-L** |  **METEOR** | **BERTScore** |\n|:------------------------------:|:-----------:|:-----------:|:-----------:|:-------------:|\n|          GPT-3.5-Turbo         |     8.71    |    22.09    |    14.48    |     24.70     |\n|    MultiFactor (MixQG-base)    |    29.12    |     45.20   |    30.01    |     54.49     |\n\nAs we can see, MultiFactor outperforms GPT-3.5-Turbo significantly in automatic scores. We further conduct human evaluation on 100 samples by asking humans to pick the better one from the questions generated by MultiFactor and GPT-3.5-Turbo (factual consistency and complexity). If both MultiFactor and GPT-3.5-Turbo outputs contains errors (semantic errors or hop error), we consider those results are Tie. The results are shown in the following:\n\n|           **Preferred**           | **GPT-3.5-Turbo win** |  **Tie**  | **MultiFactor win** |\n|:---------------------------------:|:---------------------:|:---------:|:-------------------:|\n|             100 samples           |           20          |     29    |          51         |\n|     Samples w.o. yes/no answer    |           18          |     23    |          51         |\n\nWhen the answer is yes/no, it is difficult to generate a comparison question without extra instructions or demos. After removing these (8 samples) and evaluate again, we still found our MultiFactor is better. \n\n\n**Q1: For your HotpotQA (full) setting, you used the distractor setting or fullwiki setting, if fullwiki setting, how did you retrieve the full document context?**\n\nWe follow the MulQG [2], GATENLL+CT [7], and QA4QG [8] experimental settings. It is distractor setting, which provides the ground-truth paragraph directly instead of retrieve from the documents pooling.\n\n[1] Pan, Liangming, et al. Semantic Graphs for Generating Deep Questions. ACL, 2020.\n\n[2] Su, Dan, et al. Multi-Hop Question Generation with Graph Convolutional Network. EMNLP, 2020.\n\n[3] Fei, Zichu, et al. Iterative GNN-Based Decoder for Question Generation. EMNLP, 2021.\n\n[4] Cao, Shuyang, et al. Controllable Open-Ended Question Generation with A New Question Type Ontology. ACL, 2021.\n\n[5] Fei, Zichu, et al. CQG: A Simple and Effective Controlled Generation Framework for Multi-Hop Question Generation. ACL, 2022.\n\n[6] Yang, Zhilin, et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.\n\n[7] Sachan, Devendra Singh, et al. Stronger Transformers for Neural Multi-Hop Question Generation. arXiv, 2020. \n\n[8] Su, Dan, et al. QA4QG: Using Question Answering to Constrain Multi-Hop Question Generation. ICASSP, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper propose to use Phrase-Enhanced Transformer (PET) to realize the MultiFactor, in which the FA-model and Q-model share the same transformer encoder  enabling better representation learning for both phrase selection and generation tasks.\n2. Comparing with previous work selecting one sentence from the raw context to reduce the context length, this MultiFactor propose to generate a long answer sentence based on the whole context and recognized named entity or phrase.\n3. The experiments results show that the MultiFactor can outperform than many previous baselines on two popular QG datasets.", "reasons_to_reject": "1. The MultiFactor rely on the named entities or phrases from the context which are recognized in advance by using SpaCy. So I have two concerns here: 1. the performance of NER will influence the final results a lot. 2. This method, MultiFactor, seems only focus on generating questions whose answer is short phrase or one entity so that it can not generate some complex questions, such non-factoid questions.\n2. This paper lack of comparing their MultiFactor with some strong baselines, such as zero-shot LLMs. Maybe the authors don't need to compare with ChatGPT, but I think GPT2 or GPT3 are needed.\n", "questions_to_authors": "1. For your HotpotQA (full) setting, you used the distractor setting or fullwiki setting, if fullwiki setting, how did you retrieve the full document context?", "paper_topic_and_main_contributions": "This paper proposed a novel QG framework, MultiFactor, based on multi-level content planning for question generation. The MultiFactor includes two components: FA-model, which take the context and short answer as input to select key phrases and generates full answers simultaneously. Then, the second part Q-model which takes the generated full answer as an additional input to generate questions.  Experimental results show that their method outperforms previous baselines on two popular QG datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pZ5SGMfte1", "reviewer": "Reviewer_uHqT", "summary": "", "questions": "See Reasons To Reject", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: how to obtain the \"full answer\" during training?**\n\nGiven the answer and gold question, we apply QA2D [1] to convert question and answer pairs to obtain pseudo (gold) full answers for training. We have mentioned this in Line 371 - 373.  We will revise the paper to make that clearer.\n\n[1] Demszky, Dorottya, et al. Transforming Question Answering Datasets Into Natural Language Inference Datasets. Stanford University. arXiv, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper is easy to understand\n(2) Regarding full answer as an answer-aware summary of the context is a very clever approach.\n(3) The experiment results seems strong\n  \n\n", "reasons_to_reject": "I'm not sure how to obtain the \"full answer\" during training.", "questions_to_authors": "See Reasons To Reject", "paper_topic_and_main_contributions": "This paper proposes to conduct automatic content planning at both the phrase and the sentence levels for generating questions.\n\nAt the fine-grained level, answer-aware phrases are selected as the focus for downstream QG. At the coarse-grained level, a full answer generation is trained to connect such (disjointed) phrases and form a complete sentence.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "cK7Ioc3bYh", "reviewer": "Reviewer_SaFE", "summary": "", "questions": "Please refer to \"paper weakness\", particularly W1 and W3.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful feedback. We provide the response to your concerns as follows.\n\n**Concern1: LLMs are very good at dealing with problems like QG. The experimental results are not convincing enough without comparing with them.**\n\nOur main idea is to conduct multi-level content planning for QG. In our paper, this idea is realized with medium-sized language models (T5 and MixQG) as base models for full-answer generation and question generation. However, our idea could also be realized with LLM, where both the F-model and Q-model are implemented with LLM. Alternatively, the F-model can be a medium size model where the Q-model is implemented with LLM. Such considerations are nice but too big for a conference paper, given that our analysis has already exceeded the page limit.\n\nNevertheless, we take your advice and compare MultiFactor with zero-shot GPT-3.5 Turbo. Specifically, we randomly select 1000 samples from the split of Fei [1, 2] test set, in the full-document setting to evaluate LLM’s zero-shot performance on automatic metrics. The results are shown in the following:\n\n|            **Model**           |  **BLEU4**  | **ROUGE-L** |  **METEOR** | **BERTScore** |\n|:------------------------------:|:-----------:|:-----------:|:-----------:|:-------------:|\n|          GPT-3.5-Turbo         |     8.71    |    22.09    |    14.48    |     24.70     |\n|    MultiFactor (MixQG-base)    |    29.12    |     45.20   |    30.01    |     54.49     |\n\nAs we can see, MultiFactor outperforms GPT-3.5-Turbo significantly in automatic scores. We further conduct human evaluation on 100 samples by asking humans to pick the better one from the questions generated by MultiFactor and GPT-3.5-Turbo (factual consistency and complexity). If both MultiFactor and GPT-3.5-Turbo outputs contains errors (semantic errors or hop error), we consider those results are Tie. The results are shown in the following:\n\n|           **Preferred**           | **GPT-3.5-Turbo win** |  **Tie**  | **MultiFactor win** |\n|:---------------------------------:|:---------------------:|:---------:|:-------------------:|\n|             100 samples           |           20          |     29    |          51         |\n|     Samples w.o. yes/no answer    |           18          |     23    |          51         |\n\nWhen the answer is yes/no, it is difficult to generate a comparison question without extra instructions or demos. After removing these (8 samples) and evaluate again, we still found our MultiFactor is better. \n\n[1] Fei, Zichu, et al. CQG: A Simple and Effective Controlled Generation Framework for Multi-Hop Question Generation. ACL, 2022.\n\n[2] Yang, Zhilin, et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP, 2018.\n\n**Concern2: Missing the results of QA4QG-large on SQuAD 1.1.**\n\nIn the original paper, QA4QG has not been tested on SQuAD1.1. Furthermore, since the source code is also not accessible, we cannot conduct experiments by ourself. As such, we cannot report results on SQuAD 1.1.\n\n\n**Concern3: What is the intuition of full answer?**\n\nIntuitively, a full answer can be regarded as an answer-aware summary of the context, from which complex questions are more conveniently generated. In other words, generating full answers helps remove noise or irrelevant information from long context, thus benefiting QG. Empirically, removing irrelevant from the context is helpful. This is supported by the fact that baselines on HotpotQA (full documents) performs worse than on HotpotQA (support facts) in Table 9. We have mentioned this in lines 075-085, lines 414-418 and demonstrated with Figure 1. We will improve our writing to make this clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. This paper is the first study that introduces the concept of full answers in an attempt at multi-level content planning for question generation (QG).\nS2. The approach is well explained and authors well guide the reader through both the methods as well as experiments. Overall, this paper is easy to follow. Both the introduction and organization are great. Besides, the title, abstract, and keywords are consistent with the content of the paper.", "reasons_to_reject": "W1. Authors state that \"we do not evaluate the performance of LLM (e.g., ChatGPT or GPT-4) due to the limitation of accessing OpenAI models from our country\". However, I think LLMs are very good at dealing with problems like QG. Therefore, the experimental results are not convincing enough without comparing with them.\nW2. Missing the results of QA4QG-large on SQuAD 1.1.\nW3. I don't really understand why generating a full answer is helpful. It would be better to provide detailed intuition or explanation.\n", "questions_to_authors": "Please refer to \"paper weakness\", particularly W1 and W3.", "paper_topic_and_main_contributions": "This paper studies the problem of generating questions from a given context and answer. To this end, the paper proposes MultiFactor, including a FA-model and a Q-model. Here, full answer generation is introduced to connect the short answer with the selected key phrases. Experimental results show that the method outperforms strong baselines on two popular dataset. Overall, this paper is easy to follow.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a multi-level content planning approach for generating complex questions, specifically at both the phrase and sentence levels. Experiments, including ablation studies, conducted on standard datasets, i.e., HotpotQA and SQuAD, show the effectiveness of the proposed solution. The paper is well-structured and easy to follow, and its findings may be of interest to the QA community.
Decision: Accept-Findings