PROTEGE: Prompt-based Diverse Question Generation from Web Articles
Authors: Vinayak S Puranik
Anirban Majumder
Vineet Chaoji
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Question Generation
Question Answering
Diversity
Fidelity

Abstract: Rich and diverse knowledge bases (KB) are foundational building blocks for online knowledge sharing communities such as StackOverflow and Quora, and applications such as conversational assistants (aka chatbots). A popular format for knowledge bases is question-answer pairs (or FAQs), where questions are designed to accurately match a multitude of queries. In this paper, we address the problem of automatic creation of such Q\&A-based knowledge bases from domain-specific, long-form textual content (e.g., web articles). Specifically, we consider the problem of question generation, which is the task of generating questions given a paragraph of text as input, with a goal to achieve both diversity and fidelity of the generated questions. Towards this goal we propose PROTEGE, a diverse question generation framework which consists of (1) a novel encoder-decoder based Large Language Model (LLM) architecture which can take a variety of prompts and generate a diverse set of candidate questions, and (2) a hill-climbing algorithm that maximizes a sub-modular objective function to balance diversity with fidelity. Through our experiments on three popular public Q\&A datasets, we demonstrate that PROTEGE improves diversity by +16% and fidelity by +8% over diverse beam search and prompt-based baselines.

PROTEGE: Prompt-based Diverse Question Generation from Web Articles
Vinayak S Puranik
Anirban Majumder
Amazon
Amazon
puranikv@amazon.com com
majunda@amazon.ccom com
Vineet Chaoji Amazon vchaoj ieroojieammzo..coom com
Abstract
for online shopping is the difficulty in identifying the right product that suits the requirement. For high-considee nigh-comsideration products such as laptops and smartphones, at times customers lack the human touch that they would otherwise experience in an offline store where trained sales agents can explain features of each product and provide high-level guidance to select the right one. The sales agent can proactively query the customer to understand her requirement, help refine her needs and finally recommend the right products. order to bridge the gap between online and offline shopping ex- perience, multi-turn goal-oriented dialog systems, also known as chatbots offer a promising direc- tion. Chatbots help users to familiarize technical concepts, acquire domain knowledge, get recom- mended products that they likely buy, closely mimicking the offline shopping experience.
Rich diverse knowledge-hases (KB) are foundational building blocks for online knowl- edge sharing communities such as StackOver- flow and Quora, and applications such as con- versational assistants (aka chatbots). A pop- ular format for knowledge bases is question- answer pairs (or FAQs), where questions designed to accurately match a multitude of queries. this paper, we address the prob- lem of automatic creation of such Q&A-based knowledge bases from domain-specific, long- form textual content (e.g.. g.. web articles). Specif- ically, we consider the problem of question gen- eration, which is the task of generating ques- tions given paragraph of text input, with a goal to achieve both diversity and fidelity of the generated questions. Towards this goal we propose PROTEGE, diverse question gener- ation framework which consists of (1) a novel encoder-decoder based Large Language Model (LLM) architecture which can take variety of prompts and generate a diverse set of candidate questions, and (2) a hill-climbing algorithm that maximizes sub-modula objective func- tion to balance diversity with fidelity. Through our experiments on three popular public Q& datasets. we demonstrate that PROTEGE im- proves diversity by + +16% and fidelity by +8% over diverse beam search and prompt-based baselines.
Towards curating large scale knowledge bank, Large Language Models (LLMs) (Devlin al., 2019; Raffel et al., 2020; Brown et al., 2020) have shown remarkable success learning in-depth knowledge from data. They do so without access to any external memory as the knowledge is imbibed in the model parameters. While this is fascinating, on the downside, the model may hallucinate (Mar- cus, 2020) and generate answers that factually incorrect. As result, modern chatbot systems em- ploy a Retrieval-Augment
1 Introduction
Generation (RAG) ar- chitecture (Lewis et al., 2020) that has two compo- nents: a) an encoder-decoder network that does the natural language understanding of user queries and language generation and b) a back-end knowledge- base (KB) that indexes relevant bits of information for the task hand. The encoder maps user inputs to a dense representation which is used to query the KB and retrieve evidences. The evidences, as well as the input, are fed to the decoder to generate the final response.
In data rich era, identifying, extracting and gen- erating responses to user's questions has become the next challenge. While search engines provide a simple interface for users to get responses to their queries, getting answers to complex queries still remains challenge (Krishna et al., 2021). As result, specialized knowledge bases that extract and store question-answer pairs have become prevalent.
Many applications rely on a knowledge base of generated guestion-answed pairs, to ensure reliable, accurate and as close to numan-generann informa- tion to their users. For instance, a key frustration
Scalable generation of knowledge-baat bases is funda- mental to the success of the RAG and the underly-

---

ing chatbot application. In this work, we investigate automatic generation of a knowledge-base. Unlike traditional RAG system (Lewis et al., 2020; Izacard et al., 2022) that indexes webpages and large docu- ments, we look for a knowledge-base in the form of question-answer pairs. Not only this helps us improve the accuracy of evidence retrieval, but also allows rich applications to be easily built on top of it e., suggesting related questions, navigation of the KB etc, for improved customer experience. Our focus on educational questions, i.e., questions that help user familiarize with product concepts ("What is the difference between SSD and HDD?") and use-case guidance ("What is the recommended configuration for gaming laptop"").
algorithm (Nemhauser al., 1978) to generate a near optimal set of questions, therefore, increasing yield and quality simultaneously.
We propose PROTEGE (PROmpT-based div question GEneration), diverse question generation framework which consists of two stages (1) novel encoder-decoder based LLM architecture which can take a variety of prompts and generate diverse of candidate questions. and (2) a greedy hill-climbing algorithm that maximizes a sub-modular objective function to balance diversity with fidelity. We demonstrate that PROTEGE improves diversity by +16% and fidelity by +8% while also improving text generation metrics, over strong baselines. Our experiments on three popular public Q&A datasets indicate that PROTEGE consistently outperforms both diverse beam search-based and prompt-based baselines.
Question generation is the task of generating questions given a paragraph of text as input. Ques- tion generation quality can be attributed to two characteristics: a) Fidelity that measures the se- mantic coherence of generated questions and our ability to answer them from the input paragraph, and b) Diversity which measures lexical and se- mantic dissimilarity between generated questions. Many previous works (Rajpurkar et al., 2018, 2016; K Kwiatkowski et al., 2019) have addressed the task of generating questions from text. While it is essen- tial to generate questions that are of high fidelity, for knowledge-baid completion, it is imperative to have a diverse question set. To promote diversity, current question generation models rely on beam search. The resulting set, however contains many structurally similar questions with minor lexical changes that warrant the same answer. There has been prior work (Elhamifar et al., 2012; Song et al., 2018; Vijayakumar et al., 2018) in NLP on diver- sity. In particular, Song et al. (Song al., 2018) addresses diversity via Determinantal Point Pro- cesses (DPP) for neural conversation models, it can be adapted for question generation task.
2 PROTEGE: Prompted Question Generation
Question generation models take a source con- text represented as sequence of sentences (2 -.),..),,,...... pass them through encoder to learn its latent representation and finally through a decoder to generate the output question y/ (91, 1/2 .... with one word y/ at a time. Given training data-set D {(x,, v)}. the model param- eters 0 are learned by maximizing the likelihood function (e,,)) log Pr(y x; The encoder and decoder are implemented as Transformer net- works (Vaswani et al., 2017). The encoder con- sists of No layers where each layer contains self-attention and feed-forward block. The en- coder takes an input through all the encoder blocks to generate an output = ENCODER (x: C BxSF= The th encoder block is a Transformer layer TRANS- (oRMER( !) which takes the input z!" from previous layer and generates the output (("), The encoder blocks are applied in sequence and fi- nally we get the output 7th output word yi+1. we take the previous words yo = y1... and the encoder output h and pass them through N decoder blocks. Each decoder block contains a self-attention, cross-attention and feed- forward layer. The decoder blocks are also applied in sequence and at the final layer it emits the next (Na)
R and passes
While these approaches are helpful in maximiz- ing diversity, they fall short in terms of generat- ing high fidelity output. Naively borrowing these techniques may allow the model to hallucinate and generate questions that are not answerable from the input paragraph. Addressing the task of diverse question generation through the lens of monotone sub-modular function (Bach, 2013) alleviates this problem and provide additional benefits. On one hand, this formulation provides flexibility in con- trolling diversity and fidelity of the output. On the other hand, we can leverage a well-known greedy
(Na), To generate the
word y++1 y.j
'Here H is the minibatch size, $ is the sequence length and F denotes the embedding dimension.

---

the feed-forward network. More specifically, each decoder layer performs the following computation.
Layorhoom
d
LN
(n-1)
+ S
(1-1))
(1)
LayerWorm
Layenhorm
D>
d
LN (1))
CA(ho: 1(1))
(2)
bp:
Cross Atrentinn
Cross Attention
( LN (n)
Ch
d(
(3)
Ensoder
he
MIXUP (d!)) df(") (p)
(4)
Layorhorm
D
v.g
LN (dln)
FF
FF (1
(5)
D
Here LN, SA, CA, FF are abbreviations for lay- ernorm (Ba et al., 2016), cross-attention (Vaswani et al., 2017), self-attention (Vaswaní et al., 2017) and feedforward layers. MIXUP is aggrega- tion layer that combines the context and prompt
(a)
(b)
Figure 1: Model Architecture of PROTEGE encoder and decoder. The left figure (a) shows the modifica- tions (components shown in color) done to the standard decoder architecture. As shown in the figure (b), the En- coder takes the context and prompt as input and generate representations he and hyp The decoder modified incorporate cross-atte cross-attention with A. and hy and mixup layer to aggregate the outputs.
cross-attention outputs d(m), dep to generate da) i.c. d X [d(n) dp d We propose various ways to implement MIXUP: a) treat as a tunable apper-parameter, b) learn À as a free parameter or via attention (Liin et al., 2017) weights. Note that by setting À [1, 0J. we recover the standard de- coder. More details about the architecture choice are described in Appendix A.
2.1 Controlled Generation
For controlled generation of questions, we feed the input document along with various types of prompts to the encoder. This requires some modifi- cation to the standard mecoder-decodde architecture. We use two encoders: one for the document (or, the context) and the other for the prompt signals (Dou al., 2021). Similar Transformer architecture, each encoder has N₂ layers where the first Ne layers use shared parameters O₂ for the context and prompt. The final encoder layers consist of an additional Transformer block for the context and prompt inputs with individual parameters Oc, Op respectively. More specifically, given context I. and prompt Fp. we run the following computation the encoder side,
2.2 Prompt Signals
We use types of prompts: a) keyword-based: define an entity dictionary based on domain knowledge search keywords such as, brands, fea- tures etc. Entities from this dictionary can be used as a prompt, b) sentence-based: we iden- tify informative sentences from the context and use them as prompt input. Further, there are two strategies to compute the prompts: a) HEURIS- TIC: extract prompts from the context based manually defined rules ML models, ORA- CLE: extract prompts from both context and the ground-truth question. Note that the ORACLE strat- egy requires the ground-truth and hence, can be used only during training whereas HEURISTIC can be used during both training and inference. While using ORACLE during training and HEURISTIC in inference leads to train, test mismatch of distribu- tion of prompts, our hypothesis that it will help establish strong correlation between the prompts and the generated questions. In Appendix B we describe the various prompt signals we have used for our experiments.
he TRANSFORMER (ENCODER (Ie; (c), 0.)
П2 TRANSFORMER (ENCODER (Ip, 9e),0p)
Our decoder attends to both the context and prompt signals hr hyp We achieve it by modifying the standard architecture follows. Unlike standard decoder, each decoder layer attends to both context and prompt embeddings from the encoder via cross- attention layers and their combined output is fed to

---

2.3
Balancing Diversity and Fidelity of Questions
shopping guidance queries to the search engine and extract questions, answer snippets and URLs from the search results. We further pre-process the extracted content to form question, answer, con- text triplets. We also leverage three popular bench- mark Q&A datasets namely, (1) SQUAD 2.0 (Ra- jpurkar et al., 2018), (2) NQ Natural Questions dataset (Kwiatkowski et al., 2019), and (3) MS M AARCO (Nguyen al., 2016a). In Appendix C we describe the pre-processing logic used to cre- ate the question, answer, context triplets from raw datasets. Table 7 in Appendix C lists the dataset statistics.
At the end of the first stage of PROTEGE, we have generated diverse set of questions by varying the prompt input to the model. However, in practice, some of these questions may be irrelevant, i.c., they can't 't be answered from the current context. In the second stage, we leverage an algorithm that selects a subset of questions which both relevant and diverse. Let's assume that the previous step has generated N questions Q - {QL, 92 ... .gn} from a document D. The objective of the current step is to select subset Q' S Q of size k, that maximizes <(Q', D)
3.2 Implementation details
77 (Qiversity(Q'++ + (1- (1-1)) relevance Q', D) (6)
Our models are based on the popular T5 (Text-to- Text Transfer Transformer) (Raffel et al., 2020) architecture. T5 models closely follow the encoder- decoder Transformer implementation originally proposed in (Vaswani et al., 2017) with minor modi- fications. For baseline models (section 3.3), we use the vanilla T5ForCondi tional Generation imple- mentation from the HuggingFace Transformer li- brary (Wolf et al., 2019). For our prompt-based controlled generation models we extend the vanilla implementation by including (as described in sec- tion 2.1) (1) an additional encoder for the prompt input which shares parameters with the original encoder, (2) a new cross-attention block in the de- coder which is initialized with pre-trained weights from the original cross-attention block.
Here 77 is a syper-parametet that balances the rel- evance (i.e., fidelity) and diversity. We discuss various choices for implementing the diversity and relevance functions. The relevance of a question set Q is determined via answerability i.c. how likely the question can be answered from the given con- text. The answerability of a question set Q' is calcu- lated as (Q D) = Eneq AB(D,g) .)) where AE is an answerability model build on top of standard LLM encoders such BERT (Devlin et al., 2019). We use n-grams define diversity of question bank. Let 2n (g) denote the set of n-grams in q af- ter removing stop-words. We define diversity as diversity(Q') = Ene(1233 Uyeq' (4)).
Note that the diversity expression promotes unique n-grams across questions and has been used standard metric to measure diversity of text gen- crated by LLMs in prior works, such as (Zhang al., 2018). can be noted that the diversity func- tion is sub-modular (Bach, 2013) which makes the objective function <(Q', D) sub-modular as well. Although maximization of sub-modular function is NP-Hard, it is well-known that the algorithm that greedily picks each item has provably good approximation guarantee (Nemhauser et al., 1978).
Hyper-paramete settings. To make it feasible to train a large number of models, for all our exper- iments we use the t5-small variant with 60MM parameters as the base implementation. We use a learning rate of 5e-5, epsilon of 1e-8 with AdamW optimizer. We use sequence length of 512. We train all models up to 10 epochs with training batch size of 4 and choose the checkpoint with the best performance on the validation set. We train our models on a single GPU of an AWS EC2 instance with a GPU memory of 64GB.
3 Experiments
3.3 Baselines
3.1 Datasets
BASELINE-BEAN Only the context is passed as input (without any additional prompts) and Diverse Beam Search (DBS) (Vijayakumar et al., 2016) is used to generate top-k questions. Diversity pa- rameters num wwwp__eam_gge groups and diivesity_penatee are fine-tuned by optimizing for diversity metries through grid scarch.
A supervised dataset for the question generation task typically consists of question and answer pairs along with "context" input. In order to prove the efficacy of our approach for specific domain of 'shopping guidance', we curate a custom dataset, termed SEARCHQA, by extracting QA pairs from a third-party search engine. We submit customized
BASELINE- WEELINE-PROMP Prompts are concatenated

---

BERT
Dataset
Model
Dist-1
Dist-2
Dist-3 1 Ent-1
Ent-21
Ent-3
Fidelity
Score 1
-RAM
0.5379
0.6963
3.5454
3.7396
3,6789
0.9009
0,8408
SEARCHQA
BASELINE PROMPPT
0.6333
0.7817
0.8489
3,4667
3.6144
3,5300
0.8895
0.8449
PROTEGE
0.7351
0.9197
0.9632
3,9869
4.2143
4,0804
0.8235
0.9085
BASELINE: BEAM
0.5589
0.6930
0.7425
3.7763
3.9316
3,8460
0.8763
0.8106
SQUAD
BASELINI-PROMPT
0.6413
0.8215
0.8792
3.9315
4.1840
4.1150
0,7958
PROTEGE
0.7490
0.9340
0.9680
4.2678
4.5285
4.4187
0.7915
0.8812
BASELINE BEAM
0.5032
0.5968
0.6529
3,5582
3.6160
3.5487
0.8991
0.7015
NQ
BASELINE PROMPT
0.6216
0.7522
0.8244
3.7311
3.8580
3,8131
0.8439
0.7447
PROTUGE
0.7028
0.8522
0.9117
4,0141
4.1824
4.1169
0.8048
0.8151
MASELINE-REAM
0.63.0
0.7170
2,7588
2.69220
2,3905
0.8986
0.7998
MS MARCO
DASRLINE-PROMIE
0.6421
0.7756
0.8672
2.8324
2.7905
2,5382
0.8660
0,7502
PROTEGE 0.7177
0.8693
0.9426
3,4144
3.4548
3,2191
0.8138
0.8278
Table Diversity metrics for PROTEGE and baselines across QA datasets.
with context to form a single input to vanilla T5. exactly the same set
from the context, the cost of missing out on other answerable questions. We compute the answerabil- ity score for each generated question and report the average.
For fair evaluation, of prompts used in the corresponding controlled generation (PROTEGE) model. For instance, sup- pose PROTEGE model uses ground-truth question entities as ORACLE prompts (training) and context entities as HEURISTIC prompts (inference), the ex- act same strategy is used for this baseline as well.
NLG metries. Finally, to evaluate the "close- ness" generated questions with respect to the ground-truth questions we also report standard NLG metrics popular in literature, namely: (1) METEOR (Banerjee and Lavie, 2005) which is measured as harmonic mean of unigram preci- sion and recall, (2) BLEU-4, cumulative gram BLEU (Papineni al., 2002) score, which an evaluation of matching grams of specific order (1- gram, 2-gram etc.) (3) ROUGE-L, a version of ROUGE (Lin and Och, 2004), which measures the longest common subsequence (LCS) between the generated and reference text.
3.4 Metrics
Diversity metries. We evaluate on two popular n-gram based lexical diversity metrics (1) Distinct- n (Li et al., 2016), which measures the percentage of unique n-grams out of total number of n-grams in set of generated questions. We report Dist-1, Dist-2 and Dist-3 metries, (2) Entropy-n (Zhang al., 2018), which measures how evenly the gram distribution for given question set. These two metrics are popularly used in literature to evalu- ate lexical diversity of generated responses (Zhang et al., 2018; Han et al., 2022; Stasaski and Hearst, 2022: Tevet and Berant, 2020). To measure se- mantic diversity we report a BERTScore (Zhang et al., 2020), which is measured as the average BERTScore of each pair of generated questions. BERTScore measures the semantic similarity be- tween a pair of generated sentences, hence lower the average BERTScore better the diversity. Fidelity metrics. To report a fidelity (or "answer- ability")) metric, we train a separate BERT-based model that takes context and a question and out- puts probability score for the question being an- swerable from the context. The ROC AUC of this BERT model was observed to be 0.84. We tune the threshold of this model operate at a preci- sion of 85%, corresponding to a recall of 30%. A higher bar on the precision allows us to select questions which are highly likely to be answerable
4 Results
Diversity results. For the SEARCHQA dataset, among several choices for prompt signals (de- scribed in section 2.2) we highlight the best results obtained in this section and describe the trade-offs among the choices in section 4.1. In all our tables highlight the first best result and underline the second best result. Note that f for metric indi- cates higher values are preferred whereas + indi- cates lower values are preferred. For SEARCHQA dataset, we observe table that across all the met- rics our algorithm (PROTEGE) does significantly better than both baseline. On 1, Dist-2 and Dist-3 metrics, PROTEGE does 16%, 18%, 13%, respectively, better than the second best result. On Ent- Ent-2 and Ent-3 metrics, PROTEGE shows improvement of 12%. 13% and 1 11% respectively, compared the second best result. PROTEGE also reduces BERTScore by 7%, and improves Fidelity by 8%. PROTEGE generates almost double the

---

Top-1
Top.2
Top-3
Dataset
Model
METEOR
BLEU-4
ROUGE-L
METEOR
BLEU-4
ROUGE-L
METEOR
BLEU-4
ROUGE-L
BASELINE: BEAM
0.3088
0.3794
0.3375
0.3598
0.4188
0.3853
0.3968
0.4449
0.4185
SEARCHQA
BASELINE PROMPT
0,2850
0.3539
0.3163
0.3359
0.3951
0.3567
0.4099
0.3798
PROTEGE
0.2767
0.3479
0.3004
0.3638
0.4661
0.3803
0.4034
0.4456
0.4161
RASELINE-BEAN
0.1678
0.2346
0.2083
0,2062
0.2674
0,2458
0.2332
0.2900
0.2721
SQUAD
DASFLINE-PROMPE
0.1487
0.2109
0.1968
0.1990
0.2583
0,2481
0.2256
0.2830
D,2694
PROTEGE
0.1593
0,2289
0.2053
0.2155
0.2826
0.2565
0.2490
0.3085
0.2833
BASULINI-BEAM
0.3998
0.4634
0.4307
0.4346
0,4884
0.4607
0.4570
0.5035
0,4792
NQ
PASELINE-PROMPP
0,3627
0,4333
0.4016
0,4192
0.4725
0,4520
0.4452
0.4888
0.4724
PROTEGE
0,3089
0,3965
0.3391
0,3855
0.4517
0.4055
0.4262
0.4786
0.4433
MASELINE-BEAN
0.4217
0.4712
0.4637
0.4748
0.5105
0.5096
0.5024
0.5298
0.5292
MS
BASELINE-PROMPT
0,3740
0.4228
0.4317
0.4423
0.4799
0.4894
0.4749
0,5048
0.5122
PROTUGE
0,3788
0,4234
0.4120
0.4631
0,4874
0,4821
0,5064
0.5187
0.5171
Table 2: NLG metrics for PROTEGE and baselines across QA datasets.
number of unique questions compared BASE- LINE-PROMPT, Thus, given the same context PRO- TEGE generates a higher number of unique ques- tions which are better both in terms of diversity and fidelity, compared to baselines.
fidelity objectives, and hence need not be closest the reference ground-truie truth. However, as we allow PROTEGE to select more questions (top-2 and top- 3) the model often generates a question closer to the ground-truth, which shows in top-2/top-3 results where PROTEGE does better than both the baselines in matching with the reference. In other words, if we allow top-2 questions, PROTEGE shows the best performance with improvement of 1.1% in METEOR (but, shows second best performance in BLEU-4 and ROUGE-L). Similarly, for top-3 ques- tions the corresponding improvements are +1.7%, +0.2% for METEOR and BLEU-4 scores. We ob- serve similar trends for SQUAD among benchmark datasets.
For the benchmark datasets, we observe in Table that across datasets, PROTEGE improves on all the diversity metrics (Dist-n & Ent-n) when compared both the baselines. For example, on the Dist-1 metric, compared to the second best (which is con- sistently BASELINE-PROMMTT) PROTEGE shows an improvement of 17%, 13% and 12%, respectively, for SQUAD, NQ and MS MARCO. On fidelity, compared to the second best, PROTEGE performs 9%, 9% and 4% better, respectively, for SQUAD, NQ and MS MARCO. Significant reduction is also observed on BERTScore. PROTEGE generates 1 unique questions (on average) more than BASELINE--ROMPT..
For the NQ and MS MARCO datasets, although PROTEGE shows significant improvement over baselines on diversity metrics, improvements are not observed on NLG metrics. We explain our hypothesis for this observation Appendix D. Human evaluation. We performed human evaluation to compare the quality of top-k gener- ated questions between PROTEGE and BASELINE- BEAM. Annotators were asked to label each set of generated questions (for given context) w.r.t., a) Readability (no. of readable & meaningful ques- tions), b) Diversity (no. of semantically unique questions), e) Fidelity (no. of questions answer- able from the context). In figure we observe that PROTEGE improves on BASELINE-BEAN with an absolute improvement of 5% on readability, 32% on diversity, 36% on answerability. Appendix describes the details of the human audits.
NLG results. We present metrics separately for top-1, top-2 and top-3 generated questions. The metries for top-k is computed using the question (among top-k) which results in the maximum ME- TEOR score with reference to the ground- truth question. As described earlier, for BASELINE- BEAM we use beam search generate the top-k questions, while for IASHLIINE- we pick the top-k questions based on generation score. For PROTEGE we select the top-k questions returned by our second-stage algorithm (section 2.3, which greedily selects the question that maximizes diver- sity and fidelity.
For the SEARCHQA dataset, in table 2 ob- serve that for the top-1 question the best metries are obtained from the BASELINF--BAN model. From our model's point of view this is expected as the topmost question is selected based on diversity and
4.1 Ablation studies
Effect of prompt signals. For the SEARCHQA datasct we experiment with a variety of (keyword- based and intence-basedd prompt signals as de-

---

100
Pre Greedy
Post Greedy
BASEL INF-BEAM
5000 PROTECE
Dalase
Didt-1
BERT
Fidelity
Dis--1
BERT
Fidelity
Sowe
SPARmIQQ
0.6158
0.8665
IUNIK
60
SQUAD
0.6011
03161
0,9030
0.7490
0,7915
0.8812
NQ
0,6254
0.414
0,7483
0,7028
0.8048
0.5151
MARCO
0:6422
0,8402
30
20
Table 4: Effect of greedy algorithm on diversity metrics, Pre-Greedy is the output prompt-based controlled generation (section 2.1), while Post-Greedy is the out- put of greedy hill-climbing algorithm (section 2.3).
10
Readao
Diverse
Ancerabbo
Figure Results of human evaluation.
0.84
scribed in 2.2. table 3 we present the effects of prompt signals on diversity metrics.
0.88
Diversity (Dizt-1)
0.80
0.87
Training prompt (ORACLE)
Inference prompt (HEURISTIC)
Dist-1
BERT
Fidelity
0.78
0.86
Score
Baseline
0.5379
0.9009
0.8408
0.76
Diversity Rideiity
Answer text
Context span (size=1)
0.8235
0.9085
0.74
0.84
spait (size=])
0,8405
0.9005
0.83
0.72
Answer text
Context span (size=:)
0.6268
0.8730
0,8817
Keywords from contexi span (sisæ=1)
keywords
0,8419
0,8977
Figure Diversity vs fidelity.
Keywords context spun (siam=2
Answer keywords
0,5735
0.8578
0.8906
Keywords from
Answer keyworks
0.6215
0,8829
0.8684
for both diversity and fidelity. In table we see that an effect of this greedy algorithm, across datasets both diversity and fi- delity metrics show a marked improvement. On an average, post-greedy Dist-I metric improves by 13% and Fidelity improves by 9%. Further, in Ap- pendix H we show the effects of greedy algorithm on all the diversity and NLG metrics.
context span (size=s)
Question
0.7459
0.8234
0.8765
Table 3: Effect of prompt signals diversity metrics.
Across all prompt choices, PROTEGE does better than BASELINE- BEAM on all metrics. Answer text (with a context span of size 1 during inference) per- forms the best on BERTScore and Fidelity metrics and second best Dist-1 metric. Question enti- ties shows the best performance Dist-1 metric, which is due the fact that the model is trained generate a question around the specific entity passed as a prompt. Based on these results, we typically use answer text as a preferred choice for the prompt. Detailed metrics are in Appendix F.
Diversity versus fidelity. Our algorithm to bal- ance diversity and fidelity of questions (section 2.3) allows us control the trade-off between di- versity and fidelity through the 7 parameter. Figure shows how controlling the parameter allows us to operate at different points for diversity and fidelity. Low 1) results in high fidelity, while high results in high diversity. For our experiments we used an r around 0.5 to achieve the right trade-off. Table in Appendix E shows the full metrics result of varying 77.
Effeet of ORACLE prompting. Across datasets, ORACLE prompting yields the best performance in terms of matching the ground-truie truth question. (Refer figure 4 and table in Appendix G). This ablation shows the efficacy of our architecture in- corporating the prompt when generating a question, i.c., providing the "exact" prompt elicits a question which is relatively closer to the ground-trui
4.2 Qualitative study
In Table 5 we provide qualitative examples of ques- tions generated by PROTEGE when compared with BASELINE- -BEM output. Due paucity of space do not include the context input which is fed to both the models. The second column shows the output of BASELINE- -EAM given the context input alone. The third column is a sample of the prompts which are fed to PROTEGE model (along with con-
Effect of greedy algorithm. As described in section 2.3, our algorithm takes the candidate set of questions generated in the first stage (prompt-based controlled generation) and in the second stage per- forms greedy algorithm, at each step optimizing

---

Prompt inputs
BASELINE- output
PROTEGE oulput
(keywords and sentences)
How much RAM does Intel Core pricessor support?
processor
ram, intel, 12, processor, ddr3
CI
The Intel Care processor series uppports DDRS RAM
processor slow?
processors support DDR4?
with freguenckss of 1,066 or 1,333 MHz.
Core processors maximum thermal design power
What is mavimuu power rarlnn of an 13 processor?
ratings ranging from 16 73 warts,
What encoder should for live streaming?
Which encoder best for OBS!
live streaming
C2
Nvidia good Tor OBS?
cpsl, EPU
OBS CPU GPU intensivee
Is GPU Nvidia graphics card good for
Which encoder best for streaming? ?
gpu, nvidia, graphics card
Is the MacRook Air lighter than MI?
machook air, weight
Is MacHook Air or M2 heavier?
The
is hetter choice
vast nuyovity of people
Is better buy MacRook Air or M2?
over the 33-nnch M2 MacBook Pro model, even thougg the has slightly better performance and longer battery life. It's also noticed whenever you slot into big eany around
Is the M2 Air better than M2 Pro?
Is the MacBook Air M2 good for traveling"
the chip than Pro?
compression, reducing file sizes, computer science binury.
What is the best compression tool
What the bys for streaming?
data representation add, bisesize add, usefal tool, part would experience buffering. regular drups, without, quality
for streaming?
C4
What is good speed for streaming?
Does compressing file reduce quality?
Is 128kbps good for streuming?
around I. 500 köps, sd
How mony kbps is normal for streuming?
Table 5: Table showing anecdotes of questions generated by PROTEGE and DASELINE-PROMPE
text). We have shown examples of prompt key- words (c.g., first row), as well as prompt sentences (e.g., second row). Finally last column shows the output of PROTEGE model given the prompt and context as input. We observe that given a con- text PROTEGE leverages the prompts effectively in generating diverse questions when compared to BASELINE-BEAN output. Especially, when sen- tences are passed prompts they often appear to be answers to the generated question.
ent as enrich the QA extraction technique by augmenting it with entity level metadata.
Conclusion and Future Work
In this paper we present PROTEGE, a transformer based two-stage question generation framework based on prompts that balances diversity of the generated questions with their fidelity. Through ex- tensive experiments multiple datasets we show that PROTEGE significantly improves diversity (by + 16) and fidelity (by +8%) compared to strong baselines. As future work, we will extend our models simultameously generate both questions and answers. preliminary experiments on the task of extracting answers for questions from a given context, we have observed that providing the "entities" in the question as additional prompt sig- nals to a BERT based model improves the answer extraction quality by up to +4.2% in Fl score. Sim- ilar applications to other NLG tasks such docu- ment summarization and FAQ creation are possible using the framework proposed in our paper. Exten- sion of our work to non English languages is part of future work.
5 Related Work
Rule-based (Heilman and Smith, 2010; Fabbri et al., 2020) and DNN based (Sun et al., 2018; Yin et al.. 2020) models are used for question generation from text corpora. Answer extraction (Rajpurkar et al., 2016; Kwiatkowski et al., 2019) or machine comprehension (Hermann et al., 2015; Jozefowicz et al., 2016) is a branch of NLP where the goal is to extract answer snippet from text documents given question input. In both cases, either the question or the answer is given as input. QA extraction models (Alberti et al., 2019; Du al., 2017; Reddy et al., 2017; Krishna and lyyer, 2019) are generally pipeline-basee which generates the question and the answer in sequence. Boros et al. (Boros et al., 2021) uses cnnettionannwering system to detect specific events in textual content (c.g., tweets, blogs). In this context, the entity in- formation is used frame template-based question (e.g., Where did the [attack] happen?) where attack is an event of interest). Zhang et al. (Zhang et al., 2021) propose combining entity linkage with a QA system. However, our objective is differ-
Limitations
One limitation of PROTEGE is that it is tightly integrated with existing transformer architecture. Therefore to test its efficacy with Large Language Models (LLMs), we would need access to the pre- trained model parameters. While this is possi- ble for publicly available Large Language Mod- cls (LLMs) such as Vicuna (Chiang et al., 2023), Falcon (Penedo et al., 2023) and LLaMA (Tou-

---

vron et al., 2023), we will miss out state-of-the-art LLMs such as GPT-4 (OpenAI, 2023) and Chat- GPT 2, Further our approach requires large GPU cluster to train which may lead to higher carbon emission.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Lunguage Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Experimental evidences suggest that when con- text span is used prompt, our model may hal- lucinate or mention incomplete product names or product family. For example, instead of "Core 17
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhenghao Jiang. and Graham Neubig, 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4830-4842, Online. Association for Computational Linguistics.
12700K CPU", it may generate a question with "Core i7 12700 CPU" which is ambiguous (i7 12700K CPU has a base frequency of 3.6 GHz in comparison to 2.1 GHz for i7 12700F). Gener- ating questions with quali-qualified product names will be direction of our future work.
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension, In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 1342-1352, Vancouver, Canada. Association for Computational Linguistics.
References
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 6168-6172 Flo- rence, Italy. Association for Computational Linguis- tics.
Ehsan Elhamifar, Guillermo Sapiro, and René Vidal. 2012. Finding exemplars from pairwise dissimilari- ties via simultaneous sparse recovery. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. 2016.
Layer normalization. Cite
Alexander Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. Template-base question generation from retrieved sentences for im- proved unsupervised question answering. In Proceed- ings of the 58th Anmual Meeting of the Asvociation for Computational Linguistics, pages 4508-4513, On- line. Association for Computational Linguistics.
arxiv:1607.064 06450.
Francis Bach. 2013. Learning with Submodular Func- tions: A Convex Optimization Perspective. Now Pub- lishers Inc., Hanover, MA, USA.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis-
Seungju Han, Beomsu Kim, and Buru Chang- 2022. Measuring and improving semantic diversity of di- alogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 934-950, Abu Dhabi, United Arab Emirates. Associ- ation for Computational Linguistics.
tics.
Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 An- nual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609-617, Los Angeles, California. Association for Computational Linguistics.
Emanuela Boros, Jose G. Moreno, and Antoine Doucet. 2021. Event detection as question answering with entity information. CoRR, abs/2104.06969.
Tom Brown, Benjamin Mann, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Esseholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Syxtems, volume 28. Curran Associates, Inc.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard
hhaps://chat.ppenai.com.

---

Few-shot learning with retrieval augmented language models.
Grave. 2022. Atlas:
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 201 6a. Ms marco: A human generated machine read- ing comprehension dataset. CoRR, abs/161 1. 1.09268.
Rafal Jozefowicz. Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling.
Tri Nguyen, Mir Rosenberg, Xia Song. Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016b. MS MARCO: A human gener- ated machine reading comprehension dataset. CoRR, abs/1611.09266
Kalpesh Krishna and Mohit lyyer. 2019, Generating question question-answer answer hierarchies. In Association for Com- putational Linguistics.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles progress in long-form question answering. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4940-4957 Association for Computational Linguistics.
OpenAL. 2023. Gpt-4 technical report.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wes Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics. 7:452 7:452-466.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweh dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Proceedings of the 34th Inter- national Conference on Neurul Information Process- ing Systems, NIPS'20, 20, Red Hook, NY, USA. Curran Associates Inc.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-67.
Pranav Rajpurkar, Robin Jia, and Percy Liang, 2018. Know what you don't know: Unanswerable ques- tions for SQUAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, 2016. A eversity-promoting ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages I 110-119, San Diego, California. Association for Computational I Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguisties.
Chin Lin and Franz Josef Och. 2004. Auto- matic evaluation of machine translation quality using longest common subsequence and skip- bigram statis- tics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL- 04), pages 605-612, Barcclona, Spain.
Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi. 2017. Generating natural language question-answe answer pairs from a knowledge graph using RNN based question generation model. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 376-385, Valencia, Spain, Association for Computational Linguistics.
Zhouhan Lin, Minwei Feng, Cícero Nogueira dos San- tos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. ArXiv,
Gary Marcus. 2020. The next decade in ai: Four steps towards robust artificial intelligence.
G. I.. Nemhauser, L. A. Wolsey, and M. L. Fisher. 1978. An analysis of approximations for maximiz- ing submodular set functions-i. Math. Program., 14( (6)2:555-294
Yiping Song, Rui Yan, Yansong Feng, Yaoyuan Zhang, Dongyan Zhao, and Ming Zhang. 2018. Towards neural conversation model with diversity net using determinantal point processes. In Proceedings of the

---

Thirty-Second AAAI Conference on Artificial Intelli- gence and Thirtieth Innovative Applications of Artifi- cial Intelligence Conference and Eighth AAAT Sym- posium Educational Advances Artificial Intelli- gence, AAAIT8/AAT 18/EAAI 18. AAAI Press.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with bert.
Wenzheng Zhang, Wenyue Hua, and Karl Stratos. 2021. Entqa: Entity linking as question answering. CoRR, abs/2110.02369
Katherine Stasaski and Marti Hearst. 2022. Semantic diversity in dialogue with natural language inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 85 98, Seattle, United States. Association for Computational Linguistics.
Yizhe Zhang. Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, Bill Dolan. 2018, Generating informative and diverse conversational re- sponses via adversarial information maximization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma. and Shi Wang. 2018. Answer-focused and position- aware neural question generation. Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 3930-3939, Brus- sels, Belgium. Association Computational Lin- guistics.
APPENDIX
A Architecture Choices
The architecture for controlled generation is de- scribed in Section 2.1. We once again highlight our key modifications to the standard encoder-decoder architecture in order to achieve objective of controlled question generation:
Guy Tevet and Jonathan Berant. 2020. Evaluating the evaluation of diversity in natural language generation. CoRR, abs/2004.025 0.0999..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Introduction of an additional encoder stack (with 1 + Nr layers) for the "prompt" input. The first .N. layers share the parameters with the encoder stack for the "context" input. The final layer (a transformer block) has its own individual parameters. This choice is not only to reduce computations, but also due to the fact that the difference between context and prompt signals is expected be a higher-level which is captured only the top layer of the encoder.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proceedings of the 3 International Conference on Neural Information Processing Sys- lems, NIPS 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee. David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. Proceed- ings of the AAAI Conference on Artificial Intelligence. 32(1).
Modification of each decoder block to attend to the "context" and "prompt" embedding via separate cross-attention layers. The output of the cross-attention blocks are combined using a MIXUP operation and fed to the feed- forward layer. It is important to note that this is fundamental change to the standard de- coder architecture. The intuition for this ar- chitecture that the cross-attention with the prompt signal influences the decoder to fo- cus on certain topics while the cross-attention with the context influences the decoder to im- prove relevance of the questions generated w.r.t. the context. Together the aggregated cross-attention influences the model to gen- crate question around the prompt while maining true to the information present in the context. This aggregation of the decoder cross- attention with the context and prompt embed-
Ashwin K. Vijayakumar, Michael Cogswell, Ram- prasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models. CoRR, abs/1610.022224.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: state-of-the-and natural language processing. CoRR, abs/19 abs//910.03771.
Xusen Yin, Jonathan May, Li Zhou, and Kevin Small. 2020. Question generation for supporting informa- tional query intents. CoRR, 2bs/2010.09692.

---

ding happens throughout all the decoder lay- crs.
dataset consists of a paragraph/context, a set of questions relevant to the context and for each ques- tion, an answer which is a phrase from within the context. We ignore unanswerable questions (where is "i_mposssiblee = True). We split the original train dataset into train (-80K rows) and dev (-5K rows) by splitting based on litles. We sample from the original validation set (consisting of -10K unique datapoints) create test set (-5K rows).
One of our baselines, BASELINE-PROMIR can be viewed as a combination of "context" and "prompt" tokens at the input level (i.e., early fusion) via cross-attention. We also performed limited experi- ments to combine the encoder outputs for the con- text (he) and prompt (hp) (i.e., mid-level fusion) and observed it to perform worse than BASELINE- PROMPT. Our experiments also suggested that single cross-attention and MIXUP was not sufficient guarantee faithfulness of the generated questions w.r.t. the prompt signal, and hence needs to be repeated in each decoder layer.
Google Natural Questions (NQ): Natural Ques- tions (K (Kwiatkowski et al., 2019) is collection of real user questions submitted to Google and answers gathered from Wikipedia by annotators. From the original dataset parse the ques- tion_text the long_ long_answer answer (and treat it context) and the short _oor__answrr (and treat it as an answer) which usually a phrase from within the context (except for yes/no answers). We filter out contexts that contain HTML tables (<Table>) and also fil- ter out very long contexts (>= 20 sentences). We sample from the original training data of -307K datapoints create train (-120K rows) and dev (-5K rows). We similarly parse the original -7.8K validation set create a test set (-3.5K rows). MS MARCO: MS MARCO (Microsoft Machine Reading Comprehension) (Nguyen et al., 2016b) is a large scale collection of datasets (machine reading comprehension, passage ranking, etc.) of which we leverage the question answering dataset. Queries (questions) are sampled from Bing logs and 10 most relevant passages for the query are generated. Human annotators then tag passages that contain an answer to the question and identify the answers from the relevant passages. From the original dataset, for given query we randomly select one answer and then randomly sample 3 passages (selecting one passage that contains the answer and two passages that do not contain the answer), shuffle and concatenate the passages form our input context. We sample from the origi- nal train and dev datasets to create a train (-100K rows), dev (-5K rows) and test (-5K rows).
B Prompt signals
In table 6 we describe the various prompt signals we have used for our experiments.
C Dataset details
Search QA (SEARCHQA): We start by creating a of custom templates for shopping guidance queries (e.g., things to consider when buying <calegory>, main features of <usecase> <cat- egory>). We expand the templates by populating the slots to create seed set of search queries (e.g.. main features of a gaming laptop). For each query, we submit search request to a third-party search engine and extract questions, answer snippets and URLs. Further, we extract the textual content from the URLs. We thus collect dataset of size -100K. After filtering out rows where we are (a) unable to extract the URL content (b) unable to locate the answer in the extracted URL content we are left with -60K datapoints. Finally, for each question, answer, URL datapoint, starting from the textual content extracted from the URL we extract mul- tiple paragraphs (each containing the answer in different locations) to create a "context" input (data augmentation). Thus, each question, answer, URL datapoint expands into question, answer, context datapoints. From the URLs set we create two mu- tually exclusive set of domain names, one each for train and test datasets (to ensure that models gener- alize across unseen domains), which allows us to create training dataset with ~100K rows, and dev and test datasets each with -5K rows each.
D NQ & MS MARCO observations
For the NQ and MS MARCO datasets, although PROTEGE shows significant improvement over baselines on diversity metrics, improvements are not observed on NLG metrics. In the case of NQ and MS MARCO datasets, the answer is often a short phrase (specifically, in NQ we use the "short answer" provided in the dataset). During inference,
SQuAD 2UA(SQUD)): The Stanford Question Answering Dataset 2.0 (Rajpurkar al., 2) a public dataset consisting of crowdsourced ques- tions on a selection of Wikipedia articles. The

---

Training (ORACEE)
Inference (HEURISTIC)
Iteratively select window of (=2)
Answer text
Text of the ground-truth answer.
the
Keywords derived from the ground-truth answer using RAKE (Rapid Automatic Keyword Extraction) algorithm.
lieratively select window of (=2)
Answer keywords
sentences from the context and derive the key keywords from the context window
Entities from ground-truth question identified using pre-defined dictionary of domain-spectte entities.
lteratively select window of k (=2)
Question entities
sentences from the context and derive the entities from the context window.
Table 6: Prompt signals.
SEARCHQA
SQUAD
NQ
MS MARCO
Test
Train
Train
Dev
Test
Train
Dev
Test
5432
124011
100000
Avg. context ( sentences)
10.00
10.00
10.00
5.12
4.85
5,38
4.18
4.23
4,16
10.18
9.69
9.67
Avg. question size (# words)
10.04
10.27
5.88
Avg. unswer size sentences)
2.12
2.15
2.15
1.00
1.00
1.00
0.79
0.80
0.77
1,15
1.15
1,16
Avg, size @# words)
39.22
39.70
39.87
3.17
3.06
3.61
3.39
3.41
3.42
12.91
14.81
15.32
Table 7: Dataset statistics.
select the top-k phrases/keywords from the con-
necessarily match the phrase/keyword that elicits ths ground-truth question.
text, using an unsupervised keyword detection al- gorithm (RAKE), as prompt signals. Hence, for most examples the specific phrase/keyword which is part of the ground-truth answer does not always get selected. Due to this reason, although we gener- ate number of answerable and diverse questions from the context, may not necessarily generate question semantically similar to the ground-truth question. On the other hand, the bascline tech- niques take as input only the context and no other controlling signal and hence, are more likely to gen- erate a question similar to the ground-truth. This issue is not seen with SEARCHQA and SQUAD datasets, where answers are usually complete sen- tences which are picked candidate prompt sig- nals during inference. Further note that for all datasets, including NQ and MS MARCO, when the model is provided with the exact ground-truth answer-based based prompt, it indeed generates a ques- tion semantically closer to the ground ques- tion as observed the Section 4.1 ("Effect of OR- ACLE prompting").
Example in NQ:
Context: The American Civil War was fought the United States from 1861 to 1865...
Ground-truth question: who took part in the american civil war
Ground-truth short answer: nationalists of the Union
Prompt
Generated question
president abraham lincoln
who was president when the civil war began where did the civil war take place in
united states
Example in MS MARCO:
Context: Jesse James (VII) Producer Actor. At first glance, Jesse James is the consunmate biker rebel...
Ground-truth question: who is married to jesse james Ground-truth short answer: Karla James, Janine Lindemulder, Sandra Bullock and Alexi DeJoria:
Below provide examples to illustrate why NQ and MS MARCO datasets the performance of PROTEGE is lower than baselines NLG metrics. As seen the examples below the question gener- ated by PROTEGE depends on the phrase/keyword that gets selected during inference, which need not
E
Ablation: Balancing diversity vs fidelity
Table 8 shows the full metrics as a result of varying
77.

---

Prompt
Generated question
0.450
METEOR
BLEU-4
Jesse James (VII) Producer Actor.
0.425
RDUIGE4
who is jesse james
0.400
0.375
0.350
Tattoos, knives, goatee, black t-shirts and skulls all around him
what is jesse james famous for
0.300
0.250
BASELINE-EEA PROTEGE HEURISTIC PROTECE ORACLE
eta
Dist-1
Dist-2
Dist-3
Ent-1
Enf-2
Ent-3
Fidelity
Figure 4: ORACLE vs HEURISTIC prompts.
0.00
0.5686
0,7338
0.8149
3.5514
3.7571
3.7190
0.8895
0.01
0.5901
0.7599
0.8383
3.6003
3.8137
3.7683
0.8894
H Ablation: Effect of greedy algorithm
0.10
0.6165
0,7898
0.8633
3.6668
3.8288
0.8882
0.25
0.6287
0,8030
0.8743
3.6984
3,8569
D.8877
Table 12 shows the effect of greedy algorithm on the full set of diversity metrics. As expected, across all datasets diversity metrics improve with greedy algorithm.
0.6408
0.8161
0.8850
3.7303
3,9518
3,8854
0.8784
0.75
0.6508
0.8263
0.8928
3.7580
3,9794
3.9084
0.8611
0.90
0.6542
0.8296
0.8952
3,7683
3,9890
3,9164
0.8447
0.99
0.6547
0.8296
0.8951
3.7709
3,9911
3.9183
0.8371
In table 13 we observe that post-greedy top-1 METEOR reduces for some datasets. This is ex- pected as the generated question from the first stage is often replaced by question which displays high diversity and fidelity. However, top-2 and top-3 the METEOR slightly increases (except for NQ) indicating that the greedy algorithm implicitly fa- vors the question more closer to the ground-truth (which is also expected to be answerable) as long as it improves the diversity.
1.00
0.6554
0.8295
0.8944
3.7651
3,9825
3 9070
0.8284
Table 8: Diversity vs Fidelity with varying y.
F Ablation: Effect of prompt signals
In tables 9 and 10 we present the complete set of diversity and NLG metrics based the choice of prompt signals. Specifically for the NL.G met- rics, regardless of the choice of prompt signal for top-1 question, BASELINE-BEAM generates ques- tions closest to ground-truth followed by answer keywords. For top-2 and top-3, the best strategy in general is to pass answer keywords as prompts during training and keywords from context spans of size 2 or 4 during inference. The best result with answer keywords is better than the best result with answer text, indicating that model benefits more when passed keywords rather than full text as guidance signal. Among different context span sizes passing a larger window (2 or 4 sentences) leads to better results. The worst performing is question entities possibly because model tends to overfit on the specific prompted entities, while it generalizes when passed with larger window of keywords//entences.
I Audit SOP
We perform human audits with 2 auditors to com- pare the generated questions between PROTEGE and BASELINE-BEL texts). For each data point, auditors record the fol- lowing details regarding the top-k generated ques- tions: (A) Are the questions readable and mean- ingful (i.e., well-formed and complete sentences)? (B) Out of the readable questions, how many ques- tions are semantically unique (measures semantic diversity)? (C) Out of the readable questions, how many questions are answerable from the context (measures fidelity)? In case of conflicts on any of the labels, third auditor re-verifies the decision to resolve the conflict. Finally, we take cumulative count for each aspect and measure the percentage of readable, unique and answerable questions.
(for a sample of -200 con-
G Ablation: ORACLE prompting
Figure 4 and table 11 (with the complete set met- ries for ORACLE vs HEURISTIC prompting) shows that on an average there is a 40+% improvement in METEOR metrics from HEURISTIC prompting compared to ORACLE prompting.

---

Training prompt (OrACla)
Inference prompe (HEURISTIC)
BERT
Dataset
Model
Dist-1
Dist-2
Dist-3
Rut-1
Fmt-2
Ent-3 1
Fidelity
Score
BASELINE-BEAA BEAM
NA
NA
0.5379
0.6963
0.7675
3,5451
3.7396
3,6759
0.9009
0.8408
Answer-Text
Context span (size=1)
0.7381
09197
3,9869
4.0804
Answer-Text
Context span (size=2)
0.6925
0,8829
0,9290
3.8926
4.1388
4.0343
0.8405
0.9005
SEARCHQA
PROTTGG
Answer-Text
Context-span (size=4)
3,6813
3.8432
Keywords from
Anewer-kyywods
0.6931
0,8800
0.9376
3.9216
4.1666
4,0688
0.8977
context span (size=1)
Keywords from context span (size=2)
Keywords
Keywonds from context span (size=4)
Answer Keywords
Question-Fmiries Contest antiries
0.7459
0.9278
0.9629
3.9967
4.2152
4.0664
0.8234
0.8765
Table 9: Effect of prompts on diversity metrics.
Top-1
Top-2
Top-3
Terining prompt
Infereeen prompt (ElzuHaTTEC TEC)
Dataset
Medel
(ORACLE)
METEOR
BLEU-4
ROUGE-L
METEOR
BLEU-
ROUGEL
METEOR
BLEU-4
ROUGEL
INS-LLNE-BEAM
NA
NA
0,3058
0,3794
0.3375
0.3596
D.418
0.3533
0.3968
0.444)
IL41KS
Amsweer Tex:
Consos (ilp=1)
0,2414
0.3186
0.2740
0.5233
0.3865
0.310
03779
0.422
IL3965
Arswa-Teex Text
Contool (sismm))
0,2588
0.3384
0,2902
0.3450
0,3992
0.3671
0.3921
0.4341
1.110
SEARCHQA
Ausweer Text
Contoo: Gizzel)
0,2421
0.3550
0.3112
0.365
0-4200
0.3866
03964
0.4440
1L4148
Arsona
Alenaa-Kyywadl
Keywords from
quan (size=1)
0.3546
0.3387
0.2955
0.3520
0.4060
0.3607
0
0.4415
D-4177
Reywords from
Answer Keywords
0,2767
0,377
0.3001
0.4161
0,3803
0,834
0.5456
span
Areacs-Keywonle
Keywords froa span
0,2938
0,9627
0.3162
0.3836
Queniien-Pane
0,2500
90H
Table 10: Effect of prompts on NLG metrics.
BASELINE-BEAM
PROTECE (HEURISTIC)
PROTECE (ORACLE)
Prompt Choice
METEOR
BLEU-4 top-1
ROUGE-L
METEOR
BLEU-4 top-1
ROUGE-L
METEOR
BLEU-
ROUGE-L
(Training)
top-1
top-1
top-1
top-1
top-1
top-1
top-1
Answer-Text
0.2821
0.3560
0.3102
0.3221
0.4043
0.3518
SEARCHQA
Answer-Keywards
0.3375
0.2938
0.3627
0.3162
0.3107
0.3915
0.3353
Question-Fatittes
0.2275
0.2502
0.5032
SQUAD
Answer-Text
0.1678
0.2346
0.2083
0.1593
0,2289
0,2053
0.3442
0.3133
NQ
Answer-Text
0.3998
0.4634
0.4307
0.3089
0.3965
0.3391
0.4224
0.4763
0.4518
MS MARCO
Answer-Text
0.4217
0.4712
0.4637
0.3788
0.4234
0.4120
0.4940
0.5364
0.5226
Table II: Effect of ORACLE prompting.
Pre-Greedy
Post-Greedy
BERT
BERT
Dataset
Dist-1
Dist-2 1
Dist-3 f
Fidelity
Dist-1
Dist-2
Dist-3
Fidelity 1
Score
Score
SEARCHQA
0.6158
0,8009
0.8743
0.8665
0.8387
0.7351
0.9197
0.9632
0.8235
0.9085
SQUAD
0.6911
0.8744
0.9222
0.8161
0.8030
0.7490
0.9340
0,9680
0.7915
0.8812
NQ
0.6254
0.7597
0.8294
0,8414
0.7483
0.7028
0.8522
0.9117
0,8048
0.8151
MS MARCO
0.6472
0.7984
0.8928
0.8402
0.7566
0.7177
0.8693
0.9426
0.8138
0.8278
Table 12: Effect of greedy algorithm diversity metrics.
Pre-Greddy
Pus-Greedy
METEOR
METEOR
METEOR
METEOR
METEOR
Dataset
top-1
top-2
top-3
top-1
top-2
top-3
SEARCHOA
0.2952
0.3646
0.4002
0.2767
0.3638
0.4034
SQUAD
0.1386
0.1996
0.2368
0.3550
0.4146
0.4435
0.3089
0.3855
0.4262
MS MARCO
0.3662
0.4562
0,5021
0.3788
014631
0.5064
Table 13: Effect of greedy algorithm on NLG metrics.


Reviews:
[{"review_id": "xNdrKtzeOH", "reviewer": "Reviewer_MjJZ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your valuable comments. Please find below our response for the feedback and questions you have shared:\n\n&nbsp;  \n***\n*“A relatively simple framework structure. The key difference from existing question generation models is the addition of an extra layer at the end of the encoder stage to extract embedding of prompt. Furthermore, these extracted embeddings are utilized in the decoder stage through cross-attention. However, applying attention across different input modalities is one of the commonly used approaches in NLP.\"*\n\nWe thank the reviewer for this insightful comment. We would like to highlight the following key modifications made to the standard encoder-decoder architecture in order to achieve our objective of controlled question generation:\n\n* Introduction of an additional encoder stack (with $1 + N_e$ layers) for the “prompt” input. The first $N_e$ layers share the parameters with the encoder stack for the “context” input. The final layer (a transformer block) has its own individual parameters. This is to reduce the computations and also due to the fact that the difference between context and prompt signals will be at a higher-level which is captured only at the top layer of the encoder. \n* We modify each decoder block to attend to the “context” and “prompt” embeddings via separate cross-attention layers. The output of the cross-attention blocks are combined using a MIXUP operation (Figure 1) and fed to the feed-forward layer. It is important to note that this is a fundamental change to the standard decoder architecture. The intuition behind this architecture is that the cross-attention with the “prompt” signal influences the decoder to focus on certain topics while the cross-attention with the “context” influences the decoder to improve relevance of the questions generated w.r.t the context. Together the aggregated cross-attention influences the model to generate a question around the “prompt” while remaining true to the information present in the context. This aggregation of the decoder cross-attention with the “context” and “prompt” embeddings happens throughout all the decoder layers. \n\nThe reviewer is correct in pointing out that applying cross-attention across different modalities (“context” and “prompt”) is similar to a single application of the MIXUP operation.  One of our baselines, BASELINE-PROMPT, can be thought of combining “context” and “prompt” tokens at the input level (early fusion) via cross-attention. We also did limited experiments (not presented in the paper) to combine the encoder output of the “context” embeddings and “prompt” embeddings (mid-level fusion) and observed it to perform worse than BASELINE-PROMPT.  Our experiments also suggested that a single cross-attention and MIXUP is not sufficient to guarantee faithfulness of the generated questions w.r.t the prompt and hence, we propose the need for cross-attention and MIXUP to be repeated in each decoder layer.  \n\n&nbsp;  \n***\n*“Lack of detail explanation. In Section 2.2, a detail explanation of heuristic method is missing”*\n\nIn Section 2.2 we have mentioned HEURISTIC as a class of rule-based methods for generating prompt signals, and we have mentioned the exact choice of the heuristics separately in Section 4.1 (“Effect of prompt signals”) as an implementation detail. Below we provide the definitions of training prompt signals (along with the corresponding inference prompts) used in Table 3 of our paper:\n\n**A) Answer text:**\n\n* For training: Text of the ground-truth answer.\n* For inference: Iteratively select a window of k (=2) sentences from the context. \n\n**B) Answer keywords:**\n\n* For training: Keywords derived from the ground-truth answer using RAKE (Rapid Automatic Keyword Extraction) algorithm. \n* For inference: Iteratively select a window of k (=2) sentences from the context and derive the keywords from the context window \n\n**C) Question entities:**\n\n* For training: Entities from ground-truth question identified using a pre-defined dictionary of domain-specific entities.\n* For inference: Iteratively select a window of k (=2) sentences from the context and derive the entities from the context window.  \n\n&nbsp;  \n***\n*“In Table 4, it is unclear the meaning of pre-greedy and post-greedy.”*\n\nWe apologize for any confusion regarding the terminology of ‘pre-greedy’ and ‘post-greedy’. We have defined the greedy algorithm in Section 2.3 via Equation 6. Further, we have explained the setup of the experiments in Section “Effect of greedy algorithm” (Lines 501 to 513) and presented the results in Table 4. Please consider the additional explanation below:\n\nOur approach (PROTEGE) works in two stages. **In the first stage,** we generate a candidate set of questions using prompt-based controlled generation (described in Section 2.1) using the architecture defined in Figure 1. For a given context, we vary the prompt signals (using heuristic rules) to generate a collection of a variety of questions. **In the second stage,** we implement a *greedy hill-climbing algorithm* that optimizes for equation 6 (i.e., greedily choose a set of generated questions from the first stage that achieves the best balance between n-gram diversity and fidelity). In Table 4, we call the output of the first stage as **‘pre-greedy’** and the output of the second stage as **‘post-greedy’**. We show the effect of this greedy hill-climbing algorithm by demonstrating that the resulting set of questions post the application of greedy algorithm has an improved *diversity* (by 13%) and *fidelity* (by 9%), as a direct effect of optimizing for the two criteria.  \n\n&nbsp;  \n***\n*“Insufficient explanation about limited performance. In table 2, the NLG metrics for fidelity are worse in NQ and MS MARCO dataset. It needs more analysis.”*\n\nThe reviewer has rightly called out that in Table 2, the NLG metrics are inferior to baselines for NQ and MS MARCO datasets. Table 2 shows the NLG metrics (METEOR, BLEU-4, ROUGE-L) across 4 datasets. Fidelity metrics are reported as part of Table 1 (last column) wherein PROTEGE shows significant improvement, including for NQ and MS MARCO datasets. Regarding the performance of NQ and MS MARCO on NLG metrics, we’d like to share the following hypothesis:\n\nIn the case of NQ and MS MARCO datasets, the answer is often a short phrase (specifically, in NQ we use the “short answer” given in the dataset). During inference, as the prompt signal we select the top-k phrases/keywords using an unsupervised keyword detection algorithm (RAKE). Hence, it is quite likely that for most examples we do not select the specific phrase/keyword which is part of the ground-truth answer. Due to this reason, although we generate a number of answerable and diverse questions for the context, we may not necessarily generate a question semantically similar to the ground-truth question. On the other hand, the baseline technique takes as input only the context and no other controlling signal and hence, is more likely to generate a question similar to the ground-truth. This issue is not seen with SEARCH-QA and SQUAD datasets where answers are usually complete sentences and during inference we pick each sentence in the context as a candidate prompt signal. The **examples below from NQ and MS MARCO** further illustrate our explanation.\n\n| **Dataset**               | **NQ**                                                                        |\n|---------------------------|-------------------------------------------------------------------------------|\n| **Context**               | _The American Civil War was fought in the United States from 1861 to 1865..._ |\n| **Ground-truth question** | _who took part in the american civil war_                                     |\n| **Ground-truth answer**   | _nationalists of the Union_                                                   |\n| **Prompt-1**              | _president abraham lincoln_                                                   |\n| **Generated question-1**  | _who was president when the civil war began_                                  |\n| **Prompt-2**              | _united states_                                                               |\n| **Generated question-2**  | _where did the civil war take place in_                                       |\n\n| **Dataset**               | **MS MARCO**                                                                                              |\n|---------------------------|-----------------------------------------------------------------------------------------------------------|\n| **Context**               | _Jesse James (VII) Producer \\| Actor. At first glance, Jesse James is the consummate biker rebel..._      |\n| **Ground-truth Question** | _who is married to jesse james_                                                                           |\n| **Ground-truth Answer**   | _Karla James, Janine Lindemulder, Sandra Bullock and Alexi DeJoria._                                      |\n| **Prompt-1**              | _Jesse James (VII) Producer \\| Actor._                                                                    |\n| **Generated Question-1**  | _who is jesse james_                                                                                      |\n| **Prompt-2**              | _Tattoos, knives, goatee, black t-shirts and skulls all around him_ |\n| **Generated Question-2**  | _what is jesse james famous for_                                                                          |  \n\nFurther we note that, for all datasets, including NQ and MS MARCO, when the model is provided the exact ground-truth answer-based prompt, it indeed generates a question semantically closer to the ground-truth question as described in the Section **“Effect of ORACLE prompting”** and the NLG metrics shown in Figure 4. \n\n&nbsp;  \n***\n*“In Figure 1, using identifiers (a) and (b) for sub-figures can make a more precise explanation.”*\n\nWe thank the reviewer for this suggestion. In the final version of the paper we will add identifiers (a) and (b) for the sub-figures of Figure 1, for a better explanation.  \n\n&nbsp;  \n***\n*“This paper makes marginal contributions (vs non-contemporaneous work)”*\n\nIn this paper, we have proposed a novel modification to the fundamental architecture of a decoder layer to achieve the objective of conditional generation. In particular, the prompt signal is encoded separately (from the context signal) which the decoder attends to and combines the representations with the vanilla cross-attention throughout all the layers. This helps the decoder gradually achieve the desired combination of generating an answerable question from the context, while grounding it around concepts provided as prompt signals. In order to leverage the most from this architecture, we have proposed a greedy hill-climbing algorithm that optimizes for a combination of diversity (so that multiple unique questions are generated) and fidelity (so that questions stay true to the context provided). We have applied our technique to a unique setting of question generation which is not among the most widely discussed topics in literature (unlike popular NLG tasks such as document summarization). Finally, although not presented in this paper, we have applied our technique to the task of answer extraction (given context and a question) and demonstrated that providing entities as prompt signals improves the extraction quality by up to **+4.2% in F1 score** over a baseline that doesn't incorporate the prompt signal.  \n\n&nbsp;  \n***\n*“The contribution depends on data that are simply not available outside the author's institution or consortium; not enough details are provided.”*\n\nWe have demonstrated the efficacy of PROTEGE through experiments on SQUAD, NQ and MS MARCO which are popular benchmark Q&A datasets that are publicly available. The post-processing details for these datasets are described in detail in Appendix A (“Dataset details”). We will also make the post-processed versions of these public datasets available as part of the final version of the paper. Similarly, the SEARCHQA dataset is created from publicly available sources and we will release this dataset as part of the final version of the paper. Our implementation details, including hyper-parameter settings are described in Section 3.2. We believe these steps will help in reproducing the claims made in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Easy-to-read and well-written paper.\n2. Analyzed through various experiments and metrics.\n3. Study about the relevance between diversity and fidelity.\n", "reasons_to_reject": "1. A relatively simple framework structure. The key difference from existing question generation models is the addition of an extra layer at the end of the encoder stage to extract embedding of prompt. Furthermore, these extracted embeddings are utilized in the decoder stage through cross-attention. However, applying attention across different input modalities is one of the commonly used approaches in NLP.\n\n2. Lack of detail explanation. In Section 2.2, a detail explanation of heuristic method is missing. In Table 4, it is unclear the meaning of pre-greedy and post-greedy.\n \n3. Insufficient explanation about limited performance. In table 2, the NLG metrics for fidelity are worse in NQ and MS MARCO dataset. It needs more analysis. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly studies how to improve the prompt-based question generation from web articles, called PROTEGE,  especially under the environment of domain-specific and long-form textual content. To achieve this, the paper proposes the encoder-decoder based LLM which can take diverse prompt, consisting of keyword-based and sentence-based, and generates question. In addition, to get the balance between diversity and fidelity, a hill-climbing algorithm with sub-modular objective function is leveraged. In this method, the diversity is calculated by the number of unique n-grams and the relevance is calculated with the answerability with standard LLM. The experiment results are evaluated with diverse datasets and analyzed with diversity metrics and fidelity metrics.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 1, using identifiers (a) and (b) for sub-figures can make a more precise explanation."}, "extra_text": {}}, {"review_id": "RTopz2q5D6", "reviewer": "Reviewer_ypQJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your valuable comments. Please find below our response for the feedback and questions you have shared:\n\n&nbsp;  \n***\n*“The study has not shown practical applications to help people for this research. We want more empirical diversity (add more languages if possible)”*\n\nWe agree that our experiments in this paper are focused exclusively on the Question Generation task. Our intention was to show the efficacy of the proposed technique strongly on a given NLG task by including experiments on a diverse variety of QA datasets (SEARCHQA and SQUAD, NQ, MS MARCO). However, our proposals including the *architectural framework* and the *greedy hill-climbing algorithm* are generic enough to be applied on a wide variety of NLG tasks. Although not included in this paper, we have applied the technique to the task of answer extraction from a given document (context) and question. In our experiments, we have observed that providing the “entities” in the question as additional prompt signals to a BERT-based model improves the answer extraction quality by up to **+4.2% in F1 score**. Similar applications to other NLG tasks such as *document summarization* and *FAQ creation* are possible using the framework proposed in our paper. Extension of our work to non English languages is part of future work.\n\n&nbsp;  \n***\n*“what is the way to make sure the questions that are generated stick to the passage and are answerable?”*\n\nWe have addressed the point about generating questions that are relevant to the context through our “fidelity” objective function and the corresponding metric. Fidelity is part of the objective function that we optimize for in the greedy hill-climbing algorithm (referred to as *relevance* in equation 6), and is also a metric that we report results for. Fidelity means answerability, i.e., whether the generated question is answerable from the given passage/context. Given a context and a question, we have trained a BERT model that outputs an answerability score. We leverage this score to select the question (from the candidate set of questions output from the first stage of PROTEGE) that not only adds maximum diversity to the already selected pool, but also is most answerable from the given passage (based on its answerability score). This process naturally avoids questions which do not stick to the passage or are irrelevant to the given context. As a result, as shown in Table 1, PROTEGE outperforms the baseline techniques on the Fidelity metric (last column) across datasets. More fundamentally, since our training data comprises of ground-truth questions that are answerable from a context, our modelling framework tends to generate questions that are relevant to the given context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study introduced a new method (PROTEGE) to generate questions based on LLM architecture. The study performed various experiments on three datasets and provided a lot of insight into the results. The article can open a period of generating Q&A data more automatically and faster than previous methods.", "reasons_to_reject": "The study has not shown practical applications to help people for this research.\nWe want more empirical diversity (add more languages if possible)\nAnd what is the way to make sure the questions that are generated stick to the passage and are answerable?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose PROTEGE - a question generation method based on the LLM architecture and based on input prompts. In addition, this study also introduces the greedy hill-climbing algorithm to balance the fidelity and diversity of the generated question.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Y1UJp7qRb", "reviewer": "Reviewer_aVs7", "summary": "", "questions": "- Is the paper formatted according to the EMNLP guidelines? There appear to be inconsistencies in the paper format, such as the font.\n- In lines 215-216, are the context and prompt encoded separately, or is there an alternative approach?\n- Have the authors tested their method on other controllable generation tasks, or is the proposed approach solely limited to question generation tasks? It would be interesting to explore the method's applicability in other scenarios.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, we thank you for your valuable comments. Please find below our response for the feedback and questions you have shared:\n\n&nbsp;  \n***\n*“The proposed method shows inferior performance against baselines when evaluated using NLG metrics (METEOR, BLEU, ROUGLE). It would be beneficial to include additional examples in the Appendix, explaning why tehse scores are relatively low.”*\n\nWe agree with the reviewer that PROTEGE shows inferior performance against baselines on NLG metrics (shown in Table 2), especially for the NQ and MS MARCO datasets. For SEARCHQA and SQUAD datasets, PROTEGE shows a marked improvement over baseline techniques on NLG metrics, for the case of Top-2 and Top-3 generated questions (please check METEOR scores for Top-2 and Top-3 in Table 2). However for the Top-1 case, PROTEGE performance is lower than baselines. Please note that **Table 1 (Diversity metrics)** and **Table 2 (NLG metrics)** are two different dimensions of results, as explained below.\n\n**Diversity vs NLG metrics:** Diversity metrics demonstrates the ability of PROTEGE to generate a diverse set of answerable (relevant) questions from a given context. NLG metrics (METEOR, BLEU-4, ROUGE-L) demonstrate how close the generated questions are to the ground-truth question. Our goal in this paper is not to generate the ground-truth question alone accurately, but to generate a set of highly diverse questions which are answerable from the given context. Hence, NLG metrics are not our primary choice of metrics. However, we still report NLG metrics to demonstrate that among our top-k generated questions, at least one of the questions is semantically similar to the ground-truth question. Table 2 shows that the top-1 question generated by PROTEGE, while being highly relevant (answerable), need not necessarily be similar to the ground-truth question for the given context (hence the lower performance). When considering top-2 or top-3 questions, PROTEGE starts generating at least one question closer to the ground-truth which is shown in the NLG performance being better than BASELINE-BEAM (for SEARCHQA and SQUAD datasets). However, in the case of NQ and MS-MARCO even at top-2/top-3 the NLG metrics are not better than baseline, which can be explained with the following hypothesis.\n\n**NLG metrics on NQ, MS MARCO:** In the case of NQ and MS MARCO datasets, the answer is often a short phrase (specifically, in NQ we use the “short answer” given in the dataset). During inference, as the prompt signal we select the top-k phrases/keywords using an unsupervised keyword detection algorithm (RAKE). Hence, it is quite likely that for most examples we do not select the specific phrase/keyword which is part of the ground-truth answer. Due to this reason, although we generate a number of answerable and diverse questions for the context, we may not necessarily generate a question semantically similar to the ground-truth question. On the other hand, the baseline technique takes as input only the context and no other controlling signal and hence, is more likely to generate a question similar to the ground-truth. \n\n**Additional examples for NQ, MS MARCO:** The examples below for NQ and MS MARCO further illustrate our explanation. Please note, although the generated questions are relevant and answerable, they are different from the ground-truth question. As suggested by the reviewer, we will add these examples to the Appendix in the final version of the paper. We thank the reviewer for this suggestion.\n\n| **Dataset**               | **NQ**                                                                        |\n|---------------------------|-------------------------------------------------------------------------------|\n| **Context**               | _The American Civil War was fought in the United States from 1861 to 1865..._ |\n| **Ground-truth question** | _who took part in the american civil war_                                     |\n| **Ground-truth answer**   | _nationalists of the Union_                                                   |\n| **Prompt-1**              | _president abraham lincoln_                                                   |\n| **Generated question-1**  | _who was president when the civil war began_                                  |\n| **Prompt-2**              | _united states_                                                               |\n| **Generated question-2**  | _where did the civil war take place in_                                       |\n\n| **Dataset**               | **MS MARCO**                                                                                              |\n|---------------------------|-----------------------------------------------------------------------------------------------------------|\n| **Context**               | _Jesse James (VII) Producer \\| Actor. At first glance, Jesse James is the consummate biker rebel..._      |\n| **Ground-truth Question** | _who is married to jesse james_                                                                           |\n| **Ground-truth Answer**   | _Karla James, Janine Lindemulder, Sandra Bullock and Alexi DeJoria._                                      |\n| **Prompt-1**              | _Jesse James (VII) Producer \\| Actor._                                                                    |\n| **Generated Question-1**  | _who is jesse james_                                                                                      |\n| **Prompt-2**              | _Tattoos, knives, goatee, black t-shirts and skulls all around him_ |\n| **Generated Question-2**  | _what is jesse james famous for_                                                                          |  \n\n**NLG metrics with Oracle prompting:** Finally we also note that, for all datasets, including NQ and MS MARCO, when the model is provided the exact ground-truth answer-based prompt, it indeed generates a question semantically closer to the ground-truth question as described in the Section “Effect of ORACLE prompting” and the NLG metrics shown in Figure 4. \n\n&nbsp;  \n***\n*“Some important details are missing in the experiment setting. For instance, the paper does not specify the number of examples used in human evaluation or the number of participants involved in the evaluation process.”*\n\nWe thank the reviewer for pointing out this important detail which may require further information. We had 2 auditors to work on a sample of ~200 data points for BASELINE-BEAM and PROTEGE outputs. For each data point (context and a set of generated questions), auditors were asked to label 3 values a) Readability (no. of questions that are meaningful & well-formed) b) Diversity (no. of semantically unique questions which elicit different answers from the context) c) Fidelity (no. of questions that are answerable from the given context). In case of conflicts on any of the labels, a 3rd auditor re-verified the decision to resolve the conflict. The Standard Operating Procedure (SOP) followed for labeling by auditors is described in Appendix F (Audit SOP).\n\n&nbsp;  \n***\n*\"Furthermore, there are concerns about the reliability of the fidelity metrics based on the trained neural network. The suitability of the trained BERT model for measuring the fidelity score of generated questions is questionable, considering results in Table 2.\"*\n\nIn the earlier point, we have provided the key differences between diversity metrics (from Table 1) and NLG metrics (from Table 2) and explained our hypothesis of why PROTEGE shows inferior performance on NLG metrics for some of the datasets. The reliability of the fidelity metrics mentioned by the reviewer is an important point that needs further information.\n\nIn order to report the answerability metrics, we train a standard BERT-based model that takes a context and a question as input, and predicts the probability of the question being answerable from the context. We leverage this score to select the question (from the candidate set of questions output from the first stage of PROTEGE) that not only adds maximum diversity to the already selected pool, but also is most answerable from the given passage (based on its answerability score). The ROC AUC of this BERT model is 0.84. We tune the threshold of this model to operate at a precision of 85%, corresponding to a recall of ~30%. A higher bar on the precision allows us to select questions which are highly likely to be answerable from the context, at the cost of losing other answerable questions. **The reliability of this answerability model is also demonstrated through human audits (line 463) where PROTEGE was found to improve upon the baseline by 36% on the fidelity score**.\n\n&nbsp;  \n***\n**Questions For The Authors-1:**\n*“Is the paper formatted according to the EMNLP guidelines? There appear to be inconsistencies in the paper format, such as the font.”*\n\nYes, the reviewer is correct to point out that the paper is formatted using Computer Modern Roman font, which is one of the approved formats for EMNLP and has been approved by the Program Chairs. However, in the final version, we will recompile the paper using the Times font which is the more popular format.\n\n&nbsp;  \n***\n**Questions For The Authors-2:**\n*“In lines 215-216, are the context and prompt encoded separately, or is there an alternative approach?”*\n\nAs indicated by the reviewer, the “context” and the “prompt” signals are encoded separately. We have introduced an additional encoder stack (with $1 + N_e$ layers) for the “prompt” input, where the first $N_e$ layers share their parameters with the encoder for the “context” input and a final layer (a transformer block) with individual parameters. This is to reduce the computations and also due to the fact that the difference between context and prompt signals will be at a higher-level which is captured only at the top layer of the encoder. Also this allows our model to learn unique representations for the “context” (a longer data with multiple concepts/topics) and the “prompt” (shorter, more specific signal such as entities, specific answer etc.). The decoder attends to the “context” and “prompt” embeddings separately via cross-attention layers and aggregates the representations through a MIXUP operation. These details are described in Section 2.1 of our paper. Alternatively, one of our baselines, BASELINE-PROMPT, can be thought of combining “context” and “prompt” tokens at the input level (early fusion) via cross-attention. We also did limited experiments (not presented in the paper) to combine the encoder output of the “context” embeddings and “prompt” embeddings (mid-level fusion) and observed it to perform worse than BASELINE-PROMPT. \n\n&nbsp;  \n***\n**Questions For The Authors-3:**\n*“Have the authors tested their method on other controllable generation tasks, or is the proposed approach solely limited to question generation tasks? It would be interesting to explore the method's applicability in other scenarios.”*\n\nWe agree that our experiments in this paper are focused mainly on the Question Generation task. Our intention was to show the efficacy of the proposed technique strongly on a given NLG task by including experiments on a diverse variety of QA datasets (SEARCHQA and SQUAD, NQ, MS MARCO). However, our proposals including the architectural framework and the greedy hill-climbing algorithm are generic enough to be applied on a wide variety of NLG tasks. Although not included in this paper, we have applied the technique to the task of answer extraction from a given document (context) and question. In our experiments, we have observed that providing the “entities” in the question as additional prompt signals to a BERT-based model improves the answer extraction quality by up to **+4.2% in F1 score**. Similar applications to other NLG tasks such as document summarization and FAQ creation are possible using the framework proposed in our paper.\n\n&nbsp;  \n***\n*\"Table 5, C1-Baseline output is missing\"*\n\nIn Table 5, C1 example the baseline model had output only two unique questions. Hence, the table shows only two outputs for baseline in C1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experimental results are promising, and the paper conducts a thorough analysis of the results in terms of diversity and fidelity. The inclusion of human evaluations in Figure 2 supplements the reliability of the findings.\n- The idea of employing separate cross-attention architecture for conditional generation is a novel contribution to my knowledge.", "reasons_to_reject": "- The proposed method shows inferior performance against baselines when evaluated using NLG metrics (METEOR, BLEU, ROUGLE). It would be beneficial to include additional examples in the Appendix, explaning why tehse scores are relatively low.\n- Some important details are missing in the experiment setting. For instance, the paper does not specify the number of examples used in human evaluation or the number of participants involved in the evaluation process. Furthermore, there are concerns about the reliability of the fidelity metrics based on the trained neural network. The suitability of the trained BERT model for measuring the fidelity score of generated questions is questionable, considering results in Table 2.", "questions_to_authors": "- Is the paper formatted according to the EMNLP guidelines? There appear to be inconsistencies in the paper format, such as the font.\n- In lines 215-216, are the context and prompt encoded separately, or is there an alternative approach?\n- Have the authors tested their method on other controllable generation tasks, or is the proposed approach solely limited to question generation tasks? It would be interesting to explore the method's applicability in other scenarios.", "paper_topic_and_main_contributions": "This paper proposes a method named Prompt-based diverse question generation (PROTEGE), which enables diverge question generation through a decoder modification and greedy hill-climbing algorithm. Specifically, the method involves forwarding the prompt to the different cross-attention layers in the decoder layer and then combining the representations from two cross-attention layers. The experimental results show that the proposed method can generate questions with improved fidelity and diversity compared to previous prompting methods that simply concatenate the prompt to the input context.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 5, C1-Baseline output is missing"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces PROTEGE, a novel technique for question generation over web articles. The basic concept is to condition generation on a prompt, either keyword-based (e.g. an entity in the article) or phrase-based (e.g. a sentence in the article). Cross-attention to this prompt is then used during generation, and different prompts can be selected to induce diversity. The authors combine this with a hill-climbing submodule that allows control over the tradeoff between diversity and fidelity. 

The experiments in the paper test the model using common QA datasets. The authors demonstrate that both fidelity and – especially – question diversity benefits from their approach. That is, given an article, the questions generated by PROTEGE are significantly better at covering the aspects of the article compared to prior work. The reliability of these findings are supplemented with human experiments, providing additional backing.

The paper is well-written and clearly presented, and the empirical findings are a convincing demonstration of PROTEGE's efficacy.|This paper presents a method called PROTEGE that does a two-step process for question generation from text sources.  First, a set of diverse questions is generated, using a pair of encoders one of the document and the other for the "prompt signals".  Next, there is a submodular objective function defined by a pair of quantities meant to control the trade-off between diversity and fidelity, and this optimized and hill-climbed when arriving at the final set of questions.

The authors could have done a slightly better job motivating this work.  One of the reviewers noted the lack of practical application.  However, this work not only pertains to the claimed area of constructing QA datasets, but also to the related—and very practical—areas of building better QA systems, and/or evaluating them.  This came out, albeit in a slightly oblique way, in the rebuttal to reviewer ypQJ.  Nevertheless, I believe this paper warrants inclusion in the conference.
Decision: Accept-Findings