An Empirical Comparison of LM-based Question and Answer Generation Methods
Year: 2023

Abstract:
Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context (e.g. a paragraph). This task has a variety of applications, such as data augmentation for question answering (QA) models, information retrieval and education. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an endto-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches. However, there are differences depending on the underlying generative LM. Finally, our analysis shows that QA models fine-tuned solely on generated questionanswer pairs can be competitive when compared to supervised QA models trained on human-labeled data.

heading: 1 Introduction
text: Question and answer generation (QAG) is the task of generating a set of question-answer pairs given an input context such as a document, a paragraph or a sentence. QAG can be applied to develop question answering (QA) models without human supervision (Lewis et al., 2019; Zhang and Bansal, 2019; Puri et al., 2020) and as a data augmentation mean for QA model understanding (Shakeri et al., 2020; Bartolo et al., 2021). Moreover, QAG is used as an aid of educational systems (Heilman and Smith, 2010; Lindberg et al., 2013), to improve information retrieval models (Pyatkin et al., 2021; Lewis et al., 2021), and as a tool for model interpretation (Perez et al., 2020; Lee et al., 2020).
QAG stems from question generation (QG) (Mitkov and Ha, 2003; Du et al., 2017; Zhou et al., 2017; Du and Cardie, 2018), which consists of generating a question given an answer on the input context. Despite QG being widely studied in the language model era (Murakhovs’ka et al., 2022;
Ushio et al., 2022), QAG is a more complex task, since the answer needs to be generated and not assumed to be part of the input. Therefore, it is unclear what types of QAG models work in practice as no comprehensive comparisons have been established so far.
In this paper, we formalize QAG as a task that generates question-answer pairs given a context, and compare three simple QAG strategies based on fine-tuning encoder-decoder language models (LMs) such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). Our three proposed approaches (illustrated in Figure 1) consist of: (1) pipeline QAG, which decomposes the task into answer extraction and question generation, learning a separate model for each subtask; (2) multitask QAG, which uses a shared single model to train both subtasks instead of independent ones; and (3) end2end QAG, which uses end-to-end sequenceto-sequence learning to generate question-answer pairs directly. Finally, we compare these three approaches on a multi-domain QA-based evaluation, where QA models are trained with the questionanswer pairs that each QAG model generates. All
14262
the QAG models are publicly released via HuggingFace (Wolf et al., 2020)1, and available on the online demo2.

heading: 2 Related Work
text: There are a few works that leverage pre-trained LMs for QAG. For example, Alberti et al. (2019) first fine-tuned BERT (Devlin et al., 2019) on answer extraction and QG, and generate questionanswer pairs by extracting an answer, on which the associated question is generated. Puri et al. (2020) followed a similar idea by fine-tuning an autoregressive LM for QG. In contrast, Shakeri et al. (2020) fine-tuned a single LM on answer extraction and QG jointly. Lee et al. (2020) trained an LSTM sequence-to-sequence model from scratch to generate question and answer sequentially. More recently, Bartolo et al. (2021) used a QAG model to generate adversarial examples for QA. Similarly, Lewis et al. (2021) improved on extractive QA by generating millions of question-answer pairs via QAG. In these two last cases, the model to fine-tune was BART (Lewis et al., 2020).
While all these studies use the three methods that we analyse in this paper (i.e. pipeline, multitask and end2end), these are not easily comparable, as there are important differences among them in terms of settings, dataset, input to the LMs, and evaluation metrics. Moreover, except for Lewis et al. (2021), none of the proposed QAG models have been made publicly available. Finally, the two most recent studies using BART (Bartolo et al., 2021; Lewis et al., 2021) have not performed any evaluation on the QAG model, as it is included as a part of a larger pipeline. We summarize the comparison of these prior works and our evaluation at Table 1.

heading: 3 Question & Answer Pair Generation
text: Given an input context c (e.g. a paragraph), QAG aims to generate natural question-answer pairs Qc related to the information in c: Qc = {(q1, a1), (q2, a2), . . . }. In what follows we describe three different approaches for QAG based on fine-tuning language models.
1https://github.com/asahi417/ lm-question-generation
2https://autoqg.net/

heading: 3.1 Pipeline QAG
text: The QAG task can be decomposed into two simpler subtasks, answer extraction (AE) and QG, where the AE model Pae first generates an answer candidate ã on a sentence s in context c, and then the QG model Pqg generates a question q̃ that is answerable by answer ã given context c. The AE and QG models can be trained independently on any paragraph-level QG datasets that consist of quadruples (c, s, a, q) by maximizing the conditional loglikelihood of:
ã = argmax a
Pae(a|c, s) (1)
q̃ = argmax q
Pqg(q|c, s, a) (2)
where the log-likelihood is factorized into tokenlevel predictions, similar to other sequence-tosequence learning settings (Sutskever et al., 2014). In practice, the input to the AE model takes the form of:
[c1, . . . , <hl>, s1, . . . , s|s|, <hl>, . . . , c|c|]
where si and ci are the i−th token of s and c respectively, | · | represents the number of tokens in a text, and <hl> is the highlighted token to mark the sentence in the context, following the QG formulation of Chan and Fan (2019) and Ushio et al. (2022). Likewise, the input to the QG model takes the answer into account by:
[c1, . . . , <hl>, a1, . . . , a|a|, <hl>, . . . , c|c|]
where ai is the i−th token of a. At inference time, we simply replace the gold answer a of the QG
model (2) by the prediction from the AE model (1), and run the inference over all the sentences in context c to obtain question-answer pairs. Consequently, the pipeline approach can generate, at most, as many pairs as sentences in c.

heading: 3.2 Multitask QAG
text: Instead of training independent models for each subtask, a shared model can be fine-tuned on both AE and QG jointly in a multitask learning manner. To be precise, we mix the training instances for AE and QG altogether, and randomly sample a batch at each iteration of fine-tuning. Each subtask is distinguished by a task prefix added at the beginning of the input text: “extract answer” (AE) and “generate question” (QG).

heading: 3.3 End2end QAG
text: Instead of breaking down QAG into two separate components, we can directly model it by converting the question-answer pairs into a flattened sentence y, and fine-tuning a sequence-to-sequence model to generate y from c. Let us define a function that maps Qc to a sentence as:
T (Qc) = “{t(q1, a1)} | {t(q2, a2)} | . . . ’’ (3) t(q, a) = “question:{q}, answer:{a}’’ (4)
where each pair is textualized with the template (4) and joined by a separator |. The end2end QAG model Pqag is then optimized by maximizing the following conditional log-likelihood:
ỹ = argmax y
Pqag(y|c) (5)

heading: 4 Evaluation

heading: 4.1 Experimental Setting
text: Data. QAG models are trained on SQuAD (Rajpurkar et al., 2016). As their outputs consist of arbitrary questions and answers, reference-based NLG evaluation metrics traditionally used in QG research (Papineni et al., 2002; Denkowski and Lavie, 2014; Lin, 2004; Mohammadshahi et al., 2022) are unsuitable. As such, we conduct an extrinsic evaluation by training QA models on the data generated by the QAG models. For this, we rely on SQuADShifts (Miller et al., 2020), an English reading comprehension dataset in four domains (Amazon/Wikipedia/News/Reddit). For both SQuAD and SQuADShifts, we rely on the train/validation/test splits provided in QG-Bench (Ushio et al., 2022).
Multi-domain QA Evaluation. Given a QAG model to be assessed, we first generate questionanswer pairs on each domain of SQuADShifts, and fine-tune DistilBERT (Sanh et al., 2019) on the generated pseudo QA pairs, where F1 and exact match on the test set are considered as the target metric. This SQuADShifts QA-based evaluation can be used to probe the robustness of the model across domains, as well as for the overall performance by averaging metrics over the domains. Our QA evaluation relies on Tune,3 an efficient grid search engine for parameter optimization, to find optimal hyperparameters during QA model fine-tuning. Base Models. For all comparison systems (i.e. pipeline, multitask and end2end), we experiment with T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) as base LMs, with the model weights t5-{small,base,large} and facebook/bart-{base,large} shared on HuggingFace.4 Moreover, we report the results of a QG model that takes the gold answers from the provided QA training set as input (QG-only). This is similar to the pipeline method but excluding the AE component.

heading: 4.2 Results
text: Table 2 shows the SQuADShifts QA evaluation results for the three approaches considered. Interestingly, the top-2 best models, BARTLARGE (multitask) and T5LARGE (end2end), outperform Gold QA (i.e., the model using the human-labeled gold annotations) in two out of four domains, as well as the average in both F1 and exact match. Even smaller models such as T5SMALL are competitive with respect to using the gold standard questionanswer pairs.
Given the results, it is unclear which approach provides the best performance, as BARTLARGE (multitask) achieves the best average F1 score (including the best results on Amazon and Reddit domains in both metrics), while T5LARGE (end2end) obtains the best average exact match (as well as the best results on Wiki and NYT domains in both metrics). Among the QAG approaches, T5 consistently works better with the end2end QAG, while BART is not well-suited when used end2end. A possible explanation is that T5 has observed sentences with structured information due to its multitask pre-training objective, while BART did not have
3https://docs.ray.io/en/latest/tune/index.html 4See Appendix A for details on the procedure to find opti-
mal hyperparameters during model fine-tuning.
such training instances as it was trained only on a denoising sequence-to-sequence objective.

heading: 4.3 Generation Size Analysis
text: In the SQuADShifts QA evaluation, the number of question-answer pairs generated by QAG models is often larger than the human-labelled gold dataset in each domain, as shown in Table 3.5 Therefore, to fairly compare the quality of generated question-answer pairs, we randomly downsampled the number of the generated question-answer pairs to match the size of the gold dataset. For this analysis we focus on the best-performing T5LARGE and BART LARGE QAG models6, and run the same SQuADShifts QA evaluation with the downsampled pairs. Figure 2 shows the average of F1 scores over 10 independent trials with different random seeds at downsampling.7 In this experiment, no model outperforms the gold QA baseline. This indicates that the human-annotated gold dataset is still more informative and data efficient than the generated question-answer pairs. Also, since the pipeline/multitask QAG models generate more pairs than the end2end model, downsampling has a larger effect on the pipeline and multitask models than the end2end model. This means that the T5LARGE (end2end) model can generate questionanswer pairs of higher quality than those generated by BARTLARGE (multitask), although they are equally competitive in the main experiment (§ 4.2).
5The size of generated question-answer pairs in each domain can be found in Appendix B.
6The end2end BART LARGE results match those from Table 2, since it had less data than the gold dataset.
7See Appendix C for the comparison of exact match.

heading: 4.4 QAG Model Comparison
text: So far, we have compared the three QAG approaches in terms of performance. However, performance is not the only criterion to consider when choosing a QAG model, since each approach has its own advantages and limitations in terms of computational cost and usability. From the perspective of computational complexity, end2end QAG is faster than the others at both of training and inference, because it can generate a number of question-answer pairs at once in a single paragraph pass. In contrast, both multitask and pipeline need to parse every sentence separately, and a single prediction consists of two generations (i.e. answer extraction and question generation). Essentially, the relative increase of computational cost from end2end QAG to pipeline/multitask QAG can be approximated by the average number of sentences in each paragraph. In terms of memory requirements, both multitask and end2end QAG rely on a single model, but pipeline QAG consists of two models, requiring twice as much memory storage. Finally, while computational-wise end2end is the lightest model, both pipeline and multitask approaches can generate a larger number of question-answer pairs on average, with the added benefit of being able to run the models on individual sentences. Table 4 shows a practical comparison of the three approaches.

heading: 5 Conclusion
text: In this paper, we formalized QAG as a task to generate pairs of questions and answers given an input context, and established baselines with three different QAG approaches. To compare them, we
conducted a multi-domain QA based evaluation that measures the performance of a QAG model by fine-tuning QA models on the QA training dataset generated by the QAG model. Our evaluation shows that end2end QAG models that generate questions and answers simultaneously are generally the most reliable. Nonetheless, establishing a multitask paradigm with separation between answer extraction and question generation can have added benefits, especially when using LMs such as BART. In general, the results are promising, as they show that these artificially-generated QA datasets rival in quality with those annotated by humans, which could save large amount of resources.

heading: Acknowledgements
text: Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship.
Limitations
In this paper, we studied paragraph-level QAG models, which limits their input up to around 500
tokens, and the same approach cannot be easily applied to longer documents. Also, the answer is an entity or a phrase consisting of a few tokens and the question requires one-hop reasoning, so our models are not able for use in generating longer answers or multi-hop questions. As far as the languages are concerned, the models studies here are English only and to adapt SQuADShifts QA evaluation in other languages, we need QA datasets to train and evaluate the QAG model in those languages.
The focus on this paper was on evaluating the quality of generated question-answer pairs. As such, we do not attempt to achieve the best QA model possible, but rather use question answering as an extrinsic evaluation. This extrinsic evaluation could be further enhanced with an intrinsic manual evaluation that we did not perform in this paper. Finally, given computational constraints, our QA evaluation is based on a single model only. Again, the goal here was not to achieve the best QA performance, but we acknowledge than using different models could lead to different results.
Ethics Statement
Since pre-trained LMs are known to inherit undesirable biases and tend to generate toxic contents in some edge cases (Schick et al., 2021), the QAG
models we developed in the paper could potentially generate a question or an answer including such texts. Nevertheless, we have done internal validation on the generated question-answer pairs and we have not found such examples in the data analysed in this paper.

heading: A Hyper Parameters
text: At each QAG model fine-tuning, we search the optimal hyperparameters such as learning rate via lmqg8, a hyperparameter search tool for sequenceto-sequence LM fine-tuning, and Table 5 shows the best hyperparameters. The maximum input length is fixed as 512, and the maximum output length is 256 for the end2end QAG and 32 for the others.

heading: B Size of QA Pairs at SQuADShifts QA evaluation
text: Table 6 shows the number of question-answer pairs generated from different QAG models in each domain at SQuADShifts QA evaluation. The size of the test sets are 4,942 (Amazon), 3,696 (Wiki), 5,032 (NYT), and 4,901 (Reddit).

heading: C Additional Results of Downsampled SQuADShifts QA evaluation
text: Figure 3 shows the exact match of the downsampled SQuADShifts QA evaluation experiment.
8https://pypi.org/project/lmqg
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We have dedicated limitation section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We have dedicated ethics statement section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: They are in the abstract and the introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We have developed models and the evaluation metric, explained in the section 2.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In the section 3, we refer the language models we used in our experiments.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We will describe the license of our models at the github repository.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss it in the limitation section.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use an existing dataset, where the original work already validated that aspect, so we haven’t done it.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? The models are explained in the section 2 and 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We have a complete table in the appendix to describe the statistics of the dataset.
C 3 Did you run computational experiments? We explain it in section 3.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We explain it in the appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We explain it in section 3 and the appendix.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We explain the evaluation metric in section 2 and 3,

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We explain it in section 3.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship. Limitations
In this paper, we studied paragraph-level QAG models, which limits their input up to around 500
tokens, and the same approach cannot be easily applied to longer documents. Also, the answer is an entity or a phrase consisting of a few tokens and the question requires one-hop reasoning, so our models are not able for use in generating longer answers or multi-hop questions. As far as the languages are concerned, the models studies here are English only and to adapt SQuADShifts QA evaluation in other languages, we need QA datasets to train and evaluate the QAG model in those languages. The focus on this paper was on evaluating the quality of generated question-answer pairs. As such, we do not attempt to achieve the best QA model possible, but rather use question answering as an extrinsic evaluation. This extrinsic evaluation could be further enhanced with an intrinsic manual evaluation that we did not perform in this paper. Finally, given computational constraints, our QA evaluation is based on a single model only. Again, the goal here was not to achieve the best QA performance, but we acknowledge than using different models could lead to different results