Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation
Year: 2023

Abstract:
Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, partly due to the inaccessibility of LLMs, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches, namely round-trip and prompt-based score, to selecting high-quality questions from a set of LLM-generated candidates. Our method works without the need to modify the underlying model, nor does it rely on human-annotated references — both of which are realistic constraints for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation. 1

heading: 1 Introduction & Related Work
text: Large Language Models (LLMs) have recently gained tremendous popularity in the NLP community (Devlin et al., 2019; Liu et al., 2019; Bao et al., 2020; Brown et al., 2020). The ever-increasing size in both models and training data renders many traditional learning methods impractical/intractable. As a result, prompt-based learning has emerged as a new paradigm tailored specifically towards leveraging the power of LLMs (Radford et al., 2019; Petroni et al., 2019; Raffel et al., 2020; Brown et al., 2020; Schick and Schütze, 2021b; Gao et al., 2021; Liu et al., 2021). In the zero-shot setting (such as in this study), a data sample is first “verbalized” into an input prompt and a ground-truth response in natural language. The prompt is then issued to a pre-trained LLM to obtain a predicted
∗ Equal contribution. 1We open-source all code and annotated data on github.
response, which is then compared to the groundtruth for evaluation. This new technique has been successfully applied to many applications including text classification (Yin et al., 2019; Schick and Schütze, 2021a), QA (Jiang et al., 2021), natural language generation (Li and Liang, 2021) and NLG evaluation (Yuan et al., 2021).
Despite the impressive results on popular NLP benchmarks, however, the back-end LLMs are usually pre-trained with general-domain data, leading to sub-optimal performance in new domains for prompt-based learning. There are two major challenges in successful applying general-purpose LLMs to specific domains. Firstly, aside from the many known issues of LLMs (Webson and Pavlick, 2021; Min et al., 2022; Zhao et al., 2021; Lampinen et al., 2022), their sheer size and/or accessibility (e.g., served via API over the internet) makes it prohibitively expensive and impractical for domain adaptation. These limitations have inspired a recent line of work known as prompt editing/tuning (Gao et al., 2021; Li and Liang, 2021; Madaan et al., 2022). Additionally, prompt-tuning often relies on the availability of ground-truth labels of the data, imposing much additional resource on the approach.
Given the ubiquity of these challenges, our study focuses on alleviating the constraints on both annotation availability and access to model parameters, making LLMs more accessible for deployment. We take a mainstream NLG task, namely question generation, as a case study (Du et al., 2017; Yuan et al., 2017; Du and Cardie, 2018; Pan et al., 2019; Liu et al., 2020; Pyatkin et al., 2021). In this task, a model is trained to generate a natural language question conditioned on a context and an answer, such that the generated question can be answered by the provided answer using the context as supporting evidence. Question generation is the corner stone for many NLP applications in education (Kurdi et al., 2020; Abdelghani et al., 2022), FAQ
12952
generation (Mass et al., 2020), information seeking (Qi et al., 2020), etc. In an educational setting, for example, a question generation system can generate demonstrations that inspire students’ curiosity and thinking (teaching), or to help assess students’ proficiency on certain knowledge or skills (examining). These use cases would benefit greatly from reduced dependency on computing resources, data availability, and the expertise required for finetuning an LM.
To align with these real-world scenarios, our goal is to obtain better outputs from an inferenceonly LLM (i.e., as a “black-box”, which is relatively more accessible, e.g., through online APIs). In particular, given the common practice of sampling multiple outputs to improve generation diversity, we propose a method that aims at selecting the best candidate based on multiple aspects of question quality in a zero-shot manner — notably without model adaptation or human annotations. Our method can be seen as a post-hoc selection process within a larger NLG pipeline, and thus is orthogonal and applicable to zero-shot and incontext learning methods (Rubin et al., 2021; Lu et al., 2022; Liu et al., 2022).

heading: 2 Problem Setting
text: Notations: Formally, we consider a dataset of context-answer pairs (c, a) both as strings. The task of question generation is to generate a question q that can be answered by a using c as supporting evidence. We use an off-the-shelf pretrained LLM-based question generator in a zeroshot setting (prompt construction detailed in Appendix A). To simulate the black-box generator scenario, we refrain from any form of model tuning. We do, however, assume access to a set of output sequences stochastically sampled from the question generator. We thus ground our study to this application scenario by sampling k questions Q = {qi ∶ i = 1, . . . , k}. For comparison as a baseline, we also denote qg as the question generated with a greedy algorithm (i.e., generating the most probable token at each time step).
Our goal is to devise an algorithm S which selects the best candidate qi∗ that maximizes some evaluation metric M ∶ Q ↦ R, i.e., S(Q) = i∗ = argmaxiM(qi). We use Ms, Ms, and Ms to denote the mean, min, and max of {M(q) ∶ q ∈ Q}, resp., and Mg for the greedy output M(qg). Semantically, Ms ≤Ms ≤Ms is tautologically true,
and a positive result on the design of S would translate to M(qS(Q)) outperforming both Ms and Mg. Datasets and model: We adopt two question generation datasets with distinctive characteristics, namely SQuAD (Rajpurkar et al., 2016) and Fairytale QA (Xu et al., 2022). SQuAD was originally proposed as an extractive QA dataset. It has been used as a sentence-level question generation task in the question generation literature (Du and Cardie, 2018; Yuan et al., 2017; Bao et al., 2020), i.e., a context c is a single sentence that contains the corresponding answer a as a sub-string. Fairytale QA has also been used for both question answering and question generation. It features paragraphlevel question generation (with c being one or more paragraphs), and the answer a is not necessarily a sub-string of c. Since we do not perform any form of model/prompt tuning, we use the testing split of both datasets, which consist of 11,877 data points for SQuAD and 1,007 for Fairytale QA.
We prompt GPT-3 (Brown et al., 2020)2 in a 0-shot manner for both question generation and selection (detailed in §3). We provide all prompts in Appendix A.
Evaluation Metrics : We use two metrics to evaluate the selected question q′ = M(qS(Q)): • Reference-based evaluation: Following prior works, we use BLEU-4 for SQuAD (Du and Cardie, 2018; Bao et al., 2020) and ROUGE-L for Fairytale QA (Xu et al., 2022)3. These metrics compare q′ against the reference question q̂ (a.k.a. the “groundtruth” question in the existing literature). • Human evaluation: we solicit human annotations on a subset of the data. We postulate that an overall score given holistically to rate a question would be highly subjective and thus less inductive to annotator agreement. Accordingly, we decompose the quality of questions into seven dimensions4, and ask human annotators to rate a question on each dimension followed by an overall rating of the question. We collect three annotations from different annotators for each data points. We provide details of the human study in Appendix B.
2We use the text-davinci-002 model. 3We adopt evaluation scripts provided with the datasets. 4Namely, grammatical correctness, offensiveness, clarity,
relevance, importance, specificity, and answerability.

heading: 3 Method
text: In this section we propose three question selection methods. As described in §2, each method is used to score k sampled questions in Q and selects the candidate with the highest score.
n-gram similarity: We use n-gram similarity between a question and its context to measure their relevance. This method reflects the intuitive assumption that favorable question be closely related to the information provided by the context. Specifically, we extract all unique n-grams5 sn(c) from a given context c, sn(q) from a question q. The n-gram similarity score is then defined as: simn = ∣sn(c) ∩ sn(q)∣∣sn(q)∣ , (1) where ∣s∣ indicates the size of set s. Round-trip: Intuitively, the answer to a generated question should be semantically equivalent to the answer that has been used to generated the question. Formally, a question generation model QG and a QA model (both with reasonable performance) should satisfy the following constraint:
q ′ = QG(c, a); a′ = QA(c, q′); a′ = a. (2)
This idea is closely related to cycle consistency in the existing literature on image generation (Zhu et al., 2017), machine translation (Artetxe et al., 2018), and QA (Alberti et al., 2019; Shah et al., 2019)). Here, we use GPT-3 as an off-the-shelf QA model to obtain a′ for each pair of c and q′, resulting in k answers A = {a′1, . . . , a′k} for the k
5In all our experiments n ranges from 1 to 5.
sampled questions in Q. We then measure the similarity between each a′i and the ground-truth answer a (F1 for SQuAD and ROUGE-L for Fairytale QA, in accordance with the evaluation setup from the original papers for the two datasets). Finally, we select the question corresponding to the generated answer a′i∗ that overlaps the most with a (i.e., that can be best answered by GPT-3). Prompts used in these experiments are detailed in Appendix A.
Prompt-based Score: We propose a two-step procedure (Figure 1) for prompting GPT-3 to answer the same set of meta-questions (i.e., questions about the quality of a given question) used for human evaluation (§2). In step 1, given a contextquestion pair, GPT-3 is prompted to answer a metaquestion as an open question (as opposed to choosing among a list of options) as well as to verbalize a reason for its answer. In step 2, GPT-3 is prompted to choose from a list of options representing the rating scale of the meta-question.
We empirically observe that without the first step, GPT-3 output tends to have a low-entropy distribution, i.e., often choosing the same option for a given meta-question disregarding the different context-question pairs. In contrast, the additional first-step appears to improve prediction diversity, which is inline with observations made in some existing studies (Nye et al., 2021; Wei et al., 2022).
Similar to human evaluation, we also prompt
GPT-3 to generate an overall score of a question (denoted overall prompt-based score or OPS). The average score of all individual meta-questions is instead denoted averaged prompt-based score (APS).

heading: 4 Results and Discussion
text: To measure the performance of a selection method (§3), we use it to select one out of k = 5 questions sampled from GPT-3, and score the selection with the evaluation metrics outlined in §2. Additionally, we test the ensemble performance with multiple methods. To ensure comparability, we normalize the scores obtained from each selection method into the range between 0 and 1, and use their average score to perform question selection.
Reference-based evaluation Reference-based evaluation are automatic metrics that are applied to the entire test sets of SQuAD and Fairytale QA. Table 1 shows that on both datasets, all question selection methods outperform Ms, the average score over all five sampled questions, validating the effectiveness of the proposed methods. While all individual methods outperform the greedy baseline Mg on SQuAD, round-trip performs the best, outperforming Mg on both datasets. It can be further improved via ensemble with n-gram and/or prompt-based scores (using uniform weights).
Note that prior studies require a large amount of labeled data for model training/fine-tuning, while GPT-3 performs zero-shot inference. Despite this major difference in learning paradigm, most GPT3-based models proposed here outperform previous results by significant margins on the SQuAD dataset — even the least performant samples Ms (lowerbound) achieve competitive results. For
Fairytale QA, however, only the best samples Ms (upperbound) outperform previous results (Xu et al., 2022), indicating margins for improvement on question selection strategies for future work.
Human evaluation Human evaluation consists of 16, 800 annotations (from 87 annotators) evenly split across the two datasets (details in Appendix B). For question generation (among many NLG tasks), model outputs may exhibit linguistic diversity while maintaining semantic equivalence. It is thus highly problematic to evaluate such outputs against a single reference (i.e, “ground-truth”). Figure 2 empirically shows that the ground-truth (GT) provided in the datasets often fail to receive the highest human ratings, on many occasions scoring lower than stochastic samples from GPT-3 (Ms). Consequently, we strongly advocate for human evaluation, which we believe is higly effective in improving generalizability of our results to real-world applications.
Another prominent observation is that n-gram and APS perform quite differently on the two datasets. On SQuAD, n-gram similarity outperforms other individual methods, with further noticeable improvements via ensemble with roundtrip. APS, on the other hand, does not work nearly as well, performing the worst for almost all metaquestions. In contrast, n-gram (particularly trigram) similarity shows the worst performance on Fairytale QA, while APS outperforms all other methods by a noticeable margin.
We posit that the reversed trend in comparing n-gram and APS can be explained by the distinct natures of the datasets. For SQuAD, the sentencelevel contexts are relatively short and simple with
strictly extractive answers (i.e., the answers being sub-strings of the corresponding contexts). As a result, paraphrasing the context can be a fairly effective strategy to generate questions (hence the stronger correlation between question quality and the c–q n-gram similarity). In contrast, with multiparagraph contexts and abstractive, open-ended answers, Fairytale QA questions are more likely posed about abstract ideas rather than simple context paraphrasing. Consequently, n-gram similarity, which favors local context paraphrasing, is less likely to perform well.
Conclusion In this study, we investigate the practical problem of selecting the best output from multiple samples generated by an LLM. Using question generation as a case study, we propose two prompt-based question selection methods. To alleviate real-world constraints on using LLMs, the proposed methods do not require model fine-tuning nor human annotation. Extensive experiments with both automatic and human evaluations evince the effectiveness of our approach on question selection.

heading: 5 Limitations
text: We acknowledge that our system has some limitations that warrants further investigation. For example, one needs to be mindful of the specific downstream applications of the proposed methods, both in terms of 1) potentially large variance in outof-distribution performance (e.g. divergent question generation applications that aim to spark children’s curiosity-driven thinking (Abdelghani et al., 2022)); and 2) of mitigating harmful/toxic contents in educational applications (Bender et al., 2021). As a result, we believe such techniques and applications are neither suitable nor safe to directly interact with children, we urge developers to use this technique in other ways, for instance, in teaching assistant application (e.g., a system that suggests examples for teachers), where the teacher can filter and modify the examples and thus making sure the content children receive is proper and safe.
We also acknowledge the prohibitively restrictive access to the GPT-3 model at the time of writing. We do believe that this constraint will relax over time, and meanwhile, hoping that our proposal can shed light on research and applications with more accessible LLMs such as GPT-J (Wang and Komatsuzaki, 2021) and BLOOM (BigScience, 2022) for future work.
While we acknowledge the many limitations
with respect to accessing GPT-3, we are not advocating against using it. On the contrary, in fact, we believe GPT-3 is still among the most cost-effective solutions especially in the context of natural language generation. The main goal of the study is thus to explore more data efficient ways of using GPT-3 to generate and evaluate questions. We strive to share our experience and insights with the community, which hopefully can be proven valuable and helpful.

heading: A Prompt Designs
text: We report an example of our prompt for question generation in Figure 3.
We report an example of our prompt for QA (used in round-trip) in Figure 4.
We report an example of our prompt in obtaining prompt scores in Figure 1.

heading: B Human Study
text: We randomly sample 50 documents from each of the two datasets SQuAD and Fairytale QA. Each document correspond to one ground-truth question and six questions generated by GPT-3 (five by stochastic sampling and one by greedy search). Each question is then rated by three human annotators wrt seven meta-questions and one over-all rating, altogether constituting 50 × 2 × (1 + 5 + 1) × 3 × (7 + 1) = 16, 800 annotations. There are in total 87 annotators involved in the annotation process, all annotators are English speakers, they are recruited from regions including Europe, the United States and United Kingdom. Each annotator on average performed 193 annotations and was paid on average $14.1 USD per hour.
We perform a basic spam filtering process on the raw annotations. We observe a 15.4% spam rate. All human scores reported in this paper are computed after spam removal.
We report the eight meta-questions we used for human annotation in Figure 5. The eight metaquestions correspond to columns in Figure 2. We collect three annotations from different annotators for every meta-question, we report the averaged human agreement rate in Table 2.

heading: C Additional Results
text: In Table 3, we report the full experiment results for reference-based evaluation.
In Table 4, we report the full results for human evaluation on SQuAD.
In Table 5, we report the full results for human evaluation on Fairytale QA.
D Implementation Details
In all experiments, we use the text-davinci-002 (175B parameters) variant of GPT-3. It is currently the most capable GPT-3 model variant. Compared to other variants, text-davinci-002’s support to inserting completions can better facilitate our question generation tasks (as shown in Figure 3).
We use a temperature of 0.7 during the sampling process of question generation. In all other use cases (e.g., QA round-trip, prompt score), we use greedy generation (temperature is set to 0).
text after green as suffix. We refer readers to the GPT-3 documentation for more details about GPT-3’s inserting completion mode.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 5 on page 5.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 5 on page 5.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 on page 1.
7 A4. Have you used AI writing assistants when working on this paper? This paper discuss an application of GPT-3, but we do not use GPT-3 in any of the paper section writing.
B 3 Did you use or create scientific artifacts? We use existing language model, namely GPT-3; we also use existing question generation dataset, namely SQuAD and Fairytale QA. We discuss them in Section 2 (page 2).

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, we cite the creators in Section 2, page 2.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. They are publicly available.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, in Section 1 and 5.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in section 2, 3 and appendix B.
C 3 Did you run computational experiments? Yes, described in section 2, 3, 4.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. We do not propose new model. We use existing language model, and properly cite the original work.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, section 4 and appendix B.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Yes, appendix B.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Yes, appendix B.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Yes, appendix B.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Yes, appendix B.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We do believe that this constraint will relax over time, and meanwhile, hoping that our proposal can shed light on research and applications with more accessible LLMs such as GPT-J (Wang and Komatsuzaki, 2021) and BLOOM (BigScience, 2022) for future work. While we acknowledge the many limitations
with respect to accessing GPT-3, we are not advocating against using it. On the contrary, in fact, we believe GPT-3 is still among the most cost-effective solutions especially in the context of natural language generation. The main goal of the study is thus to explore more data efficient ways of using GPT-3 to generate and evaluate questions. We strive to share our experience and insights with the community, which hopefully can be proven valuable and helpful