Learning to Plan and Generate Text with Citations (2024)

Abstract: The increasing demand for the deployment of LLMs in information-seeking scenarios has spurred efforts in creating verifiable systems, which generate responses to queries along with supporting evidence. In this paper, we explore the attribution capabilities of plan-based models which have been recently shown to improve the faithfulness, grounding, and controllability of generated text. We conceptualize plans as a sequence of questions which serve as blueprints of the generated content and its organization. We propose two attribution models that utilize different variants of blueprints, an abstractive model where questions are generated from scratch, and an extractive model where questions are copied from the input. Experiments on long-form question-answering show that planning consistently improves attribution quality. Moreover, the citations generated by blueprint models are more accurate compared to those obtained from LLM-based pipelines lacking a planning component.

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated remarkable abilities to engage in creative conversations (Thoppilan et al., 2022; OpenAI, 2023), summarize information from contextual cues (Goyal et al., 2022; Zhang et al., 2023), and deliver zeroshot performance on a wide range of previously unseen predictive and generative tasks (Brown et al., 2020; Chowdhery et al., 2022). They are also becoming increasingly useful in information-seeking scenarios, ranging from answering simple questions (Roberts et al., 2020; Rae et al., 2021) to generating responses to search-like queries (Nakano et al., 2021; Menick et al., 2022).
The increasing demand for the deployment of LLMs in information-seeking scenarios has further spurred efforts in creating verifiable systems, which
∗Equal contribution. †Work done during internship at Google.
generate responses to queries along with supporting evidence. The evidence can take the form of a URL pointing to a short segment of text which supports an answer (Bohnet et al., 2022), an attribution report with evidence snippets (Gao et al., 2022), quotes cited verbatim from pages retrieved from a search engine (Menick et al., 2022), and references to passages extracted while browsing (Nakano et al., 2021; Gao et al., 2023). In fact, this last type of evidence has been recently adopted in the form of in-line citations by commercial search engines such as BingChat1 and perplexity.ai2.
Regardless of how the evidence is presented, recent approaches tend to rely on a retrieval system (e.g., a commercial search engine) to obtain passages relevant to a query, while an LLM conditions on them to generate a response (Menick et al., 2022; Nakano et al., 2021; Bohnet et al., 2022). Other work generates an answer to the input query first and subsequently retrieves relevant evidence in a post-processing step (Bohnet et al., 2022). Alternatively, the retrieved evidence can be used to further revise the generated response rendering it more consistent with the evidence (Gao et al., 2022).
Despite recent efforts, it remains an open question how to best develop models with a built-in mechanism for attribution to external evidence. A related question is whether said mechanism contributes to generating more factually faithful output. Large-scale evaluation studies paint a worrying picture. Liu et al. (2023b) find that long-form responses from existing search engines frequently contain unsupported statements or inaccurate citations, while Bohnet et al. (2022) show that model performance on attribution varies greatly (between 46% and 71%) across different architectures for the simpler question answering task.
In this paper, we explore the attribution capabilities of plan-based models which have been
1https://bing.com/new 2https://perplexity.ai
11397
shown to be less prone to hallucinations and more controllable (Moryossef et al., 2019; Puduppully et al., 2019; Narayan et al., 2021, 2023; Huot et al., 2023b,a). We focus on long-form question answering (Fan et al., 2019; Vig et al., 2022; Xu and Lapata, 2022) which aims to generate summaries from a set of passages that answer a specific query. We simulate how a search engine might synthesize passages of high relevance to a user query by assuming access to a retriever, and some way of verifying the output, i.e., by citing sources (see Figure 1). Our models operate on retrieved passages and learn to plan and generate summaries with attribution. On account of being more expressive, plan-based models allow us to formalize different forms of attribution, e.g., plans can be verified via citations to passages, while summaries can be verified through
citations to the plan, passages, or both. Our models conceptualize text plans as a sequence of questions operating as blueprints for generation, determining what to say and in which order (Narayan et al., 2023; Liu et al., 2023a; Huot et al., 2023b). Questions as a planning mechanism are ideally suited to attribution, since they provide a natural link between retrieved passages and their summaries. We define two models which differ on whether the question-based plan is generated (see Figure 1(b)) or copied from input passages (Figure 1(c)) and explore whether explicit planning has any bearing on citation quality. Our contributions can be summarized as follows:
• We develop automatic methods to annotate training data with plans and citations, and finetune several Transformer models (Vaswani
et al., 2017) to generate attributed text.3
• Experimental results on the AQuAMuSe dataset (Kulkarni et al., 2020) demonstrate that plans consistently improve attribution quality. Furthermore, summary quality improves with an extractive blueprint model.
• Out-of-domain experiments on the ALCE benchmark (Gao et al., 2023) show that, once acquired, attribution is a robust skill (across information-seeking tasks). In terms of attribution quality, our models are competitive with (and sometimes better than) pipelines that heavily rely on large language models.

heading: 2 Related Work
text: LLMs have made significant advances in recent years in generating high quality natural language responses (Brown et al., 2020; Chowdhery et al., 2022). Despite impressive capabilities, they also demonstrate high levels of opacity: it is unclear where the information they generate comes from, which can undermine their trustworthiness (Cheng et al., 2022).
A variety of techniques aim to remedy this by building language models that provide references to support generated text. The majority of existing work has focused on models which learn to generate citations. For example, Nakano et al. (2021) and Menick et al. (2022) use reinforcement learning from human preferences to train language models to answer questions and provide supporting evidence (in the form of retrieved snippets). Bohnet et al. (2022) generate both an answer and a URL to a web page from which a paragraph is subsequently selected as support for the answer. They also experiment with attribution as a post-processing step where no learning is involved and use retrieval to select sources supporting their model’s output.
Along similar lines, Gao et al. (2022) propose a two-stage approach where generated text is first post-edited to be made attributable (to web content) and then revised accordingly. In addition to modeling, efforts have been also directed to the creation of evaluation protocols and benchmarks for assessing whether a statement is supported by provided evidence (Rashkin et al., 2021; Bohnet et al., 2022; Liu et al., 2023b).
Our work focuses on plan-based models which have been shown to be less opaque and more
3Our data and code are available from xxx.yyy.zzz.
controllable (Moryossef et al., 2019; Puduppully et al., 2019; Narayan et al., 2021, 2023; Huot et al., 2023b). We draw inspiration from Narayan et al. (2023) who formulate plans as a sequence of question-answer pairs. We develop various planbased models which can perform attribution, and quantify the extent to which the two mechanisms interact and whether they are mutually beneficial.

heading: 3 Problem Formulation
text: We follow a formulation of query-focused summarization common in the literature (Xu and Lapata 2020; Vig et al. 2022; inter alia). Let q denote an information-seeking request (e.g., “What is so special about the pulmonary arteries and veins?” in Figure 1) and P = {p1, . . . , pn} be a set of passages most relevant to q (see top left in Figure 1). We adopt a two-step approach where k most relevant passages are first retrieved and possibly reranked based on the query alone, and then fed to a secondary model which synthesizes the passages into a final summary S = {s1, . . . , sm} consisting of sentences si. An attributed summary further includes supporting evidence. For the sake of simplicity, we assume this is expressed by embedded in-line citations (see bottom left in Figure 1). Each si has a set of citations Ci = {ci,1, . . . , ci,k} where ci,j is a passage ID signifying that sentence i is citing said passage. For the summary (a) in Figure 1, C1 = {[1]}, and C2 = {[2]}.
In the following we first explain how to train a sequence-to-sequence model to generate citations, and then describe the proposed plan-based models.

heading: 3.1 Sequence-to-Sequence Model
text: Datasets for query-focused summarization such as AQuAMuSe (Kulkarni et al., 2020) or ASQA (Stelmakh et al., 2022) typically consist of queries Q, passages P , and target summaries S. A vanilla sequence-to-sequence model, generates summary S given passages P and query q, hence modeling the conditional probability p(S|P, q). In the case of attributed generation, the model is trained on summaries with citations assumed to be part of the summary’s token vocabulary. In Figure 1, the query, passages, and summary (a) with in-line references to passages [1] and [5] would constitute a training instance for this model.
In order to generate text with attribution, we must somehow annotate target summaries with citations. In experiments, we obtain silver-standard
citations, by measuring the entailment between passage pj ∈ P as the premise and sentence si ∈ S as the hypothesis. We annotate Si with ci,j for the passage pj with the highest entailment score (see Section 4.1 for more details).

heading: 3.2 Blueprint Models with Attribution
text: Narayan et al. (2023) formalize three types of blueprint models, which are purportedly suited to different generation tasks (e.g., long vs short output/input). We repurpose their end-to-end variant as it is a straightforward extension of the standard approach described above. Our model encodes query q and passages P , and then generates, B;S, the concatenation of blueprint B and output summary S in one decoding step. We define B as a set of questions B = {b1, . . . , bk} that are answered in S and act as a proxy for content selection and planning. This definition is more flexible (compared to question-answer pairs), as we do not require answers to be extracted from passages, and are not limited to a specific style of answers or questions such as those manifested in SQuAD (Rajpurkar et al., 2018). Questions can be more general while answers can be abstractive, and represent a variety of syntactic constructs beyond noun phrases (e.g., verbs, clauses, even sentences).
We assume the model has access to training data consisting of queries, passages, blueprints, and summaries (See Figure 1). Blueprints are not generally available, we explain how we obtain these automatically in the next section and define two blueprint models: an abstractive model (Figure 1(b)) and an extractive model (Figure 1(c)); these differ on whether the question-based plan is generated or copied from the input.
Abstractive Model We augment the training data with blueprints by generating questions for each target summary. We adopt an overgenerateand-rank strategy, generating multiple questions to be filtered based on their overlap with the summary.
A challenge with blueprint models is that the output may be answering a mix of general and specific questions. To avoid generating only one style of question, we employ two datasets originally developed for question answering, namely SQuAD (Rajpurkar et al., 2018) and Natural Questions (Kwiatkowski et al., 2019b). QA datasets typically consist of question/paragraph/answer triples but can be repurposed for question generation; to produce general-purpose questions, we assume the
paragraph is the input and the question is the output, while for more specific questions, we concatenate the paragraph with the sentence where the answer span is found and the model’s aim is to generate a question for the specific answer. We fine-tune T5-3B (Raffel et al., 2020) to obtain these two flavors of question generation models (see Appendix A for training details).
For each question generation model, we sample (using beam search) 10 questions per sentence and another 10 per summary so as to have a diverse question set. For each summary sentence, we then select the question with the highest lexical overlap from the union of questions (i.e., the questions specific to that sentence and the set of summary-level questions). Blueprint questions are sorted according to the order of appearance of their corresponding summary sentences. We add a special character in the blueprint (– in Figure 1) to delineate the questions corresponding to each sentence. We provide an example of how questions are generated and subsequently filtered in Appendix C.
Extractive Model In the extractive model, the questions align more closely with the input passages. Specifically, we explicitly verbalize the questions each passage might answer, i.e., Pext = {(p1,Q1), . . . , (pn,Qn)}, where pi are retrieved passages and Qi corresponding questions. Blueprint B then consists of a subset of questions whose answers are found in the input. The extractive model is functionally equivalent to the abstractive one, the only difference being that it encodes passages Pext (instead of P). This model also outputs B;S, however, the decoder learns to copy questions from the input in order to construct B. The extractive model has two advantages. Firstly, it reduces the risk of hallucinations and irrelevant information since the questions are directly copied from the input. Secondly, attribution comes for free, as we know which passage gave rise to which questions (see Figure 1). An obvious drawback is computational efficiency, as questions must be generated for every input passage.
We create a large number of questions for each passage using the same generation models described above. We then greedily select five questionsQi for each pi such that they have highest lexical overlap with the passage and minimal overlap with each other. Subsequently, we construct the extractive blueprints under the assumption that their questions ought to be aligned to the input passages
and output summary. We first discard questions that cannot be answered in the target summary using an answerability classifier, a T5 model trained on a repurposed version of SQuAD v2 (Rajpurkar et al., 2018) that predicts whether a question can be answered by a passage (see Appendix B for training details). The remaining questions are compared against the target summary and for each sentence we select a question based on lexical overlap. As in the abstractive model, a special character is added to mark which blueprint questions correspond to which sentences (see Appendix C for an example).
More formally, we summarize the abstractive and extractive models in Equations (1) and (2) respectively,
p(B;S | q, {p1, ..., pn}) (1) p(B;S | q, {(p1, Q1), ..., (pn, Qn)}) (2)
Note that Qi is generated only using pi so there is no dependency on the target data. While the target blueprint and its summary (B;S) have been created based on the input, the model is trained to generate one after the other.

heading: 4 AQuAMuSe Experiments

heading: 4.1 Experimental Setting
text: Data The bulk of our experiments use AQuAMuSe (Kulkarni et al., 2020), a long-form question answering dataset, which simulates how a search engine might synthesize documents of high relevance to a user query. It consists of queries taken from Natural Questions (Kwiatkowski et al., 2019a), passages from Common Crawl, and multisentence summaries from Wikipedia. We use the splits released in Kulkarni et al. (2020) which contain 6,599, 714, and 849 examples for training, development, and testing, respectively. The average query/response length is 9.2/107.3 words, and the number of input passages is 10.
All our models operate on the same passages which are reranked using T5-R (Huebscher et al., 2022), a T5 11B-based encoder trained with a classification loss on the MS MARCO dataset (Nguyen et al., 2016). T5-R reorders passages p ∈ P based on their relevance with query q. Target summaries in AQuAMuSe (Kulkarni et al., 2020) were annotated with in-line citations (see summaries in Figure 1) using an entailment classifier (T5-11B; Raffel et al. 2020) fine-tuned on the ANLI dataset (Nie et al., 2020); neutral and contradicting pairs
were classified as non-entailing (Honovich et al., 2022a). We consider citations part of the summary’s token vocabulary, but exclude them when evaluating summary quality.
Comparison Models We compare several models, examining how attribution and planning interact: a sequence-to-sequence baseline which does not have a plan or perform attribution (−Blueprint −Attribution); a model which generates citations without a blueprint (−Blueprint +Attribution; Section 3.1); its mirror image, i.e., a model with a blueprint (abstractive or extractive) but no attribution (+BlueprintA|E −Attribution); and a blueprint model with attribution (+BlueprintA|E +Attribution). In all cases, we fine-tune a LongT5 model (Guo et al. 2022, 3B parameters).
Evaluation Metrics We evaluate the quality of the summaries and for models that output citations whether these are correct. We measure the summary’s relevance using ROUGE-L4 (Lin, 2004) which computes the longest common subsequence between output summaries and reference text. We perform this comparison after removing any citations present in the response.
We quantify the extent to which generated summaries are faithful to their input using textual entailment (Maynez et al., 2020; Falke et al., 2019; Narayan et al., 2022; Honovich et al., 2022b). Our classifier is trained on the Adversarial NLI dataset (Nie et al., 2020); for each sentence in the summary, we check whether it is entailed by the input passages (and report the average across all sentences to obtain an overall score). Again, we remove citations from the output to measure faithfulness.
In addition, we evaluate citation quality by checking whether the citations mention the right passages. We extend AutoAIS (Bohnet et al., 2022), a recently proposed metric for automatically measuring the attribution of short answers (in a QA setting). For long-form responses, we take sentences with in-line citations and check whether these are entailed by all the passages they cite (Amplayo et al., 2023). We then report the proportion of entailed sentences. We use the same ANLI classifier (Nie et al., 2020) described above.
For blueprint models, we would additionally expect the plan to convey information relayed by the input passages. There is no point in introducing an intermediate representation if it does not facili-
4https://pypi.org/project/rouge-score/
tate grounding to the input. We measure blueprint quality, by leveraging an answerability classifier (see Appendix B for details) that checks whether a question is answered by a passage. We report the proportion of questions in the blueprint that are answered by the passages it references.5 Questions in the blueprint cite passages implicitly through their respective summary sentences. Recall that by construction there is a one-to-one correspondence between blueprint questions and summary sentences (which in turn cite input passages).

heading: 4.2 Results
text: The extractive blueprint performs best across evaluation metrics. Table 1 presents our results on AQuAMuSe when evaluating summary relevance and faithfulness. In the first block we compare models that do not generate citations. As can be seen plan-based models deliver better summaries in terms of ROUGE-L, with slightly enhanced faithfulness (see ANLI column),with the extractive blueprint model taking the lead. Our blueprint models are overall better compared to Narayan et al. (2023), achieving a new state of the art in the AQuAMuSe dataset. However, as mentioned earlier, the two approaches are not directly comparable as they differ in terms of the input they expect and blueprint definition. In the second block of Table 1 we compare models with an attribution mechanism. In general, faithfulness increases for plan-based models and relevance improves by
5Note that we are not simply measuring whether the question is answerable by any input passage, but rather if it is answerable by the cited passage.
10 points for the extractive blueprint model. In Table 2 we evaluate attribution quality for models which output citations. We observe gains in AutoAIS for our blueprint models over performing attribution without a plan. Attribution quality is slightly higher for the extractive model, which also creates more grounded blueprints, 97.97% of the questions are answered by the passages it references. We show example output in Appendix H.
The extractive blueprint generates the most abstractive summaries. We investigate the hypothesis that blueprint models might perform better simply because it is easier for them to copy sentences or passages from the input. In Figure 2 we evaluate the proportion of unique n-grams in the generated summary (compared to the input passages) for different n-gram sizes (where n = {3, 5, 10, 20, 40, 80}) for the +Attribution models in Table 1. As can be seen, the extractive model generates the most abstractive responses, while the−Blueprint baseline copies longer chunks of text from the input. This suggests that blueprints help models consolidate information from multiple sources, while still being faithful to the input.
Blueprint plans allow us to define different citation formats. Our experiments have so far adopted in-line citations as a common format for all models. We now explore alternatives which are unique to our plan-based models and examine whether these have any bearing on performance. Specifically, we (a) augment in-line passage citations with references to blueprint questions; (b) place in-line citations in the blueprint (after each question) rather than in the output summary; and (c) have implicit citations for extractive models only; in this case, we do not generate any citations but attribution can be inferred on the basis of the questions being aligned to summary sentences and copied from the input passages. Figure 3 shows
examples and definitions of these formats. Table 3 reports results on plan quality (checking whether the questions can be answered by the cited passages) and citation correctness (AutoAIS). We see that formats assuming an explicit connection between passages and questions improve answerability, however, at the expense of AutoAIS. Overall, in-line citations strike the best balance between answerability and faithfulness. We further examine the quality of blueprint plans in Appendix D.
Human evaluation corroborates automatic attribution results. In addition to automatic evaluation, we conducted a human study to assess the attribution quality of our models. Participants were asked (see Appendix E for details on our instructions and study) to verify whether the citations in the generated summaries are correct, classifying each sentence as fully supported (FSupp) by the referenced passages, partially supported (PSupp), unsupported (NSupp), or contradictory (Contra). We elicited judgments on 50 instances (randomly sampled from the AQuaMuse test set) for the systems shown in Table 4 (first block). We find that the extractive system is perceived as the most accurate among attribution models achieving the highest proportion of correct citations.

heading: 5 ALCE Experiments

heading: 5.1 Experimental Setting
text: Data We further examine whether the attribution capabilities of our models transfer to other domains, by performing (zero-shot) experiments on a recently collated benchmark called ALCE (Gao et al., 2023). ALCE contains evaluation sets from
ASQA (Stelmakh et al., 2022), which focuses on ambiguous queries and long-form answers, ELI5 (Fan et al., 2019), which focuses on how/why/what questions that require explanations, and QAMPARI (Rubin et al., 2022) where the answers are lists of entities. These datasets were selected such that the answers are factual, long-form, and require information from multiple sources. In experiments, we use the passages provided by the ALCE benchmark with no further reranking. Details on ALCE and representative examples are in Appendix G.
Comparison Models Gao et al. (2023) use ALCE to examine the citation abilities of LLMs. They report experiments with two base LLMs, namely ChatGPT (gpt-3.5- turbo-0301) and LLaMA (Touvron et al., 2023), both of which were prepended with detailed instructions and a few demonstrations for in-context learning. Their ChatGPT pipelines include (a) Vanilla, an end-toend model where the input is the query and top-5 passages; (b) Summarization condenses the passages first with a separate LLM and thus can process more of them; (c) Snippet extracts snippets from passages using an LLM; (d) Inline Search, where instead of providing passages in the input, the LLM calls search in-line whenever needed; and (e) Closed Book, where no passages are provided and attribution happens post-hoc (i.e., for each sentence, find the best passage to cite). LLaMAbased pipelines are versions of the Vanilla approach described above with either (f) LLaMA-13B, or (g) Vicuna-13B (Chiang et al., 2023).6 We compare our abstractive model (+BlueprintA+Attribution) against these pipelines. We assume that a question generation model is not available at transfer time to obtain questions for input passages (and create blueprints for the extractive model).
Evaluation Metrics We follow Gao et al. (2023) and evaluate system output in terms of correctness and attribution. Correctness is measured differently for each dataset; for ASQA, they calculate the recall of correct short answers (Stelmakh et al., 2022); for ELI5, they use InstructGPT (text-davinci-003; Ouyang et al., 2022) to generate sub-claims from the target response, and an NLI model (TRUE; Honovich et al., 2022a) to predict whether the response entails each sub-claim; for QAMPARI, they calculate precision and recall by checking the exact match to the gold-standard list of answers, considering recall to be 100% if the prediction includes at least five correct answers (Rubin et al., 2022); here, we only use recall to measure correctness since precision is calculated over a list of predicted answers and our models output one long answer (given the AQuAMuSe training). Attribution is evaluated using citation recall, which determines if the output is entirely supported by cited passages, and citation precision, which identifies any irrelevant citations. We report
6We did not include systems that additionally use response reranking and interaction strategies as these are orthogonal improvements and do not consistently improve performance.
F1 for metrics that have both precision and recall.

heading: 5.2 Results
text: The abstractive blueprint competes with LLMs in attribution. Our results are summarized in Table 5. On average, we observe that our model is better than competing systems in terms of attribution. We suspect this is due to the fact that it has been explicitly fine-tuned with outputs that have citations, which in turn improves attribution, rendering the model robust across datasets. In terms of correctness, our model performs on average better than LLaMA-13B but worse than the other systems, which is not surprising as it is applied to new datasets without access to in-domain training. In contrast, comparison LLM pipelines have been exposed to few-shot demonstrations.
Humans find Blueprint citations more accurate compared to those produced by LLMs. To further validate the results in Table 5, we conducted human evaluation comparing ChatGPT Vanilla, Vicuna-13B, and our abstractive blueprint model. Using the same instructions from Section 4.2, we collected judgments over 100 randomly sampled instances (50 from ELI5 and 50 from ASQA).7 As shown in Table 4, our model is perceived to have the most accurate citations, significantly outperforming both Vicuna-13B and ChatGPT Vanilla.

heading: 6 Conclusion
text: In this paper we focused on retrieval augmented generation with citations. We explored the attribution capabilities of plan-based models and proposed different variants depending on how the plans are created (i.e., abstractively or extractively). Our experiments revealed several interesting find-
7For Gao’s (2023) systems, we use the output from https://github.com/princeton-nlp/ALCE/tree/main/ human_eval. Appendix F shows results for each dataset.
ings. Summary quality improves for blueprint models when these learn to generate citations, both in terms of relevance and faithfulness. Conversely, this is not the case for models without an intermediate planning stage, in fact attribution slightly hurts performance (see second line in Table 1). The attribution mechanism is also important; formulating the attribution as in-line citations seems beneficial as well as a tight alignment between the input and the blueprint questions. Finally, attribution is a transferable skill for blueprint models, across datasets and information-seeking tasks.
Limitations
In this work, we have focused exclusively on encoder-decoder models. Future work could explore fine-tuning (on attribution annotations) a wider range of models, including decoder-only ones. In the future we would further like to explore more efficient ways of indexing passages with questions, as well as using the questions as an additional training signal for improving information retrieval. Finally, an important consideration with generative models is the problem of misinformation. While the work we present here makes a step towards improving the faithfulness and factual consistency of summarization systems, it is important to note that our models still make mistakes, in particular in out-of-domain scenarios.

heading: Acknowledgments
text: We thank the anonymous reviewers for their feedback. We are grateful to Hannah Rashkin, Slav Petrov, and Dipanjan Das for their comments on earlier versions of this paper.

heading: A Question Generation Implementation
text: Two question generation models were used to obtain questions for input passages and blueprints: one model was trained on SQuAD v2 (Rajpurkar et al., 2018) and another one on Natural Questions (Kwiatkowski et al., 2019b). Both question generators were built on top of a T5 3B (Raffel et al., 2020) base model, and expect either of the following input formats:
1. General-Purpose QG, where the model generates a question that can be answered anywhere in the passage:
Generate question »> [PASSAGE]
2. Sentence-Specific QG, where the model generates a question specific to a sentence in the passage:
Generate question »> [PASSAGE] » [SENTENCE]
where [PASSAGE] and [SENTENCE] are the passage and the sentence where the answer to the question (to-be generated) is found.

heading: B Answerability Classifier Implementation
text: Our answerability classifier repurposes the SQuAD v2 (Rajpurkar et al., 2018) dataset that includes unanswerable questions. Specifically, a T5 11B model is fine-tuned to accept as input a question and a passage in the following format:
question: [question] context: [passage]
and to generate Yes as output in cases where [question] can be answered given [passage] as context and No otherwise.

heading: C Blueprint Selection Examples
text: C.1 Abstractive Blueprints Table 6 shows a small-scale example of how questions are selected in abstractive blueprints. Given the summary in the example, we generate three SQuAD-style questions (marked with subscript Sq) and three NQ-style questions (marked with subscript N ) for each summary sentence. In addition, we generate three SQuAD and three NQ summary-level questions, giving us a total of 24
questions. Note that in our experiments, we use 10 SQuAD/NQ questions instead of three. For each sentence, we then select the question with the highest lexical overlap from the union of the sentence-specific and summary-level questions. In our example, the question for S1 is selected from the summary-level set. Finally, we collate these questions together with diacritic – to create the blueprint.
C.2 Extractive Blueprints
Table 7 shows a small-scale example of how questions are selected in extractive blueprints. For each of the five input passages (10 in our experiments), we generate the same amount of questions using the method described above. That is, for each passage we generate sentence-level and summary-level questions (similar to what is shown in Table 6) using both the SQuAD and the NQ question generation models. We then greedily select five questions for each passage, such that they have the highest lexical overlap with the passage and minimal overlap with each other. Once we have the passage questions, we are ready to select the questions in the blueprint. We first filter out questions that cannot be answered using an answerability classifier (see Appendix B and AF column in the table). We then select, for each sentence in the summary, the question from with the highest lexical overlap(SS column in the table). Finally, we collated these questions together with diacritic – to create the blueprint.

heading: D Grounding and Controllability
text: We further examine whether the planning aspect of the blueprint models remains intact when they are also expected to generate summaries with references to the input. We measure the extent to which the output is grounded to the blueprint plan using the answerability classifier. Specifically, we check if the questions in the blueprint are answerable by the summary as a whole and by individual sentences (and report the average). Table 9 shows our results for blueprint models with and without attribution. As can be seen, extractive blueprint models are less grounded compared to their abstractive counterparts, even though the former are better at attribution (see Table 2).
We further illustrate how to improve the attribution quality of abstractive models, thereby showcasing the controllability aspect of blueprint models.
We simply remove (post-hoc) generated blueprint questions that are unanswerable based on the input passages, and then generate the summary using the filtered blueprint. This post-processing substantially improves attribution by 7.09 AutoAIS points (compared to Table 2) while retaining or marginally improving output quality on other dimensions (63.12 ROUGE-L, 89.45 ANLI, 93.08, Answerability, and 81.25 AutoAIS).

heading: E Experimental Instructions
text: Figure 4 presents the experimental instructions used in our human elicitation study. To ensure high quality annotations, we created a screener test to determine the raters’ suitability for the task. We also limited the maximum tasks per annotator per study to six to avoid annotation fatigue. A total of 108 annotators participated in the elicitation study for AQuAMuSe, and a total of 88 and 249 annotators participated in the studies for ASQA and ELI5, respectively. The annotator agreement scores are as follows: 69.87% on AQuAMuSe, 93.00% on ASQA, and 89.04% on ELI5. Our annotators were paid adequately by our suppliers adhering to the supplier code of conduct.

heading: F Human Evaluation Results on ASQA and ELI5
text: Table 10 reports our human evaluation results for ASQA and ELI5, respectively. Results mostly align with the combined results in Table 4, showing that our model significantly produces more accurate citations.

heading: G The ALCE Benchmark
text: ALCE (Gao et al., 2023) is a collection of datasets aimed at evaluating LLM citation capabilities. They contain factual questions which require longform answers aggregating information over multiple sources (i.e., 100-word passages). We summarize various statistics on these datasets in Table 8 and describe them below.
ASQA (Stelmakh et al., 2022) focuses on ambiguous questions which have multiple interpretations (see Table 11 for an example). The questions were taken from AmbigQA (Min et al., 2020), while long-form answers were crowdsourced by synthesizing information from multiple documents. For ALCE (Gao et al., 2023), ASQA questions were additionally paired with Wikipedia passages
(2018-12-20 snapshot) which purportedly contained the answers.
QAMPARI (Rubin et al., 2022) is an open-domain QA dataset where answers are lists of entities, drawn from different passages. All questions in QAMPARI have at least 5 answers, with an average of 13 answers. Multi-answer questions were automatically generated using manually defined templates, answers were collated from Wikipedia, and examples were verified and paraphrased by crowdworkers (see Table 11 for an example). For ALCE (Gao et al., 2023), QAMPARI questions were also paired with Wikipedia passages (2018-12-20 snapshot).
ELI5 (Fan et al., 2019) mostly constists of how/why/what questions that require in-depth long answers and multiple passages as evidence. Questions (and answers) were elicited from the subreddit Explain Like I’m Five (ELI5) where users are encouraged to provide answers which are compre-
hensible by a five year old (see Table 11). For ALCE (Gao et al., 2023), ELI5 questions were paired with passages from Sphere (Piktus et al., 2021), a filtered version of Common Crawl.
ALCE (Gao et al., 2023) contains 1,000 randomly selected examples from each dataset (development set). It does not provide training data as it is aimed at assessing the citation capabilities of LLMs.

heading: H Example Output
text: Tables 12, 14, and 16 present output examples for the baseline system (−Blueprint +Attribution) and our two blueprint variants (+BlueprintA|E +Attribution). Blueprints created by the plan-based systems are respectively shown in Tables 13, 15, and 17. In general, we observe that the baseline system considers a smaller number of input passages when generating its response, while plan-based summaries contain more diverse passage references. As far as the blueprints are concerned, we find that the abstractive system generates (on average) more questions compared to the extractive model and the reference. Although the questions making up the blueprints of the two models are not identical, we see that they cover similar topics. For instance, in Table 13 the blueprints emphasize topics like the mean temperatures of a tropical monsoon climate, its precipitation, dry seasons, and variance in temperature.

heading: Future Work
text: Future work could explore fine-tuning (on attribution annotations) a wider range of models, including decoder-only ones. In the future we would further like to explore more efficient ways of indexing passages with questions, as well as using the questions as an additional training signal for improving information retrieval. Finally, an important consideration with generative models is the problem of misinformation. While the work we present here makes a step towards improving the faithfulness and factual consistency of summarization systems, it is important to note that our models still make mistakes, in particular in out-of-domain scenarios