Locally-attributable Grounded Text Generation (2024)

Abstract: Recent efforts to address hallucinations in Large Language Models (LLMs) have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. Yet, these citations often point to entire documents or paragraphs, burdening users with extensive verification work. In this paper, we introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named “Attribute First, then Generate”, breaks down the conventional end-to-end generation process into three intuitive steps: content selection, sentence planning, and sequential sentence generation. By initially identifying relevant source segments (“select first”) and then conditioning the generation process on them (“then generate”), we ensure these segments also act as the output’s fine-grained attributions (“select” becomes “attribute”). Tested on Multi-document Summarization and Long-form Question-answering, our method not only yields more concise citations than the baselines but also maintains—and in some cases enhances—both generation quality and attribution accuracy. Furthermore, it significantly reduces the time required for fact verification by human assessors.1

heading: 1 Introduction
text: Grounded text generation, which includes tasks like summarization and question-answering, aims to produce content that is derived from specific sources, either user-provided or retrieved via retrieval mechanisms (e.g., RAG; Lewis et al., 2020). To facilitate verification of models’ adherence to these sources, recent years have seen a growing interest in attributed text generation (Bohnet et al.,
* Equal contribution. 1Our code is publicly available at https://github.com/
lovodkin93/attribute-first-then-generate
2023), which aims to create text alongside supporting evidence. This method enhances models’ credibility by enabling factuality verification. It also aids in detecting and addressing factual errors, a critical need given the observed frequency of such errors, termed “hallucinations”, in model outputs compared to source texts (Mishra et al., 2024a).
While these attributions are aimed to facilitate factuality evaluation by focusing people’s attention on relevant supporting texts, current approaches often yield rather coarse attributions, pointing back to whole documents or paragraphs. Such attributions, though better than having none, require human assessors to exhaustively sift through many irrelevant details in the cited content, resulting in a time-consuming and somewhat ineffective factchecking process. Can we do better?
In this work, we extend attributed text generation to also consider attribution conciseness, reformalizing the task as Locally-attributable Text Generation (see Section 2). We introduce a criterion for attributions to be precise, targeting only the most relevant text snippets, from specific sentences down to subsentence spans, while avoiding non-essential details. This criterion complements the full coverage requirement, which stipulates that every generated fact be backed by a cited snippet. As illustrated in Fig. 1, given a query and several documents, the model output includes citations to relevant spans within the source documents for each generated sentence instead of broadly attributing the entire documents (Gao et al., 2023b). These spans represent the minimal set of relevant source snippets needed to cover the content of the output sentences.
Following our localization criterion, we propose a novel attribution-driven generation approach, aimed to ensure both conciseness and full coverage, termed “Attribute First, then Generate”. Rather than jointly producing text with citations (Gao et al., 2023b) or getting the attribution post-generation (Bohnet et al., 2023), we
3309
Documents
Prompt:
propose to first choose relevant details from the source text(s), and then focus the entire generation process on them. Specifically, we decompose the conventional end-to-end grounded generation approach into three intuitive steps, as depicted in Figure 2: (1) content selection (choosing relevant details from source texts), (2) sentence-level planning (organizing and grouping content for sentence fusion), and (3) sentence-by-sentence generation (based on the selected and structured content). Notably, by dictating the generation process, the initially selected details de facto serve as attributions. This ensures a closer attribution adherence to the generated output, as the entire generation process is now guided by these attributions. It also offers more precise supporting evidence, in the form of these selected details, thereby fulfilling our objectives of attribution accuracy and conciseness.
Our method, adaptable to various grounded generation tasks, requires only slight adaptations in the content selection step to cater to specific lexical needs, e.g., salience in summarization or query relevance in question-answering. We showcase its effectiveness in both Multi-document Summarization (MDS) and Long-form Question-answering (LFQA), exploring both finetuning and few-shot prompting strategies. Evaluations across automatic and human assessments reveal that our strategy not
only meets but in some cases surpasses existing baselines in generation quality while also achieving high attribution accuracy. Importantly, it also leads to significantly shorter citations, thereby reducing the effort needed for fact-checking by over 50%.
Altogether, our work is the first to propose localized attributions and to develop a method geared to provide such concise citations. This level of granularity in attribution enhances the efficiency of the fact-verification process, by focusing users’ attention on the most relevant details. We thus suggest that future research will build upon our paradigm to further advance locally-attributed generation.

heading: 2 Locally-attributable Grounded Text Generation
text: The task of Locally-attributable Grounded Text Generation involves processing a given set of documents D to produce an output text y. The set of documents may either be provided by the output of a retrieval system or by the user. The output, consisting of sentences s1, . . . , sn, should both fulfill the base underlying task (e.g., summarizing for MDS or generating a response for LFQA) and also include attributions C1, . . . , Cn pointing back to supporting text from D. For these attributions to effectively facilitate factuality verification by human
2
readers, it is vital that they support as much of the output as possible, while also achieving the highest granularity in both the input and output.
Concretely, output granularity entails assigning the smallest information units in the generated text y—in this work, individual sentences si—with a corresponding attribution set Ci, consisting of one or more supporting citations cji ∈ Ci. Simultaneously, input granularity requires each citation cji to point to a specific span of text hk ∈ D, possibly non-consecutive. Importantly, these spans should be as concise as possible while ensuring completeness in attribution, i.e., that every piece of information in the output, which in our setting refers to each output sentence si, is fully backed by the union of its citations Ci. For example, in Fig. 1, Ci is depicted as squared brackets at the end of each sentence with pointers to spans hk, shown as colored highlights in D (we use “highlights” and “spans” interchangeably).

heading: 3 Modeling
text: This work aims to integrate source attribution with the content-grounded generation process, to achieve more targeted attribution. Our approach is based on uncovering the inherent decision-making mechanisms involved in text generation. We first describe our suggested Attribute First, then Generate scheme in Section 3.1. Then, we propose two strategies for this framework’s application: a prompt-based in-context learning approach (Section 3.2) and fine-tuning designated components (Section 3.3). For each approach, we provide a brief overview of our implementation strategy for the framework’s phases, while a more in-depth elaboration can be found in Appendix C.

heading: 3.1 Attribute First, then Generate
text: To reliably achieve the desirable fine-grained attribution, we introduce the Attribute First, then Generate paradigm, outlined in Fig. 2. This method is structured around three key steps, mirroring the inherent decision-making processes involved in text generation: content selection, sentence-planning, and sentence-by-sentence generation. Though applicable to any grounded text generation task, we specifically test its application in Multi-document Summarization (MDS) and Long-form Questionanswering (LFQA), adapting only the content selection step to each task’s unique requirements.
Content Selection The approach begins with identifying relevant spans from the source that would contribute information to the generated output ((a) in Fig. 2). These spans then function as a more focused grounding content, guiding the lexical choices in the generation process. Notably, as these chosen segments constitute the primary information for the resulting text, they effectively serve as its (fine-grained) attribution.
Sentence Planning This step is designed to achieve sentence-level attribution in the output through the introduction of an intermediate sentence-level planning step ((b) in Fig. 2). Here, the selected spans from the previous step are organized into coherently ordered clusters, where the spans grouped within each cluster naturally fit within the same sentence. This step, which breaks down the constrained generation process into smaller, more manageable steps, serves two purposes: it simplifies the model’s task by conditioning each generation iteration on a single cluster rather than the entire set of selected spans, while also leading to the desired, sentence-level attribution in the output, in the form of the corresponding cluster’s highlights.
3
Sentence-by-sentence Generation In the final step ((c) in Fig. 2), the model executes the text generation according to the outlined plan, adopting a sentence-by-sentence approach to preserve the built-in attribution from the previous steps. Specifically, it uses the next set of highlights Ci+1 and all preceding sentences s1:i, to generate the next sentence si+1, by maximizing p(si+1|s1:i, Ci+1). This ensures each sentence expresses the content of the designated highlights while also integrating seamlessly with the preceding output.

heading: 3.2 In-context Learning
text: For each of the substeps delineated in the previous section, we propose a prompt-based in-context learning (ICL) strategy, where a model is guided with specific instructions and carefully selected few-shot examples. The content selection step ((a) in Fig. 2) is formulated as D → h1, . . . , hm, where D is the set of source documents and hi corresponds to a verbatim span from D. Namely, a model is tasked with outputting relevant spans from D, separated by a designated markup (see prompts in Figs. B.1 and B.5). In MDS, these spans are aimed to capture salient information, while for LFQA, the model, presented with a query, is instructed to identify spans answering that query. These spans are then located within the source documents via string-matching, with any undetectable span being omitted (for more details, see Appendix C.1). Notably, in all subsequent steps, we “highlight” the identified spans within the document set D via designated markups before and after each span, akin to the methodology used in prior work (Gehrmann et al., 2018; Slobodkin et al.,

heading: 2022, 2024).
text: For the sentence planning step ((b) in Fig. 2), the documents, now “highlighted”, are concatenated, truncating each at the final highlighted span to manage input length and focus the model’s attention. To highlight, we use two designated markups, <highlight_start> and <highlight_end>, which we insert into the text before and after each consecutive ‘highlighted’ span, respectively (see prompts in Figs. B.2 and B.6). The model is instructed to organize h1, . . . , hm into an ordered set of highlight clusters C1, . . . , Cn, where Ci = {hk}|Ci|k=1, to guide the subsequent generation phase. For more details, see Appendix C.2.
Finally, in step (c), the model is guided to generate the output one sentence at a time, adhering to the structured plan. Specifically, we separate
the generation process into a series of model calls, each focusing on a single cluster’s highlights, concatenated in the same manner as in step (b). The model is also presented with all previously generated sentences s1:i. The task is then structured as maximizing p(si+1|s1:i, Ci+1), guiding the model to produce the subsequent sentence si+1, considering both the current cluster of highlights Ci+1 and the previously-generated sentences s1:i.2 For more details, see Appendix C.3 as well as the full prompts structure in Figs. B.3 and B.7.
Chain-of-Thought. We also explore a variation that combines the sentence-planning and sentenceby-sentence generation stages ((b) and (c)), inspired by the Chain-of-Thought paradigm (CoT; Wei et al., 2022). Given the selected content from step (a), the model is tasked with generating sentence-level plans, deciding on highlights to include in each synthesized sentence, alongside producing these sentences (see prompts in Figs. B.4 and B.8). The model is guided to produce the final output, after forming the plan, focusing on unifying and refining sentences for coherence. For sentence-level attribution, we link each sentence in the output with the highlights from its planning stage in the CoT process.3 For more details, refer to Appendix C.3.

heading: 3.3 Fine-tuning
text: We also explore fine-tuning models for the various steps of our framework, aligning the inputoutput formulation of each subtask to its ICL counterpart in Section 3.2. During content selection ((a) in Fig. 2), our model is taught to copy taskspecific segments from D verbatim, and to separate them using a special markup (see Appendix C.1 and Figs. B.9 and B.12). To further ensure strict lexical fidelity to the source text, a constrained decoding approach is adopted during inference (see Appendix C.6 for more details).
In the sentence planning step ((b) in Fig. 2), we use special tokens to ‘highlight’ the previously selected spans, and teach our model to copy verbatim the “highlighted” spans in a clustered and ordered manner, using special tokens to separate between different clusters and different highlights within a cluster (see Appendix C.2 and Figs. B.10 and B.13 for more details). This phase similarly incorporates
2In Section 5.1 we show that conditioning on s1:i improves coherence.
3We use spaCy sentencizer (Honnibal and Montani, 2017).
4
a constrained decoding technique to maintain exact lexical matching to the highlighted content. Finally, for the sentence generation step ((c) in Fig. 2), we use special tokens to separate between the highlighted documents, the prefix, and for the LFQA setting the query (see Appendix C.3 and Figs. B.11 and B.14).
Joint Selection and Planning. We also explore a joint strategy that combines the content selection and plan generation stages ((a) and (b)). Here, the entire document set D serves as input, and the output is an ordered series of “highlight” clusters C1, . . . , Cn. To ensure exact copying of source spans, we employ the aforementioned constrained decoding strategy. For more details, see Appendix C.2.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: To construct the training, development, and test splits for the different components within our framework, we employ source-output alignment datasets that map corresponding spans between inputs and outputs. These alignments are adapted to suit the data requirements of each framework phase: for content selection, the alignments’ source-side spans serve as the target labels; in sentence planning, source spans corresponding to the same output sentence are clustered, with the order of output sentences determining the order of these clusters; for the sentence-by-sentence generation phase, the output sentences act as the target outputs for their respective source clusters.
For MDS, we use an alignment dataset derived from the DUC, TAC benchmarks4 and the MultiNews dataset (Fabbri et al., 2019), comprising of pairs of summaries, corresponding source articles and summary-source alignments from Ernst et al. (2022, 2024). For LFQA, we rely on humanannotated alignments of large language model responses with citations (Liu et al., 2023). By filtering responses that were rated as verifiable, we ensure high-quality alignment for this setting. Further details and dataset processing information are provided in Appendix A.

heading: 4.2 Models and Baselines
text: Our study investigates both ICL prompt-based LLMs and fine-tuned LMs.
4https://duc.nist.gov/
In-context Learning. For our ICL approach (Section 3.2), we utilize the Gemini model (Gemini Team Google, 2023), applying designated few-shot prompts tailored to each subtask.5 For each setting, we also implement two baseline models for comparative analysis. For the first baseline, we prompt Gemini to perform the overarching task, namely summarization for MDS and answering a query for LFQA, end-to-end (see Appendix C.4). This baseline is primarily used to ensure the generated text’s quality is not compromised in our framework. For the second baseline, we implement ALCE (Gao et al., 2023b), a prompt-based approach designed to enable the simultaneous generation of text and its attributions through in-line citations (e.g., [1] to cite source 1). See Appendix C.5 for more details.
Fine-tuned Models. For each of our proposed subtasks, we fine-tune models on our derived datasets (Section 3.3). We choose to use PRIMERA (Xiao et al., 2022), a Longformer Encoder-Decoder model (LED; Beltagy et al., 2020) optimized for MDS, for both of our MDS and LFQA settings. Our choice of model stems from PRIMERA’s leading performance in the MDS setting, attributed to its capability to handle long contexts and its pre-training technique. Further, we found that PRIMERA outperformed the Long-T5 large model (Long-T5Large; Guo et al., 2022) on our LFQA development set, where both models contain roughly the same number of parameters (Long-T5: 750M, PRIMERA: 447M). Analogous to our ICL approach, we also fine-tune a variant to perform the corresponding task comprehensively for both settings.

heading: 4.3 Evaluation
text: Following our task definition in Section 2, we aim to assess two aspects of the produced outputs. Firstly, we evaluate the generated texts’ quality, focusing on coherence and task fulfillment—specifically, summary quality in MDS tasks and answer relevance in LFQA, consistent with standard evaluation practices in each respective task (Fabbri et al., 2021b; Stelmakh et al., 2022). Secondly, we examine the attribution quality, looking at how well each sentence is supported by its accompanying citations and the conciseness of these citations (simply measured by the length of the cited texts). Our evaluation methodology combines automated metrics and human judgments.
5We used the Gemini API, which is free for academic purposes: https://ai.google.dev/models/gemini
5
Automatic Evaluation. To assess compliance with each task, we employ the ROUGEL (Lin, 2004) and BERTSCORE (Zhang et al., 2020) metrics. These compare generated texts with reference outputs, namely gold summaries for MDS and reference answers for LFQA.
For assessing citation conciseness, we report the average length (number of tokens) of the cited content per output sentence. To evaluate citation quality, we employ AUTOAIS, an NLI-based metric for attribution evaluation, in line with prior studies (Bohnet et al., 2023; Gao et al., 2023b) which have shown its correlation with human-judgment of attribution quality. Following those works, we employ the TRUE model from Honovich et al. (2022) for our NLI framework.6 The final AUTOAIS score is the average number of sentences predicted as attributed by the classifier.
Lastly, we also report the average number of non-attributed sentences per instance.7
Human Evaluation. The evaluation of generated attributed texts consists of two steps. First, anno-
6We also examined a more recent NLI model (TrueTeacher; Gekhman et al., 2023), and have found TRUE to correlate better with human judgment. See Appendix B for more details.
7Given the absence of reliable automatic metrics for fluency, we restrict its evaluation to the human evaluation, following standard practices (Fabbri et al., 2021a).
tators are asked to evaluate text coherence on a 5-point Likert scale, and in the LFQA setting also its helpfulness in addressing the query. Next, annotators proceed to evaluate the quality of citations. We follow the sentence-level attribution evaluation protocol from Gao et al. (2023a), which extends the Attributable to Identified Sources (AIS; Rashkin et al., 2022) human evaluation framework of attribution to the sentence-level. Annotators judge whether each output sentence is fully supported by its attribution, scoring 1 for full support and 0 otherwise. The final AIS score is the average of these scores across sentences. For more details, see Appendix G.

heading: 5 Results and Analyses
text: We provide results of our automated evaluation in Tables 1 and 2 and human evaluation in Tables 3 and 4. Considering the resource-intensive aspects of human evaluation, we limit this assessment to the ALCE baseline and our paradigm’s most efficient model, the ICL variant ATTR. FIRSTCoT, chosen for its optimal balance between task performance and attribution accuracy, as indicated by the automatic evaluation.
We first note that our approach, which extracts focused spans from the input documents, leads to
6
significantly shorter attributions than the ALCE baseline, where citations coarsely point to entire documents. For example, the ATTR. FIRSTCoT variant’s citations are on average 45 times shorter than ALCE’s.8 Importantly, we find that assessing our paradigm’s attribution takes annotators roughly half the time compared to those of ALCE, emphasizing the benefit of attribution localization in streamlining the manual fact-checking process.
In terms of attribution accuracy, our method surpasses the ALCE baseline in the LFQA domain (both on the automatic AUTOAIS and the human AIS metrics), while in MDS, although ALCE shows higher accuracy, it leads to twice as many unattributed sentences. We also note that while in the MDS setting our method may decrease attribution accuracy compared to ALCE’s full-document citations, readers can always fall back to examining the full document if the provided highlights are insufficiently supporting. Indeed, analyses of the human annotations revealed that around 42% of the 109 sentences deemed unsupported were partially supported, with the remainder of the attributing details frequently found in immediate proximity to the highlighted sections (usually within the same or adjacent sentences).9 This observation indicates that while our sentence fusion step occasionally includes slightly more information than the selected highlights (later used as fine attributions), it mostly adds details from the surrounding context.10
Hence, while suggesting room for improvement in strict adherence to highlights, this does not significantly hinder fact-checking effectiveness.
Importantly, the promising results on attribution do not compromise the quality of outputs. Our automatic analyses indicate minimal impact on
8See examples of fine-grained attributions generated by our approach in Appendix E.
9See partially-attributed examples in Appendix H. 10This trend is also observed in our automatic evaluation, where extending sub-sentence attributing spans to their corresponding full sentences improves AUTOAIS. See Appendix D.
summary quality in MDS and even an improvement in LFQA, as confirmed by the ROUGEL and BERTSCORE metrics. These results are further evident in our human evaluation, both in terms of the output’s fluency and (for LFQA) in helpfulness for answering the query. Notably, our in-context variants tend to produce higher-quality outputs than fine-tuned models. This apparent advantage, observed also in attribution accuracy, is likely due to Gemini’s stronger ability to generalize to longcontext settings, as well as its larger model size.
Overall, our experiments show that our approach leads to orders of magnitude shorter attributions, with little impact on the attribution accuracy and the output’s quality. We leave further improvements in fine-tuning the individual designated components to future research.

heading: 5.1 Do Models Benefit from Seeing the Prefix?
text: We also examine whether providing models with the context of previously generated sentences indeed aids the generation process, particularly in enhancing output cohesion as defined by Maimon and Tsarfaty (2023). Cohesion involves sentences being “referentially linked”, e.g., via co-referring mentions, or linked through discourse connectors. To analyze the impact of the prefix on cohesion, we modified the sentence generation prompt to exclude the prefix and conducted the sentence fusion step (step (c) in Fig. 2) using the ICL variant for both MDS and LFQA tasks, maintaining identical
7
sentence-level plans. Starting from the second sentence to isolate the impact of the prefix, we examined 20 instances per task. Our analysis of 113 sentences reveals that approximately 30% were more cohesive when including the prefix. Specifically, 23% showed improved referential linkage, mostly attributed to better use of co-reference, and 7% demonstrated better discourse connector usage.11
Additionally, our findings suggest that prefixes play an increasing role in cohesion as the output progresses (see Fig. 3), implying that cohesion gains importance as the text develops.12

heading: 6 Related Work
text: A few recent works have proposed several methods to generate text accompanied by attribution, with varying levels of granularity in the source and the generated text. In Long-Form Question Answering (LFQA), the LaMDA system (Thoppilan et al., 2022) provides attribution for the entire response in the form of URLs pointing to entire documents. In GopherCite (Menick et al., 2022), the entire response is attributed to fine-grained snippets from the documents. ALCE (Gao et al., 2023b) provides attribution for each generated sentence to one or multiple entire input documents. SEMQA (Schuster et al., 2023) is a recent multisource semi-extractive dataset for LFQA, which encourages models to integrate factual spans copied verbatim from the input, along with free-text connectors to ensure coherence. This method is by design fine-grained both at the source and the response level using string matching. Yet, this semiextractive paradigm is limited in scenarios requiring a higher level of abstraction, such as Multidocument Summarization (MDS). In contrast to all those approaches, our localized attribution achieves granularity both in the input and the output, without restricting the model to extractive text generation.
In information seeking, recent works have proposed to find attribution in post-hoc, by retrieving documents and matching each part of the answer to specific pieces of text from the retrieved documents (Bohnet et al., 2023; Gao et al., 2023a). These methods face a combinatorial challenge because the relevant snippets of text in the source documents need to be determined.
The field of attributed text generation is part of a
11See Appendix I for examples. 12The observed plateau in cohesion levels in LFQA is attributed to its outputs generally not exceeding three sentences.
broader endeavor to address hallucinations in models’ outputs and increase their overall trustworthiness. In this context, much work has been done on characterizing the sources of hallucinations (Dziri et al., 2022; Rawte et al., 2023; Yao et al., 2023; Li et al., 2024) as well as on strategies to identify and mitigate them (Azaria and Mitchell, 2023; Manakul et al., 2023; Min et al., 2023; Slobodkin et al., 2023b; Mishra et al., 2024b), thereby advancing the reliability of model-generated content.
Another related line of research focuses on textplanning for improving output quality and reducing hallucinations (Gehrmann et al., 2018; Narayan et al., 2021, 2023; Zhao et al., 2020), by introducing an additional step of planning before the final text generation. This area of study includes efforts to decompose the generation process into distinct phases, with some works focusing on the contentselection step (Gehrmann et al., 2018; Cho et al., 2019; Ernst et al., 2021), while others focus on the content consolidation phase (Lebanoff et al., 2019, 2020; Slobodkin et al., 2022, 2023a; Hirsch et al., 2023). Our work also leverages this planning paradigm to improve the factuality of the output. However, unlike previous methods, our pipeline is carefully designed to also provide high-quality finegrained attributions. Perhaps most related to our method is the work by Ernst et al. (2022) that divides the MDS process into detecting salient propositions, clustering them, and transforming them into distinct sentences. However, their method results in outputs that resemble bullet-point summaries, consisting of single-proposition sentences, rather than producing more fluid, multi-proposition sentences typical of natural text. In contrast, our method maintains the coherence and natural flow of generated texts without being restricted to singleproposition sentences.
Text-planning is also related to the Chain-ofThought (CoT) approach that demonstrated improved performance on math problems (Wei et al., 2022), coding and programming puzzles (Chen et al., 2021; Schuster et al., 2021), and other reasoning tasks (Saparov and He, 2023). Both approaches build on the idea of decomposing the generation task to allow an initial reasoning and planning stage, before generating the predicted output. In CoT, the planning and output prediction are typically performed in a single generation call, whereas in text-planning the planning stage is separated and could be performed either by the same LM or using different models and operations. Therefore, CoT
8
is more suitable for in-context learning whereas text-planning works well with finetuned specialized models. We explore both options in this work.

heading: 7 Conclusion
text: In this work, we present “Attribute First, then Generate”, a novel scheme for attributable grounded text generation, comprised of separating the generation process into multiple steps, such that attribution is achieved as a by-product of the generation. To the best of our knowledge, we are the first to provide such fine-grained attributions in both the generated text and the references. Through automatic and manual evaluations, we showed that our decomposed generation pipeline either matches or outperforms existing baselines in terms of taskspecific metrics. Importantly, our approach yields orders of magnitude shorter attributions, shortening the fact-checking time by almost 50%, with little impact on the attribution accuracy. While this work focused on a particular implementation of content selection and generation, we encourage future explorations to extend our “Attribute First, then Generate” paradigm to other configurations. We hope that our research will motivate future work in this direction, improving the quality of text generation while also providing helpful concise attributions to external sources, thereby increasing models’ trustworthiness and safe downstream use.
Limitations
Our Attribute First, Then Generate approach tends to use more computing resources and is also slower compared to direct end-to-end generation, as detailed in Appendix J. This is because it requires several steps, each involving a separate call to the model.
Furthermore, our pipeline is comprised of several interlinked components, which can potentially lead to error propagation. However, this potential issue is partially alleviated by the inherent robustness of the individual components. For instance, should the content selection mechanism identify a broad array of text fragments, including some that are irrelevant, subsequent stages in the process are designed to filter out such non-essential information.
Ethics Statement
Crowdsourcing. For our human evaluation we used the Amazon Mechanical Turk13 (MTurk) crowdsourcing platform. Recruitment of annotators was conducted via email invitations dispatched from the MTurk platform, aimed at individuals who had previously demonstrated high performance in similar research endeavors. Compensation was provided not only for the actual annotation tasks but also for an initial onboarding phase, which included familiarization with the annotation guidelines and going over feedback from the authors. The pricing for each annotation instance was calibrated based on an estimated completion time, aiming to maintain a compensation rate of approximately 12$ per hour. Additionally, we closely monitored the actual time required for task completion and offered posthoc reimbursements in compensation for instances where the required time exceeded our initial estimate by more than 5%.
Datasets licenses. The DUC and TAC datasets were acquired according to the required NIST guidelines (duc.nist.gov), and we do not publish any data from these datasets. The annotation data from Liu et al. (2023) is published with the MIT license. The Multi-News dataset (Fabbri et al., 2019) is published with a non-commercial license intended for research and educational purposes. Accordingly, our human evaluation annotations will also be limited to non-commercial use.
AI assistants AI assistants were exclusively employed to enhance the grammatical structuring of the text.

heading: Acknowledgements
text: This work was supported by the Israel Science Foundation (grant no. 2827/21).

heading: A Datasets Preprocessing
text: Datasets statistics are reported in Table A.1. For LFQA, we utilize published human annotations of LLM responses with in-line citations (Liu et al., 2023). We use the existing train, dev, and test splits, and we apply a few filters on this dataset as follows. Responses from LLMs can contain statements that are not supported by the retrieved documents and
12
are annotated with “Partial Support” or “No Support”. We classify each statement as supported by the input documents if either one of its citations is annotated with “Full Support” or if the union of all citations is annotated with “Full Support”. We remove examples from the dataset containing unsupported statements. In addition, the dataset also includes human annotations for perceived utility (on a Likert scale between 1 to 5). We only keep examples with perceived utility 4 and 5 (“Strongly Agree” and “Agree”). Lastly, the dataset contains evidence excerpts for each generated statement, which we use as citations. We remove examples from the dataset if we cannot align the evidence excerpt with the source documents.
For the training processes in the MDS setting, we use the alignment dataset from Ernst et al. (2022), automatically extracted from the DUC and TAC summarization datasets.14 For the development and test splits, we use the alignment dataset from Ernst et al. (2024), extracted from the Multi-News summarization dataset (Fabbri et al., 2019). The dataset consists of 100 summaries with a corresponding set of source news articles, comprising an average of 3 articles per summary. Additionally, for each such pair of summaries and set of documents, the authors trained qualified annotators to identify aligning spans, following the approach delineated in Slobodkin et al. (2022).
14https://duc.nist.gov/

heading: B Meta-Evaluation for AUTOAIS
text: Prior work that suggested using NLI models for automatic attribution evaluation (Bohnet et al., 2023; Gao et al., 2023b,a) has specifically advocated using the TRUE model (Honovich et al., 2022). Subsequent developments have introduced an enhanced NLI model, named TrueTeacher (Gekhman et al., 2023). In our study, to determine the most appropriate NLI model for the automatic assessment of attribution within the AUTOAIS framework, we conduct a correlation analysis between the outcomes of these models and our human-generated annotations. We also evaluate an alternative method that averages the models’ probabilities for the entailing label (’1’) across all sentences, rather than binary outcomes. To calculate correlation, we use both Kendall-Tau and Spearman’s rank correlations, as suggested in (Deutsch et al., 2022). We also apply bootstrapping (Efron, 1987) by performing 1000 samplings of 150 instances (with repetition) and calculating correlation scores for each such subset. We report the average correlation and 95% confidence intervals for each metric. Our findings, presented in Table A.2, indicate that despite the advancements offered by the TrueTeacher model, the TRUE model still aligns more closely with human judgment. Furthermore, the approach of averaging probabilities for the label ’1’ correlates significantly worse for both models. Consequently, we opt to continue utilizing the TRUE model for our AUTOAIS evaluations.
C Implementation Details
In this section, we describe the implementation details for all the component described in Section 3, both the ICL and fine-tuning approaches. ROUGEL and BERTSCORE were calculated with the evaluate library 15 (Lin, 2004).
For our ICL experiments, we use the GeminiPro model (Gemini Team Google, 2023), last updated on December 2023, which is available via API calls.16 In all LFQA experiments, the query is included as part of the input for each component. For all subtasks and baselines, we perform a grid search on the optimal number of in-context examples (1-3 in the MDS setting and 1-4 in the LFQA setting) and temperature hyperparameter (0.1,0.3,0.5,0.7,0.9) by evaluating each combination on the respective development set. For all
15https://huggingface.co/docs/evaluate/en/index 16https://ai.google.dev/models/gemini
13
fine-tuned variants, we fine-tune PRIMERA for both tasks. We also tested training LONGT5, but found PRIMERA performs significantly better. The number of parameters in the PRIMERA model is 447M. All fine-tuned models were trained for 10 epochs and the best model checkpoint was chosen by ROUGEL, unless mentioned otherwise. The training of each model took approximately 1 hour on one Nvidia A100 80GB GPU. Hyperparameter training included 3 epochs for each model and used the following sweep configuration: learning rate (max: 5e-4; min: 5e-8), warmup steps (max: 300; min: 0), weight decay (0.0, 0.2, 0.5).
C.1 Content Selection
ICL In the ICL method, we present the model with the source texts and guide it to copy specific segments from those texts (see Figs. B.1 and B.5 for prompt examples in the MDS and LFQA settings, respectively). We use a designated
“<SPAN_DELIM>” markup as a delimiter between such consecutive spans, for better parsing. Within the MDS framework, the model is tasked with identifying salient excerpts, whereas in the context of LFQA, the model is also provided with a query, and is asked to find the parts of the texts that are relevant to answering this question. Further, following our analyses of each setting’s development sets, we restrict the model to an average and maximum number of words to choose (an average of 200 and 100 words and a maximum of 900 and 200 words for MDS and LFQA, respectively). Lastly, to manage source texts that exceed the model’s input size limit, we truncate the fewest necessary sentences from those texts to ensure they fit within the specified limit.17 After performing hyperparameter tuning, in the MDS we set T = 0.1 and the number of in-context examples to 2, while in LFQA we set T = 0.3 and the number of examples to 4.
Fine-tuned Examples are provided in Figs. B.9 and B.12. The input to the model is the set of all documents, concatenated with a special markup “<doc-sep>”, which was also used in pre-training in PRIMERA. To fit the input in a context length of 4096, we trim each document to the length of 4096 divided by the number of documents in each example. The output is a set of highlights separated with a new special markup “<highlightsseparator>”. We run hyperparameter-tuning on
17We use spaCy (Honnibal and Montani, 2017) to separate documents into sentences.
each setting based on the development set. The hyperparameters used for MDS are warmup steps of 229, weight decay of 0.5, and a learning rate of 4e-4. Conversely, the hyperparameters used for LFQA are warmup steps of 234, weight decay of 0, and a learning rate of 1e-4. We chose the best model checkpoint by calculating Intersection-overUnion (IoU) between the gold highlights spans and the predicted highlights spans.
C.2 Generation Planning
ICL In the ICL setting, we present the model with the source text where we insert the mark-ups
“<highlight_start>” and “<highlight_end>” before and after each pre-selected span, (see Figs. B.2 and B.6 for the MDS and LFQA settings, respectively). We also enumerate the highlights across all documents, and for the LFQA setting, we also add the query. The model is directed to cluster highlighted segments that are likely to co-occur in a sentence in the same order it would then merge them into an output. We discovered that providing the model with task-specific motivation enhances its performance; for instance, in MDS, the objective is to fuse highlights into a summary, whereas in LFQA, it aims to formulate a response to the query. To facilitate the parsing of the model’s response, we encourage it to generate its response as a list of single-value dictionaries, with “cluster” as the key and a list of indices corresponding to the highlights within the current cluster as the value. For hyperparameter tuning, we divide the gold “highlights” into pairs. Pairs that correspond to the same target output sentence are categorized as positive, whereas others are considered negative. Subsequently, we evaluate the F1 score to assess the alignment between the predicted and gold clusters. This involves calculating the True Positives, False Negatives, and False Positives for the highlights pairs, based on the predicted clusters. Following our analyses, we opt for 2 examples and T = 0.5 in the MDS setting, and 1 example and T = 0.1 in the LFQA setting.
Fine-tuned Examples are provided in Figs. B.10 and B.13. There are two variants for generation planning: generative clustering and joint. In the generative clustering approach, the input to the model is the set of all highlights, concatenated with a special markup “<highlight-separator>”, which is the same as the output from the content selection model. The output is then a set of clusters,
14
where each cluster is separated by a special markup “<cluster-separator>”, and inside each cluster we have a set of highlights in the same format as the input. We run hyperparameter-tuning on each setting based on the development set. The hyperparameters used for MDS are warmup steps 0, weight decay 0, and learning rate 5e-5. The hyperparameters used for LFQA are warmup steps 298, weight decay 0, and learning rate 2e-4. At inference time, we used a beam size of 1. We use the same logits processor described in Appendix C.1 to enforce the model to generate valid highlights.
In the joint approach, the input to the model is the original source documents, similar to the input in the content selection model. In contrast, the output is the clustered highlights. Similar to the content selection model, we use the same logits processor to enforce generating valid highlights. The hyperparameters used for LFQA are warmup steps 298, weight decay 0, and learning rate 2e-4.
C.3 Attributed Fused Output
ICL For the ICL variant, we iteratively show the model only the current cluster’s highlights, in the same manner, we do so in the generation planning ICL experiments (see Appendix C.2). Additionally, we present the model with what it generated to that point and guide it to generate the next sentence in the paragraph, which should cover all and only the highlighted spans (see Figs. B.3 and B.7). We use 3 examples and T = 0.3 for the MDS setting, and 2 examples with T = 0.1 for the LFQA setting.
For the Chain-of-Thought approach, which merges generation planning with the fusion steps, we present the model with the “highlighted” source texts and ask it to fuse those highlights into a coherent output. Additionally, we guide the model, both in the instructions and through in-context demonstrations to iteratively cluster highlights and fuse them into a sentence, before generating the final output (see Figs. B.4 and B.8). We find that 3 incontext examples and T = 0.1 work best for MDS, while for LFQA it is 1 example and T = 0.3 that yield the best results on the development set.
Fine-tuned Examples are provided in Figs. B.11 and B.14. The input to the attributable fusion model is a single cluster of highlights, where highlights are separated by a special markup. The output is then a single sentence of the response. This model is called iteratively, and at each iteration, we include the prefix of the already generated re-
sponse. The output also includes citations for every sentence in the generated response based on the highlights used to generate the particular sentence. The hyperparameters for the fusion model are for MDS are warmup steps 54, weight decay 0.2, and learning rate 1e-4. The hyperparameters used for LFQA are warmup steps 175, weight decay 0.5, and learning rate 2e-5.
C.4 End-to-end Generation
We include as a baseline end-to-end generation without decomposing the process into several steps.
ICL For the ICL setting, we instruct the model to perform the corresponding setting’s task, namely summarization for MDS, and answering a query based on the supplied document for LFQA. In addition to the varying number of in-context examples, we also evaluate the model in a zero-shot setting. Our analyses show that for MDS, 1 example and T = 0.5 give the best performance, while for LFQA 3 examples and T = 0.7 are optimal.
Fine-tuned The input to the model is the set of all documents, concatenated with a special markup “<doc-sep>”, which was also used in pre-training in PRIMERA. Similar to our content selection, we trim each document to the length of 4096 divided by the number of documents in each example. The output is simply the full response, a summary for MDS, or an answer for LFQA. The hyperparameters used for both MDS and LFQA are warmup steps 27, weight decay 0.2, and learning rate 4e-5.
C.5 ALCE Variant
As another baseline, we also implement the ALCE approach (Gao et al., 2023b) for attributed generation, where the model is guided to jointly generate a text with in-line citations, pointing to the supporting source documents. We use the same prompt as in the original paper. Additionally, we find that for both settings, 2 examples and T = 0.5 yield the best results on the respective development sets.
C.6 Constrained Decoding
Both the content selection component and the planning component are designed to exactly copy text spans from source documents. To guide the generation to precisely follow the lexical structure of the input’s text, we employ a constrained decoding strategy for the fine-tuned models. This strategy
15
relies on a logits processor18, allowing the model to avoid certain tokens by assigning their scores to minus infinity. Specifically, the model is restricted from generating n-grams absent from the source documents. Additionally, to optimize the models for producing concise and informative highlights, we impose additional constraints, including setting minimum thresholds on the number of words in each highlight and the overall number of highlights generated. Based on hyperparameter-tuning we set a minimum of three words for each highlight in both MDS and LFQA, and a minimum number of 30 highlights for MDS and 5 highlights for LFQA. These constraints contribute significantly to optimizing the performance and reliability of the finetuned generation models. Lastly, in the planning component, the model organizes and forms clusters, each encapsulating highlights from the given input. To prevent excessively large clusters, we impose a maximum threshold of two highlights per cluster. This facilitates the generation of coherent sentences from each cluster.

heading: D Full-sentence AUTOAIS
text: Tables C.1 and C.2 present the results on the fullsentence AutoAIS vs. regular AutoAIS
18https://huggingface.co/docs/transformers/ internal/generation_utils#logitsprocessor

heading: E Fine-grained Attribution Examples
text: Table D.1 shows fine-grained attributions examples, generated by ATTR. FIRSTCoT in the MDS setting.

heading: F Human Study
text: For our human study, we followed the Controlled Crowdsourcing Methodology (Roit et al., 2020) to detect qualified annotators via Amazon Mechanical Turk.19 We reached out to potential candidates, who had exhibited high proficiency in similar tasks in the past, through email invitations sent from the MTurk platform. During onboarding, candidates reviewed the annotation guidelines and completed a closed qualification round consisting of five annotation tasks. This process, overseen by an author of the paper who provided individualized feedback, required approximately 20 minutes to complete, with participants compensated 4.5$ for their efforts. Ultimately, six out of eleven candidates were selected for the actual evaluations, with compensation rates set at 2$ per ALCE instance and 1$ otherwise, reflecting the varying complexity of the instances. To ensure fair compensation, we also monitored the actual time required to complete each instance and offered post-hoc reimbursements when this time exceeded our initial estimates by more than 5%.
Moreover, to evaluate the consistency of attribution annotations, we computed the inter-annotator agreement (IAA) using Fleiss’ Kappa coefficient (Fleiss, 1971), based on a sample of 101 sentences and their attributions from 23 instances across both models, each annotated by 3 workers.20 The resulting Kappa coefficient was 0.37, indicating a fair level of agreement (Landis and Koch, 1977).

heading: G Annotation Interface
text: Screenshots from our annotation interface are presented in Figs. E.1 and E.2. Our interface mirrors the design from Liu et al. (2023), adapted to support citations with highlights. In addition, the documents are presented within the interface rather than via external links. Within this interface, annotators assess each sentence to determine the level of support provided by its accompanying citations, choosing one of the following options: “Full Support”, “Partial Support”, “No Support”, “Citation Has Support but also Refutes Statement”, “Statement is Unclear, Can’t Make Judgement”. Finally, when calculating the final AIS score, we
19www.mturk.com 20We treat annotators’ scores as binary, akin to Appendix G.
16
17
treat instances labeled as “Partial Support” as nonattributed, while instances classified as “Citation Has Support but also Refutes Statement” are labeled as attributed, akin to Gao et al. (2023a).

heading: H Partially-attributing Examples
text: Table H.1 shows a few attribution examples labeled by human annotators as partially-supporting.

heading: I With Prefix Versus without Prefix
text: Table H.2 presents several instances from our test set when the iterative sentence-level fusion was only conditioned on each iteration’s cluster, versus when it was also conditioned on the prefix.

heading: J Latency and Total Number of Input and Output Tokens
text: In Table H.3 we present the average number of tokens processed in the input and also the average number of generated tokens for each of our pipeline’s subtasks, and also for our baselines. Additionally, in Table H.4 we present the average time in seconds it takes to run each of our pipeline’s subtasks, as well as our baselines.

heading: K Post-generation Attribution
text: We also investigated a post-generation attribution variant, where first an unattributed output was generated, and then relevant attribution was retrieved. To that end, we first generated the task-specific output, using our baseline from Section 4.2. Then, to obtain attributions, we adapted the RARR algorithm, as outlined by Gao et al. (2023a), to suit our context. In contrast to its original application of retrieving relevant documents, our adaptation employs the grounding texts directly. The process involves generating queries from each generated sentence about the various facets it encompasses, identifying 4-sentence paragraphs within the source texts that correspond to each query through a querydocument relevance model, and answering query based on these paragraphs and the output sentence. The comparisons of these responses determine the selection of attributing paragraphs; only those that either entail or contradict are retained as valid attributions. The results, as detailed in Table K.1, show that this method produced both longer and less accurate attributions compared to those derived from our approach.
18
Prediction
1 Output: Despite economic challenges and low approval ratings, Obama’s campaign finance report shows he can stillattract major cash.
Attribution: . . . The president’s campaign finance report shows he can still pull in major cash despite a stagnant economy, dipping approval ratings and grumblings among some liberal supporters that he has not done enough for their cause. . . .
2 Output: The Sheraton Waikiki is the only hotel on Oahu that provides on-site marriage licenses, and it is runninga new ad campaign in mainland LGBT publications.
Attribution: . . . The Sheraton Waikiki is the only hotel on Oahu providing on-site marriage licenses. General Manager Kelly Sanders says now the hotel is running a new ad campaign in mainland LGBT publications. . . .
3 Output: Tarek and his wife Christina tried for two years to have a son, going through two failed IVF attempts and amiscarriage before welcoming their son Brayden James in August.
Attribution: . . . The couple, who are already parents to daughter Taylor, tried for more than two years to get pregnant with their son, going through two failed attempts at IVF - including one resulting in a miscarriage at eight weeks, just before Tarek’s diagnosis. Now in remission, the new dad, who welcomed son Brayden James with Christina, 32, in August, is positive about his family’s future." . . .
Table H.1: Example predictions of the ATTR. FIRSTCoT which were annotated with partial support. The information inferred from the words in bold is not supported by the highlights provided by the model, but is supported by sentences in close proximity.
Prediction
1 w/ prefix: The Ohio River starts at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsyl-vania, and flows 981 miles to its ending point in Cairo, Illinois. The river flows in a southwesterly direction from western Pennsylvania to its mouth on the Mississippi River at the southern tip of Illinois.
w/o prefix: The Ohio River starts at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsylvania, and flows 981 miles to its ending point in Cairo, Illinois. The Ohio River is a 981-mile (1,579 km) long river in the United States.
2 w/ prefix: Idina Menzel and Taye Diggs, who starred together in the Broadway musical "Rent," have decided to endtheir marriage. Their main concern is their son. The couple , who met while starring in the Broadway musical "Rent," have a 4-year-old son, Walker Nathaniel. Diggs and Menzel , both 42, met as costars in the original production of the Broadway musical Rent in the mid-1990s. They worked at their marriage, going through tough times like everyone else, but they loved each other very much.
w/o prefix: Idina Menzel and Taye Diggs, who have been married for 10 years, have a 4-year-old son named Walker. Idina Menzel and Taye Diggs have asked for privacy during this time, as their main concern is for their son. Taye Diggs and Idina Menzel have decided to separate after ten years of marriage, but their main concern is for their son. Taye Diggs and Idina Menzel , who met in the 1990s while starring in the Broadway musical Rent, have asked for privacy during this difficult time. Taye Diggs and Idina Menzel , who met in the mid-1990s while starring in the original production of the Broadway musical Rent, married in 2003 and had a son, Walker, in September 2009.
3 w/ prefix: By exposing students to different religious texts, practices, cultures, and social teachings, they can gaina deeper understanding of the world around them and develop critical thinking skills. However , some Americans believe that teaching religious doctrine in public schools infringes on the First Amendment right to free exercise of religion.
w/o prefix: By exposing students to different religious texts, practices, cultures, and social teachings, they can gain a deeper understanding of the world around them and develop critical thinking skills. Some Americans believe that excluding religious sentiment from public schools infringes on the First Amendment right to free exercise of religion.
Table H.2: Example predictions of the ICL ATTR. FIRST model when prompted with prefix and without prefix.
19
Content Selection Generation Planning Sentence Fusion CoT End-to-end ALCE Input Output Input Output Input Output Input Output Input Output Input Output
IC L MDS
6879 (± 1893) 551 (± 467) 6425 (± 1728) 11 (± 22) 17003 (± 13203) 133 (± 80) 10273 (± 1558) 478 (± 220) 6879 (± 1893) 177 (± 55) 7341 (± 1974) 193 (± 44)
LFQA 10727 (± 3206) 142 (± 161) 2536 (± 1651) 9 (± 13) 3734 (± 1784) 45 (± 28) 2962 (± 635) 127 (± 89) 10727 (± 3206) 91 (± 85) 8476 (± 3809) 95 (± 89)
FT
MDS 1661 (± 116) 456 (± 30) 1041 (± 75) 456 (±30) 348 (± 7) 29 (± 0.6) - - 1661 (± 116) 283 (± 9) - - LFQA 2883
(±162) 121 (±11) 348 (±36) 121 (±11) 209 (±18) 27 (+ -1) - -
2883 (± 162) 60 (± 5) - -
Table H.3: Average number of tokens (and standard deviation) for each of the steps in our pipeline, as well as our two baselines.
Content Selection Generation Planning Sentence Fusion CoT End-to-end ALCE
IC L MDS 33.1 5.2 5.6 17.1 7.0 9.2
LFQA 6.3 4.7 6.8 3.3 6.1 6.7
FT MDS 68.8 5.7 2.4 - 1.7 - LFQA 10.6 3.4 1.6 - 1.2 -
Table H.4: Average time (in seconds) each of the steps in our pipeline takes per instance, as well as for our two baselines.
Task ROUGEL BERTSCORE AUTOAIS LENGTH NO ATT. (%) MDS 19.0 86.1 67.8 137.7 0.7 LFQA 33.1 89.6 50.2 149.0 9.6
Table K.1: Results of the post-generation attribution approach, adapted from the RARR algorithm, on the Multi-document summarization (MDS) and the Longform Question-answering (LFQA) tasks.
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Figure E.1: A screenshot from the first step in our human evaluation annotation interface for the LFQA setting, which is a replication of the interface presented by Liu et al. (2023). The annotator is presented with a query and a response and has to rate two aspects of the response on a Likert scale between 1 to 5. The first aspect is the fluency and cohesiveness of the response. The second is the helpfulness of the response.
35
Figure E.2: A screenshot from the second step in our human evaluation annotation interface for the LFQA setting, which is an adaption of the interface suggested by Liu et al. (2023). The annotator is presented with citations provided by the model. For each sentence, they have to rate the quality of the citations. Highlights are presented if available, with arrows to navigate between different highlights. Completed sentences are presented with a green arrow to indicate that the annotator can continue to the next sentence.
36

heading: Future Work
text: Now in remission, the new dad, who welcomed son Brayden James with Christina, 32, in August, is positive about his family’s future. ". Table H. 1: Example predictions of the ATTR. FIRSTCoT which were annotated with partial support. The information inferred from the words in bold is not supported by the highlights provided by the model, but is supported by sentences in close proximity. Prediction
1 w/ prefix: The Ohio River starts at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsyl-vania, and flows 981 miles to its ending point in Cairo, Illinois. The river flows in a southwesterly direction from western Pennsylvania to its mouth on the Mississippi River at the southern tip of Illinois. w/o prefix: The Ohio River starts at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsylvania, and flows 981 miles to its ending point in Cairo, Illinois. The Ohio River is a 981-mile (1,579 km) long river in the United States. 2 w/ prefix: Idina Menzel and Taye Diggs, who starred together in the Broadway musical "Rent," have decided to endtheir marriage. Their main concern is their son. The couple , who met while starring in the Broadway musical "Rent," have a 4-year-old son, Walker Nathaniel. Diggs and Menzel , both 42, met as costars in the original production of the Broadway musical Rent in the mid-1990s. They worked at their marriage, going through tough times like everyone else, but they loved each other very much. w/o prefix: Idina Menzel and Taye Diggs, who have been married for 10 years, have a 4-year-old son named Walker. Idina Menzel and Taye Diggs have asked for privacy during this time, as their main concern is for their son. Taye Diggs and Idina Menzel have decided to separate after ten years of marriage, but their main concern is for their son. Taye Diggs and Idina Menzel , who met in the 1990s while starring in the Broadway musical Rent, have asked for privacy during this difficult time. Taye Diggs and Idina Menzel , who met in the mid-1990s while starring in the original production of the Broadway musical Rent, married in 2003 and had a son, Walker, in September 2009. 3 w/ prefix: By exposing students to different religious texts, practices, cultures, and social teachings, they can gaina deeper understanding of the world around them and develop critical thinking skills. However , some Americans believe that teaching religious doctrine in public schools infringes on the First Amendment right to free exercise of religion. w/o prefix: By exposing students to different religious texts, practices, cultures, and social teachings, they can gain a deeper understanding of the world around them and develop critical thinking skills. Some Americans believe that excluding religious sentiment from public schools infringes on the First Amendment right to free exercise of religion. Table H. 2: Example predictions of the ICL ATTR. FIRST model when prompted with prefix and without prefix. 19
Content Selection Generation Planning Sentence Fusion CoT End-to-end ALCE Input Output Input Output Input Output Input Output Input Output Input Output
IC L MDS
6879 (± 1893) 551 (± 467) 6425 (± 1728) 11 (± 22) 17003 (± 13203) 133 (± 80) 10273 (± 1558) 478 (± 220) 6879 (± 1893) 177 (± 55) 7341 (± 1974) 193 (± 44)
LFQA 10727 (± 3206) 142 (± 161) 2536 (± 1651) 9 (± 13) 3734 (± 1784) 45 (± 28) 2962 (± 635) 127 (± 89) 10727 (± 3206) 91 (± 85) 8476 (± 3809) 95 (± 89)
FT
MDS 1661 (± 116) 456 (± 30) 1041 (± 75) 456 (±30) 348 (± 7) 29 (± 0. 6) - - 1661 (± 116) 283 (± 9) - - LFQA 2883
(±162) 121 (±11) 348 (±36) 121 (±11) 209 (±18) 27 (+ -1) - -
2883 (± 162) 60 (± 5) - -
Table H. 3: Average number of tokens (and standard deviation) for each of the steps in our pipeline, as well as our two baselines. Content Selection Generation Planning Sentence Fusion CoT End-to-end ALCE
IC L MDS 33. 1 5. 2 5. 6 17. 1 7. 0 9. 2
LFQA 6. 3 4. 7 6. 8 3. 3 6. 1 6. 7
FT MDS 68. 8 5. 7 2. 4 - 1. 7 - LFQA 10. 6 3. 4 1. 6 - 1. 2 -
Table H. 4: Average time (in seconds) each of the steps in our pipeline takes per instance, as well as for our two baselines. Task ROUGEL BERTSCORE AUTOAIS LENGTH NO ATT. (%) MDS 19. 0 86. 1 67. 8 137. 7 0. 7 LFQA 33. 1 89. 6 50. 2 149. 0 9. 6
Table K. 1: Results of the post-generation attribution approach, adapted from the RARR algorithm, on the Multi-document summarization (MDS) and the Longform Question-answering (LFQA) tasks. 20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Figure E. 1: A screenshot from the first step in our human evaluation annotation interface for the LFQA setting, which is a replication of the interface presented by Liu et al. (2023). The annotator is presented with a query and a response and has to rate two aspects of the response on a Likert scale between 1 to 5. The first aspect is the fluency and cohesiveness of the response. The second is the helpfulness of the response. 35
Figure E. 2: A screenshot from the second step in our human evaluation annotation interface for the LFQA setting, which is an adaption of the interface suggested by Liu et al. (2023). The annotator is presented with citations provided by the model. For each sentence, they have to rate the quality of the citations. Highlights are presented if available, with arrows to navigate between different highlights. Completed sentences are presented with a green arrow to indicate that the annotator can continue to the next sentence. 36