Verifiable Generation with Subsentence-Level Fine-Grained Citations (2024)

Abstract: Verifiable generation requires large language models (LLMs) to cite source documents supporting their outputs, thereby improve output transparency and trustworthiness. Yet, previous work mainly targets the generation of sentencelevel citations, lacking specificity about which part of the sentence is backed by which cited source. This work studies verifiable generation with subsentence-level fine-grained citations to locate the generated content that is supported by the cited sources in a more precise way. We first present a dataset, SCIFI, comprising 10K Wikipedia paragraphs with subsentence-level citations.1 Each paragraph in SCIFI is paired with a set of candidate source documents for citation and a query that triggers the generation of the paragraph content. On SCIFI, we then evaluate the performance of state-of-the-a rt LLMs and strategies for processing long documents designed for these models. Our experiment results reveal key factors that can enhance the quality of citations, including the expansion of the source documents’ context to be accessible to the models and the implementation of specialized model tuning.

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated remarkable capabilities in seeking and synthesizing information from given documents (Touvron et al., 2023; OpenAI et al., 2023), and many LLMpowered tools are available to the general public. However, concerns have emerged regarding their outputs’ factual accuracy, faithfulness to the source documents, and trustworthiness in general (Zhang et al., 2023; Peskoff and Stewart, 2023).
To address these concerns, recent research has introduced a new generation paradigm, verifiable generation, where LLMs are required to include citations to source documents in the model outputs,
1Our data is available at https://shuyangcao.github. io/projects/subsentence_citation/.
to support their statements (Bohnet et al., 2023; Gao et al., 2023). Verifiable generation enables users to trace the information back to its source and verify its correctness, enhancing the transparency and reliability of models. Nonetheless, existing work typically provides sentence-level citations. They are unable to indicate which part of the sentence is supported by each referenced source document, leaving the effort to users to infer the connection between the content and its citation (Liu et al., 2023a; Schuster et al., 2023). This ambiguity hinders the user’s ability to verify the information efficiently and understand the scope of the supporting documents.
We argue that verifiable generation with a finer granularity is critical for further improving the transparency and trustworthiness of LLMs. Compared to sentence-level citations, fine-grained citations, such as subsentence-level annotations, allow for more precise localization of the informa-
15584
tion that is sourced from the referenced document and support easier assessment of its accuracy, as shown in Figure 1. Furthermore, the importance of fine-grained citations to readers is quantitatively evinced by their prevalence in the popular Wikipedia pages. For instance, pages that are more frequently viewed are more likely to use citations with finer granularity, as demonstrated in Figure 2.
In this work, we aim to investigate LLMs’ ability of producing output with fine-grained citations. Specifically, we focus on subsentence-level citations, which are commonly used in informationrich documents like encyclopedias and research papers. To facilitate the study, we first collect SCIFI, containing Subsentence-level Citation of Fine granularity based on 10K paragraphs from Wikipedia, where rich citation information is available. For each paragraph in SCIFI, documents that are cited in the Wikipedia page where the paragraph belongs to are provided as the citation candidates. We also create a query based on the content of each paragraph to guide LLMs to generate relevant content.
We benchmark state-of-the-art LLMs on our dataset, including OpenAI GPT (OpenAI et al., 2023), Llama2 (Touvron et al., 2023), Vicuna (Zheng et al., 2023), and Mistral (Jiang et al., 2023). To consume the lengthy source documents, we explore three document reading strategies, that respectively target leading context of all source documents, a large portion of the context of all source documents, and full context of selected source documents. For open-source LLMs, we also examine the effect of fine-tuning with the training samples in our dataset. For evaluation, we assess the citation behavior, citation quality, and answer quality of model output. We find that: (1) complete source document context improves citation quality in LLMs; (2) larger model sizes increase answer quality but not citation quality; (3) fine-grained citation generation requires supervised fine-tuning.
Our contributions are summarized as follows: (1) We collect SCIFI which consists of 10K queries paired with reference answers that are rich in subsentence-level fine-grained citations to the source documents, enabling training models for verifiable generation with finer granularity. (2) We analyze performance of state-of-the-art LLMs augmented with various document processing strategies on SCIFI, highlighting directions that could advance model development for verifiable generation.

heading: 2 Related Work
text: Verifiable Generation. Forming in-line citations in verifiable generation challenges LLMs’ abilities to ground their generation in source documents and perform accurate attribution. To teach LLMs to include in-line citations in their outputs, early work fine-tunes LLMs with human written demonstrations (Nakano et al., 2022) or model-generated samples verified by human annotators (Menick et al., 2022), but their privately hosted training data prevents follow-up studies. The introduction of more capable LLMs (Jiang et al., 2023; OpenAI et al., 2023) makes it feasible to prompt LLMs with wellcrafted instructions to cite source documents during generation (Gao et al., 2023), and such behavior has been activated in online systems that are based on LLMs (Liu et al., 2023a), though the quality of the generated citations leaves large room for improvement (Malaviya et al., 2023). To enhance the citation quality, recent studies consider fetching source documents that better entail the output content (Li et al., 2023) or enabling LLMs to refine its outputs (Sun et al., 2024). Nevertheless, existing work largely focuses on verifiable generation with sentence-level citations, without clearly indicating the exact portion in the output that is supported by the source documents. This work, on the other hand, explores verifiable generation with finer granularity.
Fact-based Evaluation. Evaluation of outputs produced by LLMs is challenging due to their openended nature. Recent work resorts to fact-based evaluation, where an output is first decomposed into independent facts and compared against facts in the reference output (Liu et al., 2023b). To circumvent data collection for fact decomposition model training, LLMs have been instructed to extract facts from the output to be evaluated (Kamoi et al., 2023). Furthermore, Min et al. (2023) leverage LLMs’ strong capability of identifying content with similar semantics and design an LLM-based fact comparison module for more accurate evaluation. While previous work utilizes fact-based framework for evaluation of factual entailment and precision, we assess both answer coverage and citation quality with fact-based evaluation.

heading: 3 SCIFI
text: To benchmark LLMs on verifiable generation with finer granularity, we first collect SCIFI, a dataset
containing questions that require synthesizing information from multiple source documents to answer and demand more fine-grained citations to precisely locate information supported by different source documents. SCIFI is based on Wikipedia, as Wikipedia articles may densely include citations to support their content.
Before data collection, we examine how prevalent fine-grained citations are used in Wikipedia articles. Out of 100K randomly picked Wikipedia articles, 85% feature subsentence-level citations. We further sort these articles based on their pageview statistics and divide them into 20 bins.2 For each article, we tallied the total number of views over a five-year period to minimize the impact of transient trends. As can be seen from Figure 2, bins with higher pageview counts show a greater tendency of fine-grained citations, highlighting their importance to readers.
Data Collection. For each article, we extract its text content from the Wikipedia dump while keeping track of the positions and referenced sources of citations in the article. We focus on cited sources that link to downloadable websites and obtain their text content, as it is infeasible to have a uniform process to download and accurately extract content for all types of sources (e.g., images, and PDF files). The position and meta information of all types of cited sources are still preserved, though, to facilitate other research problems such as identifying citation-worthy content.
We further formulate our dataset into a questionanswering dataset by creating queries that serve as
2The pageview statistics are obtained via the Wikimedia pageview API: https://wikitech.wikimedia.org/wiki/ Analytics/AQS/Pageviews.
constraints of the model generated content. Queries are crafted for paragraphs with dense citations rather than entire articles, for more appropriate target lengths to support more sophisticated LLM techniques (e.g., in-context learning). We rank paragraphs with at least 3 citations based on citation density, by dividing the total number of citations by the total number of sentences. From the top 25% of these ranked paragraphs, we randomly sample 10K paragraphs to promote paragraphs with more fine-grained citations. For each selected paragraph, we prompt GPT-4 to generate a query asking about the content in the paragraph. Having queries to guide and constrain model outputs allows for more robust evaluation of content quality by anchoring to the reference paragraphs.
Eventually, each sample in SCIFI contains a reference paragraph, the query generated for the paragraph, and all source documents that are cited by the article where the reference paragraph belongs. The dataset is split into training and test sets with 9K and 1K samples.
Statistics. We report statistics of SCIFI along with recent datasets that involve citations in the outputs, including WICE (Kamoi et al., 2023), HAGRID (Kamalloo et al., 2023), and EXPERTQA (Malaviya et al., 2023) in Table 1. As each sample in WICE is a single sentence with citations, for fair comparisons, we compute citation density only for sentences containing at least one citation on all datasets. Our dataset contains samples with high citation density and moderate target lengths. More importantly, it includes citations with finer granularity, while others focus on sentence-level citations. More details of collection and statistics of SCIFI are included in Appendix A.

heading: 4 Experiment Setups
text: Task Setup. Given a sample in SCIFI, the LLM to be tested takes as input the query and candidate source documents to generate a paragraph with
fine-grained citations. Due to the sheer volume and excessive length of all available source documents, it is impractical to input them into the LLM simultaneously. Therefore, we provide the LLM with positive source documents—those cited by the reference paragraph, and 5 randomly selected negative source documents that are not cited by the reference paragraph. We shuffle these source documents before feeding them into the LLM.
Models. Though we limit the number of source documents input to the model, their concatenation remains lengthy. To allow the model to process the source documents, we consider (1) truncating source documents to the same size such that their concatenation can be consumed by the model (Truncated); and (2) providing summaries of source documents to the model (Summary). Additionally, we design a two-stage framework that iteratively selects the documents to be covered in next sentence by reading their summaries and writes the next sentence by consuming the original text of the selected documents. This selection process reduces the size of the document pool for the generation phase, thereby allowing the inclusion of more complete document context.
We examine the efficacy of both proprietary and open-source backbone models. As for proprietary models, we test GPT-3.5 with 16K context length and GPT-4 with 8K context length. The 7B and 13B variants of Llama2, 7B variant of Vicuna, and 7B variant of Mistral are chosen for the opensource backbones. We use their RLHF-tuned version for all open-source LLMs. Besides in-context learning that is used for all experimented models, we additionally perform supervised fine-tuning for open-source models with 4,000 samples in the training set of SCIFI to investigate the effect of fine-tuning.
Evaluation Metrics. We target the assessment of LLMs’ ability to produce subsentence-level finegrained citations, precisely cite the supporting documents, and cover sufficient information for answering the question. We first calculate the citation density of model outputs, which is the average number of citations in each output sentence.
For citation quality, we follow Rashkin et al. (2023) and measure how well the output statements entail the cited sources. Unlike sentencelevel citations which can be paired with the entire sentence for entailment assessment, evaluation of fine-grained citations requires segmenting the sen-
tence and mapping citations to the sentence portions they support. Inspired by fact-level entailment (Min et al., 2023), we decompose model outputs into individual facts and pair citations with facts using heuristics (Appendix C). Following Gao et al. (2023), we run an off-the-shelf entailment model (Honovich et al., 2022) to obtain the entailment levels between generated citations and facts.
Fact-level evaluation is also applied to assess answer quality. For each fact in the reference paragraph, we check if it entails the model output. The aggregation of the fact-level entailment scores reflects the coverage of reference facts (i.e., recall).

heading: 5 Results
text: Source documents with more complete context benefit citation quality, as indicated by the factlevel entailment scores of citations produced by different document reading strategies with in-context learning (Table 2). Summaries can inform models of the major content in the source documents, while truncation only exposes leading content and prevents accurate connection between generated content and supporting documents, thus consis-
tently yielding worse citation quality. Two-stage generation allows for the most complete document context, boosting the citation accuracy, yet its effectiveness relies on strong instruction-following capabilities of the LLMs.3
Increasing model sizes promotes answer quality. Across different backbone models and strategies, the coverage of reference facts increases after switching to a larger model within the same family, though larger model sizes do not guarantee an enhancement in citation quality. This reveals that the pre-training designs of different backbone LLMs might all aim for stronger question-answering capabilities, but assign varying significance to their citation and attribution capabilities.

heading: Generation of fine-grained citations requires
text: additional training. We observe that the density of fine-grained citations generated by the same backbone LLMs remains stable across different document reading strategies. By contrast, models generate substantially more fine-grained citations after supervised fine-tuning, as shown in Table 3. However, the effectiveness of supervised fine-tuning in enhancing citation quality varies across models. We think that supervised fine-tuning encourages LLMs’ behaviors of generating fine-grained citations. Yet, the development of LLMs’ abilities to correctly link sentence parts with supporting documents requires more specialized and sophisticated training procedure, which highlights the challenge of this task. Future directions may include design builtin citation or attribution mechanisms during LLM pretraining (Khalifa et al., 2024).
3The two-stage strategy is only adopted by the OpenAI GPT families, as other models could not consistently follow the output format designed for the strategy, resulting in invalid results.

heading: 6 Conclusions
text: We study verifiable generation with subsentencelevel fine-grained citations. SCIFI, a benchmark containing 10K subsentence-level citation-rich paragraphs together with candidate cited sources and queries, is collected to support the training and evaluation of models on this task. On SCIFI, experiments with state-of-the-art LLMs and various processing strategies demonstrate the importance of source document context and training with citation-rich data.

heading: Acknowledgments
text: This work is supported in part by the National Science Foundation through grant IIS-2046016. Shuyang Cao is supported by a Bloomberg Data Science Ph.D. Fellowship. We thank ARR reviewers for their feedback.

heading: Limitations
text: In our experiments, we employ three strategies for LLMs to handle lengthy source documents and observe improved performance when the strategy enables a more comprehensive context. Yet, more sophisticated strategies can be designed, and techniques that expand input limits of LLMs can be explored, which could potentially lead to higher performance in our benchmark study.
When evaluating the attribution quality, we leverage an existing entailment model tailored for assessing the entailment relation between short passages. However, we use it to measure the entailment relation between an extracted fact and a long source document. Although we follow the technique in previous work (see Appendix C.3) to extend the application of the off-the-shelf entailment models to long documents, more accurate evaluation can be achieved by developing entailment models specialized for long documents.

heading: Ethical Considerations
text: Our benchmark enables the evaluation of LLMs’ ability to generate subsentence-level citations. With subsentence-level citations, LLM developers are able to present LLM outputs that contain precise pointers directing users to supporting sources of different parts of output sentences. While this would enhance user trust in LLMs, it is worth noting that our dataset comprises texts that are formally written in Wikipedia and the candidate supporting documents are from reliable online sources.
An LLM with outstanding performance on our dataset might cite documents with fake facts if the candidate documents are from unreliable sources, which might further propagate incorrect information. Users of our benchmark should also consider the reliability of their candidate supporting documents when examining the reliability of LLMbased applications.

heading: A Details of Data Collection

heading: A.1 Wikipedia Article and Cited Source Collection
text: Wikipedia paragraphs in SCIFI are extracted from the Aug 20, 2021 Wikipedia dump. For each source that links to a downloadable website, we retrieve its HTML file from Internet Archive4 and extract its metadata and text content using Trafilatura.5
A sample example is shown in Table 13. Our dataset will be made publicly available un-
der the CC BY 4.0 license.6

heading: A.2 Topic Distribution
text: We use the topic model provided by Wikimedia7 to determine the topic of the page from which each paragraph in SCIFI is extracted. For each page, we select its top 3 predicted topics, considering that a single page could cover multiple topics. Top 10 topics are presented in the Table 4. Paragraphs in our dataset come from pages of diverse topics.

heading: A.3 Additional Statistics
text: SCIFI has 39292 sentences in total, 30.1% of which have more than one citation. As subsentencelevel citations frequently occur at the end of clauses
4http://web.archive.org/ 5https://github.com/adbar/trafilatura 6https://creativecommons.org/licenses/by/4.0/ 7https://meta.wikimedia.org/wiki/Machine_
learning_models/Production/English_Wikipedia_ article_topic
marked with punctuation, we also check the percentage of subsentence-level citations in SCIFI that are not attached to punctuation. We find that 30.2% of subsentence-level citations are not located around punctuation, indicating a decent level of diversity in subsentence-level citations.

heading: A.4 Prompt for Query Generation
text: We use GPT-4 to create query for each paragraph in SCIFI. The prompt we use is shown in Table 5.

heading: B Additional Results
text: We additionally test the model performance in the oracle setup, where only positive source documents (i.e., those cited by the reference paragraph) are fed into the LLM (Table 6). In the oracle setup, trends of the results are similar to those in the regular setup, with all models tending to produce answers of high quality while maintaining the citation quality. This indicates that removing source documents irrelevant to the query offers minimal help to LLMs for verifiable generation.

heading: C Details of Experiment Setups

heading: C.1 Model Prompting
text: We use 2-shot examples for all experiments with incontext learning. Prompts for the Truncated and Summary strategy are shown in Table 7 and 9. Summaries of the source documents are generated by GPT-3.5 with 16K context length using the prompt in Table 8.
In the Two-stage strategy, the LLM is given two different prompts to perform document selection and answer sentence generation. When selecting source documents, the LLM is informed of the current answer and all its previous selections, as shown in Table 10. When generating the next answer sentence, the LLM is provided the current answer and the original text of the selected source documents (Table 11).
All models are only prompted once due to the API cost.

heading: C.2 Supervised Fine-tuning
text: We use LLaMA-Factory8 for fine-tuning Llama2 models in our experiments. We adopt LoRA (Hu et al., 2021) and fine-tune the model for 3 epochs with a learning rate of 5e-5 and a batch size of 16. All LoRA-compatible projection layers are tuned,
8https://github.com/hiyouga/LLaMA-Factory/
with a rank of 32 and a α of 64. Training of each model is conducted using 2 Nvidia A40 GPUs and takes 4 hours to complete.

heading: C.3 Evaluation Metrics
text: We leverage fact-level entailment to evaluate the citation quality and answer quality. We follow the prompts in previous work (Min et al., 2023; Kamoi et al., 2023) and use GPT-3.5 to conduct fact decomposition for each output sentence separately.
To map citations to an extracted fact, we first map the extracted fact back to a segment in the original sentence. For accurate mapping, we again use GPT-3.5 to identify segments in the original sentence that best represents the extracted fact, with the prompt in Table 12. We then rank generated citations based on their distances to the sentence segment associated with the extracted fact. If two citations have the same distance to the end of the
sentence segment, the one after the sentence segment is ranked higher, as we hypothesize that the citation supporting a fact is likely to occur after its completion in the sentence. The top-ranking citation is mapped to the extracted fact.
When evaluating the entailment level, directly pairing the cited source document with the extracted fact is infeasible due to the length of the cited document. Following Kamoi et al. (2023), we divide the cited document into chunks of 256 tokens, calculate the extracted fact’s entailment level against each chunk, and take highest entailment level as the final entailment score.

heading: NASA

heading: Future Work
text: Future directions may include design builtin citation or attribution mechanisms during LLM pretraining (Khalifa et al. , 2024). 3The two-stage strategy is only adopted by the OpenAI GPT families, as other models could not consistently follow the output format designed for the strategy, resulting in invalid results