LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

## Abstract

Abstract Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering user questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to their potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations, improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs’ performance in Long-Context Question Answering with Citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the LongCite-45k dataset, successfully enabling their generation of accurate responses and fine-grained sentence-level citations in a single output. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o.
We also discover that SFT with citation information effectively reduces hallucinations and enables a more uniform utilization of context.
Our code & models are at: https://github.com/THUDM/LongCite .

## 1 Introduction

Recent years have witnessed significant advancement in long-context large language models (LLMs), enabling them to address various user questions, such as information extraction and summarization, based on lengthy texts that surpass 100,000 tokens (Anthropic, ; Zeng et al., ; Reid et al., ). Despite their remarkable capacities, current long-context LLMs typically do not provide citations to specific context snippets to support the statements they generated, making it challenging for users to verify model outputs given the substantial context lengths. This significantly impacts the reliability and trustworthiness of long-context LLMs, especially considering that they still struggle with hallucinations (Ji et al., ; Huang et al., ) and are prone to generate unfaithful content.

On the other hand, recent works in web browsing and open-domain QA have allowed LLMs to generate responses with in-line citations through retrieval-based generation (RAG) or post-hoc methods (Nakano et al., ; Gao et al., ; ; Menick et al., ). Nevertheless, these approaches still expose notable limitations in long-context scenarios: RAG often leads to compromised answer quality due to incomplete context information, while post-hoc methods prolong the user waiting time due to the more complicated pipeline. In addition, their generated citations typically refer to entire web pages Nakano et al. (); Bohnet et al. () or coarsely chunked snippets Gao et al. (), thereby requiring users to further pinpoint the specific support evidence for the final verification.

To overcome the above limitations, this work explores directly employing long-context LLMs to generate accurate responses with fine-grained sentence-level in-line citations. We first propose LongBench-Cite, an automatic benchmark, to evaluate LLMs’ performance on the task of long-context question answering with citations (LQAC), and find that current LLMs obtain unsatisfactory results
(Sec. 2). Specifically, we find that many citations produced by current LLMs are either irrelevant, cannot fully support the response, or have a coarse granularity. Meanwhile, we observe that generating citations on the fly via in-context learning generally results in responses with lower correctness compared to vanilla long-context QA.

To further enhance the inherent capacity of LLMs for generating fine-grained citations from lengthy contexts, it is essential to construct a high-quality SFT dataset. To this end, we introduce CoF (abbr. for “Coarse to Fine”), a novel pipeline that utilizes off-the-shelf LLMs to automatically construct long-context QA instances with precise sentence-level citations (Sec. 3). CoF comprises four stages: (1) Starting with a long text material, CoF first invokes the LLM to produce a query and its associated answer through Self-Instruct (Wang et al., ). (2) Next, CoF uses the answer to retrieve several chunks (each has a fixed length of 128 tokens (^1^11In this work, we uniformly use GLM4-9B’s tokenizer to count tokens.)) from the context, which are then fed into the LLM to incorporate coarse-grained chunk-level citations within the answer. (3) The LLM subsequently identifies relevant sentences from each cited chunk to produce fine-grained citations. (4) As a final step, instances with an insufficient number of citations are discarded. Our experiments validate the superiority of CoF over other LQAC strategies in terms of answer correctness and citation quality. With CoF, we construct LongCite-45k, a large-scale SFT dataset that consists of 44,600 high-quality LQAC instances with contexts up to 128,000 tokens.

Finally, we utilize LongCite-45k to fine-tune GLM-4-9B (Zeng et al., ) and Llama3.1-8B (Vavekanand & Sam, ), two latest open-source long-context models (Sec. 4). The enhanced models, namely LongCite-9B and LongCite-8B, support a max context window of 128,000 tokens and are capable of generating accurate responses along with precise, fine-grained citations in one pass. Evaluation on LongBench-Cite indicates that our trained models achieve significantly better citation quality compared to even much larger proprietary models. Specifically, our 8B/9B size model outperforms GPT-4o by 6.4%/3.6% in term of citation F1 score and also achieves twice finer granularity. Meanwhile, we observe that SFT on LQAC data can alleviate hallucinations of LLMs and enable them to utilize context information more uniformly and comprehensively, instead of only focusing on a specific part of the context. This results in a further improvement in response correctness over vanilla long-context SFT. We also conduct extensive analyses and human evaluation to further verify the effectiveness of our approach.

To summarize, our work makes the following contributions:

1. We introduce LongBench-Chat, an automatic benchmark for the task of LQAC, and reveal the limited performance of current long-context LLMs.

2. We propose CoF, which utilizes off-the-shelf LLMs to automatically construct high-quality long-context QA instances with fine-grained sentence-level citations. Using this method, we construct LongCite-45k, a large-scale SFT dataset for LQAC.

3. We successfully train LongCite-8B and LongCite-9B using LongCite-45k dataset, allowing the generation of accurate responses and fine-grained citations in one pass. Our experiments show that SFT on LQAC data not only enhances the capacity for generating citations from lengthy contexts but also further improves response correctness.

## 2 Longbench-Cite: Benchmark Long-context QA with Citations

### 2.1 Problem Definition

Figure: Figure 1: Comparison between chunk-level and sentence-level citations.
Refer to caption: x1.png

We formalize the task of long-context question answering with citations (LQAC) as follows: given a long context $\mathcal{D}$ and a query $q$, the LLM is required to return a response $\mathcal{A}$, which consists of $n$ statements $s_{1},\dots,s_{n}$, and each statement $s_{i}$ cites a list of snippets $\mathcal{C}_{i}=\{c_{i,1},c_{i,2},\dots\}$ from $\mathcal{D}$. In this work, LLMs need to segment their responses into statements based on semantic integrity by enclosing each statement with two special tokens <statement>and </statement>. As illustrated in Figure 1, we consider two types of citations:

- •
Chunk-level citations, where the context $\mathcal{D}$ is divided into indexed chunks with a fix length of 128 tokens, and each citation $c_{i,j}$ is in the form of [$k$], referring to the $k$-th chunk;
- •
Sentence-level citations, where $\mathcal{D}$ is divided into indexed sentences using NLTK (Bird, ), and each $c_{i,j}$ takes the form of [$k$] or [$a$-$b$], referring to the $k$-th sentence or the snippet that includes the $a$-th to $b$-th sentences in $\mathcal{D}$, respectively.

Previous works (Gao et al., ) for open-domain QA and citation only explore the chunk-level citations. However, the crude segmentation applied for chunk-level citations often results in incomplete cited sentences that affect user experience. In this work, we mainly focus on sentence-level citations because they ensure semantic integrity better, allow for finer-grained citation, and are thus more user-friendly.

### 2.2 Data Collection

To evaluate LLMs’ performance on LQAC task, we curate a new benchmark LongBench-Cite by collecting data from existing bilingual long-context benchmarks LongBench (Bai et al., ) and LongBench-Chat (Bai et al., ), covering multiple key user-intensive tasks in both English and Chinese. Specifically, LongBench is a comprehensive benchmark with an average length of 7k words (English) and 13k characters (Chinese), and we select two single-doc QA datasets MultiFieldQA-en/zh (Bai et al., ), two multi-doc QA datasets HotpotQA (Yang et al., ) and DuReader (He et al., ), and one summarization dataset GovReport (Huang et al., ) from it. LongBench-Chat comprises 50 real-world queries with long contexts ranging from 10k to 100k in length, covering various scenarios such as document QA, summarization, and coding, and we adopt all the queries. The detailed data statistics are listed in Table 1.
For all datasets, we require LLMs to generate long-form responses with citations.

**Table 1: Data Statistics in LongBench-Cite. ‘Source’ means the origin of the context. ‘Avg Len’ denotes the average number of words/characters of contexts in English/Chinese datasets.**
| Dataset | Task | Source | Avg Len | Language | #data |
| --- | --- | --- | --- | --- | --- |
| MultiFieldQA-en | Single-Doc QA | Multi-field | 4,559 | English | 150 |
| MultiFieldQA-zh | Single-Doc QA | Multi-field | 6,701 | Chinese | 200 |
| HotpotQA | Multi-Doc QA | Wikipedia | 9,151 | English | 200 |
| Dureader | Multi-Doc QA | Baidu Search | 15,768 | Chinese | 200 |
| GovReport | Summarization | Government Report | 8,734 | English | 200 |
| LongBench-Chat | Multi-task | Real-world Query | 35,571 | English/Chinese | 50 |

### 2.3 Automatic Evaluation

LongBench-Cite evaluates models’ responses based on the two dimensions:

- •
Correctness: Whether the response is accurate and comprehensive for the query.
- •
Citation quality: Whether the response is entirely supported by the cited snippets, no irrelevant snippets are cited, and the cited snippets are fine-grained.

In the following, we introduce automatic metrics for each dimension.

#### 2.3.1 Evaluation of Correctness

For the correctness dimension, we adopt the evaluation method in Bai et al. (), which is specially designed for long-form responses. Specifically, we first remove citation-relevant tokens from LLM response, then ask GPT-4o to rate the response based on the query and groundtruth answers via few-shot (for LongBench-Chat) or zero-shot prompting (for other datasets). The detailed prompts can be found in Figure 4, 5, and 6. In addition, to investigate whether adding citations will hurt or improve models’ long-context QA performance, we propose a new metric correctness ratio:

$$ $\text{CR}=\text{C}/\text{C}_{\text{LQA}}\times 100\%$ (1) $$

Here, C and $\text{C}_{\text{LQA}}$ respectively denote the correctness in LQAC setting and vanilla long-context QA setting, where we simply feed the concatenated context and query into the LLM to get a response.

#### 2.3.2 Evaluation of Citation Quality

To evaluate the citation quality, we select citation F1 calculated using citation recall and citation precision (Gao et al., ) as the main metric, where the former examines if the model response is fully supported by cited snippets and the later detects irrelevant citations. Compared with Gao et al. (), which uses NLI model TRUE (Honovich et al., ) for automatic examination, we further improve the measurement method with GPT-4o to better adapt to long-context QA scenarios. Human evaluation (Sec. 4.3) demonstrates our improved method has a stronger agreement with human. Besides, we use citation length to measure the granularity of citations and avoid trivial results.

Citation Recall. We score citation recall (0/0.5/1) for each statement and average over all statements in the model response. Specifically, for each statement $s_{i}$ that cites at least one snippet ($\mathcal{C}_{i}\neq\emptyset$), we concatenate all snippets in $\mathcal{C}_{i}$ and ask GPT-4o to judge whether the concatenated text fully supports (1 point), partially supports (0.5 point), or does not support (0 point) $s_{i}$. On the other hand, most LLM responses contain several “functional sentences” such as “The proposed method has the following advantages:” and “In summary, …” that do not require citation. Therefore, for each statement $s_{i}$ that has no citation ($\mathcal{C}_{i}=\emptyset$), we feed $s_{i}$ along with the query and the whole response into GPT-4o and prompt it to determine if $s_{i}$ is a starting sentence, transition sentence, or a summary or reasoning based on the previous response content. If so, $s_{i}$ needs no citation and directly receives a citation recall of 1; otherwise, the recall is 0. The prompts are shown in Figure 7 and 8.

Citation Precision. We calculate citation precision for each citation (0/1 for irrelevant/relevant citations) and average over all citations in the response. Here, a cited snippet $c_{i,j}$ is relevant if and only if it entails some key points of the statement $s_{i}$, i.e., at least partially supports $s_{i}$. We also employ GPT-4o as the judge using the prompt in Figure 9. In contrast, Gao et al. () may overlook partially supporting cases due to the limited capacity of the NLI model it uses.

Citation F1. Citation F1 is a comprehensive metric to evaluate the citation quality of a response:

$$ $\text{F1}=(2\cdot\text{P}\cdot\text{R})/(\text{P}+\text{R})$ (2) $$

where P and R denote the citation precision and recall of the response, respectively.

Citation Length. Since the sentence-level citation allows citing snippets with different lengths, we use citation length, which is the average token number of cited snippets in the response, to quantify the granularity of citations. A lower average citation length indicates the response has finer-grained and more concise citations and is thus easier for users to validate. In addition, measuring average citation length can avoid trivial hacks for citation F1 such as citing the whole context for each statement.

### 2.4 Benchmarking Results of Current Long-context LLMs

We first evaluate 7 popular long-context LLMs (3 proprietary and 4 open-source models, details listed in Table 8) on LongBench-Cite using LAC-S (long-context answering with citations in senetence level) strategy, where the model needs to read the lengthy context entirely and generate the answer along with sentence-level citations in one pass. We select LAC-S strategy as the default setting due to its efficiency, losslessness of context information, and no reliance on additional retrieval systems. A further discussion about the pros and cons of different LQAC strategies can be found in Sec. 3.2. As illustrated in Figure 10, we number each sentence $sent_{i}$ in the context by adding a prefix “<$\text{C}i$>” and prompt the LLM with one demonstration. The evaluation results of citation quality and correctness are presented in Table 2 and Table 3, respectively. Our findings are as follows:

**Table 2: Citation recall (R), citation precision (P), citation F1 (F1), and citation length (CL) of different models on LongBench-Cite using LAC-S strategy. The best and second results are bolded and underlined, respectively.**
| Model | Avg | Longbench-Chat | MultifieldQA | HotpotQA | Dureader | GovReport |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | CL | R | P | F1 | R | P | F1 | R | P | F1 | R | P | F1 | R | P | F1 |  |
| Proprietary models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 65.6 | 220 | 46.7 | 53.5 | 46.7 | 79.0 | 87.9 | \ul80.6 | 55.7 | 62.3 | 53.4 | 65.6 | 74.2 | 67.4 | 73.4 | 90.4 | 79.8 |
| Claude-3-sonnet | 67.2 | 132 | 52.0 | 67.8 | 55.1 | 64.7 | 85.8 | 71.3 | 46.4 | 65.8 | 49.9 | 67.7 | 89.2 | 75.5 | \ul77.4 | 93.9 | \ul84.1 |
| GLM-4 | 65.4 | 169 | 47.6 | 53.9 | 47.1 | 72.3 | 80.1 | 73.6 | 47.0 | 50.1 | 44.4 | 73.4 | 82.3 | \ul75.0 | 82.8 | \ul93.4 | 87.1 |
| Open-source models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GLM-4-9B-chat | 27.2 | 96 | 25.9 | 20.5 | 16.7 | 51.1 | 60.6 | 52.0 | 22.9 | 28.8 | 20.1 | 45.4 | 48.3 | 40.9 | 5.7 | 8.2 | 6.3 |
| Llama-3.1-8B-Instruct | 19.7 | 100 | 14.1 | 19.5 | 12.4 | 29.8 | 44.3 | 31.6 | 20.2 | 30.9 | 20.9 | 22.0 | 25.1 | 17.0 | 16.2 | 25.3 | 16.8 |
| Llama-3.1-70B-Instruct | 40.4 | 174 | 25.8 | 32.0 | 23.2 | 53.2 | 65.2 | 53.9 | 29.6 | 37.3 | 28.6 | 38.2 | 46.0 | 35.4 | 53.4 | 77.5 | 60.7 |
| Mistral-Large-Instruct | 51.5 | 132 | 19.8 | 23.9 | 19.0 | 71.8 | 80.7 | 73.8 | 34.5 | 40.9 | 32.1 | 58.3 | 67.0 | 60.1 | 67.9 | 79.6 | 72.5 |
| Our trained models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LongCite-8B | 72.0 | 85 | 62.0 | 79.7 | 67.4 | \ul74.7 | 93.0 | 80.8 | \ul59.2 | \ul72.1 | \ul60.3 | \ul68.3 | 85.6 | 73.1 | 74.0 | 86.6 | 78.5 |
| LongCite-9B | \ul69.2 | \ul91 | \ul57.6 | \ul78.1 | \ul63.6 | 67.3 | \ul91.0 | 74.8 | 61.8 | 78.8 | 64.8 | 67.6 | 89.2 | 74.4 | 63.4 | 76.5 | 68.2 |

**Table 3: Correctness in LQAC setting (C) using LAC-S strategy, correctness in vanilla long-context QA setting ($\text{C}_{\text{LQA}}$), and correctness ratio (CR) of different models on LongBench-Cite. We mark the cases where adding citations improves/hurts correctness (i.e., $\text{CR}>1$ / $\text{CR}<1$) in green/red.**
|  | Avg | Longbench-Chat | MultifieldQA | HotpotQA | Dureader | GovReport |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | C | $\text{C}_{\text{LQA}}$ | CR | C | $\text{C}_{\text{LQA}}$ | CR | C | $\text{C}_{\text{LQA}}$ | CR | C | $\text{C}_{\text{LQA}}$ | CR | C | $\text{C}_{\text{LQA}}$ | CR | C | $\text{C}_{\text{LQA}}$ | CR |
| Proprietary models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 69.4 | 78.2 | 88% | 61.6 | 77.4 | 80% | 84.0 | 88.3 | 95% | 74.5 | 80.8 | 92% | 81.0 | 83.3 | 97% | 46.0 | 61.3 | 75% |
| Claude-3-sonnet | 77.6 | 78.3 | 99% | 73.8 | 77.8 | 95% | 88.6 | 88.1 | 101% | 81.3 | 75.3 | 108% | 75.8 | 80.3 | 94% | 68.4 | 70.1 | 98% |
| GLM-4 | 73.7 | 77.2 | 95% | 69.4 | 79.8 | 87% | 87.6 | 88.1 | 99% | 76.3 | 76.5 | 100% | 76.0 | 75.8 | 100% | 59.4 | 65.9 | 90% |
| Open-source models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GLM-4-9B-chat | 62.3 | 70.8 | 88% | 60.4 | 67.8 | 89% | 74.2 | 84.9 | 87% | 68.5 | 71.5 | 96% | 49.3 | 68.1 | 72% | 59.3 | 61.6 | 96% |
| Llama-3.1-8B-Instruct | 52.1 | 60.2 | 86% | 53.2 | 61.6 | 86% | 63.9 | 73.3 | 87% | 64.0 | 64.5 | 99% | 29.8 | 39.4 | 76% | 49.6 | 62.1 | 80% |
| Llama-3.1-70B-Instruct | 62.0 | 65.5 | 95% | 60.8 | 64.6 | 94% | 78.4 | 78.3 | 100% | 71.3 | 75.3 | 95% | 43.3 | 42.5 | 102% | 56.3 | 66.9 | 84% |
| Mistral-Large-Instruct | 73.6 | 76.4 | 96% | 63.8 | 67.8 | 94% | 88.0 | 85.3 | 103% | 77.0 | 77.3 | 100% | 79.0 | 83.3 | 95% | 60.4 | 68.3 | 88% |
| Our trained models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LongCite-8B | 71.7 | 67.6 | 107% | 69.0 | 68.6 | 101% | 87.0 | 83.6 | 104% | 70.8 | 69.0 | 103% | 68.5 | 62.3 | 110% | 63.0 | 54.4 | 116% |
| LongCite-9B | 70.4 | 65.6 | 109% | 67.6 | 64.6 | 105% | 84.1 | 83.3 | 101% | 71.8 | 67.5 | 106% | 69.0 | 66.3 | 104% | 59.6 | 46.4 | 128% |

1. Open-source LLMs, especially models with smaller sizes, have poor citation quality and lag far behind proprietary LLMs. Though achieving correctness close to proprietary LLMs, open-source LLMs have obvious difficulty in citing supporting evidence for their generated statements. We attribute this to (1) poor instruction-following and in-context learning ability: open-source models often generate citations that do not conform to the prescribed format; (2) relatively weak evidence-searching ability: they often fail to find evidence for some statements (i.e., $\mathcal{C}_{i}=\emptyset$), or only find partially supporting or irrelevant evidence.

2. The citation quality of proprietary LLMs is still unsatisfactory. The citation F1 of proprietary LLMs on LongBench-chat and HotpotQA is only around 0.5, which means less than half statements in their responses are fully supported by the citations. Furthermore, their average citation length is even larger than chunk-level citation (whose citation length is 128), reflecting a coarse citation granularity. For example, the citation length of GPT-4o reaches 220 and each cited snippet contains about 6 sentences on average.

3. Generating responses and citations in one pass via in-context learning hurts the long-context QA performance. On most datasets, current LLMs have correctness ratios less than 100%, indicating that compared to standard long-context question answering, generating responses and citations at once through in-context learning always leads to correctness degradation due to the distribution shift from the post-training data.

Overall, the performance of current LLMs on LQAC remains to be improved. To this end, we will explore automatic construction of SFT data in the following section to further enhance LLMs’ capabilities for generating fine-grained sentence-level citations from lengthy contexts.

Figure: Figure 2: Overview of our CoF pipeline. The pipeline consists of four steps: (1) Generating long-context QA instance via Self-Instruct; (2) Using the answer to retrieve $k$ context chunks and generating chunk-level citations; (3) Extracting sentence-level citations for each statement from the cited chunks. (4) Filter out LQAC instances with few citations.
Refer to caption: x2.png

## 3 CoF: Automatic SFT Data Construction For LQAC

To utilize off-the-shelf LLMs for automatically constructing high-quality SFT data for LQAC, we propose CoF, a post-hoc retrieval- and extraction-based pipeline that obtains precise sentence-level citations from Coarse to Fine. As illustrated in Figure 2, CoF consists of four steps: (1) Given a long context material, CoF first employs the LLM to generate a query and corresponding answer through Self-Instruct Wang et al. (). (2) CoF then uses sentences in the answer to retrieve roughly $k$ chunks from the context, which are subsequently input into the LLM to add coarse-grained chunk-level citations into the answer. (3) Next, the LLM generates fine-grained sentence-level citations for each statement by extracting supporting sentences from the corresponding chunk-level citations. (4) Finally, instances with too few citations are filtered out. In the following, we will introduce each step of CoF in detail and validate its effectiveness on LongBench-Cite.

### 3.1 Pipeline Details

QA Instance Generation.
Considering that generating the answer and citations in one pass might affect answer correctness, we decide to first construct long-context QA pairs and then add citations into the answers in subsequent steps. The post-hoc characteristic also allows our pipeline to augment any long-context QA datasets with citations. For QA instance generation, we adopt the method of Bai et al. (), which first employs the LLM to propose a query according to the given lengthy context and then requests it again to obtain the answer via vanilla long-context QA. They also incorporate different task type descriptions into the prompts (Figure 11), such as summarization, information extraction, and multi-hop reasoning, to guarantee the diversity of generated queries.

Chunk-level Citation Generation.
After constructing the query and answer, we split the context into 128-token chunks and use each sentence in the answer to retrieve $l_{\text{max}}$ chunks. We retain top-$l$ chunks for each sentence, where $l=\min(l_{\text{max}},(k+n_{\text{sent}}-1)/n_{\text{sent}})$ and $n_{\text{sent}}$ denotes the number of sentences, so that about $k$ chunks are retained in total. Then we feed all these chunks, which are sorted according to their position in the context, along with the query and answer into the LLM, and ask the LLM to segment the answer into statements and generate chunk-level citations for each statement using one-shot learning. Figure 12 shows the prompt we use. Compared with generating citations for each statement individually, aggregating all retrieved chunks and generating citations at once can not only reduce the calls of LLM but also improve the citation recall due to the high relevance between the statements.

Sentence-level Citation Extraction.
Besides the coarse granularity, another drawback of chunk-level citation generated in step 2 is that the precise supporting evidence may be located at the beginning or end of the chunk where the sentences are incomplete. Therefore, to achieve fine-grained citations, we first expand each cited chunk by concatenating it with its preceding and succeeding chunks. Next, we retain and number complete sentences in the expanded chunk, and instruct the LLM to extract fine-grained supporting snippets from the chunk by outputting number spans such as [6-8], which refers to the 6th to 8th sentences, or outputting ”No relevant information” if no supporting snippet is found in the chunk. The prompt includes 3 examples and is shown in Figure 13. At last, we remove irregular spans and re-number the others according to the sentence position in the original context to obtain the final sentence-level citations.

Data Filtering.
In the final filtering stage, we discard the instance if less than 20% of the statements in the answer have citations. If an answer has too few citations, we assume it is not factual-grounded enough in the context and may leverage the internal knowledge of LLMs, which often results in hallucinations.

**Table 4: Citation F1 (F1), correctness (C), correctness ratio (CR), and citation length (CL) of different LQAC strategies on LongBench-Cite using GLM-4. We merge MultifieldQA-en/zh for brevity.**
| Method | Avg | Longbench-Chat | MultifieldQA | HotpotQA | Dureader | GovReport |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | CR | CL | F1 | C | CR | F1 | C | CR | F1 | C | CR | F1 | C | CR | F1 | C | CR |  |
| one-pass methods |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LAC-C | 51.6 | 95% | 128.0 | 33.9 | 67.8 | 85% | 55.7 | 87.3 | 99% | 41.2 | 75.3 | 98% | 59.5 | 76.3 | 101% | 67.7 | 59.1 | 90% |
| LAC-S | 65.4 | 95% | 169.0 | 47.1 | 69.4 | 87% | 73.6 | 87.6 | 99% | 44.4 | 76.3 | 100% | 75.0 | 76.0 | 100% | 87.1 | 59.4 | 90% |
| RAC-C | 72.5 | 87% | 128.0 | 69.7 | 59.0 | 74% | 79.1 | 80.7 | 92% | 57.7 | 69.8 | 91% | 75.7 | 77.3 | 102% | 80.3 | 49.9 | 76% |
| RAC-S | 79.1 | 79% | 48.0 | 76.3 | 66.4 | 83% | 86.3 | 85.7 | 97% | 58.1 | 53.3 | 70% | 83.7 | 76.5 | 101% | 91.1 | 29.0 | 44% |
| post-hoc methods |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| post-LC-C | 47.3 | 100% | 128.0 | 27.8 | 79.8 | 100% | 48.2 | 88.1 | 100% | 34.5 | 76.5 | 100% | 52.1 | 75.8 | 100% | 74.1 | 65.9 | 100% |
| post-LC-S | 57.3 | 100% | 147.0 | 34.3 | 79.8 | 100% | 65.3 | 88.1 | 100% | 40.0 | 76.5 | 100% | 64.2 | 75.8 | 100% | 82.8 | 65.9 | 100% |
| post-RC-C | 63.8 | 100% | 128.0 | 61.0 | 79.8 | 100% | 65.3 | 88.1 | 100% | 49.3 | 76.5 | 100% | 67.8 | 75.8 | 100% | 75.8 | 65.9 | 100% |
| post-RC-S | 62.8 | 100% | 48.0 | 63.4 | 79.8 | 100% | 64.8 | 88.1 | 100% | 48.6 | 76.5 | 100% | 69.7 | 75.8 | 100% | 67.5 | 65.9 | 100% |
| CoF | 65.8 | 100% | 89.0 | 66.1 | 79.8 | 100% | 65.6 | 88.1 | 100% | 50.6 | 76.5 | 100% | 67.4 | 75.8 | 100% | 79.1 | 65.9 | 100% |

### 3.2 Pipeline Validation

Before large-scale data construction, we first test CoF (without query generation and final filtering) on LongBench-Cite to validate its effectiveness. We compare CoF with the following LQAC strategies:

- •
LAC-C/LAC-S: the LLM reads the entire context and generates response and chunk-level/sentence-level citation in one pass.
- •
RAC-C/RAC-S: the LLM reads top-$k$ chunks/sentences retrieved using the query and generates response and chunk-level/sentence-level citation in one pass.
- •
post-LC-C/post-LC-S: the LLM first generates a response via vanilla long-context QA, then adds chunk-level/sentence-level citations into the response by finding supporting evidence from the whole context.
- •
post-RC-C/post-RC-S: the LLM first generates a response via vanilla long-context QA, then uses the response to retrieve about $k$ chunks/sentences from the context, and adds chunk-level/sentence-level citations by finding supporting evidence from the retrieved text (similar to step 2 of CoF).

We use GLM-4 as the backbone LLM and Zhipu Embedding-2 as the retriever for all strategies and set retrieval hyper-parameters $l_{\text{max}}=10$ and $k=40$. The results in Table 4 show that:

1. Similar to other post-hoc strategies, CoF is able to preserve the high-quality answers produced through vanilla long-context QA, well preventing correctness degradation. Specifically, GLM-4 perfectly maintains the answers’ original contents unchanged when adding citations, thereby achieving 100% correctness ratios. In contrast, though attaining higher citation F1, one-pass methods typically generate answers with lower correctness, failing to fully leverage LLMs’ long-context QA capacities.

2. CoF achieves the highest citation F1 and relatively small citation length among post-hoc methods, highlighting its ability to generate precise, fine-grained citations. Compared to post-LC-C and post-LC-S, post-hoc retrieval-based methods (i.e., post-RC-C, post-RC-S and CoF) benefit from a more focused evidence search space, typically yielding better performance. Furthermore, CoF’s superiority over post-RC-C indicates that the step of sentence-level citation extraction effectively pinpoints supporting sentences and also filters out irrelevant chunks. Though post-RC-S achieves an even shorter citation length than CoF (49 v.s. 89), we empirically found that direct retrieval-based generation results in too many discontinuous citation numbers, making subsequent training difficult (details in Sec. 4.2.2).

### 3.3 LongCite-45k: a Large-scale SFT Dataset for LQAC

After validating the efficacy of CoF, we utilize this framework to construct LongCite-45k, a large-scale SFT dataset for LQAC. Specifically, we first collect 50k documents from the pre-training corpus of GLM-4, covering 9 varied domains including books, encyclopedias, academic papers, codes, etc. These documents are mainly in English and Chinese and their lengths range from 256 to 128k tokens. We then apply CoF, using GLM-4 as the backbone LLM and Zhipu Embedding-v2 as the retriever, to generate a QA pair with sentence-level citations for each document, resulting in 44,600 high-quality LQAC instances after the filtering stage. As illustrated in Figure 2(d), the input part of each instance consists of a task instruction, a long document, and a query, and the output part is an answer with sentence-level citations.

## 4 LongCite: Teach Long-context LLMs to Generate Citations

In this section, we conduct model training experiments to determine whether SFT on LongCite-45k can enhance LLMs’ ability for the LQAC task, enabling them to generate both accurate responses and precise citations within a single output. We discuss the training details and experimental results as follows.

### 4.1 Training Details

We select two latest open-source base models, namely GLM-4-9B (Zeng et al., ) and Llama-3.1-8B (Vavekanand & Sam, ), for the training experiments. Both of the two models have been continually pre-trained on lengthy texts and support a context window of 128k tokens, thereby being suitable for SFT on LQAC data. Following Bai et al. (), we combine LongCite-45k with 76k general SFT instances from ShareGPT (Chiang et al., ) to ensure the model’s general capacities. We name the models after SFT as LongCite-9B (abbr. for GLM-4-9B-LongCite) and LongCite-8B (abbr. for Llama-3.1-8B-LongCite).

Meanwhile, to investigate whether SFT on LQAC data will influence models’ long-context QA correctness compared to standard long-context SFT (i.e., SFT on vanilla long-context QA data), we additionally train the two base models using the pure long-context QA pairs (without the task instruction and citations) in LongCite-45k, and we name the trained models as LongSFT-9B (abbr. for GLM-4-9B-LongSFT) and LongSFT-8B (abbr. for Llama-3.1-8B-LongSFT). When calculating correctness ratios for LongCite-9B/8B, we use LongSFT-9B/8B to obtain the correctness in vanilla long-context QA setting (i.e., $\text{C}_{\text{LQA}}$).

All models are trained using 4 nodes with 8$\times$H800 80G GPUs. We adopt Megatron-LM (Shoeybi et al., ) with context parallelism to support a maximum training sequence length of 128k tokens, and use packing training with loss weighting (Bai et al., ) to improve training efficiency. We set the batch size to 8 and the learning rate to 1e-5. We train each model for 4,000 steps, which is about 2 epochs and takes 18 hours.

### 4.2 Experimental Results

#### 4.2.1 Main Results

We show the citation quality and correctness of our trained models on LongBench-Cite in Table 2 and 3, respectively. Here are our main findings:

1. LongCite-8B and LongCite-9B achieve the best citation qualities among all models. Compared to three powerful proprietary models, i.e., GPT-4o, Claude-3-Sonnet, and GLM-4, LongCite-8B/9B improves the overall citation F1 by 6.4/3.6, 4.8/2.0, and 6.6/3.8, respectively. Besides, the average citation length of LongCite-8B and LongCite-9B is also significantly shorter than that of proprietary models and chunk-level citations, indicating finer citation granularity. Surprisingly, LongCite-8B and LongCite-9B even attain higher citation F1 than the data construction pipeline CoF (72.0 and 69.2 v.s. 65.8), implying a potential for continuous self-improvement. In addition, the similar citation length between the trained models and CoF demonstrates that not only the evidence-locating skill but also the citation granularity can be learned through SFT.

Specially, we observe a performance gap between LongCite models and proprietary models on GovReport. We attribute this to the coarser citation granularity of proprietary models (e.g., the citation lengths of GLM-4 and LongCite-9B on GovReport are 188 and 86, respectively), which gives them unfair advantages in the evaluation of citation recall and precision.

2. SFT with citation information further boosts the long-context QA correctness. Different from in-context LQAC where the LLMs typically generate responses with lower correctness than vanilla long-context QA (Sec.  2.4), SFT on LQAC data consistently improves the response correctness on all the datasets compared to vanilla long-context SFT (i.e., $\text{CR}>100\%$). In particular, the correctness of LongCite-8B/9B increased by 16%/28% over LongSFT-8B/9B. Besides, the overall correctness of our trained model is also comparable with the officially post-trained models (i.e., GLM-4-9B-chat and Llama-3.1-8B-Instruct), validating the rationality of generating long-context QA pairs using self-instruct in our CoF pipeline.

To further explore the reasons for the correctness improvement, we manually compared the responses generated by LongCite-9B and LongSFT-9B and found that the improvement mainly comes from two aspects (we present 3 cases in Table 9, 10, and 11 to illustrate our interpretation): (1) SFT with citation information enhances the evidence locating ability of the model and helps to prevent from hallucination; (2) LongCite models can utilize context information more uniformaly. Specifically, when faced with a query that requires a global view, the generated citation numbers allow LongCite models to be aware of that current response content has covered which parts of the context, so that they can utilize different parts of context more uniformly, resulting in a more comprehensive response. In contrast, LongSFT models tend to use more information from the head part of the context and only roughly utilize or even ignore the rest of the context.

Figure: Table 5: Performance of models using different training data on LongBench-Chat.
Refer to caption: x3.png

#### 4.2.2 Further Analysis

Ablation on LongCite-45k dataset. To verify that the enhanced LQAC ability is obtained from the LongCite-45k dataset instead of standard long-context SFT, we test LQAC performance of LongSFT-9B on LongBench-Chat using the one-shot prompt in Sec. 2.4. The results in Table 5 indicate that LongSFT-9B performs poorly on LQAC task. Similar to the open-sourced LLMs, LongSFT-9B always generates nonconforming citations or no citations.

Ablation on data filtering. To demonstrate the effect of data filtering in CoF pipeline, we train LongCite-9B with the unfiltered data. Table 5 shows that data filtering effectively improves citation quality.

Comparison with data constructed through post-RAC-S strategy. We attempt constructing LQAC data by applying post-RAC-S strategy, whose performance is comparable with CoF (Sec. 3.2), to add citations for the QA pairs in LongCite-45k. However, as shown in Table 5, the model trained with post-RAC-S data achieves much worse citation F1 than LongCite-9B. We believe the main reason is that post-RAC-S directly recalls sentences that are not necessarily adjacent from the context, resulting in many discontinuous citation numbers (such as [3][7][15]…), which makes subsequent training difficult. In contrast, CoF extracts sentence-level citations from bigger chunk-level snippets and uses number spans to represent citations. These methods contribute to maintaining the semantic coherence of the cited information, which is advantageous for training purposes.

Correlation between correctness and citation quality. To explore the correlation between correctness and citation quality, we divide LongCite-9B’s responses on LongBench-Cite into three groups according to their correctness and compute the mean and standard deviation of citation F1 for each group. As illustrated in Figure 3, responses with higher correctness typically have higher citation qualities, demonstrating a mutually promoting relationship between these two attributes.

**Table 6: Citation quality evaluated by human, GPT-4o and ALCE on LongBench-Chat.**
| Model | Human scores | GPT-4o scores | ALCE scores |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R | P | F1 | R | P | F1 | R | P | F1 |  |
| GLM-4 | 61.2 | 67.5 | 60.2 | 47.6 | 53.9 | 47.1 | 46.1 | 29.1 | 30.8 |
| LongCite-8B | 79.6 | 88.9 | 82.6 | 62.0 | 79.7 | 67.4 | \ul59.6 | \ul39.5 | \ul42.0 |
| LongCite-9B | \ul72.8 | \ul84.2 | \ul75.8 | \ul57.6 | \ul78.1 | \ul63.6 | 64.2 | 45.1 | 47.1 |

### 4.3 Human Evaluation

To verify that our automatic evaluation of citation quality using GPT-4o correlates with human judgment, we conduct a human evaluation on three models: GLM-4, LongCite-8B, and LongCite-9B. Specifically, we anonymized their responses on LongBench-Chat, including 150 responses, 1,064 statements, and 909 citations in total, and manually annotated the citation recall and precision following the same standard as GPT-4o evaluation. We also compare GPT-4o evaluation with ALCE (Gao et al., ), which utilizes NLI model TRUE (Honovich et al., ) to measure citation recall and precision. As shown in Table 7, the relative rankings produced by human and GPT-4o are consistent, indicating that improvements in GPT-4o scores also reflect improvements in human preferences. In addition, the absolute scores from GPT-4o typically aligned more closely with human scores compared to ALCE. On the other hand, we observed that GPT-4o scores are generally lower than human scores because the cited snippets often contain unclear pronouns like “he/she” and “our method”. We believe that incorporating an anaphora resolution step may alleviate this problem but will also increase the evaluation costs. Furthermore, the Cohen’s kappa coefficients between GPT-4o and human are significantly higher compared to ALCE (Table 7), demonstrating a substantial agreement for citation recall (0.593 when treating “partially support” as “not support” following ALCE) and citation precision (0.655). When taking human annotations as gold labels, GPT-4o also achieves high accuracy (75.0% for citation recall and 88.8% for precision).

## 5 Related Works

Long-context LLMs.
A mature approach for extending the context window of LLMs involves continued pre-training of base LLMs on extensive long texts followed by alignment using diverse long-context QA pairs (Cai et al., ; Zeng et al., ; Vavekanand & Sam, ). However, because of the difficulty of annotations, most long-context QA data is automatically synthesized by LLMs themselves (Bai et al., ; Xiong et al., ), which cannot strictly guarantee the faithfulness of the answers. This leads to potential hallucinations of the aligned LLMs, i.e., fabricating content not present in or consistent with the context. Therefore, users often require a way to verify the accuracy and reliability of the information provided by LLMs, especially in highly sensitive domains such as law and finance. Our work explores how to enable long-text models to produce responses with fine-grained citations, thereby enhancing the verifiability and trustworthiness of the long-context LLMs.

Question Answering with Citations.
Recently, question answering with citations has been extensively studied in the fields of open-domain QA (Nakano et al., ; Bohnet et al., ; Gao et al., ; ; Schimanski et al., ). Most of these methods rely on retrieval-augmented generation or post-hoc processing, which are not well-suited for long-context QA scenarios due to information loss or excessive latency. Furthermore, the coarse granularity of their citations negatively impacts user experience (Huang et al., ). Our work, however, leverages long-context LLMs to generate responses and precise sentence-level citations in a single pass, providing advantages in terms of response correctness, efficiency, and user friendliness. Moreover, current methods for citation evaluation largely depend on NLI models that have limited capacities (Honovich et al., ; Liu et al., ; Gao et al., ; Fierro et al., ). In contrast, we utilize GPT-4o as a judge and consider more complex scenarios, thereby achieving a higher agreement with human assessments.

## 6 Conclusion

In this work, we explore enhancing LLMs’ capacity to generate fine-grained citations from lengthy contexts. We first propose LongBench-Cite, an automatic benchmark to reveal current LLMs’ limited performance on long-context question answering with citations (LQAC). We then introduce CoF, a novel pipeline that uses off-the-shelf LLMs to automatically generate long-context QA instances with precise sentence-level citations, to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we successfully train LongCite-8B and LongCite-9B with LongCite-45k, allowing the generation of accurate responses and fine-grained citations in one pass. Extensive analyses and human evaluation further verify the effectiveness of our approach. We believe that this work lays a solid foundation for further research on LQAC and contributes to the development of more reliable and trustworthy LLMs.

## Appendix A Model Cards

We list the details of our evaluated models in Table 8.

**Table 8: Model cards.**
| Model name | Model version | Context window |
| --- | --- | --- |
| Claude-3-Sonnet (Anthropic, ) | claude-3-sonnet-20240229 | 200,000 tokens |
| GPT-4o (OpenAI, ) | gpt-4o-2024-05-13 | 128,000 tokens |
| GLM-4 (Zeng et al., ) | GLM-4-0520 | 128,000 tokens |
| GLM-4-9B-chat (Zeng et al., ) | - | 128,000 tokens |
| Llama-3.1-8B-Instruct (Vavekanand & Sam, ) | - | 128,000 tokens |
| Llama-3.1-70B-Instruct (Vavekanand & Sam, ) | - | 128,000 tokens |
| Mistral-Large-Instruct (Jiang et al., ) | Mistral-Large-Instruct-2407 | 128,000 tokens |

## Appendix B Case Study

We present three cases in Table 9, 10, and 11 to help interpret the improvement of correctness (the detail interpretation is in Sec. 4.2.1).

**Table 9: Case study. LongSFT-9B hallucinates the office location of Duke Energy, directly copying that of Affiliated Managers Group, while LongCite-9B gets the correct answer due to SFT with citations. We mark the wrong and correct statements in red and green, respectively.**
| Query: Are both Duke Energy and Affiliated Managers Group based in Massachusetts? |
| --- |
| LongSFT-9B:<br>Based on the information provided, Duke Energy has offices in several locations including West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom. Affiliated Managers Group has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom. So they both have an office in Massachusetts. |
| LongCite-9B:<br><statement>Based on the information provided, Duke Energy is headquartered in Charlotte, North Carolina, while Affiliated Managers Group has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom.<cite>[25-25][219-219]</cite></statement> <statement>So only Affiliated Managers Group is based in Massachusetts.<cite></cite></statement><br>Citations:<br>[25-25]: Duke Energy Corporation is an American electric power and natural gas holding company headquartered in Charlotte, North Carolina.<br>[219-219]: (NYSE: AMG) is a strategic partner to independent investment management firms globally, with equity stakes in a number of partner-owned traditional investment managers, hedge funds, and specialized private equity firms which it calls “Affiliates.” The company has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom. |

**Table 10: Case study. LongCite-9B utilize more information from the middle part of the context, resulting in a more detailed response than LongSFT-9B. We mark the coarse and detailed counterparts in their response in red and green, respectively**
| Query: Please write a one-page summary of the above government report. |
| --- |
| LongSFT-9B:<br>The Department of Homeland Security (DHS) has established policies and processes for managing its multi-billion dollar acquisition investments. These policies and processes are primarily set forth in its Acquisition Management Directive 102-01 and Acquisition Management Instruction 102-01-001. The department uses these policies and processes to deliver systems that are intended to close critical capability gaps, helping enable DHS to execute its missions and achieve its goals.<br>The DHS acquisition management policy establishes that a major acquisition program’s decision authority shall review the program at a series of predetermined acquisition decision events to assess whether the major program is ready to proceed through the acquisition life-cycle phases. Depending on the program, these events can occur within months of each other, or be spread over several years.<br>The report found that 10 of the 24 programs assessed were on track to meet their schedule and cost goals, while 14 programs were not on track. The report made two recommendations to DHS to improve its acquisition management policies and processes. |
| LongCite-9B:<br><statement>The report assesses the Department of Homeland Security’s (DHS) major acquisition programs and their progress towards meeting schedule and cost goals. It also evaluates DHS’s policies and processes for acquisition management, resource allocation, and requirements development to determine if they reflect key portfolio management practices.<cite></cite></statement><br><statement>Key findings include:<cite></cite></statement><br><statement>- Of the 28 programs reviewed, only 10 were on track to meet their schedule and cost goals as of January 2018. The remaining 14 programs had slipped on schedule, cost, or both.<cite>[89-97]</cite></statement><br><statement>- The most common reasons for schedule slips were new requirements, technical challenges, and external factors like staffing shortages or funding cuts.<cite>[105-106][108-108]</cite></statement><br><statement>- Some programs had cost breaches due to scope increases, technical challenges, or contractor labor costs.<cite>[119-119][121-121][127-127]</cite></statement><br><statement>- DHS has taken steps to improve its policies and processes, but more is needed. For example, it should require components to submit affordability information when programs re-baseline due to breaches.<cite>[178-178][234-234][258-258]</cite></statement><br><statement>- The report makes two recommendations for DHS to improve its acquisition management: require components to submit affordability information when programs re-baseline and assess the results of post-implementation reviews to identify opportunities to improve performance across the portfolio.<cite>[258-259]</cite></statement><br><statement>- DHS concurred with the recommendations and has identified actions it will take to address them.<cite>[261-261]</cite></statement> |

**Table 11: Case Study. The citation numbers in the response make LongCite-9B aware of which parts of the document the current response has covered and thus help it utilize context information uniformly to generate a more comprehensive summary, while LongSFT-9B only focuses on the front part of the extensive document and ignores the rest parts.**
| Query: Please write a one-page summary of the above government report. |
| --- |
| LongSFT-9B:<br>The US Agency for International Development (USAID) has obligated over $5 billion in bilateral assistance to the Palestinians in the West Bank and Gaza since 1993, primarily using funds appropriated through the Economic Support Fund (ESF). USAID provides project assistance and debt relief payments to Palestinian Authority (PA) creditors, and implements most ESF programs, including those related to private sector development, health, water and road infrastructure, local governance, civil society, rule of law, education, and youth development. USAID assistance to the West Bank and Gaza is conducted under antiterrorism policies and procedures outlined in an administrative policy document known as Mission Order 21. As of March 31, 2018, USAID had obligated about $544.1 million (over 99 percent) and expended about $350.6 million (over 64 percent) of approximately $544.5 million in ESF assistance allocated for the West Bank and Gaza in fiscal years 2015 and 2016. |
| LongCite-9B:<br><statement>The report discusses USAID’s Economic Support Fund (ESF) assistance to the Palestinians in the West Bank and Gaza from FY2015-FY2016. Key points include:<cite></cite></statement><br><statement>- USAID obligated over $544 million in ESF assistance for the West Bank and Gaza in FY2015-FY2016, with about $350 million expended.<cite>[8-8]</cite></statement><br><statement>- Most assistance went to project assistance ($215 million in FY2015, $184 million in FY2016) for economic growth, infrastructure, education, health, etc. Some went to debt relief payments to PA creditors ($75 million in FY2015, $70 million in FY2016).<cite>[9-11]</cite></statement><br><statement>- USAID conducted required vetting of PA creditors to ensure assistance did not support terrorism. Vetting occurred within 12 months before debt relief agreements were signed.<cite>[36-36][40-40][42-42]</cite></statement><br><statement>- External assessments identified weaknesses in PA financial systems for medical referrals and petroleum procurement. USAID took additional steps to mitigate risks.<cite>[52-52][62-62]</cite></statement><br><statement>- Financial audits of the debt relief grants did not identify any issues.<cite>[66-66][70-70]</cite></statement><br><statement>- USAID does not plan to resume direct cash payments to the PA due to concerns about misuse. Payments to creditors are considered more transparent.<cite>[49-49]</cite></statement><br><statement>In summary, the report concludes that USAID generally followed proper procedures for providing ESF assistance to the Palestinians in the West Bank and Gaza, with some concerns about PA financial systems. Debt relief payments to creditors were properly vetted.<cite></cite></statement> |

## Appendix C Prompts

Figure: Figure 4: prompt for correctness evaluation on LongBench-Chat.

Figure: Figure 5: Prompt for correctness evaluation on MultiFieldQA-zh/en, HotpotQA, and Dureader.

Figure: Figure 6: Prompt for correctness evaluation on GovReport.

Figure: Figure 7: Prompt for evaluating citation recall when the statement has at least one citation.

Figure: Figure 8: Prompt for evaluating citation recall when the statement has no citation.

Figure: Figure 9: Prompt for evaluating citation precision.

Figure: Figure 10: One-shot learning prompt for the LAC-S strategy.

Figure: Figure 11: Prompt for English question generation in the CoF pipeline. For each long text material, we randomly select one of the four task prompts and let the LLM generate five questions to ensure that the questions cover content from multiple spans within the long text. We then randomly choose one of these questions. For long Chinese documents, we translate the corresponding prompts into Chinese and obtain Chinese questions.

Figure: Figure 12: Prompt for chunk-level citation generation in the CoF pipeline.

Figure: Figure 13: Prompt for sentence-level citation extraction in the CoF pipeline.

We list the prompts used in this work in Figure 4, 5, 6, 7, 8, 9, 10, 11, 12, 13.

## Appendix D Evaluation Cost

On LongBench-Cite, a run of GPT-4o evaluation for correctness/citation quality costs about $4/$25.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - Construction of LongBench-Cite: Is LongBench-Cite just a concatenation of existing datasets? Is there any additional processing to create LongBench-Cite examples (and if so what is it)? If it is just a concatenation, then what is the advantage of combining it into one dataset vs. evaluating on them each separately? 
- Can you explain precisely what is being compared in the Cohen’s Kappa calculation? Is it correctness at the response level, or citation quality at the sentence level or something else?
- Can we just use CoF to do the LQAC task? What would be the disadvantage? 
- Not a question: When you list multiple Figures, often only one is hyperlinked in the text; all should be hyperlinked
- If you use GPT-4o for all of the evaluation steps, is it fair to compare GPT-4o (and have it evaluate itself)?
- Did you notice any “lost-in-the-middle” type phenomenon (https://arxiv.org/abs/2307.03172)?
- How much do you think the format of the citations you are asking for affects the sentence-level citations results of proprietary models? In other words, do you think there is a way to reformulate the task (prompt) such that those models would perform much better? Did you try many different prompts?
- Is it possible/necessary for your models to cite non-contiguous sentences, e.g., [1-2;5] ?
soundness: 3
strengths: **Interesting Task.** Studying fine-grained citations (at the sentence-level) is interesting. Efforts in developing models with the capability to provide fine-grained citations is useful. In particular, this could help with evaluation of the veracity of model outputs

**Results.** The results show that the fine-tuned models generally outperform proprietary models in long-context QA with citations, which is a nice result. Furthermore, results seem to imply that proprietary models may have been adequate at chunk-level citations but struggle more with sentence level citation. Finally, the "correctness ratio" measurement, which compares how models perform with and without having to cite, is interesting and clearly shows shows that making the models output citations alongside correct answers seems to degrade model performance on QA. 

**Complete Work.** The paper presents a relatively complete exploration of an idea: a new variant of QA with citations is proposed; an evaluation metric for the task is proposed; existing models are evaluated on the task; a dataset is constructed for fine-tuning, and used to train new models for the task; those models perform well; analysis is presented.

**Human evaluation** It’s great to have human evaluation. I liked the analysis of LongCite-9B vs LongSFT-9B; I think it could be fleshed out more. The manual evaluation of citation recall/precision strengthens the paper, although I think the correlation is not entirely convincing since--if I understand correctly--it only includes 3 data points (GLM-4, LongCite-8B and -9B). The presentation could perhaps be strengthened; it took me some time to correctly interpret the correlation from the table. The Cohen’s Kappa result is more convincing, although I wasn’t sure how it was computed (questions below).
summary: This paper introduces LongBench-Cite, a Long-Context QA benchmark that requires models to add citations in their responses. Different from other similar works, this work evaluates citations at the sentence level, rather than at the chunk level. That is, models in this task are supposed to cite specific sentences in context that support their responses rather than documents or large chunks of text. The authors also construct LongCite-45k, which is a supervised fine-tuning dataset for the task studied, and use this dataset to train two models. Their 8B and 9B parameter, fine-tuned models outperforms GPT-4o. The results of their experimentation show that existing models are not proficient in sentence level citation. Moreover, requiring sentence level citations degrades model performance. Finally, fine-tuning improves model capability in QA and sentence level citations a reasonable amount.
weaknesses: **Clarity and Missing Details.**  The writing is a bit hard to follow at times, for example in the description of the construction of the dataset (see questions below). Another area that was hard to follow were the pipeline details. Specifically, what retrieval method is used to retrieve chunks in the chunk-level citation generation (I know it's Zhipu, but could more details be provided)? How were the hyperparameters chosen (l, k, & n)? In pipeline validation, how are the evaluations carried out (done manually? Or automatically)? 

**Over-reliance on LLM Evaluation.** Either GPT-4o or GLM-4  is used for: 1) evaluating correctness of QA answers, 2) evaluating correctness of citation (precision and recall), 3) filtering out starting, transition, summary and reasoning sentences, 4) all steps of the CoF pipeline for dataset construction. Your human evaluation showing correlation between GPT-4o evaluations and human evaluations is critical to dispelling some concern over possible systemic errors from GPT-4o corrupting your results, but the concern isn’t totally alleviated. For example, it seems like one way to get a high score is to emit many sentences that GPT-4o would deem “starting, transition, summary or reasoning” since these need no citations according to the evaluation scheme. I think a discussion of this reliance and where things could go wrong should be included in a limitation section, or additional measurements and/or human evaluations should be conducted to address potential issues.

**Missing Discussion of Sensitivity to Prompt.** Since the proprietary models are only prompted, and the task itself has a somewhat atypical format (e.g., with each sentence tagged with an index) it would be good to discuss whether any tuning of the prompts was conducted (especially for the proprietary models) and/or any observations related to the sensitivity of the models to the prompt. This would help alleviate concern that the proprietary may have performed better if only their prompts were tuned better.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Typo: Line 82: "LongBench-Chat" should be corrected to "LongBench-Cite".
2. Typo: In Figure 4, lines 802-803, "Example Answer 2" should be corrected to "Example Answer 3".
3. Why are the standards for correctness evaluation inconsistent across different datasets in LongBench-Cite, and why are the scoring ranges different? For example, LongBench-Chat does not consider comprehensiveness, and its score range is 1-10, while GovReport’s score range is 1-5, and others are 1-3. Additionally, why does the evaluation of LongBench-Chat require a few-shot approach, while others are zero-shot?
4. How is the correctness score in Table 3 calculated? Is it the average score across the entire dataset? However, the correctness score range varies across different datasets. Was this unified into a percentage scale here? The authors did not clearly explain this.
5. Why is the performance of GLM-4-9B-chat on the GovReport dataset exceptionally weak in Table 2? Was there any specific analysis conducted?
6. Compared to chunk-level citation, are there any other statistics changes in sentence-level citations, such as the number of citations or the proportion of a statement citing multiple sources
soundness: 2
strengths: 1. The structure of this paper is clear and easy to understand
2. The experimental design (choice of model, dataset, and evaluation criteria) is reasonable.
3. The author noticed that adding citations can harm response correctness, which was not mentioned in previous work
4. Both human evaluation and experiments demonstrate the effectiveness of the trained LongCite-8B and LongCite-9B
summary: This paper first evaluates the performance of LLMs in generating sentence-level citations in long-context QA. Then, they distill GLM-4 to construct high-quality fine-grained citation data and fine-tune GLM-4-9B and Llama-3-8B to generate sentence-level citations. Experimental results show that simply SFT can improve citation quality and response correctness.
weaknesses: 1. Lack of significant novelty: Firstly, the concepts of long-context LLM citation [1] and fine-grained citation [2][3] have already been discussed in previous works. This paper merely combines these two concepts, stitching them together into a single paper. More importantly, the authors even lack discussion and comparison of these important prior works. Secondly, the implementation approach for fine-grained citation heavily overlaps with [2][3], as both rely on distilling advanced LLMs to extract fine-grained spans. The main difference is that this paper extends the context length and distinguishes citation forms by splitting the context into sentences.
2. Unoriginal conclusions: Previous works on attributed text generation have already demonstrated that simply SFT can significantly enhance a model's attribution abilities [2][3][4]. I did not learn anything novel from the authors' conclusions, with the different prerequisite being a long-context.
3. Insufficient benchmarking: The paper only adopts the LAC-S strategy for benchmarking, which is insufficient. It is difficult to deduce whether the model’s poor sentence-level citation quality is more influenced by poor instruction-following or weak evidence-searching ability based on these evaluations.
4. Lack of baseline comparisons: The method in [2] achieves sentence-level citation through distilling Gemini, using different models for content selection and sentence planning. The authors lack a fair comparison with such method.
5. Evaluation of correctness: The authors’ evaluation of correctness includes both accuracy and comprehensiveness, which are actually different dimensions but equally important. These should be separately assessed instead of being unified under correctness. Otherwise, it becomes challenging to discern whether fluctuations in correctness are due to errors or insufficient coverage. The causes and consequences of these two are entirely different. I would prefer referring to correctness as response quality or overall quality.
6. High cost of constructing SFT data: The entire data construction process heavily relies on calling the proprietary model GLM-4, which is not only high-cost but also hard to reproduce.
7. Quality of the question and answer generated by self-instruct: This is crucial for the diversity and quality of CoF data, but the quality cannot be guaranteed. The authors lack specific analysis of the quality of the questions and answers generated in the first step. Will the generated questions focus too much on local information in the background, resulting in insufficient answers?
8. Control over granularity: LongCite-45k contains 128,000 tokens. Assuming an average sentence length of around 20-30 tokens, this would result in over 4,000 sentence indices, which is too fine-grained. I am concerned about its practicality in real-world usage.

I am willing to improve the score if the author can effectively address these concerns. 

[1] Attribute or Abstain: Large Language Models as Long Document Assistants. Buchmann et al. EMNLP 2024

[2] Attribute First, then Generate: Locally-attributable Grounded Text Generation. Slobodkin et al. ACL 2024

[3] Learning Fine-Grained Grounded Citations for Attributed Large Language Models. Huang et al. Findings of ACL 2024

[4] Training Language Models to Generate Text with Citations via Fine-grained Rewards. Huang et al. ACL 2024

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: If the authors can clearly and explicitly explain the unique challenges addressed by this work in the long-context scenario, I would consider increasing the score.
soundness: 3
strengths: 1. The overall structure is complete, and the writing is well-organized.
2. A comprehensive body of work has been conducted for the LQAC task, including data construction and model training.
summary: This paper primarily focuses on the task of Long-context QA with Citation, including the following key points:
1. adoption of sentence-level citation; 
2. proposal of an automated data construction method; 
3. improved model performance on the LQAC task after training.
weaknesses: Indeed, as mentioned above, this paper presents a complete body of work around LQAC, with comprehensive content that is self-contained. However, the core issue with this paper lies in its low level of innovation:

1. The proposed LQAC task essentially combines aspects from *short-text QA*, *fine-grained citation*, and *long-context generation with citation*, making the innovation somewhat incremental.
2. I believe that in addition to discussing the general challenges of generation with citation, research on the LQAC task should emphasize the unique characteristics of **long-context scenarios**, such as *citation consistency across contexts*, *citation redundancy*, and *potential contradictions in citations*. This would better highlight the differences between this work and prior research.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: Could you explain why the chunk size must be fixed at 128 tokens instead of matching the sentence length?
soundness: 2
strengths: - The paper is well-structured, effectively integrating the benchmark, training data, and fine-tuned models for the LQAC task.

- The evaluation results on LongBenchCite demonstrate that the fine-tuned models achieve state-of-the-art citation quality, surpassing even advanced proprietary models, including GPT-4o.
summary: This paper focuses on the Long-Context Question Answering with Citations (LQAC) task. To assess large language models (LLMs) in generating accurate answers with appropriate sentence-level citations, the authors introduce an automated benchmark called **LongBench-Cite**. They further develop a supervised fine-tuning dataset **LongCite-45k** using a **Coarse to Fine (CoF) pipeline**. Leveraging this SFT dataset, they train **two LLMs** that achieve state-of-the-art citation quality on their curated benchmark.
weaknesses: - **Lack of Motivation and Novelty**: In addition to the coarse granularity issue, the authors point out that the chunk-level citation may also result in incomplete sentences. This problem arises because the authors have rigidly divided the context into fixed-length chunks of 128 tokens. A potential solution could be splitting the text based on sentence terminators, such as periods or semicolons. Given that the authors assert that sentence-level citation is superior to chunk-level citation, what motivates the inclusion of chunk-level citation in the CoF pipeline? Additionally, since most methodologies (Evaluation Method / Dataset Construction / Model Training) in this paper are adapted from existing work [1], the novelty is unclear.

- **Obscure and Inconsistent Noun Naming.** There are several confusing noun pairs in the paper. The authors should read through their paper before submission. For instance:
    - *(LongBench-Chat, LongBench-Cite):* In the abstract (line 17), the authors state, "We first introduce LongBench-Cite, ...," whereas in the introduction (line 82), they refer to "We introduce LongBench-Chat, ...". This inconsistency is perplexing, and it is unclear if one of these is meant to reference a published work [1].
    - *(LongCite-9B, LongSFT-9B) and (LongCite-8B, LongSFT-8B):* The naming convention is confusing, making it hard for readers to discern whether these models are synonymous or represent distinct concepts.

[1] Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., ... & Li, J. (2024). Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058.