LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA
Authors: Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, Juanzi Li
Accepted at: acl25

LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-Context QA

Jiajie Zhang1†, Yushi Bai1†, Xin Lv2, Wanjun Gu1, Danqing Liu1, Minhao Zou1 Shulin Cao2, Lei Hou1, Yuxiao Dong1, Ling Feng1, Juanzi Li1*
1Tsinghua University   2Zhipu AI

*Work done when JZ and YB interned at Zhipu.AI.
*Corresponding author.

Abstract

Though current long-context large language models (LLMs) have demonstrated impressive capacities in answering various questions based on extensive text, the lack of citations in their responses makes user verification difficult, leading to concerns about their trustworthiness due to the potential hallucinations. In this work, we aim to enable long-context LLMs to generate responses with fine-grained sentence-level citations on the fly, thereby improving their faithfulness and verifiability. We first introduce LongBench-Cite, an automated benchmark for assessing current LLMs’ performance in long-context question answering with citations (LQAC), revealing considerable room for improvement. To this end, we propose CoF (Coarse to Fine), a novel pipeline that utilizes off-the-shelf LLMs to automatically construct long-context QA instances with precise sentence-level citations, and leverage this pipeline to construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we train LongCite-8B and LongCite-9B using the constructed dataset, successfully enabling the generation of accurate responses and fine-grained citations in one pass. The evaluation results on LongBench-Cite show that our trained models achieve state-of-the-art citation quality, surpassing advanced proprietary models including GPT-4o. We also discover that SFT with citation information can further improve the correctness of model responses compared to standard long-context SFT.

1 Introduction

Recent years have witnessed significant advancement in long-context large language models (LLMs), enabling them to address various user questions based on lengthy texts that surpass 100,000 tokens (Anthropic, 2024b; Zeng et al., 2024; Reid et al., 2024). Despite their remarkable capacities, current long-context LLMs typically do not provide citations to specific context snippets to support the statements they generated, making it challenging for users to verify model outputs given the substantial context lengths. This significantly impacts the reliability and trustworthiness of long-context LLMs, especially considering that they still struggle with hallucinations (Huang et al., 2023) and are prone to generate unfaithful content.

On the other hand, recent works in search engines and open-domain QA have allowed LLMs to generate responses with in-line citations through retrieval-based generation (RAG) or post-hoc methods (Nakano et al., 2021; Gao et al., 2023a,b; Menick et al., 2022). Nevertheless, as shown in Figure 1, these approaches still expose notable limitations in long-context scenario: RAG often leads to compromised response quality due to incomplete context information, while post-hoc methods prolong the user waiting time due to more complicated pipelines. In addition, their generated citations typically refer to entire web pages (Nakano et al., 2021) or coarsely chunked snippets (Gao et al., 2023b), thereby requiring users to further locate the specific supporting evidence for the final verification.

In light of these challenges, we are curious: Can we directly employ long-context LLMs to generate responses with sentence-level citations on the fly based on the whole context, thereby ensuring high response quality, fine citation granularity, and normal user waiting time simultaneously? To this end, we first propose LongBench-Cite, an automatic benchmark to evaluate LLMs’ performance on the task of long-context question answering with citations (LQAC), and find that existing long-context LLMs obtain unsatisfactory results (Sec. 3). Specifically, we find that citations produced by these LLMs either cannot fully support their response, or have a coarse granularity. Meanwhile,
we observe that generating citations on the fly via in-context learning generally results in less correct responses compared to vanilla long-context QA.

To further enhance the inherent capacity of LLMs for generating fine-grained citations from lengthy contexts, it is essential to construct a high-quality SFT dataset. To this end, we introduce CoF (abbr. for “Coarse to Fine”), a novel pipeline that utilizes off-the-shelf LLMs to automatically construct long-context QA instances with precise sentence-level citations (Sec. 4). CoF comprises four stages: (1) Starting with a long text material, CoF first invokes the LLM to produce a query and its associated answer through Self-Instruct (Wang et al., 2023). (2) Next, CoF uses the answer to retrieve several chunks from the context, which are then fed into the LLM to incorporate coarse-grained chunk-level citations into the answer. (3) The LLM subsequently identifies evidence sentences from each cited chunk to produce fine-grained citations. (4) As a final step, instances with insufficient citations are discarded. Our experiments validate the superiority of CoF over other LQAC strategies in terms of answer correctness and citation quality. Using CoF pipeline, we construct LongCite-45k, a large-scale SFT dataset that consists of 44,600 high-quality LQAC instances with contexts up to 128,000 tokens.

Finally, we use LongCite-45k to fine-tune GLM-4-9B (Zeng et al., 2024) and Llama3.1-8B (Vavekanand and Sam, 2024), two latest open-source long-context models (Sec. 5). The enhanced models, namely LongCite-9B and LongCite-8B, support a max context window of 128,000 tokens and are capable of generating accurate responses along with precise, fine-grained citations in one pass. Evaluation on LongBench-Cite indicates that our trained models achieve significantly better citation quality compared to even much larger proprietary models. Specifically, our 8B/9B size model outperforms GPT-4o by 6.4%/3.6% in terms of citation F1 score and achieves twice finer granularity. Meanwhile, we observe that SFT with citation information can alleviate hallucinations of LLMs and enable them to utilize context information more uniformly and comprehensively, instead of only focusing on a specific part of the context. This results in a further improvement in response correctness over standard long-context SFT. We also conduct extensive analyses and human evaluation to further verify the effectiveness of our approach.

2 Related Works

Long-context LLMs. The current mainstream approach for extending the context window of LLMs involves continued pre-training of base LLMs on extensive long texts followed by alignment using diverse long-context QA pairs (Zeng et al., 2024; Vavekanand and Sam, 2024). However, because of the difficulty of annotations, most long-context QA data is automatically synthesized by LLMs themselves (Bai et al., 2024; Xiong et al., 2023), which cannot strictly guarantee the faithfulness of the answers. This leads to potential hallucinations of the aligned LLMs, i.e., fabricating content not present in or consistent with the context. Therefore, users often require a way to verify the accuracy and reliability of the information provided by LLMs. Our
work explores how to enable long-context models to produce responses with fine-grained citations, thereby enhancing the verifiability and trustworthiness of long-context LLMs.

Question Answering with Citations. Recently, question answering with citations has been extensively studied in open-domain QA (Nakano et al., 2021; Bohnet et al., 2022; Gao et al., 2023a,b), and some works (Slobodkin et al., 2024; Huang et al., 2024) also explore fine-grained citations for more precise attribution. These methods (either prompting or SFT data construction) use retrievers or LLMs to pre-select relevant context before generating responses and citations. However, such paradigm is not suitable for long-context scenarios since missing any important context information will lead to compromised response quality (Appendix D). In addition, Buchmann et al. (2024) evaluates several prompting approaches for chunk-level citation generation in long-context QA. Nevertheless, the RAG method and more complex pipelines they tried also suffer from context information loss or excessive latency. Our work, however, enables long-context LLMs to produce accurate responses and precise sentence-level citations in a single generation, providing advantages in terms of response quality, verifiability, and efficiency. Moreover, previous methods for citation evaluation only consider limited cases due to rigid NLI models they use (Honovich et al., 2022; Gao et al., 2023b). In contrast, we improve the evaluation to fit more complex long-context scenarios and utilize GPT-4o as a judge, thereby achieving a higher agreement with human assessments.

3 Longbench-Cite

3.1 Problem Definition

We formalize the task of long-context question answering with citations (LQAC) as follows: given a long context \( \mathcal{D} \) and a query \( q \), the LLM is required to return a response \( A \), which consists of \( n \) statements \( s_1, \ldots, s_n \), and each statement \( s_i \) cites a list of snippets \( C_i = \{c_{i,1}, c_{i,2}, \ldots\} \) from \( \mathcal{D} \). In this work, LLMs need to segment their responses into statements based on semantic integrity by enclosing each statement with two special tokens <statement> and </statement>(as illustrated in Figure 1).

We consider two types of citations:
• Chunk-level citations, where \( \mathcal{D} \) is divided into 128-token \( ^\dagger \) chunks, and each citation \( c_{i,j} \) is in the form of \([k]\), referring to the \( k \)-th chunk;
• Sentence-level citations, where \( \mathcal{D} \) is divided into indexed sentences using NLTK, and each \( c_{i,j} \) takes the form of [a-b], referring to the snippet that includes the \( a \)-th to \( b \)-th sentences in \( \mathcal{D} \).

Most previous works (Menick et al., 2022; Gao et al., 2023b; Buchmann et al., 2024) for citation generation only explore chunk-level citations. However, the coarse granularity of chunk-level citations requires users to sift through many irrelevant details in the cited content. Therefore, in this work, we mainly focus on sentence-level citations because they allow for finer-grained citation and are thus more user-friendly.

3.2 Data Collection

Previous citation-generation benchmarks and methods mainly focus on open-domain QA (Gao et al., 2023b; Huang et al., 2024) and are not adaptable to the long-context scenario. To evaluate LLMs’ performance on LQAC task, we curate a new benchmark LongBench-Cite by collecting 1,000 long-context QA instances from existing bilingual long-context benchmarks LongBench (Bai et al., 2023) and LongBench-Chat (Bai et al., 2024), covering multiple key user-intensive tasks in both English and Chinese. Specifically, LongBench is a comprehensive long-context-understanding benchmark and we select two single-doc QA datasets MultiFieldQA-en/zh (Bai et al., 2023), two multi-doc QA datasets HotpotQA (Yang et al., 2018) and DuReader (He et al., 2018), and one summarization dataset GovReport (Huang et al., 2021) from it. LongBench-Chat consists of 50 real-world long-context queries and we adopt all the queries. Detailed statistics are listed in Table 6. For all datasets, we require LLMs to generate long-form responses.

3.3 Automatic Evaluation

LongBench-Cite evaluates models’ responses based on two dimensions:
• Correctness: Whether the response is accurate and consistent with the groundtruth.
• Citation quality: Whether all statements in the response are supported by the cited snippets, no irrelevant snippets are cited, and the cited snippets are fine-grained.

3.3.1 Evaluation of Correctness

For the evaluation of correctness, we first remove citation-relevant tokens from the LLM response,
then ask GPT-4o to rate the response based on the query and groundtruth. Such LLM-as-judge method has been shown to align well with human for open-end long-context QA (Bai et al., 2024). The detailed prompts can be found in Figure 5, 6, and 7. Furthermore, to investigate whether adding citations will hurt or improve models’ long-context QA performance, we propose a new metric correctness ratio:

\[
\text{CR} = \frac{\text{C}}{\text{C}_{\text{LQA}}} \times 100\%
\]

Here, C and \( \text{C}_{\text{LQA}} \) respectively denote the correctness in LQAC setting and vanilla long-context QA setting (i.e., simply feeding the concatenated context and query into the LLM to get a response).

3.3.2 Evaluation of Citation Quality
To evaluate the citation quality, we select citation F1 calculated using citation recall and citation precision as the main metric, where the former examines if the model response is fully supported by cited snippets and the later detects irrelevant citations. Compared with Gao et al. (2023b), which uses NLI model TRUE (Honovich et al., 2022) for automatic examination, we further improve the measurement method with GPT-4o to better adapt to long-context QA scenarios. Human evaluation (Sec. 5.3) demonstrates our method has a stronger agreement with human. Besides, we use citation length to measure the granularity of citations and avoid trivial results.

Citation Recall. We score citation recall (0/0.5/1) for each statement and average over all statements in the model response. Specifically, for each statement \( s_i \) that cites at least one snippet (i.e., \( C_i \neq \emptyset \)), we concatenate all snippets in \( C_i \) and ask GPT-4o to judge whether the concatenated text fully supports (1 point), partially supports (0.5 point), or does not support \( s_i \) (0 point). On the other hand, we observe that most LLM responses contain several “functional sentences” such as “The proposed method has the following advantages:” and “In summary, ...” that do not require citation. Therefore, for each statement \( s_i \) that has no citation, we prompt GPT-4o to determine if \( s_i \) is a starting sentence, transition sentence, or a summary or reasoning based on the previous response content. If so, \( s_i \) needs no citation and directly receives a citation recall of 1; otherwise, the recall is 0. The prompts are shown in Figure 8 and 9.

Citation Precision. We calculate citation precision for each citation (0/1 for irrelevant/relevant citations) and average over all citations in the response. Here, a cited snippet \( c_{t,j} \) is relevant if and only if it entails some key points of the statement \( s_i \), i.e., at least partially supports \( s_i \). We also employ GPT-4o as the judge using the prompt in Figure 10. In contrast, (Gao et al., 2023b) may overlook partially supporting cases, due to the limited capacity of the NLI model it uses.

Citation F1. Citation F1 is a comprehensive metric to evaluate the citation quality of a response:

\[
\text{citation F1} = \frac{2 \cdot \text{citation recall} \cdot \text{citation precision}}{\text{citation recall} + \text{citation precision}}
\]

Citation Length. Since the sentence-level citation allows citing snippets of different lengths, we use citation length, which is the average token number of cited snippets in the response, to quantify the granularity of citations. A lower average citation length indicates the response has finer-grained and more concise citations and is thus easier for users to validate. In addition, measuring average citation length can avoid trivial hacks for citation F1, such as citing the whole context for each statement.

3.4 Results of Existing Long-context LLMs
We first evaluate 7 popular long-context LLMs (3 proprietary and 4 open-source models, details listed in Table 7) on LongBench-Cite using LAC-S (long-context answering with citations in senetence level) strategy, where the model reads the entire context and generates response with sentence-level citations in one pass. We select LAC-S as the default setting because it directly evaluates models’ LQAC ability without relying on additional retrieval systems. A further comparison of different strategies can be found in Appendix D. As illustrated in Figure 11, we prompt these LLMs with one demonstration. The evaluation results of citation quality and correctness are presented in Table 1 and 2, respectively. Our findings are as follows:

1. Open-source LLMs have poor citation quality and lag far behind proprietary LLMs. They have obvious difficulty in citing supporting evidence for their generated statements. We attribute this to (1) poor instruction-following ability: small LLMs often generate citations that do not conform to the prescribed format; (2) weak evidence-searching ability: larger open-source models still often cite irrelevant sentences.

2. The citation quality of proprietary LLMs is still unsatisfactory. Specifically, their average
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Avg</th>
    <th colspan="2">Longbench-Chat</th>
    <th colspan="2">MultifieldQA</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">Dureader</th>
    <th colspan="2">GovReport</th>
  </tr>
  <tr>
    <th>F1</th><th>CL</th>
    <th>R</th><th>P</th>
    <th>R</th><th>P</th>
    <th>R</th><th>P</th>
    <th>R</th><th>P</th>
    <th>R</th><th>P</th>
  </tr>
  <tr>
    <td colspan="13"><i>Proprietary models</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>65.6</td><td>220</td><td>46.7</td><td>53.5</td><td>46.7</td><td><b>79.0</b></td><td>87.9</td><td>80.6</td><td>55.7</td><td>62.3</td><td>53.4</td><td>65.6</td><td>74.2</td><td>67.4</td><td>73.4</td><td>90.4</td><td>79.8</td>
  </tr>
  <tr>
    <td>Claude-3-sonnet</td><td>67.2</td><td>132</td><td>52.0</td><td>67.8</td><td>55.1</td><td>64.7</td><td>85.8</td><td>71.3</td><td>46.4</td><td>65.8</td><td>49.9</td><td>67.7</td><td><b>89.2</b></td><td><b>75.5</b></td><td><b>77.4</b></td><td><b>93.9</b></td><td>84.1</td>
  </tr>
  <tr>
    <td>GLM-4</td><td>65.4</td><td>169</td><td>47.6</td><td>53.9</td><td>47.1</td><td>72.3</td><td>80.1</td><td>73.6</td><td>47.0</td><td>50.1</td><td>44.4</td><td><b>73.4</b></td><td>82.3</td><td>75.0</td><td><b>82.8</b></td><td><b>93.4</b></td><td><b>87.1</b></td>
  </tr>
  <tr>
    <td colspan="13"><i>Open-source models</i></td>
  </tr>
  <tr>
    <td>GLM-4-9B-chat</td><td>27.2</td><td>96</td><td>25.9</td><td>20.5</td><td>16.7</td><td>51.1</td><td>60.6</td><td>52.0</td><td>22.9</td><td>28.8</td><td>20.1</td><td>45.4</td><td>48.3</td><td>40.9</td><td>5.7</td><td>8.2</td><td>6.3</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td><td>19.7</td><td>100</td><td>14.1</td><td>19.5</td><td>12.4</td><td>29.8</td><td>44.3</td><td>31.6</td><td>20.2</td><td>30.9</td><td>20.9</td><td>22.0</td><td>25.1</td><td>17.0</td><td>16.2</td><td>25.3</td><td>16.8</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td><td>40.4</td><td>174</td><td>25.8</td><td>32.0</td><td>23.2</td><td>53.2</td><td>65.2</td><td>53.9</td><td>29.6</td><td>37.3</td><td>28.6</td><td>38.2</td><td>46.0</td><td>35.4</td><td>53.4</td><td>77.5</td><td>60.7</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct</td><td>51.5</td><td>132</td><td>19.8</td><td>23.9</td><td>19.0</td><td>71.8</td><td>80.7</td><td>73.8</td><td>34.5</td><td>40.9</td><td>32.1</td><td>58.3</td><td>67.0</td><td>60.1</td><td>67.9</td><td>79.6</td><td>72.5</td>
  </tr>
  <tr>
    <td colspan="13"><i>Our trained models</i></td>
  </tr>
  <tr>
    <td>LongCite-8B</td><td><b>72.0</b></td><td><b>85</b></td><td><b>62.0</b></td><td><b>79.7</b></td><td><b>67.4</b></td><td>74.7</td><td><b>93.0</b></td><td><b>80.8</b></td><td>59.2</td><td>72.1</td><td>60.3</td><td>68.3</td><td>85.6</td><td>73.1</td><td>74.0</td><td>86.6</td><td>78.5</td>
  </tr>
  <tr>
    <td>LongCite-9B</td><td><b>69.2</b></td><td><b>91</b></td><td><b>57.6</b></td><td><b>78.1</b></td><td><b>63.6</b></td><td>67.3</td><td><b>91.0</b></td><td><b>74.8</b></td><td><b>61.8</b></td><td><b>78.8</b></td><td><b>64.8</b></td><td>67.6</td><td><b>89.2</b></td><td>74.4</td><td>63.4</td><td>76.5</td><td>68.2</td>
  </tr>
</table>

Table 1: Citation recall (R), citation precision (P), citation F1 (F1), and citation length (CL) of different models on LongBench-Cite using LAC-S strategy. The best and second results are bolded and underlined, respectively.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Avg</th>
    <th colspan="2">Longbench-Chat</th>
    <th colspan="2">MultifieldQA</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">Dureader</th>
    <th colspan="2">GovReport</th>
  </tr>
  <tr>
    <th>C</th><th>CR</th>
    <th>C</th><th>CR</th>
    <th>C</th><th>CR</th>
    <th>C</th><th>CR</th>
    <th>C</th><th>CR</th>
    <th>C</th><th>CR</th>
  </tr>
  <tr>
    <td colspan="13"><i>Proprietary models</i></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>69.4</td><td>78.2</td><td>88%</td><td>61.6</td><td>77.4</td><td>80%</td><td>84.0</td><td>88.3</td><td>95%</td><td>74.5</td><td>80.8</td><td>92%</td><td>81.0</td><td>83.3</td><td>97%</td><td>46.0</td><td>61.3</td><td>75%</td>
  </tr>
  <tr>
    <td>Claude-3-sonnet</td><td>77.6</td><td>78.3</td><td>99%</td><td>73.8</td><td>77.8</td><td>95%</td><td>88.6</td><td>88.1</td><td>101%</td><td>81.3</td><td>75.3</td><td>108%</td><td>75.8</td><td>80.3</td><td>94%</td><td>68.4</td><td>70.1</td><td>98%</td>
  </tr>
  <tr>
    <td>GLM-4</td><td>73.7</td><td>77.2</td><td>95%</td><td>69.4</td><td>79.8</td><td>87%</td><td>87.6</td><td>88.1</td><td>99%</td><td>76.3</td><td>76.5</td><td>100%</td><td>76.0</td><td>75.8</td><td>100%</td><td>59.4</td><td>65.9</td><td>90%</td>
  </tr>
  <tr>
    <td colspan="13"><i>Open-source models</i></td>
  </tr>
  <tr>
    <td>GLM-4-9B-chat</td><td>62.3</td><td>70.8</td><td>88%</td><td>60.4</td><td>67.8</td><td>89%</td><td>74.2</td><td>84.9</td><td>87%</td><td>68.5</td><td>71.5</td><td>96%</td><td>49.3</td><td>68.1</td><td>72%</td><td>59.3</td><td>61.6</td><td>96%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td><td>52.1</td><td>60.2</td><td>86%</td><td>53.2</td><td>61.6</td><td>86%</td><td>63.9</td><td>73.3</td><td>87%</td><td>64.0</td><td>64.5</td><td>99%</td><td>29.8</td><td>39.4</td><td>76%</td><td>49.6</td><td>62.1</td><td>80%</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct</td><td>62.0</td><td>65.5</td><td>95%</td><td>60.8</td><td>64.6</td><td>94%</td><td>78.4</td><td>78.3</td><td>100%</td><td>71.3</td><td>75.3</td><td>95%</td><td>43.3</td><td>42.5</td><td>102%</td><td>56.3</td><td>66.9</td><td>84%</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct</td><td>73.6</td><td>76.4</td><td>96%</td><td>63.8</td><td>67.8</td><td>94%</td><td>88.0</td><td>85.3</td><td>103%</td><td>77.0</td><td>77.3</td><td>100%</td><td>79.0</td><td>83.3</td><td>95%</td><td>60.4</td><td>68.3</td><td>88%</td>
  </tr>
  <tr>
    <td colspan="13"><i>Our trained models</i></td>
  </tr>
  <tr>
    <td>LongCite-8B</td><td><b>71.7</b></td><td><b>67.6</b></td><td><b>107%</b></td><td><b>69.0</b></td><td><b>68.6</b></td><td><b>101%</b></td><td><b>87.0</b></td><td><b>83.6</b></td><td><b>104%</b></td><td><b>70.8</b></td><td><b>69.0</b></td><td><b>103%</b></td><td><b>68.5</b></td><td><b>62.3</b></td><td><b>110%</b></td><td><b>63.0</b></td><td><b>54.4</b></td><td><b>116%</b></td>
  </tr>
  <tr>
    <td>LongCite-9B</td><td><b>70.4</b></td><td><b>65.6</b></td><td><b>109%</b></td><td><b>67.6</b></td><td><b>64.6</b></td><td><b>105%</b></td><td><b>84.1</b></td><td><b>83.3</b></td><td><b>101%</b></td><td><b>71.8</b></td><td><b>67.5</b></td><td><b>106%</b></td><td><b>69.0</b></td><td><b>66.3</b></td><td><b>104%</b></td><td><b>59.6</b></td><td><b>46.4</b></td><td><b>128%</b></td>
  </tr>
</table>

Table 2: Correctness in LQAC setting (C) using LAC-S strategy, correctness in vanilla long-context QA setting (C<sub>LQA</sub>), and correctness ratio (CR) of different models on LongBench-Cite. We mark the cases where adding citations improves/hurts correctness (i.e., CR > 1 / CR < 1) in green/red.

citation length is even larger than chunk-level citation (whose citation length is 128), reflecting a coarse citation granularity. For example, the citation length of GPT-4o reaches 220 and each cited snippet contains about 6 sentences on average.

3. Generating responses and citations in one pass via in-context learning hurts long-context QA performance. On most datasets, current LLMs have correctness ratios less than 100%, indicating that compared to standard long-context QA, generating responses and citations at once through in-context learning always leads to correctness degradation due to the distribution shift from the post-training data.

Overall, the performance of existing LLMs on LQAC remains to be improved. Therefore, we will explore the construction of SFT data in the next section to further enhance LLMs’ abilities to generate fine-grained citations from lengthy contexts.

4 CoF Pipeline

The core of constructing high-quality SFT data for LQAC is guaranteeing precise citations while maintaining the correctness of answers. To this end, we propose CoF, a post-hoc retrieval- and extraction-based pipeline that utilizes an off-the-shelf LLM to obtain sentence-level citations from Coarse to Fine. As illustrated in Figure 2, CoF consists of four steps: (1) Given a long context material, CoF first employs the LLM to generate a query and corresponding answer through Self-Instruct (Wang et al., 2023). (2) CoF then uses sentences in the answer to retrieve roughly k chunks from the context, which are subsequently input into the LLM to add coarse-grained chunk-level citations into the answer. (3) Next, the LLM generates fine-grained sentence-level citations for each statement by extracting supporting sentences from the cited chunks. (4) Finally, instances with too few citations are filtered out. In the following, we will introduce each
Figure 2: CoF pipeline consists of four steps: (1) Generating long-context QA instance via Self-Instruct; (2) Using the answer to retrieve \( k \) context chunks and generating chunk-level citations; (3) Extracting sentence-level citations for each statement from the cited chunks. (4) Filter out LQAC instances with few citations.

step of CoF in detail.

4.1 Pipeline Details

QA Instance Generation. Considering that generating the answer and citations in one pass might affect answer correctness, we decide to first construct long-context QA pairs and then add citations in subsequent steps. The post-hoc characteristic also allows our pipeline to augment any long-context QA datasets with citations. For QA instance generation, we adopt the method of Bai et al. (2024), which first employs an off-the-shelf LLM to propose a query based on a given long context and then requests it again to obtain the answer. They also incorporate different task type descriptions into the prompts (Figure 12), such as summarization, information extraction, and multi-hop reasoning, to guarantee the diversity of generated queries.

Chunk-level Citation Generation. After constructing the query and answer, we split the context into 128-token chunks and use each sentence in the answer to retrieve \( l_{\max} \) chunks. We retain top-\( l \) chunks for each sentence, where \( l = \min(l_{\max}, (k + n_{sent} - 1)/n_{sent}) \) and \( n_{sent} \) denotes the number of sentences, so that about \( k \) chunks are retained in total. Then we feed all these chunks, which are sorted according to their position in the context, along with the query and answer into the LLM, and ask the LLM to segment the answer into statements and generate chunk-level citations for each statement using one-shot learning. Figure 13 shows the prompt we use.

Sentence-level Citation Extraction. Besides the coarse granularity, another drawback of chunk-level citation generated in step 2 is that the precise supporting evidence may be located at the beginning or end of the chunk where the sentences are incomplete. Therefore, to achieve fine-grained citations, we first expand each cited chunk by concatenating it with its preceding and succeeding chunks. Next, we retain and number complete sentences in the expanded chunk, and instruct the LLM to extract fine-grained supporting snippets from the chunk by outputting number spans such as [6-8], which refers to the 6th to 8th sentences, or outputting "No relevant information" if no supporting snippet is found in the chunk (see Figure 14 for the prompt). At last, we remove irregular spans and re-number the others according to the sentence position in the original context to obtain the final sentence-level citations.

Data Filtering. In the final filtering stage, we discard the instance if less than 20% of the statements in the answer have citations. If an answer has too few citations, we assume it is not factually-grounded enough in the context and may leverage the internal knowledge of LLMs, which often results in hallucinations.

4.2 Pipeline Validation

Before data construction, we first test CoF (without QA generation and final filtering) on LongBench-
Cite to validate its efficacy. Specifically, we compare CoF with various LQAC strategies, and the details are presented in Appendix D. The results in Table 8 show that CoF achieves high citation F1, fine citation granularity, and perfectly maintains the correctness of answers at the same time.

4.3 LongCite-45k Dataset

Using CoF pipeline, we construct LongCite-45k, a large-scale SFT dataset for LQAC. Specifically, we first collect 50k documents from the pre-training corpus of GLM-4, covering 9 varied domains including books, encyclopedias, academic papers, codes, etc. These documents are mainly in English and Chinese and their lengths range from 256 to 128k tokens. We then apply CoF to generate an LQAC instance for each document. We use GLM-4 as the backbone LLM and Zhipu Embedding-2 as the retriever, and set retrieval hyper-parameters \( l_{\max} = 10 \) and \( k = 40 \). This results in 44,600 high-quality LQAC instances after the filtering stage (detailed statistics and API cost are in Appendix A and B). As illustrated in Figure 2(d), the input part of each instance consists of a task instruction, a long document, and a query, and the output part is an answer equipped with sentence-level citations.

5 LongCite Models

In this section, we conduct model training experiments to determine whether SFT on LongCite-45k can enhance LLMs’ ability for LQAC, enabling them to generate accurate responses and precise citations within a single output.

5.1 Training Details

We select two latest open-source base models, namely GLM-4-9B (Zeng et al., 2024) and Llama-3.1-8B (Vavekanand and Sam, 2024), for the training experiments. Both of the two models support a context window of 128k tokens, thereby being suitable for SFT on LQAC data. Following Bai et al. (2024), we combine LongCite-45k with 76k short SFT instances from ShareGPT (Chiang et al., 2023) to ensure the model’s general capacities. We name the trained models as LongCite-9B (abbr. for GLM-4-9B-LongCite) and LongCite-8B (abbr. for Llama-3.1-8B-LongCite). Detailed training hyper-parameters are listed in Appendix E.

Meanwhile, to investigate whether SFT on LQAC data will influence models’ long-context QA correctness compared to standard long-context SFT (i.e., SFT on vanilla long-context QA data), we additionally train the two base models using the pure long-context QA pairs (without the task instruction and citations) in LongCite-45k, and we name the trained models as LongSFT-9B (abbr. for GLM-4-9B-LongSFT) and LongSFT-8B (abbr. for Llama-3.1-8B-LongSFT). When calculating correctness ratios for LongCite-9B/8B, we take the correctness of LongSFT-9B/8B as \( C_{LQA} \).

5.2 Experimental Results

5.2.1 Main Results

We show the citation quality and correctness of our trained models on LongBench-Cite in Table 1 and 2, respectively. Here are our main findings:

1. LongCite-8B and LongCite-9B achieve the best citation qualities among all models. Compared to three advanced proprietary models, i.e., GPT-4o, Claude-3-Sonnet, and GLM-4, LongCite-8B/9B improves the overall citation F1 by 6.4/3.6, 4.8/2.0, and 6.6/3.8, respectively. Besides, the average citation lengths of LongCite-8B and LongCite-9B are also significantly shorter than that of proprietary models, indicating finer citation granularity. Comparison in Appendix D also shows the superiority of LongCite models over RAG and post-hoc strategies.

2. SFT with citation information further boosts the long-context QA performance. Different from in-context LQAC where the LLMs typically generate responses with lower correctness (Sec. 3.4), SFT on LongCite-45k dataset consistently improves the response correctness on all datasets compared to vanilla long-context SFT (i.e., CR > 100%). Detailed case study in Appendix G indicates this improvement is because SFT with citation information can (1) alleviate hallucinations of LLMs and (2) enable LLMs to utilize context information more uniformly, resulting in more comprehensive responses. In addition, the overall correctness of our trained model is also comparable with the officially post-trained models (i.e., GLM-4-9B-chat and Llama-3.1-8B-Instruct), validating the rationality of QA instance generation through Self-Instruct in our CoF pipeline.

3. LongCite models have obvious advantages over RAG and post-hoc methods. Combining with evaluation results in Table 8, we can see the superiority of LongCite models over conventional RAG and post-hoc methods: On the one hand, the correctness ratio of RAG methods is only around
<table>
  <tr>
    <th>Model</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>CL</th>
    <th>C</th>
  </tr>
  <tr>
    <td>LongCite-9B</td>
    <td>57.6</td>
    <td>78.1</td>
    <td>63.6</td>
    <td>112</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>w/ standard SFT</td>
    <td>7.6</td>
    <td>15.6</td>
    <td>6.3</td>
    <td>86</td>
    <td>57.4</td>
  </tr>
  <tr>
    <td>w/o data filtering</td>
    <td>57.4</td>
    <td>71.2</td>
    <td>61.2</td>
    <td>115</td>
    <td>67.4</td>
  </tr>
</table>

Table 3: Performance of models using standard long-context SFT (i.e., LongSFT-9B) or unfiltered data on LongBench-Chat.

80%, indicating severe degradation of LLMs’ long-context QA performance due to incomplete context information, while LongCite models can further improve the performance; On the other hand, LongCite models also achieve higher citation F1 than all post-hoc methods and allow convenient one-pass generation, while post-hoc methods need repeatedly call the LLM and thus at least double the user waiting time. Surprisingly, LongCite-8B/9B even attains higher citation F1 than the data construction pipeline CoF (72.0/69.2 v.s. 65.8), implying a potential for continuous self-improvement.

5.2.2 Ablation Studies

Ablation on LongCite-45k dataset. To verify that the enhanced LQAC ability is obtained from LongCite-45k dataset instead of standard long-context SFT, we evaluate LongSFT-9B on LongBench-Chat using one-shot learning as Sec. 3.4. The results in Table 3 indicate that LongSFT-9B performs poorly on LQAC task. Similar to the open-sourced LLMs, LongSFT-9B always generates nonconforming citations or no citations.

Ablation on data filtering. To show the effect of data filtering in CoF pipeline, we train LongCite-9B with the unfiltered data. Table 3 shows that data filtering effectively improves citation quality.

Further analysis can be found in Appendix F.

5.3 Human Evaluation

To verify that our automatic evaluation of citation quality using GPT-4o correlates with human judgment, we conduct a human evaluation (annotator information in Appendix H) on GLM-4, LongCite-8B, and LongCite-9B. Specifically, we anonymized their responses on LongBench-Chat, including 150 responses, 1,064 statements, and 909 citations in total, and manually annotated the citation recall and precision following the same instructions as GPT-4o evaluation. We also compare GPT-4o evaluation with ALCE (Gao et al., 2023b), which utilizes NLI model TRUE (Honovich et al., 2022) to measure citation recall and precision. As shown in Table 4,

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Human scores</th>
    <th colspan="3">GPT-4o scores</th>
    <th colspan="3">ALCE scores</th>
  </tr>
  <tr>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GLM-4</td>
    <td>61.2</td>
    <td>67.5</td>
    <td>60.2</td>
    <td>47.6</td>
    <td>53.9</td>
    <td>47.1</td>
    <td>46.1</td>
    <td>29.1</td>
    <td>30.8</td>
  </tr>
  <tr>
    <td>LongCite-8B</td>
    <td><b>79.6</b></td>
    <td><b>88.9</b></td>
    <td><b>82.6</b></td>
    <td><b>62.0</b></td>
    <td><b>79.7</b></td>
    <td><b>67.4</b></td>
    <td>59.6</td>
    <td>39.5</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>LongCite-9B</td>
    <td>72.8</td>
    <td>84.2</td>
    <td>75.8</td>
    <td>57.6</td>
    <td>78.1</td>
    <td>63.6</td>
    <td><b>64.2</b></td>
    <td><b>45.1</b></td>
    <td><b>47.1</b></td>
  </tr>
</table>

Table 4: Citation quality evaluated by human, GPT-4o and ALCE on LongBench-Chat.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Citation recall</th>
    <th colspan="2">Citation precision</th>
  </tr>
  <tr>
    <th>Kappa (\( \kappa \))</th>
    <th>Acc</th>
    <th>Kappa (\( \kappa \))</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>0.544/0.593*</b></td>
    <td>75.0/80.2*</td>
    <td><b>0.655</b></td>
    <td><b>88.8</b></td>
  </tr>
  <tr>
    <td>ALCE</td>
    <td>0.247*</td>
    <td>64.7*</td>
    <td>0.146</td>
    <td>47.4</td>
  </tr>
</table>

Table 5: Agreement between GPT-4o/ALCE and human. * means treating “partially support” as “not support”.

the relative rankings produced by human and GPT-4o are consistent, indicating that improvements in GPT-4o scores also reflect improvements in human preferences. In addition, the absolute scores from GPT-4o typically aligned more closely with human scores compared to ALCE. On the other hand, we observed that GPT-4o scores are generally lower than human scores because the cited snippets often contain unclear pronouns like “he/she” that may confuse the judge LLM. We believe that incorporating an anaphora resolution step may alleviate this problem but will also increase the evaluation costs. Furthermore, Table 5 shows that the Cohen’s kappa coefficients between GPT-4o and human are significantly higher than ALCE, demonstrating a substantial agreement for citation recall (0.593 when treating “partially support” as “not support” following ALCE) and citation precision (0.655). When taking human annotations as gold labels, GPT-4o also achieves high accuracy (75.0% for citation recall and 88.8% for precision).

6 Conclusion

In this work, we explore enhancing LLMs’ capacity to generate fine-grained citations from lengthy contexts. We first propose LongBench-Cite, an automatic benchmark to reveal existing LLMs’ limited performance on LQAC. We then introduce CoF, a novel pipeline that uses an off-the-shelf LLM to automatically generate long-context QA instances with precise sentence-level citations, and construct LongCite-45k, a large-scale SFT dataset for LQAC. Finally, we successfully train LongCite-8B and LongCite-9B, allowing the generation of accurate responses and fine-grained citations in one
pass. Extensive analyses and human evaluation further verify the effectiveness of our approach. We believe that this work lays a solid foundation for further research on LQAC and contributes to the development of more reliable and trustworthy LLMs.

7 Limitations

We discuss several limitations of our work in this section: (1) Though LongCite can enhance the verifiable of long-context LLMs and reduce hallucinations to some extent, the trained model may still fabricate unfaithful content or wrong citations. We believe preference alignment can further alleviate these issues. (2) Since our goal is to achieve better performance than the most advanced proprietary models, our SFT data construction also relies on a strong proprietary LLM (i.e., GLM-4). To support reproducing and further research, we will open-source our code, LongCite-45k dataset, and our trained models. (3) In the evaluation for correctness, there is a risk that GPT-4o will assign higher scores for itself, as found by previous works. However, since we focus more on the correct ratio, which is a relative ratio, the influence will be alleviated. In addition, because the evaluation of citation recall and precision is similar to simple NLI tasks, GPT-4o would not present obvious bias. (5) A possible way to hack our citation evaluation is to emit many sentences that GPT-4o would deem “starting, transition, summary, or reasoning” since these need no citations according to the evaluation scheme. In this case, we can ignore these “functional sentences” when calculating the citation recall to defend such hack.

8 Ethical Considerations

We have already desensitized the training data. All the models and datasets used in this work are publicly published with permissible licenses.

9 Acknowledgement

This work is supported by Beijing Natural Science Foundation (L243006), National Natural Science Foundation of China (62476150), Natural Science Foundation of China (NSFC) 62495063 and 62425601. This work is also supported by Zhipu AI.

References

Anthropic. 2024a. Anthropic: Introducing claude 3.5 sonnet.

Anthropic. 2024b. Anthropic: Introducing the next generation of claude.

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. CoRR, abs/2401.18058.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308.14508.

Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roei Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. CoRR, abs/2212.08037.

Jan Buchmann, Xiao Liu, and Iryna Gurevych. 2024. Attribute or abstain: Large language models as long document assistants. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8113–8140. Association for Computational Linguistics.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. RARR: researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 16477–16508. Association for Computational Linguistics.

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6465–6488. Association for Computational Linguistics.

Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu,
Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018. Dureader: a chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering@ACL 2018, Melbourne, Australia, July 19, 2018, pages 37–46. Association for Computational Linguistics.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kuklansky, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3905–3920. Association for Computational Linguistics.

Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, et al. 2024. Learning fine-grained grounded citations for attributed large language models. arXiv preprint arXiv:2408.04568.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. CoRR, abs/2311.05232.

Luyang Huang, Shuyang Cao, Nikolaus Nova Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1419–1436. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, H. Francis Song, Martin J. Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. CoRR, abs/2203.11147.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332.

OpenAI. 2024. Openai: Hello gpt-4o.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-Lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053.

Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. 2024. Attribute first, then generate: Locally-attributable grounded text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3309–3344. Association for Computational Linguistics.

Raja Vavekanand and Kira Sam. 2024. Llama 3.1: An in-depth analysis of the next-generation large language model. ResearchGate.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13484–13508. Association for Computational Linguistics.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madhan Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023. Effective long-context scaling of foundation models. CoRR, abs/2309.16039.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2369–2380. Association for Computational Linguistics.

Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Yao Hu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue
Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. 2024. Chatglm: A family of large language models from GLM-130B to GLM-4 all tools. CoRR, abs/2406.12793.
<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th>Source</th>
    <th>Avg Len</th>
    <th>Language</th>
    <th>#data</th>
  </tr>
  <tr>
    <td>MultiFieldQA-en</td>
    <td>Single-Doc QA</td>
    <td>Multi-field</td>
    <td>4,559</td>
    <td>English</td>
    <td>150</td>
  </tr>
  <tr>
    <td>MultiFieldQA-zh</td>
    <td>Single-Doc QA</td>
    <td>Multi-field</td>
    <td>6,701</td>
    <td>Chinese</td>
    <td>200</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>Multi-Doc QA</td>
    <td>Wikipedia</td>
    <td>9,151</td>
    <td>English</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Dureader</td>
    <td>Multi-Doc QA</td>
    <td>Baidu Search</td>
    <td>15,768</td>
    <td>Chinese</td>
    <td>200</td>
  </tr>
  <tr>
    <td>GovReport</td>
    <td>Summarization</td>
    <td>Government Report</td>
    <td>8,734</td>
    <td>English</td>
    <td>200</td>
  </tr>
  <tr>
    <td>LongBench-Chat</td>
    <td>Multi-task</td>
    <td>Real-world Query</td>
    <td>35,571</td>
    <td>English/Chinese</td>
    <td>50</td>
  </tr>
</table>

Table 6: Data Statistics in LongBench-Cite. ‘Source’ means the origin of the context. ‘Avg Len’ denotes the average number of words/characters of contexts in English/Chinese datasets.

![Bar chart showing context length distribution of LongCite-45k dataset](page_370_670_708_312.png)

Figure 3: Context length distribution of LongCite-45k dataset.

A  Data Statistics

The detailed data statistics of LongBench-Cite are listed in Table 6. For the SFT dataset LongCite-45k, we show the distribution of context length (measured by GLM-9B tokenizer) in Figure 3, and list the citation-relevant statistics in Table 10.

B  API Cost

On LongBench-Cite, a run of GPT-4o evaluation for correctness/citation quality costs about $4/$25. When using GLM-4 to construct LongCite-45k dataset, we spend 3.2 CNY (about $0.44) on average for each LQAC instance.

C  Model Cards

We list the details of our evaluated models in Table 7.

D  Comparison of Different LQAC Strategies & Validation of CoF Pipeline

Before large-scale data construction, we first test CoF (without query generation and final filtering) on LongBench-Cite to validate its efficacy. We compare CoF with the following LQAC strategies:

• **LAC-C/LAC-S**: the LLM reads the entire context and generates response and chunk-level/sentence-level citation in one pass.

• **RAG-C/RAG-S**: the LLM reads top-\( k \) chunks/sentences retrieved using the query and generates response and chunk-level/sentence-level citation in one pass.

• **post-LAC-C/post-LAC-S**: the LLM first generates a response via vanilla long-context QA, then adds chunk-level/sentence-level citations into the response by finding supporting evidence from the whole context.
<table>
  <tr>
    <th>Model name</th>
    <th>Model version</th>
    <th>Context window</th>
  </tr>
  <tr>
    <td>Claude-3-Sonnet (<a href="https://anthropic.com/">Anthropic</a>, 2024a)</td>
    <td>claude-3-sonnet-20240229</td>
    <td>200,000 tokens</td>
  </tr>
  <tr>
    <td>GPT-4o (<a href="https://openai.com/">OpenAI</a>, 2024)</td>
    <td>gpt-4o-2024-05-13</td>
    <td>128,000 tokens</td>
  </tr>
  <tr>
    <td>GLM-4 (<a href="https://arxiv.org/abs/2401.07662">Zeng et al.</a>, 2024)</td>
    <td>GLM-4-0520</td>
    <td>128,000 tokens</td>
  </tr>
  <tr>
    <td>GLM-4-9B-chat (<a href="https://arxiv.org/abs/2401.07662">Zeng et al.</a>, 2024)</td>
    <td>-</td>
    <td>128,000 tokens</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct (<a href="https://arxiv.org/abs/2401.07662">Vavekanand and Sam</a>, 2024)</td>
    <td>-</td>
    <td>128,000 tokens</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Instruct (<a href="https://arxiv.org/abs/2401.07662">Vavekanand and Sam</a>, 2024)</td>
    <td>-</td>
    <td>128,000 tokens</td>
  </tr>
  <tr>
    <td>Mistral-Large-Instruct (<a href="https://arxiv.org/abs/2309.12817">Jiang et al.</a>, 2023)</td>
    <td>Mistral-Large-Instruct-2407</td>
    <td>128,000 tokens</td>
  </tr>
</table>

Table 7: Model cards.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Avg</th>
    <th colspan="3">Longbench-Chat</th>
    <th colspan="3">MultifieldQA</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">Dureader</th>
    <th colspan="3">GovReport</th>
  </tr>
  <tr>
    <th>FI</th>
    <th>CR</th>
    <th>CL</th>
    <th>FI</th>
    <th>C</th>
    <th>CR</th>
    <th>FI</th>
    <th>C</th>
    <th>CR</th>
    <th>FI</th>
    <th>C</th>
    <th>CR</th>
    <th>FI</th>
    <th>C</th>
    <th>CR</th>
    <th>FI</th>
    <th>C</th>
    <th>CR</th>
  </tr>
  <tr>
    <th colspan="18"><i>one-pass methods (GLM-4)</i></th>
  </tr>
  <tr>
    <td>LAC-C</td>
    <td>51.6</td>
    <td>95%</td>
    <td>128.0</td>
    <td>33.9</td>
    <td>67.8</td>
    <td>85%</td>
    <td>55.7</td>
    <td>87.3</td>
    <td>99%</td>
    <td>41.2</td>
    <td>75.3</td>
    <td>98%</td>
    <td>59.5</td>
    <td>76.3</td>
    <td>101%</td>
    <td>67.7</td>
    <td>59.1</td>
    <td>90%</td>
  </tr>
  <tr>
    <td>LAC-S</td>
    <td>65.4</td>
    <td>95%</td>
    <td>169.0</td>
    <td>47.1</td>
    <td>69.4</td>
    <td>87%</td>
    <td>73.6</td>
    <td>87.6</td>
    <td>99%</td>
    <td>44.4</td>
    <td>76.3</td>
    <td>100%</td>
    <td>75.0</td>
    <td>76.0</td>
    <td>100%</td>
    <td>87.1</td>
    <td>59.4</td>
    <td>90%</td>
  </tr>
  <tr>
    <td>RAG-C</td>
    <td>72.5</td>
    <td>87%</td>
    <td>128.0</td>
    <td>69.7</td>
    <td>59.0</td>
    <td>74%</td>
    <td>79.1</td>
    <td>80.7</td>
    <td>92%</td>
    <td>57.7</td>
    <td>69.8</td>
    <td>91%</td>
    <td>75.7</td>
    <td>77.3</td>
    <td>102%</td>
    <td>80.3</td>
    <td>49.9</td>
    <td>76%</td>
  </tr>
  <tr>
    <td>RAG-S</td>
    <td>79.1</td>
    <td>79%</td>
    <td>48.0</td>
    <td>76.3</td>
    <td>66.4</td>
    <td>83%</td>
    <td>86.3</td>
    <td>85.7</td>
    <td>97%</td>
    <td>58.1</td>
    <td>53.3</td>
    <td>70%</td>
    <td>83.7</td>
    <td>76.5</td>
    <td>101%</td>
    <td>91.1</td>
    <td>29.0</td>
    <td>44%</td>
  </tr>
  <tr>
    <th colspan="18"><i>post-hoc methods (GLM-4)</i></th>
  </tr>
  <tr>
    <td>post-LAC-C</td>
    <td>47.3</td>
    <td>100%</td>
    <td>128.0</td>
    <td>27.8</td>
    <td>79.8</td>
    <td>100%</td>
    <td>48.2</td>
    <td>88.1</td>
    <td>100%</td>
    <td>34.5</td>
    <td>76.5</td>
    <td>100%</td>
    <td>52.1</td>
    <td>75.8</td>
    <td>100%</td>
    <td>74.1</td>
    <td>65.9</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>post-LAC-S</td>
    <td>57.3</td>
    <td>100%</td>
    <td>147.0</td>
    <td>34.3</td>
    <td>79.8</td>
    <td>100%</td>
    <td>65.3</td>
    <td>88.1</td>
    <td>100%</td>
    <td>40.0</td>
    <td>76.5</td>
    <td>100%</td>
    <td>64.2</td>
    <td>75.8</td>
    <td>100%</td>
    <td>82.8</td>
    <td>65.9</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>post-RAG-C</td>
    <td>63.8</td>
    <td>100%</td>
    <td>128.0</td>
    <td>61.0</td>
    <td>79.8</td>
    <td>100%</td>
    <td>65.3</td>
    <td>88.1</td>
    <td>100%</td>
    <td>49.3</td>
    <td>76.5</td>
    <td>100%</td>
    <td>67.8</td>
    <td>75.8</td>
    <td>100%</td>
    <td>75.8</td>
    <td>65.9</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>post-RAG-S</td>
    <td>62.8</td>
    <td>100%</td>
    <td>48.0</td>
    <td>63.4</td>
    <td>79.8</td>
    <td>100%</td>
    <td>64.8</td>
    <td>88.1</td>
    <td>100%</td>
    <td>48.6</td>
    <td>76.5</td>
    <td>100%</td>
    <td>69.7</td>
    <td>75.8</td>
    <td>100%</td>
    <td>67.5</td>
    <td>65.9</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>CoF</td>
    <td>65.8</td>
    <td>100%</td>
    <td>89.0</td>
    <td>66.1</td>
    <td>79.8</td>
    <td>100%</td>
    <td>65.6</td>
    <td>88.1</td>
    <td>100%</td>
    <td>50.6</td>
    <td>76.5</td>
    <td>100%</td>
    <td>67.4</td>
    <td>75.8</td>
    <td>100%</td>
    <td>79.1</td>
    <td>65.9</td>
    <td>100%</td>
  </tr>
</table>

Table 8: Citation F1 (FI), correctness (C), correctness ratio (CR), and citation length (CL) of different LQAC strategies on LongBench-Cite using GLM-4. Detailed description of these strategies are in Appendix D. We merge MultifieldQA-en/zh for brevity.

• post-RAG-C/post-RAG-S: the LLM first generates a response via vanilla long-context QA, then uses the response to retrieve about \( k \) chunks/sentences from the context, and adds chunk-level/sentence-level citations by finding supporting evidence from the retrieved text (similar to step 2 of CoF).

We use GLM-4 as the backbone LLM and Zhipu Embedding-2 as the retriever for all strategies and set retrieval hyper-parameters \( l_{\max} = 10 \) and \( k = 40 \). The results in Table 8 show that:

1. Similar to other post-hoc strategies, CoF is able to preserve the high-quality answers produced through vanilla long-context QA, well preventing correctness degradation. Specifically, GLM-4 perfectly maintains original answer contents unchanged when adding chunk-level citations, thereby achieving 100% correctness ratios. In contrast, though attaining higher citation F1, one-pass strategies (especially RAG methods) typically generate answers with lower correctness, failing to fully leverage LLMs’ long-context QA capacities.

2. CoF achieves the highest citation F1 and relatively small citation length among post-hoc methods, highlighting its ability to generate precise, fine-grained citations. Compared to post-LAC-C and post-LAC-S, post-hoc retrieval-based methods (i.e., post-RAG-C, post-RAG-S and CoF) benefit from a more focused evidence search space, typically yielding better performance. Furthermore, CoF’s superiority over post-RAG-C indicates that the step of sentence-level citation extraction effectively pinpoints supporting sentences and also filters out irrelevant chunks. Though post-RAG-S achieves an even shorter citation length than CoF (49 v.s. 89), we empirically found that sentence-level retrieval-based generation results in too many discontinuous citation numbers (such as [3][7][15]...), making subsequent training difficult (details in Appendix F).

E Training Hyperparameters

All the LongCite and LongSFT models are trained using 4 nodes with 8×H800 80G GPUs. We adopt Megatron-LM (<a href="https://arxiv.org/abs/1909.08053">Shoeybi et al.</a>, 2019) with context parallelism to support a maximum training sequence length of 128k tokens, and use packing training with loss weighting (<a href="https://arxiv.org/abs/2401.07662">Bai et al.</a>, 2024) to improve training
![Bar chart showing Citation F1 mean and std. w.r.t correctness of LongCite-9B’s responses.](page_246_183_755_377.png)

Figure 4: Citation F1 mean and std. w.r.t correctness of LongCite-9B’s responses.

<table>
  <tr>
    <th>Model</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>CL</th>
    <th>C</th>
  </tr>
  <tr>
    <td>LongCite-9B w/ CoF data</td>
    <td>57.6</td>
    <td>78.1</td>
    <td>63.6</td>
    <td>112</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>w/ post-RAG-S data</td>
    <td>50.6</td>
    <td>57.2</td>
    <td>50.1</td>
    <td>91</td>
    <td>66.8</td>
  </tr>
</table>

Table 9: Performance of models using CoF data and post-RAG-S data on LongBench-Chat.

efficiency. We set the batch size to 8 and the learning rate to 1e-5. We train each model for 4,000 steps, which is about 2 epochs and takes 18 hours.

F  Further Analysis

Correlation between correctness and citation quality. To explore the correlation between correctness and citation quality, we divide LongCite-9B’s responses on LongBench-Cite into three groups according to their correctness and compute the mean and standard deviation of citation F1 for each group. As illustrated in Figure 4, responses with higher correctness typically have higher citation qualities, demonstrating a mutually promoting relationship between these two attributes.

Comparison with data constructed through post-RAG-S strategy. We attempt constructing LQAC data by applying post-RAG-S strategy, whose performance is comparable with CoF (Sec. D), to add citations for the QA pairs in LongCite-45k. However, as shown in Table 9, the model trained with post-RAG-S data achieves much worse citation F1 than LongCite-9B. We believe the main reason is that post-RAG-S directly recalls sentences that are not necessarily adjacent from the context, resulting in many discontinuous citation numbers (such as [3][7][15]...), which makes subsequent training difficult. In contrast, CoF extracts sentence-level citations from bigger chunk-level snippets and uses number spans to represent citations. These methods contribute to maintaining the semantic coherence of the cited information, which is advantageous for training purposes.

Performance of LongCite-9B on GovReport. We find that the performance of LongCite-9B on the GovReport dataset in Table 1 is relatively weak. By checking the generated response, we find that for 31 out of 200 test instances of GovReport, LongCite-9B directly outputs the summary without generating citations. A feasible fix method is to force the model to first generate the special token "<statement>" so that it will generate citations for each instance. This leads to 80.7 citation F1 on GovReport.

Citation-relevant statistics. We list the statistics about citations generated by LongCite-9B and LongCite-8B on LongBench-Cite in Table 10.

G  Case Study

To further explore the reasons for the correctness improvement of LongCite Models, we manually compared the responses generated by LongCite-9B and LongSFT-9B and found that the improvement mainly comes from two aspects (3 cases in Table 11, 12, and 13 to illustrate our interpretation): (1) SFT with citation information enhances the evidence locating ability of the model and helps to prevent from hallucination (Table 11); (2) LongCite models can utilize context information more uniformly (Table 12
<table>
  <tr>
    <th></th>
    <th>#statements per response</th>
    <th>#cited snippets per statement</th>
    <th>#sentences per cited snippet</th>
    <th>proportion of statements citing multiple snippets</th>
  </tr>
  <tr>
    <td>SFT dataset (LongCite-45k)</td>
    <td>10.11</td>
    <td>0.81</td>
    <td>2.30</td>
    <td>13.0%</td>
  </tr>
  <tr>
    <td>LongCite-8B</td>
    <td>5.65</td>
    <td>1.06</td>
    <td>1.86</td>
    <td>22.9%</td>
  </tr>
  <tr>
    <td>LongCite-9B</td>
    <td>5.75</td>
    <td>0.92</td>
    <td>1.86</td>
    <td>17.2%</td>
  </tr>
</table>

Table 10: Citation-relevant statistics of the SFT dataset and responses generated by our trained models on LongBench-Cite.

and 13). Specifically, when faced with a query that requires a global view, the generated citation numbers allow LongCite models to be aware of that current response content has covered which parts of the context, so that they can utilize different parts of context more uniformly, resulting in a more comprehensive response. In contrast, LongSFT models tend to focus more on the head of the context and only roughly utilize or even ignore the rest of the context. We observe this in 13 out of 20 sampled instances from GovReport dataset.

H Information of Annotators in Human Evaluation

The annotators for human evaluation consist of 3 PhD students and 2 undergraduate students. All of them are male and their major is computer science. PhD/undergraduate students are paid 400/200 CNY per day for internship.

I Prompts

We list the prompts used in this work in Figure 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.
Query: Are both Duke Energy and Affiliated Managers Group based in Massachusetts?

LongSFT-9B:
Based on the information provided, Duke Energy has offices in several locations including West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom. Affiliated Managers Group has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom. So they both have an office in Massachusetts.

LongCite-9B:
<statement>Based on the information provided, Duke Energy is headquartered in Charlotte, North Carolina, while Affiliated Managers Group has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom.<cite>[25-25][219-219]</cite></statement> <statement>So only Affiliated Managers Group is based in Massachusetts.<cite></cite></statement>

Citations:
[25-25]: Duke Energy Corporation is an American electric power and natural gas holding company headquartered in Charlotte, North Carolina.
[219-219]: (NYSE: AMG) is a strategic partner to independent investment management firms globally, with equity stakes in a number of partner-owned traditional investment managers, hedge funds, and specialized private equity firms which it calls “Affiliates.” The company has principal offices in West Palm Beach, Florida; Prides Crossing, Massachusetts; Stamford, Connecticut; and London, United Kingdom.

Table 11: Case study. LongSFT-9B hallucinates the office location of Duke Energy, directly copying that of Affiliated Managers Group, while LongCite-9B gets the correct answer due to SFT with citations. We mark the wrong and correct statements in red and green, respectively.

[Instructions] You are asked to evaluate the quality of the AI assistant’s answers to user questions as an impartial judge, and your evaluation should take into account factors including correctness (high priority), helpfulness, accuracy, and relevance. The scoring principles are as follows: 1. Read the AI assistant’s answer and compare the assistant’s answer with the reference answer. 2. Identify all errors in the AI Assistant’s answers and consider how much they affect the answer to the question. 3. Evaluate how helpful the AI assistant’s answers are in directly answering the user’s questions and providing the information the user needs. 4. Examine any additional information in the AI assistant’s answer to ensure that it is correct and closely related to the question. If this information is incorrect or not relevant to the question, points should be deducted from the overall score. Please give an overall integer rating from 1 to 10 based on the above principles, strictly in the following format: “[rating]]”, e.g. “[5]”.
[Question] {Question}
[Reference answer begins] {Groundtruth} [Reference answer ends]
Below are several assistants’ answers and their ratings:
[Assistant’s answer begins] {Example Answer 1} [Assistant’s answer ends]
Rating: [[{Rating for Example Answer 1}]]
[Assistant’s answer begins] {Example Answer 2} [Assistant’s answer ends]
Rating: [[{Rating for Example Answer 2}]]
[Assistant’s answer begins] {Example Answer 3} [Assistant’s answer ends]
Rating: [[{Rating for Example Answer 3}]]
Please rate the following assistant answers based on the scoring principles and examples above:
[Assistant’s answer begins] {Response} [Assistant’s answer ends]
Rating:

Figure 5: prompt for correctness evaluation on LongBench-Chat.
Query: Please write a one-page summary of the above government report.

LongSFT-9B:
The Department of Homeland Security (DHS) has established policies and processes for managing its multi-billion dollar acquisition investments. These policies and processes are primarily set forth in its Acquisition Management Directive 102-01 and Acquisition Management Instruction 102-01-001. The department uses these policies and processes to deliver systems that are intended to close critical capability gaps, helping enable DHS to execute its missions and achieve its goals.
The DHS acquisition management policy establishes that a major acquisition program’s decision authority shall review the program at a series of predetermined acquisition decision events to assess whether the major program is ready to proceed through the acquisition life-cycle phases. Depending on the program, these events can occur within months of each other, or be spread over several years.
The report found that 10 of the 24 programs assessed were on track to meet their schedule and cost goals, while 14 programs were not on track. The report made two recommendations to DHS to improve its acquisition management policies and processes.

LongCite-9B:
<statement>The report assesses the Department of Homeland Security’s (DHS) major acquisition programs and their progress towards meeting schedule and cost goals. It also evaluates DHS’s policies and processes for acquisition management, resource allocation, and requirements development to determine if they reflect key portfolio management practices.</cite></statement>
<statement>Key findings include:</cite></statement>
<statement> Of the 28 programs reviewed, only 10 were on track to meet their schedule and cost goals as of January 2018. The remaining 14 programs had slipped on schedule, cost, or both.<cite>[89-97]</cite></statement>
<statement>- The most common reasons for schedule slips were new requirements, technical challenges, and external factors like staffing shortages or funding cuts.<cite>[105-106][108-108]</cite></statement>
<statement>- Some programs had cost breaches due to scope increases, technical challenges, or contractor labor costs.<cite>[119-119][121-121][127-127]</cite></statement>
<statement>- DHS has taken steps to improve its policies and processes, but more is needed. For example, it should require components to submit affordability information when programs re-baseline due to breaches.<cite>[178-178][234-234][258-258]</cite></statement>
<statement>- The report makes two recommendations for DHS to improve its acquisition management: require components to submit affordability information when programs re-baseline and assess the results of post-implementation reviews to identify opportunities to improve performance across the portfolio.<cite>[258-259]</cite></statement>
<statement>- DHS concurred with the recommendations and has identified actions it will take to address them.<cite>[261-261]</cite></statement>

Table 12: Case study. LongCite-9B utilize more information from the middle part of the context, resulting in a more detailed response than LongSFT-9B. We mark the coarse and detailed counterparts in their response in red and green, respectively
Query: Please write a one-page summary of the above government report.

LongSFT-9B:
The US Agency for International Development (USAID) has obligated over $5 billion in bilateral assistance to the Palestinians in the West Bank and Gaza since 1993, primarily using funds appropriated through the Economic Support Fund (ESF). USAID provides project assistance and debt relief payments to Palestinian Authority (PA) creditors, and implements most ESF programs, including those related to private sector development, health, water and road infrastructure, local governance, civil society, rule of law, education, and youth development. USAID assistance to the West Bank and Gaza is conducted under antiterrorism policies and procedures outlined in an administrative policy document known as Mission Order 21. As of March 31, 2018, USAID had obligated about $544.1 million (over 99 percent) and expended about $350.6 million (over 64 percent) of approximately $544.5 million in ESF assistance allocated for the West Bank and Gaza in fiscal years 2015 and 2016.

LongCite-9B:
<statement>The report discusses USAID’s Economic Support Fund (ESF) assistance to the Palestinians in the West Bank and Gaza from FY2015-FY2016. Key points include:<cite></cite></statement>
<statement>- USAID obligated over $544 million in ESF assistance for the West Bank and Gaza in FY2015-FY2016, with about $350 million expended.<cite>[8-8]</cite></statement>
<statement>- Most assistance went to project assistance ($215 million in FY2015, $184 million in FY2016) for economic growth, infrastructure, education, health, etc. Some went to debt relief payments to PA creditors ($75 million in FY2015, $70 million in FY2016).<cite>[9-11]</cite></statement>
<statement>- USAID conducted required vetting of PA creditors to ensure assistance did not support terrorism. Vetting occurred within 12 months before debt relief agreements were signed.<cite>[36-36][40-40][42-42]</cite></statement>
<statement>- External assessments identified weaknesses in PA financial systems for medical referrals and petroleum procurement. USAID took additional steps to mitigate risks.<cite>[52-52][62-62]</cite></statement>
<statement>- Financial audits of the debt relief grants did not identify any issues.<cite>[66-66][70-70]</cite></statement>
<statement>- USAID does not plan to resume direct cash payments to the PA due to concerns about misuse. Payments to creditors are considered more transparent.<cite>[49-49]</cite></statement>
<statement>In summary, the report concludes that USAID generally followed proper procedures for providing ESF assistance to the Palestinians in the West Bank and Gaza, with some concerns about PA financial systems. Debt relief payments to creditors were properly vetted.<cite></cite></statement>

Table 13: Case Study. The citation numbers in the response make LongCite-9B aware of which parts of the document the current response has covered and thus help it utilize context information uniformly to generate a more comprehensive summary, while LongSFT-9B only focuses on the front part of the extensive document and ignores the rest parts.
You are asked to evaluate the quality of the AI assistant’s answers to user question as an impartial judge, and your evaluation should take into account factors including correctness (high priority), and comprehensiveness (whether the assistant’s answer covers all points). Read the AI assistant’s answer and compare against the reference answer, and give an overall integer rating in 1, 2, 3 (1 = wrong or irrelevant, 2 = partially correct, 3 = correct and comprehensive) based on the above principles, strictly in the following format:“[[rating]]”, e.g. “[[2]]”.
Question:
{Question}
Reference answer:
{Reference answer}
Assistant’s answer:
{Response}
Rating:

Figure 6: Prompt for correctness evaluation on MultiFieldQA-zh/en, HotpotQA, and Dureader.

You are asked to evaluate the quality of the AI assistant’s generated summary as an impartial judge, and your evaluation should take into account factors including correctness (high priority), comprehensiveness (whether the assistant’s summary covers all points), and coherence. Read the AI assistant’s summary and compare against the reference summary, and give an overall integer rating in on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the evaluation criteria, strictly in the following format:“[[rating]]”, e.g. “[[3]]”.
Question:
{Question}
Reference answer:
{Reference answer}
Assistant’s answer:
{Response}
Rating:

Figure 7: Prompt for correctness evaluation on GovReport.
You are an expert in evaluating text quality. You will receive a user’s question about an uploaded document, a factual statement from an AI assistant’s response based on that document, and a snippet from the document (since the document is too long to display in full). Your task is to carefully assess whether this statement is supported by the snippet. Please use the following scale to generate your rating:
- [[Fully supported]] - Most information in the statement is supported by or extracted from the snippet. This applies only to cases where the statement and parts of the snippet are almost identical.
- [[Partially supported]] - More than half of the content in the statement is supported by the snippet, but a small portion is either not mentioned or contradicts the snippet. For example, if the statement has two key points and the snippet supports only one of them, it should be considered [Partially supported].
- [[No support]] - The statement is largely unrelated to the snippet, or most key points in the statement do not align with the content of the snippet.
Ensure that you do not use any information or knowledge outside of the snippet when evaluating.
Please provide the rating first, followed by the analysis, in the format “Rating: [...] Analysis: ...”.

<question>
{ Question }
</question>

<statement>
{ Statement }
</statement>

<snippet>
{ Concatenation of Cited Snippet }
</statement>

Figure 8: Prompt for evaluating citation recall when the statement has at least one citation.
You are an expert in evaluating text quality. You will receive a user’s question regarding their uploaded document (due to the length of the document, it is not shown to you), an AI assistant’s response based on the document, and a sentence from the response. Your task is to determine whether this sentence is a factual statement made based on the information in the document that requires citation, rather than an introductory sentence, transition sentence, or a summary, reasoning, or inference based on the previous response.
Ensure that you do not use any other external information during your evaluation.
Please first provide your judgment (answer with [[Yes]] or [[No]]), then provide your analysis in the format “Need Citation: [[Yes/No]] Analysis: ...”.

<question>
{Question}
</question>

<response>
{Model Response}
</response>

<statement>
{Statement}
</statement>

Figure 9: Prompt for evaluating citation recall when the statement has no citation.

You are an expert in evaluating text quality. You will receive a user’s question about an uploaded document, a factual statement from an AI assistant’s response based on that document, and a snippet from the document (since the document is too long to display in full). Your task is to carefully assess whether the snippet contains some key information of the statement. Please use the following grades to generate the rating:
- [[Relevant]] - Some key points of the statement are supported by the snippet or extracted from it.
- [[Unrelevant]] - The statement is almost unrelated to the snippet, or all key points of the statement are inconsistent with the snippet content.
Ensure that you do not use any information or knowledge outside of the snippet when evaluating.
Please provide the rating first, followed by the analysis, in the format “Rating: [...] Analysis: ...”.

<question>
{Question}
</question>

<statement>
{Statement}
</statement>

<snippet>
{Cited Snippet}
</statement>

Figure 10: Prompt for evaluating citation precision.
Please answer the user’s question based on the given document. When a factual statement S in your response uses information from some chunks in the document (i.e., <C{s1}>-<C{e1}>, <C{s2}>-<C{e2}>, ...), please append these chunk numbers to S in the format "<statement>{S}<cite>[{s1}-{e1}][{s2}-{e2}]...<cite></statement>". For other sentences such as introductory sentences, summarization sentences, reasoning, and inference, you still need to append "<cite></cite>" to them to indicate they need no citations. You must answer in the same language as the user’s question.

Here is an example:

{An Example}

Now get ready to handle the following test case.

[Document Start]
<C1>{Sentence 1} <C2>{Sentence 2} <C3>{Sentence 3} ...
[Document End]

[Question]
{Question}

[Remind]
Please answer the user’s question based on the given document. When a factual statement S in your response uses information from some chunks in the document (i.e., <C{s1}>-<C{e1}>, <C{s2}>-<C{e2}>, ...), please append these chunk numbers to S in the format "<statement>{S}<cite>[{s1}-{e1}][{s2}-{e2}]...<cite></statement>". For other sentences such as introductory sentences, summarization sentences, reasoning, and inference, you still need to append "<cite></cite>" to them to indicate they need no citations. You must answer in the same language as the user’s question.

[Answer with Citations]

Figure 11: One-shot learning prompt for the LAC-S strategy.
Prompt for General type task:
{Long Text Material}
Given the above text, please propose 5 English questions that are diverse and cover all parts of the text, in the following format: “1: ”, “2: ”, ...

Prompt for Summary type task:
{Long Text Material}
Given the above text, please propose 5 English questions that require summarization or integration from multiple parts, make sure they are diverse and cover all parts of the text, in the following format: “1: ”, “2: ”, ...

Prompt for multi-hop reasoning type task:
{Long Text Material}
Given the above text, please propose 5 English questions that require multi-hop reasoning, make sure they are diverse and cover all parts of the text, in the following format: “1: ”, “2: ”, ...

Prompt for Information Extraction type task:
{Long Text Material}
Given the above text, please propose 5 English information-seeking questions, make sure they are diversified and cover all parts of the text, in the following format: “1: ”, “2: ”, ...

Figure 12: Prompt for English question generation in the CoF pipeline. For each long text material, we randomly select one of the four task prompts and let the LLM generate five questions to ensure that the questions cover content from multiple spans within the long text. We then randomly choose one of these questions. For long Chinese documents, we translate the corresponding prompts into Chinese and obtain Chinese questions.
Your task is to add citations to the existing answer. Specifically, when a factual statement S in the answer uses information from context snippets 11, 12, ..., ln, please add citations by appending these snippet numbers to S in the format "<statement>{S}<cite>[{11}][{12}]...[{ln}]</cite><statement>". For other sentences such as introductory sentences, summarization sentences, reasoning, and inference, you still need to append "<cite></cite>" to them to indicate they need no citations. Except for adding citations, do not change the original content and format of the existing answer.

Here is an example:

{An Example}

Now get ready to add citations for the following test case.

[Contexts Start]
Snippet [1]
{Chunk 1}

Snippet [2]
{Chunk 2}

Snippet [3]
{Chunk 3}

...
[Context End]

[Question]
{Question}

[Existing Answer Start]
{Answer}
[Existing Answer End]

[Answer with Citations]

Figure 13: Prompt for chunk-level citation generation in the CoF pipeline.
You will receive a passage and a factual statement. Your task is to identify the parts in the passage (i.e., chunks <C{s1}>-<C{e1}>, <C{s2}>-<C{e2}>, ...) that support some key points of the statement, and output the chunk number in the format:
""
[s1-e1]
[s2-e2]
...
...

If the passage contains no key information relevant to the statement, you must output "No relevant information".

Here are some examples:

{Example 1}

{Example 2}

{Example 3}

Now get ready to process the following test case.

[Passage Start]
<C1>{Sentence 1} <C2>{Sentence 3} <C3>{Sentence 3} ...
[Passage End]

[Statment]
{statement}

[output]

Reviewer paper summaries: The paper introduces a benchmark LongBench-Cite that evaluates LLMs' performance on answering long-context questions with citations to assess LLM trustworthiness. In addition, the paper also introduces a Coarse-to-Fine pipeline to automatically generate QA pairs with citations (LongCite-45K) to train open source models to generate answers with citations for long-context questions. The paper also introduces two automatic evaluation metrics (correctness and citation quality) to evaluate on LongBench-Cite. The automatic evaluation and human evaluation results show that current open source LLMs are worse at generating responses with faithful citations than black-box LLMs, but finetuning open source LLMs can improve LLMs' ability to generate with responses with more correct citations.
Summary of strengths: 1. The paper introduces a pipeline Coarse-to-Fine (CoF), a self-instruct inspired pipeline to generate QA instances with citations. Using this pipeline to generating training data to finetune opensource LLMs improves their ability to generate more correct citations even than black-box models, which verifies the effectiveness of CoF.

2. Reasonable research findings and conclusions.

3. Extra human validations to improve credibility of results.
Summary of weaknesses: 1. Possible inflation of scores for automatic evaluation metrics: line 283-286 states that if a sentence has no citation, it directly receives a citation recall of 1. In the first place, why include sentences that do not need a citation in the evaluation metric calculation? If a model generates a response with many sentences without a citation, wouldn't that beat the metric achieve a similar score with a model generates many citations?

2. Some of the claims are missing support statements.
Line 406-411: why does aggregating all retrieved chunks and generating citations at once can improve the citation recall? Any results, analysis or related work to support this claim?

Line 546-549: Any statistics or ablation study results to support that LongSFT models use more information from the head of the context? What percentage of the citations use information from the head of the context ?
Comments, suggestions and typos: Line 566-568: who are the annotators? What are the difference of backgrounds of the annotators? If there are multiple annotators annotating the same instance, what is the annotator aggrement?
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper tackles the problem of long context question answering with citations. In particular, the authors aim at generating fine-grained (sentence-level) citations along with the response to questions, with the goal of improving faithfulness and verifiability.
In order to achieve such goals, the paper presents an automatic benchmark (LongBench-Cite) to assess the performance of LLMs in the long-context QA setting. Then, they propose a pipeline (Coarse to Fine, CoF), that leverages off-the-shelf LLMs to build long context QA instances with precise sentence-level citations. They also perform Supervised Fine Tuning on top of 8B and 9B models with the new dataset showing interesting results. The paper may be interesting for many researchers that tries to improve the factuality and interpretability of the output of LLMs, by enabling better and more fine-grained citations of sources.
Summary of reasons to publish: The reviewers agree that the paper and related dataset could be useful for the community for the problem of long context QA with citations. In particular, the proposed pipeline is interesting for generating such a resource. The reviewers also like the discussions and findings in the paper regarding the usage of SFT vs. ICL.