L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?

## Abstract

Abstract Long-context models (LCMs) have made remarkable strides in recent years, offering users great convenience for handling tasks that involve long context, such as document summarization.
As the community increasingly prioritizes the faithfulness of generated results, merely ensuring the accuracy of LCM outputs is insufficient, as it is quite challenging for humans to verify the results from the extremely lengthy context.
Yet, although some efforts have been made to assess whether LCMs respond truly based on the context, these works either are limited to specific tasks or heavily rely on external evaluation resources like GPT-4.
In this work, we introduce L-CiteEval , a comprehensive multi-task benchmark for long-context understanding with citations, aiming to evaluate both the understanding capability and faithfulness of LCMs.
L-CiteEval covers 11 tasks from diverse domains, spanning context lengths from 8K to 48K, and provides a fully automated evaluation suite.
Through testing with 11 cutting-edge closed-source and open-source LCMs, we find that although these models show minor differences in their generated results, open-source models substantially trail behind their closed-source counterparts in terms of citation accuracy and recall.
This suggests that current open-source LCMs are prone to responding based on their inherent knowledge rather than the given context, posing a significant risk to the user experience in practical applications.
We also evaluate the RAG approach and observe that RAG can significantly improve the faithfulness of LCMs, albeit with a slight decrease in the generation quality.
Furthermore, we discover a correlation between the attention mechanisms of LCMs and the citation generation process.

## 1 Introduction

The rapid development of Long-context Models (LCMs) provides users with numerous conveniences in resolving long-context real-world tasks, such as code analysis (Zhu et al., ) and long document summarization (Reid et al., ).
Recently, the community has gradually intensified its efforts to enhance the faithfulness of generative artificial intelligence (Manna & Sett, ), which is of paramount importance for LCMs.
This is because tasks that involve long context usually require LCMs to respond based on the provided context rather than relying solely on models’ intrinsic knowledge.
Therefore, there is an urgent need for a benchmark to verify whether LCMs truly leverage context for responding and reflect those models’ capability on long-context tasks.

To date, substantial efforts have been made to develop benchmarks for evaluating LCMs.
These endeavors aim to achieve several key objectives:
(1) ensuring that the benchmarks include a comprehensive range of task scenarios and varying context lengths;
(2) employing automated metrics to guarantee the reproducibility of evaluations;
(3) incorporating an appropriate volume of test data to maintain evaluation efficiency; and
(4) offering sufficient interpretability (e.g., providing evidence to support the responses).
As shown in Fig. [1](#S1.F1), taking three representative long-context benchmarks as examples:
LongBench (Bai et al., ) primarily evaluates the accuracy of LCMs’ responses across a range of realistic and synthetic tasks, with a context length of up to 24K tokens;
Ruler (Hsieh et al., ) focuses on using synthetic data to test LCMs’ capabilities in information retrieval over long sequences, with context lengths exceeding 48K tokens;
and LongCite (Bai et al., ) assesses whether models respond based on the content within the context, employing GPT-4 as a judge.
These benchmarks, based on their purpose, can be roughly divided into two categories: (1) evaluating long-context understanding and (2) assessing model faithfulness.
The former evaluates model outputs using large volumes of test data to infer LCMs’ capabilities but lacks interpretability to the generated results.
The latter are mainly based on short-context datasets (e.g., in LongCite, the maximum sequence length only reaches 32K, comprising just 5.88% of the benchmark) and rely on external resources like GPT-4 to judge faithfulness, making the evaluation results hard to reproduce.

Figure: Figure 1: Overview and comparison among different representative benchmarks for LCMs.
Refer to caption: /html/2410.02115/assets/x1.png

In this work, we introduce L-CiteEval, a comprehensive multi-task benchmark for long-context understanding with citations.
As shown in Fig. [2](#S2.F2), given the question and long reference context, L-CiteEval requires LCMs to generate both the statements and their supporting evidence (citations).
There are 5 major task categories, 11 different long-context tasks, with context lengths ranging from 8K to 48K in L-CiteEval.
To address the timeliness and the risk of data leakage in testing (Ni et al., ; Apicella et al., ), we incorporate 4 latest long-context tasks as the subsets in L-CiteEval, ensuring that the evaluation remains up-to-date and robust.
Different from previous benchmarks for long-context understanding that primarily assess LCMs based on their predicted answers, L-CiteEval evaluates model performance based on both the generation quality (whether the predicted answer is correct) and citation quality (whether the provided citations can support the corresponding answer).
To extend the context length of short-context data, we design a rigorous data construction pipeline to extend the sequence length and mitigate the perturbation introduced from the additional context.
Additionally, to facilitate the ease of use and ensure reproducibility, L-CiteEval offers an automatic evaluation suite.
Considering that the prediction from LCMs can be influenced by both the task difficulty and the context length, we propose two benchmark variants: L-CiteEval-Length and L-CiteEval-Hardness.
These two variants strictly control the variables within the evaluation, focusing solely on context length and task difficulty to assess LCMs’ capabilities.

We test 11 cutting-edge and widely-used LCMs, including 3 closed-source and 8 open-source models, which feature different sizes and architectures.
We also explore whether the Retrieval-Augmented Generation (RAG) technique can improve the faithfulness of LCMs.
Evaluation results indicate that there is a minor difference between open-source and closed-source models regarding generation quality, while open-source models substantially trail behind their closed-source counterparts in terms of citation quality.
Utilizing the RAG technique exhibits a notable improvement in the faithfulness of open-source models, but it slightly impacts the generation quality.
Furthermore, we reveal a correlation between the model’s citation generation process and its attention mechanism (i.e., retrieval head (Wu et al., )), demonstrating the validity of our benchmark and offering insights for future evaluations of LCM faithfulness and the development of advanced LCMs.

## 2 Related Works

### 2.1 Long-context Understanding Benchmarks

Currently, there is a growing body of work dedicated to evaluating the long-context understanding capabilities of LCMs.
The majority of benchmarks for LCMs are built based on real-world tasks that inherently encompass long context, including but not limited to long-document QA, long-document summarization, and long-term conversations (Li et al., ; Shaham et al., ; An et al., ; GoodAI, ; Bai et al., ; Dong et al., ; Zhang et al., ; Lee et al., ; Levy et al., ).
Recently, InfiniteBench (Zhang et al., ) has pushed the boundaries of benchmarks based on real-world tasks by extending the context length beyond 100K tokens.
However, real-world tasks exhibit a variety of forms and evaluation methods, and existing evaluations are applied inconsistently across different works.
Additionally, the generated results can also be influenced by the intrinsic knowledge of LCMs.
To make evaluations more controllable and eliminate the influence of the LCMs’ intrinsic knowledge, synthetic benchmarks are often employed (Hsieh et al., ).
Among those synthetic benchmarks, task formats can be custom-defined into various types, such as retrieval-based tasks that require the model to extract specific information from a long context (Kamradt, ; Mohtashami & Jaggi, ; Xiao et al., ; Liu et al., ; Wang et al., ), many-shot in-context learning (Agarwal et al., ; Bertsch et al., ), fact reasoning (Kuratov et al., ; Karpinska et al., ), etc.
In this work, we introduce L-CiteEval, which contains both real-world tasks and synthetic tasks for long-context understanding with citations.
By requiring LCMs to provide evidence to support their predictions, we can also mitigate the challenge of being unable to test whether LCMs respond based on their intrinsic knowledge or the provided context.

Figure: Figure 2: Task format and pipeline of L-CiteEval benchmark.
Refer to caption: /html/2410.02115/assets/x2.png

### 2.2 Citation Generation

The citation generation task aims to verify whether the model predictions are supported by the referenced source (Li et al., ).
To evaluate the citations generated by models, Rashkin et al. () first proposed attributed to identified sources (AIS) evaluation framework to measure the faithfulness of the model outputs.
Then, some works began to improve the AIS framework in different tasks (such as single-document QA (Bohnet et al., ) and fact checking (Honovich et al., )) and domains (such as science (Funkquist et al., ) and commerce (Liu et al., )).
To enhance the evaluation precision of citations within the generated text, Qian et al. (); Kamalloo et al. (); Li et al. () made great contributions based on the QA tasks.
With the advancement of generative AI, citation generation has begun to require models themselves to generate citations that support their predictions (Gao et al., ).
More recently, Bai et al. () introduced LongCite, which represents the first attempt at citation generation in long context question-answering tasks.
Compared with LongCite, L-CiteEval is (1) more comprehensive – it covers a wider range of tasks, supports longer context lengths, and strictly categorizes tasks by length intervals; (2) more reproducible – it relies entirely on automatic evaluation metrics without reliance on GPT-4 or human judgments; and (3) more efficient – the task and data distribution are well-designed in L-CiteEval, enabling users to utilize a limited amount of testing data to reflect the LCMs’ capabilities.

## 3 L-CiteEval: Task and Construction

### 3.1 Problem Definition and Evaluation Metrics

##### Problem Definition

As shown in Fig. [2](#S2.F2), given the long context $T$ and question $Q$, a LCM is expected to generate the response $R$, which contains several statements $\mathcal{S}=\{s_{1},s_{2},\cdots,s_{n}\}$ and their corresponding citations $\mathcal{C}=\{c_{1},c_{2},\cdots,c_{n}\}$.
The context $T$ is divided into chunks of varying lengths based on the specific task, with each chunk representing a citation segment.
Specifically, we set large citation chunk sizes for information-concentrated tasks like Single-Document QA to ensure segment integrity while using small citation chunk sizes for information-dispersed tasks like summarization to maximize the number of citations that LCMs can leverage to support the generated results.
The model can then utilize these citation segments to support the statement $s_{i}$ within the response.
In terms of output format, we require each statement $s_{i}$ to be strictly followed by a supporting citation chunk index $c_{i}$, which can also serve as an enclosure.

##### Automatic Evaluation

During the verification stage, the model response is evaluated from two aspects: the response quality and citation quality.
As shown in Tab. [1](#S3.T1), for response quality, we employ different evaluation metrics tailored to each specific task, e.g., Precision (Prec.) and Recall (Rec.) for QA tasks and Rouge-L (Lin, ) for summarization tasks.
As for citation quality, following Gao et al. (), we adopt Citation Recall ($CR$) to reflect whether the model statements are fully supported by the citations; Citation Precision ($CP$) to detect irrelevant citations; and citation $F_{1}$ score to represent the overall citation performance.
Besides, we report citation number $N$ to show how many citations the model uses to support its output.
Different from previous works that utilize an NLI model (Gao et al., ) to automatically determine whether citations support the corresponding statements, we adopt a long-context NLI model deberta-base-long-nli (Sileo, ), to better align with long-context scenarios.
We describe the calculation of $CR$ and $CP$ in Appendix [A](#A1).

**Table 1: Statistic of tasks in L-CiteEval. The citation chunk size for each task is {task}:{size}.**
| Tasks | Source | Evaluation<br>Metric | Length Distribution | Total |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0$\sim$8k | 8$\sim$16k | 16$\sim$24k | 24$\sim$32k | 32$\sim$40k | 40$\sim$48k |  |  |  |  |
| Single-document QA   (NarrativeQA: 256, Natural Questions: 256) |  |  |  |  |  |  |  |  |  |
| NarrativeQA | (Kočiskỳ et al., ) | Prec., Rec. | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| Natural Questions | (Kwiatkowski et al., ) | Prec., Rec. | - | - | 40 | 40 | 40 | 40 | 160 |
| Multi-document QA   (HotpotQA: 128, 2WikiMultihopQA: 128) |  |  |  |  |  |  |  |  |  |
| HotpotQA | (Yang et al., ) | Prec., Rec. | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| 2WikiMultihopQA | (Ho et al., ) | Prec., Rec. | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| Summarization   (MultiNews: 128, GovReport: 128, QMSum: 128) |  |  |  |  |  |  |  |  |  |
| MultiNews | (Ghalandari et al., ) | Rouge-L | 20 | 20 | 20 | 20 | 20 | - | 100 |
| GovReport | (Huang et al., ) | Rouge-L | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| QMSum | (Zhong et al., ) | Rouge-L | 20 | 20 | 20 | 20 | - | - | 80 |
| Dialogue Understanding   (LoCoMo: 256, DialSim: 256) |  |  |  |  |  |  |  |  |  |
| LoCoMo | (Maharana et al., ) | Prec., Rec. | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| DialSim | (Kim et al., ) | Prec., Rec. | 40 | 40 | 40 | 40 | 40 | 40 | 240 |
| Synthetic Task   (NIAH: 256, Counting Stars: 128) |  |  |  |  |  |  |  |  |  |
| NIAH | (Kamradt, ) | Rouge-1 | 20 | 20 | 20 | 20 | 20 | 20 | 120 |
| Counting Stars | (Song et al., ) | Accuracy | 30 | 30 | 30 | 30 | 30 | 30 | 180 |

### 3.2 Benchmark Construction

There are 5 main categories in the L-CiteEval benchmark: Single-document QA, Multi-document QA, Summarization, Dialogue understanding, and Synthetic tasks, covering both realistic and synthetic tasks.
We report the data source for each task in Table [1](#S3.T1),
For each task, we utilized the same construction process to handle the dataset.
As shown in Fig. [3](#S3.F3), the construction process for each task in the L-CiteEval benchmark consists of 3 steps, including (1) Seed Data & Padding Data Sampling, (2) Padding Data Filtering, and (3) Length Extension.

##### Step1: Seed Data & Padding Data Sampling

Considering the large amount of data in each source dataset, we first sample a portion of testing dataset $\mathcal{D}_{seed}$ as the seed data, from which we can subsequently construct the benchmark.
However, some source datasets, e.g., LoCoMo (Maharana et al., ), exhibit short context.
Consequently, we sample data from the remaining source dataset to serve as the candidate padding data $\mathcal{D}_{pad}$ for length extension.
We divide all the sampled data ($\mathcal{D}_{seed}$ and $\mathcal{D}_{pad}$) into citation chunks of approximately equal size, with sentences as the basic unit.
As mentioned above, we utilize different citation chunk sizes for different tasks.
For tasks involving concentrated information, e.g., single-document QA, we employ smaller chunk sizes, while for tasks involving dispersed information, e.g., summarization, we use larger chunk sizes.
We report the citation chunk size for each dataset in Table [1](#S3.T1).

##### Step2: Padding Data Filtering

Using padding data to extend the length of the short-context dataset would introduce additional contextual information and could potentially influence the generated results.
Therefore, we eliminate the padding data that might affect the predictions based on overlapping entities in the text.
Specifically, we apply spaCy(^1^11[https://spacy.io/usage/models](https://spacy.io/usage/models)), a Named Entity Recognition model $f_{\theta}$, to extract all the entities $E$ from the question ($E^{(Q)}_{seed}$) and reference context ($E^{(T)}_{seed}$) in $\mathcal{D}_{seed}$, as well as the entities from the reference context ($E^{(T)}_{pad}$) in padding data.
Then, we keep the padding samples $\mathcal{D}_{pad}^{*}$ that share a small entity overlaps with the seed data, which can be written as:

$$ $\mathcal{D}_{pad}^{*}=\left\{\mathcal{D}_{pad}^{\prime}\mid\mathcal{D}_{pad}^{\prime}\sim\mathcal{D}_{pad},~{}~{}E_{pad}^{(T)}=f_{\theta}(\mathcal{D}_{pad}^{\prime}),~{}~{}|E_{seed}^{(T)}\cap E_{seed}^{(Q)}\cap E_{pad}^{(T)}|\leq\delta\right\},$ (1) $$

where $\delta$ is the threshold to control the entity overlap between seed data and padding data.
In this paper, we set this $\delta=5$ as a strict criterion to filter out data that may potentially impact the results.

Figure: Figure 3: Benchmark construction pipeline.
Refer to caption: /html/2410.02115/assets/x3.png

##### Step3: Length Extension

After obtaining the padding data $\mathcal{D}_{pad}^{*}$, we leverage these data to extend the context length of seed data $\mathcal{D}_{seed}$.
As shown in Figure [3](#S3.F3), we have three different benchmark settings, including L-CiteEval and its two variants: L-CiteEval-Quality and L-CiteEval-Length.
Specifically, for the L-CiteEval benchmark setting, given the target length interval of the dataset, we first sort the data according to the context length within each task.
We then randomly sample contexts from $\mathcal{D}_{pad}^{*}$ to extend the context length and fill in the missing target length intervals.
The L-CiteEval benchmark is designed to benchmark the models comprehensively. Thereby, the seed data and context extension data for all samples are different.
For the L-CiteEval-Length benchmark, which aims to test the model’s performance from the context length perspective, we use the same set of seed data and different sets of padding data to extend to various context lengths.
For the L-CiteEval-Hardness benchmark that is designed to benchmark models based on question difficulty, we first quantify and rank the difficulty of each question according to the model’s generation quality(^2^22Specifically, we categorize the difficulty level of each sample based on GPT-4o because GPT-4o has been proven to exhibit the highest preference similarity with human annotators (Yadav et al., ).).
Then, we categorize the difficulty into three levels: easy (where the model mostly provides correct answers), medium, and hard (where the model mostly produces incorrect answers).
We use the same padding data to extend the context length for each difficulty level.
We use GPT-4 as the evaluator to classify the sample difficulty, as it shows the best generation quality.

##### Benchmarks Overview

For clarity, we list the differences among the three benchmarks below:

- •
L-CiteEval is designed to evaluate the comprehensive capabilities (generation quality and citation quality) of LCMs, which is constructed with different seed data (varying question difficulty) and padding data sources (varying context). This benchmark includes 2,080 testing samples, with 11 tasks across 5 categories.
- •
L-CiteEval-Length is designed to evaluate the LCMs from the context length perspective, which is constructed with the same seed data source (same question difficulty) but different padding data sources (varying context).
This benchmark consists of 4 tasks across 4 categories, i.e., NarrativeQA (Single-Doc QA), HotpotQA (Multi-Doc QA), GovReport (Summarization), and Counting Stars (Synthetic task), with each task containing 200 testing samples.
For each task, we establish three context length intervals: 8K, 16K, and 32K.
- •
L-CiteEval-Hardness is designed to evaluate the LCMs from the task difficulty perspective, which is constructed with the different seed data source (varying question difficulty) but same padding data sources (same context). This benchmark shares the same data distribution and volume with L-CiteEval-Length, except that the scoring is based on task difficulty (Easy, Medium, and Hard) rather than context length.

## 4 Experiments

**Table 2: Statistic of LCMs. $*$ means the model utilizing YaRN (Peng et al., ) to extend the base context length. $\dagger$ denotes the MoE model, where activated parameters are enclosed in parentheses.**
| Model | Ctx. Size | #Param | Architecture | Open-source |
| --- | --- | --- | --- | --- |
| GPT-4o (20240513) (OpenAI, ) | 128K | \faLock | \faLock | ✗ |
| o1-mini (OpenAI, ) | 128K | \faLock | \faLock | ✗ |
| Claude-3.5-Sonnet (20240620) (anthropic, ) | 200K | \faLock | \faLock | ✗ |
| Qwen2.5-3B-Instruct (Team, ) | 32K (128K^∗) | 3B | Decoder-Only | ✓ |
| Phi-3.5-mini-instruct (Abdin et al., ) | 128K | 3.8B | Decoder-Only | ✓ |
| Llama-3.1-8B-Instruct () | 128K | 8B | Decoder-Only | ✓ |
| GLM-4-9B-Chat (GLM et al., ) | 128K | 9B | Decoder-Only | ✓ |
| Mistral-NeMo-Instruct-2407 (Mistral, ) | 128K | 12B | Decoder-Only | ✓ |
| Qwen2-57B-A14B-Instruct (Yang et al., ) | 32K (128K^∗) | 57B (14B^†) | MoE | ✓ |
| Llama-3.1-70B-Instruct () | 128K | 70B | Decoder-Only | ✓ |
| Llama3-ChatQA-2-70B (Xu et al., ) | 128K | 70B | Decoder-Only | ✓ |

We conduct experiments with 11 latest LCMs, including 3 closed-source LCMs and 8 open-source LCMs, each with a context window size of at least 128K tokens, encompassing different parameters (ranging from 3B to 70B) and model architectures (dense model and MoE model).
The statistic of LCMs is shown in Tab. [2](#S4.T2).
We provide one demonstration within the prompt for each task to make the model’s output format more standard, i.e., one-shot learning during the inference time, and employ the same instruction for every LCM.
Demonstration of model prediction, question, and instruction for each task is shown in Appendix [E](#A5).
We benchmark all the LCMs with L-CiteEval and then select 6 representative LCMs (including 1 closed-source LCMs and 4 open-source LCMs) to further evaluate on L-CiteEval-Length and L-CiteEval-Hardness benchmarks.

### 4.1 Model Performance on L-CiteEval

We report the citation quality in Tab. [3](#S4.T3) ( information-concentrated tasks that require models to seek local information in several citation segments) and Tab. [4](#S4.T4) (information-dispersed tasks that require models to seek global information from the entire context) and report the generation quality in Tab. [5](#S4.T5).

**Table 3: Citation quality of LCMs in information-concentrated tasks within L-CiteEval.**
| Models | Single-Doc QA | Dialogue Understanding | Needle in a Haystack |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ | $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ | $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ |  |
| \faLock   Closed-source LCMs |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 32.05 | 38.12 | 33.48 | 2.02 | 53.90 | 64.25 | 56.76 | 2.17 | 76.25 | 76.67 | 76.39 | 1.12 |
| Claude-3.5-sonnet | 38.70 | 37.79 | 37.43 | 3.54 | 54.45 | 50.48 | 51.45 | 2.83 | 65.00 | 68.33 | 65.97 | 1.04 |
| o1-mini | 29.83 | 35.33 | 31.66 | 3.38 | 45.54 | 50.74 | 47.21 | 2.63 | 25.42 | 28.33 | 26.25 | 1.58 |
| \faUnlock   Open-source LCMs |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-3b-Ins | 7.13 | 5.83 | 6.00 | 1.75 | 9.53 | 9.71 | 8.41 | 2.33 | 12.08 | 12.50 | 12.22 | 1.12 |
| Phi-3.5-mini-Ins | 21.06 | 20.46 | 19.14 | 2.86 | 20.39 | 24.27 | 20.57 | 2.27 | 11.11 | 12.50 | 11.53 | 1.20 |
| Llama-3.1-8B-Ins | 22.68 | 24.73 | 22.64 | 2.59 | 51.86 | 57.58 | 53.50 | 2.08 | 34.31 | 35.83 | 34.72 | 0.99 |
| Glm-4-9B-chat | 29.00 | 28.66 | 28.05 | 2.21 | 54.54 | 55.62 | 53.58 | 1.78 | 46.53 | 50.83 | 47.78 | 1.23 |
| Mistral-Nemo-Ins | 4.34 | 3.68 | 3.76 | 0.68 | 23.91 | 24.33 | 23.50 | 1.35 | 11.11 | 12.50 | 11.53 | 1.18 |
| Qwen2-57B-A14B-Ins | 4.90 | 3.43 | 3.82 | 1.27 | 22.63 | 22.54 | 21.61 | 1.80 | 15.28 | 15.83 | 15.42 | 1.17 |
| Llama-3.1-70B-Ins | 25.89 | 26.89 | 26.11 | 1.23 | 51.71 | 56.20 | 53.19 | 1.76 | 46.67 | 46.67 | 46.67 | 0.82 |
| ChatQA-2-70B | 21.75 | 22.54 | 21.92 | 1.12 | 47.67 | 51.25 | 48.77 | 1.29 | 38.33 | 38.33 | 38.33 | 0.95 |

**Table 4: Citation quality of LCMs in information-dispersed tasks within L-CiteEval.**
| Models | Multi-Doc QA | Summarization | Counting Stars |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ | $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ | $\mathbf{CP}$ | $\mathbf{CR}$ | $\mathbf{F_{1}}$ | $\mathbf{N}$ |  |
| \faLock   Closed-source LCMs |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o | 57.48 | 58.50 | 56.10 | 1.71 | 34.37 | 54.28 | 41.60 | 22.86 | 83.37 | 81.18 | 81.71 | 4.54 |
| Claude-3.5-sonnet | 66.85 | 55.62 | 58.58 | 2.44 | 36.70 | 55.03 | 43.45 | 17.70 | 73.01 | 75.83 | 73.15 | 4.81 |
| o1-mini | 49.95 | 49.60 | 48.58 | 1.78 | 20.23 | 33.61 | 24.83 | 19.58 | 34.06 | 46.46 | 38.45 | 6.73 |
| \faUnlock   Open-source LCMs |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-3b-Ins | 13.17 | 8.04 | 9.37 | 1.96 | 7.72 | 12.15 | 9.09 | 9.52 | 3.82 | 1.48 | 2.01 | 1.66 |
| Phi-3.5-mini-Ins | 11.89 | 10.25 | 10.53 | 1.71 | 10.90 | 10.94 | 9.60 | 8.23 | 4.19 | 3.67 | 4.09 | 3.48 |
| Llama-3.1-8B-Ins | 43.41 | 42.15 | 41.64 | 1.62 | 19.57 | 23.03 | 20.83 | 18.31 | 16.87 | 18.26 | 19.18 | 4.19 |
| Glm-4-9B-chat | 47.91 | 44.75 | 45.09 | 1.64 | 29.16 | 37.29 | 31.92 | 11.38 | 18.15 | 15.69 | 16.21 | 4.52 |
| Mistral-Nemo-Ins | 17.61 | 15.45 | 15.85 | 0.70 | 11.21 | 14.85 | 12.40 | 5.45 | 3.09 | 2.92 | 3.26 | 2.32 |
| Qwen2-57B-A14B-Ins | 17.30 | 12.07 | 13.61 | 1.06 | 4.01 | 3.37 | 3.19 | 3.81 | 4.37 | 4.37 | 4.24 | 4.24 |
| Llama-3.1-70B-Ins | 49.64 | 54.02 | 50.74 | 1.42 | 25.50 | 31.99 | 27.91 | 11.78 | 66.85 | 61.74 | 63.73 | 4.37 |
| ChatQA-2-70B | 47.20 | 49.51 | 47.92 | 1.10 | 19.57 | 23.60 | 20.89 | 11.81 | 14.02 | 11.22 | 13.22 | 3.49 |

#### 4.1.1 Analysis of Citation Quality

##### Open-source LCMs versus Closed-source LCMs

Overall, there is still a significant performance gap between open-source LCMs and closed-source LCMs (excluding o1-mini), especially in tasks involving the reasoning step.
Specifically, we can observe that: (1) closed-source LCMs generally provide more accurate citations (larger $F_{1}$ score) and tend to cite more segments with the context (larger value of $N$); (2) in the Dialogue Understanding task, the performance of the strongest open-source LCMs (Llama-3.1-70B-Instruct) has approached that of the closed-source LCMs. However, in other tasks requiring reasoning, particularly in synthetic tasks, although strong open-source LCMs like GLM-4-9B-Instruct cite a similar number of segments as the closed-source models, the quality of these citations is lower, resulting in a performance gap of nearly 20 $F_{1}$ points.

##### Performance of Open-source LCMs

In general, there is significant room for open-source LCMs to improve, and medium-sized open-source LCMs (Llama-3.1-8B-instruct and GLM-4-9B-Chat) are highly competitive, with performance that matches or even exceeds that of large LCMs (Llama-3.1-70B-instruct).
More concretely, our findings are:
(1) The improvement in citation quality does not directly correlate with the increase in model parameters. As the number of model parameters increases, citation performance does not consistently improve, but overall, large LCMs (70B) perform well, and medium-sized LCMs (8B and 9B) show very promising results;
(2) The actual activated parameters of LCMs are crucial, as evidenced by the MoE LCM (Qwen2-57B-A14B) exhibiting significantly lower citation quality, even under-performing small dense LCMs such as Phi-3.5-mini-instruct;
(3) Training data diversity is essential for LCMs. Taking ChatQA-2-70B, which is primarily trained on QA task datasets, as an example, we can observe that ChatQA-2-70B performs exceptionally well on Single-Doc QA tasks and Multi-Doc QA tasks but struggles significantly with the synthetic tasks and summarization tasks.

##### Performance of Closed-source LCMs

Among closed-source LCMs, GPT-4o and Claude-3.5-sonnet demonstrate strong performance on L-CiteEval, with GPT-4o surpassing all the experimental open-source LCMs across all tasks in citation quality.
Notably, while o1-mini achieves unparalleled results in reasoning tasks such as GSM8K (Cobbe et al., ) and Livecodebench (Jain et al., ), its citation generation capability significantly deteriorates in long-text scenarios.
Particularly in synthetic tasks and summarization tasks, which require LCMs to search for dispersed key information and use the retrieval information to respond, o1-mini’s performance is significantly inferior to strong open-source LCMs, such as Llama-3.1-70B-instruct.
This suggests that the o1-mini model falls short in retrieving key information from the context for responding.

**Table 5: Generation quality of LCMs on L-CiteEval, where $\dagger$ denotes the NIAH results, $\ddagger$ denotes the Counting Stars results, and Summ. denotes the summarization task.**
| Models | Single-Doc QA | Multi-Doc QA | Summ. | Dialogue | Synthetic |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prec. | Rec. | Prec. | Rec. | Rouge-L | Prec. | Rec. | Rouge-1^† | Acc^‡ |  |
| \faLock   Closed-source LCMs |  |  |  |  |  |  |  |  |  |
| GPT-4o | 11.78 | 70.37 | 10.34 | 87.38 | 20.15 | 9.81 | 65.35 | 89.24 | 91.88 |
| Claude-3.5-sonnet | 5.96 | 71.96 | 4.30 | 80.77 | 22.06 | 3.71 | 57.80 | 88.33 | 69.65 |
| o1-mini | 10.30 | 66.44 | 7.36 | 64.25 | 19.22 | 7.02 | 54.27 | 54.98 | 57.29 |
| \faUnlock   Open-source LCMs |  |  |  |  |  |  |  |  |  |
| Qwen2.5-3b-Ins | 8.91 | 60.28 | 3.82 | 52.41 | 22.39 | 4.58 | 40.77 | 84.49 | 26.81 |
| Phi-3.5-mini-Ins | 8.62 | 62.34 | 7.82 | 64.54 | 19.48 | 11.39 | 52.77 | 73.83 | 61.32 |
| Llama-3.1-8B-Ins | 10.11 | 68.13 | 7.66 | 68.84 | 20.90 | 11.07 | 58.84 | 85.11 | 33.75 |
| Glm-4-9B-chat | 11.22 | 67.25 | 7.88 | 77.97 | 21.42 | 7.69 | 51.25 | 90.81 | 58.82 |
| Mistral-Nemo-Ins | 10.53 | 59.71 | 8.78 | 67.70 | 20.83 | 9.27 | 49.26 | 87.88 | 18.06 |
| Qwen2-57B-A14B-Ins | 12.93 | 61.71 | 15.25 | 57.53 | 22.95 | 14.32 | 52.23 | 91.30 | 63.61 |
| Llama-3.1-70B-Ins | 15.23 | 67.08 | 12.50 | 76.40 | 22.29 | 19.62 | 62.91 | 88.18 | 89.03 |
| ChatQA-2-70B | 43.25 | 61.20 | 34.95 | 55.64 | 22.06 | 26.57 | 58.34 | 70.14 | 78.68 |

#### 4.1.2 Analysis of Generation Quality

From Table [5](#S4.T5), we can find:
(1) In Single-Doc QA, Multi-Doc QA, and Dialogue understanding tasks, closed-source LCMs significantly outperform open-source LCMs in recall scores.
This indicates that the statements of closed-source LCMs contain the correct answers.
However, closed-source LCMs tend to generate excessive statements to substantiate the results, consequently leading to lower precision scores.
In Summarization and Synthetic tasks, the gap between closed-source and strong open-source LCMs is small, as the corresponding evaluation results are close, e.g., 22.06 Rouge-L score of Claude-3.5-sonnet versus 22.95 Rouge-L score of Qwen2-57B-A14B-Instruct in Summarization tasks;
(2) Open-source LCMs tend to achieve better performance as the model parameters increase.
Combined with the mediocre citation quality of large LCMs mentioned above, we speculate that larger LCMs rely more on their internal knowledge (which might include task-specific information) rather than responding based on the provided context.
Consequently, their outputs are more often drawn from inherent knowledge rather than the context itself.
This finding is also consistent with the current research (Intel, ).

### 4.2 Model Performance on L-CiteEval-Length and L-CiteEval-Hardness

#### 4.2.1 Impact of Context Length for LCMs

We report the LCMs’ performance on L-CiteEval-Length in Fig. LABEL:fig:result_l_citeeval_length.
When keeping task difficulty constant but extending the context length, we can observe an overall decline in open-source LCMs’ performance.
Specifically, the smallest model, Llama-3.1-8B-Instruct, is the most affected by longer contexts.
For instance, in the HotpotQA task, its performance drops by around 20 points as the context length increases from 8K to 32K.
Larger models, such as Llama-3.1-70B-Instruct, are slightly impacted.
However, the closed-source LCM (GPT-4o) maintains a relatively stable performance, showing minimal degradation.
This suggests that open-source LCMs are more susceptible to irrelevant context, leading to a drop in both generation and faithfulness.
More details and model performance on L-CiteEval-Length benchmark are shown in Appendix [C](#A3).

#### 4.2.2 Impact of Task Difficulty for LCMs

We divide each task into different difficulty levels based on the generation quality of GPT-4o.
The LCMs’ performance on L-CiteEval-Hardness is shown in Fig. LABEL:fig:result_l_citeeval_hardness.
We observe that as task difficulty increases, the generation quality of LCMs generally decreases (except for the synthetic task Counting star, which open-source LCMs consistently perform poorly on).
However, citation quality does not display a consistent trend, though all LCMs demonstrate similar patterns across tasks.
This aligns with our intuition that faithfulness is not strongly correlated with task difficulty.
Besides, these results also underscore a gap between citation quality, which reflects the model’s ability to retrieve information from the context, and the generation quality of LCMs.
More details and model performance on L-CiteEval-Hardness benchmark are shown in Appendix [D](#A4).

## 5 Analysis

Given outstanding performance retrieval-augmented generation (RAG) on long-context understanding tasks (Li et al., ; Yu et al., ), we explore whether RAG can enhance long-context understanding in citation generation tasks.
Furthermore, we will analyze the relevance between the citations produced by LCM and its internal attention mechanisms.

Figure: Figure 5: Performance of RAG on 3 tasks in L-CiteEval, where the left group shows citation quality and the right group shows generation quality. Natural Q. refers to the Natural Question task.
Refer to caption: /html/2410.02115/assets/x4.png

### 5.1 Impact of RAG for Long-context Understanding with Citations

##### RAG Settings

We utilize the dense retriever GTR-T5-XXL (Ni et al., ) to identify the citation segments related to the question within the context.
For each question, we select the top 32 citation segments with the highest retrieval scores and concatenate these segments as input to the LCMs.
We conduct experiments on 6 tasks from the L-CiteEval benchmark.
Due to space constraints, we present the results for three representative tasks in Fig. [5](#S4.F5) and show all the results in Appendix. [B](#A2).

##### Result Analysis

We can observe that RAG can significantly enhance the citation quality of LCMs.
When equipped with RAG, the Llama-3.1-70B-Instruct model achieves substantial improvements over the baselines and demonstrates comparable or even superior performance compared to GPT-4o.
The Llama-3.1-8B-Instruct model also shows notable enhancement in citation quality.
However, overall, RAG may lead to a slight decline in generation quality, which could be attributed to the retrieval process of RAG resulting in the missing of some contextual information, preventing LCMs from leveraging the remaining information for accurate response.

Figure: Figure 6: Pearson correlation analysis between generated citations and attention mechanisms. The x-axis represents the number of correct citations produced by the model, and the y-axis represents the number of correct citation segments attended by the attention. The red curve indicates the fitted correlation, with closer alignment to the green curve signifying a higher correlation.
Refer to caption: /html/2410.02115/assets/x5.png

### 5.2 Relevance between Citation Generation and Attention Mechanism

Recently, Wu et al. () highlighted that LCMs can accurately identify token-level salient information within the context.
We explore whether the process of citation generation by LCMs is also reflected in the attention mechanisms.
Let the ground truth citation segment within the context be denoted as $g_{j}$.
Following Wu et al. (), we can use the retrieval score to determine whether the LCM’s attention focuses on the segment containing $g_{j}$ when generating the citation for $g_{j}$.
We find the positions that receive the most attention from all the attention heads.
If a position is located in the segment containing $g_{i}$ and the model’s output citation is exactly $g_{i}$, or if neither matches, we consider this a “correct retrieval”.
Otherwise, it is an “incorrect retrieval”.
We conduct the experiments on two tasks (HotpotQA and 2WikiMultihopQA) with two strong LCMs (Llama-3.1-8B-Instruct and GLM-4-9B-Chat).
We plot the number of citations generated by the models and the number of citation segments identified by the attention heads in Fig. [6](#S5.F6).
Ideally, if all citation positions exhibit “correct retrieval”, each data point would be distributed along the diagonal (i.e., the green dot line in  [6](#S5.F6)).
We utilized Pearson correlation analysis to calculate the correlation coefficient (r) between the generated citations and those retrieved by the attention mechanism, finding all the correlation values exceed 0.7.
This reveals the underlying mechanism by which we can leverage the model’s citation output to verify whether the model is truly responding based on the given context.

## 6 Conclusion

In this paper, we introduce L-CiteEval, a multi-task benchmark for long-context understanding with citations.
There are 5 major task categories, 11 different long-context tasks, with context
lengths ranging from 8K to 48K in L-CiteEval.
For reproducibility of evaluation results and the ease of use, we develop an automatic evaluation suite.
Additionally, considering the multitude of variables that affect model generation results, we developed two benchmark variants: L-CiteEval-Length and L-CiteEval-Hardness, which evaluate the LCMs from the context length and task difficulty aspects.
Experiments on 11 cutting-edge and widely used LCMs indicate that open-source LCMs are prone to generating responses based on their intrinsic knowledge rather than the context, while closed-source LCMs tend to provide more explanations, which significantly reduces generation accuracy.
We also find that RAG technology can significantly enhance the faithfulness of open-source LCMs, although it may lead to some loss in generation quality.
Furthermore, we reveal a correlation between the model’s citation generation process and its attention mechanism, demonstrating the validity of the citation generation approach and providing insights for future evaluations of LCM faithfulness.

## Appendix A Citation Precision and Recall Calculation

We provide the calculation process of Citation Precision ($CP$) and Citation Recall ($CR$) in Algo. [1](#alg1).

Figure: Algorithm 1 Calculate Citation Precision, Recall, and F1 Score

## Appendix B RAG Performance on L-CiteEval

Figure: Figure 7: Performance of RAG on 6 tasks in L-CiteEval, where the top group shows citation quality and the bottom group shows generation quality.
Refer to caption: /html/2410.02115/assets/x6.png

In this section, we provide all the RAG results, where we conduct experiments on 6 tasks with 5 different LCMs.
We present the comparison among each model in Fig. [7](#A2.F7).

## Appendix C Model Performance on L-CiteEval-Length

**Table 6: Model performance on L-CiteEval-Length.**
| Models | 0$\sim$8k | 8$\sim$16k | 16$\sim$32k |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\mathbf{F_{1}}$ | Rec. | $\mathbf{F_{1}}$ | Rec. | $\mathbf{F_{1}}$ | Rec. |  |
| NarrativeQA |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 62.08 | 62.63 | 46.67 | 61.36 | 33.25 | 64.84 |
| Qwen2.5-3b-Ins | 17.50 | 56.19 | 4.58 | 58.09 | 1.25 | 56.96 |
| Llama-3.1-8B-Ins | 43.01 | 61.99 | 39.17 | 64.41 | 40.27 | 62.55 |
| Qwen2-57B-A14B-Ins | 12.50 | 58.52 | 0.00 | 51.12 | 12.92 | 53.41 |
| Llama-3.1-70B-Ins | 59.17 | 63.42 | 51.67 | 63.24 | 47.50 | 62.86 |
| HotpotQA |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 65.67 | 95.67 | 63.50 | 95.25 | 63.75 | 89.62 |
| Qwen2.5-3b-Ins | 3.81 | 70.42 | 6.58 | 65.21 | 4.76 | 55.62 |
| Llama-3.1-8B-Ins | 51.83 | 85.25 | 40.56 | 81.04 | 40.83 | 67.75 |
| Qwen2-57B-A14B-Ins | 12.50 | 85.62 | 7.29 | 72.92 | 6.83 | 62.92 |
| Llama-3.1-70B-Ins | 67.50 | 89.42 | 56.67 | 90.25 | 49.17 | 85.25 |
| GovReport |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 56.68 | 23.07 | 48.82 | 21.48 | 44.45 | 20.65 |
| Qwen2.5-3b-Ins | 21.12 | 27.66 | 13.08 | 28.16 | 3.43 | 22.92 |
| Llama-3.1-8B-Ins | 57.08 | 24.27 | 38.28 | 24.15 | 18.46 | 19.25 |
| Qwen2-57B-A14B-Ins | 6.55 | 29.51 | 2.09 | 30.52 | 1.71 | 24.20 |
| Llama-3.1-70B-Ins | 57.55 | 25.41 | 43.60 | 23.43 | 17.64 | 21.62 |
| LoCoMo |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 78.13 | 68.07 | 73.91 | 66.93 | 72.24 | 68.77 |
| Qwen2.5-3b-Ins | 16.40 | 55.18 | 10.81 | 45.12 | 6.77 | 43.87 |
| Llama-3.1-8B-Ins | 76.51 | 68.68 | 63.54 | 68.39 | 63.91 | 61.33 |
| Qwen2-57B-A14B-Ins | 55.92 | 63.76 | 22.92 | 58.18 | 16.13 | 59.29 |
| Llama-3.1-70B-Ins | 75.45 | 73.21 | 71.27 | 70.53 | 64.38 | 57.89 |
| Counting Stars |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 97.30 | 93.33 | 92.71 | 83.33 | 92.95 | 88.75 |
| Qwen2.5-3b-Ins | 2.67 | 37.08 | 5.17 | 32.50 | 0.00 | 29.58 |
| Llama-3.1-8B-Ins | 42.93 | 42.08 | 35.64 | 33.75 | 18.70 | 20.00 |
| Qwen2-57B-A14B-Ins | 27.21 | 45.00 | 10.51 | 77.92 | 0.89 | 46.25 |
| Llama-3.1-70B-Ins | 76.96 | 56.67 | 74.93 | 66.25 | 65.14 | 58.33 |

We report all the evaluation results in Tab. [7](#A4.T7), where we test with 5 LCMs on 5 tasks in L-CiteEval-Length.

## Appendix D Model Performance on L-CiteEval-Hardness

**Table 7: Model performance on L-CiteEval-Hardness.**
| Models | Easy | Medium | Hard |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\mathbf{F_{1}}$ | Rec. | $\mathbf{F_{1}}$ | Rec. | $\mathbf{F_{1}}$ | Rec. |  |
| NarrativeQA |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 40.83 | 100.00 | 46.25 | 69.67 | 54.92 | 19.16 |
| Qwen2.5-3b-Ins | 11.67 | 75.00 | 4.58 | 60.02 | 7.08 | 36.22 |
| Llama-3.1-8B-Ins | 27.92 | 94.17 | 52.08 | 69.78 | 42.44 | 25.0 |
| Qwen2-57B-A14B-Ins | 5.00 | 75.00 | 15.42 | 63.13 | 5.00 | 24.92 |
| Llama-3.1-70B-Ins | 43.75 | 94.17 | 55.83 | 70.76 | 58.75 | 24.60 |
| HotpotQA |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 66.50 | 100.00 | 71.42 | 100.00 | 55.00 | 80.54 |
| Qwen2.5-3b-Ins | 3.81 | 71.25 | 3.67 | 66.46 | 7.68 | 53.54 |
| Llama-3.1-8B-Ins | 52.67 | 88.75 | 41.39 | 82.29 | 39.17 | 63.00 |
| Qwen2-57B-A14B-Ins | 12.50 | 83.12 | 5.62 | 73.33 | 8.50 | 65.00 |
| Llama-3.1-70B-Ins | 65.83 | 91.25 | 59.17 | 92.50 | 48.33 | 81.17 |
| GovReport |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 49.95 | 15.26 | 51.27 | 10.86 | 48.74 | 9.24 |
| Qwen2.5-3b-Ins | 14.32 | 16.28 | 9.31 | 14.65 | 14.00 | 14.37 |
| Llama-3.1-8B-Ins | 40.35 | 15.55 | 34.75 | 12.09 | 38.72 | 12.57 |
| Qwen2-57B-A14B-Ins | 3.48 | 30.02 | 3.26 | 25.37 | 3.61 | 28.85 |
| Llama-3.1-70B-Ins | 37.47 | 16.36 | 35.46 | 13.04 | 45.86 | 11.98 |
| LoCoMo |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 78.52 | 100.00 | 71.37 | 85.30 | 74.39 | 18.47 |
| Qwen2.5-3b-Ins | 8.44 | 69.12 | 15.85 | 60.09 | 9.70 | 14.96 |
| Llama-3.1-8B-Ins | 76.17 | 96.62 | 70.07 | 82.06 | 57.72 | 19.73 |
| Qwen2-57B-A14B-Ins | 44.17 | 84.23 | 15.58 | 73.67 | 35.21 | 23.32 |
| Llama-3.1-70B-Ins | 81.64 | 93.56 | 67.24 | 79.3 | 62.21 | 28.76 |
| Counting Stars |  |  |  |  |  |  |
| GPT-4o-2024-05-13 | 100.00 | 100.00 | 100.00 | 100.00 | 82.96 | 65.42 |
| Qwen2.5-3b-Ins | 1.33 | 36.67 | 4.51 | 40.00 | 2.00 | 22.50 |
| Llama-3.1-8B-Ins | 40.18 | 32.50 | 30.05 | 28.33 | 27.04 | 35.00 |
| Qwen2-57B-A14B-Ins | 21.71 | 49.17 | 5.74 | 57.08 | 11.16 | 62.92 |
| Llama-3.1-70B-Ins | 77.16 | 54.17 | 69.21 | 58.75 | 70.66 | 68.33 |

We report all the evaluation results in Tab. [6](#A3.T6), where we test with 5 LCMs on 5 tasks in L-CiteEval-Hardness.

## Appendix E Cases study

We provide all the prompts as well as all the model generation results for each task from Fig. [16](#A5.F16) to Fig. [43](#A5.F43).

Figure: Figure 8: GPT-4o in NarrativeQA

Figure: Figure 9: Claude-3.5-sonnet in NarrativeQA

Figure: Figure 10: Phi-3.5-mini-Instruct in NarrativeQA

Figure: Figure 11: Llama-3.1-8B-Instruct in NarrativeQA

Figure: Figure 12: Qwen2-57B-A14B-Instruct in NarrativeQA

Figure: Figure 13: Llama-3.1-70B-Instruct in NarrativeQA

Figure: Figure 14: GPT-4o in HotpotQA

Figure: Figure 15: Claude-3.5-sonnet in HotpotQA

Figure: Figure 16: Phi-3.5-mini-Instruct in HotpotQA

Figure: Figure 17: Llama-3.1-8B-Instruct in HotpotQA

Figure: Figure 18: Qwen2-57B-A14B-Instruct in HotpotQA

Figure: Figure 19: Llama-3.1-70B-Instruct in HotpotQA

Figure: Figure 20: GPT-4o in MultiNews

Figure: Figure 21: Claude-3.5-Sonnet in MultiNews

Figure: Figure 22: Phi-3.5-mini-Instruct in MultiNews

Figure: Figure 23: Llama-3.1-8B-Instruct in MultiNews

Figure: Figure 24: Qwen2-57B-A14B-Instruct in MultiNews

Figure: Figure 25: Llama-3.1-70B-Instruct in MultiNews

Figure: Figure 26: GPT-4o in LoCoMo

Figure: Figure 27: Claude-3.5-Sonnet in LoCoMo

Figure: Figure 28: Phi-3.5-mini-Instruct in LoCoMo

Figure: Figure 29: Llama-3.1-8B-Instruct in LoCoMo

Figure: Figure 30: Qwen2-57B-A14B-Instruct in LoCoMo

Figure: Figure 31: Llama-3.1-70B-Instruct in LoCoMo

Figure: Figure 32: GPT-4o in NIAH

Figure: Figure 33: Claude-3.5-Sonnet in NIAH

Figure: Figure 34: Phi-3.5-mini-Instruct in NIAH

Figure: Figure 35: Llama-3.1-8B-Instruct in NIAH

Figure: Figure 36: Qwen2-57B-A14B-Instruct in NIAH

Figure: Figure 37: Llama-3.1-70B-Instruct in NIAH

Figure: Figure 38: GPT-4o in Counting Stars

Figure: Figure 39: Claude-3.5-Sonnet in Counting Stars

Figure: Figure 40: Phi-3.5-mini-Instruct in Counting Stars

Figure: Figure 41: Llama-3.1-8B-Instruct in Counting Stars

Figure: Figure 42: Qwen2-57B-A14B-Instruct in Counting Stars

Figure: Figure 43: Llama-3.1-70B-Instruct in Counting Stars

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Are the evaluation metrics entirely automatic or not? How are the precision and recall evaluated on the QA benchmarks for the quality evaluation?
soundness: 2
strengths: 1. The proposed benchmark contains more tasks and longer contexts that can better evaluate LCMs.

2. The authors conduct comprehensive experiments on an extensive collection of LCMs.
summary: The authors introduce a multi-task benchmark called L-CiteEval to evaluate long-context models (LCM) from the perspective of generation quality and citation quality (i.e., how the response is correctly supported by information from the context). Compared to existing long-context evaluation benchmarks, L-CiteEval has a wider range of tasks, data points with longer context lengths, and strictly categorizes tasks by length intervals. The authors also curate two variants, L-CiteEval-Length and L-CiteEval-Hardness, to decouple the difficulty of the tasks from the long-context understanding in the evaluation. The authors test 11 LCMs on L-CiteEval and find that 1) open-source models underperform closed-source models in terms of citation quality; 2) RAG improves LCMs' faithfulness; 3) There is a correlation between the model’s citation generation process and its attention mechanism.
weaknesses: 1. Besides more tasks, longer contexts, and controlable lengths, the introduced benchmark L-CiteEval has limited novelty compared to the existing benchmark LongCite. The authors fail to demonstrate the necessity of L-CiteEval - what are the research questions that can be answered only by L-CiteEval but not LongCite. 

2. In the related works, the authors mention that "L-CiteEval relies entirely on automatic evaluation metrics without reliance on GPT-4 or human judgments" and list it as an advantage of L-CiteEval compared to LongCite. But in the limitation section (Section A), the authors state "current evaluation metrics still heavily rely on human judgment or model outputs." as the limitation of the work. This is confusing.

3. There exist errors in the manuscript. For example, the x-axis of Figure 4(a) is inconsistent with its caption and its reference section (Section 4.2.1). This causes miscomprehension of the section and the figure. A typo on line 253, L-CiteEval-Quality -> L-CiteEval-Hardness.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) Are certain tasks particularly challenging for LCMs in terms of citation faithfulness?
2) Did the authors observe any trends in model performance as context length increased? Are there critical thresholds beyond which models struggle with citation or response quality?
soundness: 1
strengths: 1) L-CiteEval thoroughly assessed LCMs on response quality and citation faithfulness, covering diverse tasks with up to 48K tokens, filling a key gap in LCM evaluation.
2) The benchmark’s emphasis on citation accuracy is crucial for research applications, ensuring models rely on provided context rather than intrinsic knowledge.
3) Evaluating both open-source and closed-source models, the paper identifies performance gaps, highlighting challenges for open-source LCMs and the benefits of Retrieval-Augmented Generation (RAG) for citation reliability.
summary: This paper introduces L-CiteEval, a benchmark specifically designed to evaluate long-context models (LCMs) on their ability to effectively use extended contexts for generating responses and maintaining citation faithfulness. With the rising use of LCMs for tasks that require processing extensive information, such as summarization and question answering over large documents, accurately assessing whether these models rely on the provided context or default to intrinsic knowledge is essential. L-CiteEval aims to address this need by providing a systematic, automated evaluation suite across 11 diverse tasks, with context lengths ranging from 8K to 48K tokens. This setup enables a robust assessment of how well LCMs handle the unique demands of long-context scenarios.

L-CiteEval evaluates models along two dimensions: response quality, which measures the relevance and accuracy of the generated answers, and citation faithfulness, which assesses whether the model’s responses are appropriately grounded in the source material. To explore the impact of context length and task complexity, the benchmark includes two variants—L-CiteEval-Length and L-CiteEval-Hardness—allowing for an analysis of model performance based on different context lengths and task difficulties. Through these variants, L-CiteEval provides a nuanced view of how well models manage extended contexts and remain faithful to provided information, which is particularly relevant for applications in research and academia where accurate source attribution is critical.

The authors benchmarked 11 long-context models, including both open-source and closed-source options, revealing significant differences in citation faithfulness. Their findings indicate that open-source models often struggle to maintain citation accuracy, with a tendency to rely on intrinsic knowledge rather than contextual information. Notably, incorporating the Retrieval-Augmented Generation (RAG) approach improved citation accuracy but slightly reduced the quality of response generation. Additionally, the study identifies correlations between a model’s citation performance and its attention mechanisms, offering insights into how model architecture may influence citation faithfulness.

While the paper makes a contribution by introducing a comprehensive benchmark, it does not propose new technical methods or model architectures. Additionally, it remains unclear how the insights gained from its evaluations will directly inform or advance the development of LCMs. Although L-CiteEval addresses a gap in assessing LCMs’ contextual understanding and citation accuracy, the paper could benefit from more explicit guidance on how these findings could be leveraged to improve LCM design and performance in practical applications.
weaknesses: 1) The paper identifies gaps in citation accuracy but explores few methods beyond RAG to improve performance.
2) Heavy dependence on automated scoring may overlook nuances; human evaluation could add reliability.
3) It is unclear how the insights gained from its evaluations will directly inform or advance the development of LCM in practice.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: NA
soundness: 2
strengths: 1.) This paper introduces L-CiteEval, a benchmark for long-context understanding with citations for several open-source and closed-source LLMs.

2.) The paper is well-written in a particular context and a due literature review has been conducted.

3.) The analysis done in the paper is good.
summary: This paper introduces L-CiteEval, a multi-task benchmark for long-context understanding with citations for LLMs. Although the paper is well-written, it is potentially unfit for ICLR and best suited for linguistics venues.
weaknesses: 1.) I found this paper unfit for ICLR. Though it is about datasets and benchmarks yet there is nothing about learning representations or core machine learning. The paper is best suited for core NLP venues. As we can go through the references section, almost every referenced paper is from arXiv, and the very few remaining ones are from computational linguistics.

2.) The x-axis in Fig.4(a) for all the subplots is wrong. It should not be (Easy, Medium and Hard, i.e. difficulty) but should be the values of varying context length. It is imperative that the authors double-check all axis labels and legends to ensure consistency with the described experiments and accurately reflect the data being presented.

3.) The legends on both the x-axis and the y-axis are missing in Fig.3 and Fig.4. Adding clear and descriptive axis labels to all figures to improve readability and interpretation of the results is a general practice that is followed in the scientific domain.

4.) The captions for all the figures should be descriptive enough instead of small one-liners. Detailed captions ensure that the figure conveys the requisite information and the reader is not compelled to read the complete main text.

5.) The claim “However, the closed-source LCM (GPT-4o) maintains a relatively stable performance, showing minimal degradation.” in line 418 is incorrect. It is clearly visible from several subplots in Fig.4(a) that there is high degradation. E.g., NarrativeQA F_1, GovReport F_1, NarrativeQA Rec., HotpotQA Rec., GovReport Rec. Moreover, just taking three points on the x-axis is not sufficient to establish a trend and thus claim a phenomenon. Since it is a benchmarking paper, it should be taken into account carefully. So, authors need to provide a more nuanced discussion of performance variations across different tasks and metrics, do extensive and accurate benchmarking, and thus derive valid inference and claims.

6.) In line 477, it should be “performance of retrieval”.

There may be other weaknesses too but I did not give the paper a second read.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Have the authors compared model performance on genuinely long texts versus artificially padded ones? Insights into how padding affects the model’s citation accuracy and response relevance could inform whether padding is a viable approach for long-context benchmarks.

Did the authors consider alternative or additional metrics that might capture nuanced citation quality more effectively? For example, integrating metrics that evaluate citation relevance based on context similarity could provide a more robust evaluation of citation faithfulness.
soundness: 2
strengths: L-CiteEval addresses an important challenge in LLM evaluation. With extremely lengthy contexts, it is increasingly difficult for humans to verify model performance manually.

The use of automated evaluation over LLM-based judgments mitigates the biases that can be introduced by using models like GPT-4 as evaluators.

While previous work, such as Bai et al. (2024)’s LongCite, already introduced citation benchmarking, L-CiteEval’s approach adds automated quality checks on both response content and citation integrity.
summary: The authors introduce the L-CiteEval benchmark to evaluate LLMs on long-context tasks with citations. L-CiteEval includes both real-world and synthetic tasks across 11 domains, with context lengths ranging from 8K to 48K tokens. Unlike existing benchmarks that often limit scope or rely on LLM judges, L-CiteEval assesses models based on both response and citation quality, using an automated evaluation suite for consistency and reproducibility. The evaluation reveals that open-source models tend to rely more on intrinsic knowledge, whereas closed-source models excel in citation accuracy. However, the authors note that current citation quality metrics may not fully capture nuanced aspects of citation faithfulness.
weaknesses: The authors mention using padding to extend short-context datasets. Relying solely on Named Entity Recognition (NER) to filter padding data might not fully eliminate unwanted noise, potentially impacting model response consistency.

L-CiteEval assembles its benchmark from pre-existing datasets, applying a padding strategy to extend short-context data to the target lengths. This approach, while practical, does not introduce new data or methodologies for long-context representation, which limits the novelty of the contribution. The primary innovation seems to be the padding process.

As the authors acknowledge, the current citation metrics used in L-CiteEval may not fully capture the intricacies of citation faithfulness, particularly in complex, multi-reference scenarios. This limitation restricts the benchmark’s ability to comprehensively assess whether models are citing accurately and contextually, which is essential in long-context tasks.