$\textit{L-CiteEval}$: A Suite for Evaluating Fidelity of Long-context Models
Authors: Zecheng Tang, Keyan Zhou, Juntao Li, Baibei Ji, jianye hou, Min Zhang
Accepted at: acl25


Reviewer paper summaries: Authors in this introduce L-CiteEval, a comprehensive suite designed to evaluate the fidelity of Long-context Models (LCMs). They assess the response quality and fidelity of LCMs, which are crucial for tasks requiring long-context understanding.
The benchmark covers 11 tasks with context lengths from 8K to 48K. The paper also goes on to describe an automatic evaluation pipeline.

This paper focuses on the long-context capability evaluation for LLMs. The main difference of this new benchmark from existing ones is the additional evaluation of citation quality. The authors also design the corresponding automatic evaluation metrics and the verification pipelines to ensure high-quality and reliable assessment.
Summary of strengths: Strengths:
1. Comprehensiveness: The study evaluates 11 state-of-the-art LCMs, offering insights into the performance differences between open-source and closed-source models.
2. Context Compression Methods: The paper explores context compression methods, demonstrating improvements in model fidelity despite a slight decline in response quality.
3. Citation Generation Analysis: The analysis of citation generation from both explicit model output and internal attention mechanisms adds a novel dimension to the evaluation of LCMs.

Relevance and Impact: Given the increasing importance of long-context understanding in NLP, this paper addresses a critical need and has the potential to influence future research and development in the field.

This paper propose a novel evaluation setting: the citation quality of LLMs when answering long-context questions, enabling a more comprehensive evaluation and understanding of the LLMs' capabilities.
Summary of weaknesses: 1. I am struggling to understand the context and use of Table 3 and 4. Authors can you explain how does showing citation metrics for different LLMs help make a case for L-CiteEval as a benchmark? The comparison doesn't even utilize other benchmarks of either long context or citation rates, to compare comprehensiveness of L-CiteEval. Please rewrite your conclusion section accordingly. 
2. It also seems like no specific model has consistent performance across all tasks in Table 3 and 4. Any comments/hypothesis? Maybe GPT4o shows this for one task, that is "Needle in haystick", but for all other tasks the numbers are inconsistent. 

Overall the paper does have merits as they introduced a new dataset but fails to convincingly demonstrate the use of this dataset.

The main concern on this paper is the contribution.

- All the datasets in this new benchmark are taken from previous work. Also, many previous long-context benchmarks have included some of these datasets for evaluation, making the evaluation results less useful [1-3].
- The citation quality evaluation is not novel. Although previous work have not include this metric under long-context scenarios, such an extension is very straightforward.

[1] Zhang, Xinrong, et al. "∞ Bench: Extending long context evaluation beyond 100k tokens." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] Yen, Howard, et al. "Helmet: How to evaluate long-context language models effectively and thoroughly." arXiv preprint arXiv:2410.02694 (2024).

[3] Shaham, Uri, et al. "Zeroscrolls: A zero-shot benchmark for long text understanding." arXiv preprint arXiv:2305.14196 (2023).
Comments, suggestions and typos: Formatting:
Several lines have inconsistent spacing - example Line 499 has inconsistent spacing
Grammatical errors
Line 156
Line 272

N/A
Overall assessment scores: 3

2.5
Metareview score: 3
Metareview: This paper introduces L-CiteEval, a benchmark of three datasets comprising several tasks for long-context evaluation. It is claimed that the main difference of this new benchmark from existing ones is to address the fidelity issue in the evaluation of long-context models. The authors also design the corresponding evaluation metrics and the verification pipelines to ensure reliable assessment.
Summary of reasons to publish: The reviewers appreciated the following points:
* The Long-context evaluation problem is an important one
* The proposed dataset addresses several known and significant limitations of existing long-context evaluation datasets, such as fixed length and fixed question difficulty. 
* The amount of work done, including benchmarking of several LLMs, is appreciated.