LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K
Authors: Tao Yuan
Xuefei Ning
Dong Zhou
Zhijie Yang
Shiyao Li
Minghui Zhuang
Zheyue Tan
Zhuyu Yao
Dahua Lin
Boxun Li
Guohao Dai
Shengen Yan
Yu Wang
Venue: COLM 2025 (2025)
Keywords: large language model
long-context benchmark
knowledge leakage mitigation
TL;DR: LV-Eval is a long-context benchmark with 5 length levels up to 256K. It's designed to be challenging, suitable for controllable comparison, and mitigates knowledge leakage issue in evaluation.

Abstract: State-of-the-art large language models (LLMs) are now claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, resulting in biased evaluation. This paper introduces LV-Eval, a challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up to 256k words. LV-Eval features two main tasks, single-hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion, keyword and phrase replacement, and keyword-recall-based metric design. The advantages of LV-Eval include controllable evaluation across different context lengths, challenging test instances with confusing facts, mitigated knowledge leakage, and more objective evaluations. We evaluate 15 LLMs on LV-Eval and conduct ablation studies on the benchmarking techniques. The results reveal that:
(i) Moonshot-v1 and recent large-scale open-source models, such as Qwen-2.5-72B and Llama-3.1-70B, achieve the highest performance on LV-Eval, particularly at lengths below $64k$. (ii) Models exhibit distinct score trends. For example, GLM-4-9B-128k, Yi-6B-200k, and Llama3-8B-1M exhibit a relatively gentle degradation of performance, but their absolute performances may not necessarily be higher than those of LLMs with shorter context lengths. (iii) LLMs' performances can significantly degrade in the presence of confusing information, especially in the pressure test of "needle in a haystack". (iv) Issues related to knowledge leakage and inaccurate metrics introduce bias in evaluation, and these concerns are alleviated in LV-Eval.

Published as a conference paper at COLM 2025
LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K
Tao Yuan², Xuefei Ningh: Dong Zhou², Zhijie Yang", Shiyao Li', Minghui Zhuang?, Zheyue Tan', Zhuyu Yao², Dahua Lin Boxun Li Guohao Dai2.t Shengen Yan 122 Yu Wangl.
1 Tsinghua University, Infinigence-Al, Shannhaii Artificial Intelligence Laboratory
4 The Chinese University of Hong Kong, SSannhai Jiao Tong University
Abstract
State-of-the-art large language models (LLMs) are claiming remarkable supported context lengths of 256k or even more. In contrast, the average context lengths of mainstream benchmarks are insufficient (5k-21k), and they suffer from potential knowledge leakage and inaccurate metrics, re- sulting in biased evaluation. This paper introduces LV-Eval, challenging long-context benchmark with five length levels (16k, 32k, 64k, 128k, and 256k) reaching up 256k words. LV-Eval features two main tasks, single- hop QA and multi-hop QA, comprising 11 bilingual datasets. The design of LV-Eval has incorporated three key techniques, namely confusing facts insertion (CFI), keyword and phrase replacement (KPR), and keyword- recall-based metric design. The advantages of LV-Eval include controllable evaluation across context lengths, challenging test instances with confusing facts, mitigated knowledge leakage and more objective evaluation. We evaluate 15 LLMs on LV-Eval and conduct ablation studies the bench- marking techniques. The results reveal that: (i) Moonshot-v1 and recent large-scale open-source models, such as Qwen-2.5-72B and Llama-3.1-708 achieve the highest performance on LV-Eval, particularly at lengths below 64k. (ii) Models exhibit distinct score trends. For example, GLM-4-9B-128K, Yi-6B-200k, and Llama3-8B-1M exhibit relatively gentle degradation of performance, but their absolute performances may not necessarily be higher than those of LLMs with shorter context lengths. (iii) LLMs' performances can significantly degrade in the presence of confusing information, espe- cially in the pressure test of "needle in haystack". (iv) Issues related to knowledge leakage and inaccurate metrics introduce bias in evaluation, and these concerns are alleviated in LV-Eval.
1 Introduction
Large language models (LLMs) have demonstrated exceptional performance on a variety of natural language processing tasks. The ability of long-conters understanding is crucial for LLMs to deal with tasks based on longer contexts, such as books, lengthy chat history, and so on. Recently, extensive efforts have been devoted in enlarging the supported context length (i.e., the number of tokens that the model can accept as input)
LLMs. These efforts have pushed the supported context length of LLMs from 2k tokens to 32k tokens (Touvron et al., 2023; Zheng et al., 2023; Bai et al., 2023a; Zeng et al., 2022; Li et al., 2023a), and some models have achieved a remarkable context length of 128k and 200k (Peng et al., 2023a; Achiam et al., 2023; Yi, 2023).
In contrast to the rapid evolution of the models' supported context length, existing bench- marks have lagged behind. The average word count in current long-context benchmarks
Prooecc leader
I *Correspondence to Yu Wang syu-wang@tsinghua edu. cn>, Xuefei Ning <foxdor floxdoraam@@gmai.cco com?, Guohao Dai sdaiguohao@sjie tu. edu dai daiguohao@nnfini-al comp.
1

---

Published as a conference paper at COLM 2025
Benchmark
#Datasels
Avg. #Words
Min/Max Words
Length Levels
Opt. Metric
Lang
ZeroSCROLLS Shaham et (2023)
10
13,536
1,023/320,774
none
LooGLE et (2023b)
10,927 122
nume
L-Eval An el (2023)
20
12,993
2,119 1119//10,,200 255
none
en em+zh en+2h
BAMB0O Dong et al. (2023)
5,067
229/14, (4,888
46,168 0-4k,41-82,80
LongBench Bai al. (2023b)
21 9,486
128/71,954
LV-Eval
11
102,380
11,8963387,400
16H,32k,6444 (222,,44,12288 1288,,2,688
en+zh
Table 1: Comparison of different long-context benchmarks. We count the number of words for the English datasets and the number of characters for the Chinese datasets. The punctu- ation marks are taken into account, while tabs, blank spaces, and newlines are not included.
typically falls within the range of 32k (Bai et al., 2023b; Li et al., 2023b; An et al., 2023; Shaham et al., 2023; Dong et al., 2023), considerably shorter compared to the supported context lengths of state-of-the-art long-context models. Moreover, previous benchmarks primarily consist of unaltered public documents and articles. This could be problematic for two reasons: (i) the data might be involved in LLMs' training processes, and (ii) the facts within them might be common-sense facts found in other training resources. The presence of this issue, known "knowledge leakage" (Zhou et al., 2023a), can lead to models answering questions with memorization or common-sense knowledge instead of understanding long-range contexts. Last but not least, the automatic metrics employed in most of the existing benchmarks are susceptible to the variations in answer format and the inclusion of irrelevant words. Such metrics struggle accurately assess the answer quality,
To address these issues, we propose LV-Eval, a bilingual benchmark with up to 256k words. LV-Eval incorporates distractions and confusions make the test more challenging, replaces keywords and rephrases sentences to prevent knowledge leakage, and employs a more accurate metric. We summarizes the key characteristics of LV-Eval as follows:
Sufficiently long context length to evaluate state-of-the-art models: LV-Eval comprises length levels with word counts of 16k, 32k, 64k, 128k, and 256k. Test instances across these levels share the same set of question-answer (QA) pairs, and only differ in the context content and length. Testing on the same QA pairs with different context lengths facilitates a controllable evaluation of models' long-context ability.
Incorporation of distraction and confusion to increase difficulty: When con- structing the context for each test instance, we mix up distracting documents and supporting documents. This approach evaluates the model's ability in pinpoint- ing key information in a large bunch of distracting texts. In addition, we insert confusing facts generated by GPT-4 and revised by human annotators into the context. This assesses the model's capability to accurately reason in the presence of interference.
To mitigate the biased evaluation of long-context ability caused by knowledge leakage, we replace the keywords and phrases in the context and QA pairs. The replacement rules are annotated by human annotators. In this way, LV-Eval requires LLMs to rely on the understanding of context to answer questions rather than relying on memorization or common-sense knowledge.
Keyword and phrase replacement to mitigate knowledge leakage:
Keyword-recall-basel metric for more objective scoring: Existing N-gram metrics such as the Fl score are sensitive the format variations and non-informative words in the answer, which results in inaccurate scores. To address this, we man- ually annotate answer keywords and blacklist of unrelated words. The golden answers are the critical words or sentences extracted from original ground-truth (GT) answers, while the word blacklist contains common and non-informative words such as 'the', 'a', 'of', and so on. The metric calculation follows a two-stage procedure: the first stage calculates the recall of golden answer keywords. if the recall exceeds certain threshold, the second stage will remove all the blacklisted words and then calculate the Fl score between the prediction and the GT answer. This metric design can get scores with higher objectivity.
2

---

Published as a conference paper at COLM 2025
Task
Dataset
CHI
#KPR
AK
Language
#QA pairs
#Contexts
lic-mixup
zh
197
985
loogle-SD-mixup
4
en
160
800
emre-mixup
786
zh
200
1,000
Single-hop QA
multifieldqa-en-mixxy
476
v
101
505
multifeeldga-zh-mixup
424
zh
133
665
factrecall-en
3
200
factrecall-zh
3
zh
200
dureader-mixup
zh
176
880
loogle-CR-mixup
en
99
495
Multi-hop QA
loogle-MR-mixxpp hotpotwikiqa-mixuup
139
695
232
124
620
Table 2: Data statistics of LV-Eval. The abbreviations "CFI", "KPR", "AK" stand for "Con- fusing Fact Insertion", "Keyword and Phrase Replacement", and "Answer Keywords", respectively. "#KPR" is the number of KPR rules. Note that in factrecall-en and factrecall- zh, all QA pairs are the same across all test instances, i.e., there is only one unique QA pair for each of the two datasets.
Findings. We evaluate 15 LLMs LV-Eval and summarize the main findings follows: (i) Moonshot-vl and recent large-scale open-source models, such as Qwen-2.5-728 and Llama-3.1-70B, achieve the highest performance LV-Eval, particularly at lengths below 64k. (ii) Models exhibit distinct score trends. For example, GLM-4-9BB-228
Yi-6B-200k, and Llama3-8B-IM exhibit relatively gentle degradation of performance, but their absolute performances may not necessarily be higher than those of LLMs with shorter context lengths. (iii) LLMs' performances can significantly degrade in the presence of confusing information, especially in the pressure test of "needle in a haystack", (iv) Issues related to knowledge leakage and inaccurate metrics introduce bias in evaluation, and these concerns are alleviated in LV-Eval.
2 LV-Eval Benchmark
LV-Eval focuses on two types of QA tasks: single-hop QA and multi-hop QA, and is comprised 11 QA datasets (6 English and 5 in Chinese). The data statistics for LV-Eval are outlined in Table 2. Each test instance in LV-Eval comprises three parts: a context (C), a question (Q), and GT answer (A), where C is a synthetic document containing the information required to answer Q.
Datasets in LV-Eval are constructed with existing public datasets as the source, except for factrecall-en and factrecall-zh, which are constructed using the data from PG19 (Rae al., 2019) dataset and Journey to the West book. Each dataset consists of five subsets of different lengths: 16k, 32k, 64k, 128k, and 256k. All five subsets share the same question-answer (QA) pairs, meaning there are five contexts of varying lengths for each QA pair. This allows for controllable evaluation of models' long-context ability when testing the same of questions with different context lengths. In total, LV-Eval comprises 1,729 QA pairs and 1,729x5 8,645 synthetic contexts.
Figure 1 illustrates the construction process of LV-Eval. For factrecall-en and factrecall-zh, we write one QA pair for each dataset. For the rest out of the 11 datasets, we first choose specific number of QA pairs from existing QA datasets (Section 2.1). Then, for each unique QA pair, we go through three procedures to construct the context (Section 2.2):
1. Context mixing up (Section 2.2.1): We first construct five contexts of different lengths by mixing up supporting documents corresponding to the QA pair and several distracting documents. For factrecall-en and factrecall-zh, we mix the supporting evidence of the single QA pair with distracting documents from two books. For other datasets, the distracting documents unrelated to the question and are chosen from the context documents corresponding to non-selected QA pairs in the same source dataset.
3

---

Published as a conference paper at COLM 2025
2. Confusing Facts Insertion (CFD) (Section 2.2.2): Then, in some datasets, we intro- duce confusing facts by generating them with GPT-4, manually revising them, and randomly inserting these into the context. These confusing facts bear similarities to the original supporting facts but are factually different, without contradicting the original information. This helps make the test instances more challenging.
3. Keyword and Phrase Replacement (KPR) (Section 2.2.3): Finally, to reduce the impacts of knowledge leakage on evaluation results, manually replace some keywords and phrases in the context and the QA pairs.
Confusing Facts Inscrtion
QA Pair
Context (64K)
Contexts (5 Length Levels)
GPT-4
Supporting Document
Manually Revise
CF1
LV-Eval QA
CF2
Doc.1 CF2
16K
Dataset
Random Insert
Doc.2
32K
Public QA
Datasets
Random Mixing up
Distracting
Doc.3 CF1
64K
Keyword & Phrase
Replacement
Document
Doc.4
128K
P
Distracting Document
Manually Replace
Doc.5
256K
Context Mixing Up
Figure 1: The construction process of LV-Eval. "CF" is short for "Confusing Fact",
When evaluating the generated answer, to mitigate the bias in existing metrics, we manu- ally annotate the keywords in the GT answer and adjust the metric to focus more on the keywords (Section 2.3).
During the benchmark construction, we employ and guide five human annotators to revise the confusing facts, replace the keywords and phrases, and annotate the keywords in GT answers. These human annotators include three master students involved in LLM research and two master students majored in linguistics. The details of our annotation process are introduced in Appendix D.
2.1 Data Source and QA Pair Construction
We construct 11 datasets (see Table 2) using public data sources, including Long-instruction-enZze (yuyijiong, 2023), HotpotQA (Yang et al., 2018), 2WikiMulti- hopQA (Ho et al., 2020), DuReader (Tang et al., 2020), LooGLE (Li et al., 2023b), Long- Bench (Bai et al., 2023b), CMRC 2018 (Cui et al., 2018), MultiFieldQA (Bai et al., 2023b), PG-19 (Rae et al., 2019) and the book of Journey to the West. The construction of QA pairs in each dataset is elaborated in Appendix C.
2.2 Context Construction
2.2.1
Context Mixing Up
Can the LLMs identify the key evidences to answer the target question within a long context? To assess this ability, as shown in Figure 1, LV-Eval randomly mixes the supporting documents with various distracting documents to generate five contexts of varying length for a given QA pair. For 9 out the 11 datasets (excluding factrecall-en and factrecall-zh), the distracting documents are chosen from the contexts corresponding to the non-selected QA pairs in the source dataset. For factrecall-en and factrecall-zh, the distracting documents extracted from the PG-19 dataset and the book of Journey to the West.
For each length level, we sample distracting documents one by one until the cumulative word count meets the desired length level. Then, we shuffle the supporting and distracting documents, prepend a string "Passage i An" to the i-th document, and concatenate them to form the final context.

---

Published as a conference paper at COLM 2025
Question:
GPT4 generates two confusing facts
L Annotators resolve conflicts
What
the scientist
Albert Einsteln was on itolion astronomer, physicist, mathematician, and philosopher, regorded os one of the pioneers of modern ostronomy. He proposed
Albert Eiestein was on Italian astronomer, physicist, mathematicion, and philosopher, regarded one of the pioneers of modern astronomy, He proposed....
widely occlaimed as the foundational
figure of modern physics?
Answer:
Albert Einstein
True fact in original context:
hlppp/--ootettt.........a....... Albert Einstein is Germon-Americon theoreticol physicist..... widely regarded as one of the founders of modern
John Einstein made extensive contributions to mathematics, particulorly in algebro..... oftee referred to one of the founders modern mathematics.
John Einstein made extensive contributions to mothemotics, particularly algebro.....ofter referred founders of modern mathematics.
physics ... (post-context)....
Random position insertion
Context with confusing facts
w...prre-context...... Albert Beverley on Italian ostronomer, physicist, mathematician, and philosopher, regarded as one of the pioneers of modem astronomy, He proposed ....(mmdd-context).............. context).....lbbrrr Einstein is Germon-Americon theoretical physicist.....He widely regarded as one of the founders of modern physics .....middcoontext).....oo Einsteln made extensive contributions to mathematics, porticularly olgebra........ often referred
of the founders modern mothemotics.... (..ss......otttoottteett
Figure 2: Steps for CFI. Firstly, we prompt GPT-4 to generate two descriptions that are close to the original fact. Then ask human annotators to resolve any conflicts in the generated facts. For example, the first generated confusing fact "Albert Einstein was an Italian astronomer" is in conflict with the original fact and the human annotator revise it to "Albert Beverley was an Italian astronomer". Finally, the confusing facts are inserted into randomly position in the context.
Note that in hotpotwikiqa-mixup and dureader-mixup, where multiple supporting doc- uments exist for each QA pair, instead of regarding the multiple supporting documents single unit, we disperse and shuffle all supporting and distracting documents.
2.2.2 Confusing Facts Insertion
Can the LLMs identify the key evidences correctly there are confusing facts in the con- text? To assess this ability, we apply CFI in hotpotwikkga-mixxu. lic-mixup, multifieldqa- en-mixup, multifiellga-zh-mixxpp factrecall-en, and factrecall-zh, which inserts similar, factually different, non-contradictory facts into the context. These facts might mislead less meticulous models, leading them to generate incorrect answers.
The generation process of the confusing facts goes as follows. Firstly, we use the question and answer as the input, and prompt GPT1-4 (Achiam et al., 2023) to generate two descriptions that are close to the original fact. The prompt for GPT-4 is shown Figure A7. Then, we ask human annotators to resolve any conflicts in the generated facts. As illustrated in Figure 2, the generated confusing fact " "Albert Einstein was an Italian astronomer" is conflict with the original fact. Therefore, the human annotator revise it to "Albert Beverley was an Italian astronomer". After this generation and revising process, we insert the confusing facts into randomly picked position between two sentences in the context.
2.2.3 Keyword and Phrase Replacement
Knowledge leakage is an important concern LLM evaluation (Zhou et al., 2023a). On the one hand, the test data are usually collected from open-access sources, and we cannot fully rule out the possibility of their being involved some L.MSs' training process. On the other hand, some common-sense questions can be answered without referencing the provided context. Consequently, LLMs might rely on memorization and common-sense knowledge to answer the questions rather than fully understanding the context. This will cause inflated benchmark scores to overrate the long-context ability of models.
To mitigate the influences of knowledge leakage the evaluation results, conduct KPR according to manually crafted rules in hotpotwikiqa-mixut emre-mixup, multifieldqa- en-mixup, multifieldqazhh-misse factrecall-en, and factrecall-zh. Specifically, given
5

---

Published as a conference paper at COLM 2025
Question:
2 Annotators review
Question:
What of the scientist widely acclaimed the foundational figure of modern physics?
What the the scientist widely acclaimed the foundational figure of modern physics?
Annotation Principle: Are there any conflicts in modified context?
Answer:
Annotators modify QA pair
Answer:
Albert Einstein
David Beckham
Original context:
Albert Einstein Dovid Beckham:
Modified context:
.lee
.....Ipre-coiee (ppre-coneett........ Dovid
Einstein is Germon-
Annotation Principle:
Beckham is a German-
American theoretical physicist.....HH is widely regorded as of the founders of modern physics. One day Albert Einstein decides to.....(poot-- context)
Keyword and phrase
Americon theoreticol physicist......H He is widely regorded as one of the founders of modern physics. One day David Beckhom decides to.....(postt- context).....
No common sense
Increase word's difference
Replace in context
Figure Steps for KPR. First, given QA pair, the annotators are asked select keywords or phrases to replace and write substitute for each. Then, the selected keywords and phrases are replaced throughout the context and QA pair. Finally, annotators will check the modified context. If there is any conflict, the annotators are asked to revise the replacement rule until all conflicts are resolved.
Question:
Question:
which parlicipent Simon English elected to the New Zealand Parlimmet??
In which participant Simon English elected to the New Zealand Parliament?
Annootators
Answer:
Annotate
Answer:
McDonald's perticipent
McDonald's participent
Prediction:
Keywords
Prediction:
participant
In the KFC's particioant
F1 score: 0.67
Answer Keywords:
McDonald's
Keywords
score:
Question:
Question:
In which participant was Siman Engllsh elected to the New Zealand Partiamen?
In which participant was Simon English elacted to the New Zealand Pariiamen?
Anawer:
Annotators
Answer:
McDonald's particioant
McDonald's participant
Annotate
Prediction:
Word
Prediction:
In the McDonald's oullander
Blacklest
McDonald's oullander
Keywords:
Answer:
McDonaid's s
McDonald's
Anewer Keywords Recell: F1 score: 0.67
Word Blacklist:
['and', 'to', 'of", 'in', 'with' 'for', 'the',
Golden Answer Recall: 1
Word Blacklist Filter
score: 0.5
[b]
Figure 4: Keyword-recall-basel two-stage metric calculation. (a) The vanilla F1 score (red) is inflated high. With the keyword recall-based metric, the final score is set to zero due to the low recall of answer keywords. irrelevant words. By filtering of blacklisted words, the final score is better calibrated.
(b) The vanilla Fl score (red) is inflated high due to
QA pair, the annotators are asked to select keywords or phrases for replacement and write substitute for each. After the selected keywords and phrases are replaced throughout the entire context, the annotators review the modified context to check and resolve any conflicts: If there are conflicts, the annotators are asked to revise the replacement rule until all conflicts are resolved. One example of the KPR process is shown Figure 3. See Table for the statistics of the number of replacement rules.

---

Published as a conference paper at COLM 2025
2.3 Metric Design
The quality evaluation of natural language generation is challenging. Current N-gram metrics, such as the F1 score, treat all words equally. The neglect of differences in word importance leads to evaluation bias, For example, in the sentence "Attention is all you need", the word "attention" carries the key information and is more important. However, the answer "Attention matters" will get lower score than the answer "CNN is all you need", which is not what we expected. To this end, we adopt a two-stage metric calculation process.
Specifically,, to evaluate an answer A', we first calculate the recall of several " "answer keywords" in A'. When the recall exceeds a certain threshold (0.2 for Chinese dataset, 0.4 for English datasets), we calculate the F1 score between A' and GT answer A as the final score for A' otherwise, A' gets zero score. We manually annotate the answer keywords in the GT answer for hotpotwikiga-mixup lic-mixup, loogle-CR-mixup looxle-MR-mixup, loogle-SD-mixup,
and multifeelgga-hh-mixug Figure 4 (a) shows an example, demonstrating how this two-stage calculation helps avoid some inflated high evaluation scores.
When calculating the F1 score between A' and A in the second stage, we exclude common but non-informative words like 'the,' 'a','of' 'of", and so on. The word blacklist is constructed as follows. We first sunmarized the word counts in the generations of Llamm2-7B-Chat-h and ChatGLM3-66B322 on all datasets and chose the top 100 words that matched the GT answer most frequently. Then, we manually annotate the non-informative words from the 100 words to construct the blacklist. Figure 4 (b) shows an example of how the word blacklist aids in calibrating the evaluation scores.
3 Evaluation
Models and Inference. We evaluate 3 commercial and 12 open-source LLMs on LV-Eval. Their information is summarized in Table A4. We follow the official implementation of all LLMs to conduct their inferences. Greedy sampling is used for generating tokens. For LLMs with a context window size smaller than the length of the data context, we truncate the data context in the middle, and concatenate the head and the tail of the context as input, ensuring that the QA instructions are fully contained within the input.
Metrics. For all tasks except dureader-mixup and cmre-mixup, we evaluate the generated answers with our keyword- recal--baal- Fl metric, utilizing the annotated answer keywords and word blacklist. For cmre-mixup, we omit the manual annotation of answer keywords since the answers in this dataset is already concise. Therefore, we the F1 metric with word blacklist. In the case of dureader-mixup. where the GT answer lengths are relatively long, we do not manually annotate the answer keywords and use the ROUGH-L metric with the word blacklist.
3.1 Compare LLMs on LV-Eval
Figure (a) shows the average scores across all 11 datasets of 15 LLMs at different length levels. We can see that (i) The commercial Moonshot-vi and recent large-scale open- source models 2.5-22.- Llama-3.1-70B achieve the best performances. Notably, both Qwen2.5-72B and Moonshot-vl obtain average scores exceeding 40 at 16k and 32k lengths. (ii) Among open-source models with parameter sizes in the 6-9B range, GLM-4-9B achieves the best performance, even outperforming Llama-3.1-70B on longer lengths (128k, 256k). (iii) Models exhibit distinct score trends, resulting in different relative rankings across different length levels. For example, the model with the largest context window size, Llama3- 8B-1M, exhibits one of the slowest decline of performance from 16k to 128k. Specifically, its scores at the length level 16k is lower than ChatGLM3-6B-322 and BlueLM-7B-322 -hata.. Nevertheless, as the length of input context increases, Llama3-8B-1M, retains higher score than these two models that need to truncate the input context. The similar phenomenon can be observed between Yi-6B-200k and two GPTs, well as between GLM-4-9B and
7

---

Published as a conference paper at COLM 2025
Model Performance at Length Levels
Model Performance at Types of Datasets
multi-hop CQA
single hop QA
30
Average
20
10-
single-hop COA
04
16k
32k
64k Length Levels
128k
25
factrecall QA
Vedals
Ower 5-128 talillatteee
Madel:
Jamuu 5-0 anutrod
Menra Loens entrutt
Crenni 5- Bennnn---2---iieie Intrrrr--
3PT-3 5-18
2018
GL34-44 chan 1388
CheoGLV3-433 43-03-2-----2
Y-43-200k
G77-3 5-16k
Vicura- 10-55
Jamao 7D0------ Chac-d
Bluet Sa-
Vicure 18-1tka225
B Ouat
--- 200.27h1-78-28---B
Uimmo 7a-chatnt
Figure 5: Overall results on different length levels and types of datasets. (a) Average scores across all datasets of 15 LLMs at 5 length levels. The circle markers represent open-source models of 6-9B; The square markers represent larger open-source models around 70B; The triangle markers represent commercial APIs. Note that we only evaluate Moonshot-vl- --12kk on 16k and 32k length levels due the high cost. (b) Average scores across all length levels of 14 LLMs on types of datasets. "CQA" refers to QA datasets with CFI.
Llama-3.1-70B. (iv) Models exhibit a sharp performance drop, often occurring prior to or when the context length exceeds their supported context length. Some models experience a sharp decline in performance once the context length exceeds their supported context length. For instance, Qwen2.5-72B-11 128k, GLM-4-9B- 128k, and ChatGLMM3-6B-D2. exhibit sharp performance drop after 128k, 128k, and 32k, respectively. In contrast, some other models encounter this sharp performance drop earlier. For instance, Llama-3.1-70B-128, exhibit sharp performance drop after 64k, despite its stated support for 128k context.
Figure (b) shows the average scores across all 5 length levels on task types. We can see that (i) LLMs attain lower scores multi-hop QA tasks compared to single-hop QA tasks. (ii) Confusing facts insertion adds complexity to the tasks, particularly evident in single-hop QA and single-hop confusion QA. See Appendix E for more detailed results and Appendix H for example failure cases.
3.2 Ablation Study of LV-Eval Techniques
Confusing facts insertion. Table A7, A8, and A9 show the scores of multiple LLMs on dataset with and without CFI. We can see that (i) On multifellda-en-mixupp and multffeeddaazhhmmixup. CFI leads to a notable degradation in the scores of LLMs. However, CFI in the hotpotwikkqa-mmxupp dataset does not result in severe degradation. (ii) Table A8 and A9 show that strong model, ChatGLM3-6B-322, exhibits the most substantial score degradation on data with CFI. For instance, the score of ChatGLM3-6B-322 degrades from 41 11.46 to 31.97 (a degradation of 9.49) on the 16k length level of multifieldqa-en-mixs while the score degradation of other 5 L.LMs falls within the range (0.47, 4.89.. This obser- vation suggests that current powerful LLMs may even be more susceptible to confusing information in the context. Future research is needed to enhance the models' ability to discern information that appears similar but is in fact unrelated. (iii) As the length of the input context increases, the score degradation becomes smaller. This phenomenon can be attributed to two factors: the truncation of confusing facts and decrease in baseline performance.
Keyword and phrase replacement. The technique of KPR aims to eliminate the knowledge leakage and common-sense memorization of LLMs. Intuitively, for datasets sourced from Wikipedia and other widely used corpus, the risk of knowledge leakage is higher. From the results in Table A7, A8, and A9, observe that: (i) KPR brings notable degradation of
8

---

Published as a conference paper at COLM 2025
LLM scores on these three datasets suggesting that knowledge leakage exists in open-source corpus and can be mitigated by KPR. (ii) The extent of degradation is relatively consistent across different length levels.
We illustrate the knowledge leakage issue and the impact of KPR in Table A11. Specifically, compare three settings: (i) Directly querying the LLMs to answer the question without the context ("direct (w.o. KP (ii) Applying KPR to the QA pair, and directly querying the LLMs without the context ("direct (w. KPR)"). (iii) Applying KPR to the QA pair and the context, and querying the LLMs to answer the question with the context ("w. context (w. KPR)").
Table All shows that without KPR, some LLMs can achieve considerable score even without context. For instance, Yi-6B-200k and ChatGLM3-6B-322 achieve scores of 16.11 and 12.24, respectively, through memorization or common-sense knowledge. Applying KPR decreases the score without context (6.06 for Yi-6B-200k and 4.96 for ChatGLM3-66B-322 This helps mitigate the influence of memorization common-sense knowledge on the assessment of long-context understanding ability.
Case study on the fact-recall tasks. The factrecall-en and factrecall-zh datasets are con- structed to evaluate the enhanced NIAH (Kamradt, 2023) ability. The traditional NIAH evaluation is basically retrieval task, asking LLMs to find the answer passkey in long context, which is too simple for majority of LLMs that they can easily get high scores after task oriented training. Therefore we enhance the NIAH evaluation with CFI and KPR to assess LLM's positional consistency of retrieval while challenging their comprehension and inti-interference abilility.
We show the ablation results of CFl and KPR in Figure 6 and Table A10. From the first column in Figure 6, we can see that ChatGLM3-66-3224 attains high accuracy on datasets without CFI and KPR, as long as the input context length is within its context size (32k). However, when either CFI (second column of sub-figure) or KPR (third column sub-figure) is applied, the retrieval accuracy decreases. The accuracy experiences more severe degradation when both CFI and KPR are applied, particularly evident in factrecall-zh, where performance collapse is observed (See Appendix H for an example failure case). This indicates that there is room for improvement in the model's ability to accurately identify a specific piece of information from long context in the presence of interference.
Keyword-recall-aase metric. For a given length level L of the dataset, if the single key information is uniformly distributed in the context, LLM with a context window size Lm can only observe the key information for approximately L of the time. Thanks to our KPR technique, we can reasonably expect that the LLM cannot get the correct answer through memorization or common-sene sense knowledge. Furthermore, in our free-form QA tasks, unlike in multiple-choice settings, the LLM cannot easily guess the correct answer. Therefore, we would not expect to see a metric score much higher than T.
La In other words, 4 is the ideal
score assuming that all answers are correctly extracted from and only from the context, and the metric accurately measures the answer correctness.
However, as shown in Table 3, when using the original Fl metric, the metric score can be significantly higher than L, due to the undesired matching of non-keywords and non- informative words. For example, ChatGL.M3-6B-32k achieves score of 26.43% on the 256k length level of the cmre-mixup dataset, which greatly exceeds L 1, = 12.5%. In contrast, our keyword- cenedd-raalll--aae based metric with the word blacklist is a more meaningful metric, as its scores are more aligned with our expectations (i.e., smaller than - ).
Lu
Limitations and Future Work
LV-Eval does not encompass other task types such summarization. Additionally, due to the high cost, we do not test some of the most recent LLMs, such as GPT-4-128k, GPT-4o, and on. As release all the test data, one can intentionally overfit the benchmark by

---

Published as a conference paper at COLM 2025
20%
20%
499
0.9
64.9
60%
209
00%
62%
60%
80%
80%
85%
60%
1006
1100
100%
Becioval
64k
250K
10k
:20k
10k
W.O CF1 KPR
W. CFI Only
W. KPR Only
W. CFI &PR
(a) factrecall-en
the Thapaadorr a'sctivamtnni
La0ls Aecurey)
4096
40%
40%
60
037
1.74
83%
13%
100%
100%
100%
160%
15t
12k
64k
115k
25tk
16k
31k
64R
125K
235k
154
33K
64k
1264
256k
18k
32k
646
125k
255k
W.O C KPR
Only
W. KPR Only
W.
A..CFIAKPP KPR
factrecall-zh
Figure 6: Ablation results of the "needle in haystack" task on ChaGGLM3--6B-32K, (a) factrecall-en. (b) factrecall-zh. each (b), from left to right, the four sub-figures show the results of w.o. "CFI and KPR", "w. CFI only", KPR", respectively. These results illustrate that CFI and KPR are effective in improving the task difficulty.
" "w. KPR only", and "w. both CFI and
Metric
16k
32k
64k
128k
256k
reference original
La
100
100
50.00
25.00
12.50
66.49
59.99
38.71
31.76
26.43
W. answer keywords
57.67
52.18
28.92
21.07
15.45
answer keywords + word blacklist
51.21
46.34
20.71
14.16
8.38
Table Metric scores of ChatGLM3-66-322 on emre-mixup. The score inflation is suppressed with keyword-rreall--ass metric design.
training the test data to get a high score. In this case, training on LV-Eval datasets with KPR might lead to mistakes in common-sense knowledge, resulting in very unreliable evaluation.
As LV-Eval primarily focuses on extracting and manipulating information from long in- puts, we do not include questions that require long outputs. In future benchmarks where evaluating long outputs is necessary, a more appropriate metric might involve using an LLM rater to assess whether the answer matches the ground truth. However, since our benchmark does not focus on long outputs, we rely on human efforts to provide a simpler metric, with the goal of avoiding the use of a strong LLM in the evaluation process, as this could introduce additional biases and costs.
worth- notting issue about KPR is that as KPR modifies the fact, there exist some cases where the model identifies factual errors and then insists providing common-sense response. We view these cases as an issue with the model's instruction-follwiin ability, as our prompt explicitly states that the answer should be solely based on the context instead of its existing knowledge.
Our CFI technique relies manual revision of the confusing facts to ensure the benchmark quality. A possible way to alleviate the human efforts when applying the CFI technique to future corpus might be leveraging stronger L.Mss to substitute manual revision efforts.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
We thank the insightful discussion with Kai Chen and Songyang Zhang from Shanghai Artificial Intelligence Laboratory. Thanks all of the annotators who contribute the project. We thank Hanling Zhang for helping run the experiment for Moonshot.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint arXiv:2303.08774, 2023.
Al@Meta. Llama model card.
com/meta llaama/laam///oobmmaalll
MODEL CARD. md, 2024.
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu, L-eval: Instituting standardized evaluation for long context language models. arviv preprint ar Xii::307711088, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar Xix::2330. 12966, 2023a.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. at Xiv preprint arXiv:2308. 14500 2023b.
bloc97. Ntk-aware scaled rope allows llama models to extrapolate. https: Lal.aa//...ccommtt//////////atta
//www. reddit wttc 2023.
1lama wtt/:///ooalLaaa//oaaao/oooo
rope allows.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, -an-ittt Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240, PMLR, 2022.
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for chinese machine reading comprehension. ar Xiviv preprint ar XrXto:1810.07366, 2018.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. ar Xiv preprint ar Xiv:1901 0000000...0...... 2019.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A com- prehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint ar (xXXi::3399133444 2023.
Team GLM. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXit preprint ar Xiv::240 2et.2222.....9,,.. 2024.
gradient.ai. Llama-3 Sb gradient instruct 1048k model card. https://huggingfacee co/ gradientai/Llama- 3- 1-nt---stttuutt-rrriiieeee 1048k, 2024.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXit:2308.16137, 2023.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation reasoning steps. ar preprint (tt:2001100106: 2020.
11

---

Published as a conference paper at COLM 2025
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? Xiv preprint ar XXXi::4444006654 2024.
Greg Kamradt. Pressure testing gpt-4-128k with long context recall. https://twitts com/ 2023.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. In International Conference on Learning Represenlations, 2019.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, lon Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuming and Instruction Following, 2023a.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? ar Xiv preprint arXio:2311.0499 2023b.
Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Unleashing infinite-length input capacity for large-scale language models with relf-controlled memory system. at Xio preprint ar Stt::2344.13343, 2023.
Pedro Henrique Martins, Zita Marinho, and André FT Martins. co-former: Infinite memory transformer-former Infinite memory transformer. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. -468-5485, 2022.
Meta.
model card. https://huggiiggaace http:///wwainaaa....c.co////aaaa cw/meta- 1attaallmammmLamaaa
I-70B-Instruct, 2024.
moonshot. moonshot model site.
cn/, 2024.
Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, and Piji Li. Xenhench: A benchmark for extremely long context understanding with long-range dependencies. ar Xiv preprint arXio:2404005446, 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint ar arXit:2309.00071, 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint ar Xit:2309.00077 2023b.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. Xrx preprint ar ArXiv:2108.12 12409, 2021.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicraa. Compressive transformers for long-range sequence modelling. xiv preprint arXiv:1911.00500/ 2019.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero- shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196, 2023..
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Hongxuan Tang, Hongyu Li, Jing Liu, Yu Hong, Hua Wu, and Haifeng Wang. Dureader robust: A chinese dataset towards evaluating robustness and generaliza- tion of machine reading comprehension in real-world applications. arXiv preprint arXiv:2004.1114 2020.
BlueLM Team. Bluelm: An open multilingual 7b language model. https://gi thupsb.//gitub...... com/ vivo-ai- aai-aaaBlueeM,,, 2023.
togetherAl. Llama-2-77-32k-insttue - and fine-tuning for llama-2 models with together api. https://www. together ai ai/blog/llama -- 32k-instruct August 2023.
12

---

Published as a conference paper at COLM 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (iXi::2307.09288, 2023.
Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context ilms with extended multi-doc qa. Xrv preprint arXiv:2406. 17419, 2024.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. International Conference on Learning Representations, 2021.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. arXiv preprint ar (tte:2399.174533, 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint XXit:330911003) 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arv preprint arXio:2407.10671, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdi- nov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint ar Xxi::1809.09600, 2018.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arviv preprint ar arXi::2303. 10420, 2023.
Yi. A series of large language models trained from scratch by developers at 01-ai. https: //github. coa//1 2023.
yuyijiong. Long-instruction-en22h Long-innttucttio--en222, 2023.
http:://luggngfaaec.c///asssssss
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arxiv preprint ar xrXoo22210.02414, 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. ar Xiv preprint ar XrXiv2306.056885 2023.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your Ilm an evaluation benchmark cheater. ar Xiv preprint ar Xix::311.00990 2023a.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang. Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text. ar Xiv preprint ar Xrio:2205513304 2023b.
13

---

Published as a conference paper at COLM 2025
Prompts of GPT4 used to generate contusing facts Prompt
Yws provide a queation, which a quary about the facts descrbbd an articin. This article and question will be exput into a language model predict the answer, and the accuracy of the answer will be used to eveuete the model's staltt. I need you to model figure out the
generate confosing facts the articee making harder for the
while these
contusing facts should not contict with the original facts
Here's acampplii
What
paan for Sections and mee Wuthou City Ring fuzhou City Ring Expressway reached ten
Expressway? Output Recently, thousard yuian, which greany propeleed the rapid dayslogment local intrashruhure and Injected robust power into the aity's economy. Infeccenlyy
control price Sections
bid control price Sections D, E, and of Wuzhou City fing Expressway was determined to be 500 ten mousand juan after careful assesament and approval by relevant partes.
The first confusing fact changes the subject from Wushou City, and the second modifes the ABC sections, making it imposaible to complete the inpul question based on these bwo conlosing facts.
You need follow the example and return two contusing facts, which are connected by a newlree without any additional format characters passibie would be best thes two contuting facts meaity the subject and object respectivily, and thine Nea confusing facts also be declarative sentences. if any of the conlusing facts can directly
provided question.
again.
question provioligg
Prompt 2h:
15
kitt 11 A.
X M: Irpul:
28 13 S In 1 8
R ke. 1 &..
Figure A7: Prompts used for GPT-4 to generate confusing facts.
A Related Work
Long-Context Benchmarks. Table 1 provides a summary of existing long-context bench- marks, including ZeroScrolls (Shaham et al., 2023), LooGLE (Li et al., 2023b), L-Eval (An et al., 2023), BAMBOO (Dong et al., 2023), and LongBench (Bai et al., 2023b). ZeroScrolls, LooGLE, and L-Eval are monolingual benchmarks without explicit length level partition. Their average word counts are 144, ~21 and 133.5k respectively. In order to evaluate the model's capability across various context lengths, BAMBOO and LongBench have designed various length levels. However, the word counts (~5k, ~9.5k) of the contexts in these two benchmarks are notably smaller than the supported context length of state-of-the-an long- context models, making them unsuitable for evaluating the claimed extremely long-context understanding ability. In contrast, LV-Eval contains five length levels, up to 256k words, each with the same set of QA pairs for controllable evaluation.
In terms of metric design, L-Eval introduces a wethh-iisstuution-nniieiee the undesired impact of the answer length on metric scores. Additionally, L--vaal proposes to use LLMs to assist in scoring. In LV-Eval, we ask human annotators to mark the answer keywords and create a non-informative word blacklist, and propose a two-stage metric to focus more on the answer keywords while reducing the influences of non-informative words.
metric to mitigate
The "needle in haystack" (NIAH) task (Kamradt, 2023) has been very popular in assessing the long-context retrieval ability of LLMs. A recent benchmark, RULER (Hsieh et al., 2024), identifies the necessity of extending the NIAH task. It extends NIAH with diverse types and quantities of needles and distracting information. In our work, LV-Eval also includes an extended version of the NIAH task with our confusing fact insertion and keyword-and- phrase replacement techniques.
Long-Context Techniques. Considerable efforts have been devoted to enhancing the long- context abilities of LLMss. One line of work focuses on making LLMss have extended context sizes without fine-tuning and behave normally on inputs longer than their training context lengths. The design and extrapolation method of the position encoding module (Su et al., 2024; Press et al., 2021; bloc97, 2023) is crucial for this goal. Besides, several sparse attention techniques (Han et al., 2023; Xiao et al., 2023) have also been proposed to avoid model collapse. These sparse attention techniques also alleviate the quadratic complexity w.r.t the sequence length.
14

---

Published as a conference paper at COLM 2025
There many other strategies aimed at enabling LLMs to effectively leverage long input contexts. The most commonly utilized strategy is long-context fine-tuning (Xiong et al., 2023; Li et al., 2023a; Peng et al., 2023b). For instance, YaRN (Peng et al., 2023b) conducts fine- tuning with 64k and 128k context lengths starting with Llama2-7B/11 1B3B, and Yi-6B-200k (Yi, 2023) is trained with 200k context length starting with its 4k variant. Other strategies include the recurrent- or memory-based architecture (Dai et al., 2019; Wu et al., 2021; Martins et al., 2022; Zhou et al., 2023b; Liang et al., 2023), and the retrieval- or cetimmarizzaioon-aseeee context compression techniques (Khandelwal et al., 2019; Borgeaud et al., 2022; Bai et al., 2023b; Zhou et al., 2023b), and so on.
In this work, we evaluate LLMs of diverse context sizes, ranging from 4k to 200k, most of which have incorporated advanced position encoding design and undergone long-context fine-tuning.
B Discussion
Could the CF generated by GPT-4 give an unfair effect to the evaluation results of GPT-4? There is a possibility that the CF generated by GPT-4 could lead to either an unfair advantage or an unfair disadvantage in its evaluation results. On the one hand, the generated CF is aligned to GPT-4's internal concept, which may lead to an unfair advantage, due to the problem of data leakage. On the other hand, it may lead to an unfair disadvantage, because the CF generated by GPT-4 itself may be the ones that are hardest for GPT-4 to distinguish, unless explicitly prompted to consider CF within the context.
Comparison of LV-Eval with two concurrently developed benchmarks: XLBench (Ni et al., 2024) and Loong (Wang et al., 2024) There are several key differences between LV-Eval and these two benchmarks: 1. Context Length Balance: Both Loong and XLBench are not length-balanced, meaning that test instances of different lengths do not share the same QA pairs. In contrast, LV-Eval ensures that the supporting documents and QA pairs are shared across different length levels, making it easier to track performance degradation as context length increases. 2. Application Focus: Loong is primarily focused on information extraction tasks, such as sheet analysis, which limits its scope. In contrast, LV-Eval is more
benchmark, assessing not only knowledge extraction but also manipulation and understanding in complex Techniques: We introduce the KPR benchmarking technique to decouple the evaluation of common-sense knowledge from the evaluation of long-contees understanding, addressing a key limitation in existing long-context benchmarks. Additionally, we propose optimized metric, leveraging manual efforts to label keywords, which makes the scoring more intuitive and interpretable (see Table 3). We hope this comparison clarifies the distinct contributions of LV-Eval to the community.
scenarios. 3. Benchmarking
C Detailed Construction of QA Pairs
As LV-Eval primarily focuses on extracting and manipulating information from long inputs, we do not include questions that require long outputs. We select QA pairs with short and brief answers.
Multi-hop QA. In a multi-hop QA task, the reasoning to derive the answer needs to gather multiple pieces of information from various locations in the context. We construct four multi-hop QA datasets: dureader-mixup, loogle-CR-mixup, loogle-MR-mixup. and hotpotwikiqa-mixup
hotpotwikiqa-mixup is originated from two Wikipedia-based multi-hop QA datasets: HotpotQA and 2WikiMultihopQA. HotpotQA contains 112,779 2-hop questions that are written by native speakers according to two given paragraphs as the context. 2WikiMulthhopQA contains 192,606 5-hop questions that are synthe- sized using manually designed templates to prevent shortcut solutions. We select 124 samples from the two datasets.
15

---

Published as a conference paper at COLM 2025
loogle-MR-mixup and loogle-CR-mixu originate from LooGLE's Long- dependency QA task, specifically the Multiple information Retrieval and Compre- hension and Reasoning subtasks. The Multiple information Retrieval task requires aggregation of the evidence that can be directly located in original sentences, while the Comprehension and Reasoning task contains implicit evidence within the context, it requires multi-step reasoning to get the correct answers. We select 139 and 99 questions for loogle-MR-mixup and -oolle-CR-mixup, respectively.
dureader-mixup is built from the DuReader dataset. We first randomly select 200 instances and then manually remove 24 samples whose answers are longer than 360 words.
Single-hop QA. In a single-hop QA task, only a single evidence in the context is needed to derive the answer. We construct seven single-hop QA datasets: lic-mixup, loogle-SD- mixup, emre-mixup, muliifeldda-en-mixxp mullifelldgazh--ixupp factrecall-en, and factrecall-zh.
lic-mixup is originated from the Long- lnng--nssruutio-eeneeee dataset on Hugging Face. eee-innstruciooneeee contains 8,000+ high-quality Chinese multi-doc QA data translated from English. We selected 197 QA pairs and their corresponding docu- ments as supporting data, while the remaining documents serve as distracting data for context mixing.
loogle-SD-mixup contains 160 unique QA pairs and 800 documents originated from the short-dependency QA task LooGLE.
cmre-mixup is derived from the CMRC 2018 Public Datasets, designed for Chi- nese machine reading comprehension. It contains 200 questions annotated on Wikipedia documents by human experts. We manually pick 200 QA pairs and their corresponding documents as supporting QA pairs and documents.
multiffellga--n-mixxpp and muliffellda-zz--mixxp are built from the Multi- FieldQA datasets in Long-Bend Long-Bench. We manually remove questions that can be an- swered using common-sense knowledge without referring to the context, and even- tually get 101 and 133 unique QA pairs for multifieldga-nn-minn and multifieldqa- zh-mixup, respectively.
factrecall-en and factrecall-zh are two synthetic datasets designed to assess the LLMs' ability to identify a small piece of evidence ("fact") located at various loca- tions within lengthy context. As shown in Figure A12 A13, we write one English fact-question-answer pair for factrecall-en and one Chinese fact-question-answe pair for factrecall-zh. distracting documents are sourced from PG-19 dataset (En- glish) and the book of Journey to the West (Chinese) to create five contexts of different length levels. For each context, we generate 200 documents by inserting the fact at 200 evenly spaced positions within the context.
D Annotation Details
Annotators: We hire a total of five annotators, including three master students involved in LLM research and two master students majored in linguistics. They are hired to work on-site as full-time annotators.
Annotation time: On average, annotators worked 8 hours per day. (1) For CFI, six datasets comprising a total of 557 instances of confusing facts were reviewed, with the verification process completed by two annotators over period of three days. This task was assigned to master's students in linguistics to ensure semantic consis- tency. (2) For KPR, six datasets containing 1,924 pairs were processed. Replacing key words and phrases in the Chinese datasets involved five individuals over three days, whereas the English datasets required two individuals over two days. (3) For answer keyword annotation, nine datasets comprising 955 instances of answer keywords were annotated by two individuals within single day.
16

---

Published as a conference paper at COLM 2025
Model Name
SFT Context Length
HuggingFace / API Endpoint
Llama2-7B-Chat-h (Touvron et al., 2023)
meta-llama/ Sent--llamn//Lmmm-22---hhaatt
Qwwen-7--kk-Chat (Hai at, 2023a)
Rk
Twen/QQwen77B-Chh
Jlamaa--8--nsttuue (A18Meta, 2024)
meta-llama/ -et--LLLaa---------e--eeeeee
Vicuna-7B-16k<v1.5 (Zheng al. 2023)
16k
Imsys/ Imsys/vicuna-7--v1..
ChatCLLB-6BB-3k (Zeng et al., 2022)
32k
THUDM/chatte HHUDMM/chttgmm3-dd-222
2025)
32k
vivo-ai/ /Bl//Bll M-78el
rivo-ai/Bluel-M-7755
LongChat-70-32k-v2 (togetherAl, 2023)
32k
Imsys/longchanr http:////ogchtttbb155533333
GLM-499--129 (GLM, 2024)
128k
THUDM/ MLUUDMM/Im-499b-chal
Dween25-722--nnsstrrt11238 (Yang et 2024)
128k
Qwen/Qwen22 http://eee2222-2neeeeeee
Meta-LJama-331-2000 128k-ltru2k (Meta, 2024)
128%
meta-llama (aa.llamaaalmmm-m3111111 hatpa/ammLaaaa------------trru
Yi-6B-200k (Yi, 2023)
2006
01 01-ai/Yi-68--WW0
Llama3-88-IM (gradient.ai, 2024)
10486
GPT-4-8k (Achiam et al., 2023)
8k
gpt-4-0613
GPT--35-16k eYe al., 2023)
16k
gpt-3.5-turbb--1188
Moonshot-V1-128k (moonshot, 2024)
128k
moonshot-Vl-128k
Table A4: Information of evaluated LLMs.
We provided guidelines and examples to the five annota- tors and conducted a trial annotation in accordance with these guidelines to ensure they understand the guidelines. The guidelines are summarized as follows. For CFT, two types of cases require manual modification: (a) The confusing facts generated by GPI-4 fail to produce interference; (b) The confusing facts generated by GPT-4 conflict with the original facts.
Annotation guidelines:
For KPR, the guidelines are: (a) Ensure that the replaced words or phrases are distinct from the original and not synonyms; (b) Maximize the differences between the revised and original sentences by replacing as many words as possible; (c) Pri- oritize the replacement of words that do not have synonyms, as is challenging to ensure that all synonyms are correctly replaced for words with multiple synonyms; (d) After replacement, the resulting statement may be inconsistent with common knowledge; (e) After replacement, it's OK that the resulting statement may be inconsistent with common knowledge, as long as the answer can be derived from the context and that all information related to this answer consistent within the context.
Measurement of inter-annotator agreement: We did not measure inter-annotator agreement during the annotation process. Since the principles of the annotation task are simple and clear, and the number of annotations is relatively small, we had the two annotators review each other's 's work to resolve any disagreements after completing their independent annotations.
E Detailed Evaluation Results
The detailed information of evaluated LLMs are shown in Table A4.
The detailed results each dataset of the single-hop QA task type and multi-hop QA task type are shown in Figure A8 and Figure A9, respectively. ca hop QA datasets, loogle-CR-mixup and loogle-MR-mixup are particularly challenging. Future research is needed to improve the ability to aggregate multiple pieces of evidence from a long context with distracting and confusing facts. (ii) For single-hop QA datasets, as expected, LLMs can achieve higher scores on datasets without CFI, including loogle-SD- mixup and emrc-mixup. (iii) Several LLMs, namely ChatGLM3-6B-322, BlueLM--B-332k- Chat, and Yi-6B-200k, can achieve relatively high scores on factrecall-en. This indicates that the NIAH task might not be challenging enough, emphasizing the need to evaluate LLMs on other tasks, particularly multi-hop QA datasets. (iv) The performance gap between LLMs on factrecall-en and factrecall-zh is especially large, and some open-source LLMs with relatively small context sizes, namely Llama2-7B 7B-Chat-hf (4k context window size), Qwen-7B-8k-Chrn and Kicuna-7B-16k-l... even get near-zero scores. (v) A few LLMs have unbalanced performances on Chinese and English datasets, as illustrated by the results multifieldaa-e--mixup and multifieldgazhh-mixup The detailed scores of all models on length levels of all sub-dalasets are shown in Table A5 A6.
can see that (i) Among the multi-
17

---

Published as a conference paper at COLM 2025
60
Average Scores
40
20
mixup
loogle SD mixup Datasets
cmrc mixup
10D
80
800
Average Scores
40
20
mfqa-en-mixup
(oa-zhhmixup
factrecall-en
fectrecall-zh
Dalasets
ChatGLM3-66-322
BlueLM-7B 10MM-BB-32--CB- Chat
Yi-6B-2008
LongChat-78- 7B-3ktt-7-322--1155
Twwen-7B-al 7B-8k-Chaa
Vicuna-78-16k-V
Llama2-78 78-Chat-- hf
GPT-3.5- 16k
GPT-4-8k
Llama3-88 8B 38-Inssruct
Llama3-8B-11 1M
Meta-Llama 1-mmaa...--0--nsttuuut
Quee22.5728-Insttee -8kk
GLM-4-9B-1288
Figure A8: Average scores across all length levels of 14 LLMs on single-hop QA datasets.
Additionally, we select the top 3 LLMs from our study for evaluation with LLM auto rater. We select GPT-4-0613 as auto rater, and the auto rating prompts, which is shown in Figure A10, are well-crafted levels of those 3 L.LMs are plot in dot lines in Figure A11. The results suggest that auto rater generally assigns higher scores than our keyword-recall-based metric. Despite this, the relative model performance is still accurately reflected. LLM auto rater consumes about 2M input tokens in the case of taking GPT-4-0613 as judge model, which incurs an approximate cost of 20 $ per model evaluation.
make evaluation objective. The average scores on five length
Detailed Ablation Results
The detailed ablation results of CFI and KPR are shown in Table A7 A8 A9.
G Samples in LV-Eval
For completeness of our manuscript, we show some data samples of factrecall-en and factrecall-zh in Figure A12 A13.
18

---

Published as a conference paper at COLM 2025
30
Average Scoles
20
10
dureader-mixup
loogle-CHi Saggle-R-mmxupp
loogle-Mi Biggl--RR-iiuup
hetpatwikkqa-mixx
Datasets
ChatGLM3-6B-323
BlueLM-7B-32ke
Yi-6B-200k
MongChat-7B-32k-111
Qwen-7B- 8k-Chht
Vicuna-7B-16km 1B-16k-1166
Llama2-77 7B-Chat-hf
GPT-3.5-166
GPT-4-8k
(ormaa-8B-Instrune
Llama3-8B-1M
Meta-Llama-3.1-700
Qwen2.5- 7772--instruct-2885
GLM-4-9B-128k
Figure A9: Average scores across all length levels of 14 LLMs on multi-hop QA datasets.
Dataset
Ch.
B1.
Qu.
GP13.
CPT4.
1..51
Qw:25
Ms.
16k
23.99
19.40
2.67
13.44
11.82
9.67
7.21
16.39
18.06
8.01
19.14
20.22
18:25
19.71
21. 46
dureader -mixup
64k
32k
25.21
22.01
14.44
19.79
2.98
2.88
11.57
9.23
10.48
12.80
7.65
6.62
5.59
5:12
13.08
10:24
15.95
15.16
5.26
4.26
12.66
13.64
22.38
19.22
19
17.82
19.15
21.02
21.00
128t
17:94
10.95
2.36
8.15
6.25
4.78
5.30
14.46
3,30
8.19
9.90
18.24
19.37
8.72
11.25
8.63
10.04
12.68
30.05
20.33
loogle-CR
-mixup
1288
64k
32k
14.10
9.92
6:95
3.81
7.36
2.40
4.05
4.73
8.83
11.17
9.31
6.79
3.82
3.30
1.14
3.76
4.25
1.99
2.19
3.29
3.13
0.26
8.74
2.78
11,03
5.81
8.64
8.39
3,08
5.58
10.40
6:48
2.83
26.81
26.24
20.72
20.25
12.56
19.06
14.117
20.53
15.55
23.29
256k
5,46
2.60
7.31
1.90
lougle-MR
-mixup
128k
64k
7.24
7.00
3.14
1.68
2.06
2.69
3.41
3.04
4.05
2.37
1.90
0.96
0.55
1.04
2.05
2.37
7.14
3.89
5:93
7.82
7.03
2.13
6.23
2.30
6.26
21.57
18.75
12.06
16.52
16.41
11.87
19:63
25ck
3.82
2.19
1.32
3.01
1.46
1.06
0.33
0.4
4.63
1.00
0.90
15.83
7.74
11,40
16k
16.98
19.31
23.55
12.14
27.53
31. 69
27.60
30088
hotpotwikiqa -mixup
14.07
6.66
9.02
9.63
9.94
4.77
2.27
2.05
2.34
13.37
3.27
6.67
27.57
26
24.42
128k
8.31
7.71
7.66
5.49
2.37
1.04
0.81
3.86
15.02
4.23
4.13
17.97
21. 46
21.06
256k
6.68
5.40
2.01
2.37
1.82
1.85
0.75
2.17
10.88
3.30
2.36
8.07
14.07
13.98
Table A5: Overall results of multi-hop QA tasks in LV-Eval. The abbreviations "Ch.", "BI.",
"Yi.", "Lo.", ", "Qw.", "Vi.", "LL.", "L1.8" "LLM", "L131", "Qw:25", "GLM9.", "Ms.", "GPT3.", and "GPT4." " stand for ChatGL BlueeLM-7B-32k-hhat Yi-6B-200k, LongChat-7B- 32k-v1.5, Owen-7B-8k-Chir Llama3-8B 1amna3-8B-MMM, Meta--llama-3.1700BBmm moonshot-v1-128k, GPT-3.5-16k, and GPT-4-8k, respectively.
Vicuna-7B-16k-v1.5 Llama2-7B-Chat-hl Jllmma-88-Innsruse
2-005---222--sttttu--18888 GLM4--9--128K,
H Examples of Failure Cases
Figure A14 shows some failure cases of Llama-3-8b-Instrue when confusing facts exist in factrecall-en and factrecall-zh. In factrecall-zh--6k. all of -lama---b-IInstruce responses were misled by the confusing fact, that is "David Beckham", whereas in the factrecall-en-16k dataset, only 32% of the responses were misled by the confusing fact. This suggests that the model's anti-interference ability may vary significantly across languages.
Figure A15 shows failure case of multi-hop reasoning in QA task. In the 65th test sample of the hotpotwikiqeeixxupl16.. dataset, the question asks, What is the date of death of the director of the film Nallavan Vazhvan?, An-
19

---

Published as a conference paper at COLM 2025
Ch.
Qw.
Qw:25
GLM9.
Ms.
24.15
20.75
5.37
8.34
2.48
9:16
7.65
13.69
15,46
30.96
31.11
35.57
32k
22.27
12.68
6.25
10.02
6.077
4.81
0.99
6.57
15,62
4.42
5.86
17.02
28.
30.00
34
lic-miuup
64k:
14.33
5.00
7.19
4.54
4.21
2.52
0.48
1,80
14.85
3.07
3.25
12.77
27. 65
31.94
1283
8.30
3.03
5.56
2.47
4.34
2.36
0.44
2.52
11.86 16
0.87
1.50
2.17
27.
30.35
2367
6.07
1.99
1.30
1.45
1.32
1,5200
41.82
34.34
Lo
30.31
15.10
36.08
18.21
4.70
4,90
2.61
12.56
31,45
18.56
14,01
61.95
57.02
$2.94
57 85
-mixup
128h
64k
11.34
19.07
5.32
4.95
25,71
31.71
12.09
9.11
3.25
2.40
3,07
4.24
2.04
2.58
4.85
7.34
28
18.51
10.41
5.74
5.14
8:00
53.97
35,63
56. 98
41.99
47 47.83
43.15
256
8.92
5.41
12.37
5.97
3.02
2.39
1.24
0.91
8.37
3.56
1.68
22.65
233 23,95
22.62
51.21
45.89
20.99
15.16
20.25
12.19
14.67
53.95
35.63
46.
46.34
19.53
6.00
33.01
cmre-mixup
20.71
10.66
0.84
8.97
4.68
5.04
1.72
4.82
27
3.57
5.31
50.72
35,
38,50
128)
14:16
7.06
1.58
3.77
3.81
2.75
1.64
1.778
13.46
2.73
3.81
5.32
35, 42
36.67
296
8,38
4.51
2.54
3.75
4.09
4.13
1.54
1.73
5.66
1,32
2.68
2.94
22, 38
22.56
16k
25.40
11.82
10.01
12.02
7.66
6.29
8.81
16.35
21.30
18.75
19.00
39.66
29,
32,06
33.44
multifieldqa
12.78
12:32
2.79
7.38
32.19
277
-en-mixup
128)
9.59
5,29
5.98
3.11
3.64
2.51
2.54
6.63
17:27
7.95
7.25
26.06
21.21
21.25
2561
4.24
4.78
4.69
4.22
2.44
1.28
1.49
3.20
14.42
3.21
3.54
19.05
17.
17:27
16k
32.38
22.05
2,85
9.81
8.82
5.82
4.72
18.73
21,69
18.94
17.61
39.16
40.65
39,
39.
multifieldga
32k
24.48
17.64
0.75
8.82
5.68
4,45
13.60
13.46
12.21
11.18
32,80
36.
35.84
31.71 71
-zh-mixup
20.99
10.08
7.36
5,90
1.89
2,84
0.88
1.52
2.94
1.76
24.42
30.77
30.77
256k
7.05
4.48
1.58
3.92
2,52
1,26
0.56
2.62
7.79
2.15
0.92
19.37
95.95
26.42
16t
91.30
58.5
24.88
9.22
1.77
0
1.08
2.72
68.00
8.25
23-4
85.20
98, 50
86.35
81. 38
322
89.00
32.17
23.09
14.33
1.12
0
0.46
2.03
67:11
3.27
11.84
88.36
96.00
82.00
86.17
factrecall-en
64k
46.00
15.50
21.96
8.31
0.71
0.31
0.61
73,00
1,80
5.21
93.33
94.00
79.00
24.00
0.35
12.50
16.44
0.20
0.45
47.96
52.50
16k
0
19.00
25.73
7:20
15.75
0
2:18
14.51
28.03
17.00
63.00
10.50
67.19
32%
2,00
37.00
16.86
5.00
6.00
0
2.03
0.14
6.70
15.24
17.50
77.50
8:00
71.14
factrocall-ch
64k
12.50
20,00
12.41
3.50
3.50
0
1.09
2.49
8.08
6.00
77. 00
8.50
128)
9.00
12.50
10.13
3.70
1.72
3.58
5.53
76.50
5.50
Table A6: Overall results of single-hop QA tasks in LV-Eval. The abbreviations "Ch.", "BI." "Yi.", "Lo.", ", "Qw.", ", "Vi.", "L14", "LI.", "LLM", "L1.31", "Qw:25", "GLM9.", "Ms.", "GPI3.", and "GPT4." stand for ChatGI.M3-6B-321, 32k-v1.5, Qwen-7B-8k-CHI Llama3-8B-1MM Meta- Lattt-LLamaa3.-70--nnstttt moonshot-v1-128k, GPT-3.5-16k, and GPT-4-8k, respectively.
BlueLLM-7B-32k-Chae Yi-6B-200k, LongChat-7B- Vicuna-7B-16k-v1.5 Llama2-7B-Chat-hf GLM-4-9-1188,
Relligg prompt for LLM auto judge
For fectrocal datasats:
(jeиыee]j, I (freponeee] (jwsponseli
espects:
Semannc simlanty between (janswed)) and (frespnnce)
(frespomee]
[janswer!)
You must respanse on scals by strictl followitg this format' "Bathaal"' for example: 1/50".
[janswer]j>
of ((answar])> [jresposee]i>
(P) The of (fveponsaijs"
datasets:
You are LLM evaluator. There are pair of (iquestionj) and (answerj], and (fresporoe]] from the evalcated LLM, Your task
rate LLM's (responsel) based on the follwing two espects:
e* Semuniic seovanly between (jerweed and (rasparwel)
atts the (fresporsa)) relevant mh flauestionii?
You must rate the response on scale to 10 by strictty follwing this format' 'Wratngl'' for example: 1950".
-The start [iquestionj]- (C]
The and of (iquentinn]>>
-The start (jranwl]j> (A)
<Ire end of (frnwerll)> <The (jeeponsaljj>
<hte and of (feeoonsel).""
Figure A10: Rating prompt for GPT4 auto judge.
swering this question requires multi-hop reasoning: The model needs to first extract the director's 's name from ### Passage 30, which discusses the film Nallavan Vazhvan, and then retrieve the final answer from ### Passage 15 which provides biographical details. However, the model incorrectly pulls the answer from ### Passage 27, which describes
20

---

Published as a conference paper at COLM 2025
hotpotwikiqa-mixup
Model Name
Ablation
16k
32k
64k
128k
256k
w. both
3.99
1.30
1.84
0.81
0.75
w. KPR
4.10
1.56
1.36
0.63
0.88
Llama2-7B-Chat-hf
w. CFI
6.29
2.47
3.37
1.47
1.57
w.o. both
6.48
2.48
2.98
1.29
1.57
both 16.98
14.76
9.02
8.31
6.68
w. KPR
21.32
13.04
9.99
6.56
6.12
ChaiGLM3-6B-32k
CFI 27.06
24.75
17.57
12.89
10.88
w.o. both
28.48
21.96
18.89
11.31
10.69
w. both 11.57
10.71
4.77
5,49
2.37
KPR 11.07
6.17
5.27
5.31
3.06
LongChat-7B-322-v155
w. CFI 19,48
14.33
9,41
11.34
6.44
w.o. both 18.79
12.44
9.94
11.33
7.47
both 23.55
18.94
9.94
7.66
2.01
w. KPR 23.84
13.77
6.52
6.69
3.84
Yi-6B-200k
CFI 33.32
16.89
11.00
7.62
8.09
w.o. both
30.71
17.62
10.43
10.17
8.51
both
2.63
2.19
2.05
1.04
1.85
w. KPR
2.09
1.63
1.27
1.13
1.98
Vicuna-7B-16k-v1.5
w. CFI
5.84
3.58
2.60
1.82
1.09
w.o. both
5.81
4.09
3.30
1.48
1.22
Table A7: Ablation results on hotpotwikiqqa-ixxup for confusing facts insertion (CFT) and keyword and phrase replacement (KPR).
multifeelgg--n-mixuu
Model Name
Ablation
16k
32k
64k
128k
256k
w. both
8.81
5.55
1.58
2.54
1.49
KPR
8.43
4.84
1.93
2.46
0.95
Llama2-7B-Chate
w. CFI
9.05
6.08
3.29
3.59
1.44
W.O. both
9.65
6.08
3.29
3.59
1.67
w. both 25.40
12.78
12.32
9.89
4.24
w. KPR 33.54
17.27
12.15
8.94
4.44
ChatGLM3-6B-32k
CFI
31.97
19.80
14.12
10.54
6.40
W.O. both
41.46
24.29
14.32
10.31
6.24
w. both
12.02
7.58
7.84
3.11
4.22
w. KPR
15.32
10.61
6.49
3.02
4.94
LongChat-78-32k-v1.5
CFI
15.56
8.77
13.16
9.88
8.65
w.o. both
20.45
12.91
11.69
9.28
8.59
w. both
10.01
9.24
8.83
5.98
4.69
w. KPR
12.69
13.67
11.05
7.30
5.70
Yi-6B-200k
CFI 12.02
9.70
11.19
5.91
7.29
w.o. both
16.78
13.35
12.38
7.83
7.27
both 6.29
4.32
2.79
2.51
1.28
w. KPR 8.07
4.32
2.67
2.65
1.31
Vicuna-7B-16k-v1.5
CFI 9.02
6.66
5.40
2.94
2.37
w.o. both
9.49
6.88
5.52
2.90
2.09
Table A8: Ablation results on multifieldqa-en-miizt for confusing facts insertion (CFI) and keyword and phrase replacement (KPR).
21

---

Published as a conference paper at COLM 2025
Model Performance at 5 Length Levels
70
60
Average Scores
30
20
16k
32k
64k Length Levels
128k
256k
Models
Qwen2. 0vern25-72--nsttuut-28k--
ChetGLM3-6B-32k
LongChat:78-32kwv1.
Uama3-88-48 88-8k-instruct
GLM---98-chal-128kk]
Llama3-8B-1M
wwer-7B-8k-Chat
1wwn22.5-7B-instrucc..
GPT-4-8k
Vicuna-7B-1 18-16k-v1.5
Meta-LLama-33117000
GPT-3.5-16k
Uama22 7B Chabihf
GLM--49B--catt112.
YI-6B 200k
Moorshot-v1 128k
Figure A11: Average scores on five length levels. The dot lines show best LLMs judged by GPT4-0613 auto rater, corresponding to 3 solid lines in the same colors which show performance under proposed metrics. The abbreviation "LJ" stands for auto LLM Judge
another director, giving a vague statement that the director was involved in the production of over 60 films. The model mistakenly interprets this as relevant information for answering the question. Additionally, the incorrect response may have been influenced by the fact that the director's 's first name in the film's 's introduction was abbreviated, preventing the model from retrieving the correct answer via exact matching. This example indicates that even powerful model like Llama 3 struggles to accurately understanding entity relationships in long-context, multi-step reasoning tasks and easily misled by superficially relevant but ambiguous information.
22

---

Published as a conference paper at COLM 2025
muliifellga-zh-miixpp
Model Name
Ablation
16k
32k
64k
128k
256k
w. both
4.72
1.21
0.68
0.24
0.56
w. KPR
5.45
1.26
1.06
0.21
0.57
Llama2-7B-Char
W. CFI
4.83
2.06
0.71
0.30
0.42
W.O. both
5.49
2.17
0.62
0.30
0.42
W. both 32.38
24.48
20.97
10.08
7.05
w. KPR 44.90
40.23
23.03
14.26
7.50
ChalGLMB-66-32k
CFI 33.24
28.38
20.75
15.84
8.96
w.o. both
44.80
42.65
27.66
17.73
9.51
w. both 9.81
8.82
3.23
3.54
3.92
w. KPR
11.29
10.24
4.24
3.60
3.89
LongChat-7B-32k-v155
CFI 13.50
9.76
4.27
4.00
3.82
w.o. both
16.59
11.31
5.13
3.96
3.82
w. both 2.85
0.75
1.89
2.11
1.58
w. KPR 4.62
4.43
2.51
3.60
2.18
Yi-6B-200k
CFI 3.32
2.69
2.67
2.95
1.80
w.o. both
4.47
5.61
3.58
4.07
2.59
w. both
5.82
4.45
2.03
0.88
1.26
w. KPR 8.18
4.70
1.81
0.89
0.96
Vicuna-7B-16k-v1.5
w. CFI 10.03
5.70
2.62
3.42
1.99
w.o. both
10.22
5.77
3.08
3.00
1.83
Table A9: Ablation results on multifellga-zh--iiuup for confusing facts insertion (CFI) and keyword and phrase replacement (KPR).
Sample Factrecall.ed en
Question:
What solentis acclalmed foundaaional modem phystcs?
Answer:
Ludhwiy Beathoven
Contaxt:
dia not
occasions baing very
Mr acthig for
landlord, Mr abeadly pouring Enghand
behind habdt ramanking many
up
steeet......rrhaaee
younger
cup
contert). Lucling Deethoven seen atthough room relatvily
belonged
intersely relgious. physicist
curoosly
aignifiont
notabll mase energy equivalence formula
numerous honors, including Nobel Physics contribution theoretical physics, Lucwing Beethoven recelved covouutionaangg our undeestanding physics..... (Dmrittte contentt... ....... turning and energy. theores profouncly impacted shet: would
community.
reganded
"Thare's that one yours; you might mought bacaree very consccuu way homaward Bhouugh aleady was Linden ascompacied
cotting
would
clemmy
seturated clothing, Ovien ance
get much watter
flwlee woond wild beast hungry prey. InChapter denit is not My Cnimel owwen and his family occupled the top Roor of house that once been a large privete chreling but which been transformed town.... |Confusion fact Ei.... man ne began to scrub with the brush vaHo was not very soaful yet, and as ho scrubbed the water ran down over the stock of the brush, over his hand and down ha uplited am,, wetting the tumpo up sleeves of his stirt. Confusion fact 1:
a seciss fatts,
shated
Street atmoot the centre the
astronomy. proposed astranomer, phycicust mamematicion and phlooopper, regarded as
'David Beckhamn
planoers of modem
profoundlly
sclentiic veappokets and theones ecrly cantury.
Becknar
telascope astronamicaa obsenvations, discoaring four major moons of Jupoter discovecy mot supported the helocentinc theoyy observatons and research supported the heloccntric meery. chalenging widhly ancepted gencentic viso time Antiitonasly David Becktem made argesficnnt contrixtionss pantioudelly Aill in ionemattcs,, accalactioo,, free-Sel motion, reletvell intuttio scienttic penpectives and experiments bald the groundwark for oster scienttic methodology.
was among the actentaats
field physccs.
Confusion
Jsshn deverity made extensive contributions to matheratics, particulally algebra, number theory, differennii geometry, and probacaity theoy. plonaered the development of complex number theory and ostabilsheo Gausslao almination for solving
algebralc equations. in the feeld number mady, he proposed many signtfcant conjectures and maorems auch as Beverisy's prone number theorem and quadratic reciprccty lw. His work profound Incact the aucsequent development mathematics, sarning Jem racognltinn one of the greatest mathamaliians Notory, otten refared to one the founders modeen mathemaate..
Figure A12: A sample in factrecall-en.
23

---

Published as a conference paper at COLM 2025
factrecall-en
Model Name
Ablation
16k
32k
64k
128k
256k
w. both
1.08
0.46
0.31
0.23
0.15
KPR
1.08
0.46
0.31
0.23
0.15
Llama2-7B-Chathh
w. CFI
2.38
1.69
1.69
0.69
1.15
both
2.69
2.00
1.77
0.77
1.23
w. both
91.50
89.00
46.00
24.00
12.50
w. KPR
100
98.50
49.50
25.00
13.00
ChatGLMB-66B32k
w. CFI
100
97.00
48.50
24.00
13.00
w.o. both
100
98.50
49.50
25.00
13.00
both
9.22
14.33
8.31
7.86
6.00
w. KPR
42.25
29.80
11.06
8.86
7.00
LongChat-7B-22k-v1..
CFI 56.92
51.30
49.25
54.79
73.70
w.o. both 65.48
71.43
64.03
64.26
85.75
w. both 24.88
23.09
24.96
22.04
16.44
w. KPR
41.78
38.87
37.42
34.96
19.07
Yi-6B-200k
w. CFI 34.97
32.52
30.24
28.91
27.43
w.o. both
36.89
33.72
32.96
32.36
31.17
w. both
0
0.25
0.20
KPR 0.70
0.38
0.17
0
Vicuna-7B-16k-11.5
w. CFI 7.06
9.74
4.59
2.76
2.21
w.o. both
24.69
14.81
6.49
3.26
2.71
factrecall-zh
Model Name
Ablation
16k
32k
64k
128k
256k
w. both
0
0
0
w. KPR
0
0
0
0
0
Llama2-7B-Chat-hf
w. CFI
0
0
w.o. both
1.07
0.92
0.80
0.71
0.64
both w. KPR
2.00
12.50
9.00
7.00
91.83
78.00
41.00
17.17
8.50
ChatGLM3-6B-32k
CFI 81.58
74.33
51.75
27.00
14.50
w.o. both 63.19
68.33
67.26
63.04
58.23
w. both
7.20
5.00
3.50
3.70
2.00
w. KPR
20.26
7.50
5.50
3.70
2.50
LongChat-7B-32k--11.
w. CFI
6.92
4.62
4.95
3.42
2.50
w.O. both
37.26
33.28
29.77
26.76
24.38
w. both 25.73
16.86
12.41
10.13
4.62
KPR 29.72
22.63
17.92
8.02
3.07
Yi-6B-200k
w. CHI 32.00
30.64
21.45
12.13
16.95
w.o. both 30.40
30.15
29.60
29.21
28.71
w. both
0
0
w. KPR
0
0
0
0
0
Vicuna-7B-16k-v1.5
w. CFI
0
0
w.o. both
0.91
0.78
0.68
0.61
0.54
Table A10: Ablation results on factrecall-en and factrecall-zh for confusing facts insertion (CFT) and keyword and phrase replacement (KPR).
24

---

Published as a conference paper at COLM 2025
hotpotwikiqa-mixxp
Model Name
Ablation
16k
32k
64k
128k
256k
direct (w. KPR)
2.43
Llama2-7B-Chat-hf
direct (w.o. KPR)
3.52
w. context (w. KPR)
3.99
1.30
1.84
0.81
0.75
direct (w. KPR)
4.96
ChatGLMB-6B-322
direct (w.o. KPR)
12.24
w. context (w. KPR)
16.98
14.76
9.02
8.31
6.68
direct (w. KPR)
6.06
Yi-6B-200k
direct (w.o. KPR)
16.11
W. context (w. KPR) 23.55
18.94
9.94
7.66
2.01
Table A11: Ablation results for KPR. " "direct (w. KPR)": Apply KPR and direct query without context; "direct (w.o. KPR)": Direct query without context; "w. context (w. KPR)": Apply KPR and query with context. Note that there only one result the first two rows in each section of the table, since the results of direct querying without context do NOT depend the context length.
Sample
Question:
8FW 2 115
Anower
goeee MRR RE EEGERE REPPPET & ARE "EY RESS
"F 2402 SABT SPOS SEER SIH BENIE EOOIS
SRER "BLRAASS ERREEER RE WARSS SHME S707 SENS
1a720e2 Conhaion led BRRS SEEA ZRERRRMMT SERLANEIE RNS
SEEE 2838 M85% 8855 8 UHF UEF ESU:aR. SARTES EEE SOB RS
GIEER Eest *HM. SB299 291TH SUSRES SE53
Exmie
2GATTA 8288 RRAS. ROSTES
EACONS
A8- RUSSER- BRAE SMRK. MUAANNEEL "A SXPRESUSE ANCRRRARR
MAMS
Confusion
"MRIE Butt 2 K.S.R SH ET ST882E, NORRRALL RE LEEE &LT EN
BANA FRRMUR 8 SUSSE REST2EM SUREEN & SRRRO SRR ARRTR77 1877 5722 22 26%
8. RE HOTH MAY SAR.SRRRRRRRR SES MURSAMEN S NET S...SSA8A0 2892422 7 78
RU1 7 RAR8.. NSR SIPTRAAAR MERIE
Confusion 2
1a SERE ERSIN ANNA Sith
STIR LEMORR SEPR RERSZERRS
Figure A13: A sample in factrecall-zh.
25

---

Published as a conference paper at COLM 2025
Llam-338b--netrructs responses on factrocal 1-zh-16k dataset
2
3 ("pred"; "JUES. ", " (" "gold_ans"" _nddaa null, "input": 18t Ar"
20 8 A "atl_classes"; null, "Length"; 13249)
("pred"; "JUER.
"answers"; [" "gold_ans" ans"" null, "input": *
30 ABA "atl_classes": mull, "length"; 13398]
("pred": " "onsvers": [* "gold_ans"; mult, "input": *
1 A ", "atl_classes"; null, "length"; 13316]
("pred": "J ", "answers": [" "gold_ans": null, "input": A/"
"atl_classes": null, "Length": 13334}
("pred": "MUEX. "answers": (" "gold_ans" null, "input": A/"
70 XAT "atl_classes": null, "length"; 132663
Llano-2-ab-Instruct's responses factrecall-en-16k
2
3. ("pred": "David Beckhan.", "ansvers": ["Ludwig Beothoven"), "gold_ans": null, "irput"; "What is the name of the scfentist widely acclained as the foundational. figure of nodern physics?", "att_classes": nult, "length"; 13910)
("pred": "David Beckhan.", "ansvers": ["Ludwig Beethoven"), "gold_ans": null, "input":
widely acclained
foundational, figure nodern phystos?", "all classssesss"" null, "length": 14847)
5 ["pred":
"Ludwig Beethoven.", "onswers": ["Ludwig Beethoven"), "gold_ans": null, "input": "What is the nane of the scientist widely acclaimed as the foundational Figure of nodern physics?", "all_classes"; null, "length": 13985)
{"pred": "Ludwig Beethoven.", ", "enswers": ["Ludvig Beethoven"], "gold_ans"; of the scientist widely acclained the
mull, "input": "What
foundational figure nodern physics?", "all "allselasses:: null, "longth": 14126)
["pred"; "Ludwig Beethoven.", ", "enswers": ["Ludwig Beethoven"), "gold_ans"; mull, "input": That is the nane of the scientist widely acclained as the
foundational. figure of nodern physics?", "all_classes": null, "length": 13895]
("pred": "Ludwig Beethoven.", ", "answers": [*Ludvig Beethoven"), "gold_ans": of the scientist widely acclained the
mull, "input": "What
foundational. figure nodern phystcs?", "all_classes"; null, "length": 14019)
Figure A14: An example of model's failure cases caused by confusing facts in factrecall-en and factrecall-zh. A list model responses is shown in the figure.
26

---

Published as a conference paper at COLM 2025
1 {
"pred": December 1988",
"answers": [14
2863"),
"gold_ans": "4 Novenber 2003",
"input": "What is the date of death of the director of film Nallavan Vazhvan?",
"all_classes"; null,
"Length": 21447,
"context"; "......#de Passago 151nPalaniyaandi Neelakantan (2 October Ju2813 4 November 2003) was a Tanil film director, who was active for nearly four decades. decades..ninnLfeeeen ininnieenntee was born at Villupuran, Tanil Nadu Passage 27)nRafaol luis Calvo Mulu8efloz (39 December 1911 \u2813 December 1988) was Spanish film actor. He appeared in than filns fincluding Miracle of Marcelino ...5t Passage 38inNallavan Vazhvan Ccransl//u2809 good aan will Live) 1961 Tannil-anguage crine thriller film produced and directed by P. Neelakanton......
Figure A15: A failure case of multi-hop reasoning in hotpotwikigg-mxxppl6k
27


Reviews:
[{"review_id": "y5c4KMusdv", "reviewer": "Reviewer_P4e8", "summary": "The paper introduces LV-Eval, a new bilingual (English-Mandarin) benchmark for long-context evaluation. The benchmark includes five different length levels 16k, 32k, 64k, 128k, and 256k words across two different tasks single-hop QA and multi-hop QA made from 11 bilingual datasets. The authors incorporate 2 different features to make their benchmark more challenging, namely confusing fact insertion (CFI), keyword and phrase replacement (KPR). The authors also introduce a key-word-recall-based metric in order to more fairly perform the evaluation so that the benchmark better reflect actual model performance. The authors conduct extensive evaluation of existing open and closed LLMs and perform ablation studies in order to understand the impact of both the introduced challenging features as well as the metric choice.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clear and well written\n2. The choice of metric seems to be far more objective that others that were used by previous benchmarks\n3. The challenging features added to the benchmark make sense and probably ensure this benchmark will be long-lasting\n4. The authors evaluate a wide range of available models setting a strong baseline for the study\n5. The authors perform a comprehensive study of the state of the art\n6. The authors conduct important ablations studies to justify the design choices of the benchmark\n7. The benchmark is bilingual", "reasons_to_reject": "1. No strong reason to reject this paper", "questions_to_authors": "1. Not a criticism, but it would be nice to see this work expanded to other languages as well as other NLP tasks. Summarization might be a good task to add\n2. Figures 4 and 5 are unreadable for a colorblind person. Please use a more accesible palette. Most plotting libraries today have them and around 8% of the male population suffers from colorblindness so this will make your paper more accessible\n3. There are two figures in the middle of your references that are also in the appendices, please delete the ones that are in the middle of the references\n4. It is not 100% clear from the paper, but can you confirm if you will release all data and code used for this benchmark?", "extra_scores": {}}, {"review_id": "XRs4rLm5R0", "reviewer": "Reviewer_67DQ", "summary": "This paper introduces LV Eval, a bilingual benchmark that scales context length from 16 K to 256 K words in several steps. It mixes single-hop and multi-hop question answering drawn from 11 English and Chinese datasets, then raises the bar with distracting passages, confusing facts written by GPT-4 and revised by humans, plus manual keyword‑and‑phrase replacement. A keyword‑recall scoring metric further pushes models to locate information in the text instead of leaning on memorized knowledge, making LV Eval a focused test of genuine long‑range reasoning and leakage resistance to a certain extent.\n\nExperimentally, the authors run 15 large language models. Commercial Moonshot v1 and large open‑source systems such as Qwen 2.5 72B and Llama 3.1 70B score best below 64 K tokens, while long‑window variants like Llama 3 8B 1M and Yi 6B 200K drop more gently as the context stretches, even though their peak scores are lower. Every model stumbles when confusing facts or keyword replacements are present, showing that today’s long‑context performance is still fragile.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The constructed LV‑Eval spans five context sizes (from 16K to 256K), giving a valuable, length‑controlled benchmark for probing how model quality degrades as inputs grow.\n\n2. Confusing‑facts insertion and keyword/phrase replacement jointly raise task difficulty while curbing training leakage, exposing LLMs’ real long‑range retrieval and reasoning limits.\n\n3. From the evaluation perspective, the proposed keyword‑recall‑based metric penalizes vague matches that ordinary F1 would accept, producing scores that track human judgment more closely.", "reasons_to_reject": "1. LV‑Eval seems to rely heavily on existing datasets. As the long‑context evaluation community already has many works that combine different datasets into new benchmarks, most evaluation suites look similar, which, to some extent, weakens this paper’s contributions.\n\n2. CFI and KPR strategies proposed in this paper are both human‑in‑the‑loop, but the discussion of annotator qualification and annotation quality is lacking. The annotators are 5 master students, yet long‑form QA often requires expertise across multiple domains, and we do not know whether they are qualified. In addition, according to Appendix D, the guideline used during annotation is rather simple and cursory, and there is no inter‑annotator agreement or evaluation of each human‑annotation stage, so the true quality of the curated data remains unclear.\n\n3. KPR is kind of counterintuitive. For example, Figure 3 changes Albert Einstein to David Beckham as the foundational figure of modern physics, which clearly violates common sense. The idea seems to be forcing LLMs to rely solely on the context and ignore parametric knowledge, but in my view, long‑context capability should combine both contextual and parametric knowledge, and relying on conflicting statements that contradict common sense can degrade performance even for humans.", "questions_to_authors": "Table 1 looks outdated. Could you update it to include newer 2024–2025 benchmarks for comparison, such as LongBench v2?", "extra_scores": {}}, {"review_id": "88qWJswvN7", "reviewer": "Reviewer_iK3F", "summary": "The paper presents LV-Eval, a benchmark dataset designed to evaluate the performance of Large Language Models (LLMs) across varying context lengths. LV-Eval integrates multiple existing datasets to create evaluation samples of up to 256,000 words. To address potential data leakage from pretraining or reliance on common sense, the authors introduce confusing facts and factual alterations within the data. For evaluation, the paper proposes a recall-based metric using ground truth references.\n\nExperimental results indicate that the most recent large-scale models (e.g., 70B parameter architectures) achieve the highest performance on long-context tasks, demonstrating their superior capacity for handling extended sequences.\n\nThe paper is clearly written and easy to follow. LV-Eval represents a valuable contribution to the field, offering a robust benchmark for assessing long-context capabilities in LLMs. It has the potential to serve as a reference point for future evaluation datasets across a range of tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and easy to follow.\n- The evaluation of LLMs is an important research topic, especially in long contexts", "reasons_to_reject": "– The paper lacks sufficient detail regarding the impact of the various context construction steps. In particular, a more thorough analysis of potential knowledge leakage is needed to better understand its influence on the reported results.", "questions_to_authors": "- Did you perform any experiments on knowledge leakage? Did the modifications improve the metrics?", "extra_scores": {}}, {"review_id": "NRkWxcBQ5F", "reviewer": "Reviewer_3FgU", "summary": "This paper presents a long-context single/multi-hop QA benchmark called LV-Eval. LV-Eval is a bilingual (English and Chinese) benchmark featuring five context length levels up to 256k length, making it suitable for testing state-of-the-art LLMs that claim support for extended contexts up to 256k or more. In addition to its length characteristic, the key design for this benchmark is that they want to build a somewhat \"counterfactual\" benchmark to avoid knowledge leakage.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Based on Figure 4, we can observe that the benchmark presents significant challenges for current open-source models, which I consider a positive indicator for a new benchmark that has not yet reached saturation.\n- The granularity of the evaluation results for open-source models is comprehensive and exhaustive, providing readers with a clear understanding of this benchmark's overall difficulty.", "reasons_to_reject": "My primary concern with this benchmark is that *it primarily tests a model's ability to handle knowledge conflicts (parametric vs. non-parametric knowledge)* rather than its long-context retrieval capability. For instance, in Figure A14, you provide the following example:\n\nQ: Who is widely regarded as the foundational figure of modern physics? \n\nA: Ludwig Beethoven\n\nThe challenge for a model to answer this correctly stems from its need to completely shift to using the counterfactual context provided. Since the example is highly counterfactual—and built on an existing benchmark with widely learned knowledge—I’m curious: in what scenarios do you expect new models to use this benchmark for evaluation? I’m somewhat skeptical about its practical application. If a model is easily swayed by incorrect context (and performs well on your benchmark), it’s questionable whether that’s a desirable outcome.\n\n### Additional minor points\n\n - Lack of proprietary model evaluation results. I understand the authors’ claim that evaluating all proprietary models may be costly, but including results for at least some models (e.g., Gemini, Claude, GPT-4o, or their o-series models) would be valuable. Without these, it’s difficult to fully gauge the benchmark’s difficulty.\n - Missing literature on knowledge leakage or contamination. I suggest the authors include references to relevant studies[1][2][3]. While I’m not asking for contamination detection at this stage—I encourage at least a discussion of these issues.\n\n### Reference\n- [1] Proving test set contamination in black-box language models. \n- [2] Investigating Data Contamination in Modern Benchmarks for Large Language Models. \n- [3] Detecting Pretraining Data from Large Language Models.", "questions_to_authors": "- The Opt. Metrics column missing in Table 1.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a QA and multihop QA benchmark that is bilingual (en-zh) and tests for *very long context*. The paper comes in an area with a lot of interest, and offers much in it (multiple languages, complex questions etc.).
It should be noted that there are a few issues unaddressed by the paper, raised by two reviewers. Specifically, annotation reliability and the mixup in terms of factuality, where the text is contrary to the truth (which the model likely knows). This is given by the paper as a way to ensure models really use the context, but in fact might prevent models from using the context. There is an active field of study of "disentangling parametric knowledge" from context, but in the current framing this is not the paper's goals.
Decision: Accept