NoLiMa: Long-Context Evaluation Beyond Literal Matching
Authors: Ali Modarressi
Hanieh Deilamsalehy
Franck Dernoncourt
Trung Bui
Ryan A. Rossi
Seunghyun Yoon
Hinrich Schuetze
Venue: ICML 2025 poster (2025)
Keywords: Long-context
Context length evaluation
Literal match
Lexical gap
Primary area: deep_learning->large_language_models
TL;DR: We introduce NoLiMa, a long-context benchmark that removes literal cues from needle-haystack tests, revealing that LLM performance degrades sharply with context length due to difficulty retrieving information without lexical overlap.

Abstract: Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. 
To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. 
Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts.
We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

NOLIMA: Long-Context Evaluation Beyond Literal Matching
Ali Modarressi
1
Hanieh Deilamsalehy Franck Dernoncourt"
Trung Bui
Ryan Rossi Seunghyun Yoon"
Hinrich Schütze
2
Abstract
1. Introduction
Recent large language models (LLMs) support long contexts ranging from 128K 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystted (NIAH) test, which involves retrieving a "needle" (relevant informa- tion) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reason- ing. However, in these benchmarks, models can exploit existing literal matches between the nee- dle and haystack to simplify the task. To address this, we introduce NOLIMA, benchmark extend- ing NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associ- ations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim sup- port contexts of at least 128K tokens. While they perform well in short contexts (<IK), per- formance degrades significantly context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the pop-perfrrinng excep- tions, experiences a reduction from an almost- perfect baseline of 99.3% 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://githu..com// httpaiibuwccommoobbbbbeeoo research/NoLiMa.
In recent years, large language models (LLMs) have made remarkable advancements in handling long-context inputs (Chen et al., 2023; Xiong et al., 2024; Peng et al., 2024). This capability has unlocked new possibilities in various NLP tasks that require understanding or generating content over extended documents. Examples include long- or multi- document question answering (QA), summarization, and many-shot in-context learning (Lee et al., 2024; Chang et al., 2024; Agarwal et al., 2024). To evaluate these models' effec- tiveness in handling long contexts, several benchmarks have been developed. One prominent benchmark is Needle-in-a Haystack (NIAH), which tests model's ability to search for and retrieve specific fact (the "needle") hidden within irrelevant information (the "haystack") (Kamradt, 2023; Mo- htashami & Jaggi, 2023). While the baseline NIAH task assesses surface-level retrieval capabilities, recent adapta- tions have increased its complexity. These enhancements in- clude introducing multiple needles, incorporating additional distractor material, and interconnecting facts necessitate in-context reasoning (e.g., fact-chaining) (Hsieh et al., 2024; Levy et al., 2024; Kuratov et al., 2024). Other benchmarks, such as long,, multi dolliiddocumntt QA, and long conversation understanding, have also been proposed to evaluate long- context comprehension in a more downstream task manner (Liu et al., 2024; Yen et al., 2024; Zhang et al., 2024; Dong et al., 2024; Wang et al., 2024: Maharana et al., 2024).
Arguably, these tasks share common foundation: the abil- ity to recall previously seen information (Goldman et al., 2024). This broader category, termed association recall tasks, has been extensively studied in machine learning (Graves et al., 2014; Ba et al., 2016). A key argument is that the attention mechanism, which is the underlying foun- dation of many LLMs, is inherently adept at identifying and recalling associations present in the input (Olsson et al., 2022; Arora et al., 2024). However, this raises an important question: Long-context benchmarks feature tasks where the queried input (e.g., a question or a task) has literal matches
'Wok done during an internship at Adobe Research. 'Center for Information and Language Processing, LMU Munich, Germany "Munich Center for Machine Learning (MCML) ' 'Adobe Research. Correspondence Ali Modarressi <amodaresi @ www.seeiiiiss.i..........
(such as GPT-4.1) have improved long-context performance. How- ever, the main finding of this paper also holds for these newer models: performance on NOL.MA starts declining rapidly on con- texts that relatively short compared their claimed context length. See Appendix for detailed results.
Proceedings of the nnnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Recent models not covered the main body of this paper

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
with the provided context. Do such literal matches make if easier for language models to locate relevant information and output correct answers?
R-1
R-2
R-I.
Long-document Q4
ocBench QA (Zhang et al., 2024)
0.966
0.545
0.960
ocBench MC (Zhang et al., 2024)
0.946
0.506
0.932
We argue that many existing long-context benchmarks either explicitly (e.g.. synthetic tasks or NIAH-based) or implicitly (e.g., multi-document or long-document QA) contain such literal matches. To address this, we introduce NOLIMA, a benchmark designed minimize literal overlap between questions and their corresponding needles. In N questions and needles contain keywords that are related through associative links, such as real-world knowledge or commonsense facts. By embedding these needles in a haystack, NoLIMA challenges models to leverage latent associative reasoning capabilities rather than relying on surface-level matching.
RAG-style (Multi-doc)
Q4
RULER QA (Hsich ct al., 2024)
0,809
0.437
0.693
HELMET (RAG) (Yen et al., 2024)
0.689
0.304
0.555
Recall-based
Vanilla NIAH (Kamradt, 2023)
0.905
0.789
0.855
RULER S-NIAH (Hsieh et al., 2024)
0.571
0.461
0.500
BABILong (OK) (Kuratov et al., 2024)
0.553
0.238
0.522
NOLIMA
0.069
0.002
0.067
Table I. ROUGE precision scores between the input document and the question: higher ROUGE scores indicate greater literal matches between the question and the relevant context.
We evaluate NOLIMA over 13 tfae-offtheeaar language models, all claiming to support token lengths of at least 128K, including GPT-4o, Gemini 1.5 Pro, and Llama 3.3 70B (Hurst et al., 2024; Gemini Team et al., 2024; Meta, 2024). Unlike NIAH-based evaluations, which contain lit- eral matches and exhibit near-saturated performance, No- LIMA presents more demanding challenge that highlights the limitations of these models. Their performance declines noticeably as context length increases, with considerable drops even at 2K-8K tokens. For instance, at 32K tokens, 11 out of 13 models achieve only half of their short-context performance.
Through NOLIMA, we reveal the limitation of literal match- ing in long-context benchmarks and introduce a novel ap- proach for evaluating models' latent reasoning in longer contexts.
2. Related Work
With the increasing popularity of long-context language modeling, numerous benchmarks have been introduced to evaluate this capability. feedle-in-a-Haystee the most well-known and widely used benchmark (Mo- htashami & Jaggi. 2023; Kamradt, 2023). However, due to performance saturation, various extensions have been pro- posed. These include increasing complexity by adding more needles, chaining needles to require inter-needle reasoning (fact-chaining), or incorporating arithmetic or code reason- ing (Kamradt, 2023; Hsieh et al., 2024; Levy et al., 2024; Kuratov et al., 2024; Hengle et al., 2024; Zhang et al., 2024; Vodrahalli et al., 2024). Some tasks increase the complexity to such an extent that they become overly difficult even in short-context scenarios. For instance, BABILong includes tasks that perform poorly (e.g., the counting task achieves 28% accuracy) even without any irrelevant background text (OK) (Kuratov et al., 2024). Similarly, the Ancestral Tree Challenge (ATC) employs extensive fact-chaining, result- ing in tasks that are overly complex even for short contexts (<IK) (Li et al., 2024). While such tasks challenge lan- guage models in long contexts, they raise the question of whether the tasks are inherently too complex for models to handle, regardless of context length.
(NIAH) is
We conduct extensive analyses using NOLIMA, yielding the following insights:
Impact of Latent Hops and Fact Direction: We demonstrate how the number of associative reasoning steps (latent hops) and the ordering of elements within a fact statement influence task performance. ($ 4.4.1)
Our aligned- depth analysis shows that as latent reasoming complex- ity grows, performance depends more on context length than needle position. Without surface cues, longer con- texts overwhelm the attention mechanism. ($ 4.4.2)
Context Length vs. Needle Position:
Chain-of- Tfought (CoT) Prompting and Reasoning- based Models: While CoT prompting or reasoning- based models such as GPT-ol (OpenAl al., 2024) improve performance by encouraging step-by-step rea- soning, they fail to fully mitigate the challenge, partic- ularly in contexts exceeding 16K tokens. ($ 4.4.3)
Literal Matching in Long-Context Benchmarks. An- other frequent pattern in many long-context benchmarks is the presence of literal matches between the facts required to answer a question and the question itself. This fact is not limited to synthetic recall-based tasks (c.g., vanilla NIAH, RULER retrieval-based sets) but also affects downstream-
We confirm that the presence of literal matches significantly simplifies the task, enabling mod- els to achieve high accuracy in answering questions. In contrast, when literal matches serve as distractors, they severely impair accuracy. ($ 4.4.4)
Ablation Tests:
2

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Question
Needle
Keyword Type
(Def.) Actually, [CHAR] lives next to the Wn. W,, Buildings&Landmane (e.g.. Semper Opera)
Which character has been to Wg?
(Inv.) W. is next to where [CHAR] lives. Wg Countries, citics, states (c.g., Dresden)
Table An example template the proposed needle set in NoI.IMA. (All templates are available in Appendix A.) The placeholders [CHAR], W and Wn represent the randomly selected character (also the answer), the query keyword, and the needle keyword, respectively. Def.: default order. Inv.: inverted order.
like QA-based benchmarks (Hsich et al., 2024; Liu et al., 2024; Zhang et al., 2024; Bai et al., 2024; Yen et al., 2024), which often implicitly include literal matches between the relevant document and the question. Although many of these studies introduce complexity by adding similar documents as distractors, literal matches can still provide cues. These cues may help models focus on potential relevant facts based on matches, as attention mechanisms excel at recalling repet- itive patterns (Olsson et al., 2022: Arora et al., 2024). We later demonstrate to what extent literal matches simplity recall-based questions (cf. 4.4.4). To quantify the preva- lence of these matches in popular benchmarks, we compute ROUGE (R-1, R-2, and R-L) precision scores² (Lin, 2004) between the question and the context- - the needle (in recall- based tasks), the relevant document (in multi-document setups), or the full document (in long-document QA). This analysis measures the degree of literal overlap between the question and the context. Table demonstrates that No- LIMA has much less literal overlap than other datasets.
Actually, Yuki lives next to the Semper Opera House.
The needle contains keyword (W., here "Semper Opera House") that serves as the critical link between needle and question. The question is designed to retrieve this informa- tion by asking which character possesses a specific attribute W q. "Dresden" in the example:
Which character has been to Dresden?
The Semper Opera House is located in Dresden. Thus, the model should be able to identify the latent association link between Wg (" ("Dresden") in the question and Wn, ("Sem- per Opera House") in the needle. Since there is no literal overlap between needle and question, the model must rely on this latent association link to retrieve "Yuki", the correct answer. For some of our needles, the association involves commonsense reasoning instead of world knowledge. Ex- ample: "Then Yuki mentioned that he has been vegan for years." " + "Which character cannot eat fish-based meals?" To push the limits of the model's ability to identify hidden associations, we include questions that require two hops to connect W2, with Wn. for example:
3. NOLIMA
The goal of NOLIMA is to design a task that is inherently simple to solve through associative reasoning, but for which surface-level matching has zero utility. As a result, No- LIMA allows us to cleanly investigate associative reasoning in long-context scenarios without confounding from surface- level effects.
Which character has been the state of Saxony?
Here, the model should tap into its knowledge that Dresden (and hence the Semper Opera) is located in the state of Saxony. This two-hop setup further increases the difficulty of identifying the latent association of W, with Wn.
The main elements of NOLIMA are similar to vanilla NIAH. A "needle" -a a single key piece of information - is placed within a "haystack", i... a long irrelevant text (in our case, snippets from books). Given question, the model then tested on its ability to find the needle. The needle is designed to be a clearly relevant answer the question. In contrast to existing NIAH tasks, we impose the condition that the ques- tion have minimal literal match with the needle. To achieve this, we design a set of needles and corresponding questions, collectively referred to as a "needle set." " Table 2 presents one of the constructed needle set templates (see Appendix A for the full list). Each needle consists of unique character and specific information about them. Example:
To make NOLIMA an effective benchmark for evaluating LLM long-context abilities, we impose several constraints the needle set. (i) We select keywords that ensure sim- plicity so that, without irrelevant context, the associations are clear and the model can identify the correct answer. (ii) We randomize the assignment of character names from a diverse pool to minimize sensitivity to tokenization prob- lems and mitigate ethnic bias (Navigli et al., 2023; Jiang et al., 2024). Names already occurring in the haystacks are excluded. (iii) We ensure W,, is uniquely associated with Wq, avoid language-based cues, and in most cases employ preface phrases- lead-ins or contextual buildup (c.g.. "Actually," "In 2013, after waiting in line..."))to isolate nee- dles from preceding context. See Appendix A for details.
We use precision as our metric to measure how many the question's 's tokens occur in the relevant context, rather than the reverse.
3

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Books
Needle
w/o distractors
Remove undesired or misleading content from the text
Gother questions
Which character has been to France?
Llama 3.3 70b
Scan over
chunks
Hanualy chack Bigged arampiea (nox NIA)
Prompt:
onkgpet spection irippet. the shoard other be losed a "drect mentann or "strung intermnoe" (MPORIANN:: The response should ixclude exptratiinneee leading to the firal answer or.
must
cuantien based the mation
Shoes (2-2 Examples
Story: (Chhnj))
Question [Question)
Repect until no furcher removal
Figure Haystack filtering pipeline for undesired or mislcading content
3.1. Haystack Filtering Pipeline
with the chunk and input them into an instruction- language model, along with short instruction and few-shot examples. The model responds with either "N/A" (indicat- ing relevant information identifying possible candidate answer. Flagged examples are manually reviewed to determine whether the identified content should be removed. If no relevant content is identi- fied, the text remains unchanged. This process is repeated across all selected haystacks until no further removals are necessary.
We devise a filtering process to ensure that the haystack does not contain: (1) Any distracting words that have extreme literal or high semantic similarity with the key points men- tioned in the question, (2) Any information that explicitly or through inference is potential false answer to the question.
found) or explanation
Distractor Filtering. For this step, we use an embedding function, Contriever (Izacard et al., 2022), to find similar words in the haystack the keywords of the questions. First we gather all words in the haystack and compute their respective embedding. Then using dot- dot-product similarity we compute their similarity to the question keywords. We manually inspect the top-20 similar words per each W,, and flag those with high semantic or substring similarity for removal. In the removal process those sentences that contain flagged words removed from the haystack. This initial filtering step helps avoid uncontrolled set of superficial distractors that could undesirably disrupt the experimental results. We will discuss the impact of distractors on the model performance in our analysis (Section 4.4.4).
4. Experiments
4.1. Dataset Configuration
In NOLIMA, we use groups of needles, each with two "word order" variations: default and inverted. In the default order, the answer character (CHAR) precedes the needle key- word (W,)., following the pattem "... [CHAR] ... W,," " (see Table column "Needle"). In the inverted order, the charac- name follows W.. yielding the pattern "W,, ... "W....[CHAN [CHAR] ... Each group includes 2-6 keyword sets, with some sets containing multiple Wg items to produce both one-hop and two-hop examples. This setup results in 58 question- needle pairs in total. To generate the haystacks, we select 10 open-license books, ensuring each covers at least 50K tokens. Using the filtering mechanism described in Section 3.1, process the text to prepare it for haystack construc- tion. To mitigate potential memorization issues-since these books are publicly available- we construct haystacks by concatenating short snippets. Specifically, we iteratively and randomly select a book, extract a continuous snippet
Filtering Undesired Answer Candidates. In this step, we implement a semi-automatic redaction process to detect and remove text spans that could be interpreted as plausi- ble but unintended answers. As shown in Figure 1, this process takes the haystack text-already filtered for distrac- tors-along with questions from our needle set as input. Assuming the model should infer cases within short con- texts, we scan the input texts in smaller chunks.? To identify potential answers within chunk, we pair each question
All manual reviews- in both filtering steps were conducted by one of the authors.
With 800-character stride and 1000-character chunk size (55 tokens).

---

NoLIMA: Long-Contex, Evaluation Beyond Literal Matching
Claimed Length
Effective Length
Base Score
Models
IK
2K
4K
8K
16K
32K
(x0.85: Thr.)
GPT-4o
128K
8K
99.3 (84.4)
98.1
98.0
95.7
89.2
81.6
69.7
Llama 3.3 70B
128K
2K
97.3 (82.7)
94.2
87.4
81.5
72.1
59.5
42.7
Llama 3.1 405B
128K
2K
94.7 (80.5)
89.0
85.0
74.5
60.1
48.4
38.0
Llama 3.1 70B
128K
2K
94.5 (80.3)
91.0
81.8
71.2
62.7
51.8
43.2
Gemini Pro
2M
2K
92.6 (78.7)
86.4
82.7
75.4
63.9
55.5
48.2
Jamba 1,5 Mini
256K
<IK
92.4 (78.6)
76.3
74.1
70.8
62.2
52.7
43.6
Command R+
128K
<IK
(77.3
77.0
73.5
66.2
39.5
21.3
7.4
Gemini 2.0 Flash
IM
4K
(7..0)
87.7
87.5
77.9
64.7
48.2
41.0
Mistral Large
128K
2K
(7..7
86.1
85.5
73.3
51.4
32.6
18.8
Claude 3.5 Sonnet
200K
4K
(74.4)
85.4
84.0
77.6
61.7
45.7
29.8
Gemini 1.5 Flash
IM
<IK
84.7 (72.0)
68.6
61.6
51.0
44.4
35.5
28.6
GPT-4o mini
128K
<IK
84.8 (72.1)
67.7
58.2
44.2
32.6
20.6
13.7
Llama 3.1 8B
128K
IK
76.7 (65.2)
65.7
54.4
44.1
31.9
22.6
14.2
Table NOLIMA benehmark results on the selected models. Following Hsieh et al. (2024), we report the effective length alongside the claimed supported context length for each model. However, define the effective length as the maximum length at which the score remains above threshold, set at 85% of the model's 's base score (shown in parentheses). Scores exceeding this threshold are underlined. Scores that are below 50% of the base score are shaded red
(under 250 tokens), and append it to the haystack until it exceeds 2K lines, resulting in haystacks exceeding 60K to- kens. In all experiments, each needle is placed 26 times at equal intervals across the evaluated context length. With randomly generated haystacks, 58 question-needle pairs, and 26 placements per context length, this setup results in 7,540 tests per context length experiment.
4.3. Evaluation Setup & Metric
During inference, use task template (see Appendix C) that instructs the model to answer the question based on the provided text. Since all questions seek the name of the character mentioned in the needle, any returned answer con- taining the correct name is considered accurate. Accuracy is reported as the proportion of tests with correct answers.
4.2. Models
Models are evaluated all tasks over context lengths of 250, 500, IK, 2K. 4K, 8K. 16K, and 32K. To take into account how models would perform on NOLIMA regardless of long-context scenario, we control the difficulty of the task by reporting a base score. Evaluations at context lengths of 250, 500, and 1K are used to compute the base score. These three are the shortest contexts. a model can solve the task at these lengths, then any deterioration of its performance greater lengths is expected be solely due to its difficulties with generalizing over long contexts. For each question- needle example, we compute the average scores over 5 haystacks, then take its maximum score across the 250, 500, and IK tests. The final base score is obtained by averaging these maximum scores across all question- needle examples. Inspired by Hsich al. (2024), we also report the effective length of each model. While they the performance of Llama 2 at 4K context length as threshold (85.6%), we define the threshold as 85% of the base score. Thus, the effective length of a model is the largest tested length that exceeds this threshold. Additionally, some plots show the normalized score, calculated by dividing the accuracy score by the base score.
For the filtering process, we opted for using the Llama 3.3 70b instruction tuned model (Meta, 2024). As a control test, for each question, we place its needle in 100 randomly selected chunks to determine whether the model (1) under- stands the filtering task and (2) is familiar with the facts and capable of inferring the answer. The model achieves a score of 99.8% in this test, indicating its ability to effectively flag conflicting information from the haystacks.
For the evaluation process, we select five closed-source models: GPT-4o, GPT-4o Mini (Hurst et al., 2024), Gemini 1.5 Pro Flash, Gemini 2.0 Flash (Gemini Team et al., 2023; 2024) and Claude 3.5 Sonnet (Anthropic. 2024), along with seven open-weight Llama models: The Llama 3.x model family (3.1 8B, 70B, 405B, and 3.3 70B) (Dubey et al., 2024; Meta, 2024), Mistral Large (Mistral, 2024), Command R+ (Cohere For AI, 2024), and Jamba 1.5 Mini (Team et al., 2024). All these models well-known and widely used in long-context setups. In our analysis reasoning- based prompting and models, we evaluate GPT-o1, GPT-o3 Mini (OpenAl et al., 2024; OpenAI, 2025), and DeepSeek-RR 70B (DeepSeck-Al et al., 2025). More de- tails regarding model versions and deployment details are described in Appendix B.
4.4. Results
Table 3 presents the performance results of all NOLIMA tests on the selected models. Most models achieve high base
5

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
scores, indicating that the designed needle set is relatively
1.0
simple answer in shorter contexts. Even models with base scores exceeding 90.0% exhibit a significantly shorter effective length than their claimed lengths, generally limited to <2K tokens, with GPT-4o being an exception. While GPT-4o demonstrates strong overall performance, it fails to generalize effectively beyond 8K tokens.' Out of the 13 models. 11 exhibit performance at 32K lengths that is half or less of their base scores. For comparison, in other benchmarks with similar settings, such as BABILong (QA1) (Kuratov et al., 2024) and RULER (Hsieh et al., 2024), Llama 3.1 70B achieves effective lengths of 16K and 32K, respectively. However, in NOLIMA, Llama 3.1 70B has an effective length of only 2K and shows a significant drop in performance at 32K lengths (42.7% vs. 94.3% base score). Models such as Claude 3.5 Sonnet, Gemini 1.5 Flash, GPT- 4o mini, and Llama 3.1 8B may have weaker base scores, but their effective lengths are calculated relative to these scores. This reveals an interesting observation: Models like Claude 3.5 Sonnet and Gemini 2.0 Flash, despite having a lower base score, may underperform in shorter contexts but demonstrate better length generalization than models with higher base scores, such as Llama 3.1 70B and Llama 3.3 70B. In fact, both Sonnet and Gemini 2.0 Flash achieve even higher raw scores in 4K-token experiments compared to some higher-base-ee base--soree models.
Normelted Soore
Normalzzed Score
0.9
0.8
0.8
0.7
0.6
One-hog
Defaull:
0.6
Twe-hop
Inverted
GPT40
0.5
GFT40
1aaa 3.3 700
0.4
Llama 3.3 70b
1K
2K
4K
8K
16K 32K
1K
2K
4K
8K
16K 32K
Context Length
Content Length
(a) One-hop vs. Two-hop
(b) Dcfault vs. Inverted
Figure 2. Impact of (a) number of hops and (b) inversion of order ("[CHAR] ... ...,,, " vs. "W.. ... CHAR") on normalized perfor- mance across GPT-4o and Llama 3.3 70B models. The red dotted line indicates the 0.85 effective threshold.
and two-hop tasks. It is evident that, for the same context lengths, questions involving two-hop latent reasoning steps are more challenging than those requiring one-hop reason- ing. Notably, the performance gap between one-hop and two-hop tasks widens with increasing context lengths. GPT- 4o demonstrates impressive generalization performance, handling both types of examples effectively even at context lengths up to 4K. A detailed breakdown of performance on one-hop and two-hop examples is provided in Appendix F, complementing the aggregate results shown in Table 3.
Model scaling generally improves performance, as seen in the progression from Llama 3.1 8B to 70B, Gemini 1.5 Flash to Pro, or GPT-4o mini to GPT-4o. However, the benefits of scaling diminish at larger scales; for example, the performance gap between Llama 3.1 70B and 405B is smaller (and sometimes worse) than that between 8B and 70B. In general, "lite" models such as Gemini 1.5 Flash, GPT-4o mini, and Llama 3.1 8B perform well in shorter contexts (<IK tokens) but fail to generalize effectively in longer contexts.
Each group of needles includes both default and an in- verted template and Figure 2(b) shows that inverted exam- ples are more challenging to answer. We argue this difficulty arises from the model's causal attention mechanism, par- ticularly in longer contexts where attention signals weaken. In the default template, the question in particular W2 can link directly to W., which generally will contain infor- mation about the character's name since the name appears earlier in the sequence. This allows the model to backtrace effectively from W,, through W,, to the character. In the inverted template, W2 may still attend to W., but since the fact is incomplete (the character hasn" 't been mentioned yet), the model cannot use that attention to resolve the question. Instead, it must rely on weaker signals encoded in the char- acter's name to establish the link, which becomes harder with longer contexts due to diminishing attention strength. While these findings shed light on the challenge, deeper mechanistic analysis beyond the scope of this paper and requires further study.
Based on the dataset construction method described in Sec- tion 4. NOLIMA can generate haystacks at any desired length. We applied this to test GPT-4o and Gemini 2.0 Flash at 64K and 128K tokens. GPT-4o maintained over 50% of its base score at 128K, while Gemini 2.0 Flash dropped to just 16.4%. Full results and evaluation details are provided in Appendix E.
1. LATENT HOPS & INVERSION
As discussed in Section 3, our needle set also includes exam- ples requiring two-hop associative linking from the question keyword to the needle keyword. To evaluate the impact on length generalization, Figure 2(a) presents the normal- ized performance of two top-performing models on one-hop
4.4.2. NEEDLE PLACEMENT DEPTH ANALYSIS
A common evaluation across NIAH-based benchmarks (Kamradt, 2023) examines the impact of needle placement within the context window. In Figure 3(a), we observe a set-in-the-middle effect (Liu et al., 2024) in 32K, where model performance dips when the needle appears in the
'In BABBLong, the effective length is also based on 8,%% of the OK base performance threshold
6

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
1.0
1.0
Accuracy
0.6
Accurucy
0.4
2000
0.2
8000
0.2
16000
0.0
100
1000
500
2000
1500
Depth (%)
Depth (%)
Tokons Degth
Tokonn Depth
(a) Full Sweep (One-hop)
(b) Full Sweep (Two--op)
(c) Last 2K (One-hop)
(d) Last 2K (Two-hop)
Figure 3. The full sweep plots (a & b) illustrate performance across the entire context window, where 0% corresponds to the beginning of the haystack and 100% the end. plots that range for various context lengths; 0 marks the end of the context, and larger values indicate positions farther from the end (up to 2K tokens inward). The color shading of each plot line represents the tested context length. To minimize noise and highlight trends more clearly, we increased the number of placements from 26 to 51 and applied moving average with window size of 12."
last 2K tokens (c & d) depict performance when needle placements are aligned within
middle of longer contexts.
depth the only changing factor in each plotline is the context length, which in turn means that the model has more tokens that it needs to attend to.
Additionally, Figure 3(b) reveals key phenomenon: longer contexts in more complex (two-hop) examples dampen the performance distribution over the full sweep depending on their length. In vanilla multi-document or NIAH-based benchmarks (Kamradt, 2023; Liu al., 2024), models per- form consistently well when the needle (or gold document) appears at the very beginning or end of the context window, with minimal impact from context length. However, in No- LIMA, as task complexity increases in two-hop scenarios, larger context sizes shift the entire trendline downward to- ward zero, with performance declining even at the edges of the context window.
Based on the final 2K results in Figure 3(c), the one-hop setup confirms our earlier observations from the full-sweep plots. The "lost-in-the-micee tiithhdeemdee"" phenomenon formance dips toward the center of the context-primarily appears in simpler tasks, Each plotline drops as it moves toward the center, reflecting its dependence placement position and the way the model encodes positional infor- mation. In contrast, the two-hop scenario appears to be influenced more by attention limitations than by position encoding alone. Figure 3(d) reveals that, rather than depth exacerbating performance drops, the plot lines remain rela- tively stable over the last 2K positions. However, context length significantly reduces the overall performance trends observed in this range. Llama 3.x models, like many other recent language models, feature rotary position embeddings (RoPE), relative PE (Su et al., 2024). For each token depth in Figure 3(d), as the relative distance between question and fact remains the same regardless of context length, position encoding does not explain the performance drop. Instead, the main limiting factor is the increased context length: as the number of tokens grows, the attention mechanism strug- gles to process information effectively. In the absence of strong surface-level cues (e.g., literal matches), locating rel- evant facts becomes challenging for the model, regardless of their position within long contexts.
per-
To further investigate this issue, we devise an alternative setup that focuses on analyzing the last 2K tokens instead align
of sweeping across the full context. Therefore, the placement positions in the last 2K tokens for all context lengths (see Figure 4). This ensures that for a certain token
2K
4K
8K
Irpat Saumach
2K
Lkrtk
4K
4.4.3. COT PROMPTING
Since NoLIMA examples require associative reasoning between the needle and question keywords retrieve the
Figure 4. Needle placements in full sweep (top) vs. last 2K tokens sweep (bottom): In the last 2K setup, placement positions are aligned in different context lengths, unlike the proportion- based positioning in full sweep.
*All figures Llama 3.3 70B. Plots without smoothing ap- plied are available in Appendix G.
7

---

NOLIMA: Long-Contex! Evaluation Beyond Literal Matching
4K
8K
16K
32K
Base
4K
8K
16K
32K
Score
One-hop
w/o CoT
90.3
84.1
73.2
56.2
Llama 3.3
70b
w/ CoT
95.6
91.1
82.6
60.6
w/o CoT w/ CoT
98.3
55.5
37.2
16.7
8.9
Increase rate
5.9%
8.3%
12.8%
7.8%
97.1
73.0
51.2
31.8
10.1
Two-hop
Reasoning models
w/o CoT
70.7
57.4
42.7
25.9
GPT-ol
99.9
92.0
78.0
60.1
31.1
w/ CoT
82.4
70.1
56.7
34.3
GPT-o3 Mini
98.8
52.8
36,9
25.5
18.9
Increase rate
16.5%
22. 1%
32.7%
32.4%
DeepSeek RI-DI.-70b
99.9
91.4
75.5
49.4
20.7
Table 4. Comparison of Shaain-of-Though (CoT) improvements in performance for Llama 3.3 70B, evaluated on both one-hop and two-hop tests.
Table S. Evaluation results of HOLIMA-Hard: Scores falling be- low 50% of the base score are highlighted in red
evaluation (Kamradt, 2023). (2) Multiple Choice (MC): questions that maintain the required latent associative rea- soning while incorporating literal matches. In this setup, the question includes four character names as answer op- tions-three from the haystack and one correct answer from the needle.
correct answer, in this part we evaluate when the model is prompted to reason in Shain-of-Thoughh (CoT) style (Wei et al., 2022) before returning a final answer (see Appendix C for more details). In Table 4, we present the results when asked for CoT compared to asking directly for the final answer. CoT prompting shows improvements over long- context tests and it shows a higher rate of improvement in two-shot. Despite the improvements, the tasks seem to remain challenging. For example, two-hop examples with CoT prompting barely achieve the scores of one-hop exam- ples without CoT and continue to perform poorly texts 16K tokens longer. The challenge with CoT prompting is that the questions in NOLIMA are straightforward. They are mentioning singular clue to the answer, meaning they cannot be further decomposed into simpler steps. This limits the benefits of CoT prompting. However, the difficulty lies in reasoning through the association between the question and the needle, which remains significant challenge for the model.
As expected, Table 6 shows that direct examples with high degree of literal overlap between the question and the needle are straightforward for the model to answer, even in long contexts, consistent with prior findings in RULER (Hsich et al., 2024). Additionally, literal matches significamtly aid the model when the questions remain unchanged, and only the multiple-choice format introduced. The inclusion of literal matches in the multiple-choiee setup provides sig- nificant guidance to the model. By offering the character names as answer options, including the correct name from the needle, the model can focus its search within a smaller scope. This dramatically simplifies the task of identifying the correct answer, as the literal match serves as a direct hint, reducing ambiguity in the reasoning process.
To assess the performance of reasoning- beasoning-based models (e.g., GPT-ol) on NOLIMA, we selected the 10 most challenging needle-question pairs from the 58 available, based on the results summarized in Table 3. We refer to this subset as NoL.MM and present the evaluation results in Table While reasoning-based models outperform CoT prompting on Llama 3.3, they still fail to achieve full-length generaliza- tion on this subset. Across all models, performance drops below the 50% mark at 32K context length. Notably, base scores are nearly perfect, demonstrating the simplicity of the task- within this designated "hard" subset. This means that even with intermediate reasoning steps, mod- els still struggle to link the needle the question in long contexts without surface-level cues.
Distracting Literal Matches. While literal matches serve as cues if they are part of the relevant fact, they can also act as distractors if they are irrelevant to the answer. In Section 2, we noted that some related benchmarks include similar documents in the context as distractors to test the model's ability to discern the correct answer from irrelevant ones. This setup creates matches between the query and both relevant and irrelevant documents or facts. In contrast,
8K
16K
32K
Direct
98.3
98.5
98.5
One-hop
84.1
73.2
56.2
- w/ Literal Match (MC)
98.7
97.4
93.1
4.4.4. ABLATION STUDY: LITERAL MATCH EFFECT
Two-hop w/ Literal Match (MC)
57.4
42.7
25,9
To examine the simplifying impact of literal matches on results, we define two new sets of tests: (1) Direct: ques- tions that explicitly ask about the fact stated in the needle by stating W,, in the question, resembling a vanilla NIAH
96.3
94.6
87.2
Table 6. Results in two literal match setups: direct and multiple
choice (MC) questions. Model: Llama 3.3 70B
8

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
1.0
RAG systems, a relevant document containing the correct an- swer may have a lexical gap with the query. So, even if such document is retrieved alongside others that likely have higher lexical similarity, language models may struggle to extract the correct answer, as they can become distracted by the lexical overlap with these other documents. This work highlights the need for benchmarks that go beyond surface- level retrieval to assess deeper reasoning. NoLIMA sets new standard for evaluating long-context comprehension and emphasizes the importance of developing approaches capable of handling complex reasoning in long contexts.
0.9
Normalized Soote
0.8
0.7
w/o Distractor
w/ Distractor
GPT-4o
Llama 3.3 70b
1K
2K
4K
8K
Context Length
Figure 5. Normalized performance comparison across GPT-4o and Llama 3.3 70B models, with and without distractors. The red dotted line marks the 0.85 effective threshold.
Impact Statement
This paper presents work aimed advancing the field of long-context language modeling by evaluating and analyz- ing the most commonly used LLMs. There are many poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.
NOLIMA allows us to explore a different scenario: when the context contains distracting words overlapping with the question, while the relevant fact has minimal overlap with the query. We insert a distractor sentence into the haystack (details in Appendix D) containing W, but entirely irrel- evant to both the needle and the question's 's intent. This setup poses a significant challenge, requiring the model to disregard irrelevant literal overlaps while identifying a relevant fact with no meaningful overlap with the query. As shown in Figure 5, such distractors have substantial impact on degrading length generalization. GPT-4o now demonstrates an effective length of just 1K, while Llama 3.3 70B performs even worse. While adding distractors slightly lowers base scores (GPT-4o: 93.8, Llama 3.3 70B: 84.4), the normalized plots still clearly illustrate a performance drop at longer lengths. These results highlight the challenge of resolving queries in contexts where irrelevant overlaps mislead the model, and the relevant fact shares no overlap with the question.
Acknowledgments
We thank Abdullatif Köksal, Leonie Weissweiler, and Amir Hossein Kargaran for their valuable feedback and support, particularly in the early stages of this project. We also appreciate the insights from our peers, and we are grateful to the anonymous reviewers for their constructive comments.
References
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S. C., Zhang, B., Faust, A., and Larochelle, H. Many-shot in-context learning. In ICML 2024 Workshop on In-Context Learning, 2024. URL https: / //openzeview.ctt net wodoumiidgooiiiiioooo wtt.ggoi7p.H........
Anthropic, A. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 2024.
5. Conclusion
NOLIMA provides a challenging benchmark for evaluating the reasoning capabilities of large language models in long- context settings. By removing literal overlaps between ques- tions and relevant information, the benchmark tests models' ability to infer and link information within extensive irrel- evant content. Our findings show that even state-of- ofh-th--rtt models struggle, especially as context length increases, re- vealing serious limits in their attention mechanism. While causal attention should theoretically access all previous to- kens, models often rely on surface-level cues in longer con- texts. This vulnerability becomes more pronounced when the context contains literal matches that fail to connect with the truly relevant fact, causing models to overlook the cor- rect information and focus instead on superficial signals. We believe our findings with NOLIMA are likely to extend to downstream applications. For instance, in search engines or
Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Re, C. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: //wp:///ie///eeeeeeemmeemeeeeeee net/ forum? (iaum?iddin
Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016.
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, I.,, Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multitask benchmark for long context understanding. In Ku, L.W,., Martins, A., and Srikumar, V. (eds.). Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Vol-

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
ume 1: Long Papers), pp. 19-33137, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.186533w1/22224.clloogg 172. URL https: htw.aaannnooolyy.og///...
Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference Empirical Methods in Natu- ral Language Processing, pp. 16376-16586, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. 00.16666//1102244mmmmmmmom 924. URL https: /tta:://aclaaana http://aaaantthooggy.... 2ww.o//ntttooooooorg/2222222222 org/2024. emnlp mallly-maing main. 924/.
acl- 1oollllonn... .ong.172/.
Chang, Y. Lo, K., Goyal, T., and lyyer, M. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https: // openreview net/ forum? d= d3dd=t33tteeee.e..
Graves, A., Wayne, G., and Danihelka, I. Neural turing machines, 2014. URL 5400.
abs/
Chen, S., Wong, S., Chen, L., and Tian, Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15590 2023.
Hengle, A., Bajpai, P., Dan, S., and Chakraborty, T. Multi- lingual needle in haystack: Investigating long-context behavior of multilingual large language models. arXiv preprint arXiv:2408. 10151, 2024.
Cohere For AI. cla--commaand-rppus------2222220 2024, URL
c4ai-command 08-22444.
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER: What's s the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https: //openrevew net/ forum? IIw.c 65y.
DeepSeek-AI, Guo, D., Yang. D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q.. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capabil- ity in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276 2024.
Dong, Z., Tang, T., Li, J., Zhao, W. X., and Wen, J.-R. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Calzolari, N., Kan, M.-.,, Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 2086-2099, Torino, Italia, May 2024. ELRA and ICCL. URL https: : ////aaaaannoooooooo// orgg 2024. wec.-reccmm
Izacard, G. Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense in- formation retrieval with contrastive learning. Transac- tions on Machine Learning Research, 2022. ISSN 2835- 8856. URL https: http:///ppeeeeeeeeeeeeppeeeeeeee net forum? id=jkN1 pxi7000
188/.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv:2407. 27783, 2024.
Jiang, B., Xie, Y., Hao, Z., Wang, X., Mallick, T., Su, W. J., Taylor, C. J., and Roth, D. A peek into token bias: Large language models are not yet genuine reasoners. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceed- ings of the 2024 Conference Natural Language Processing, pp. 4722-4756 Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10. (0.18663//1120044cmml--mmo 272. URL httpp/////.aaaathooggg..oo/ emnlp-main.25 272/.
Empirical Methods in
Gemini Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
2024.
Gemini Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Kamradt, G. Needle in Sryssack-pressuree testing 1lms. Github Repository, pp. 28, 2023.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwa- sawa, Y. Large language models are zero-shot reason- ers. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: openreview. net/ forum?id=e27Bb5yyy2
Goldman, O., Jacovi, A., Slobodkin, A., Maimon, A., Dagan, I. and Tsarfaty, R. Is really long con- text if all you need is retrieval? towards gen- uinely difficult long context NLP. In Al-Onaizan, Y.,
10

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D. I., Sorokin, A., and Burtsev, M. BABILong: Test- ing the limits of LLMs with long context reasoning-in- a-haastack. In The eiirt Conference on Neural Information Processing Systems Datasets and Bench- marks Track, 2024. URL https: openreview. net/ forum? www.nniiddddmdmmmmmmmmmm
Bangkok, Thailand, August 2024. Association for Com- putational Linguisties. doi: 10. 10.18655//1/2022aall 747. URL https: http://aaaanncolo..orogg//////// hal-long.747..
Meta, A. Llama 3.3 model card. 2024. URL
https://githinn commmeta--lamma
1lama - mode 1a
Kwon, W. Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E.., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposhum on Operating Systems Principles, 2023.
3/MODEL_CABL md.
Mistral, A. Mistral large 2. Mistral Large 2 Blog- post, 2024. URL https: //tm///tsaaaaaeeeeeeeeeee
mistral-Large-2451
Lee, J., Chen, A., Dai, Z., Dua, D., Sachan, D. S., Bo- ratko, M., Luan, Y., Amold, S. M. R., Perot, V., Dalmia, S., Hu, H., Lin, X., Pasupat, P., Amini, A., Cole, R., Riedel, S., Naim, I. Chang, M.-W., and Guu, K. Can congecontext language models subsume retrieval, rag, sql, h
Mohtashami, A. and Jaggi, M. Random-access infi- nite context length for transformers. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: https::/apenree/////s///// /twp///veeeeeeeeeeeeeeeeee net/ forum? i 1d=7eHn64w0vy.
and more?, 2024. URLhttps:: /ttp///aavvvv///////vv//eee/.aaa wwwoorggaaaooo abs/ 2406 18.06.10 1.666.3122 13121.
Navigli, R., Conia, S., and Ross, B. Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality, 15(2), June 2023. ISSN 1936 1955. doi: 10.1145/3597307. URL https: /doi org/10. 1145/ 3597307
Levy, M., Jacoby, A., and Goldberg, Y. Same task, more tokens: the impact of imput length on the reasoning perfor- mance of large language models. In Ku, L..W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 15339-15353, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/20024acl-looo 818. URL https: http///aaanntoooy..orrg/p// acl 1olg.oon.o. 818/.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and induction heads, 2022. URL https:// arxiv. abs 2202222222222 ...00..10 11895.
Li, M., Zhang, S., Liu, Y., and Chen, K. Needlebench: Can lims do retrieval and reasoning in million context window?, 2024. URL https: www.aiiww.ggaaae....e.e.. 2407.11963
OpenAL. URL 13-mini-system--cn
Openai 03-mini system card. 2025.
https: http///ppnna..ccoo com/ index/
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: //aclanthology org/ 1013//
OpenAL, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., al. Openai ol system card. arXiv preprint urXiv:2412.16720, 2024.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157- 00068. URL https:
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. YaRN: Ef- ficient context window extension of large language mod- els. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:///eepreeii..eeme net / forum? id=:HBfx22ull...
173, 2024. doi: 10. 00638. / //aclanthology: org. ta.l-1....l1...
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbi- eri, F., and Fang, Y. Evaluating very long-term conver- sational memory of LLM agents. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Volume Long Papers), pp. 13851-13870
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127066 2024.
Team, G. Gemma 3. 2025. URL https: http::/ //g0:./////../ hgt.//lo/////oo///////o/o. Gemma 3Report.

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Team, J., Lenz, B., Arazi, A., Bergman, A., Manevich, A., Peleg, B., Aviram, B., Almagor, C., Fridman, C., Padnos, D., et al. Jamba-1.5: Hybrid transformer-mambe models at scale. arXiv preprint arXiv: 12570, 2024.
Computational Linguisties: Human Language Technolo- gies (Volumee I: Long Papers), pp- 4643-4663, Mex- ico City, Mexico, June 2024. Association for Compu- tational Linguistics. doi: 10.18653/v1//2244..u 260. URL https: /ass://aalaaa /aalaaacachhccccc www.oggg......... naacl- laacl-1ong.22 1on..oong...0
Vodrahalli, K., Ontanon, S., Tripuraneni, N., Xu, K., Jain, S., Shivanna, R., Hui, J., Dikkala, N., Kazemi, M., Fatemi, B., Anil, R., Dyer, E., Shakeri, S., Vij, R., Mehta, H., Ramasesh, V., Le, Q. Chi, E., Lu, Y., Fi- rat, O., Lazaridou, A., Lespiau, J.-B., Attaluri, N., and Olszewska, K. Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024. URL https: /awvvvv.vvvvv///////e//eeeevvvvv org/ abs / 2409 9. 12640.
Yen, H., Gao, T., Hou, M., Ding, K.,, Fleischer, D., Izsak, P., Wasserblat, M., and Chen, D. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694, 2024.
Zhang, X., Chen, Y., Hu, S., Xu. Z., Chen, J., Hao, M. K., Han, X., Thai, Z. L.,, Wang, S., Liu, Z., and Sun, M. oobench: Extending long context evaluation beyond 100k tokens, 2024. URL htips: /ar.iv.oogg.gaaa.......ooo.. /ba// 2402.13712
Wang, M., Chen, L., Cheng. F., Liao, S., Zhang. X., Wu, B., Yu, H., Xu, N., Zhang, L.., Luo, R., Li, Y., Yang, M.. Huang, F., and Li, Y. Leave no document behind: Benchmarking long-context LLMs with extended multi- doc QA. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods Natural Language Processing, pp. 5627- 5646, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2024. amnlp-main.322 URL https:/ /tle//aclaan /aclaaacnnnoooy..... org org/ emnlp main.322//
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of-thoughh prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022.
Wolf, T., Debut, I.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P. Rault, T., Louf, R., Funtow- icz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In Liu, Q and Schlangen, D. (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Process- ing: System remonntaations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.118663/112000.mmll-mmm aclanthology org/2023 ewneemnneepee demos.ee.........eoooeee..e....e 6/-
URL https://
Xiong, W., Liu, J.. Molybog, I., Zhang, H., Bhargava, P., Hou, R. Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M.. Wang, S., and Ma, H. Effective long-context scal- ing of foundation models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for
12

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
A. Needle Set Design& & Considerations
In Table 7, we demonstrate the full needle set that we use in NOLIMA. In designing the needle templates, there are multiple considerations involved. First, all templates in the needle set begin with a small introductory phrase or least one word (e.g.. "Actually," "In 2013.") to distinguish themselves from the preceding context. This ensures that the needle's keyword or character is not inadvertently linked to the prior context. Since newline is appended at the end of each needle, this issue is mitigated if the keyword or character appears at the end of the needle.
Question
Needles
Keyword Types
Def.
There was [CHAR] who was an engineer liv- W, Countries, cities, states ing in Wn.
Which character has been to Wg?
Inv.
There was engineer living in W.. named [CHAR].
W
Countries, cities, states
Def.
Actually, [CHAR] lives next the Wn. W, next to where [CHAR] lives.
W Buildings & Landmarks
Which character has been to W
Inv.
W
Countries, cities, states
Def.
2013, after waiting line for hours, [CHAR] finally saw the original Wa painting up close. In 2013, the original Wa painting was seen up close by [CHAR], finally, after waiting in line for hours.
W,.
Buildings & Landmarks
Which character has been W
Inv.
W Countries, citics, states
Def.
A message came in from [CHAR] saying, "I'm
W, Dictary restriction
W,. and nothing more.
(c.g., lactose intolerant)
Which character cannot drink W
Inv.
message came saying, "I'm W.." from [CHAR].
W Drinks & Beverages
Def.
Then [CHAR] mentioned that he has been W,, for years.
W, Dietary (e.g.. vegan)
restriction
Which character cannot. eat W
Inv.
There was a Wn, guest, named [CHAR].
W Foods
Table 7. Our proposed needle set templates in NOLIMA. The placeholders [CHAR]. Wg. and W., represent the randomly sclected
character (also the answer), the query keyword, and the needle keyword, respectively, Def.: default order. Inv.; inverted order.
Another consideration is that the needle keyword should be uniquely associated with the query keyword. For instance, in the following sentence:
There was an engineer living in Cambridge, named Yuki.
Although the term "Cambridge" is commonly associated with the "United Kingdom," it is not uniquely so; it could also refer to cities in the United States, Canada, or other countries. Additionally, we aim to avoid relying on language-specific markers. Many cities have distinctive elements in their names, such as orthographic features, morphological structures, or cultural naming conventions, that hint at their linguistic or geographic origins. By minimizing the influence of such markers, the needle design ensures a more rigorous evaluation of the model's ability to make meaningful connections based on learned knowledge rather than surface-level linguistic cues. For each template, we manually curated 2-6 keyword pairs, resulting in total of 28 keyword pairs. Taking into account the order of fact statements, this generates 58 needle-question pairs.
B. Models
In Table 8, we list all the models selected for evaluation. Models that are open weights were deployed using the vLLM library (Kwon et al., 2023), with weights obtained from HuggingFace (Wolf et al., 2020).
C. Task Prompt Templates & Inference Settings
In Table 9, present the task prompts used across all evaluations. While do not employ the commonly used "Let's think step by step" prompt in the Shain-of-Though (CoT) setup (Kojima et al., 2022), our prompt encourages the model to elaborate and expand its reasoning sufficiently before producing a final answer. To manage the extensive testing scope-7,55e tests per context length-wee we limit reasoning to three sentences or a maximum of 192 generated tokens. In the CoT setup, a test is considered successful the final answer (on the newline) includes the correct answer. This differs with the non-CoT setup, where success is determined based on whether the correct answer is present within the
13

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Model
Context Length
Open Weights?
Model Revision
GPT-4.1
IM
No
9pt-4.1-2025-04-114
GPT-4.1 mini
IM
No
9pt-9_11mmin1-22255-----11
GPT-4.1 nano
1M
No
9P0-2.--aan--2225-4441
GPT-4o
128K
No
gpt-4o-2024-11120
GPT-40 mini
128K
No gpt-4o-mln1-20220771
Llama Maverick
IM
Yes
Llama 4 Scout
10M
Yes mB2_:_10ao/1111244455000----1111
Llama 3.3 70B
128K
Yes
mata--1aaraaliaamm--333---------
Llama 3.1 405B
128K
Yes
Llama 3.1 70B
128K
Yes oete=---aann111mmmm33-110-21-222
Llama 8B
128K
Yes
Gemini 1.5 Pro
2M
No gemini-1..-prr--02
Gemini 1.5 Flash
IM
No
gemini-i.--Elash-000
Gemini 2.0 Flash
IM
No
gemini-2.0-flast:
Gemini 2.5 Flash
IM
No
Gemma 27B
128K
Yes
google/gema-3-277b-ii
Gemma 3 12B
128K
Yes
30oglae/gema-33-12bitt
Gemma 4B
128K
Yes 900gla/gema-3-4b-tt
Claude 3.5 Sonnet
200K
No atterogecccaaade-333--ea22222222
Jamba 1.5 Mini
256K
Yes
a22:aab/AA722--Gmaaa-15555.111
Command R+
128K
Yes Colerrafooss/ccai...coeeeemmmeap
Mistral Large 2
128K
Yes
Reaxoning-based models modelss
GPT-ol
128K
No
gpt-o1-2024-12-17
GPT-o3 Mini
128K
No
gpt-o3-mini-2025--11
DeepSeck RI-DL-70b
128K
Yes
Table 8. Details of the selected models used for evaluation.
generated output. For all standard instruction models, use greedy decoding during generation. For reasoning- based models, we utilize the default sampling decoding mechanism for GPT-ol and GPT-o3 Mini, while R1-based models employ top-P sampling with 0.95 and temperature of 0.6. In addition, we cap the maximum number of generated tokens in reasoning-based models at 1536 tokens, including both reasoning and output tokens. In all models, we apply each model's instruction-tuned chat templates.
D. Distractor Design
To construct and integrate the distractor sentences mentioned in Section 4.4.4, devised two templates, applied uniformly across all needle-question pairs. Depending on the Wg. use one of the following templates:
There was an article about W, the daily newspaper.
or
There was photo of W, in the daily newspaper.
Some instances of W, may naturally include an article (e.g., "a" "an"), making them better suited the second template, while others fit the first. Regardless of the choice, the templates are designed remain neutral and unrelated to the intent of the question the fact stated by any needle.
To minimize interference with the needle, we randomly place the distractor sentence while ensuring token distance of at least 20% of the context length. For example, in IK-token test, the distractor must be at least 200 tokens away from the needle. Additionally, to avoid any advantage from proximity to the beginning or end the context (which may gain extra attention), we restrict placement to between the 20% and 80% marks of the context length. Together, these two constraints leave a span of 40%-60% of the context length available for random placement of the distractor sentence.
14

---

NoLIMA: Long-Context Evaluation Beyond Literal Matching
Mode
Prompt Template
You will answer question based on the following book snippet:
(haystack w/ needle}
Use the information provided in the book snippet the question. Your answer should short and based on either explicitly stated facts or strong, logical inferences.
w/o CoT
Question: {question]
Return only the final answer with no additional explanation or reasoning.
You will answer question based on the following book snippet:
(haystack needle)
Use information provided in the book snippet answer the question. Be aware that some details may not stated directly, and you may need to INFER the answer based on the given information. Begin with brief explanation of your reasoning in NO MORE THAN THREE (3) sentences. Then, return the final answer on new line.
w/ CoT
Question: {question)
Table 9. Details of prompt templates utilized in our evaluation.
Claimed Length
Effective Length
Basc Score (x0.85: Thr.)
Models
1K
2K
4K
8K
16K
32K
64K
128K"
GPT-4.1
IM
16K
97.0 (82.5)
95.6
95.2
91.7
87.5
84.9
79.8
69.7
64.7
GPT-4o
128K
8K
99.3 (84.4)
98.1
95.7
89.2
81.6
69.7
62.4
56.0
Gemini 2.5 Flash
IM
2K
94.4 (80.2)
90.1
86.1
79.4
68.2
57.9
48.4
Gemini 2.0 Flash
IM
4K
89.4 (76.0)
87.7
87.5
77.9
64.7
48.2
41.0
33.0
16.4
Llama Maverick
IM
2K
90.1 (76.6)
81.6
78.3
68.8
49.0
34.3
24.5
Gemma 27B
128K
<1K
88.6 (75.3)
73.3
65.6
48.1
32.7
20.2
9.5
Gemma 12B
128K
1K
87.4 (74.3)
74.7
61.8
39.9
27.4
16.8
7.3
Llama 4 Scout
10M
1K
81.7 (69.4)
72.3
61.8
50.8
35.5
26.9
21.6
GPT-4. Mini
IM
<IK
80.9 (68.8)
66.7
62.8
58.7
51.9
46,2
38.8
GPT-4.1 Nano
IM
1K
80.7 (68.6)
60.8
48.2
36.7
28.8
19.5
9.4
Gemma 4B
128K
<IK
73.6 (62.6)
50.3
35.3
16.4
7.5
2.3
0.9
Table 10. NOLIMA benchmark results for GPT-4o, Gemini 2.0 Flash, and additional recent models. For models with stronger performance
32K, extend evaluation to 64K and 128K using the same setup. Scores above the effective threshold are underlined; scores below 50% of the base score shaded in red
E. Results Beyond 32K & Recent LLMs
Based on the dataset configuration outlined in Section 4.1, we construct haystacks by randomly concatenating snippets extracted from the filtered books dataset. This setup enables evaluation across scalable context lengths, including 64K and 128K tokens, and can be extended further as model limits allow.
Two practical adjustments were made in this evaluation setup, without altering the core methodology: (1) Reduced Placement Count: Due cost limitations, we reduce the number of needle placements from the default 26 to 11 placements per context length. This change minimizes API usage while still preserving meaningful coverage across the haystack. (2) Token Limit Constraints: GPT-4o has strict 128,000 token limit, including both input and output tokens. To accommodate the task prompt and model response, we limit the haystack length to 17,,000 tokens for this model. To ensure a fair comparison, we use the same haystack length for Gemini 2.0 Flash.
The results of this evaluation are presented in Table 10, focusing on GPT-4o and Gemini 2.0 Flash 64K and 128K context
127,500-tooken haystack used based on model token limit constraints.
15

---

NoLIMA: Long-Contex, Evaluation Beyond Literal Matching
1.0
1.0
WW
Accurscy
0.6
Accurary
Accurucy
W
0.4
2000
0.2
8000
0.2
16000
100
1000
500
1500
Degth (%)
Depth (96)
Tokana Degth
Tokons Depth
(a) Sweep (One-hop)
(b) Full Sweep (Twoo-op)
(c) Last 2K (One-hop)
(d) Last 2K (Two-hop)
Figure 6. Unsmoothed needle placement depth plots corresponding to the smoothed results in Figure 3. These plots reflect raw performance values prior to applying moving average.
lengths. While GPT-4o does not fully meet its claimed performance at the maximum context length, it nonetheless maintains over 50% of its base score at 128K, indicating relatively strong performance at that scale. In contrast, Gemini 2.0 Flash shows a sharper decline, dropping to 16.4% 128K.
To cover recent model releases, Table 10 also includes entries for newer models such the GPT-4.1 series and Gemini 2.5 Flash (Team, 2025; Gemini Team et al., 2024). For those that showed stronger performance at 32K, we also extend the evaluation to 128K using the same setup. GPT-4.1 shows clear improvements over prior models; however, its effective context length remains around 16K-well below the claimed MM-and its performance drops below 65% on 128K context lengths.
F. One- & Two-hop Results
Tables 1l and 12 present detailed results the one-hop and two-hop subsets of the NOLIMA benchmark, evaluated across the selected models. The tables follow the same format and thresholding criteria as Table 3, reporting both the claimed and effective context lengths. Note that the base scores-and consequently the thresholds-are computed separately for each subset. Although both one-hop and two-hop subsets show strong base scores, two-hop tasks generally yield shorter effective context lengths. This suggests that while models can perform well on complex reasoning in short contexts, their performance degrades more rapidly as context length increases, indicating reduced length generalization under greater reasoning demands.
Claimed Length
Effective Length
Base Score (x0.85: Thr.)
Models
1K
2K
4K
8K
16K
32K
GPT-4o
128K
16K
99.3 (84.4)
97.7
97.5
87.3
79.8
LJama 3.3 70B
128K
8K
97.7 (83.1)
97.1
93.6
90.3
84.1
73.2
56.2
Llama 3.1 405B
128K
4K
91.7 (78.0)
88.7
87.3
80.2
68.4
59.6
49.4
Llama 3.1 70B
128K
4K
96.4 (82.0)
95.2
89.8
82.7
76.9
66.3
56.9
Gemini 1.5 Pro
2M
4K
90.8 (77.2)
85.7
85.5
81.9
72.3
63.4
55.1
Jamba 1.5 Mini
256K
2K
(79.1)
80.0
80.2
71.5
62.9
Command R+
128K
2K
917.877.)
79.4
78.4
75.6
$2.2
26.9
10.4
Gemini 2.0 Flash
IM
4K
92.5 (78.6)
91.5
93.5
89.6
78.4
61.8
52.8
Mistral Large
128K
4K
870.8))
82.5
86.1
80.3
62.5
44.1
27.5
Claude 3.5 Sonnet
200K
8K
(76.9)
89.9
91.6
89.5
78.2
61.1
45.2
Gemini Flash
IM
IK
85.7 (72.9)
76.4
71.8
57.0
48.7
41.3
GPT-4o mini
128K
IK
88.4 (75.2)
81.0
73.6
57.6
45.4
30.2
20.0
Llama 3.1 8B
128K
IK
83.0 (70.5)
75.7
69.3
60.7
49.6
35.7
22.7
Table 11. NoLIMA benchmark results one-hop examples. Base scores and effective lengths computed using only the one-hop subset. Scores above the effective threshold underlined, while scores that below 50% of the base shaded in red
WWithout reasoning (thinking budget= = 0).
16

---

NoLIMA: Long-Contex, Evaluation Beyond Literal Matching
Claimed Length
Effective Length
Base Score
Models
1K
2K
4K
8K
16K
32K
(x0.85: Thr.)
GPT-4o
128K
8K
99.3 (84.4)
98.7
98.7
95.8
85.9
74.6
57.4
Llama 3.3 70B
128K
IK
96.7 (82.2)
90.6
79.8
70.7
57.4
42.7
25.9
Llama 3.1 405B
128K
2K
95.3 (81.0)
89.4
82.0
67.4
49.8
34.6
23.8
Llama 3.1 70B
128K
IK
92.0 (78.2)
85.9
72.0
57.0
45.2
33.8
26.4
Gemini Pro
2M
IK
94.9 (80.7)
87.1
79.4
53.6
45.7
Jamba 1,5 Mini
256K
<IK
91.7 (77.9)
71.8
66.6
62.0
50.7
40.0
28.4
Command R+ Gemini 2.0 Flash
128K
<IK
97..5
74.0
67.4
54.7
23,8
14.3
3.8
IM
2K
85.7 (72.8)
83.1
80.0
63.6
47.8
31.4
26.4
Mistral Large
128K
2K
(79.5)
90.4
84.7
64.7
18.4
Claude 3.5 Sonnet
200K
2K
(71.4)
79.8
74.5
63.0
41.5
26.9
10.9
Gemini 1.5 Flash
IM
<IK
83.5 (71.0)
59.1
49.0
35.6
28.8
19.2
12.9
GPT-4o mini
128K
<IK
80.4 (68.3)
51,4
39.3
27.6
16.9
8.8
5.9
Llama 3.1 8B
128K
<IK
68.9 (58.6)
53.4
36.0
23.6
10.0
6.5
3.8
Table NOLIMA benchmark results on two-hop examples. Base scores effective lengths are computed using only the two-hop subset. Scores above the effective threshold are underlined, while scores that are below 50% the base score are shaded red
G. Raw Needle Placement Depth Plots
Figure presents the needle placement depth plots corresponding Figure prior the application of the moving average employed in the main figure.
17


Reviews:
[{"review_id": "XJPBhb5V0c", "reviewer": "Reviewer_7az5", "summary": "The paper introduces NoLiMa, a benchmark designed for advanced needle-in-a-haystack (NIAH) tests with minimal lexical overlap between questions and the relevant information (needles) within the context. NoLiMa comprises 56 question-needle pairs, each paired with contexts of varying lengths. The benchmark minimizes literal overlap between questions and the relevant information, compelling models to rely on latent associations (one-hop and two-hop reasoning) and world knowledge rather than surface-level pattern matching.\n\nThe study evaluated 12 long-context Large Language Models (LLMs) on these question-needle pairs with different context lengths, including a short context used as a baseline to assess the models' fundamental capability to answer questions without the influence of long contexts. The results indicate that 10 out of 12 models experienced a performance drop below 50% of their baseline at 32K tokens. Even the strongest model, GPT-4o, saw a significant decline from 99.3% (baseline score) to 69.7% at 32K tokens. These findings suggest that the long-context reasoning capabilities of LLMs have been overstated in existing benchmarks that rely on literal matches between questions and context.Additionally, the paper provides a comprehensive analysis of latent hops, fact direction, and the position of needles within the context.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.\n\n**[Multi-hop retrieval]**: Multi-hop retrieval, which involves fact chaining, is discussed in our \"Related Work\" and \"Introduction\" (Hsieh et al., 2024; Levy et al., 2024). NoLiMa focuses on a prior step—locating relevant fact(s) from long contexts, which is a prerequisite for effective multi-hop reasoning.\n\nNote: The U-NIAH paper was released after the submission deadline.\n\n\n**[Insights on attention weights]**: Analyzing attention weights at long context lengths is extremely memory-intensive. For example, with 8K tokens, a LLaMA 3.1 8B model yields 32(layers) × 32(heads) × 8000**2 = 65 billion weights (~260 GB) for a single input. That said, we did limited analysis comparing inverted vs. default cases on some examples. We found that in inverted settings, W_n gets relatively more attention than the character name when W_n appears earlier—loosely supporting the theory in lines 300–306 (col 2). But due to high computational cost and the analysis being somewhat outside our main scope, broader experiments (e.g., on LLaMA 3.3 70B or full dataset or longer lengths) weren't feasible.\n\n\n**[Non-lexical synonyms]**: While using non-lexically matched synonyms helps avoid literal matching, their embeddings are often similar enough to still act as a match. Moreover, not all our examples rely solely on hyponym-hypernym reasoning—some, like the dietary cases (e.g., lactose intolerance <==> milk), require commonsense reasoning.", "claims_and_evidence": "The main claim of the paper is that model performance significantly degrades as context length increases when literal matching is removed. This assertion is generally supported by the experiments, such as the observation that 10 out of 12 models dropped below 50% of their baseline performance at 32K tokens on the NoLiMa benchmark.\nHowever, my concern is that NoLiMa only addresses a limited aspect of the problem. The non-lexical overlap in NoLiMa primarily reflects hyponym-hypernym relationships, such as \"city\" to \"state.\" There are many other factors to consider when assessing long-context reasoning without lexical overlap, including synonyms, entity abbreviations, and cross-lingual elements.", "methods_and_evaluation_criteria": "I find the methods and evaluation criteria to be sound.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "The experimental designs in the paper are sound and well-executed. The experiments cover a diverse range of models, including state-of-the-art models and smaller models, as well as open-sourced and closed-sourced models. The experiments include variations to provide comprehensive analysis, such as comparing CoT vs non-CoT approaches and one-hop versus two-hop reasoning.\n\nWhile the current experimental setup is necessary, I believe that providing an analysis of the attention mechanisms in open-source LLMs could further benefit the community. This would help in understanding the results and improving future LLMs. The current experiments focus on aggregated results from a limited dataset size, which may reduce the depth of insight gained.", "relation_to_broader_scientific_literature": "The findings in this paper could significantly benefit the LLM community by drawing more attention to non-lexical-overlap long context tasks. The proposed dataset serves as a valuable benchmark for evaluating Needle-in-a-Haystack capabilities.", "essential_references_not_discussed": "Multi-hop retrieval has been widely discussed in retrieval-augmented generation (RAG) work. The paper could benefit from a brief discussion comparing the NoLiMa dataset with other RAG datasets to highlight its unique contributions and potential synergies.\n\nMulti-hop Question Answering https://arxiv.org/pdf/2204.09140\nU-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack https://arxiv.org/pdf/2503.00353v1", "other_strengths_and_weaknesses": "See my other comments.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Are there any insights derived from the attention weights in LLMs in the experiments on NoLiMa?\nAre LLMs robust to long contexts with non-lexical synonyms rather than one-hop hyponym-hypernym reasoning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZEnqTltadP", "reviewer": "Reviewer_bVpt", "summary": "This paper present a benchmark which extend NIAH with the needle set requiring models to infer latent associations beyond literal matching. It shows current long context LLMs will have performance degradation in their proposed benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.\n\n**[32K limit]**: NoLiMa has no inherent context length limit. As stated (lines 200–206, 2nd col.), haystacks are built from random story snippets (using the filtered stories) and can be extended to any length. Since all models drop below the 0.85×base score at 16K and 32K (10 out of 12 drop below 0.5×base score), extending further offers little to no extra insight.\n\nNevertheless, to demonstrate this, **we will include results at 64K and 128K** for two models: **GPT-4o**, the top-performing model in Table 3, and **Gemini 2.0 Flash**, a new model we plan to add.\n\n**[Differences with [1]]**: While [1] requires global reasoning over entire novels, NoLiMa focuses on the retrievability of single facts. Notably, questions in [1] often include key cues (e.g., character names) that help narrow down relevant spans—similar to our multi-choice setup in Section 4.4.4. Even without strong literal overlap, the presence of answer choices allows the model to locate relevant regions and reason accordingly.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "There are previous works for more realistic long context benchmark like LongBench V1 and V2, and complex version of NIAH like RULER, but the author's perspective is still novel.", "essential_references_not_discussed": "No, as far as I know.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The proposed benchmark is novel in terms of the problem definition, focusing on avoiding literal matching and constructing latent associations that are important for current long-context evaluation.\n2. I appreciate the benchmark creation process described in Figure 1, which avoids conflicting information and distractor keywords while ensuring the purity of the evaluation task.\n3. The experiment is comprehensive and the results are interesting - the scores for 32K are notably low, and the analysis of latent hops & inversions provides valuable insights.\n\nWeaknesses:\n\n1. The context length is only limited to 32K; it could be extended to 128K to examine what happens when further generalizing the context length.", "other_comments_or_suggestions": "No.", "questions_for_authors": "I am curious about the differences and relationship between this paper and [1], which requires global reasoning ability and might also not have severe literal matching problems.\n\n[1] One Thousand and One Pairs: A “novel” challenge for long-context language models, Karpinska et al.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7A9Nm57suL", "reviewer": "Reviewer_wAvn", "summary": "This work provides an examination of the capability of large language models (LLMs) to handle long-context information retrieval tasks.\nThe authors propose a new benchmark, NoLiMa, which is designed to test the ability of LLMs to find relevant information in long texts without relying on direct lexical cues.\nThis benchmark is an important contribution to the field, as it addresses the limitations of existing needle-in-a-haystack (NIAH) tests by reducing literal matches and forcing models to rely on more complex inference mechanisms.\nThe paper's evaluation of 12 popular LLMs across different context lengths is revealing.\nThe findings that performance degrades as context length increases are critical for understanding the current limitations of LLMs.\nThe in-depth analysis about the impact of latent hop and CoT shed further light to the understanding of long-context reasoning capabilities in LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to thank the reviewer for their thoughtful review and constructive feedback.\n\n**[Our works contribution compared to RULER]:**\n- Our related work section highlights how extensively literal matching is involved across various long-context benchmarks—including RULER.\n- We show that literal matches play an impactful role in affecting the results (in Section 4.4.4) an analysis that **cannot be done with RULER due to their existing literal matches**.\n- Since RULER is heavily based on lexical matching, it does not support an analysis of latent hops (4.4.1) and needle placement with reasoning variability (4.4.2). We would appreciate it if the reviewer could describe how such an analysis could be done on RULER.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This work provides an empirical study. There is no theoretical proof.", "experimental_designs_or_analyses": "Yes. I have checked all the results and analysis in Section 4. They make sence to me.", "relation_to_broader_scientific_literature": "This work contributes to the existing evaluation and understanding of long-context LLMs.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The main weakness, in my opinion, is that the main findings of this work are very similar to those of RULER [1].\nTruely, there are some differences: this work tries to minimize the lexical overlap between the needle and the haystack while RULER does not.\nBut I don't see the necessity of this setting, as RULER also finds out that existing long-context LLMs have a limited effective context length.\nMoreover, this work provides some in-depth analysis about long-context reasoning (e.g., the impact of latent hops and CoT reasoning), but I think these analysis can also be done on RULER.\n\nTherefore, I think this work has made a solid but incremental contribution.\n\n[1] RULER: What’s the real context size of your long-context language models?", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yPbBh9cUo0", "reviewer": "Reviewer_q6NK", "summary": "The authors present NoLiMa, which is a long-context benchmark. What differentiates it from previous long-context benchmarks is that there is a small n-gram overlap between the question text and relevant context. The authors describe the benchmark creation process, including filtering steps to remove distractors/conflicting information. They evaluate 12 popular models on their benchmark and find that effective context lengths are much less than previously claimed (i.e., benchmark performance drops notably as length of the haystack increases). The authors also examine the effects of number of latent hops, inversion, needle depth, CoT, and adding distractors. The overall conclusion is that long-context understanding remains a challenge for even the strongest models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we want to thank the reviewer for their thorough and detailed review and constructive feedback.\n\n**[Limited needle set]**: While we agree the set is limited, many comparable works—such as RULER or vanilla NIAH—use even fewer or similarly limited questions.\n\n**[The Eiffel tower painting example]**: As noted in 148–150 (2nd Col) and Appendix A, the keywords W_q and W_n were carefully chosen to ensure unique and true associations, which is also reflected in the near-perfect base scores of top models.\n\n**[CoT Effect]**: Our point is not to dismiss CoT, but to highlight that it’s not a fully robust solution in this setting. While it does improve performance, the gains diminish as context length increases, which remains a core challenge.\n\n**[Other models]**: Gemini 2.0 and Deepseek R1 were released *after or near the submission deadline*. \nGemini 2.0 released on February 5th (6 days after the ddl.) and Deepseek R1 on January 20th (10 days before the ddl.).\nNevertheless, we will include additional results on Deepseek R1 (distill) and GPT o1 and o3-mini models to the CoT section of the camera-ready version. On a challenging NoLiMa subset, **o1, o3-mini, and R1-distilled-70B** all drop below 50% of the base score despite scoring near 100% on the base task, highlighting the difficulty even for reasoning-models. Also, for the main evaluation in Table 3 we will add results on **Gemini 2.0 Flash**.\nMoreover, the evaluation code and dataset will be publicly released to enable testing on future models.\n\n**[In Figure 2 is 0% depth...]**: In Fig. 2(a) and (b), 0% depth marks the start of the sequence. In the last 2K (tokens depth), 0 means the end (Fig. 4) -- We'll clarify this in the camera-ready version.\n\n**[What does \"minimize sensitivity to tokenization problems\" mean]**: Since the answers to the dataset questions are character names, relying on a fixed name could introduce bias in model performance. Some models may split names into more sub-tokens than others (e.g., \"Ve ro nica\" vs. \"Vero nica\"). Rotating character names helps reduce potential biases caused by this tokenization variance.\n\n**[Distractor filtering example and the removal process in \"Conflicting Information Filtering\"]**: We can include some examples in the appendix. Here's one from the stories before filtering:\n\n *“…Yale, but then he (Steve) rebelled, cashed in his med-school scholarship, and went to Paris to study photography…”*\n\nFor the question *“Which character has been to France?*, this passage introduces a conflict with the needle. Steve is a valid answer based on this span, but we want the model under evaluation to select only the needle as the relevant fact. To prevent this, the filtering model flags such conflicts—where additional spans introduce alternative correct answers—and we remove the entire conflicting span, expanding to the nearest sentence boundaries to preserve fluency.\n\n\n**[Is Figure 2 averaged over all models]**: No, Figure 2 is only for LLaMA 3.3 70B. A high-resolution sweep, like Fig 2—since it involves twice the placements—would be too costly on closed models.\n\n**[Potential Inclusions]**: Thank you for the great feedback! We can include the one-hop vs. two-hop breakdown for Table 3 and the unsmoothed Figure 2 in the appendix. Regarding question difficulty, it tends to vary with keywords and model behavior. As for observable patterns, as discussed in the paper, two-hop and inverted questions are generally more challenging.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The overall experimental design seems reasonable", "relation_to_broader_scientific_literature": "This paper is related to analysis of long-context capabilities of LLMs. There has been substantial work trying to increase long-context comprehension and to measure this ability. This work builds on the evaluation side of things.", "essential_references_not_discussed": "I'm not aware of any, but it's possible that is due to a lack of awareness on my part.", "other_strengths_and_weaknesses": "**Strengths**\n- I found the tables and figures very nice - particularly Table 3\n- Nice Rouge comparison to previous works\n- Overall a straightforward idea with solid results\n\n**Weaknesses**\n- More major\n  - The needle set is very limited. There are only 3 total questions (\"Which character has been to...\", \"Which character cannot drink...\", \"Which character cannot eat...\"\n  - The third template in the needle set seems ambiguous. For example, seeing a painting of the Eiffel Tower does not imply one has been to France.\n- More minor\n  - The authors generally seem to discount the usefulness of CoT (e.g., \"The challenge with CoT prompting is that the questions in NOLIMA are straightforward. They are mentioning a singular clue to the answer, meaning they cannot be further decomposed into simpler steps.\") However, CoT seems to *really* help (up to like 20%-range). I can see that even with CoT the task remains hard, but I feel like in interesting question is why CoT works so well, and the authors seem to ignore this.\n  - Adding results for newer models (e.g., Deepseek, Gemini 2) would make the paper stronger\n  - Some parts were a little unclear (maybe I missed something)\n    - In Figure 2 is 0% depth at the end of the sequence or the start? I'm assuming end.\n    - What does \"minimize sensitivity to tokenization problems\" mean on line 145?\n    - Would be nice to see an example or two in the \"Distractor Filtering\" section\n    - How are problematic parts \"removed\" in the \"Conflicting Information Filtering\" section\n    - Is Figure 2 averaged over all models\n  - Potential inclusions\n    - It would be nice to see Table 3 broken down into e.g., one-hop vs two-hop in the appendix\n    - It would be nice to see the unsmoothed Figure 2 in the appendix\n    - Would be interesting to see accuracy broken down by question - are some of the questions just harder than others?", "other_comments_or_suggestions": "Please see \"Other Strengths and Weaknesses\"", "questions_for_authors": "I've recommended \"Accept\" because I think there is enough to this paper to be useful to the community. I don't think that changes during the reviewing process would lead me to change this evaluation. Nevertheless, I hope the authors will consider my feedback in the \"Other Strengths and Weaknesses\" section. I feel like some of the suggested changes/clarifications would be easy to make, and would strengthen the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**: The paper presents a non-trivial long-context benchmark which does not boil down to simple text-matching-based retrieval. This allows the authors to uncover new serious deficiencies in long context performance of language models. The reviewers appreciate the paper's contributions, with only minor concerns that appear addressed in my opinion. Thus I recommend accept.

**More details**:
The reviewers agree that the experiments support the main claims, are comprehensive (`bVpt`) over a diverse set of models (`7az5`), and well-executed (`7az5`). The reviewers also say that the results about the deficiencies of language models on the benchmark are interesting (`bVpt`), in-depth (`wAvn`), solid (`q6NK`) and useful to the community (`q6NK`).

No serious weaknesses were pointed out. There was a point about the benchmark capturing only about hyponym-hypernym relationships, but the authors note that there are commonsense-based relations too. Reviewer `wAvn` suggests that prior work (RULER) is highly similar to this work (or rather that all the analyses here can be done there too), but I agree with the reviewers that RULER involves lexical matches and it's not clear that these analyses can be immediately done there.
Decision: Accept (poster)