Positional Biases Shift as Inputs Approach Context Window Limits
Authors: Blerta Veseli
Julian Chibane
Mariya Toneva
Alexander Koller
Venue: COLM 2025 (2025)
Keywords: Long-context understanding
positional biases
TL;DR: This paper examines how input length, relative to a model’s context window, affects positional biases in LLMs.

Abstract: Large Language Models (LLMs) often struggle to use information across long inputs effectively. 
Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears at the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. 
However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest.
To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model’s context window. 
Our findings reveal that the LiM effect is strongest when inputs occupy up to 50\% of a model’s context window.
Beyond that, the primacy bias weakens, while recency bias remains relatively stable.
This effectively eliminates the LiM effect; instead, we observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input.
Furthermore, our results suggest that successful retrieval is a prerequisite for reasoning in LLMs, and that the observed positional biases in reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.

Published as a conference paper at COLM 2025
Positional Biases Shift as Inputs Approach Context Window Limits
Blerta Veseli', Julian Chibane?, Mariya Toneva,, Alexander Koller'
Saarland Informatics Campus, Saarland University, Germany
2 Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
3 Max Planck Institute for Software Systems, Saarland Informatics Campus, Germany
Abstract
Large Language Models (LLMs) often struggle to use information across long inputs effectively. Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest. To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model's context window. Our findings reveal that the LiM effect is strongest when inputs occupy up to 50% of model's context window. Beyond that, the primacy bias weakens, while recency bias remains relatively stable. This effectively eliminates the LiM effect; instead, we observe distance-based bias, where model performance is better when relevant information is closer to the end of the input. Furthermore, our results suggest that successful retrieval a prerequisite for reasoning in LLMs, and that the observed positional biases reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.
1 Introduction
Large language models (LLMs) are increas- ingly leveraged in real-life scenarios, fac- ing tasks such as search (Kelly et al., 2023;
0.95
0.90
Ziems et al., 2023), coding (Nam et al., 2024; Sun et al., 2025) or summarization (Chang
et al., 2024; Zhang et al., 2024a). Beyond their in-weight knowledge, they are typi- cally provided task-specific information at inference time. To successfully perform such tasks, LLMs require retrieval and rea- soning capabilities to a) find relevant infor- mation and b) infer over the retrieved infor- mation to satisfy the user's queries. Since relevant information for the user query can span large web search results, legal docu- ments, book excerpts, and the like, this calls for LLMs that are capable of successfully retrieving and reasoning over excessively large inputs. While in recent years, the con- text windows of LLMs have grown substan- tially, their ability to fully exploit these tended windows does not scale proportion- ally.
Accuracy
0.80
0.75
6.65
0.60
Relevant Text
Relathve Length
305
832
625 333 25
0.75
Figure 1: The "Lost in the Middle" (LiM) ef- fect describes how models favor information from the beginning (primacy bias) and end (re- cency bias) over the middle an input. Our findings reveal that as inputs reach model's context window size, the primacy bias drops and the LiM effect disappears. We show this effect across models.

---

Published as a conference paper at COLM 2025
Understanding how LLMs make use of long inputs has been active question in NLP. Notably, Liu al. (2024) shed new light on this by identifying positional biases- - specifi- cally, primacy and recency biases where models have improved accuracy of retrieving information positioned at the beginning and end of an input. This leads to the Lost in the Middle (LiM) effect, in which model performance declines when relevant information is located in the middle of an input. This effect, however, has not been consistently observed research using different LLMs has not found these LiM effects (Zhang et al., 2024b). This naturally leads to the question: What conditions underly the Lost in the Middle effect: where, when, and for what type of models and data does it occur?
When examining this question and the previous literature, we observed that prior works not only differ in the models that they evaluated, but also differ drastically in the length of inputs that were leveraged for evaluation. The study by Liu et al. (2024), along with others (Hsieh et al., 2024; Liu et al., 2024), that demonstrated the LiM effect, evaluated sequences of up to 6K tokens, while Zhang et al. (2024b), using approximately 100K-token inputs, could not confirm the LiM effect. This observation hints at one aspect of the favorale conditions, leading to more precise question: How does LLM's input length interact with its positional biases?
We address these research questions by introducing relative input length, Lec, which represents the ratio of an input's length to the model's context window size. Previous positional analyses did not completely max out model's context window but focused on shorter and less expensive inputs (e.g,, less than 50% of the context window for 8K models and under 10% for models with over 32K tokens). Instead, we investigate position effects across the full context window. We place relevant information at different positions in the input text (first, middle, last) and pad it with irrelevant information following Levy et al. (2024). Additionally, we define simple yet concrete metrics measure primacy, recency, and lost-in-the-mite midde biases directly from the data.
In addition to retrieval tasks, we investigate reasoning tasks and observe similar positional biases. To examine whether retrieval is necessary precursor to reasoning in LLMs, we analyze reasoning success probabilities conditioned on successful versus unsuccessful retrieval. Our findings confirm that successful retrieval is indeed prerequisite for effective reasoning, and when retrieval fails, reasoning performance drops significantly. Moreover, we find that positional biases in reasoning largely stem from those in retrieval; conditioning on successful retrieval largely eliminates these biases.
Our approach enables four key findings:
First, in contrast to inconsistent findings in the literature, we find the LiM effect is prominent across all tested models with different context window sizes.
Second, by evaluating LiM with respect to model's maximum context window size, we find that LiM is present for all models when Liel I 0.5. These findings reconcile inconsistent earlier findings: works that tested LiM shorter inputs observed it because the inputs made up less than 50% of the context widow, whereas works that focused on long-conten. benchmarks exceeding 100K tokens did not detect LiM, as the effect tends to vanish or diminish significantly for inputs that occupy larger portion of the context window.
Third, we characterize why the LiM effect vanishes beyond 50% of an LLM's context size. A simple scenario could be that overall performance gets closer to chance levels across all positions as input length increases, as found in Liu et al. (2024), which can mask position-specife differences. However, our findings indicate that the disappearance of the LiM effect is largely driven by strong decrease in favoring initial positions as input length grows (i.e., decrease in primacy bias) - we observe that accuracy for initial positions drops to the same level as, or even below, that of middle positions.
I For example, running our full set of experiments on GPT-4 Turbo (128K) cost approximately 10,000 US dollars, while evaluating an open-source model such as Llama 3.1 70B (128K) requires roughly 24 hours on four H100 GPUs (each with 80GB VRAM) for just one dataset.
2

---

Published as a conference paper at COLM 2025
We find that successful retrieval is crutial precursor to effective reasoning in LLMs; when retrieval fails, reasoning performance drops significantly, Moreover, we show that positional biases in reasoning are largely inherited from those in retrieval.
2 Related Work
Long-Context Benchmarks use Absolute Input Lengths. While LLMs are increasingly equipped with larger context windows, their ability to efficiently process long inputs does not scale proportionally. Recent benchmarks ranging from realistic datasets (Karpinska et al., 2024; An et al., 2024; Zhang et al., 2024b; Bai et al., 2024; Kočiský et al., 2018; Kryscinski et al., 2022; Shaham al., 2023) to synthetic ones (Hsieh et al., 2024; Song et al., 2025; Yuan et al., 2024; Modarressi et al., 2025; Tian et al., 2025; Wang et al., 2024; Kuratov et al., 2024) aim to evaluate long-context capabilities. While most of these benchmarks contain only sequences of predefined lengths, others (Hsieh et al., 2024; Song et al., 2025; Wang et al., 2024) allow adapting input length as needed. Existing benchmarks often evaluate models with different context window sizes using the same absolute input lengths. This creates an imbalance- - for example, a 32K-token input may fully saturate one model's context but occupy only fraction of another's ss - making it hard to compare how well models use their available capacity. To address this, we define input length relative to context window size, enabling more meaningful comparisons and insights into scaling behavior.
Inconsistent Insights about Positional Biases. Models often exhibit positional biases, such as performing better on information at the beginning (primacy) or end (recency) over the middle (Lost in the Middle, LiM) (lvgi et al., 2023; Liu et al., 2024; Kamradt, 2023). But findings on positional biases are inconsistent (Bai et al., 2024; Modarressi et al., 2025; Hsieh et al., 2024; Wang et al., 2024; Tian et al., 2025; Song et al., 2025; Zhang et al., 2024b). Some studies report no LiM effect (Zhang et al., 2024b; Song et al., 2025), while others observe it only at a single input length among many tested (Modarressi et al., 2025). Primacy bias has been consistently observed in tasks where models are presented with a list of candidate answers (Wang et al., 2024; 2023), with Wang et al. (2024) further noting that the effect intensifies with input length for some models. However, these studies differ substantially in input lengths and tasks tested. For example, Modarressi et al. (2025) evaluate tasks that require substantial reliance on in-weight knowledge; Song et al. (2025) use multi-evidence setups that distribute information evenly across the input; and only Zhang al. (2024b) tests input lengths beyond 100K tokens. By contrast, studies on shorter contexts (S 6K tokens) and more controlled synthetic tasks consistently find positional biases (Levy et al., 2024; Liu et al., 2024; Xu et al., 2024). To address this, we normalize input lengths and employ controlled tasks that (1) present relevant information in a single location at a time, and (2) minimize reliance on in-weight knowledge, isolating positional biases under input usage.
Retrieval Effects Reflected in Reasoning Results. To improve long- reasoning, several studies have proposed retrieee-thnn-thing approaches that show substantial gains by encouraging models to first output the relevant information and then the inferred answer. Some approaches rely on prompting strategies (Cattan et al., 2024; Li et al., 2024b;a), similar to CoT (Wei et al., 2022) or analogical prompting (Yasunaga et al., 2024), while others improve performance by for example fine-tuning (Qiu et al., 2025). At the same time, these works find that when models did not output the correct facts first, this results in reasoning errors (Li et al., 2024a; Levy et al., 2024). Although these results indicate that retrieval is important for reasoning, im improvements might stem from decomposing complex single- step task into simpler substeps. Moreover, prior works so far find positional biases for retrieval and reasoning separately and do not show their connection. To address this, we construct retrieval-reasoniig exact information needed for the corresponding reasoning question. This enables assessing reasoning performance under retrieval success or failure. We find that positional biases in reasoning are largely inherited from those in retrieval.
minimal pairs, where each retrieval question targets the
3

---

Published as a conference paper at COLM 2025
Dataset
Premises
Reasoning Question
Retrieval Question
GT
Julie younger than Julian, Samantha younger than Julie.
Samantha younger than Julian?
Is Julie younger than Julian?
MonoRel
Jimmy is slower than Michael. Scott is slower than Jimmy.
Michael slower than Scott?
Is Michael slower than Jimmy?
X
John's kitchen is marble-floored.
Ethan in marble-floored room?
Is John's kitchen marble-floored?
PIR
Ethan in John's kitchen.
Michael is in Anna's great hall. Anna's great hall
Michael white-walled room?
Ts Anna's great hall white-walled?
green-walled.
If X is furry and is good then is tall. Erin is furry. Erin is good.
Erin tall.
Erin is furry and good.
RuleTaker
If X is tall and X is good then X is flat. Fiona is lazy. Fiona is tall.
Fiona is flat.
Fiona is tall and good.
X
Ella found two boxes: Box 0 contains the wire. Box 1 contains the ball. Ella puts the key into Box Ella moves the contents from Box to
After performing all operations, Box contains nothing.
The key is put into Box 1.
Box
Hannah found two boxes: Box contains the wire. Box contains the ball. Hannah moves the contents from Box 0 to Box 1. Hanna puts the hat into Box O.
After performing all operations, Box contains the wire and the ball and the hat.
The wire is put into Box 1.
Table 1: Datasets Overview. Each row illustrates a representative instance from a dataset, consisting of set of premises, corresponding pair of retrieval and reasoning questions, and a ground truth (GT) label - where a checkmark (v) denotes a true-labeled example and a cross (X) denotes false-labeled one. Each retrieval question targets the specific information needed to answer its paired reasoning question. While the retrieval question directly answered by one of the premises, the reasoning question requires combining multiple premises to infer an unstated conclusion. All examples are shortened for brevity. See full examples in Appendix A.1.
3 Putting Positional Biases in Relation to Context Window Size
LLMs exhibit positional biases, such performing better on information at the beginning (primacy) or end (recency) over the middle (Lost in the Middle, LiM) (Ivgi et al., 2023; Liu et al., 2024; Kamradt, 2023). Prior work is a) inconsistent about if and when positional biases occur and b) fails connect positional biases in retrieval with those reasoning. We address b) by constructing dataset of Retrieval-Reasoning Minimal Pairs (Section 3.1), where models can not rely on in-weight knowledge but have to use the text input, and where the same atomic (sentence) information needed for the retrieval and reasoning task. To adress a) we propose to measure model accuracy with respect to relative input length, ie., the length of the input normalised by the model's context window size (Sec. 3.2). We propose concrete metrics to measure primacy, recency, and Lost in the Middle (LiM) biases in Sec. 3.3, and give an overview of evaluated models in Sec. 3.4.
3.1 Datasets: Retrieval-Reasooinng Minimal Pairs
In our analysis, we use four reasoning datasets: MonoRel, PIR, (Simplified) RuleTaker by Levy et al. (2024), and entity tracking dataset (BoxTracker) by Kim & Schuster (2023) (see Table 1). MonoRel and PIR each consist of two premises. MonoRel consists comparative statements expressing relationships among three people (A, B, and C). For example: "A is slower than B, B is slower than C" with the corresponding reasoning question being: "Is A slower than C?". The comparative adjective varies (e.g., slower, younger, taller, faster,

---

Published as a conference paper at COLM 2025
Position of Relevant Text in Input
First
Middle
Last
Relevant Text Within Distractor Text
Relative Input Length Lyel
0.06
0.06
0.06
Accuracy per Lyel for each Position
800p 80O
a0r0.06
M
ace1
M
acc
ace1
Method Overview. Schematic overview of how relevant text (orange) is positioned within distractor text (gray) at three locations (first, middle, last) across varying input lengths (6% to 100% of the context window). For each position P E {First, Middle, Last} and each relative input length Lye (0.6, 0.25,, 0.38, 0.5, 0.75, 1}, we measure accuracy acef.n. illustrating how well the model performs when relevant information placed at position P for a relative input length Lyet-
Figure 2:
0.12,,
etc.). PTR uses spatial premises involving a location L,, a person P, and an attribute A, such as: "Location L possesses a certain attribute A; person is at that location L." It then poses questions like: "Is person P at a location with attribute A?". Rule'Taker includes two factual premises and one logical rule, requiring the model to apply the rule infer a conclusion based on the two facts. BoxTracker consists of sequences of four statements: two that describe the initial world state (e.g,, the contents of each box), and two that describe actions altering that state (e.g., moving or inserting objects). The goal is to track how the contents of a given box change over time.
In addition, we constructed retrieval question in correspondence to each reasoning ques- tion, creating reasoning rettrival minimal pairs. Each retrieval question targets a specific piece of information required for the reasoning process. The answer to the retrieval question is stated explicitly in one of the premises and requires no inference. In summary, we end up with four datasets, each containing 100 instances, with two questions (retrieval and reasoning) per instance.
Following Levy et al. (2024), premises are embedded into short narratives to produce more cohesive and natural-looking paragraphs. The result is referred to as relevant text in the following. MonoRel, PIR, and Rule'laker are already formulated as balanced binary classification tasks for reasoning (50% true-labeled, 50% false-labeled), and we extend this pairs.
balance to the corresponding retrieval questions in our minimal We also adapt the BoxTracker dataset to follow the same format for consistency (see Table 1).
3.2 Method: Varying Input Length and Positions
To investigate positional biases, we systematically vary two factors: the input length and the position of relevant text within that input.
We define relative input length Lrel as proportion of the input text length Linput to a model's context window size Lmaxi
Linput
L
(1)
Lmax
For our experiments we choose Lęt E {0.06, 0.25, 0.38, 0.5, 0.75, aligned with prior work Levy et al. (2024), whose absolute input lengths (500, 1000, 2000, 3000 tokens) map to relative values (0.06, 0.12, 0.25, 0.38) given a 8K-token context window. This allows for more direct comparison with related work. We also tested evenly spaced values (e.g., 0.1, 0.4, 0.6, 0.8, 1); see Figure 11, Appendix A.2.
The relevant text (see section 3.1) is then extended to the target length Linput LTel Lmax by padding with distractor text. We adopt the padding variants proposed by Levy et al.
5

---

Published as a conference paper at COLM 2025
MonoRel
PIR
RuleTaker
BoxTracker
Model
Context Window
RT
RA
RT
RA
RT
RA
RT
RA
Llama-3.1-70B
128K
1.0
1.0
1.0
1.0
0.98
0.65
1.0
0,60
Llama-3.3-70B
128K
1.0
1.0
1.0
1.0
0.98
0.73
1.0
0,88
Llama-3-70B
8K
1.0
0.98
1.0
1.0
0.77
0.63
0.88
0.86
Mistral-Small-24B
32K
1.0
1.0
1.0
1.0
1.0
0.91
0.96
0.78
Qwen-2.5-32B
32K
0.95
0.92
1.0
0.98
0.97
0.89
0.97
0,92
Gemma-2-27B
8K
1.0
1.0
1.0
1.0
0.96
0.88
0.87
0.83
Table 2: Models Overview. Here we show an overview over the models and their accuracy on base instances for reasoning (RA) and retrieval (RT). Models are selected based on their performance on base instances and with the goal to test as many different context window sizes possible.
(2024) but omit the duplicate padding variant, which extends inputs by repeating the same instance multiple times. Since this places the same relevant information at multiple positions simultaneously, it would undermine our position-sensitive analysis. To then analyze positional effects, we place the relevant text at the beginning F, middle M, or end of the input sequence. Figure shows overview of the full approach.
3.3 Metrics: Quantifying Positional Biases
To measure the LiM effect as well as primacy and recency biases, we define the following metrics by making use of the definition of each positional bias. The LiM effect is present when accuracy for middle positions acc is lower than accuracies for first acc and last positions and acc*, Accordingly, we define the metric LiMi (LiM intensity) as follows:
(acc Rcc*)) + (aco* - accM)), if acc > acc and acch > acc 0,
LiMi
(2)
otherwise.
LiMi is designed such that it equals zero if the LiM effect not present. Similarly, we define PriMi (primacy bias intensity) and ReCi (recency bias intensity) as follows:
PriMi acc
-
(3)
ReCi acc"
acc
(4)
Unlike LiMi, PriMi, and ReCi can have negative values. Negative values for these biases indicate a decrease in the bias.
To compute LiMi, PriMi, and ReCi relative to context window size, use the position- specific accuracy values accF acc and acc per relative input length Lye (see Figure last row) to compute one LiMi, one PriMi and one ReCi value per Lyel-
3.4 Models
We select diverse set of open-source LLMs to evaluate positional bias (see Table 2). Our primary selection criteria are: (1) coverage of a wide range of context window sizes (from 8K to 128K), and (2) sufficient performance on non-padded instances (i.e., accuracy over baseline 0.5). The first criterion ensures that findings generalize across varying context window sizes, while the second ensures meaningful signal during the positional bias evaluation. We use only the instruction-tuned versions of these models. We use only instruction-tuned versions of these models.
Our model selection supports comparisons not only across varying context window sizes, but also along two axes: (1) model generation, e.g., Llama-3.1-70B vs. Llama-3.3-70B (both 128K context, same size), (2) model family, e.g., Mistral-Small--3 32K context, similar size). Table summarizes the tested models, their context windows, and
vs. Qwen2.5-32B (both

---

Published as a conference paper at COLM 2025
0.15
0.15
0.15
0.10
0.10
0.10
PriMi
0.08
-0.10
0.4
0.8
10
Relatice Imput Length
Relative Length
Relative Length
Retrieval
Reasoning
Figure 3: Quantitative positional bias evaluation. Visualization positional bias intensities across varying input lengths Leel, expressed as proportion of the model's 's context window. The primacy bias intensity (PriMi) is shown in the left subplot, recency bias intensity (ReCi) in the middle, and LiM intensity (LiMi) the right. Retrieval results are shown in green, and reasoning results in orange. Solid lines indicate the mean positional bias intensity across datasets and models, with shaded areas denoting the standard error of the mean (SEM). We observe that primacy bias decreases, while recency bias increases, as Lrel grows. Furthermore, the LiM effect is most pronounced for Lel I 0.5, peaking around Lel 0.25 for reasoning and Lrel ~v 0.38 for retrieval.
performance non-padded inputs. To assess the effect of model size, include results for Llama-3.1-80 1-BB (128K context) in Appendix A.4, Figure 12, enabling direct comparison to Llama-3.1-70B while keeping the main paper focused on higher-capacity models.
Findings
In this section, we analyze how positional biases evolve input length approaches the model's context window limit. Up to 50% of the context window, we observe a more pronounced Lost in the Middle (LiM) effect, while the primacy bias decreases and the recency bias increases as input length approaches context window size (Section 4.1). Beyond 50% of model's context window, primacy bias fades significantly, introducing distance- based bias favoring information closer to the end of an input. An effect that makes LiM disappear (Section 4.2). Finally, our results indicate that reasoning relies on successful retrieval and that positional biases in reasoning are substantially inherited from retrieval (Section 4.3).
4.1 Positional Biases Emerge Consistently Relative to Context Window Size
In this section, we analyze whether positional bias trends remain consistent across models when input length is normalized by each model's context window (relative input length).
Evaluation. Following Sec. 3, we evaluate the six selected models on our datasets. We observe similar trends actoss datasets and models, we present averaged results in Figure We analyze retrieval and reasoning separately. Figure shows the resulting PriMi, ReCi, and LiMi curves for retrieval and reasoning.
Results. From Figure 3, the LiM effect (LiMi) is strongest when Les S 0.5, across models with varying context window sizes. Beyond this relative input length, the LiM effect declines sharply. This finding helps reconcile earlier research: works that tested LiM on shorter sequences possibly observed because they stayed below the 50% threshold, whereas long- context benchmarks exceeding 100K tokens did not detect it, as the effect tends to vanish or diminish significantly for inputs that occupy larger portion
the context window. The primacy bias (PriMi) remains relatively stable up to L,el R 0.5, after which it begins to drop steeply. In the case of retrieval, it even falls below zero at L = 0.5, indicating that accuracy for hisstpooitioned information becomes lower than for iisiilepositioned information. The recency bias (ReCi) exhibits mirrored pattern: its intensity increases steadily up to Tyel 0.5, then levels off beyond that point. Given LiMi's definition (Eq. 2), its decline with
7

---

Published as a conference paper at COLM 2025
Gemma-22-27
Liama---70
Nist Small-240
Llama-3.3-00
Llama-3.1 7-8
Qwin225-322
04
FISt
madle
Postion of
114
Figure 4: Reason for disappearing LiM. Different models are depicted in columns and tasks in rows, the x-axis represents the relevant information position, y-axis represents task accuracy, Again, we find an accuracy minima in the middle (LiM effect) of shorter inputs (light red curves) Liel S 0.5. We see the LiM effect disappearing when relative input length Lel gets longer (darker res curves) Let 2 0.5. For long sequences (dark red), task accuracy at the end stays consistent (consistent recency bias), while accuracy at the beginning drops significantly (disappearing primacy bias), removing the LiM and concluding in distance- based bias. We show the results per dataset in Appendix A.3.
input length could stem from edge accuracies dropping to match the middle, or both falling below it, a "found in the middle" effect. Prior work favors the first hypothesis: overall accuracy drops with longer inputs (Levy et al., 2024). Yet, trends in primacy and recency suggest a third possibility; only the first position degrades, while the last holds. Since PriMi drops below zero, the fading LiM effect likely actually reflects loss of primacy bias.
In the next section, we analyze why the LiM effect fades as input length approaches the context window limit and how this interacts with other positional biases.
4.2 Positional Biases Shift Toward Distance-Based Bias with Increasing Input Length
In Section 4.1, we observed that LiM effect diminishes beyond 50% of model's context window, while the primacy bias decreases and the recency bias increases as relative input length grows. Here, we show what happens to positional biases after the threshold of 50% and explore why LiM diminishes-offere offering insights into how models utilize their inputs as they approach the limits of their context windows.
Evaluation. We present the results per model, averaged across the datasets in Fig. 4,nin- stead of averaging additionally across models, as in Fig. 3), and perform a more fine-grained result analysis next.
Results. Our findings indicate that the diminishing LiM effect after Let R 0.5 is largely driven by weakening primacy bias accuracy at the initial position drops to match fall below that of the middle. As shown in Figure 4, the typical V/U-shaped LiM curve fades as inputs approach the maximum context size, suggesting that models shift how they utilize input when reaching full capacity. Rather than distinct positional preferences (e.g., primacy or recency), performance increasingly reflects a distance-based bias: relevant information closer to the end yields higher accuracy. In reasoning with Llama-3-70B, Figure 4 shows that while both first and last positions are initially favored, accuracy at the first drops sharply beyond the halfway mark, whereas the last remains stable. At full input length, the model ranks positions as: last > middle > first. This shift illustrates how primacy effects diminish with longer inputs, while later positions retain their advantage. Similar trends hold across other models: all exhibit stronger recency and weaker primacy biases as inputs grow longer. The distance-based bias is especially pronounced in Llama-3-70B, Llama-3.1-70B, and Qwen- 2.5-32B. At full length (Lyy) = 1), performance at middle and last positions even surpasses that at shorter lengths (0.5 < Iyel < 1) in reasoning. Wu et al. (2025) imply connection
8

---

Published as a conference paper at COLM 2025
Model
P(RA RT 0)
P(RA 1
RT
Difference
% Increase
Llama-3.1-70B
0.42
0.76
0.34
81
Llama-3.3-70B
0.35
0.39
111
Llama-3-70B
0.52
0.76
0.24
46
Mistral-Small-24B
0.39
0.72
0.33
85
Qwen-2.5-32B Gemma-2-27B
0.38
0.82
0.44
116
0.50
0.79
0.29
58
Table 3: Retrieval success substantially boosts reasoning accuracy across models. We report the reasoning accuracy P(RA 1) conditioned whether retrieval was successful (RT = 1) or not (RT = 0), alongside the absolute difference and percentage increase in accuracy. Values shown are averaged across all datasets.
between LiM and positional biases in training data. Based on that, we hypothesize that the LiM's prominence at shorter input lengths may result from models being pretrained more heavily short sequences, from such biases being more common in shorter texts. We show that the observed positional bias trends generalize to real-world tasks by replicating our experiments retrieval-augmented QA dataset (Figure 10, Appendix Á.2).
4.3 Reasoning Failures correlate with Positional Biases in Retrieval
Sections 4.1 and 4.2 showed how LiM, primacy, and recency biases evolve input lengths approach a model's context window size. We observed that these positional effects appear in both retrieval and reasoning, following strikingly similar patterns. Retrieval and reasoning are closely connected: reasoning over long inputs typically requires first locating the relevant information, process also described as retrieval. This prerequisite role of retrieval in reasoning, combined with the shared positional patterns shown previously, raises two key questions: (i) How does retrieval success or failure impact reasoning accuracy? and (ii) To what extent are positional biases in reasoning propagated through retrieval?
Evaluation. To investigate the relationship between retrieval RT and reasoning RA, we compare the conditional probabilities of correct reasoning given failed retrieval, P(RA
RT 0), and correct reasoning given successful retrieval, P(RA = RT = 1). successful retrieval is indeed required precursor of successful reasoning, significant and constant performance increase of reasoning in case of sucessful retrieval, RT = 1) P(RA- = RT = 0). Moreover, P(RA 1 RT 1) offers measure of reasoning performance, disentangled from retrieval errors. If positional biases in reasoning are substantially propagated through retrieval, then disentangling from retrieval errors should reduce or eliminate these biases in reasoning. Conversely, reasoning accuracy conditioned on failed retrieval P(RA effects, as it still would inherit position-dependent errors the retrieval stage introduces. We, therefore, use the contrast between these two conditional probabilities to quantify both the impact of retrieval on reasoning accuracy and the extent to which positional biases are propagated through the retrieval stage.
expect
ie. P(RA
RT 0) should exhibit stronger positional
Results. As shown in Table 3, correct retrieval boosts reasoning accuracy by up to 116%, pattern consistent not only on average but also across datasets (Table 4, Appendix A.4). Since all base instances are explicitly controlled to have equal complexity within the same dataset, it is unlikely that the consistent advantage of P(RA corresponding instances being inherently casier. Referring back to question (i) how retrieval success or failure impacts reasoning performance, these results provide strong evidence that retrieval is not merely correlated with, but supports, accurate reasoning in LLMs.
RT = =) stems from
Beyond showing that successful retrieval supports accurate reasoning, we asked (ii) to what extent positional biases in reasoning are propagated through retrieval. Figure revealed strikingly similar positional bias trends in both retrieval and reasoning, with reasoning consistently about 10 percentage points less accurate than retrieval. This pattern is confirmed in Figure 4: as Lel increases, both retrieval and reasoning reduce their primacy bias, while

---

Published as a conference paper at COLM 2025
P[R==1 | RT=0)
PRM=1 [T=1)
0.9
Probability
0.6
0.4
middle
first
middle
last
Position of Relevant Text
Position Relevant Text
Figure 5: Positional biases in reasoning conditioned on retrieval accuracy. Reasoning accuracy P(RA = 1) across positions of relevant text (first, middle, last), conditioned on whether retrivval RT was correct P(RA- = 1 RT = 1) (left) or incorrect P(RA = RT 1) (right). When retrieval fails, models show pronounced drop in reasoning accuracy for middle-positioned information, a manifestation of the LiM effect. In contrast, when retrieval succeeds, models show higher robustness to positional biases, resulting in almost flat curves for some models (Llama 3.1 70B, Mistral).
recency bias remains stable. These parallels raise the question of whether reasoning's positional biases are inherited from retrieval, with the 10% performance gap possibly reflecting reasoning's greater intrinsic difficulty. To examine this, we plot P(RÁ 1| RT 0) (left) and P(RA- 1 RT =} (right) in Figure 5 across different positions of relevant text. When retrieval fails, reasoning accuracy exhibits stronger positional biases most notably a pronounced drop at middle positions (the LiM effect). In contrast, when retrieval succeeds, accuracy remains much more stable across positions, and for some models, such as Mistral-Small-244 and Llama-3.3-70B, the curves are nearly flat, indicating minimal positional biases. These findings suggest that positional biases in reasoning are largely though not entirely propagated through retrieval, as reasoning becomes markedly more robust to position when retrieval succeeds.
Conclusion
We investigated how positional biases behave as input length reaches the context window passitiiooo
size of an LLM. Previous literature is mixed on whether these positional biases exist for diverse LLMs. To address this, we conducted an empirical analysis in which input length is defined relative we developed metrics that enable the measurement of positional biases across these relative input lengths. Our analysis shows that considering positional biases with respect to the rela- tive input length is crucial for identifying trends across models with varying context window sizes. Specifically, we found that the Lost in the Middle (LiM) effect, where performance drops when relevant information appears in the middle of the input, consistently occurs for input lengths up to about 50% of a model's context window. Beyond this threshold, the LiM effect diminishes, primarily because the primacy bias tends to fade as input length increases, counteracting the emergence of the LiM effect. As input length approaches a model's maximum context capacity, we observe an emerging distance-based bias: model performance tends to be higher for information appearing closer to the end of the input se- quence. Moreover, our evaluation using retrieval-reasoning minimal pairs, where a retrieval and reasoning question are posed for the same problem, indicates a strong correlation between failures in retrieval and reasoning. We show that positional biases on reasoning are largely inherited from those on retrieval, demonstrating that retrieval is precursor of long-context reasoning. Overall, our findings provide valuable insights into how language models process long inputs, up to a point where these inputs actually max out model's context capacity, and they offer new evaluation protocols for understanding and improving the performance of long-context models.
model's context window rather than as fixed variable. In addition,
10

---

Published as a conference paper at COLM 2025
Acknowledgments
Funded by the Deutsche forschungsgemeinschaf (DFG, German Research Foundation)
-GRK 2853/1 "Neuroexplicit Models of Language, Vision, and Action" " project number 471607914.
References
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14388-14411, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL /acttatos//cllnnhooooooy wwg. acl long..no..... 776/
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilinguul, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL hatp:::///aaathhloog..oo org/ 20g//222.22o acl- long. 172/.
Arie Cattan, Alon Jacovi, Alex Fabrikant, Jonathan Herzig, Roee Aharoni, Hannah Rashkin, Dror Marcus, Avinatan Hassidim, Yossi Matias, Idan Szpektor, and Avi Caciularu. Can few-shot work in long-context? recycling the context to generate demonstrations, 2024. 2024 13632.
URL
Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The 12th International Conference on Learning Representations, 2024. URL /tatt::///xxv....t wtg/ 2310. 00785.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? In Proceedings of the 1st Conference on Language Modeling (COLM), 2024. URL https://arxiv.. abs////b/////
you
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short- text models. Transactions of the Association for Computational Linguistics, 11:284-299, 2023.
URL httpss://aaaatthllll..oooo hww.ooolgggooooooooooooooooooort wrw.orgg/////o 2023. tac1-1.17/.
G. Kamradt. Needle in a haystack Ilm test. https: / /tti:/giitii wigggthubbb.. www.tttcoooooooc..cooooooototttt ctt/ NeedleInAHaystack, 2023. Accessed: 2025-07-09.
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A "novel" challenge for long-context language models. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17048-17088 Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthoooogy. wwt2224..mmll.....m..
948/.
Dominique Kelly, Yimin Chen, Sarah E. Cornwell, Nicole S. Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L. Rubin. Bing chat: The future of search engines? Proceedings of the Association for Information Science and Technology, 60(1):1007-1009 October 2023. URL https://dl. acm. org/ prg//oi//ee /asi//abb/1 1002/pra2 927.
Najoung Kim and Sebastian Schuster. Entity tracking in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Amnull Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3835-3855, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: // //acl www.aaahhooog.orrgga/. acl acl-long. 213/.
11

---

Published as a conference paper at COLM 2025
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The Narrative reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. URL https:
//aclanthology. orgg/18--12200 1023/.
Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection datasets for long-form narrative summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6536-6556 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL pettps://aclanthonn org/2022. fol2..innn findings-eniennn emmlp. 488/.
Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: lesting the limits of ilms with long context reasoning-in-at haystack. Advances in Neural Information Processing Systems, 37:106519-00655,, 2024. URL https:/ /trps://ariiv orgg/bs//405/ 10149.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15339-15353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: //aclanthology... www.:olooo..oggg/...0/ org/2024 acl long. 818/.
Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, and Yixuan Su. AIr? A retrieve-then-reason framework for long-context question answering, 2024a. URL htt.:::::/aarai..rr//a//////////
org /twg//////////////// 032277.
Yanyang Li, Shuo Liang, Michael Lyu, and Liwei Wang. Making long-context language models better multi-hop reasoners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2462-2475, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL https: www.aaaaaalooyoo... org/ 2024/024...0 acl-long. 135/.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. URL https:
//aclanthology. org.org///// 2024. tacl-1.9/.
Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schuetze. Nolima: Long-context evaluation beyond literal matching. In 42nd International Conference on Machine Learning, 2025. URL https:
waaxkv..org/bbs/2eeee 05167.
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an 11m help with code understanding. In Proceedings of the JEEF/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. URL
wrw. 2207000877
Yifu Qiu, Varun R. Embar, Yizhe Zhang, Navdeep Jaitly, Shay B Cohen, and Benjamin Han. Eliciting in-context retrieval and reasoning for long-contex large language models. In Wanxiang Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 3176-3192, Vienna, Austria, July 2025. Association for Computational Linguistics. URL https:// aclanthology org/ 2025. findings-acl. 165/
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, Singapore, December 2023, Association for Computational Linguistics. URL https://aclaanthooogy..o org/2033 findings emnlp. 536/.
12

---

Published as a conference paper at COLM 2025
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars: A multi-evidence, position- aware, and scalable benchmark for evaluating long-context large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eu- genio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 3753-3763 07633,76,,,00 Abu Dhabi, UAE, January 2025. Association for org/ 201./22255..0 col coling-main. 253/.
Computational Linguistics. URL ht
Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. Source code summarization in the era of large language models. In Proceedings of the 47th International Conference on Software Engineering (ICSE), 2025. URL https: /awxiv..ivvvv./a////v/ org/ a20/ 07959.
Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, and Xiaojiang Liu. Distance between relevant information pieces causes bias in long-context LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 521-533, Vienna, Austria, July 2025. Association for Computational Linguistics. URL https:// aclanthology orgoorg////// 2025. findings-ade ac1.
Blerta Veseli, Simon Razniewski, Sneha Singhania, and Gerhard Weikum. Evaluating language models for knowledge base completion. 20th Extended Semantic Web Conference (ESWC), 2023, 2023. URL http::///aaxxvv..aaabbbbb. 11082
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3712-3724 Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL ht https: http::////aaathhhlogyyyy . org/ 2024. naacl-long. 205/.
Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. Primacy effect of ChatGPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, PP. 108-115, Singapore, December 2023. Association for Computational Linguistics. URL https: /clannhooooyy.. org/2023. emnlp-main 8/.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Öh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-248377. Cur- ran Associates, Inc., 2022. URL httpps: neurei neurips. cc/paper siles/paper/ att.ffeeeeedd555111124444444aaa. abtaaa/ orr.a//444444141111111ooooooooor .df.
Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the emergence of position bias in transformers. In 42nd International Conference Machine Learning, 2025. URL
pettps//aarxxt orgg/bs//500/ 01951
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The 12th International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. URL https://arxiv. org/abs/2310. Dre/abs/2310.03025.
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogi- cal reasoners. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 17019-17045, 2024. URL https: httttt::p//prroeedingg..... iclr cc/paper files//pperr202/ffilee /2044file/ 4990dad2c1
.dd.
13

---

Published as a conference paper at COLM 2025
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. Lv- eval: A balanced long-come contect benchmark with 5 length levels up to 256k, 2024. URL wtt.rv..orr/bbs/222.5555555
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summariza- tion. Transactions of the Association for Computational Linguistics, 12:39-57, 2024a. URL https://aclanthologgy org/2024, tacl-1.3/.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. coBench: Extending long context evaluation beyond 100K tokens. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), PP. 15262-15277, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL https: hawpa//alaannoooo.ooooooooo ort. acl acl-long. 814/.
Noah Ziems, Wenhao Yu, Zhihan Zhang, and Meng Jiang. Large language models are built-in autoregressive search engines. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 2678, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https: / http://acaanntoooo.orr////////
findings-ace acl. 167/.
14

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Full-Length Examples of Base Instances
The datasets MonoRel, PIR, and Simplified RuleTakerr are taken from Levy et al. (2024), while Box is generated using the framework proposed by Kim & Schuster (2023). For each dataset, we present a true-labeled example on the left and a false-labeled example on the right. The reasoning questions are taken from the original datasets, while the corre- sponding retrieval questions were constructed to form bessoning-retrevee Each retrieval question isolates the factual information required to answer the associated reasoning question. While we showed shortened examples in the main paper (see Table 1), we present the full-length instances below ie, premises (in bold) embedded within short narratives.
minimal pairs.
Julle Baker younger than Julian Barton. unchanging the northmm star. it' suth is clear as day, undeniatlle the sky is bluc. This a seallt by that they both live with tact that shapes their their lives thread that runs through the tapestry of thei shared exper onces. it's simple truth strogghftoword thet, but one that repeating: Juree Baker an Barton It's a fast that as able as the course sun across the sky, immutatle as the tuming the earth. younger established and is well known orrong their cirzle of friends and family It's nota mater debate speaulation but simple nuth that Samantha Arnold younger Jukn Baloer Their upe diffeeene ségraficantt lts whoogh to make Juke Baker teel cenoe of responsibllity towards Samontha Aroold
that remains constant,
Dylan Oconnor slower Jason Campbell. This dbserved time and time again. Whather their daiy roudinss Joson Campbel always sooms 1o be one step anoad. it's not Dylan Coomonts particutarly slow, just Jason Compbell exceptons fost When they'r givon a task, Jason Compoctl is always the first one to complete it it's a mater of sAll intelligenon, but Jason Campbell slower than Charles Paters. is escatiisedd time and time again. Whether it's in the momiing when they're gotting evening ng they'n preparing claner
been work
Interactinns their onship
diference
has beon
fast
mendy
always finishos first He's methodical in medicuious, and yes, slower It's not ho's tozy motivation not compettion, course, but wern Charles Peters would be the cleer winnw Shelk jusi fleter: Jowan Campbell stower then Charlee Peners. Itfs as simple as that No matter the task no materr the cay, Charles Peters stooys twlor Dampbell pattem. rlyhh their
Reasoning Question: Samontha Arrri than Julann Barton?
Reasening Questoon Is Chartes Faters slower than Oconwor? Retteval Quesstoo: ls Jason Campbel deowrr than Dylan Ozonnor?
Retreval Duessoon: ls Samaniha Amakd younger than Julle Baker?
Figure 6: MonoRel.
John's grand batiroom blue walled, a 'ruht thas rescmatte with the very founcotion the strusture in which Is housed. The moment one sots with is arrmaiilatlly
John's living room is blue walled, a siatemmnt that reverberates with unwaverrnng certiinty trooughut entine esbablsshmer The
earr ademmnt of dheagr and amiiennce reinforing this reality rut simply an obssevstin madie the occauiinaa ststtrr rather is well-tocmented and unioorsslly aczappied fact that Joherss grand badroom blue codled. The coradebercy with which this fact is presented mataraatee, as every description, every brochure, every
grand
staps into the stainly, it palaabll that Jonn's livng room is blue wallad, with overy element within its confines meticulously aligned to aftmm blue walled. it mere observatio that is livig room blue walled; that disply ingrained in the consciousness of everyone associated with the place. The architects and designers have left no stone untumed in blue walled, sentiment that ochoed overy comner the room. This not accidental feature; it deliberate design choice that has been emphasized repestedy, making it clear that John's livin
John's grand ballnamm insembbly tightights bellboom is brus walled is a detai that has been mattuiouuly plamned and that
grand
ensuring that Johr's livin
bolfrom blue walled, iniegnal iss incaepiion and has bwen fuiiiuuly meisteree trooughroui is sniskems
ensuring that John's
Bridget Burke in John's grand ballroom, a statemont that has been contimed
blue walld.
ipppanecly and has bocome a well addnwwlddged within context pertains to
Matthew Ortiz is in John's living room, fact that has become as much part of the place as the wells and the celling The truth that Matthew Oriz is well-sotublahhed
The reality that Bridget Burke s in John'ss grand ba from not samponary condition
porsistent and enduring state atfirs. tis recognized that nosociation between Bridger Burke and John's grand bollromm is so strong that the menton of Jonn's grand baliroom inmediately brings to mind the fact that Bridget Burke there. constancy twilnoom sometting five new been oboenoed and notedd by all who are prlcy to thks been
is almost redundiant mention it yet is mentioned, again and again, testament to its unassal lable veracity. There a seriain myyhm to the repettionn cadence reinforee the knowledge that Matthew
John's liing room
sitution, where Bridger
Jonn's grand
living roam ovary haration. it a really that has sattled into the consciousnese of all who are aware of the space, a steady drumboat that resorates with the John'' living room
information itis trut we ghtt of mentox Bridge Burk is John's grand hal room.
often that has
the
phrase: Marthew
Reveoning Queetion ls Bridget Burke in blue walled room? Retrivaal Question: is Bridget Burke John's grand balrroom?
Reasoning Quastion: Is Matthew Ontiz a red walled room?
Retrieval Quession: Is John's living room red walled?
Figure 7: PIR.
Julia found two Bowes: Box and Box 1. Box D contains a cross. B contains gitt, Jula puts the caise Box Julia moves the contents Box 0 Box 1,
Slivee found two Bowes: Box and Box 1, Box O contains a git, Box contains a cross. Silvia moves the contents Box to Box Siivia puts the coat into Box
Reasoning Question: After par nerforming all operationn, contains the cross and tha cake and the gift. Retrieval Question: The cake was put into Box 0.
Reasoning Qusetion: After parforming operations, Box contains
coat.
Retrieval Question: The gift vas put into Box
Figure 9: BoxTracker.
15

---

Published as a conference paper at COLM 2025
X good and small then loud.
If tall and X big then is kind.
Dave is small. Dove, being small often finds himself having navigate through a world that is not always accommodsting to hiss size Whether it's strugglin reach
Dave is tall. Dave, who is tail, known for towering height. Standing an impressive heigtt. Dave's tell stature hard mises People often how tall Deve is, is diffningg characleristic his appearance, With his tail frame, Dave stands head and shoulders above the crowd, quite lttrally Whether he warking down the street or sttending balherings, Dava's tsllneess ahwsys noticsable. talinicis has become part of his idently, and is somothing that pesple have come to expect from him. Dave's taliness not just one time observation, it a poseessess
high shaivos being overlooked crowe. Dave's small stature a constant presence in his daily life Desplie the chal lenlgegg he faces, Dave has leameed to sdopt and make the most of his pethis frame. Ha everyday prob wrobiems, using step stools and reaching tools access things that are out of his reach. Dave's smallinees has also given him unique world
expert finding creative solutions
perspactive
him..
Dave is good. Dave known for his goodness. He has always been good. and overyone around him can altest to that. From a young age, Dave dieplayed natural ind
Dave is red. Dave, the person we all know and love, is red. Yos, you heard it right, Dave is red. It's not something out of the ordinary, it's just a simple fact that
towards kindness and compassion. His friends
his loes,
red. It's characteristic
and family have always admirad his ability to sea the good in others and lend a helping hand whenever neaded. Deve's goodness is not just finited to his parsonal relationships workspaace Dave known for hts integrity and his lllignees to go above and beyond to ensure the sucoees of his team, His colleggues often seok his and guidenccs, knowing that will always provide fair and thoughtfuul perspective
that coffines him, that sets him spart from the rest. When you see Dave working down the street, you can't help but notios his rednessi It's become of who he People have to expect to accept it the usual. Dave's redness is not something that surprises anyone anymore
his professional it well. the
his identity
Reasoning Queetion: Dave loud
Reseoning Question: Dave is kind.
Reirieval Quadtion Dave is small and good.
Retricoal Quastion: Dave till and big.
Figure 8: Simplified RuleTaker.
A.2 Additional Experiments
A.2.1 Real-World Task
34
Tedto Qeer2:100
MAN
12:48
5.50
Yadtor SONLK
Figure 10: Retrieval-Augmented QA. To assess whether the trends observed in this paper extend to real-world settings, we constructed a retrieval-based QA dataset. We selected long-tail factual questions (Veseli et al., 2023) that a set of models (Gemma-2-27B, Llama- 3-70B, Mistral) fail to answer without supporting evidence but succeed with access to the corresponding Wikipedia page. This preselection ensures that the answer depends on retrieval rather than memorized knowledge from pre-training, mirroring the controlled con- ditions of our synthetic datasets. Each instance consists of the answer-containing Wikipedia page concatenated with irrelevant ones, with only one document per input providing the necessary information (similar to (Liu et al., 2024)). Unlike our binary tasks, the model here must generate the object of (subject, relation, object) triple (e.g., (France, hasCapital, Paris)) where the object typically spans multiple tokens. As models varied in how consistently they followed the instruction to output only the object, we report whether the ground truth object appears anywhere in the answer instead of the exact match accuracy. We follow the same setup as in the main paper: varying the position of the relevant document (first, middle, last) and input length (5 levels), resulting in 1500 instances (100 questions x 3 positions x 5 lengths). We included Mistral-Small-225 and Llama-3.1-88 to compare them with ntheir ewer and larger versions Mistral-Small-2244 Llama-3.11700 3.--BB).. Performance trends align with the synthetic setting: accuracy declines when relevant information appears earlier in longer inputs, and the LiM effect is most prominent in shorter contexts.
16

---

Published as a conference paper at COLM 2025
A.2.2 Comparing Relative Input Lengths
Ulma-3-778
Gemma-2-277
Ulama
0.4
ACCUPOCY
0.0
03
0.
0.4
6.6 fndd
first
middle
misdle
middle
Position of Relevant Text
Position of Relevant Toxt
Figure 11: Comparison between different Led. We compare the different relative input lengths L.el (0.12, 0.25, 0.38, 0.5, 0.75, 1.0] with Leel (0.1,0.2, 0.4, 0.6, 0.8, 1.0), and ob- serve no difference the overall trends.
A.2.3 Comparing Model Sizes
Mistral-Small-248
Owen22.5 5-326
Ulma-3311 1-88
Uama-3.1-708
03
0.4
0.8
0.6
0.1
trst
middle
Wet first
middle
Ins
test
midde
lnt first
middle
Inst
Position Relevant Text
Position Relevant Text
Figure 12: Comparison across models of different sizes with matched context lengths. On the left, we show Mistral-Small--24B and Qwen2.5-32B (both with 32K-token context windows); on the right, Llama-3.1-8B and Llama-3.1-708 (both with 128K-token windows). Despite the differences in model scale, we observe similar positional trends within each context length setting: performance drops when relevant information appears earlier in the input as input length increases and the LiM is most noticable for shorter input lengths.
A.3 Dataset-specific Visualizations
In this section we show visualizations on model and dataset difference without averaging. Overall, our findings on positional bias are consistent across both models and datasets, even when examining individual conditions. Retrieval tasks show highly stable patterns i.e., Lost in the Middle (LiM) effect for relative input lengths up to 0.5 and steep primacy drops beyond that point. This likely reflects the relatively uniform complexity of retrieval tasks, which does not vary substantially across datasets. Reasoning tasks also display consistent positional trends, but detecting these effects becomes more difficult when model performance approaches chance level, random guessing obscure sensitivity to input position. This issue is most evident in more complex datasets like Simplified RuleTaker and Box.
17

---

Published as a conference paper at COLM 2025
Gommor22220 2-278
Uamo 3-708
Mistri- Small 248
Usima 3.3
Libr 3.
Owen 528
1nz
midd
cds
brt Pm
midde
1:t Erst
miids
1ast frat
middo
featonng
Figure 13: MonoRel.
Gomma- 2 278
Lamo
Misro 248
Ubima 3.3-788
Ubma 3.1 708
Owen2.5 328
04
midde
bet
risde
fri
Relevant
Figure 14: PIR.
Uama
Misto Small
Usms
Tost of Relevent Text
Figure 15: Simplified RuleTaker.
m558
mitde
reldse
fra
Relevant
Figure 16: BoxTracker.
18

---

Published as a conference paper at COLM 2025
A.4
Conditional Probabilties
Model
Dataset
P(RA=1 RT=0)
P(RA=1|RT=1) Difference
Increase (%)
MonoRel
0.56
0.85
0.29
52
PIR
0.57
0.81
0.24
42
Llama-3.1-70B
RuleTaker
0.25
0.70
0.45
180
BoxTracker
0.30
0.66
0.36
120
MonoRel
0.49
0.88
0.39
80
PIR
0.39
0.84
0.45
115
Llama-3.3-70B
RakeTaker
0.15
0.71
0.56
373
BoxTracker
0.36
0.55
0.19
53
MonoRel
0.73
0.91
0.18
25
PIR
0.49
0.94
0.45
92
Llama-3-70B
RuleTaker
0.45
0.64
0.19
42
BoxTracker
0.42
0.53
0.11
MonoRel
0.62
0.92
0.30
49
PIR
0.60
0.87
0.27
Gemma-2-27B
RuleTaker
0.47
0.67
0.20
43
BoxTracker
0.34
0.70
0.36
106
MonoRel
0.49
0.83
0.34
0.69
PIR
0.27
0.76
0.49
181
Mistral-Small-24BB
RuleTaker
0.42
0.65
0.23
55
BoxTracker
0.39
0.62
0.23
MonoRel
0.29
0.84
0.55
189
0.34
0.87
0.53
156
Qwen2.5-32B
Taker
0.61
0.83
0.22
36
BoxTracker
0.29
0.76
0.47
162
Table 4: Reasoning accuracy (RA 1) conditioned on retrieval success (RT versus failure (RT = 0) across datasets and models. In balanced binary tasks (as is the case here), P(RA=1 RT=0) values near 0.5 indicate chance-level performance. Across all models and datasets, reasoning accuracy increases substantially when retrieval succeeds rising by least 25% and up to 373% highlighting the critical role of retrieval in enabling correct reasoning. P(RA
RT 1) consistently being higher might but is unlikely to stem from the corresponding instances being inherently easier. The base instances are artificially generated and therefore control for complexity: all examples within the same dataset follow the same logical structure and reasoning depth, with identical distributions of operations and entities. This supports the interpretation that successful retrieval itself is the key driver of improved reasoning performance.
19


Reviews:
[{"review_id": "YWcp3JkiXp", "reviewer": "Reviewer_rJZo", "summary": "This work investigates the positional biases of LLMs in different relative input lengths. \nFour LLMs with different context window sizes are involved in experiments, including Gemma 2 27B (8K), Llama 3-70B (8K), Mistral-Small-3-24B (32K), and Llama 3.1 8B (128K).  \nA metric, LiMi, is introduced to measure the intensity of the ``Lost in Middle'' (LiM) phenomena of positional biased. \nTwo types of tasks (retrieval and reasoning) on four datasets are evaluated in experiments. \nExperimental results confirm the Lost in Middle effect, and also point out the raise of distance bias as the increase of input length.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work analyzes the problem of positional biases in different input lengths, providing another perspective to this serious issue that is still yet to tackle in current LLMs. \n2. A quantitative approach is proposed to evaluate the Lost in Middle effect. Interesting findings are also shown and discussed.", "reasons_to_reject": "1. The choice of the models in experiments could be more thoughtful. In the current version, the four models are chosen from different sizes (number of parameters), generations, manufacturers (Google, Meta, Mistral), and context window lengths. It could be more analytical to compare the models with some dimensions fixed. For example, it is interesting to know the comparison between Llama 3.1 8B and Llama 3.1 70B as both are the same generation, same claim context window size but different in model sizes and capabilities. It is also interesting to compare Llama 3.1 70B with Llama 3.3 70B. In other words, the chosen and the comparison of the models could be organized in a more systematic and thoughtful manner.", "questions_to_authors": "1. Wrong symbol \"¿\" in the caption of Table 1.", "extra_scores": {}}, {"review_id": "SzavwYW9sy", "reviewer": "Reviewer_db9o", "summary": "The paper conducted a comprehensive analysis that uses an input length relative to a model’s context window size, rather than the length itself as a fixed variable. The findings are that the relative input length plays a crucial role in positional biases. Specifically, it was observed that the LiM effect is prominent when input sequences occupy up to 50% of a model’s context window. Beyond that, a primacy bias is gradually overshadowed by a distance-based bias, effectively eliminating the LiM effect.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducted a comprehensive analysis that uses an input length relative to a model’s context window size, rather than the length itself as a fixed variable. The findings in the paper can reconcile the inconsistency in the findings in the previous work. While it is natural and simple, utilizing an input length relative to a model’s context window size is reasonable and can yield interesting results.", "reasons_to_reject": "While the authors showed only averaged results across datasets and models in Sec. 4.1, from Fig. 5 in Sec. 4.2, different models look having different characteristics in the results. So, discussing the differences among the models would be valuable and fruitful. Furthermore, the results across different datasets might have different characteristics. So, I hope the authors will investigate and discuss the differences, if any. They can be shown in the appendices.", "questions_to_authors": "We sometimes found strange citation formats in the draft. Please try to properly use \\cite rather than \\citet, when necessary.", "extra_scores": {}}, {"review_id": "m4lVS0mWTn", "reviewer": "Reviewer_DcAC", "summary": "This paper investigates positional biases in LLMs, focusing on the \"Lost in the Middle\" effect, where models struggle with middle-positioned information. By defining input length relative to a model’s context window, the authors show LiM dominates when inputs occupy ≤50% of the context window, but diminishes beyond this threshold as primacy bias weakens and a distance-based bias emerges. They also demonstrate a strong correlation between retrieval failures and reasoning errors. These findings reconcile inconsistent prior results and highlight the importance of evaluating positional effects relative to model capacity.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper introduces a novel relative input length framework to systematically analyze positional biases across models with varying context windows.\n* The paper provides comprehensive validation using multiple models and datasets, resolving contradictions in prior research.", "reasons_to_reject": "* The paper is limited to synthetic or modified datasets, raising concerns about generalizability to real-world long-context tasks.\n* The paper also does not establishes causality between retrieval failures and reasoning errors.\n* The paper lacks deeper analysis on the cause and implication of the phenomenon discovered.", "questions_to_authors": "* I am wondering why the relevant information’s position is not evenly spaced in investigation (e.g. 0.2 0.4 0.6, etc.) but in current way shown in Figure 2’s caption and Section 3.2 Line 155\n*  Is current evaluation data representative of long-context reasoning and retrieval tasks? For instance, there may exist even more complex tasks that may involve multiple hops of reasoning (which of course is more difficult)\n* Current model used is mainly based on context length. Will the model family / model scale itself show different trends in results?\n* The 50% threshold mentioned in paper seems is from empirical results, but is there any explanation why the threshold is 50%? How to attribute this phenomenon and what does it imply?\n* LiM here is seen as a relative concept. As from equation 1, the LiMi score is relative to acc^F and acc^L. However, from figure 1, the middle position performance indeed becomes lower and lower as relative input length increases. Could this be really call “Lost in the middle” diminishes? As the performance does not rise up — it’s only the decrease in beginning performance that changes the shape of the curve.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a thorough empirical study of needle-in-the-haystack across different context lengths. Namely, accuracy is studied as a function of both needle position as well as the ratio of the input to model's max context length. The findings are interesting, that lost-in-the-middle is consistent but also gives way to lost-everywhere-except-the-end as inputs reach the max context length. This intuitively makes sense, as data used for long-context extension likely has a strong recency bias as well.

The reviews are consistently positive-leaning. I see no reason to reject this paper, as the measurement and evaluation presented is the most important bit. Fixing the issue this paper finds is likely a question of data.

Comments
1. Typo in Section 4.1 Title: "Dispite"
2. Appendix A is empty
3. Figure 5 Caption: Should this be disappearing primacy bias?
Decision: Accept