Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?
Authors: Jonathan Roberts
Kai Han
Samuel Albanie
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
Long Context
Evaluation
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: As the context limits of Large Language Models (LLMs) increase, the range of
possible applications and downstream functions broadens. In many real-world
tasks, decisions depend on details scattered across collections of often disparate
documents containing mostly irrelevant information. Long-context LLMs appear
well-suited to this form of complex information retrieval and reasoning, which has
traditionally proven costly and time-consuming. However, although the development of longer context models has seen rapid gains in recent years, our understanding of how effectively LLMs use their context has not kept pace. To address
this, we conduct a set of retrieval experiments designed to evaluate the capabilities
of 17 leading LLMs, such as their ability to follow threads of information through
the context window. Strikingly, we find that many models are remarkably thread-
safe: capable of simultaneously following multiple threads without significant loss
in performance. Still, for many models, we find the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as
the context window grows. Our study also highlights the important point that token counts from different tokenizers should not be directly compared—they often
correspond to substantially different numbers of written characters. We release
our code and long context experimental data.

Published as a conference paper at ICLR 2025
NEEDLE THREADING: CAN LLMs FOLLOW THREADS THROUGH NEAR-MILLION-SCALI HAYSTACKS?
Jonathan Roberts'
Kai Han
Samuel Albanie
University of Cambridge
*The University of Hong Kong
https://needle hhreaddigg.ithhu....
ABSTRACT
As the context limits of Large Language Models (LLMs) increase, the range of possible applications and downstream functions broadens. many real-world tasks, decisions depend on details scattered across collections of often disparate documents containing mostly irrelevant information. Long-context LLMs appear well-suited to this form of complex information retrieval and reasoning, which has traditionally proven costly and tome-consuming. However, although the develop- ment of longer context models has seen rapid gains in recent years, under- standing of how effectively LLMs use their context has not kept pace. To address this, we conduct
of retrieval experiments designed evaluate the capabilities of 17 leading L.Mss, such as their ability to follow threads of information through the context window. Strikingly, we find that many models remarkably thread- safe: capable of simultancously following multiple threads without significant loss in performance. Still, for many models, we find the effective context limit is sig- nificantly shorter than the supported context length, with accuracy decreasing as the context window grows. Our study also highlights the important point that to- ken counts from different tokenizers should not be directly compared- they often correspond to substantially different numbers of written characters. We release our code and long context experimental data.
INTRODUCTION
In recent years, LLMs and multimodal LLMs have been shown to possess remarkable ca- pabilities (Bubeck al. 2023) across tasks including software engineering (Hou et al. 2023). geospatial reasoning (Roberts et al. 2023a b). medicine (Wu et al.) (202)), mathe- matical and scientific figure understanding YYue al., 2024) and finance (Liu al. 2023b).
Decliine and Fall of
Gasinl 1.5 Pro
Comp cereiete
of
Shakespeore
Bibte (x,77)
500k
Ron and Feace
272
Journey to best
Moby
357k
Jarba 1.5
Claude 2/1/3/5.5
apr-44 Turba/a/0
1204
An expansion of compute resources, coupled with technical innovations (Liu et al.] 2023a), is enabling contemporary frontier models to be trained on ever increasing volumes of data and longer context limits the maximum number of tokens they can process at once. To contextu- alise the number of tokens leading models can process simultaneously, at just over 300k to- kens'. the classic novel Moby-Dick (Melville) 1851) could fit into the reported 2M token con-
LLaMA 3.1 38/T08/4055
138a
Mistral Lazgu /Setoo
1120
Phs-3 Kediue
Claude
Bein logge
Books
Gemint 1.0
LLaMA 88/708
LLMs
Coateod: Leagth fm1121on 11478 3:1 takoua))
Contextualising context lengths of LLMs and classic literature Books sourced from Project Gutenberg (2024).
Figure 1:
text window of Gemini 1.5 Pro (Reid et al.
2024) almost 5 times. As shown in Fig. [] most books and even book series contain fewer tokens
than the longest model context windows.
Using the 1.L.M-3.11 tokenizer (Dueey et al. Emails: jdr533 3am uk, ka kaihanxhhku. nk, samuel a baniaaiee academi c@gma com
2024

---

Published as a conference paper at ICLR 2025
A longer context offers potential benefits performance, for example, many-shot in-context learn- ing (Agarwal et al. 2024) in which hundreds or thousands of examples are appended to the model input. Another consequence is the wider range of possible applications and attainable downstream tasks. In particular, with a longer context, models can better perform real-world scenarios, such as legal document retrieval, academic research, understanding tax frameworks, and solving crimes and puzzles. In these cases, decisions are made and conclusions drawn based on large quantities of information distributed across many sources and formats. The ability to hold information on the scale of multiple full-length novels or hundreds of academic papers and documents in-context, makes models well-suited to this type of task.
The rate of development of longer context models has outpaced the understanding of how well they use their long context and can navigate Moreover, current benchmarks are considered inadeuutte and lacking (Bai et al. 2023 Zhang et al. 2024). Specifically, we identify three limitations of the extant literature related to long context understanding. (1) Performance saturation: Building on the 'needle in a haystack' test (Kamradt 2023), numerous benchmarks focus simple retrieval-based experiments. Frontier models can perform these tasks excellently, achieving perfect or near-pert pear-perfect scores (Reid al. 2024. Anthropic 2024a Dubey al. 2024). leaving little headroom and useful insights to be gained. (2) Limited context length: In most long-context benchmarks, evaluations limited to sub- 100k contexts, falling short of the context limit of frontier LLMS by an order of magnitude. (3) Lack of granular takeaways: Due to the use of real documents or tendency to aggregate multiple tasks into an overall metric in most works, isolating specific trends is challenging other than the macro-trend that performance degrades as context length increases.
As such, there is opportunity for a set of challenging experiments, suitable reach the limits of fron- tier models. To this end, we design and conduct a series of retrieval- long context experiments of varying degrees of difficulty, across a range of context sizes up to 900k (Gemini 1.5) tokens. Our investigation includes novel needle threading tasks, which entail following a thread of linked pieces of information across different parts of the context and retrieving the final value. We also explore a more difficult multi-threading variation, which requires tracking multiple threads simultaneously, and assess whether the LLMs are thread-safe. We evaluate a suite of 17 LLMs on these tasks and observe performance decreases in longer contexts. Coupled with the finding that tokenization differs significantly between models, we introduce a task-specific effective context limit metric.
In summary, core contributions are: (1) We introduce challenging multi-step threading and multi- turii-thrediing retrieval tasks and evaluate 17 leading LLMs. (2) For simple needle retrieval tasks, show that increased context length reduces performance, while increasing the number of needles retrieved concurrently has relatively limited impact on stronger models.
(3) We show that leading LLMS are remarkably thread-safe- their thread following performance is largely unaffected by con- current queries. (4) We compare tokenizers, highlighting significant differences in token counting. (5) We propose a task-specific and configurable model-agnostic effective context limit metric.
2 RELATED WORK
Evaluation of the long context capabilities of large language models is recent yet burgeoning field of research. Numerous works focus on evaluating I. L.MSs at long-document understanding tasks,
such as question answering (An et al. 2023 Bai et al. 2023 Dong al. [023] Kuratov al.. 2024]
Shaham et al.) 2023 Li et al.) 2023 to decrease with increasing context length. Related tasks involve the summarisation and citation of insights across documents (Laban et al. 2024) and claim verification (Karpinska et al. (0024). which proves challenging for frontier models. While these benchmarks provide robust evaluations across variety of tasks, they typically focus on smaller context lengths. with most including only limited explorations beyond 100k. Although there are benefits to realism by using real documents for these tasks, there are drawbacks. Specifically, timely annotation and curation are required, making it difficult to decompose performance as a function of variables such as context depth and length.
Yuan et al.] 2024). in which performance is generally found
Other works focus more abstract retrieval tasks (e.g., Kamradt (2023), ), allowing clearer take- aways at the cost of real-world relevance. An influential work is Liu et al. (2024), which empirically demonstrated that the position of relevant information within an LLM's 's context significantly impacts
performance, with the best performances attained when information is at the beginning or end of the
context. Similar behaviour is reported in some subsequent works (Xu et al. 2023 An et al. 2024
2

---

Published as a conference paper at ICLR 2025
Dong al. 2023 Hsich et al. 2024b Laban al. 2024) (and in some cases (Levy al. 2024) but others have failed to replicate the findings (Zhang et al. 2024, Song et al. 2024). Song et al. (2024) introduces retrieval paradigm involving the accumulation of information throughout the context window, along with a more challenging variant that includes misleading information. Despite re- vealing interesting behaviour, there is limited headroom for frontier models on these tasks. Some recent related works include more challenging retrieval experiments, involving multiple steps. One example is the Ancestral Trace Challenge (Li et al. 2024), which proves challenging but evaluated to relatively modest context lengths (up to 2k tokens). Another example is Variable Tracking (Hsieh) al. 2024a). however, results on these tasks are included as part of wider set of experiments rather than being analysed in detail separately. We evaluate our difficult needle threading tasks to context lengths up to 630k tokens and comprehensively ablate and decompose the results.
3 TASKS
Taking inspiration from prior works (Liu al. 2024 Hsich al. 2024a Zhang al., 2024). we fo- cus our experimentation on abstract tasks containing synthetically generated data. By using synthetic data, (I) we avoid potentially expensive question-ant question-and-ansee and-answer curation and annotation, (2) ensure high-quality and noise-free data, and (3) we gain fine-grained control over the sequence length and other task parameters, allowing direct influence on difficulty. The abstract setting removes almost all natural language semantics, enabling the derivation of insights more closely linked to the parameters of the context window. We use string-serialised JSON objects containing key-value pairs of random UUIDs for our core experiments. Each UUID is a unique 32-character, 128-bit value string. The prompts used for each task follow this general structure:
<Task description>> ("9a159850-22) 20g.
3ode8eca9-8fd44 70044-0000.0bf166--447774444445
<Output instructions">
Key(s): -4664b247008744-3be------- 1190882-1191//dd0e
Corresponding value(s):
In the following subsections, we outline our long-conted understanding tasks. To complement the textual descriptions, we also include schematic of each task in Fig[]] We conduct each experiment on a set of 'haystacks' of different sequence lengths, m, where each haystack (H) is a set of key- {1,2,3, ..
value pairs: H {(K,, Vo)
Single Needle. In this simple, motivating task the goal is to provide the corresponding value (V₂) to single specified key (K,). For each haystack, we place needles at a fixed set of placement depths.
Multiple Needles. Building on the previous task, the goal of this task is to provide all the corre- sponding values specified set of between 2 and 25 keys. We consider two different placement methods: (1) Random keys are randomly sampled (without replacement).
(2) Clustered after randomly sampling an initial key, all subsequent keys are sampled adjacently (motivated by the observation that informative cues for a given query often cluster together in real world applications).
Conditional Needles. Rather than providing specific keys, the goal of this task is retrieve the values corresponding to all keys matching specified criteria. In this case, we modify target keys by replacing a randomly selected character with a special character such as *g* or '&'. The expected values are those corresponding to keys containing the special character.
Threading. We define Threading Task by initially selecting a subset of n indices j = ji.,j,...,,J) from H, where ju E {1,2,.., m). We then iterate over the indices j for i > 1, replacing in H, Kin + Vi to form a thread. Given single start key (K,)), ) the end goal is to find the value at the end of the thread (V,.). We evaluate thread lengths up to n=25 steps and experiment with different thread directions: (i) Forward where the position of each subsequent pair in the thread occurs later in H (i.e., ju < j2 ... < <n),,, (ii) Backward where the positions of subsequent pairs occurs earlier in (i.e., jn > j2 > >> jn) and (iii) Random - where each subsequent pair in the thread occur any available position in H. regardless of direction.
3

---

Published as a conference paper at ICLR 2025
Multi-Threading. For this task, modify H to include more than thread. The goal is to determine the final value of each thread, given only the starting keys. We investigate different combinations of thread lengths, number of threads and thread direction.
Branched Threading. In this variation, we add branching to the threads. Specifically, at each index in the thread (except the first key), modify 2 more keys (number based the specified branch- ing factor, b) to equal one the previous values. At each step, there are b possible continuations, only one of which continues. The overall goal determine the final value of the longest thread.
EXPERIMENTS
Baselines. To build comprehensive characterisation of the capabilities of current frontier long con- text models, we evaluated set of 17 LLMs on our challenging long context retrieval experiments. Since the majority of frontier long context models are closed-source, we centre our evaluation on closed-source baselines. However, we also evaluate subset of open-source models as comparison. Where possible, we focus on chat or instruction-uns -unnedd variants each L.M as their greater tendency to follow instructions enables a broader range of tasks and eases automatic evaluation. Specifically, we evaluate models from the closed-source GPT-4 (OpenAl 2023 2024), Gemini 1.0 (Gemini Team al. 2023) and 1.5 RReid et al. 2024), Claude 3 (Anthropic 2024a) and 3.5 (Anthropic) 2024b),
Start by
Valee
Cunfiguration
Single
Multiple
Needles
Keys, k
Vales,
Randor, and
Chomadd #
Randon, u=l.
Choterod, neq,
TCM
Threading
Moldi threadlng
Branched threading
Furward,
L=3
Randon,
threods
RP-1, fad vinc, 1-3
aF-3, L-2
6-9
Figure 2: Schematics for long-context key-value retrieval tasks. See for descriptions.

---

Published as a conference paper at ICLR 2025
and Reka (Oraazabal et al. 2024) series and the open- Jamba 1.5 (Team al. 2024). Mistral
A1 2024a), and LLaMA 3.1 (Dubeyy et al. 2024) model series. Reported context lengths for each model are shown in Fig. [
Prompting, We used simple prompting strategy throughout our experimentation that consisted of single basic user prompt containing the question and output format instructions for each task. In keeping with prior works (Roberts et al., [2024a] by OpenAl 2024b). we do not modify the system prompt or tailor the prompt for each model. With the exception of providing examples of the desired output format, we do not use few-shot examples or explicitly encourage reasoning. We include the specific prompts used in each task in the
Inference. All inference was carried out in zero-shot setting. To aid reproducibility, we set model iyperparameters that encourage as deterministic generation possible. Concretely, we use greedy search decoding strategies in which the most probable token is selected from the model vocabulary V each step, conditional on the preceding tokens i.e., Wn+1 arg max P(w) W1, u2,,...; w,,) We achieve this by specifying random seeds and setting the temperature parameter to zero. We evaluate the LLMS via the VertexAl (Google 2024) {Gemini, Claude, Jamba, LLaMA 3.1, and Mistral), OpenAl (OpenAl 2024a) (GPT), and Reka (AI 2024b) {Reka) APIs. We aimed to evaluate each model as close to their context limits as possible, however, due to API restrictions this was not always feasible. More inference details can be found in the
Evaluation. Following recent work (Roberts et al. 2024b), we use a strong LLM (Gemini 1.5 Flash) to parse the output from the evaluated LLMs into a specific format before evaluation via exact matching with the expected answer. As most models exhibit strong output following abilities, this LLM--ased reformatting and evaluation has been demonstrated to correlate strongly with other evaluation measures in (Roberts et al.) 2024a). For most models, this was only necessary for tasks requiring multiple values as the answer. For tasks requiring k values as answers, we only evaluate the top k answers provided by the models, any other additional answers were disregarded.
Tokenization. Context limits are typically reported in tokens and models are com- pared though this a consistent, model- agnostic metric. However, although minor variations tokenization schemes might be expected across tokenizers, our prelim- inary experiments revealed significant dif- ferences, outlined in Fig. 3 A UUID pair is represented by 50 tokens by GPT- 4o while Gemini 1.5 uses 75. Over longer contexts this difference is notable: Gem- ini 1.5 Flash's reported context limit of IM tokens is equivalent to 700k GPT-4o to- kens. References token counts through- out this section refer to text tokenized us- ing the LLaMA 3. tokenizer.
Bamker EETT gaine
1330
100
3839
+000> sobwen
830
0.06
420
0-
0.2
0.6
0.5
0.0
1.0
1.2
1.4
1.2
1.0
Facber characters (esllions)
Figure Tokenization, LLMs tokenize UUIDs sig-
nificantly different granularities.
In the following subsections, we report the results on the tasks outlined in } Experi- ments were carried out on haystacks of 12 different sizes ranging from Ik to 630k tokens (measured in LLaMA I tokens). For most models, we repeat each experiment on different sets of haystacks and report the average performance, however, in some cases, only repeat was feasible due to rate limit restrictions. More details, full results, and branched threading results can be found the.
4.1 SINGLE NEEDLE
As a motivating task, we evaluate the ability of the models to accurately retrieve values correspond- ing to keys at fixed depths in 10% increments in the haystacks. We show heatmaps for subset of the models in Fig. and overall depth-averaged model performance on this task the At shorter contexts, the models perform this simple task well. However, in most cases, the retrieval accuracy decreases for longer context lengths. This suggests that while the models can perform inference inputs up their context limits, most have smaller "effective" limit from which they
5

---

Published as a conference paper at ICLR 2025
OPT-4o Claude
Ousioi Flash
100
80
60
Longth
MalA
Figure 4: Single Needle heatmaps. For most models, the effective context length is less than the context limit, At longer contexts, retrieval precision decreases towards the middle of the context.
GPT-4o. Cluntared
SP7-46. Randon
Gwnint 1.5 Flach, Clustered
Gentnt 1.5 Flach. Randon
Suatez of Hondles
15
Length (1k Llals
Figure 5: Multiple Needles heatmaps. Context length has substantially greater effect on perfor-
mance than needle placement positions or the number of needles.
Meabbe Saudles
Context Length (1k 1AMM 3.1 tokeas)
Figure 6: Conditional Needles heatmaps. Needles prove easier retrieve when clustered.
can accurately extract information. Notable exceptions are GPT-4o and Jamba-1.5 Large, which attain perfect scores throughout. From the heatmaps, it is apparent that for the majority of models, accuracy decreases towards the middle of the context, supporting the findings of et al. (024).
4.2 MULTIPLE NEEDLES
Building the previous task, evaluate the capability simultaneously retrieve values corre- sponding to 11.2:34,.510.15220.22 imput keys from the haystacks. We report overall results aver- aged over all numbers of needles for each context size Fig. and heatmaps for selected models in Fig. [ which show decomposition of performance as a function of the number of needles and needle placement (randomly placed clustered). Considering the overall result, we observe a sim- ilar macro-average trend as in the single needle task, where performance decreases at larger context sizes. However, in this case, owing the higher degree of difficulty the performance drop-off steeper, with several models' accuracy reduced to below 20% as their context limits are approached. This faster performance degradation suggests the effective context limits for this task are even shorter than when retrieving single needle. As before, GPT-4o achieves near- near-perfect score. The heatmaps for Gemini 1.5 Flash show retrieval accuracy is unaffected by the relative placement the nee- dles. Furthermore, context length has far larger impact performance than the number of needles which has very limited impact on performance for the stronger models.
4.3 CONDITIONAL NEEDLES
Sharing similar structure the multiple needles tasks, the conditional needles task assesses the ability to retrieve the values corresponding to 1,2,3,4,55100155,,,,,55, unspecified input keys that meet the condition of containing the ** character. Compared the multiple needles task, similar

---

Published as a conference paper at ICLR 2025
100
100
Acsunne
Accason
Clasde
Sccoet
107-44 0133
1ei111 1-0 Fleso
90
1082 1.5 1arge
4L444 3-1 4050
1J.04 5-1 1b
200
2003
0022
103
200
300
5003
Floss
11aMs 3-1 tabwna)
12a4s
-usteed Necs
Figure 7: Overall accuracy for Multiple Needles (left) and Conditional Needles (right). Shaded regions show 95% confidence intervals.
overall trend is observed. Fig. shows an arguably steeper initial performance decrease at shorter context lengths followed by a shallower decline towards the longer context lengths, resulting in lower overall scores. More differences between the tasks can be seen in the heatmaps in Fig. 6 One clear observation is that the placement of the conditional needles directly impacts the ability the models to retrieve the corresponding values: retrieval accuracy is higher when the relevant key-value pairs are clustered rather than randomly placed. Also, when randomly placed, performance noticeably decreases when the number of needles increases. We found similar model performance with different conditional characters, though was notably lower for
4.4 THREADING
Figure 8: Overall accuracy for Threading (left) and Multi-threading (right). Shaded regions
show 95% confidence intervals.
GPT-4o wint
GPT-66
Gendnt 1.5 Flash
Clande 3.5 Sonnet
Forwaard
Forward
Forward
Forword
20
Backward
Backvard
Backward
Rickvard
Langth
Figure 9: Threading. For most models, forward-travellin threads are easier to follow.
Having demonstrated the models' capabilities perform single-step retrieval-basee tasks (at least at shorter context lengths), we now move towards challenging multi-step ceasoning-basee retrieval. Concretely, at each context size, we test how accurately each model can retrieve the final value from threads of length: (2.3.4.5.6.7788,9, 10, Threading introduces directionality the relative position in the context window subsequent pieces of the thread. We repeat each evaluation on threads going in forward, backward and random directions (see Fig. 2 Overall results are displayed in Fig. S and example heatmaps are shown in Fig. 9 Average accuracies are significantly lower for this task reflecting the added difficulty of following the thread through the context. For many models, e.g., Gemini 1.5 Flash (darker red) and Claude Haiku (darker blue), the accuracy plateaus nearly zero at higher context lengths. The heatmaps reveal two clear trends. Firstly, performance

---

Published as a conference paper at ICLR 2025
decreases both with increasing context length and thread length. Second, the direction of the thread matters. Except for Claude 3.5 Sonnet, all models achieve much better accuracies on threads moving forward through the context compared to threads travelling backwards.
4.5 MULTI-THREADEE
We extend the threading task by adding extra threads for the models to simultaneously retrieve final values from. We evaluate on thread lengths of (2.3.4.5,10) for [2,3,4,5] separate threads and repeat for 'forwards', 'baccwards', 'random directions', and "all random' directions. The averaged accuracies for each context size are shown in Fig. S The lack of clear differences between the heatmaps for vs 5 threads suggests that within the experimental range of thread lengths, the models are thread-safe and performance is not significantly degraded by simultaneously following additional threads. This is further illustrated in Fig. I0 in which Claude 3.5 Sonnet shows no performance degradation up to 25 threads and GPT-4o and Gemini 1.5 Pro show gradual decline.
Percentage ot threads Beisleeee
40
GPT-40
20
Clanda
Nunber of Threade
Figure 10: Frontier LLMs are thread-safe. Each point represents an average over 10 repeats retrieving randomly directed threads with length of 3 in 20k LLaMA 3.1 token haystack.
4.6 AGGREGATING HAYSTACK METRICS
To directly compare the overall performance of the models, take equally weighted average over the Single Needle, Multiple Needles, Conditional Needles, Threading and Multi-threading task scores. The results are presented Tab. [T We find that the best model depends on the context size: for the smallest contexts GPT-4o is best, at the longer contexts Gemini 1.5 Pro is superior, and Claude 3.5 Sonnet the best performing from 2.5 to 32k. Across the board, the closed-source models outperform the open-source models.
Model
Accuracy (%)
1.2k
2.5k
5k
32k
64k
128k
180k
250k
500k
630k
Gemini
87.7
81.1
76.7
78.6
74.8
72.7
69.2
Gemini Flash
80.7
73.3
70.1
67.5
65.7
53.9
53.3
46.1
21.3
19.7
Jamba 1.5 Large
70.8
63.5
60.2
57.5
47.1
43.9
43.4
40.4
Jamba 1.5 Mini
55.4
50.4
44.8
39.0
33.3
30.4
27.2
20.4
Claude 3.5 Sonnet
91.5
88.7
84,9
80,9
79.4
75.9
63.2
50.6
48.0
Claude 3 Sonnet
82.0
73.7
67.9
$2.0
44.6
44.7
39.9
38.8
37.6
Claude 3 Haiku
71.8
65.7
62.8
59.3
53.3
50.3
43.0
37.2
37.4
GPT-4o
93.2
86.1
81.6
74.1
71.9
68.6
64.9
60.9
GPT-4o mini
75.7
67.9
64.7
61.8
58.3
56.3
51.3
42.9
Reka Core
59.8
53.8
17.0
33.5
29.6
27.0
24.9
Reka Flash
58.8
43.5
31.2
29.8
26.8
25.4
20.4
14.1
LLaMA 3.1 8b
54.9
49.8
45.3
40.9
33.6
29.0
26.0
13.7
LLaMA 3.1 70b
78.1
68.9
66.0
61.9
57.1
52.5
38.5
4.5
LLaMA 3.1 405b
76.7
77.1
70.5
69.8
62.8
55.2
39.3
19.6
Gemini 1.0 Pro
59.7
46.9
42.5
40.9
27.8
Table Overall results averaged across Single Needle, Multiple Needles, Conditional Needles, Threading and Multi-threading tasks. The highest scoring models at each context size is bold.
4.7 EFFECTIVE CONTEXT LENGTH
The observed macro-trend of reduced performance at longer context windows implies the models' ability to fully use their context window weakens as it grows. short, there a context size beyond which the models cannot effectively reason over and retrieve from. We propose effective
8

---

Published as a conference paper at ICLR 2025
630
1000
1000
Claso J.D. S.canel
Jaste_ 1-0 b..sage
Glass Ponse:
C1098 3- 1.4.550
Sua. Claartere (tlusands)
500
(thoueson LiatA 50000))
Saa. Charactes (Chousanda)
(P7-taua10))
(1 1.1.
-800
(Mbroseddd L1.02 3.1 mabon))
800
900
Oenaes 0-2co
Deks, Cocw
Gensed 2,700
600
Contert Lengta
800
Lengeh
600
.50
400
-20
2800
180
200
:28
200
64
64
32
52
10 15
20
Degth 00)
Meedlex
Figure 11: Contour plots showing 'effective context length frontiers' for the Single Needle (left) and Multiple Needles (right) tasks. Raw contours were used for the determination of the effective context lengths in Tab. 2 To improve visual clarity, the contours displayed have been smoothed using a Gaussian filter with o=1.5.
Model
Context Limit (Ik chars)
Effective Context Size (Ik chars) (proportion of limit, %
Single Ncedle
Multiple Needles
Conditional Needles
Threading
Multi-threading @+5 steps
@ 10 ncedles
@ needles
@5 steps
Gemini 1.5 Pro
2472
315 (139)
430 (17%)
220 (9%)
(00%)
0 (0%)
Gemini 1.5 Flash
1236
132 (115)
294 (24%)
44 (4%)
0 (0%%
0 (0%%
Jamba 1.5 Large
295
295 (100%)
295 (100%)
10 (3%)
0 (0%%
0 (0%)
Jamba 1.5 Mini
295
87 (29%)
17 (6%)
10 (3%)
0 (0%%
0 (0%%
Claude 3.5 Sonnet
309
169 (55%)
309 (100%)
121 (39%)
4(1%)
3 (1
Claude 3 Sonnet
309
309 (100%)
309 (100%)
[4 (5%)
0 (0%%
0 (0%%
Claude Haiku
309
87 (28%))
201 (6558)
((0%)
0 ((0%))
GPT-4o
214
214 (100%) )
214 (100%)
14 (7%)
(3%%
3 (1 (1.))
GPT
214
120 (569))
(825)
(20%))
(0%)
(0%)
Reka Core
214
5 (2%)
5 (296)
3 (19)
0 (0%)
0 (0%)
Reka
214
(25)
(4%)
(1
(0%)
(0%)
LLaMA 3.1 8b
214
(7%)
22 (10%)
34 (16%)
0 (0%)
0 (0%)
LLaMA 3.1 706
214
22 (10%)
114 (53%)
(16%)
(0%)
0 (0%)
LLaMA 3.1 405b
214
138 (6415)
124 (581E)
60 (28%))
0 (0%)
3 (1 11)
Gemini 1.0 Pro
24 (63%))
31 (82%)
0 (0%)
(0%)
0 (0%)
Table 2: Effective context lengths. @ X indicates the effective limit on the task when the named
parameter equals X.
context length metric for each task that leverages the granularity of our experiments rather than simply estimating average. For each task, we create dense grid of points along the two key experimental variables (see axes of heatmaps) and interpolate the average accuracy at each point. We then determine a contour corresponding to a threshold accuracy level (taken here to be 75%). This contour represents the effective frontier, beyond which retrieval is unreliable. For the Single Needle task, we conservatively take the minimum value of the contour to provide metric that is independent of context position. For the other tasks we take the corresponding contour value at specific point on the x-axis, for example, where Num. Needles = 10 or Thread Length = 5. Example contour plots are shown in Fig. 11 Tab. coontans the computed effective context length metrics for each task. Given the discrepancies between tokenizers, we base our metric on the model-agnostic number of characters in the input rather than token count. The results show that most models have an effective context length far less than their advertised context limit.
4.8 NATURAL LANGUAGE ABLATION
To supplement the preceding experiments. we conduct natural language experiments that serve as closer analogues to real-world applications. Initially, we take sentences from The History of the De- cline and Fall of the Roman Empire, by Edward Gibbon (see Fig as a proxy for the UUID pairs in the abstract tasks. We prompt ol -preview (OpenAl) 2024) to generate list of plausible yet fictional Roman events (i.e., not included in the text). Using these events, we construct "threads" of linked sentences of the form '... Event A and then Event B."..., 'Event B and then Event C.,... and replace sentences in the text with them. We evaluate the threading task in this setting on haystacks from 1k to 630k token context lengths with threads of 2-25 steps (see Fig. 12 As in the abstract set- ting (Fig. D. following threads in the natural language text proves challenging for the models, with similar poorer performance observed at longer contexts. The preference towards forward-travellinn

---

Published as a conference paper at ICLR 2025
GPT-4o
Backward
Backvard
Backward
Sackvard
820
E10
Context Length (au LlaHk 3.1 toksus)
Figure 12: Threading through natural text showing a clear preference for forward moving threads.
GPT-4o
Gemini 1.5 Pzo
40
Threada
Figure 13: Multi-threading through natural text. Each point represents average over 5 repeats retrieving randomly directed threads with length of 3 ~00 LLaMA 3.1 token haystack.
threads is apparent in this setting, with almost backward- travelling threads correctly trieved. We also conduct multi-threading experiments using this approach (this time with additional simultaneous threads) and present results in Fig- [3] Each point represents average over 5 repeats retrieving randomly directed threads with length the threading experiments for which the results and insights are largely the same across the abstract and natural text settings - this multi-threading task in the natural language setting proved much more challenging for the models. Moreover, we find the task be challenging when retrieving multiple threads that are all forward or all randomly directed. Thus, the multi multi-threading results are nuanced with strong performance in the synthetic setting and weaker performance the natural text setting.
mackward-travelling
in 20k LLaMA 3.1 token haystacks. Unlike
CONCLUSIONS
We introduce set of retrieval experiments covering simple single-needle retrieval, more difficult muediple-needle and conditional-needu retrieval and finally, challenging needle threading and multi- threading retrieval. All experiments the same distribution as the relevant text. By curating the haystacks synthetically, we have granu- lar control across specific independent variables enabling performance and extract the following interesting takeaways after evaluating 17 LLMs on our tasks. (i) At long context lengths, the retrieval precision of frontier LLMs decreases towards the middle of the context; (ii) Clustering needles has little effect when tasked with retrieving specific needles but noticeably increases performance when retrieving all needles meeting condition; (iii) Most LLMs achieve higher accuracies when retrieving threads moving forwards through the context ver- sus backward directed threads; (iv) The evaluated I.LMs show proficiency keeping track of mul- tiple threads simultaneously. Thus, we go further than most prior long context benchmarks, which provide only coarse, macro-trends. After revealing notable differences between tokenizers and ob- serving poorer performances on larger haystacks, particular, we propose a contour- based task sss--fpcciiic metric that is independent of tokenization. For given task setting, the metric defines the maximum context size which model effectively perform. We release our code and tasks for the community to use and we hope that our findings encourage further long context understanding research.
carried out on haystacks where the distractor text is from
decompose key variables affecting
derive an effective context limit metric. In
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the UKRI Centre for Doctoral Training in Application of Artificial Intelligence to the study of Environmental Risks (reference EP/S022261/I), an Isaac Newton Trust grant, research gift from Google, an EPSRC HPC grant, the Hong Kong Research Grant Council Early Career Scheme (Grant No. 27208022), and HKU Seed Fund for Basic Research. Samuel would like to acknowledge the support of Z. Novak and Novak in enabling his contribution.
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, al. Many-shot in-context learning. arXiv preprint arXiv:2404.1100 2024.
Mistral AL Mistral Large https://aattalllaa///ooossaaaaap 2024a.
July
Reka AL. Reka Al API.
2024b.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint 2 1088, 2023.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian MtanGGungg Lou. Make your Ilm fully utilize the context. arXiv preprint 24X.x::4444.681111 2024.
Anthropic.
Introducing the next generation of Claude. hutps://ww.anthropic.ccomm
news/ -lwude-3-aaaaaeeeeee 1y Mar 2024a.
Anthropic. Claude 3.5 Sonnet.
httpsss/www.nntrrppcc..co///
claude-3- 3--ee------ooeeeee Jun 2024b.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongehang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint urXiv:2303.127113 2023.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint MrXiv:2309.13345 2023.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint 1rX1v:2332.11885, 2023.
Google. Vertex AL.
2024.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic litera- ture review. arXiv preprint arXiv:2308. 10620, 2023.
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint ar arXiv:2404.06m 2024a.
11

---

Published as a conference paper at ICLR 2025
Cheng- Yu Hsich, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Ku- mär, James Glass, Alexander Ratner, Chen- Yu Lee, Ranjay Krishna, et al. Found in the mid- dle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406. 16008, 2024b.
Greg Kamradt. Linteessneeddeeiaaaaattt.... http:///////gggttcc....////aaaaa
TnteeleeIAHaastaackc 2023. Accessed: 2024-09-09.
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A" novel" challenge for long-conteds language models. arXiv preprint urXiv:2406.1622 2024.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. In search of needles in 10m haystack: Recurrent memory finds what ilms miss. arXiv preprint vrXi::2402.10790, 2024.
Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-context Ilms and rag systems. arXiv preprint arXiv:2407.01370, 2024.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402 2024.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939, 2023.
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. NeedleBench: Can LLMs Do Retrieval and Reasoning Million Context Window? arXiv preprint arXIv:2407.11963, 2024.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.01889, 2023a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157 2024.
Xiao-Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. Fingpt: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485, 2023b.
Henman Melville. Moby-Dick; or The Whale. Project Gutenberg, 1851.
https://www.
battebberg..or///bookks.
OpenAl. GPT-4V(ision) System Card. https://chh.openna..cc///ppppppp
System_Card.pdl 2023.
OpenAl. GPT-4o mini: advancing cost-efficiiet
go
July 2024.
OpenAl. OpenAl ol: A Large Language Model for Complex Reasoning, OpenAl website, Decem- ber 2024. http://oppenai.co//ll//
OpenAl. API Reference. ht http:///aaa.tt......aaacaoooooor 2024a.
OpenAl. http://///tg/hoo/ooo///////aa//a 2024b.
2024b. Accessed:
15-05-2024.
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models. arXiv preprint arXiv:2404.12387 2024.
Project Gutenberg. Project gutenberg.
2024, Accessed:
2024-09-23.
12

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05550. 2024.
Jonathan Roberts, Timo Lüddecke, Sowmen Das, Kai Han, and Samuel Albanie. GPT4GEO: How Language Model Sees the World's Geography. arXiv preprint arXiv:2336.000020 2023a.
Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs. arXiv
preprint arXiv:2311. 14656, 2023b.
Jonathan Roberts, Kai Han, and Samuel Albanie. GRAB: A Challenging GRaph Analysis Bench- mark for Large Multimodal Models. arXiv preprint arXiv:2408. / 1817,, 2024a.
Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. SciFIBench: Benchmarking large multimodal models for scientific figure interpretation. Neural Information Processing Systems, 2024b.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding, arXiv preprint (rXiv:2305.1416 2023.
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-sturn stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403. 1802, 2024.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid transformer- mamba models at scale. arXiv preprint arXiv:2408. /2570, 2024.
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang, al. Can gpt-4y (ision) serve medical applications? case studies on 4yt-4v for multimodal medical diagnosis. arXiv preprint arXiv:2310.0990 2023.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025. 2023.
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with length levels up to 256k. arXiv preprint arXiv: arXiv:2402.051333 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, et ocbench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1; Long Papers), pp. 15262-15277, 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
We structure our appendix into the following 8 parts:
A Results for the Branched Threading task: $A
Inference metrics such as API service response times:
B
Details of the prompts used for each task: $C
D Specific API model versions used for inference:
D
E Full per-task results for each model:
F Discussion of the limitations of this work:
Description of API- Apseb restrictions encountered during this work: SG
Tables detailing the number of repeats carried out at different context lengths per model for each of the 5 core tasks: S
A BRANCHED THREADING
Clande Hadisa
Clande Sonnet.
Clande 3.5 Soznet
GPT-4a
GPT-4a
Gemini 1.0 Pro
Genini 1.5 Flash
Gemini 1.5 Pro
Janba 1.5 Large
Accuzacy (8)
60
LLaMA
Raka Core
Reka Flach
100
200
300
400
500
800
Context:
Length (1k LZaMA 3.1 takena)
Figure 14: Branched threading. Shaded regions display 95% Wilson confidence intervals.
We carried out branched threading investigation to evaluate the models' ability to accurately re- trieve the final value of threads of length 12.3,4.5.6,7.8.9,,10 where there is branch at each step. We repeat this for branching factors 7,,,,, and present the averaged results in Fig. [1] Similar to the threading tasks, retrieval accuracy drops significantly the context length increases.
B INFERENCE METRICS
14

---

Published as a conference paper at ICLR 2025
FF2-4
Gontnt_14 11b_pro
12.0
3.5, S.5_Snn
Noar Beqpaaze tite
Near Besposoo T160
10.0
15.0
10.0
GPC-4o_ sini
6.0
5.0
1.5
Clande, 3. b_Sonnet
2.0
10
Longth
tolonns)
Number
Figure 15: Mean response times for the nat- ural text (single) threading experiment. Each point corresponds to an average over 65 points (13 thread lengths 5 repeats).
Figure 16: Mean response times for the nat- ural text multi-threading experiment. Each point corresponds to an average over points (from 5 repeats).
C PROMPTS
C.1 SINGLE NEEDLE
Extract the value corresponding the specified key in the JSON object below.
{
001044477-874--bbbe 2t1e::9hhhb--e-----1111122000000 11291/2d066""" 1120::011000000000------11111110
....
"bae328al 4413--11-333-- 4bbba32333322 12-4ee192- -0038-811cc8::24044"" "ca8bf5c3-Iede-7dee
Only write the corresponding value, nothing else. Key: "<key>"
Corresponding value:
C.2 MULTIPLE NEEDLES
Extract the values corresponding to the specified keys in the JSON object below.
{
Only write the list of corresponding values in square brackets, nothing else. Keys: |<keys>>]
Corresponding values:
C.3 CONDITIONAL NEEDLES
Extract the values corresponding to the keys that contain the character "<charr"" in the JSON object below.
90u11998009P6222aaaa114444400000 "54422277-4744223334446611111112 ....
Only write the list of corresponding values in square brackets, nothing else. Corresponding values:
15

---

Published as a conference paper at ICLR 2025
C.4 THREADING
The specified key corresponds to value JSON object below. However, that value might equal another key the JSON object. The value corresponding to this new key might also equal another key in the JSON object. This chain could continue beyond. Extract the final value in the chain. If the value corresponding the first key does not equal another key, then the final value is the value corresponding to the first key.
{
99 http://cwwo.......oooooooo/oooaa
"bae http::/888.02.1..........e....00 "1222aab-90---00hh99111777146663
...
Only write the corresponding value the end of the chain, nothing clsc. Key: "<key>" Corresponding final value:
C.5 MULTI-THREADING
The specified keys each correspond to values the JSON object below. However, the values might equal others key in the ISON object. The value corresponding to each new key might also equal another key in the JSON object. This chain could continue beyond. Extract the final values cach the chain. If the valuc corresponding the first key does equal another key, then the final value is the value corresponding to the first key.
{
"98155980-220 201002226--0000222 2bab-alll 4bbbaal1444crfdeb003999" 55de8eca9-8fd "d64b2470 8779-3333--668---111
32ac328al 44f3-144aaaa----- 1444--777111d333-444002hhhhh "1 112.::
"5d88d1 12-c4cc-7991-11-- 0ee777991-d33---- -t-1-d00008-1111522444444.
Only write the corresponding values the end of each chain in square brackets, nothing else. Keys:
"<keys>"
Corresponding final values:
C.6 BRANCHED THRREADING
The specified key corresponds to value in the JSON object below. However, that value might equal other keys in the JSON object. The values corresponding these keys might also equal other keys in JSON object. This branched chain could continue beyond. Follow the longest chain and extract the final value at the end of the chain.
http:///w.......aa.......a...aa. h1tp::/1111112220044066663400420 "5d88d1 (ad86al112ccee779l.00343441--... http:///.e...aaaaaaooooeeeeeaeaa re88bS533ladee770000000--
Only write the corresponding value the end of longest chain, nothing else. Key; "<key>"
Corresponding final value:
C.7 LLM REFORMATTING SINGLE VALUE OUTPUT
A generative model has answered question to which the answer is 32-character hexadecimal string UUID. in The output from the model answering the question is "cunformatted. Inneformatted_mmdllll wwt.e/oooooooooooeeeeeeeeeeeeeee .eepponse>"... An Extract just the 32-character hexadecimal UUID string from the output. Keep the dashes but remove any whitespace, other characters (such as punctuation quotes), any additional text and explanation, Return only the extracted 32-character hexadecimal UUID, without any additional text or explanation. If answer provided, return "None".An
16

---

Published as a conference paper at ICLR 2025
C.8 LLM-REFORMATTING MULTIPLE VALUE OUTPUT
A generative model has answered question which the answer is list of 32-character hexadecimal strings. (n The output from model answering the question "Kunformatted model response>>"""""" An Extract just the list of 32-character hexadecimal UUID strings from the output. Keep the dashes but remove any whitespace, other characters (such as punctuation or quotes), and any additional text and explanation. An Format the list a list of strings, with each string in the list being 32-character hex- adecimal UUID string. For example: ['12345678-122) 1-1-788-113-578--- 1234-6888-77 1367612245688 '87654321-4321- 8765-4321 14921-777686654321111 I''m Return only the extracted list, without any additional text or explanation. Do not include any additional syntax, like ""yython",, in your answer. If no answer is provided, return "n
D MODEL VERSIONS
Closed-source model API versions
GPT-4o mini: gpl-4oo-iini22224440--8
GPT-4o: gpt-4o-202--08-06
Gemini-Pro- 10mini--.0-pro-002
Gemini 1.5 Flash: geminn-l..5fllss-prreieee--
Gemini 1.5 Pro: gemini-l.-5ppo-prreeeex-0011111
Claude 3 Haiku:
Claude 3 Sonnet: e 20240029
Claude 3.5 Sonnet: e 202400000
Reka Flash: reka-flashh-22244999
Reka Core: reku-core-202-404
FULL RESULTS
Claude Sonnet
Accusery 00
Claude 3.5 Sonnet
60
CPT-4a
6PT-4o mini
Genini
Canini 1.5
Canins 1.5 Pro
Janba 1.5 Large
Janba 1.5 Mini
30
CLaNA
LLaMA 70b
Raka Core
Raka Flash
100
200
300
400
500
600
Kistral Large
Contect Lungth sakums)
Mistral Kano
Figure 17: Single Needle overall performance with 95% Wilson confidence intervals.
17

---

Published as a conference paper at ICLR 2025
Claude 3.5 Sounet
10
Length
LiaM
tokens)
Figure 18: Multi-threading. Concurrently following N threads does not degrade performance.
Model
Accuracy (%)
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5
100.0
100.0
100.0
100.0
100.0
76.4
30.9
Gemini 1.5 Flash
100.0
100.0
100.0
100.0
100.0
94.5
83.6
89.1
74.5
34.5
32.7
Jamba 1.5 Large
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Jamba 1.5 Mini
100.0
100.0
98.2
98.2
96.4
100.0
94.5
78.2
72.7
Claude 3.5 Sonnet
100.0
100.0
100.0
100.0
100.0
100.0
98.2
90.9
87.3
Claude 3 Sonnet
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
94.5
Claude Haiku
100.0
100.0
100.0
100.0
98.2
100.0
94.5
83.6
GPT-4o
100.0
100.0
GPT-4o mini
100.0
100.0
100.0
100.0
100.0
98.2
94.5
80.0
Reka Core
100.0
100.0
0.0
94.5
87.3
89.1
87.3
61.8
Reka Flash
100.0
100.0
76.4
83.6
85.5
76.4
56.4
50.9
LLaMA 3.1 8b
96.4
98.2
100.0
94.5
98.2
89.1
87.3
50.9
LLaMA 3.1 706
100.0
96.4
96.4
98.2
96.4
89.1
89.1
18.2
ILaMA 405b
100.0
100.0
98.2
100.0
Gemini 1.0 Pro
100.0
100.0
100.0
98.2
76.4
Mistral Large
100.0
100.0
100.0
100.0
98.2
Mistral Nemo
100.0
100.0
100.0
100.0
12.7
Table 3: Single Needle depth-averaged results. Reka Core 0.0 at 5k is likely due to safety restraints
(output generated due 'context').
Model
Accuracy
(%)
1.2k
2.5k
Sk
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
100.0
100.0
100.0
100.0
100.0
99.8
97.4
96.3
94.7
76.7
34.6
30.0
Gemini 1.5 Flash
100.0
98.9
100.0
100.0
99.9
86.7
86.3
84.0
67.7
46.3
18.5
10.0
Jamba Large
99.6
99.4
99.5
98.0
95.5
92.6
88.4
83.9
Jamba 1.5
67.0
63.0
56.6
46.4
35.0
21.4
13.5
Claude Sonnet
100.0
100.0
100.0
99.9
99.7
99.6
99.1
97.3
85.9
Claude Sonnet
100.0
100.0
100.0
100.0
99.5
98.6
97.0
93.8
91.7
Claude Haiku
99.9
100.0
99.4
99.7
98.5
96.9
94.9
80.2
67.0
GPT-4o
100.0
100.0
100.0
100.0
100.0
100.0
99.9
99.8
GPT-40 mini
99.9
99.8
99.0
98.6
97.2
95.6
85.5
70.5
Reka Core
97.6
82.7
64.7
50.0
54.8
42.9
31.6
0.0
Reka Flash
94.9
77.9
68.2
55.2
48.1
49.8
45.0
19.4
LLaMA 3.1 8b
98.0
94.7
88.1
78.3
63.6
51.8
40.9
16.8
T.I.MA 70h
100.0
100.0
100.0
99.9
97.7
91.2
73.2
LLaMA 405b
88.2
98.6
94.0
88.2
17.7
Gemini 1.0 Pro
99.8
99.9
98.2
97.4
58.5
Table 4: Multiple Needles overall results.
18

---

Published as a conference paper at ICLR 2025
Model
Accuracy (%)
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini Pro
98.6
98.3
95.2
97.3
93.6
95.7
92.4
85.6
77.9
86.2
59.9
Gemini Flash
96.3
96.9
94.6
94.3
90.2
86.8
78.8
78.8
66.7
64.1
52.2
54.8
Jamba 1.5 Large
98.0
92.4
85.4
71.0
30.7
25.0
27.1
Jamba Mini
80.5
66.3
46.0
30.7
19.6
15.9
20.3
Claude 3.5 Sonnet
88.9
92.2
89.8
88.3
87.1
87.7
71.4
45.3
51.4
Claude 3 Sonnet
99.9
99.9
98.1
45.0
16.1
17.0
Claude 3 Haiku
99.2
94.3
90.2
84.9
60.9
50.8
21.8
28.9
33.5
GPT-4o
100.0
99.8
99.2
97.5
91.2
92.8
89.9
82.3
GPT-4o mini
98.2
98.3
92.9
88.9
80.1
77.4
76.7
63.9
Reka Core
56.9
61.2
16.9
21.7
4.7
2.8
5.6
Reka Flash
68.8
37.7
6.7
6.6
0.2
0.0
0.0
0.0
LLaMA 3.1 8b
52.9
51.2
34.1
31.0
4.9
2.5
0.4
0.0
LLaMA 3.1 70b
97.2
98.4
99.1
97.1
85.4
80.5
30.0
1.8
LLaMA 3.1 405b
100.0
100.0
99.8
98.5
94.7
85.6
16.7
0.2
Gemini 1.0 Pro
54.0
17.4
11.0
8.0
1.1
Table 5: Conditional Needles overall results.
Model
Accuracy (%)
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
57.8
42.2
35.0
37.8
29.4
25.0
23.3
23.3
Gemini 1.5 Flash
46.7
33.9
25.6
18.3
16.7
13.9
10.0
6.7
2.8
0.0
0.6
Jamba Large
23.9
12.2
8.3
5.6
0.6
0.0
Jamba Mini
7.8
3.3
1.7
1.7
0.0
0.0
Claude 3.5 Sonnet
78.3
72.2
61.7
53.3
52.2
43.9
13.3
5.6
4.4
Claude 3 Sonnet
40.0
26.7
17.2
7.2
6.7
2.8
1.1
0.0
0.0
Claude 3 Haiku
25.6
10.0
7.2
3.3
1.7
0.0
1.7
0.6
1.1
GPT-4o
75.0
61.1
51.1
30.0
23.3
16.1
14.4
7.2
GPT-4o mini
37.2
22.8
14.4
8.3
5.0
0.0
0.0
0.0
Reka Core
27.8
22.2
0.0
0.0
0.0
0.0
0.0
Reka Flash
19.4
0.0
2.8
2.8
0.0
0.0
0.0
I.LMMA 3.1
13.2
1.4
0.7
0.0
0.0
0.0
LLaMA 70b
13.0
1.9
0,0
LLaMA 40Sb
58.3
20.8
29.2
12.5
0.0
Gemini 1.0 Pro
23.3
8.9
2.2
0.6
1.1
Mistral Large
68.9
45.0
31.1
10.6
1.1
Mistral Nemo
12.2
7.2
2.2
0.0
0.0
Table 6: Threading overall results.
Model
Accuracy (%)
1.2k
2.5k
Sk
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
82.2
65.1
53.2
57.9
50.7
44,9
34.6
Gemini 1.5 Flash
60.5
36.9
30.4
25.1
18.5
10.5
7.8
4.0
2.2
0.3
0.5
Jamba Large
32.5
13.5
8.0
13.0
3.8
1.2
1.2
Jamba 1.5 Mini
18.9
10.8
13.6
7.9
2.5
1.0
0.0
0.0
Claude 3.5 Sonnet
90.1
79.1
72.8
62.8
58.2
48.5
33,9
13.8
11.1
Claude 3 Sonnet
69.9
42.1
24.2
7.6
1.0
5.1
1.5
0.0
1.6
Claude 3 Haiku
34.1
24.2
17.4
8.7
7.4
4.0
2.3
1.6
1.6
GPT-4o
90.9
69.5
57.5
42.9
44.9
34.1
19.9
15.2
GPT-4o mini
43.0
18.6
17.3
13.1
9.3
10.3
0.0
0.0
Reka Core
16.8
2.9
3.5
1.5
1.3
0.0
0.2
Reka Flash
11.1
1.7
0.2
0.6
0.0
LLaMA
14.0
3.3
1.1
1.5
LLaMA 70b
55.1
21.6
4.1
1.8
0.4
LLaMA 3.1 405b
91.6
71.5
43.7
22.7
14.5
2.2
2.4
0.3
Gemini 1.0 Pro
21.6
8.2
1.3
0.3
1.9
Mistral Large
71.3
49.2
34.9
14.4
8.7
Mistral Nemo
19.0
14.4
9.7
7.7
3.1
Table 7: Multi-Threading overall results.
19

---

Published as a conference paper at ICLR 2025
F LIMITATIONS
We note several limitations to our work. First, we restrict our study to the use of synthetic data. While this has significant benefits (fine-grained controllability, automatic provision of per- fect ground truth), our benchmark does not capture differences in LLM behaviour that are domain- specific (for instance, LLMs may be more performant on some distributions than others). Second, as discussed below, the scale of our experiments (particular the number of experimental repeats) was limited by cost for the larger models.
G API RESTRICTIONS
The design of our experiments was guided in part by the following API-hased restrictions and limi- tations:
Cost. For the most expensive models (e.g., Gemini 1.5 Pro, Claude 3.5 Sonnet), running just single repeat on one task could cost hundreds of dollars. Therefore, in some cases, the evaluation of these models could not be repeated extensively, limiting the statistical strength of our experiments.
Context restrictions. Some models were only available for API-based inference in lim- ited capacity (e.g., Mistral), in which it was not possible provide ipputs that approach the context limit. As such, we could only evaluate these models as close to the context limit as we could.
Latency. As result of latency introduced by low server throughput indirectly via low rate limits at the time of writing, for some models (e.g., LLaMA 3.1), it was not possible extensively conduct repeats.
H REPEATS
Model
Num. Repeats
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
5
5
5
Gemini 1.5 Flash
5
5
5
5
5
5
Jamba 1.5 Large
5
5
5
5
5
5
5
Jamba 1.5 Mini
5
5
5
5
5
5
5
Claude 3.5 Sonnet
5
5
5
5
Claude Sonnet
5
5
Claude Haiku
5
5
5
5
5
GPT-4o
5
5
5
5
5
5
5
GPT-4o mini
5
5
5
5
5
5
Reka Core
5
5
5
Reka Flash
5
5
5
LLaMA 3.1 8b
5
5
5
5
5
5
LLaMA 3.1 70b
5
5
5
5
5
5
5
LLaMA 3.1 405b
5
5
5
5
5
Gemini Pro
5
Mistral Large
5
5
Mistral Nemo
5
5
5
5
Table 8: Number of repeats carried out for the Single Needle task.
20

---

Published as a conference paper at ICLR 2025
Model
Num. Repeats
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
5
5
5
5
5
Gemini 1,5 Flash
5
5
5
5
5
5
5
5
5
5
Jamba 1.5 Large
5
5
5
5
5
5
5
Jamba 1.5 Mini
5
5
5
5
Claude 3.5 Sonnet
5
5
5
Claude Sonnet
5
5
5
5
5
5
Claude 3 Haiku
5
5
5
5
5
5
5
GPT-4o
5
5
5
5
GPT-4o mini
5
5
5
Reka Core
1
1
Rcka Flash
1
1
1
LLaMA 3.1 8b
2
2
2
2
2
2
2
LLaMA 3.1 70b
2
2
2
2
LLaMA 3.1 405b
1
Gemini 1.0 Pro
5
5
Table 9: Number of repeats carried out for the Multiple Needles task.
Model
Num. Repeats
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
5
5
5
5
5
5
1
Gemini 1.5 Flash
5
5
5
5
5
Jamba 1.5 Large
5
5
5
5
Jamba 1.5 Mini
5
5
5
5
5
5
Claude 3.5 Sonnet
5
5
5
5
5
5
5
Claude Sonnet
5
5
5
5
5
Claude Haiku
5
5
GPT-4o
5
5
5
5
5
GPT-4o mini
5
5
5
5
5
5
Reka Core
1
Reka Flash
1
LLaMA 3.1 8b
LLaMA 3.1 70b
1
LLaMA 3.1 405b
1
Gemini 1.0 Pro
5
5
Table 10: Number of repeats carried out for the Conditional Needles task.
Model
Num. Repeats
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
5
5
5
5
Gemini 1.5 Flash
5
5
5
5
5
5
5
5
Jamba 1.5 Large
5
5
5
5
5
5
5
Jamba 1.5 Mini
5
5
5
5
Claude 3.5 Sonnet
5
Claude Sonnet
5
5
5
5
5
5
Claude 3 Haiku
5
5
5
5
5
5
5
GPT-4o
5
5
5
5
5
GPT-4o mini
5
5
5
Reka Core
1
Reka Flash
1
1
1
LLaMA 3.1 8b
4
4
4
4
4
4
4
4
LLaMA 3.1 70b
3
3
2
LLaMA 3.1 405b
1
1
Gemini 1.0 Pro
5
5
5
Mistral Large
5
5
5
5
Mistral Nemo
5
5
5
Table 11: Number of repeats carried out for the Threading task.
21

---

Published as a conference paper at ICLR 2025
Model
Num. Repeats
1.2k
2.5k
5k
10k
20k
32k
64k
128k
180k
250k
500k
630k
Gemini 1.5 Pro
1
1
1
Gemini 1.5 Flash
5
5
5
5
Jamba 1.5 Large
1
1
1
1
Jamba 1.5 Mini
1
1
I
1
Claude 3.5 Sonnet
5
5
5
5
5
5
Claude Sonnet
Claude Haiku
5
5
5
5
5
5
GPT-4o
1
1
GPT-4o mini
1
Reka Core
Reka Flash
LLaMA 3.1 8b
1
LLaMA 3.1 70b
1
LLaMA 3.1 405b
1
Gemini 1.0 Pro
5
Table 12: Number repeats carried out for the Multi-threading task.
22


Reviews:
[{"review_id": "Jjkr0BcQ9I", "reviewer": "Reviewer_Q7CU", "summary": "In this paper, the authors introduce simple single-needle retrieval, multiple-needle and conditional-needle retrieval and challenging needle threading and multithreading retrieval. Experiments on haystacks consisting of key-value pairs of UUIDs shows the retrieval precisions of 17 LLMs vary on different context lengths, multiple-needle and multiple threading conditions.", "questions": "What is the time complexity (running/response time) of each LLM that is used for evaluation in different experiment settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper focuses on an interesting problem: how effectively LLMs use their context. \nThe observations from the experiments are inspiring, for example, many models are remarkably thread-safe: capable of simultaneously following multiple threads without significant loss.", "weaknesses": "Only one synthetic dataset is used for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQ0N45lTJ6", "reviewer": "Reviewer_n6xp", "summary": "The paper evaluates the performance of Large Language Models (LLMs) as their context limits increase, focusing on their ability to handle complex information retrieval across multiple documents. Through experiments with 17 leading LLMs, the study finds that while many models effectively manage multiple threads of information, their actual efficient context range is often shorter than their maximum allowed, leading to reduced performance as the context window expands. The research also highlights inconsistencies in token counts across different tokenizers. Results and methodologies are made available for further study.", "questions": "Please refer to the previous section", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Note: This is a review by an emergency reviewer. \n\nThe paper addresses an intriguing and valuable research problem by exploring the utility of synthetic data for abstract tasks in the context of long-sequence processing.\n\nIntroducing multi-threading tasks as part of the experimental setup is particularly innovative. \n\nThe findings of the paper offer practical insights that could inform both the academic and industry sectors about the design and tuning of language models for specific applications.", "weaknesses": "Note: This is a review by an emergency reviewer. \n\nThe major concern is that there's a gap between experimental design and real-world applications. The study employs highly abstract tasks using synthetic data with no natural language semantics, deviating considerably from the typical environments where large language models (LLMs) operate. The string-serialized JSON objects and UUIDs as key-value pairs fail to engage the models in natural language processing—core to their training and operational objectives. Consequently, the findings have limited applicability to real-world scenarios that demand comprehension, generation, and manipulation of actual linguistic content. This gap undermines the relevance of the research to practical applications of LLMs, which are primarily designed to interact with and generate coherent, contextually appropriate natural language.\n\nThe authors mention that they have released their code and experimental data for public use. However, the authors didn't upload supplemental materials to the open review nor include an anonymous link of their data and code. I would suggest sharing the code through an anonymous GitHub repository or similar platform. This would greatly aid other researchers and reviewers in replicating and understanding the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Twhz3DKdqF", "reviewer": "Reviewer_bsZQ", "summary": "This paper investigates the ability of Large Language Models (LLMs) to handle complex information retrieval and reasoning tasks across long contexts. The authors conduct a series of retrieval experiments with 17 leading LLMs to assess their capability to follow information threads within extensive context windows, revealing that while many models can manage multiple threads without significant performance loss, their effective context limits are often shorter than their technical limits. The study also emphasizes the variability in token counts between different tokenizers, which affects model performance comparisons. A key contribution is the introduction of a task-specific effective context limit metric, which provides a more nuanced understanding of model capabilities in long-context scenarios.", "questions": "1. Have you considered conducting smaller-scale validation experiments with natural language data to verify if the findings generalize?\n\n2. For the branched threading task, why was it only evaluated on a subset of models and smaller context sizes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides an extensive evaluation of 17 leading LLMs across various long-context retrieval tasks, offering a thorough analysis of their performance in handling complex information retrieval and reasoning.\n\n2. Through innovative needle threading and multi-threading experiments, the study creates scenarios where LLMs must follow chains of information across different parts of the context, effectively testing their limits in long-context understanding.\n\n3. Analysis reveals significant differences in tokenization between models, crucial for understanding the discrepancies in reported context lengths and making accurate comparisons between LLMs.\n\n4. The authors propose a task-specific and configurable metric independent of tokenization, enabling more precise assessment of models' reasoning capabilities over context.", "weaknesses": "1. The study's exclusive use of synthetic data (UUID key-value pairs) may not accurately reflect performance on natural language tasks or domain-specific applications.\n\n2. The paper may be limited in techinical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l0OC6Daq4n", "reviewer": "Reviewer_wcWb", "summary": "This paper executed a set of experiments to evaluate the capabilities of 17 LLMs to perform retrieval tasks with long contexts. It introduces challenging multi-step threading and multi-threading retrieval tasks to test the models' ability to track information through extended contexts. The results show that increased context length reduces retrieval performance, with accuracy generally decreasing as the context window grows. The authors also demonstrate that many leading LLMs are thread-safe. Additionally, they highlight significant differences in token counts across different tokenizers.", "questions": "Given that the experiments rely on synthetic, abstract tasks, how do you anticipate the performance trends observed in your study would translate to real-world, domain-specific applications with more complex and noisy data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) This paper studies an important aspect of LLMs -- their ability to manage long contexts -- which hasn't been fully explored yet.\n\n(2) The paper is well-written.\n\n(3) The task designs are comprehensive and insightful, especially the comparison between forward and backward threading.\n\n(4) The experiments are extensive, and the results provide valuable insights into LLM performance with long contexts.", "weaknesses": "(1) Code and data are not currently shared.\n\n(2) The experiments in the paper rely on abstract retrieval tasks using synthetic data. These tasks can not reflect the complexity found in real-world, domain-specific applications. In practice, data often includes ambiguous language, diverse formats, and contextually relevant nuances. Moreover, the experiment design cannot test the model's ability to understand semantics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)