HELMET: How to Evaluate Long-context Models Effectively and Thoroughly
Authors: Howard Yen
Tianyu Gao
Minmin Hou
Ke Ding
Daniel Fleischer
Peter Izsak
Moshe Wasserblat
Danqi Chen
Venue: ICLR 2025 Poster (2025)
Keywords: long-context language models
benchmarking
Primary area: datasets and benchmarks

Abstract: Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the underlying reasons behind these practices and find that existing benchmarks often provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions---the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.

Published as a conference paper at ICLR 2025
HELMET: HOW TO EVALUATE LONG-CONTEXT LAN- GUAGE MODELS EFFECTIVELY AND THOROUGHLY
Howard Yen P Tianyu Gao Minmin Hou Ke Ding'
Daniel Fleischer Peter Izsak' Moshe Wasserblat Danqi Chen PPrinceton Language and Intelligence, Princeton University *Intel (hyen, tianyug, d
wwwpcctooonnnnoommmm.. prr.iiictttss.prrtetttoooooo edu https: https///ppiintooeeeeeee wwppoeeetonnnn...e. github. D/HELMET
ABSTRACT
Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the derlying reasons behind these practices and find that existing benchmarks of- ten provide noisy signals due limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How Evaluate Long-context Models Effectively and Thoroughly), comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues previous benchmarks by adding controllable lengths up 128K tokens, model-based eval- uation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, find that (1) synthetic tasks like NIAH do not reliably predict down- stream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream perfor- mance; ultimately, we advocate for holistic evaluation across diverse tasks.!
demonstrate that HELMET offers more reliable and
INTRODUCTION
Long-context language models (LCLMs) un- lock a myriad of applications, from summariz- ing long documents to learning new tasks the fly with thousands of examples. Many re- cent benchmarks have sought to evaluate lan- guage models' long-context abilities (Zhang al., 2024b; An et al., 2024: Shaham et al., 2023; Bai et al., 2024, inter alia). However, recent developments in long-context process- ing (Chen et al., 2023; Xiong et al., 2023; Peng et al., 2024; Fu et al., 2024) still rely ei- ther on perplexity haystack tasks (NIAH; Kamradt, 2024; Hsich et al., 2024). Frontier LCLMs (Dubey et al., 2024; Team et al., 2024c; OpenAl, 2023; Team
Table
Most LCLMs evaluate on synthetic (Syn.) tasks. ? base models. no: BBBENCH
ZEROSCROLLS
Model
Syn. PPL
RAG ICL
QA AlI QS QL SQ All
Gemini-1.5 GPT4
X
X
X
X
x
X
X
X
X
X
X
Claude-3.5
X
X
Llama-3.1
/
/
X
X
Phi-3
X
X
/
X
X
X
Jamba-1.5 Qwen2
X
X
X
X
X
X
X
Command R
X
X
X
X
X
X
X
synthetic needle-in-a-
Xiong
X
X
Chen al. Peng el al. Fu et al."
/
X
X
X
X
x
X X
X
X
X
X
OOu data and code available at https:/ :/tpss////iii wttp..c.o//rrineettooo

---

Published as a conference paper at ICLR 2025
NIAH
RULER
ooBench
HELMET (Ours)
Perfomence
100
80
80
50
50
60
60
40
40
40
40
30
30
00
00
00
00
BB=708 40 PlashePro
8B-70B 40 Flash>Pro
88-708
Flash<Pro
884706
Flash-Pro
Figure 1: Long-context benchmark results of frontier LCLMs (Llama-3.1 8B/70B, GPT-44-mini, GPT-40-08-06 and Gemini-1.5 Flash/Pro) at 128K input length. NIAH is saturated for almost all models; RULER (Hsieh et al., 2024) and x. (Zhang et al., 2024b) show unexpected trends for Llama-3.1 (Dubey et al., 2024). In contrast, HELMET demonstrates more consistent rankings of these frontier models.
et al., 2024a) also mostly report NIAH, sometimes with arbitrary subsets of other datasets, as shown in Table 1. Such inconsistencies complicate comparisons between different models. It is also un- clear whether synthetic NIAH tasks or the few chosen datasets offer a holistic picture of models' long-context abilities and their performance in real-world applications.
Why don't I model developers agree on these evaluation?? We take closer look and find that existing benchmarks suffer from many critical design flaws, including:
Insufficient coverage of downstream tasks: Existing benchmarks either focus on synthetic tasks (Hsieh et al., 2024) or include only simple question answering (Zhang al., 2024b). Other works study particular aspects of LCLMs, such as summarization (Chang et al., 2024), in-context learning (Li et al., 2024c), and retrieval-augmenn do not provide a holistic evaluation of LCL.Ms.
generation (RAG; Lee et al., 2024), but they
Inadequate lengths: Most natural language datasets in existing benchmarks (Shaham et al., 2023; An al., 2024; Table 4) are too short to effectively test frontier long-context abilities (usually 2 128K).
Unreliable metrics: For commonly used long-document QA and summarization tasks, most existing benchmarks still rely on metrics like ROUGE (Lin, 2004), which are often noisy and unreliable (Goyal et al., 2023; Deutsch et al., 2022; Chang et al., 2024).
Incompatibility with base models: Many LCLM developments focus on base models without instruction tuning, but most existing benchmarks require models to be instruction-tuned--err developers can only rely on synthetic tasks or perplexity.
Consequently, existing benchmarks either not applicable to long-context works (inadequate lengths or incompatibility) or provide highly noisy signals (insufficient coverage or unreliable met- rics). We summarize the shortcomings of existing benchmarks in Table 2. For the three benchmarks that support a 128K2 context length (NIAH, RULER, and oo BENCH), we use them to evaluate six frontier models and report the numbers in Figure 1. We see that NIAH does not reflect differences across models; RULER and oo BENCH show unexpected trends-on RULER, Gemini Flash outper- forms Gemini Pro; on oo BENCI, the 70B Llama model underperforms compared the 8B one. This raises concerns about the reliability of the benchmarks and how they can differentiate long- long-context models, which likely contributes the lack of use of these benchmarks in model development.
To address these challenges. we present HELMET (How to Evaluate Long-context Models Effectively and Thoroughly). We curate a diverse set of application-cennrric long-context tasks across seven categories. Beyond widely adopted categories like synthetic recall, long-document question answering (QA), and sunmarization, we also add many-shot in-context learning (ICL), retrieval- augmented generation (RAG), passage re-ranking, and generation with citations (Gao et al., 2023).
We further address the shortcomings of existing benchmarks: (1) we ensure all datasets support input lengths of 128K tokens and are easily extendable to longer contexts ($2.1); (2) introduce reference-basee model evaluation for QA and summarization and show that it significantly improves over n-gram overlap metrics ($2.2); (3) refine the prompts and in-context demonstrations used for all tasks, reducing evaluation noise caused by different output formats and allowing base models to
Throughout the paper, we use binary prefixes K= 2
2

---

Published as a conference paper at ICLR 2025
Table 2:
Comparison of long-context benchmarks: ZeroSCROLLS (Shaham al., 2023). Long- Bench (Bai et al., 2024). L-Eval (An et al., 2024), RULER (Hsieh et al., 2024), cc BENCH (Zhang al., 2024b), and our HELMET. L: input tokens. t: All datasets have < 128K except one dataset. :; L-Eval uses LLMs compute reference-free, pairwise win-rates; we design reference- based model evaluation for specific tasks.
Type of tasks
Benchmark features
Re-
Long- QA
Synthetic Recall
Robust Eval.
Controll- able
Cite RAG
Summ ICL
L Z 128k
rank
ZeroSCROLLS
X
X
LongBench
X
J
X
/
/
/
x
L-Eval J
/
X
xt
X
RULER
X
X
X
X
X
X
J
cOBENCH
X
X
X
X
X
HELMET (Ours)
V V
V
V
be evaluated robustly across most categories ($2.3). Together, HELMET enables a holistic evalua- tion of long-context capabilities and provides more reliable signals for model development. Figure demonstrates that HELMET can clearly differentiate models with varying capabilities and reflect comparisons consistent with human perception. More discussion on HELMET's improvements over previous benchmarks and direct comparisons can be found in SA.
To understand the progress of LCLMs and how different long-context capabilities correlate with one another, we evaluate a comprehensive list of 59 LCLMs of various architectures, scales, and training approaches. Our analysis reveals that (1) synthetic tasks poorly indicate ownstream performance ($3.1). (2) different categories in HELMET show distinct trends ($3.2), and (3) open-source models significantly lag behind closed ones on tasks that require reasoning over long contexts or following complex instructions-thh the gap further widens context length increases ($3.3). Finally, find that the RAG category strikes a good balance between ease of use, stronger correlation with downstream tasks, and compatibility with base models. Ultimately, it is imperative evaluate LCLMs across diverse spectrum of categories, We hope that our insights provide a more effective way to evaluate LCLMs for future model development and benchmarking.
OUR BENCHMARK: HELMET
In this work, seek overcome the shortcomings of existing benchmarks by meeting the fol- lowing desiderata: (1) diverse coverage across different tasks and capabilities of LCLMs, (2) con- trollable context lengths that support more than 128K input tokens, and (3) reliable evaluation for both base and instruction-tune and how they improve upon existing evaluation benchmarks in terms of settings and metries. overview of HELMET is shown in Table 3.
models. In this section, we describe the datasets used in HELMET
2.1 REALISTIC AND DIVERSE LONGGCCNTEXTT APPLICATIONS
Retrieval-augmmented generation (RAG). We use open-domain question answering (ODQA) which requires retrieving from a knowledge corpus and then generating correct answers (Chen et al., 2017)-as a representation of aurieetl-- generation (RAG) applications. We utilize Natu- ral Questions (NQ; Kwiatkowski et al., 2019), TriviaQA (TQA; Joshi et al., 2017), HotpotQA (Yang et al., 2018), and PopQA (Mallen et al., 2023). We use the gold passage (the passage with the answer) from Petroni et al. (2021), otherwise select any passage that contains the answer.
Given an input length L, we first determine the number of passages that can fit within L tokens, then retrieve A passages" from the corpus* that do not contain the answer as distractors. This differs from previous works that randomly sample passages from the corpus (Lee et al., 2024) and is more
We use A Tababa-Ni.. N.PP/gT h.PP/gte-11 aelee-argeee arge-en-vl 1..5 5 for retrieval (Zhang et al., 2024a). WWe use Wikipedia 2019-8-01 dump, split into 100-word passages (Petroni al., 2021).
3

---

Published as a conference paper at ICLR 2025
Table 3: Overview of evaluation datasets. We select datasets that cover various important long- context capabilities. SubEM: substring exact match.
Category
Dataset
Metries
Description
Natural Questions
SubEM
Factoid question answering
Retrieval- augmented generation
TriviaQA
SubEM
Trivia question answering
PopQA
SubEM
Long-tail entity question answering
HotporQA
SubEM
hop question answering
Generation with
ALCE ASQA
Rocall, Cite
Answer umbiguous questions with citations
citations
ALCE Qampari
Recall, Cite
Answer factoid questions with citations
Passage re-ranking
MS MARCO
NDCG@10 10
Rerank passage for a query
TREC Coarse
Accuracy
Question type classification, 6 labels Question type classification, 50 labels Task intent classification, 68 labels Banking intent classification, 77 labels Intent classification, 151 labels
Many-shot
TREC Fine
Accuracy
in-context
NLU
Accuracy
learning
BANKING77
Accuracy
CLINCI50
Accuracy
Narrative
Model-hased
Book and movie script QA
Long-document
Be QA
ROUGE F1
Novel QA with entity replacement
QA
xBBNNC MC
Accuracy
Novel multiple-choice QA with entity replacement
xCBENH Sum
Model-hased
Novel summarization with entity replacement
Summarization
Multi- LexSum
Model-based
Summarizing multiple legal documents
ISON KV
SubEM
Retrieve key in JSON dictionary
RULER MK Needle
SubEM
Retrieve the needle (a number) within noisy needles
Synthetic recall
RULER MK UUID
SubEM
Retrieve the needle (a UUID) within noisy needles
RULER MV
SubEM
Retrieve multiple values for one needle (key)
realistic and challenging. For NQ, TQA, and PopQA, we take the top k distractors and insert the gold passage at six evenly distributed positions following Liu et al. (2023). For HotpotQA, which requires two gold passages, we combine them and the top k - 2 distractors and randomly shuffle them into three permutations. We use substring exact match (SubEM: whether the answer is included in the output), following previous work (Asai et al., 2024a). See $B.1 for more details.
Generation with citations (Cite). We leverage ALCE (Gao et al., 2023) to evaluate LCLMs on a realistic application of answering questions while providing correct attributions (Bohnet et al., 2022). Given multi-faceted questions and relevant passages, models required to generate a long- text answer and cite supporting passage IDs at the end of each sentence. This tests models' ability to utilize the passages in the context and also follow the instructions about citation formats. We use the ASQA (Stelmakh et al., 2022) and QAMPARI (Rubin et al., 2022) subsets from ALCE. For an input length L., we first determine the number of passages k, and use the top retrieved passages from Wikipedia as contexts. The model's outputs are evaluated on correctness and citation quality, and we report the average across all metrics. See $B.2 for more details.
Passage re-ranking (Re-rank). Re-ranking retrieved passages based on their relevance to the query an important application of LCLMs (Sun et al., 2023). The task requires the model retrieve relevant information, compare, and reason over different parts of the contexts. We use the MS MARCO dataset (Bajaj et al., 2018), where each instance contains a query and passages retrieved by BM25 (Robertson & Zaragoza, 2009) from the Internet. Each passage has annotations of relevance label-perfect, highly relevant, or not relevant. We determine the number of passages from the input length L, and randomly sample k passages with balanced labels for each test query. The model is prompted with the query and the shuffled & passages and is instructed to output the top- 10 document IDs ranked by relevance. We report NDCG@ @ 10. Details are in BB.3.
Many-shot in-context learning (ICL.. In-context learning (ICL. is a key ability that enables LLMS to adapt to new tasks on the fly (Brown et al., 2020). Recent studies (Ratner et al., 2023; Xu et al., 2024; Li et al., 2024c; Bertsch et al., 2024) explore performing many-shot ICL (with thousands of examples) with LCLMs. Following Bertsch et al. (2024), we focus datasets with large la- bel spaces: TREC-coarse, TREC-fine (Li & Roth, 2002), BANKING77 (Casanueva et al., 2020), CLINC150 (Larson et al., 2019), and NLU (Liu et al., 2019). We adjust the number of shots to trol the input length L, and the number of examples in each class balanced. We report accuracy on the test set.
4

---

Published as a conference paper at ICLR 2025
One difference from previous works is that map original natural language labels (e.g., location) into numbered labels (i.e., 0, 1) to test how well a model can learn new tasks instead of relying on its pre-trained priors (Wei et al., 2023; Pan et al., 2023; Min et al., 2022). More details are in $B.4.
Long-document question answering (LongQA), We use Narra- tiveQA (Kočiský et al., 2018) and the English book QA and multiple choice (MC) subsets from BENCI (Zhang et al., 2024b) for evaluat- ing long-document QA. We select those tasks for their abundant con- text lengths (Table 4). We truncate the document from the end based on L. We use ROUGE FI for <BENC QA (answers are mostly en- tity names) and accuracy for oBBENCH MC. For NarrativeQA, where the answers be long text and open-ended, we design and use model-based evaluation ($2.2).
Table 4: Dataset lengths.
Datasets
Medium Max
ZeroSCROLLS
QASPER GovReport QUALITY
6K
12K
12K
33K
9K
11K
SQUALITY
8K
10K
Summarization (Summ). Summarization tests LCLMs' ability to synthesize information across the contexts. We choose Multi-LexSum (legal document summarization) and the English summarization task from BENCH (novel summarization) for their extensive lengths (Ta- ble 4). We truncate the document from the end based the evalu- ation length L. We use our model-based evaluation ($2.2) for both datasets instead of the commonly used ROUGE, as better reflects human judgment.
HELMET
NarrativeQA
73K 518K
oc BENCH QA 191K 835K
oc BENCH MC 167K 835K
BENCH Sum 154K 835K
Multi-LexSum 90K 5M
Synthetic recall. Synthetic recall tasks, such as needle-in-a-haystaee (NIAH), stress test models' ability to recall relevant information (the "needle") from long contexts. They have gained popularity for being easy to (as they can test any arbitrary length) and easy control (can placing the "needle" at any position). For this category, we select multiple synthetic recall tasks from RULER extended version of NIAH; Hsich al., 2024) and also add JSON KV retrieval task (Liu al., 2023), which we find more challenging. We intentionally select the synthetic tasks that correlate well with application-dricc driven tasks; in-depth discussions are $3.1. Following previous works, we report the percentage of the ground truth answers that are substrings in the generation (SubEM). Refer to $B.5 for more details.
2.2 RELIABLE EVALUATION METRICS
Existing long-contes benchmarks (Zhang et al., 2024b; Shaham et al., 2023) largely rely on n-gram overlap metrics like ROUGE (Lin, 2004), which have been shown to correlate poorly with human judgment for tasks with long outputs, such as summarization (Goyal al.. 2023; Deutsch et al., 2022; Krishna et al., 2023). L-Eval (An et al., 2024) uses LLMs to score reference-free "win rates." which neglect the available answer annotations and always require evaluating model pairs. Instead, design reference -based model evaluation method for long-document QA and summarization that is more reliable and easy to use.
Question answering. In NarrativeQA, we prompt GPT-4o with the question, the ground truth, and the model output to check for fluency and correctness. The flueney score is either 0 (incoherent or repetitive) or I (fluent), and the correctness score takes on the value of 0 (incorrect), I (partly correct), 2 (correct but not fully relevant), and 3 (correct and relevant). We take the product of the two the final score, normalizing to range of [0, 100).
Summarization. Following previous works (Kamoi et al., 2023; Zhang Bansal, 2021), we first decompose the gold summary into atomic claims and use GPT-4o to check if each claim is supported by the generation (recall) and if each sentence in the generation is supported by the reference sum- mary (precision). We then compute the Fl score from the recall and precision scores. Additionally, we ask GPT-40 to evaluate fluency (0 or 1) and take its product with the Fl score as the final score. In each step, prompt GPT-4o with handwritten examples.
5

---

Published as a conference paper at ICLR 2025
Multi-LexSun eeSSum
InfBench Sum
ROUGE? 2
Ours V
ROUGE ?
Ours
GPT-4o-08
22.2
170
Geminn--15-Pro
23.5
17.
31.2
34.5
Claude-3.5 S
22.5
20.7
20.4
20.2
19.8
43.8
41.
41.8
42.1
44.0
14.2
16.2
16.9
14.0
14.5
14.1
25.9
51.2
29.1
29.2
1lama-3.2-1B-lnng
211
225
222
22.2
22.
15.2
18.8
20.5
18.5
15.1
14.1
15.5
15.8
165
16.4
0.3
13
2.4
2.7
311
Llama-3. 3.2-3B-Ins
20.9
20.3
218
22.8
342
16.4
114
12:1
Llama-311-88IIsst
1amaa-31-708-Inst
23.3
23.4
41.4
05.8
177
18.3
Mistral-78-Inst-vOt
23.1
22.1
22.3
21.4
212
25.5
21.
17.
18.1
15.8
11.5
12.3
12.6
12.6
12.4
13
31
2.5
2.9
2.5
Mistral-7B-Inst-v0.
21.9
22.6
23.2
20.0
15.6
83.3
33.3
32.3
10.0
15.0
15.0
13.7
11.9
8.2
10.2
11.0
1.8
0.2
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
Figure 2: Comparison between ROUGE-I, Fl and our model-based evaluation metric on summa- rization tasks. Our metric shows more consistent trends: it reflects the performance gain on GPT-4o with increased input length, while ROUGE remains almost the same; our metric also clearly differ- entiates models while ROUGE shows little distinction.
Empirically, our reference-based model evaluation reflects more consistent trends, as shown in Fig- ure 2: (1) Llama-3. 1--8--8Bnne Inst achieves similar ROUGE scores to GPT-4o, while our evaluation re- veals a significant gap. (2) Our metric better identifies incoherent generations and shows lower performance for models with smaller context windows, such as the Mistral models. (3) Our metric exhibits a substantially more positive trend for GPT-4o as input length increases, whereas ROUGE- L remains within 2-point absolute difference. We further validate the model-based evaluation through human studies, which suggest our new metries strongly correlate with human judgments: for example, on BENCC Sum, our metric reaches human model agreement of Cohen's 's 0.91 for summary precision and K = 0.76 for recall. More details on the human studies are in $B.
2.3 ROBUST PROMPTING AND CONTROLLED EVALUATION SETTINGS
Robust prompting reduces noise and enables evaluation base models. Many long-context benchmarks require models to follow instructions and only support evaluating instruction-tune models (Zhang al., 2024b; Shaham al., 2023). However, many model developments do not incorporate instruction tuning (Chen et al., 2023; Fu et al., 2024), leaving these models reliant on perplexity-based evaluation synthetic tasks. To support long-context research efforts, we design our benchmark so that at least subset of the datasets accommodates evaluating base models.
Existing benchmarks mostly use zero-shot prompting (Shaham et al., 2023; Zhang et al., 2024b), which leads to inconsistent output formats, especially for base models. For example, the model may output long answer in RAG when a short answer is required. We add two-shot demonstrations in the prompt for all tasks to address this problem. 6 For long- -occmmettnntt QA and summarization, we replace the original document with a placeholder phrase to reduce the number of input tokens in the ICI. example. As shown in Table 8, both base and instruction-tuned models significantly benefit from the demonstrations.
Furthermore, we employ the length-instruccion-ennaaeee generation tasks (i.e., summarization), which has been shown to have substantially more consistent and reliable evaluations (An et al., 2024). As a result, we find that our reproduction of previous datasets, such as oo BeEEC QA tasks, better reflects the capabilities of LCLMs, as shown in Table 7. The use of demonstrations and improved instructions more accurately depicts how models perform in real applications.
evaluation from L-Eval for long-
Controlled input length and difficulty. An important dimension to consider when evaluating LCLMs is the input length L, as longer inputs can provide more information while challenging the model's ability to process distracting contexts. As we discussed in 12 we can control the input length for each task by either adjusting the number of retrieved passages, the number of demon- strations, truncating the document text to fit within the specified lengths. This allows us to study model performance at or beyond the length of current frontier LCLMS (2 128K).
"Except for ICI (the number of shots varies) and RULER (we follow the original formatting).

---

Published as a conference paper at ICLR 2025
As shown in Figure 1 and 2, HELMET provides rankings consistent with human perception of model performance. The diverse tasks, controllable lengths, and robust evaluation metrics and prompting enable thorough examination of LCLMs across multiple dimensions.
3 ANALYSIS
We evaluate 59 LCLMs with HELMET. To our best knowledge, this is the most thorough and controlled comparison of long-contex. models on diverse applications. These models cover closed- source models, such GPT-4, Claude, and Gemini, well open-source model families, such as Llama (Dubey et al., 2024), Mistral (Jiang et al., 2023), Phi (Abdin et al., 2024), and Qwen (Qwen al., 2025). We also consider models that use different architectures full-attention transformers (Vaswani et al., 2017), sliding-window attention (Beltagy et al., 2020), and hybrid models with SSM modules (Dao & Gu, 2024). We also benchmark position extrapolation models such as YaRN (Peng et al., 2024) and LongRoPE (Ding et al., 2024). We list all the models evaluated in Table 15. We evaluate each model input lengths: L E {8K, 16K, 32K, 64K, 128K). where
the number of 2lama-2 tokens (Touvron et al., 2023), and use greedy decoding for all models to ensure consistency. We randomly sample 100 to 600 examples from each dataset; more details are in $D
3.1 SIMPLE SYNTHETIC TASKS ARE POOR PREDICTORS OF REAL-WORLD PERFORMANCE
Many model developers rely on simple synthetic tasks, such as NIAH, for evaluating long-context language mod- els, but is unclear if these tasks accurately represent real-world performance. To this end, we calculate Spear- between synthetic and real- man's rank correlation -utned models. First, Fig- world tasks for 35 instruction-tun ure 3 shows that none of the synthetic tasks achieves an average correlation higher than 0.8. We also make the following observations.
NIAH
0.44
0.71
0.75
0.76
0.72
0.68
RULER MK
0.84
0.79
0.87
0.74
RULER MV
RULER All
0.51
0.77
0.85
0.79
0.83
0.75
Recall
0.61
0.74
0.85
0.80
0.85
0.77
Not all synthetic tasks are created equal. The original NIAH, which places a needle in the middle of unrelated essays and asks the model to retrieve it, exhibits weak cor- relation with real-world tasks: all correlations are < 0.8. Similarly, the popular RULER average score-which in- cludes not only NIAH variants but also synthetic aggre- gation, multi-hop tracing, and does not yield strong correlations (all < 0.85).
RAG
0.85
0.92
0.89
0.78
ICL
Cite
Re-rank
LongOA
Summ
Avg.
Figure 3: Spearman's 's rank correlation at 128K input length, calculated across 35 instruction-tuneet models.
We take a closer look at different RULER tasks and find that harder recall-type tasks are more reflective of real-world categories-fie for example, RULER MK, which places distracting needles around the target needle. Despite the overall low correlation, we believe these tasks can still serve useful sanity check during model development. We compile several such RULER tasks, along with ISON KV, to form the HELMET synthetic recall set (more discussions in $F.
Tasks with noisier, more distracting contexts better differentiate models. To understand why synthetic tasks exhibit weak correlation with real-world tasks, we plot the performance of different models NIAH, RULER MK (one of our recall tasks), and HotpotQA (one of our RAG tasks) in Figure 4. We use BENCH QA as a representative real-world task. We find that most models achieve either perfect or near-zero performance the middle and resulting in poor separability between models. In contrast, RULER MK, which introduces more distracting contexts, better distributes model performance between 0% and 100%, leading to clearer differentiation.
the original NIAH, leaving few data points in
RAG is better proxy for real-world tasks. Finally, find that RAG datasets, such as HotpotQA, consistently achieve higher correlation with other real-world tasks. Figure also shows that Hot- PotQA exhibits almost linear relationship with the QA dataset. Similar to synthetic tasks, RAG
7

---

Published as a conference paper at ICLR 2025
n=35 Spearman rho-0.63 p=5.le-05
n= Spearman rho- p=3.2e-09
n= Spearman rho- p-2.9e-12
InfiniteBench QA
20
40
60
80
100
20
40
60
80
100
0
10
20
30
40
50
60
NIAH
RULER MK
HotpotQA
Figure 4: Distribution instruction-tuned models' performance on ooBENCH QA with respect to NIAH, RULER MK, and HotpotQA.
tasks are easy to control and assess models' recall abilities. However, since all passages are retrieved and relevant to the query, RAG contexts are more distracting and therefore harder saturate.
3.2 DIVERSE LCLM APPLICATIONS CALL FOR DIVERSE EVALUATION
In long-context language modeling, realistic tasks often only used in isolation (Karpinska al., 2024: Li et al., 2024c; Dubey et al., 2024), which limits the understanding of LCLMs in a broader context. In this work, cross-examine model performance over wide range of real tasks, and find that different categories do not consistently correlate with each other, as shown in Figure 5.
show moderate correlation due their shared re- trieval component. However, the added complexity of generating citations in ALCE results in lower cor- relation with other Naturari RAG and passage re-ranking moderately correlate due the shared retrieval component As shown in Figure 9, generating correct answers and producing valid ci- tations are not strongly correlated, suggesting that instruction following and recalling facts within long contexts are distinct capabilities.
Recall
0.88
RAG
0.88
1
0.72
0.85
0.92
0.89
0.5
Cite
0.72
0.84
0.34
Re-rank
0.85
LongQA
0.82
0.92
0.72
0.87
0.85
0.51
Furthermore, some tations and in-context learning- not correlate well with other categories. Intuitively, summariza- tion tests for the model's ability aggregate infor- mation across the entire input, while ICL evaluates tasks from many examples.
with ci-
Summ
0.89
0.38
ICL
0.61
0.5
0.34
0.36
0.52
0.38
Recall
RAG
Cite
Re-rank
LongQA Summ ICL
its ability to learn Such capabilities are orthogonal to recall facts long contexts. Such capabilities are orthogonal recalling facts in long contexts. Therefore, model developers should evaluate across these distinct axes form more holistic understanding of model's 's capabilities (see additional analysis in $E.2).
Figure 5: Spearman rank correlation between
different categories at =22K.
3.3 MODEL PERFORMANCE ACROSS TASKS AND LENGTHS
We show the performance of iustruction-tuned models on HELMET at five different lengths in Figure 6, and the full results illustrated in Figure 10. We analyze the model performance across two critical dimensions of long-context language modeling: task complexity and input length.
Open-source models lag behind closed-source models on complex tasks. First, we consider the performance of frontier LCLMs at the longest input length of 128K tokens. We find that the closed- source models than ICL, the crosed-source models outperform the open-source models on all tasks. The gap is relatively small synthetic recall and LongQA, where the task is retrieve information from the context. There is stark contrast in the generation with citations and re-ranking performance. where the closed-source models are 30 to 40 absolute points better than the best open-source models.
GPT-4o-08 and Gemini-1.5-Pro- stand out as the strongest LCLMs. Other
8

---

Published as a conference paper at ICLR 2025
Recall
RAG
Cite
Re-rank
GPT-4
88.5
97.6
912
73.5
75.1
73.5
70.9
6A.7
46.0
28.1
12
87.8
76.1
54.7
25.8
GPT-4o-05
832
50.2
912
90.1
82.4
12.5
72.4
72.5
72
70.4
46.2
43.7
42.3
4244
40.4
05.3
77.9
61.7
57
47.7
GPT-4o
100.0
100.0
100.3
100.0
99.9
73.9
74.0
72.4
71.5
70.2
45.4
46.7
45.4
45.4
44.3
05.9
80.5
68.5
58.4
50.0
GPT-4o-mini
98.8
96.1
96.1
89.5
72.1
68.0
36.1
33.7
80.8
71.1
58.8
Claude-3 5-S-nnnnt
96.2
9/2
38.1
36.8
32.2
75.2
41.0
Gemini-1.5 Flash
93.2
49.7
49.4
80.8
Gemini-1 5-oPro
87.6
87.3
86,3
87.2
81.0
73.6
73.4
72.3
71.8
71.1
47
46.8
44.8
45.2
43.5
87.6
84.1
76.5
68.4
68.7
Llama-3.1- 1-88
10000
100.0
99.7
992
99.2
70.8
68
66.5
65.0
39.5
36.3
25.7
10.9
13.1
2.8
67.0
51.7
43.4
32.8
14.0
Llama-3. 3.100B
100.0
100.0
98.9
90.5
50.7
74.0
71.8
70.9
69.6
56.2
46.3
41.2
40.1
29.0
7.5
03.5
73.5
00.2
41.0
24.5
Llama-3. 3-708
100.0
100.0
98.6
98.6
81.8
72.7
63.2
56.1
40.4
38.6
10.9
84.8
75.7
65.0
Mistral-Nemo
30.1
68.6
32.7
MegaBean-Miarai
85.9
94.2
89.5
64.7
$7.0
22.0
56.4
40.33
38.0
25.6
Ministal-8B 8B
89.9
97.7
86.1
30.9
12.9
65.7
86.0
63.5
47.1
36.2
35.7
24.1
7.6
0.7
0.4
67.4
43.4
32.5
0.0
0.0
Phi-3-mini-128k
87.8
56.8
96.1
95.4
50.1
62.6
61.4
69.4
66.3
46.7
23.3
18.5
10.1
2.1
0.6
63.1
33.1
23.5
14.0
6.8
Phi-3-amall-
94.0
90.6
03.4
75.00
22.3
67.1
64.7
62.6
58.5
33.9
17.9
13.5
6.9
3.5
3.0
48.1
35.5
27.4
18.9
1.9
Phi-3-med-1288
58.9
78.3
69.3
24.5
05.1
62.3
44.9
39.8
26.4
52.6
35.7
26.1
Phi-3.5-min
22.2
35.4
26.5
Qwen2.66 5-728
103.6
100.0
98.4
71.4
38.4
72.5
72.0
76.2
63.1
43.0
42.7
40.8
38.0
25.1
8.0
86.0
75.3
80.3
41.0
24.5
Qwen2.5-7B-1M
96.9
98.5
94.1
92.0
86.0
67.7
58.3
64.9
64.0
09.0
36.2
17.9
8.6
9.5
4.0
71.3
57.4
45.9
32.1
24.0
Qwen2 Qwen2.5-148-.4 5-14-1MMM
100.0
36.9
88.4
99.2
$7.8
12.2
70.8
68.9
68.0
62.0
41.9
36.4
32.8
28.2
27.1
74.3
57.8
46.9
38.0
27.9
Jamba-1.5-MMi
98.4
97.3
03.2
50.0
66.2
64.1
61.8
57/6
14.4
8.8
48.3
38.4
25.4
ProLong
98.2
99.4
90.5
63.2
34.6
27.1
60.4
47.4
31.1
22.6
16K
32K
64K
128K
32K
64K
129K
BK
15K
32K
64K
128K
8K
16K
32K
64K
128K
LongOA
Summ
ICL
Ave
GPT-4
46
46.7
282
29:1
28.8
37.8
33.5
80.8
82.0
46.0
68.
57.5
48.3
GPT-4o-05
63.5
64.3
62.0
29.6
43.5
79.9
64.2 63.9
63.3
60.1
GPT-4o-08
65.6
59.3
34
82.6
86.8
66.2
GPT-4o-min
85.3
36.6
46
49.
54.8
28.7
33
36.1
30.0
40.8
77.3
79.6
81.0
81
81
61 2
6144
60.1
57.7
56,1
Claude-3.55 Claude-3.5SSonnel 5.S-onnet
292
23.1
27.2
126
12,6
28.9
33
36.5
35.6
36.8
87:1
88.7
69.8
892
61.0
58
502
49.5
44.5
36.4
Gemini-15 1.5miis-.5--aashh
33.9
42.4
46.3
52.3
57.3
25.0
28
38.5
38.3
39.4
74.2
68.7
56.6
39.7
28.1
62.1
62.0
60.2
56.11
62.1
Gemini-1 5-Prro
41.5
46.3
52.6
59.8
30.0
46.4
78.7
81.1
82.0
63.7
66.4
64.0
Llama-3.1
Llama-3.1-708
54.0
56.3
31.6
71.6
76.0
61.8
68.2
Llama-3.3-700
34.7
41.8
49.8
52.2
52.0
24.9
30.0
38.4
32.5
33.3
70.2
75.0
76.3
82:1
77.8
61
62:1
61
58.8
48.2
Mistral-Nemo
31.0
33.4
27.3
21.4
22.5
25..5
19.8
20.8
19.3
18.5
B5
76.0
80.2
R23
84.0
55.
402
38.4
28.7
25.7
MegaBeam-MMisral
20.7
30.7
32.2
34.9
37.5
20.3
22.
24.1
28.0
28.0
72:1
78:4
80.8
83.8
86.2
50
4
48.4
47.4
45.4
Ministral-8B
34.9
23.7
25.0
22.7
25.6
14.4
11.5
74.8
53.0
47.7
28.6
Phi-3-mini-1288
Phi-3-small-1228
25.0
28.5
38.0
41.8
27.5
18.6
20.3
24.4
6.6
67.8
79.1
79.6
472
45,7
4:32
24,9
Phi-3-med- 128k
22.7
21.2
19.1
15.4
26,9
21.4
20.3
23.0
27.6
24.8
62.6
05.5
67.8
73.0
73.2
51.0
46.5
41.0
37.7
29.1
Phi-3.5-min
24.7
28.4
29.3
30.8
28.5
19.6
21.1
22.9
24.9
28.3
63.8
70.2
75.2
78.0
79.5
48 8
48.
44.8
42.0
33.6
Qwen2.5 5-72B
382
40.5
44.4
41.5
38.0
282
36.8
31.4
76.4
80
81.8
65.6
83.2
63.4
633
61.8
51.4
38.2
Qwen2.557B-11
37.9
36.5
30.8
27.9
Owen2.5 5-14B-1M
79.8
58.
Jamba-1.55 1.5MMn
3358
40.0
48.1
51.0
54.2
17.1
17.3
17.7
18.1
81.0
87.0
89.
91.0
53.4
50.
50.8
48.3
46.8
ProLong
28.3
36.9
37.6
40.8
43.9
21.7
22.3
22.2
28.0
29
69.1
75
78.3
03.
86.9 8
54.4
53.7
51.1
503
49.4
8K 18K
32K
64K
128K
8K
18K
32K
64K
128K
8K
18K
32K
64K
128K
8K
16K
32K
84K
128K
Figure 6: Results of HELMET. All models are instruction-tunel and have claimed context win- dow of 128K tokens or more.
Performance degradation with longer inputs is category-dependent. Most frontier models largely retain performance on recall and RAG with longer inputs; however, even the best models experience significant degradation as context length increases on tasks like re-ranking and genera- tion with citations. As illustrated in Figure 7, performance degradation at longer lengths becomes more pronounced as task complexity increases from left to right. On generation with citations, open-source models completely collapse at 128K, while GPT-4o remains relatively stable. This underscores the importance of evaluating models on more complex long-context applications.
No clear winner across all categories. As we observe from the previous sections, the different cate- gories do not always correlate with each other. This is evident the varying oop-perfonnnng models across categories: for instance, GPT-4o excels in recall and generation with citations, while Gem- ini performs better in passage re-ranking and long-document QA. Furthermore, many open-source models outperform closed-source models on ICL, potentially because heavy instruction tuning neg- atively impacts ICL. We provide qualitative examples in Table 19. Thus, evaluating models across multiple axes is essential. In the appendix, we also present additional analysis, such the perfor- mance of positional extrapolation methods (SE.3), the lost-in-the the-middle phenomenon ($E.4), and the performance of Claude ($F.6).

---

Published as a conference paper at ICLR 2025
NIAH
RAG
Re-rank
Cite
GPT-40-08
100.0
100.0
100.0
100,0
100.0
73.9
74.0
72.4
71.5
70.2
86.9
30.5
68.5
58.4
50.0
45.4
46.7
45.4
45.4
44.3
G.
86.8
50.0
96.0
34.0
362
73.6
73.4
12.3
71.8
71.
07.6
84.1
76.5
08/4
59.7
47.2
45.8
66.5
46.2
43.5
Llama-3.1 3.1-8B-Inst
100.0
100.0
102.0
100.0
100.0
70.5
68.5
68.6
96.5
68.6
07.0
61.7
43.4
32.8
14.0
36.3
29.7
10.9
18.1
29
Llame-3.1 1i-me-3.--70--Ins
100.0
100.0
102.0
100.0
100.0
74.3
71.5
10.8
08.5
96.2
83.5
735
62.2
41.0
24.5
64.3
41.
40.1
28.8
75
Jamba-1 Miiiii
100.0
100.0
103.0
100.0
01.8
57.3
62.1
14.6
14.4
0.
5.6
Owen2-7B-Inst
Qwen2-57B-inst
100.0
100
100.0
87.0
36.0
65.4
12.7
38.9
36.0
11.3
4.1
38
8K 16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
Figure 7: Results of selected instruct models on various lengths and increasing complexity of tasks. Notably, Qwen2 relies on position extrapolation, while other open-models are trained at or greater than 128K context window.
4 RELATED WORKS
Long-context language models. Frontier models such as GPT-4 (OpenAI, 2023). Gemini (Team al., 2024b), and Claude claim to have expanded their context window beyond 100K tokens. In the open-source community, there are also efforts to train models with longer input lengths (Dubey et al., 2024; Fu al., 2024; AI al., 2024; Gao al., 2024), explore position extrapolation techniques (Peng et al., 2024; Chen et al., 2023; Ding et al., 2024), and experiment with efficient architectures (Beltagy et al., 2020; Bertsch et al., 2023; Gu & Dao, 2024; Dao & Gu, 2024; Yen et al., 2024;
Lieber et al., 2024, inter alia).
Synthetic tasks. Synthetic tasks are often used to evaluate LCLMs since they can be procedurally generated, enabling arbitrarily long input lengths and controlled "needle" placement (Tay et al., 2021; Liu et al., 2023). In particular, Needle-in-a-Hasstee (NIAH; Kamradt, 2024) inserts a "nee- dle" at specific depths of a long essay (i.e.. the haystack) and asks the model to recall the fact. Recent works have expanded upon it and designed new procedures to test different aspects of LCLMs (Hsieh et al., 2024; Li et al., 2024b; Levy et al., 2024; Arora et al., 2023; Laban et al., 2024; Goldman et al., 2024, inter alia). However, they do not study how results on synthetic tasks transfer to real ap- plications. In contrast, we evaluate both synthetic and downstream tasks and investigate how they correlate with each other.
Long-context benchmarks. As discussed in the main paper, existing benchmarks either are limit to relatively short context lengths (Shaham et al., 2022: 2023; Li et al., 2024a; An al., 2024; Bai et al., 2024; Dong et al., 2024; Kwan et al., 2024) or lack rigorous evaluation methods (Zhang et al., 2024b; Yuan et al., 2024). Many works instead focus on specific domains, such as question answering (Wang al., 2024b; Karpinska et al., 2024; Wang et al., 2024a;c), in-context learning (Li et al., 2024c: Bertsch et al.. 2024; Xu et al., 2024; Anil et al., 2024; Agarwal et al., 2024), summarization (Chang et al., 2024; Kim et al., 2024; Shen et al., 2022), RAG (Lee et al., 2024). In this work, we construct a comprehensive benchmark that tests across diverse downstream tasks at long input lengths and also present unified comparison and analysis across 59 LCLMs.
CONCLUSION
In this work, we first identify the shortcomings of long-context evaluation settings and existing mnechmarks-ove--eliiane on synthetic tasks, limited coverage of realistic applications, and un- reliable metrics among others. We seek to address these issues by constructing HELMET, an application-cent benchmark with diverse domains and reliable evaluation settings. We then present comprehensive evaluation of 59 frontier LCLMs across multiple dimensions, including different tasks, input lengths, and model types. Our analysis shows that synthetic tasks poorly pre- dict downstream performance and that different categories exhibit distinct capabilities and trends. Thus, evaluating models on a diverse set of world tasks is essential. Furthermore, open-source models still lag behind closed-source models complex tasks at longer lengths. Finally, hope that our benchmark and comprehensive evaluation serve as a valuable resource for future research in long-context language modeling.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have publicly released all of our code and data to ensure reproducibility. Experimental settings outlined in the main text ($3) and the appendix ($D), and we will provide additional details and instructions in the code repository. Furthermore, we plan on releasing the model generated outputs and statistics to enable further analysis and comparison.
ACKNOWLEDGMENTS
We would like to acknowledge Mengzhou Xia, Howard Chen, Xi Ye, Yinghui He, Lucy He, Alexan- der Wettig, Sadhika Malladi, and other members of Princeton Language and Intelligence for their helpful feedback and discussion. We also thank Adithya Bhaskar and Joie Zhang for helping with validating the model-based evaluation. Tianyu Gao is supported by an IBM PhD Fellowship. This work is gratefully supported by NSF CAREER award (IS-2239290), grant from Intel, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chur Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi. Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce. Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang. Yu Wang. Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: arxiv..iv.v.//vv. org/ abs/ 404 14219.
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Be- hbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, 2024. URL https: /tts:///axiiiii wwg//bggaaeo...ooooooooo.....o.. / 2204 11018.
01. AI, :. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang. Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01 2024.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14388-1441) Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:
//ac /aclannhologyy org/ / 22024.024222 1.24.a. acl long.
Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024.

---

Published as a conference paper at ICLR 2025
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language mod- cls, 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: https///pppeeveeeeeee net.ewwwweeeeeeee.eeeeeeeeeeee.e wwt.viiwwwee/orummt formmformmm?c hSyW5go0v8.
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen tau Yih. Reliable, adaptable, and attributable language models with retrieval, 2024b. URL wwtphttp///// /ttp///aavvvv//. wwtp/r///////eo/ee///////eeeeeee 2abs/24033 03187.
Yushi Bai, Xin Lv, Jiajie Zhang. Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. LongBench: A bilin- gual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: /aa/laaannttt /atpaaaaaannooooo.aaaoaa orgoor////////o 2024. acl soll-onge 72.
Payal Bajaj, Daniel Campos, Nick Craswell, Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: human generated machine reading comprehension dataset, 2018. URL httpt:// http:p:s//axxxv.ooo orgg /1l1 09268.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document Transformer. 0555 2020. URL https: //t////axiii /tt.//ivvvvv.////////..e///.aavl httppp///wwwwwwoo////o////////// ab00 05150.
Amanda Bertsch, Uri Alon, Graham Neubig. and Matthew R. Gormley. Unlimiformer: Long-range transformers with unlimited length input. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https: wpwp//weenneeeeee//e/ee/..ooe.wm net www.::iid-wwwww.......
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. In-context learning with long-contex: models: An in-depth exploration. In First Workshop on Long-Context Foundation Modely @ ICML 2024, 2024. URL wwt.h:t/////ee http:///ppeeeeeeeeeeeee forum? 40=44Amm7
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Risenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037, 2022. URL https::/p//... /ttp///axvvvv///v//////eee/..wvv org www.oorgg/......... http:/df/2211/ 22lff/221..02 080377 .ddf.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https: / htppppaprs..// www.sspcpppaaa.ecmmmmem httpc///ppppe soc/paper/2020 2a.00////hhss/ 1447c0d6bfcb45ee 96741 8bfbaacl 42f64a Abstract. html.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient in- tent detection with dual sentence encoders. In Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, lñigo Casanueva, and Rushin Shah (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Comversational AI, pp- 38 48-45, Online, July 2020. Association for Computational Linguisties. doi: 10. 10.18653/v1/2020 nlp4convai- 1.5. URL https: / //acc//aaccce aclanthologyy org/2020 www.convaiiiiinmmmmmmmmmmm
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: A systematic explo- ration of leng-tength summarization in the era of LLMs. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: /ttp////pee//eeeeee/eeeee/eeeeee net forum?ide 7Ttk3RzDeus
12

---

Published as a conference paper at ICLR 2025
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. In Regina Barzilay and Men Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870- 1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1171. URL http:::/p//./.....t.. //acaclanmn antho logy. hrtp////11/1117700
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the trec 2019 deep learning track, 2020. URL https:://arat org abs / /2003.0033. 07820.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023, URL https:: //tp:///axiii wwgg 01.30700699
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL httpp::/arxiv...r.tt/ org/ 2abbs/2455 21060.
Daniel Deutsch, Rotem Dror, and Dan Roth. Re-examining system-level correlations of auto- matic summarization evaluation metrics. In Marine Carpuat, Mathe-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 5038-6052. Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10. 10188663311222..mmm moaimmaaii222 URL https: https/////aaaattooaa...o org/2022. naacl-maing 442.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending I1m context window beyond 2 million tokens, 2024.
URL https:tt///// / /ttx/aaxvvvo.oo/ org/ abs 2470..3753.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. BAMBOO: A comprehen- sive benchmark for evaluating long text modeling capacities of large language models. In Nico- letta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation ELLECCCOLINEE 2024), pp. 2086 2099, Torino, Italia, May 2024. ELRA and ICCL. URL https: ac anthology org/ 2024 wrer- ma in..cmmaiin..................... 188.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar. Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone. Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong.
13

---

Published as a conference paper at ICLR 2025
Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic. Brian Gamido. Britt Montalvo, Carl Parker. Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute. Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk. Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Baan-Baptste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A. Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu
14

---

Published as a conference paper at ICLR 2025
Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vitoo, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL /tapss://axxvv.ooo
abs/ / 2407 21783.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465-6488, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023..mmm. 1/2023.emnlp main. 398. https: ////accaantttt www.aaaaaooooooooooooooooooooooa org 2223. emnlp-ma i 38.
Tianyu Gao, Alexander Wettig. Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2024. URL https: /awxiv.vvvvv.///// org/abs////ooooooooooo..../... /2100 02660.
Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. really long context if all you need is retrieval? towards genuinely difficult long context nlp, 2024. URL https:: /aaxiivaaivivvvv/ org/ abs/ 00402.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. News Summarization and Evaluation in the Era of GPT-3, 2023, URL https: http///axvvvirr.e/ee 2209. 12356.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling Conference on Language Modeling, 2024. URL IRthhttps://ppeereeiee net forum?id- tEYskw1VY2.
modelngc with selective state spaces. In First
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? In First Conference Language Modeling, 2024. URL https: /twpr//ieep/eeeee...///om/. wwt. forum? 1d=kIoBbc76Sy.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 18.18653/v 1/165//11111-114., URL https::////// htt/a/aaaaannooogoo/oooo /ortthoologyogg/////oooo P17-1147.
Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. WiCE: Real-Worke Entailment for Claims in Wikipedia. arXiv preprint arXiv:2303.01432, 2023. URL https://arxx org/abs/ 22333. 01 432.
Garrett Kamradt. Needle in haystack pressure testing 1lms, 2024. URL https://github. com/ http//kkmaaatttnnm Meaaraatt/LLMTtttteeaeeeeaeaaaa
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A "novel" challenge for long-context language models, 2024. URL org/abs/ 2406. 6.6.6.66........ 64.
Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit lyyer. FABLES: Evaluating faithfulness and content selection in book- length summarization. In First Conference on Language Modeling, 2024. URL https:// openreview. wttp://.eee/uouueeeee htte://oorrmmiieeeessssaaaa
15

---

Published as a conference paper at ICLR 2025
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10.1 00023.
URL https::p////... / /acc/cccccc/ antaon oog.ghogggoooooooo...ooo.oo Q1 - 1023.
Kalpesh Krishma, Erin Bransom, Bailey Kuehl, Mohit lyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. LongEval: Guidelines for human evaluation of faithfulness in long-form summa- rization. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Confer- ence of the European Chapter of the Association for Computational Linguistics, pp. 1650-1669, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 18653vlll 121/.
1023.eacl-main. 121. URL https: /aw.aaaannnnnooooaaaaaaaaaooaott org / 2023.02.33222
Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Yuxin Jiang, Lifeng Shang. Qun Liu, and Kam-Fai Wong. M4LE: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15568-15599) Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 1/22244///22222aclllll 832. URL ht https:: /aals://aaaaaa anthhloghhooo org/ 2024. 11244.ac-lomm long. 832/-
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Ming-Wei
Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/ URL https: http:///aaaaatttoaago orgg 1110Q0190222
Philippe Laban, Alexander R. Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of haystack: A challenge to long congeconextt lims and rag systems, 2024. URL https://arxix
org/abs 2407. 01370.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P Processing (EMNLP-IJCNLP), pp. 13 1311-1316, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.1865//1/DD1--1111. URL http:::s//allaathhoooooo org httpl/gy..orgg191111/1 9
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. Can long-context language models subsume retrieval, rag, sql. and more?, 2024. URL https: //axivv.//iiviva//a org/ 0bbs/2406.135 13121.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length the reasoning performance of large language models. In Association for Computational Lin- guistics (ACL), 2024.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE: Can long-context language models understand long contexts? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pp. 16304-16333, Bangkok, Thailand, August 2024a. Association for Com- putational Linguistics. URL https: htt.:ttt///caaathhlllee
accl long. 859.
Mo Li. Songyang Zhang. Yunxin Liu, and Kai Chen. Needlebench: Can 1lms do retrieval and reasoning in million context window?, 2024b. URL 11963.
www./orrg/aaoo abs///b//// abbs/2407.
Tianle Li, Ge Zhang, Quy Due Do Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. CoRR, abs/2404.02060, 2024c. URL https: / /oii. www.door///...ooooooo..ooooo..oe org/o0. 10. 48550/ / 2404. 02060.
16

---

Published as a conference paper at ICLR 2025
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. URL https: http///acaaatttttg/o// .org/ C02-1150.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Frez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba language model, 2024. URL https: http://aaxvv.raoaaa/////e/ 240.. 19887.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL hstps: / wwcccannhhloooooo ogy..oooooo.oooooo.....o.oooo... waw.oogy......o..... 4
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2023. URL https://aag
Semanticscholie ort.://pccCooooCC........ wot.oopup...:20000505000 259360665.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking natural lan- guage understanding services for building conversational agents. ArXiv, abs/1903.05566, 2019.
URL https: /ttp///ppi.aa semant inschooaac org htrp/CorrpsIDD.. 7T0IID::7663333 76660838.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non- parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Asso- ciation for Computational Linguistics (ACL), pp. 9802-9822, Toronto, Canada, July 2023. As- htt55///1/12222alllooooogg
sociation for Computational Linguistics. doi: / //aclanthology org/2023. acl-long...... 5.
URL https:
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zeulemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-1 1164, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/vI/ 2022.emnlp- 759. URL https: / ac www.annnnooooyyy org. wttp://o///022.....mmmmmmaaaaaaa wmn. 759.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hermandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wos Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Empirical Methods in Natural Language Processing (EMNLP), pp. 2022. URL https: / http://aaaaaaaltooaaoeeeoo org/2022. emnlp-main. 669.
OpenAL. GPT-4 Technical Report, 2023.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in- context: Disentangling task recognition and task learning. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8298-8319 8319, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 10.18663/v1/222. 2e122...innins--ee findings-acl.527. URL https: htta//aaaanaclloooooeeee org/2023. findings acl.-acl... 527.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///ppeeeeee.e.eeeeee net wtt.l//wwweetttttmmmmeeeeee
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 5523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 1/255/ wawaacllmmmi.2.... URL https: //cc anthal ogy. .rrg 2021 naacl -aa 200.
17

---

Published as a conference paper at ICLR 2025
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https::///// aaxiv. org/ abs/ 241. 15115.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ ques- tions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), 2016. URL https: http://aaaaaatoooooco
1264/.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Brown, and Yoav Shoham. Parallel context windows for large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceed- ings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6383-6402, Toronto, Canada, July 2023, Association for Computational Lin- URL ht htsps:tptt... / //a//aaaantttt hww.aannoooooooo.oooooooooo org/2023.
Amnon Shashua, Kevin Leyton-Brown,
guisties. doi: 10. acl-long. 352.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3(4):333-388) apr 2009. ISSN 1554-0669. doi: 10.1561/1500000011. URL wtthtttt////e/ / /t/./oiiooooo/ org 100 wet...111111110000
Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs. arXiv preprint (xXiv:2255/12666 2022. URL https: // //arxiv.orgg org/ abs 2205 12665.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: /accllaaaaa www.anthhooogy... org/2022. emr.ppmme ma in.iinn 823.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/11/2033. wwt.:yy...ggg22o0e fi findings emmnlp.
findings-emnlp5366. URL patts://acllle http:////ccaaaatoll.....oo/// wtw.hhoooogg..o///223...
536.
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. Multi- lexsum: Real-world summaries of civil rights lawsuits at multiple granularities. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
URL https://////.. / http///peeeeeeeeeeeeeeeeeeeeeeet net.wwwwwwwwe.eeeeeeoomeee..eeee fooe/forroe Forummidd---- 1dd
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wa Wei Chang. ASQA: Factoid questions meet long-form answers. In Proceedingy of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273-8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. https: https/////aaaaathhooooee org/2022 emnlp-man cosp-mmain... 566.
URL
Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2021. URL httptt 04 69.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 14918 144377, Singapore, December
18

---

Published as a conference paper at ICLR 2025
2023. Association for Computational Linguisties. doi: (0.18653/w122223.mmml main.923. URL attps://aclantholll org/2023 emnlp-main. .rt../20...e.....mmma....... 923.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: / //openreview... net/ ho///formmm/// wwwoommiiiiiyyeeee -Vc-gggrcee.k.kk...
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptisee Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai. Anja Hauth. Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler. Timothy Lilli- crap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hen- nigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du Ed Chi, Heng-Tze Cheng, Eric Ni. Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li. Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen. Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapa- thy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Mar- tin Baeuml, Zhifeng Chen, Laurent EI Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lu- cas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Sehucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hart- man, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Re- itter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Bala- guer, Jackie Xiang, Pidong Wang. Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Gana- pathy, Smit Sanghavi, Ajay Kannan, Ming- Wei Chang. Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoe, Alex Kaskasoli, Séhastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hycontack Lim, Sarah Hodkinson, Pranav Shyam. Johan Ferret, Steven Hand, Ankush Garg, Tom Le Painc, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipan- jan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim. Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Lan- don, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev. Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fer- nando, Will Hawkins, Behnam Neyshabur Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang. Fan Yang, Shuo yin Chang, Paul Komarek, Ross Mcllroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc,
19

---

Published as a conference paper at ICLR 2025
Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang. Joshua Maynez, Mary Phuong. Taylor Tobin, Andrea Tacchetti, Maja Trebacz. Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao. Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexi- ang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Toma- sev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Chang- pinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang. Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo. Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang. Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kass- ner, Subbrajit Roy, Ethan Dyer, Víctor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy. Elspeth White. Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thortoo,, Fangxiaoyu Feng, Wojeiech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Unlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pel- lat, Vladimir Feinberg, James Cobon Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Has- san, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani. Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Al- bert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Nor- bert Kalb, Vamsi Bedapudi, Adam Kurzrok. Shuntong Lei. Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliamo, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling. Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lom- briser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg. Yifan He, Oleksii Duzhyi, Anton Älgmyr. Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna. Aleksandr Chuklin. Josie Li. Carric Spadine, Travis Wolfe, Kareem Mohamed, Sub- habrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay. Akanksha Mau- rya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana- Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejezuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Mal- colm Rose Harriott, Mudit Bansal, Alexei Robsky. Geoff Bacon, David Greene. Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih- Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik
20

---

Published as a conference paper at ICLR 2025
Shah, Hung Nguyen, Noah Ô Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang. Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Chum Lee, Wojeiech Fica. Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Doo- ley. Srividya Pranavi Potharaju, Fileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry. Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranck-Shraded An- drew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski. Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mu- jika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Sid- dharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LVV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoom, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Woj- ciech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung. Kai Yang. Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanezyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieil- lard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, John- son Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao. Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikotaj Rybinski, Ashwin Sreevatsa, Jennifer Prendki, David So- ergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Di- ana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Mar- cus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijaykkumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene. Duc Dung Nguyen, Paula Kurylowicz. Cassidy Hardin, Lucas Dixon, Lili Janzer. Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bu- lanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Han- nah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang. Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHosseen Bateni, Den- nis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp. Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattic, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver
21

---

Published as a conference paper at ICLR 2025
Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thomton,, Jordi Pont-Tuset, Pradyumna. Narayana, Jing Li, Saaber Fatchi, John Wieting, Omar Ajmeri, Be- nigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang. Yeqing Li, Nir Levine, Ariel Stolovich, Reheca Santamaria-Fernandeer Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlic Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christo- pher A. Choquette-Chon
Yunjie Li, TJ Lu, Abe Ityyceeiih,, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna
David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antomio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju. Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Ram- mohan Belle, Lei Wang. Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty. Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang. Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Gold- enson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz. Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Miku- lik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M.. Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong. Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko. Christoph Him- schall, Ashwin Sethi, Xingyu Federico Xu. Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive. Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mu- dit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim. Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024a. URL https: /ttp///axivv/vv/////v//////aaaaa org/abs/////eoooooooo/ooo/aob 2312 11805.
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng.
22

---

Published as a conference paper at ICLR 2025
Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang. Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos, Mahdis Mahdich, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang, Dasha Valter, Maja Trebacz, Kiran Vo- drahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Sid- dhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent El Shafey, Junhyuk Oh, TTm Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becea Roelofs, Sid Lall, Amit Marathe, Betty Chan, San- tiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho, Sebastian Riedel, Karel Lenc, Khin Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin, Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai, Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da- Woon Chung. Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Brt-Orgil Batsaikhan, Ma- teo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoc Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang. Fangyu Liu, Lampros Lam- prou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang. Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominík Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope, Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May, Zhen Yang, Johan Schalkwyk, Christina Butter- field, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Mar- vin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thornton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap. Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann. Xiang Zhou, Diana Mineu, Yujia Li, Ravi Addanki, Kalpesh Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, Hyun- Jeong Choe, Alex Tomala, Chalence Seffranek-Shrader Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wei Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićevíć, Do- minik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, lain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky, Thanumalayan Sankaranarayana Pillai, Dan Hurt, An- geliki Lazaridou, Xi Xiong. Ce Zheng, Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang, Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros, Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang. Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia El Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos. Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, lean-Baptiste Lespiau. Josh Newlan, Zeyncep Cankara, Soo Kwak, Yun- han Xu, Phil Chen, Andy Coenen, Clemens Meyer, Katerina Tsihlas, Ada Ma, Juraj Gottweis,
Krishna,
23

---

Published as a conference paper at ICLR 2025
Jinwei Xing. Chenjie Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes-Fitt, Heng Chen, David Reid, Keran Rong, Hongmin Fan. Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilie, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang. Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, El- naz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Su- san Zhang, Michael Azzam, Khe Chai Sim, Sergi Caelles, James Keeling, Abhanshu Sharma, Andy Swing. YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittycheriah, Soheil Has- sas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang. Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Due Dung Nguyen, James Svensson, Le Hou, Sarah York, Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp, Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang. Cosmo Du, Bo Li, Zizhao Zhang. Mariko linuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Blo- niarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carrie Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon- Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, An- ton Briukhov, Neil Houlsby, Mia Glaese, Balaji Lakshminarayanaan Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu, Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Solomon Chang. Julian Schritwiiesrr, Elena Buchatskaya, Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux, Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang. Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Ruben- stein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aish- warya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao, Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Fe- lix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac Eric Ni, Alexander Pritzel, Adam Iwan- icki. Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srini- vasan, Hannah Muckenhirn, Honglong Cai, Amol Mandhane, Mukarram Tariq. Jack W. Rae, Gary Wang, Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Gar- rette, Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang. Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hut- ter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee. Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thorton,, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong, Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Bums, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, lonel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C.
Amold,
24

---

Published as a conference paper at ICLR 2025
Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng, Hui Li. Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Centamaria-Fermandee Wojeiech Fica, Xinyun Chen, Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srini- vasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando, Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le, Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong. Boyu Wang, Sujeevan Rujayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Di- ane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann- Rice, Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjic Zhou, Lilly Taylor, Jennifer Prendki. Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hi- lal Dib, Jeff Stanway, Frank Perbet, Neje Trdin, Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz Bahargam, Mark Epstein, Sujoy Basu, Li Lao. Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang. Amir Globerson, Grant Uy. Andrew Bolt, Olivia Wiles, Mi- lad Nasr, Ila Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galie, John Carpenter, Kevin Villela, Haibin Zhang, Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mah- moud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan Popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora, Dessie Petrova, Danielle Eisenbud, Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni, Yannis Assael, Thomas Brovelli, Prateek Jain, Mibajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Ros- gen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi, Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Chinmot. Solomon Kim, William Zeng. Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Choquette-Choo, James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velie, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham, Kathryn Tunyasuvunakool, Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri. Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim. Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham. Evan Rosen, Lisa Wang, Qijun Tan, David Engel. Francesco Pongetti, Dario de Ce- sare, Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan I.i, Asaf Aharoni, Trieu Trinh, Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana, Austin Matthews, CJ Carey, Matthew Johnson, Kremena Gora- nova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Man- ish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Bop- pana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Ko- rehemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq lqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chai- tanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev, Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang, Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo, Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang
25

---

Published as a conference paper at ICLR 2025
Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrini, DongHyun Choi, Ivan Philipps, Ziyue Wang, Ioana Bica, Ankush Garg, Jarek Wilkiewicz, Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger. Sammy Jerome, Ab- hishek Chakladar, Alek Wenjiao Wang. Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kepa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejezuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd AI Merey. Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukeuoglu, Jeffrrey Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024b. URL https:://ariis wwwp//t///////oooo///.o../.o..s/ 2403. 05530.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhl- gay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedi- gos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zus- man, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg. Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz. Alon, Ro'i Belson, Roi Co- hen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Yehoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba- 1.5: Hybrid consformer-mamleee models at scale, 2024c. URL https: / arxii.viivv....///aaaaaaaaaavvva org/ abs/ 2408. 12570.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NIPS), 30, 2017. URL https: //pp:e///ppppp//p wwwp.cc./...com...m............t pccppppe///117//// html.
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-LEval: Eval- uating long-context LLMs with length-adaptable benchmarks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I; Long Papers), pp. 37 3712-3724, Mexico City, Mexico, June 2024a. Association for Computational orq/
Linguistics. doi: 10. 2024. www.nnaalllnnn ong. 205.
URL https:
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Xiangkun Hu, Zheng Zhang, Qian Wang, and Yue Zhang. Novelqa: Benchmarking question answering on documents exceeding 200k tokens, 2024b. URL https: http://wwxxv///aaaaa//// 24C3.12766
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang. and Yongbin Li. Leave no document behind: Benchmarking long-contess LLMs with extended multi-doc QA. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Em- pirical Methody in Natural Language Processing, pp. 5627-5646, Miami, Florida, USA, Novem-
26

---

Published as a conference paper at ICLR 2025
ber 2024c. Association for Computational Linguisties. doi: 10. 10.18653/w1120044cmmpppppo org/2024 emnlp-main. 322/.
URL ptttps://aclanthhllooooooo
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023. URL https:/ /tappp://axxiii /ttp/aaxxvvv..///.//ee.//a.a.llt
http:///wwwwww.o///////////o wwwp:///////////e///// 2200. 03846.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of- the-art natural language processing, 2020. URL https: atwiv.iiiivv....v org/ abs/ /000s//9910 1910.03771. 03771.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023.
Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long- congecontext language models with lifelong ICL and task haystack. In First Workshop on Long-Come Context Foundation Models @ ICML 2024, 2024. URL https:tt///// https////ppeeeeieeeeeee net forum?ide (atmmmddd--tttttinnn
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei. Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: aarxivvvvv. orgg als// 200o/24407 2407.10671.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang. Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-11m technical report, 2025. URL https: /arxiv......../////a/ org/ abs/////////// / 22012551112 15383.
arg/abs//
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2369- 2380, Brussels, Belgium. 1259. URL https: / www.cannnoooogoo. wtt/ 1259.
2018. Association for Computational Linguistics.
doi:
Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel con- text encoding. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pp. 2588-2610, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 1/655//1/2022..a---22 142. URL https: / /acl anthh 22042024. acl-longe 42.
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao. Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. Lv-eval: balanced long- context benchmark with 5 length levels up to 256k, 2024. URL httpt:tpp//// http///axvvvv/// wwt.v.o.rg//////// 0.02..5133
Shiyue Zhang and Mohit Bansal. Finding a balanced degree of automation for summary eval- uation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 6617-6632, 2021. doi: 11118663/112222.emmll wet000/e.mmppmmm........... m31..531.
URL
org
2021 emn emnlp-main 531.
27

---

Published as a conference paper at ICLR 2025
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval, 2024a. URL https:: settps:/arrivv /aaaiiivvvv wwtp:ooggggoooooooooooooooo 2407 . 9669.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. sobench: Extending long context evaluation beyond 100k tokens, 2024b. URL https: / arxiv. org/ /bgg/bs///// 241.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: / //openrevi ew. net. forum? d=uccHPGDlac.
Judging LLM-as-a-juut
28

---

Published as a conference paper at ICLR 2025
A COMPARISON WITH OTHER BENCHMARKS
Due to the lack of large-scale human studies on long-context language models, discerning the true rankings of LCLMs remains challenging. Therefore, we rely previous works and empirical obser- vations to guide our expectations. Specifically, the Gemini 1.5 report demonstrates that larger mod- els consistently outperform smaller ones at handling long contexts- qualitatively and quan- titatively across the Gemini, GPT, and Claude model families (Section 5.2. Tables 4 and 5 Team al., 2024b). Although their evaluation suite not publicly available, these results align with com- monly observed patterns in language model performance. Given that obtaining ground truth rank- ings through human evaluation is prohibitively expensive and practically infeasible, relying on such well-established patterns is reasonable. In this section, we further examine HELMET's improve- ments over existing benchmarks through direct comparisons of model rankings and performance, supported by careful ablation studies.
A.1 RESULTS
We build upon previous datasets and refine evaluation settings to better reflect model capabilities. Specifically, we evaluate models on coBENCH, another benchmark that evaluates long-context at 128K tokens. The results reproduced from the original authors' code are shown in Table 6. The numerical results for Figure I are presented in Table 5.
Upon closer inspection, we find that set of open-sourced models' ranking and performance on HELMET and BECH in Table 9. In addition to the previously noted performance discrepancy between Llama-3.1 1--maa-.--88-Inntteee Instuunttt and Ilama-3. 1.1.-..------iii-i
we observe that all Llama-3.2 models degenerate performance on BBBEN at input length 128K. However, on HELMET, the Llama- 3.2 models, especially Llama-3,22 qualitative examination, we find that the Llama 3.2 models are able to produce coherent and useful generation long contexts with better prompting strategies from HELMET, such adding in- context learning examples.
rank well against other open-source models. Upon
Our ablation study few-shot demonstrations in Table 8 shows that 2-shot demonstrations sub- stantially improve base model performance, better reflecting real-world usage patterns. As shown in Table 7, improved instructions and prompting enable smaller Llama-3.2 models compete ef- fectively with larger models. Thus, HELMET provides provides a better and more realistic reflection of how these models would be used in practice over previous benchmarks. These findings demonstrate that HELMET provides a more accurate and practical assessment of model capabilities compared to previous benchmarks.
Table 5: Results with 128k input length across different benchmarks.
Model
Claimed Length
NIAH
RULER
BENCH
HELMET
GPT-4o-mini GPT-4o-08
128000
100.0
80.8
$1.9
$4.1
128000
100.0
93.3
57.1
Gemini-I Demini-I.5-Flash Gemini- 1.5-Pro
1.5-Flash
1048576
100.0
86.6
50.8
50.7
2097152
45.3
65.3
60.3
62.7
Llama-3, 1amma-311-80 I.--
131072
100.0
81.3
44.1
47.0
1.mma-3.1-70B-
131072
100.0
75.8
39.7
49.3
A.2 DISCUSSIONS
Many existing long-co conteco language modeling benchmarks studied in isolated settings, such as synthetic tasks (Hsich et al., 2024; Li et al., 2024b; Levy et al., 2024; Laban et al., 2024), in-context learning (Anil et al., 2024; Bertsch et al., 2024; Li al., 2024c; Agarwal et al., 2024), summarization settings (Lee et al., 2024). However, these works lack unified evaluation across diverse downstream tasks.
(Chang et al., 2024; Kim et al., 2024), and retrieval-augmente
There are also benchmarks that sought to unify different datasets, such as BoBENCH (Zhang et al., 2024b), LongBench (Bai al., 2024), L--val (An et al., 2024), IV-Eval (Yuan et al., 2024), and
29

---

Published as a conference paper at ICLR 2025
Table 6: Results on Be al., 2024b). We report numbers from running the original authors repo: https: //gttuu.bb. com/ monn/ppeeMM In inffnntetn initeBench. The original code did support evaluation of the Gemini and Llama models the time of access, so we evaluate them by following the script and template for GPT-4 and open-source models, respectively. We exclude coding tasks from the evaluation suite since out of the scope for general long-context language modeling tasks.
Model
MC
QA
Sum
Diag
Cale
Find
Number
PassKey
KV
Avg.
GPT4
75.9
23.7
10.7
0.0
47.1
100.0
100.0
58.8
44.9
GPT--4-005
88.2
37.9
23.7
28.5
0.0
58.6
100.0
100.0
94.2
55.4
GPT-4o-08
86.5
26.0
21.5
51.0
0.0
58.9
100.0
100.0
99.8
57.1
GPT-44omini
78.2
19.1
24.8
21.5
0.0
69.7
100.0
100.0
80.4
51.9
Gemini- 1
42.1
30.0
0.0
47.4
100.0
100.0
31.4
50.8
Semini-1.5-Prp
77.5
27.7
29.0
97.5
0.0
58.0
100.0
100.0
70.4
60.3
Llama-3,1-4
56.8
8.8
14.3
0.5
0.0
22.0
99.7
100.0
18.8
33.0
-lama-3.1-B--1-
67.2
15.5
26.7
23.0
0.0
33.1
100.0
55.0
44.1
Llama-3.1-700
17.5
0.0
32.3
100.0
100.0
13.4
35.1
Llama-3.1-22 1ama-3.1-770B-Ims.
75.5
23.3
18.0
0.0
43.1
99.7
100.0
2.6
39.7
Llama-3.2 3.2- BB
2.2
1.4
8.6
4.5
0.0
0.0
1.5
0.0
0.0
2.0
IJama-3.2 3.2-1 B-Innstruct
3.5
1.4
12.5
5.5
0.0
0.0
0.0
0,0
0.0
2.4
Llama-3.2 3.2-3B
1.3
1.2
7.6
4.5
0.0
0,0
1.7
0.0
0.0
1.9
Llama-3.2-3BB 2amm--32-3B-Instruue.
2.2
1.6
13.3
4.5
0.0
0.0
1.7
1.7
0.0
2.8
Table We compare the results on ooBENCH multiple-choice (MC) and question answering (QA) tasks between the original authors and HELMET. In our implementation, we leverage refined prompts and carefully crafted parsing scripts ensure robust and fair evaluation. For example, we find that Llama-3.2 models are actually much better long-context language models than <BENCH would suggest.
Original
HELMET
Model
Claimed Length
MC
QA
MC
QA
GPT-4o-mini
128000
88.2
37.9
71.0
44.5
GPT-4o-05
128000
86.5
26.0
76.0
GPT-4--08
128000
78.2
19.1
73.0
47.9
Gemini- 1.5ini--..5-ae F.5-Fashh
1048576
76.0
42.1
79.1
51.7
Gemini- 1.5-Pro
2097152
77.5
27.7
83.3
50.9
Llama-3. 3.1-BB
131072
56.8
8.8
54.0
38.8
Llama-3. 3.--BBBBBBBBB---- 1-8B-Inst
131072
67.2
15.5
49.0
40.3
Llama-3.2-13 3.2-IB
131072
2.2
21.0
Llama-3.2-14 3.2--BBIIstt
131072
3.5
16.0
13.3
Llama-3.2-42 3.2-3B
131072
1.3
1.2
36.0
31.0
Llama-3.2 3.2-BBBInBIB
131072
2.2
1.6
42.0
20.6
ZeroSCROLLS (Shaham et al., 2023). Many of these benchmarks are still limited by the context length, evaluation metrics, or both. Most similar our work, BoBENCHZaang et al. (2024b) also evaluates models at context lengths at 128K tokens. However, their evaluation settings are lim- ited a few domains syntheticc, QA, and summarization. Although they also evaluate coding tasks, the domain is limited to code-specialized models outside of the realm of general-purposed language models. Furthermore, their summarization evaluations lack robust evaluation and still rely on ROUGE scores. Similarly, LV-Eval (Yuan et al., 2024) evaluates long-context models across different lengths but is limited to the QA tasks. Furthermore, we are the first to evaluate existing LCLMs comprehensively-ae ables previously unavailable insights into the correlation across diverse tasks and the landscape of current models.
evaluate 59 models of different sizes and architectures, which en-
30

---

Published as a conference paper at ICLR 2025
Table 8: We evaluate the performance of models on a subset of HELMET tasks with O-shot and 2-shot demonstrations at 128k input length to understand the impact of ICL on model performance, averaged across three random seeds. The standard deviation across three runs are shown in the subscript. We observe that the performance is generally higher for 2-shot demonstrations compared to O-shot demonstrations. Crucially, the 2-shot examples enable base models to achieve higher results that reflect the model's 's long-context capabilities in realistic settings, such as for MSMARCO and ISON KV è denotes base models.
JSON
MSMARCO
oC.BENCH QA
Model
0 shot
shot
shot
shot
shot
shot
0 shot
2 shot
0 shot
GPT-4o-mini
92.3s.s
93.72.1
57.8
60. 4..0
24.8a.7
31.4:.1
72.02.a
69.34.7
28. 28.81.9
47.14.4
GPT-44-00
99.3o.s
36.77.1
59.4ao
63.02.1
42.7p.9
46.8:
73.7u.
76.96.
49.11.9
51.314
GPT-4o-08
100.0o.0
100.0a.o
58.52.
61.94.5
16.60.3
48.71.0
75.32.5
72.70.6
29.80.9
18.61.
Gemini-1.55 Flasaah
98.3o.6
99.01.0
53.Ta.9
51.5u.
44.93.0
49.9c
73.9a.a
77.1a
48.2
49.92.
Gemini-1.5-4 1.5-Pr
97.7o.6
93.72.a
62.22
39.22.7
57.10.9
58. 38.40.s
83.6g.a
83.30.7
46.41.8
17.2a.2
Llama-3.1 3.1-8B'
77. 77.3g.1
98.01.0
4
44.9s.9
0.1u.0
7.50.s
$5.3s.s
53.3a.0
36.12.
36.62.1
Llama 3.1-8B Inst
96.0r.o
95.7aus
48.41.3
48.92.
5.4p.o
13.81.
49.72.
52.02.
29.12.
37.92.
Laama-3,22 3,2-1B'
34.0s.s
34.32.1
25.1
23.22.
0.36.
4.8o.5
23.0a.s
22.71.5
18.92.5
18.99.
Juma-3.2-1BBInns
6.3
9.3a.a
23.344
22.91.4
0.0g.0
0.5o.a
16.03.6
16.32
12.60.4
12.7o.5
Llama-3.2-3B'
30.3g.1
51.0s.9
40.22.0
42.3
0.1o.0
6.0g
43.01.0
10.77.2
31.111
31.62.2
Jlama-3.2-3B-Im
31.32.
36.72.8
44.22.
42.8s.8
0.2g.0
0.90.2
36.02.0
40.0a.5
17.81.2
19.70.8
Table 9: Comparison of model rankings on HELMET and BENCH
Model
HELMET
Model
InfBench
Llama-3.1-700
49.3
Llama-3. 1lama-3.1-8R-Mm
1-8B-Inst
46.7
1lmma-3.188--mm
47.0
Llama-3.1-700 1-70B-Inss
43.7
Llama-3.1-708
41.3
Yi-34B-2008
43.1
Yi-34B-200k
38.3
Alama-3.1-70B
38.6
Llama-3.2-3 3.2-3BIIIntt
36,9
Yi-9B-200k
37.6
Llama-3. 1-8B
35.6
Llama-3. 1-8B
35.7
Yi-9B-200k
33.0
Yi-6B-200k
32.0
31.9
-lama-3.2-3B-o
2.8
Yi-6B-200
26.3
3.ama-3.2-1B----
2.6
Llama-3.2-1D 3.2ma-3.--IB-Inst
24.6
Llama-3.2-1B
2.0
Allama-3.2-11
21.2
Llama-3.2-3B
1.8
To summarize, HELMET improves upon existing benchmarks by providing comprehensive eval- uation across diverse tasks and carefully designed evalution metrics and prompting strategies. Con- sequently, HELMET provides accurate reflection of the models" capabilities in practice.
B DATASETS
B.1 RETRIEVAL-AUGMEETED GENERATION
Natural Questions (NQ; Kwiatkowski et al., 2019) is large-scale dataset for open-domain question answering featuring real user queries. TriviaQA (TQA: Joshi et al., 2017) comprises trivia questions and their corresponding answers. HotpotQA (Yang et al., 2018) contains questions that require multi-passage reasoning. We source these datasets from KILT (Petroni et al., 2021), which provides annotations linking each query to its corresponding gold passages containing the answers.
PopQA (Mallen et al., 2023) consists of questions about long-tail entities. To minimize the impact of pre-training memorization, we filter the dataset to include only subject entities with popularity scores below 1000. Since gold passages as positive or negative by checking for the presence of the ground truth answer.
not available for PopQA, we classify retrieved passages
Notably, we populate the context with hard negative passages, retrieved from the same corpus the positive passages using a real retriever. This approach presents a significantly greater challenge than using randomly sampled passages. This design choice better reflects real-world strrival--augenter
31

---

Published as a conference paper at ICLR 2025
generation tasks, where models must effectively distinguish between relevant and irrelevant infor- mation.
B.2 GENERATION WITH CITATIONS
Generation with citations represents a crucial task for enhancing language model trustworthiness and verifiability (Bohnet et al., 2022; Gao et al., 2023; Asai et al., 2024b). We employ ALCE assess models' capability to generate properly cited text. Following the original methodology. we utilize Wikipedia as the retrieval corpus and GTR (Ni al., 2022) as the retriever. We omit MAUVE-based fluency evaluation since models typically generate fluent text, and in cases where they don't, ' the other metrics already approach zero. Instead, we focus on correctness and citation quality, reporting their average.
B.3 PASSAGE RE-RANKING
We evaluate models' passage re-ranking capabilities (Sun et al., 2023) using the MS MARCO pas- sage ranking dataset (Bajaj et al.. 2018). Our evaluation uses annotated datasets from the TREC Passage Re-ranking challenge (Craswell al., 2020). Each dataset instance consists of a query and a list of passages with associated relevance scores. The dataset combines Bing user queries with pas- sages retrieved via BM25 from web pages. Passages labeled perfect, highly relevant, or relevant. For each input length L, we determine the number of passages & that can be included. To eliminate positional bias, balance the label distribution and randomize passage order. We create three different permutations of the k passages for each input. The final performance is reported in NDCG@ 10. While higher values of are possible for NDCG evaluation, the computational cost of generating numerous passage IDs during inference leads to maintain NDCG@ 10 our metric.
B.4 IN-CONTEXT LEARNING
For in-context learning, we implement label mapping strategy that compels models utilize in- context examples rather than relying on pre-trained knowledge for classification tasks. Each label randomly mapped to integer (0. 1,..., - 1), where represents the number of unique labels in the dataset, following established practices (Pan et al., 2023). We then replace all label texts with their corresponding integer mappings throughout the dataset. Following Li et al. (2024c), organize examples into demonstration rounds, with each round containing exactly one example per label in randomized order. The input is constructed by concatenating these demonstration rounds until reaching the target input length L. Unlike other task categories where we evaluate 100 samples, for ICL datasets we evaluate 500 samples. We balance the label distribution among the evaluation set.
B.5 SYNTHETIC TASKS
For RULER tasks, generate data using the original authors' scripts (Hsich et al., 2024), employ- ing the Llama-2 tokenizer (Touvron et al., 2023) to standardize input text across all models. The RULER suite comprises the following tasks:
NIAH Single (three variants):
NIAH (variant 2): Most similar to the original NIAH (Kamradt, 2024)
NIAH Single Repeat (variant 1): Uses repeated phrases instead of Paul Graham essays as context
NIAH Single UUID (variant 3): Similar variant but uses UUIDs as retrieval values
NIAH MultiKey (MK, three variants):
NIAH MK Essay (variant 1): Involves retrieving one gold key among three irrelevant
keys, using Paul Graham Essays as context
-IIH MK Needle (variant 2): Uses needle-based context structure
NIAH MK UUID (variant 3): Similar to variant but employs UUIDs as retrieval values for all needles
-
32

---

Published as a conference paper at ICLR 2025
NIAH MultiValue (MV): Requires retrieving four different numbers associated with a sin- gle key from irrelevant essay context
NIAH MultiQuery (MQ): Involves retrieving correct values for four distinct keys from irrelevant essay context
Variable Tracking (VT): Requires tracking variable values through sequential operations
Word Extraction Tasks:
Common word extraction (CWE)
Frequent word extraction (FWE)
Both tasks require identifying the most frequently occurring words
Question Answering Tasks:
SQuAD (variant 1; Rajpurkar al., 2016)
HotpotQA (variant 2; Yang et al., 2018)
- Notable distinction: Our implementation uses retrieved passages rather than random samples, making the task more realistic and challenging
For comprehensive details, please refer to the original paper (Hsieh et al., 2024).
For JSON KV tasks, we generate JSON dictionary containing randomly generated UUIDs both keys and values, similar to Liu et al. (2023). For each dictionary, construct six queries, asking the model to retrieve the value for each key at six evenly spaced positions within the context.
B.6 MODEL-BASER EVALUATION
Automatic evaluation metrics, such ROUGE, are known to be unreliable and uncorrelated with hu- man judgments (Goyal et al., 2023; Chang et al., 2024). However, existing long-conted benchmarks still largely rely these metries (Bai al., 2024; Zhang et al., 2024b). In this work, we seek to more reliably evaluate language models at long-contexts by leveraging LLMs judges, inspired by pre- vious works (Zheng et al., 2023). For all model- evaluations, we use GPT- 202oo02024-2 2024-05-13 as the judge.
Long-document question answering. For NarrativeQA, found that the models can often output answers that are correct but have little surface form or lexical overlaps with the ground truth answers in preliminary experiments, resulting in lower-than-expected performance. This is often due the long lengths of the ground truth answers, which gives more possibilities of how to write it than simple named entity that is often the case of other QA datasets.
Therefore, we employ an LLM to judge if the model output is fluent and semantically similar to the ground truth. Given the question, the ground truth answer, and the model -ener-ggeeeeted output, we ask the LLM to judge if (1) the model output is fluent and (2) the generated output is correct and relevant to the question with the ground truth as a reference. The detailed prompts, precise definitions, and instructions are found in Table 10.
In Table 14, we also find that model evaluation can useful catching extremely cases. For example, Claude scores low in terms of Fl due to its tendency to output extra, assistant-like text, which penalized by the n-gram overlap metric, and appears to be worse than the much smaller Llama-3.2-3B-1 model. However, the model-based evaluation is able to catch this issue, and Claude scores higher than Llama-3.2-3B sizes.
which users might have expected given the model
Summarization. At high level, we ask the model to check for three properties: fluency, precision, and recall. Fluency can take on value of either 0 (incoherent, incomplete, and/or repetitive) (fluent and coherent). Precision is the percentage of model-generated sentences that are supported by the gold summary (we use the long summary from Multi-LexSum here). Recall is the percentage of the key points that are supported by the model-generated output. We calculate the F1 score from precision and recall and multiply it with the fluency score for the final score.
We first ask the model to generate a list of key points or atomic claims from the gold summary, following previous works that show that LLMS accurately decompose long texts (Kamoi al.,
33

---

Published as a conference paper at ICLR 2025
2023: Gao et al., 2023). We manually checked 105 claims originating from 25 Multi-LexSum sum- maries and found that the claims were all factually correct. Although we found one out of 25 instances where the model missed possible key point in the given summary, found that GPT-4o is almost always reliable for the decomposition task. For Multi-LexSun we use the short summary to obtain the key point as the annotation contains both long and short summary. These key points are then saved for Multi-LexSum and BoBENCH Sum.
We show the detailed prompts for the summarization tasks in Table 11, 12, and 13, which are mod- eled after previous works (Kamoi et al., 2023; Chang et al., 2024; Kim et al., 2024). These previous works have shown that LLMs can effectively judge model outputs (Zheng et al., 2023), but we conduct human analysis to further verify the evaluation metric.
From qualitative analysis, we found that the model is consistently able distinguish between fluent and non-fluent outputs, where we agree with the model judgments 100% of the time for randomly sampled Multi-LexSum and ooBENCH Sum outputs. We then check if the GPT-4o judgments for precision and recall agree with human judgments. To this end, we sample 10 generated summaries for Multi-LexSum and BENCI each (from Gemini-1.554 five key point evaluations for each summary. We follow a similar procedure where we check if the model output supports the key point. We observed Cohen's x = 0.76 for ooBENCH Sum and conduct a similar human evaluation, and found a A = 0.91 for xBENNH Sum and k. = 0.83 for Multi-LexSumm suggesting near-perfect agreement.
and Llama-3.1-70B-Im
and check
0.72 for Sultm, suggesting substantial agreement. For precision,
Inspecting the disagreements, we find that that most of them arise from the partially supported cases. For instance, the key point may include specific details, such the names of government departments or Court Justices' names, that are not explicitly mentioned in the generated summary. and the model judge is typically more lenient about the exclusion of these small details while humans are more strict. However, this is also subjective to the preference of the human.
Qualitatively, We find that model-based evaluation can catch cases where the model is overly repet- itive and scores high ROUGE-L score as a result, such as Mistral-7B-Inst-vll on co BENH Sum. For instance, the Mistral model may generate an output consisted of the sentence "The author's object is to show that the study of grammar is necessary part of education" repeated for hundreds of times. This summary would receive an ROUGE-L score of 12.3 while the GPT-4o judge would penalize the model for being overly repetitive and incoherent and assign the output final score of 0.0. Thus, our GPT-4o judge penalizes the model for being overly repetitive, and the final metric reflects this issue.
C MODELS
Please see Table 15 for the detailed information of the models used in this work.
D EXPERIMENTAL SETUP
As previously described. we are able evaluate models across different input lengths. Thus, we evaluate all models at L E (8192, 16384, 32768, 65536, 131072}. For the proprictary models, GPT, Gemini, and Claude, we rely on the provider's API. For all open-source models, we evaluate on H100 GPUs with 80GB of memory. We use the HuggingFace framework (Wolf et al., 2020) to load and generate model outputs. We apply instruction-tunee models' chat template whenever applicable. We use FlashAttention2 (Dao, 2023) and BF16 for faster inference. Our compute is limited to 8 H100 GPUs; thus, we are not able to run some of the larger models, such as Command-E R or Jamba- Large, at 128K tokens. We evaluate 600 examples for JSON KV, NQ, PopQA, and TQA, 300 examples for the MSMARCO and HotpotQA, 500 examples for ICL, and 100 examples for the remaining datasets.
34

---

Published as a conference paper at ICLR 2025
Table 10: Model-based evaluation prompt for long-document question answering.
impartial judge and evaluate the quality of the provided answer which attempts to answer the provided question based on a provided context. Although you are not given the context, you will be given a set of correct answers that achieves full scores on all netrics, and you need to assess the provided answers using the correct answers.
Please act
Below your grading rubric:
incomplete): Incoherent sentences, repetitive sentences (even if not by exact words), incomplete answers, or gibberish. Note that even if the answer is coherent, if it is repetitive or incomplete, it should be given a score of 0. Score (ccherent, non-repetitive answer) Coherent, non-repetitive, fluent, grammatically correct answers.
Fluency: Score (incoherent, repetitive,
Correctness: Score 0 (Inccrrect); The answer does not agree with the provided correct answers at all. Score 1 (partly correct): Partly agree with one of the provided correct answers (for example, the question asks for date and a person; the answer gets the date right but the person wrong). - Sccre 2 (correct but not fully relevant): Fully agrees with one of the provided correct answers but mentions other completely irrelevant information. Note that extra details provided in the answer, even not mentioned in the correct answers, should NOT be seen as irrelevant as long as they are relevant to the questi on to reascnable extend. - Score 3 (correct and relevant): Fully agrees with one of the provided correct answers and only provides information relevant to the question. Note that if the answer is longer than the correct answer, as long as everything in the answer relevant to the questicn, it should still be given score 3. For example, if the correct answer is "hh North Pole and the answer is "Theey are headed for the North Pole'', it should still be given a score of 3.
Now, read the following question, answer, and correct answers. First think step-by-step and provide your reasoning and assessment on the answer. Then output your in the following json format: [l' "''Euuecc''' 0, "correctness" 1}}.
Question: (question]
Correct answers: cnorrect.answent Answer: (parsed.output)
E ADDITIONAL RESULTS
E CORRELATION BETWEEN SYNTHETIC AND DOWNSTREAM TASKS
We show the correlation between all synthetic and RAG datasets with other downstream tasks in Figure 8. We see that the synthetic tasks are generally less correlated with the downstream tasks, whereas the RAG datasets are more correlated with the downstream tasks. Furthermore, there ap- pears be a pattern between the complexity of the synthetic task and its correlation with other tasks the more complex the task, the more correlated it is with the downstream tasks. Further- more, noisier variants tend to be more reflective of real-world applications, as seen the difference between RULER MK Essay, where the context is irrelevant essays, and RULER MK Needle, where the context is other distracting needles. Upon closer inspection, we note that Needle MK Nee- dle/UUID, JSON KV, and NIAH MV generally have relatively high correlation with other realistic datasets, and are able to test the recall abilities of the model. Thus, we select these four datasets as part of the HELMET synthetic recall subset. We validate this selection by checking the correlation
35

---

Published as a conference paper at ICLR 2025
Table 11: Model-based evaluation prompt for summarization fluency score.
Please act impartial judge and evaluate the fluency of the provided text. The text shculd be coherent, non-repetitive, fluent, and grammatically correct.
Below is your grading rubric: Score 0 (incoherent, repetitive, incemplete); Incoherent sentences, repetitive sentences (even if not by exact words), incomplete answers, or gibberish. Note that even if the answer is coherent, if it is repetitive or incemplete, it should be given score of 0. Examples:
Incomplete: "Summary:''
Inccherent: " Summary: The plaintiff the the the the able the the the
the the the the the the the able the the the the the " Repetitivy "Summary: The U.S. government brought a criminal case against four defendants. Summary: The U.S. government brought criminal case against four defendants. Summary: The U.S. government brought criminal case against four defendants. Sunmary: The U.S. government brought a criminal case against four defendants."
Score 1 (coherent, non-repetitive answer) Coherent, non-repetitive, fluent, grammatically correct answers. If the text coherent, non-repetitive, and fluent, but the last sentence is truncated, it should still be given sccre of 1. Examples: apprenticeship test that had disparate impact Black apprenticeship applicants. The Equal Employment Opportunity Commission (KHOC) filed this lawsuit on December 27, 2004, in U.S. District Court for the Southern District of chio.'' rhe plaintiffs sought declaratery and injunctive relief, as well as attorneys' fees and costs, under the Americans with Disabilities Act, the Rehabilitation Act of 1973, the Secial Security Act, and the Nursing Home Reform Act. The case was certified as class action on behalf of all Medicald-eligible adults with disabilitics in Cook County, Illincis, who are being, or may in the future be, unnccessarily confined nursing facilities and with appropriate supports and services may be able to live in community setting. The defendants denied the allegations and argued that the plaintiffs" claims were not typical of the class and that the class definition was too broad. The case is ongoing, with discovery and expert testimony scheduled for the fall ot
This case is about
Now, read the provided text, and evaluate the fluency using the rubric. Then output your score in the following json format: (f''fluncyy': 1}).
Text: "(text]''
between this set and other downstream tasks, and found that generally has higher correlation than using synthetic datasets individually. There may be more optimal methods for selecting synthetic datasets, and leave this future work.
E.2 CORRELATION BETWEEN DATASETS
We plot the correlation between all HELMET datasets in Figure 9, In general, the datasets in each category are strongly correlated with each other. One exception to this observation is the lack of correlation between the ALCE datasets. This suggests that writing the correct citations is different skill from answering questions with facts. We also observe some low correlation between datasets for long-document QA and ICI, but this is likely due to the diversity within each categories.
36

---

Published as a conference paper at ICLR 2025
NIAH 0.44
0.71
0.75
0.76
0.72 0.68
RULER S Repeat- 0.49
0.71
0.74
0.67
0.69 0.66
RULER UUID 0.5 0.68
0.72
0.74
0.74
0.67
RULER MK Essay 0.56
0.71
0.77
0.78
0.74
0.71
RULER MK Needle
0.48
0.73
0.84
0.79
0.87
0.74
RULER MK UUID
0.49
0.69
0.84
0.81
0.85
0.74
RULER MQ
0.54
0.75
0.82
0.78
0.79
0.74
RULER MV
0.61
0.71
0.77
0.83
0.79
0.74
CWE
0.27
0.77
0.8
0.73
0.79
0.67
FWE
0.31
0.66
0.78
0.68
0.75
0.63
0.51
0.56
0.6
0.5
0.58
0.55
RULER SQUAD
0.45
0.72
0.87
0.86
0.88
0.76
RULER HotpotQA 0.46
0.83
0.89
0.83
0.89
0.78
RULER 0.51 0.77
0.85
0.79
0.83
0.75
JSON 0.55
0.67
0.78
0.76
0.83
0.72
Recall
0.61 0.82
0.77
NQ
HotpotQA
PopQA
TriviaQA
RAG
ICL
Cite
Re-rank
LongQA
Summ
Avg.
Figure 8: Spearman's correlation at 128K input length, calculated across instruction-tune mod-
els, between all synthetic and RAG tasks and other downstream tasks.
E.3 POSITIONAL EMBEDDING EXTRAPOLATION REMAINS A CHALLENGE
A key component of LCLMs is its positional embeddings, it's essential to how the model pro- cess positional information and extrapolate to long sequences. Thus, we also consider models that leverages positional extrapolation during inference. Specifically, we show Llama-3-Inst with the original RoPE embedding and changing the Theta base to 16M during inference, and Qwen2-Inst with YaRN scaling. Formally, RoPE defines 0₂ = b 4-24/1D), where 0, is the angle at the d-th hid- den state, b is constant called the base, and [ is the size of the hidden state (Su et al., 2021).
37

---

Published as a conference paper at ICLR 2025
.SON KV
E30 [:01
5:11
1.67
0:41
5.7%
9.2%
0.79
5AN 0.7
C:M
Necole-o 87
X75
0.55
4.39
6:0
6.31
8.79
0.06
0.31
AM MK UURD-9 6.0
A77
042
0.4
6.00
033
0.2
675
1.09
6.79
8:00
0.79
8.04
0.05
RAT
043
0.0
0.09
6.71
8:00
0.79
045
0.00
Rocal MM 0:00
0.00
6:90
6.22
0.72
0.7
6.99
0.30
9:00
0.79
8:85
0.05
0.03
8:12
0.00
0.05
6.71
Notpot
75 0.00
0.05
9.94
0:55
6.5
0.94
5.55
8.0
8.50
0.24
8.0
0.47
9.6
0:47
0.50
Tixu24 Cal
0.00
6.0
6:22
0.2
6.97
6.71
8:20
0.2
0.56
1AG Gde
0.00
0.00
0.92
630
9.24
0.7
6.90
6.22
6.00
8:22
00.00
8,4
0.5
0120
A6 Comeetross
5. Cde
364
642
C
026
6.9
6.47
6.00
8.52
8.52
8:00
0.20
9.20
042
ASQA Diee Socal
365
022
0.0
9.7
0.80
6.11
8.00
8:22
0.20
024
0.00
ASDA Ote Frechton 64
622
6/A
9.9
032 6.6
0.59
6.64
6.79
0.64
8.94
8:22
0.77
8.70
000
822
032
CAMPARI Correctress
0.55
073
0.65
0.73
0.22
CAMPANI Dis Racal XX Cdt9
304 639
0.0
065
6.9
0.75
K.44
8.03
8.99
9:00
0/22
8:00
0.50
9:54
037
4
0.30
GANTAR Cie Trechion- 0/99
0/6
Cls-asi c.2
6.2
6.2
MEMARC0 9.N can c.n
A77
0.0
0.3
3.82
362
6.00
3312
0.53
0.54
6.20
0..00
044
KatatNeGA C.7
0.00
5.43
5.50
5.64
5.00
0.00
britlnch 9.75 646
375
0.00
6.00
6.43
0.94
1.0
0.21
8.22
057
0.00
G12
625
0.5
6:00
0.9
6.4
8.09
6.70
8:00
0.70
8A2
047
3:40
LengbA G39
300
6.2
632
6:22
9.72
6.90
6:22
0.79
0.00
0.5
0.54
Irtlench S.m- G/L
629
0.64
3.00
6.34
0.71
6.52
6.6L
8.00
8.29
0.91
827
024
3:40
8M 6:00
4.1
0.03
0.0
5:19
6.7
5.9
5:19
022
9.47
0:41
0.06
Summ
TREC Coune
0.
5:35
0.3
0:33
322
8.27
0.18
5.09
0.12
0.1M
8.7
038
TREC Ase
BAMGNG73
6A2
302
631
6.3
8:00
0:00
CLNICT56
622
654
C.L
8:00
M.U 624 A62 647 631 305
652
6.2
830
0.22
032
CAl
635
0.24
6:30
Our F:00
tmt a and 2m
HON Hi
Aass we Groude
Becall
Hespostae
Popide
ASDA Coneensay
Asick Ohe Secial
AsdA dee Prechiion
Galpee Conmettnnt
naspani Car Aeciali
COMPANL Cae Aeerlll
AESMANCO
ontherch Che
a MC
Lengiiae
Intherchh Ston
Mati- Ledtenioo
Santin
Clarse
Figure 9: Spearman's 's correlation at 128K input length, calculated across 35 instruction-uus -tuned mod- els, between all HELMET datasets and category averages.
.lama-3 sets b = 500, 000 during training, and we evaluate with both the original base and setting 6 = 16, 000, 000.
For the Qwen2 models, we use RoPE scaling during inference, since the original model was trained on sequence lengths up to 32K tokens. Specifically, we follow the recommended scaling factor specified in their HuggingFace model card".
We find that existing positional embeddings still struggle at out-of-distribution length, with both families of models dropping sharply in performance past L 32768, show in Figure 7. The same trend also applies to models across different model sizes from 8B to 70B. Finally, altering the posi- tional embedding may lead to degradation at shorter lengths, which is evident for Llama-3-8B-Ing on ODQA and ICL. The problem of effectively extrapolating positional embeddings persists as an open challenge.
E.4 PRESENCE OF LOST IN THE MIDDLE
Previous works found that models often struggle with recalling facts in the middle of the input, a phenomenon named lost in the middle (Liu al., 2023). In this work, we extend the previous
hhttps://hugging ct/ wwwp///ee/weee/ee 78- instr uproceessttrttts ng- long-texta
38

---

Published as a conference paper at ICLR 2025
analysis to input length up to 128K tokens. We place the needle six different evenly spaced depths in the context, and evaluate the models' ability to retrieve the needle. In our study, the needle may be either a key in the JSON dictionary the answer to question. We show the results in Figure 11 for JSON K Figure 12 for Natural Questions, Figure 13 for PopQA, and Figure 14 for TriviaQA. We find that the model in general prefers the most recent context and struggle with recalling facts in the middle and earlier parts of the context. Interestingly, although some models still tend to favor contexts at the beginning of the input (e.g., Llama-3 and Llama-3.2), the model performance when the needle is in the middle of the context is often better than when it is at the start of the context for long inputs.
E.5 COMPARISON BETWEEN BASE AND INSTRUCTION-TUNED MODELS
Previous benchmarks largely focus evaluating instruction- models (Shaham et al., 2023; Zhang et al., 2024b). As a result, developers of long- contect base models often turn to perplexity and synthetic tasks to evaluate their models (Fu et al., 2024; Yen et al., 2024; Al et al., 2024). However, as we have seen in this work, it is important to evaluate LCLMs on diverse downstream tasks to get full picture of the model's capabilities. Furthermore, comparing base and instruction- tuned models is essential for understanding the impact of instruction tuning on model performance long-contexts. Thus, we built HELMET be compatible with both base and instruction- tuneed models through in-context learning examples. Taking closer look at the results in Figure 10, we find that instruction-tunie -unnng generally improves the performance of the model across all tasks. Instruction-tuning at long contexts may yield more benefits than simply scaling up the model sizes; for example, Llama-3. 1-B. Inst outperforms Llama-3. 1-70B on most tasks.
E.6 PERFORMANCE OF CLAUDE
In this subsection, investigate the relatively-low performance of the Setuude-.5--omme model in comparison to other proprietary models, such as GPT-4o and Gemini. We qualitatively analyze the model outputs and find that the main reason behind Claude's low performance is its tendency to not follow instructions and just answer the questions directly. For example, it often does not output the citation markers for Cite (Table 17). does not output rankings in the passage re-rankings task (Table 18), and does not output the classification label for ICL. (Table 18). Sometimes it even refuses to answer the question due to copyright concerns (Table 17). Thus, it has lower score on these tasks. We argue that it is important for LCLMs to follow the user instructions in real applications should penalize models that do not. Sometimes, the output may be truncated due to the restriction on the maximum number of generated tokens (Table 16). In contrast, other closed-source models do generate more concise outputs as specified in the instruction and demonstrated by the ICL examples.
E.7
FULL RESULTS
We provide all the detailed evaluation numbers on each dataset each length here: https:/ // shorturl avV..
39

---

Published as a conference paper at ICLR 2025
Table 12:
Model-based evaluation prompt for summarization recall score. here we show the demon-
strations for Multi-LexSum.
Please act as impartial judge and evaluate the quality of the provided summary of a civil 1awsuit. The sunmary is based set of legal decuments, and it shculd contain a short description of the backkrrund, the parties invelved, and the outcomes of the case. The text should contain all the major points in the expert-writed sumnazy, which are given to Below is your grading rubric: Recall: Evaluate the provided sunmary by deciding if each of the key points is present the provided summary. A koy point is considered present if its factual information is well-supported by the previded summary. - Score: the number of key points present in the provided summary. Examples: use the following examples guide your evaluation.
Example 1:
Key points: 1. The case challenged curfews in Los Angeles and San Bernardino, California. 2. The curfews were issued in response to the naticnwide protests following the police killing of George Floyd in Minneapelis. 3. The complaint argued that the curfews violated free speech, free assembly, free movement, and Due Process. 4. The complaint also argued that the San Bernardino curfew violated the Establishment Clause. 5. The complaint sought injunctive and declaratery relief.
The plaintiffs voluntarily dismissed the on July 7, 2020. 7. The dismissal occurred because the city had rescinded the curfews and not attempted to reinstate them.
Summary: "In June 2020, Black Lives Matter Los Angeles and several individuals filed lawsuit in the U.S. District Court for the Central District of California against Los Angeles Mayor Bric Garcetti, other city officials, and the City of San Rernardino, challenging the constitutionalio of curfew crders imposed during protests against police violence. The plaintiffs, represented by the ACIU of Southern California, argued that the curfews violated their First Amendment rights to free speech and assembly, as well as their freedom of movement, by suppressing political protests and other activities. The lawsuit also claimed that the curfews were not narrowly tailored to address any emergency and lacked sufficient notice. However, the plaintiffs voluntarily dismissed the case in July 2020 after the defendants lifted the curfew orders and did not reinstate then in the following weeks.'!
Reascning: The summary states that the plaintiffs challenged the constitutionality of curfew erders against Los Angeles and San Bernadino, so key point 1 is present. The summary does not mention that the curfew orders were issued in response to the nationwide protest that resulted from the police killing of George #loyd in Minneapol so key point 2 is missing. The summary does mention that the complaint argued that the curfews violated the First Amendment rights to free speech and assembly, so key point the complaint argued that the San Bernardino curfew violated the Establishment Clause, key point 4 is missing. The summary does not mention that the complaint scught injunctive and declaratory relief, so key point is missing. The summary mentions that the plaintiffs voluntarily dismissed the case in July 2020 after the defendants lifted the curfew orders and did net reinstate them in the following weeks, so key peint 6 and 7 are present. Finally, key points 1, 3, 6, and 7 are present in the summary, se the recall score is Output: (("'reaal''; 4]}
present. The summary does not mention that
Example 2: lomitted for brevity]
Now, read the provided summary and key points, and evaluate the summary using the rubric. First, think step-by-step and provide your reasoning and assessment on the answer. Then output your score in the following jeon format: ((''recil'': 2)}. Key points: (keypoints) Summary: " "ssummary'"
40

---

Published as a conference paper at ICLR 2025
Table 13: Model-based evaluation prompt for summarization precision score. Here we show the demonstrations for Multi-LexSum.
impartial judge and evaluate the quality of the provided summary of a civil lawsuit. The summary is based on a set of legal documents, and it should contain a short description of the background, the parties invelved, and the outcones of the case.
Please act
Evaluate the provided Belcw is your grading rubric: Precision: provided summary is summary by deciding if each sentence supported by the information provided in the expert summary, A sentence is considered supported if its major facts align with the infornation in the expert summary. A sentence is still considered supported even if some of its minor details, such dates, entity names, laws and previous court cases, are not explicitly mentioned the expert sumnary. A sentence is not supported if its major facts are not mentioned contradicted in the expert summary. of sentences in the provided summary that supported by the expert summary. Examples: use the fallowing examples to guide your evaluation.
the names of
Score: the number
Example 1:
Expert summary: "This lawsuit, brought in the the U.S. District Court for the Central District of California, plaintiffs were represented by attorneys from the ACIU of Southern California. This lawsuit followed nation-wide protests that occurzed in respense to the killing of George Floyd by a police officer in Minncapelis. While most pretests were peaceful, some ended in violence, property destruction, ricting, and looting. Many cities, including Los Angeles and San Bernardino, issued curfews in an attempt to quell these ricts. [emitted for brevity]"'
filed June 3, 2020. The
Provided summary: "In June 2020, Black Lives Matter Los Angeles and several individuals filed Lawsuit in the U.S. District Court for the Central District California against Los Angeles Mayor Eric Garcetti, other city officials, and the city of San Bernardino, challenging the constitutionalle of curfew erders imposed during protests against police viclence. The plaintiffs, represented by the ACLU of Southern california, argued that the curfews violated their First Amendment rights to free speech and assembly, as well as their freedom of movement, by suppressing political protests and other activities. The lawsuit also claimed that the curfews were not narrowly tailored to address any emergency and lacked sufficient notice. However, the plaintiffs voluntarily dismissed the case in July 2020 after the defendants lifted the curfew orders and did not reinstate them in the following weeks."!
Reasoning: ing: The first sentence in the provided aurmary is well supported by the expert summary even though some entity names are not explicitly menticned. The second sentence is also well supported by the expert summary, as it mentions the ACIU of Southern Californis and the First Amendment rights. The third sentence is not supported by the expert summary, it does not mention the lack of narrow tailoring or sufficient notice. The fourth sentence summary, as it mentions the voluntary dismissal of the case in July 2020. Therefore, the precision score is 3. Output: (('precision"':
well supported by the expert
3, " "ssentence.count".. 4]} Example 2: [omitted for brevity]
Now, read the provided summary and expert summary, and evaluate the summary using the rubric. First, think step-by-step and provide your reasening and assessment follewing json format: {f' preciision"" 2, " "ssentencs_count". 6)). Expert summary: "fexpert_summary"" Provided summary: ' "{suumary)''
the answer. Then output your score in the
41

---

Published as a conference paper at ICLR 2025
Table 14: Comparison between ROUGE-L F1 scores, which is commonly used in previous works but cannot identify errors in generations, and our GPT-4o-based evaluation metric, which better reflects user experience and achieves better separability. We use model-based evaluation for NarrativeQA (NQA) and oc BENCCISSummarizttonn (co (ENNUSS))))) and Multi-LexSum(MLSS)
Narrative
<BNCC Sum
Multi-LexSun
Model
Fl
GPT
R-L
GPT
R-L
GPT
GPT-4o-05
46.5
55.3
17.4
34.8
25.5
55,4
GPT-4o-08
43.1
51.3
16.8
31.1
24.9
56.1
Claude-3.5-sonnee
16.2
43.3
14.2
30.5
22.0
50.9
Gemini-1.5-Flash
39.0
42.9
17.0
28.7
24.5
51.1
Ge5-ni-I.5-Pro
42.8
50.9
1
32.0
25.8
58.1
Llama-3 1amaa-3-B-IInss
1.0
6.7
6.8
0.0
8.0
3.3
Llama-3-88 -aaa--3---Inns-Theee
3.3
4.1
7.6
0.0
12.4
22.3
Mama---70B-Inns--Tee
4.9
10.9
5.9
0.0
14.4
24.0
-lama-3.1-8B-Imm
35.1
47.7
17.4
16.2
25.2
46.9
1ama-3,1-700-BB.
38.3
54.0
17.5
19.8
25.2
51.6
1lama-3.2-1B-11
14.5
18.2
16.6
2.9
23.1
20.3
Llama-3.2-30 3.nsstt
23.8
36.7
16.5
12.4
23.7
43.6
12.3
16.0
12.6
4.2
22.1
16.1
Distral-7B-Inst-Vee
14.9
21.0
13.4
1.0
20.0
23.4
Mistral-7B-Inst-v..33
21.0
22.0
12.3
0.1
19.0
24.4
42

---

Published as a conference paper at ICLR 2025
Table 15: Length denotes the training length of the model or, if not known, the claimed context window. For MoE models, we denote number of active/total parameters. We change the RoPE (Su al., 2021) 8 during inference for some models and denoted with -B. Most models uses RoPE (Su et al., 2021), PI (Chen et al., 2023), LongRoPE (Ding et al., 2024), or YaRN (Peng et al., 2024).
Name
Length
Architecture
Positional Embh
Params
Inst.?
Proprietory
gpt-4-0125-preview
128000
2o--4o-2024-55.11
gpt-to-2024-08-06
128000
cluub-33--ommmtt222440000
gemiin-[_5-faah-001
1048576
7
gemini-L.5pro-000
2097152
7
Llama-2 (aase-2- (Touvron et al., 2023)
11aMA-277B.32
32768
Full attentitnn
Llama-2-7B-32K, Instruct
32768
Full-attention
7B
(aama-2-76-90k-biie
80000
Pull-sttention
Dynamic NTK
7B
Yam-Llama-227h--666
66536
Full-attention
Yam-Llumm-22 2-7h- 128k
131072
Full-attentioe
YARN
7B
Llama-3-bace (Duhey al., 2024)
1lama-3-88
8192
Full attention
RoPE:
801
Llama-3-44 3-8B-Instucc
Full-attention
Llama-3-8B-9
8192
Full-attention
RoPE
8B
8192
Full-attention
RoPE
Laama-3 3-70B--
8192
Fall-attention
RoPF:
70B
Llama33-7B--mme 1-stttccct-
8192
Fall-attentia
RoPE
70B
Fall-attention
Llama Instrcr
131072
Full attention
RoPE
80
Llama-3.
Full-atlention
Llama-3 1-70B-Instruct
131072
Full-atteation
RoPE
708
Llama-3.3 3,3-77BB-Instru
131072
Full Fall-atiention
RePE
70B
Llama-3.2-1B
131072
Full-attention
RoPE
Llama-3.2-18 1----s---tt
131072
Full- Full-attention
RoPE
1B
3.ama-3.2.38
Fall-attention
Llama-3.2-33 Instrnctt
131072
Full attention
RoPE
3B
ProLong (Gao al.,
ProLong
524288
Full aull-attention
RoPE
8B
Mistral-based (Jiang et al., 2023)
Mistral-TR TB-IIstrract-O..
8192
Sliding window
RoPF
7B
Instruct-v0.2
Full-attention
Mistral-BB-vol
32768
Full-attention
RoPE
7B
Mistral-7B- 1a-aal77--stt
32768
Full-attention
RoPE
Mistral- Nemo Nemo-Base-2285 Base-24077
128000
Full-attention
RoPE
12B
Mistral-Nemos Nemo-Instruut22400
128000
Full-attention
RoPE
12B
MegaBeam- Mistral-7--511
Full-attention
RoPE
Ministral SB. Instract-2441
131072
Full-attn Sliding window
RoPE
8B
et al., 2024)
Yi-6B-200K Yi-9B-200K
200000
Pull-sttention Full-attention
RoPE RoPE
GB
Yi-34B-200K
200000
Full-atteation
348
Vi-1.5-98-32K
32768
Full-attention
RoPE
Plu-3 (Abdin et al., 2024)
33-mimi-118K-insttuus
full-attention Blocksparse attention
LongRoPE
Phi-3-small- 1ai-112kkinnnune
131072
LougRoPE
78
Phi-3-medium- 128k-instruct
131072
Full-atiention
LongRoPE
14B
Phi-3.5-miini-nsttue
131072
Full-attention
LongRoPE
Qvien2 (Yang 2024)
Qwen2-70 7B
32768
Full-atlention
YaRN
7B
Qwea2-7B BB-Itttrct
32768
Full Full-attention
YaRN
7B
Qwea2-578-A148
32708
Full-attention
YaRN
14B/57B
X
Qwea2-57B. 10--2-77--A485 -nn---nrtttt
32768
Full attention MoE
YaRN
14B/57B
Qwem2.5 (Qwen et al., 2025)
Qwen2.5.1.50
131072
Dual-Chuna Attention
YaRN
1.5R
X
Qwen2.5.1.50
Chunk Attention
Qwea2. 5-3B
131072
Dual-Chunk Attention
YaRN
3B
X
Qwen2.5.2 L-3
131072
Dual-Chunk
YaRN
Qwen2.5-75
131072
Dual-Chun) Atlention
YaRN
7B
131072
Dual-Chunk Attention
YaRN
7B
Qwee2.5-77BB-nme
131072
Dual-Chunk Attention
72B
Qween2.5.1 S.I (Yang et al., 2025)
Owee2.55-BB-Ins. Qwen2.5-14B-IIntt-IMM
1,010,000
Dual-Chunk Attention
YaRN
7B
1,010,000
Dual-Chunk Attention
YoRN
14B
Jambo (Licher al., 2024) (Team et 2024c)
Jamba-x0.1
262144
Hybrid (Mamba + Full-attention) MoE
None
128/528
X
Jamba- MiniMini
Hyhrid (Mamha Full-attention)
None
43

---

Published as a conference paper at ICLR 2025
Recall
RAG
Cite
Re-rank
GPT-4
19:5
59.5
91
735
75.1
73.5
10.9
00.1
61.7
46.0
42.7
26
3.1
12
57.0
78.1
06.7
25
9.
GPT-4c-05
03.2
82.2
21
90
82.4
72.5
72.4
72.6
72.7
70.4
46.
43.7
42.5
42.4
40.4
68.3
77.
67.7
67.7
47.7
GPT-4c-08
13:.0
103.0
100.0
100.0
W83
73.9
74.0
12.4
71.5
70.2
45.4
46
45.4
45.4
46.9
68.9
90.5
68.
58
50.0
GPT-4o mini
88.8
29.5
20.1
96
836
72.
72:1
70.0
68.7
68
96.1
35.7
30.0
27:1
24.3
88.8
71.
58.5
42.0
31.2
Cleude-3.5-Sone
88.8
96.6
9:2
86.7
56.4
35.0
36
32.2
41.0
34,8
17.9
72
Gamink-1.5-Flaa
83.6
83.8
70.6
48.7
49.4
90.8
74.0
Gemini-1 1-5rro
87.5
13:2
17.8
0.0
82.4
11.5
34.4
30.8
14.8
Lamma-388-88
19/33
81.6
85
39-4
0.0
95.9
2
61.5
40
1.3
22.4
20.7
13:1
4.8
37.1
26.5
14.4
0.1
Llama-3-708-4
91.0
85.2
86.2
36.9
0.0
$8.1
86.2
62.6
66.5
0.0
23,9
22.2
15.3
8.7
2.8
44.1
41.2
20.3
0.0
Llame-3-70B-
87.5
85.0
B1
43.6
0.0
70,5
68.0
68.G
60.4
0.0
43.5
46.2
37.7
7.6
0.0
52.1
73.0
68.0
26.2
0.0
1lama-3.1.88
107.3
12:3
10.1
1::10
76T0
65.5
65.5
63.8
61.5
17C
10.
51
2.6
4
58.5
94.9
22.7
9.3
Llama-3.1-88
100.0
1001.
18
852
FO
65.88
26.77
10.9
43.4
14.0
1.70-
09.0
89.2
16.2
1am0-3.1-708
120.0
100.0
98.3
90.7
71.5
70.9
69.5
60.2
41.0
24.5
1-18
75.0
60.5
252
38.1
37.6
6.5
6.8
574
2.5
26.2
18.3
9/4
L.ama-3.2-1B
72.8
62.65
46,9
49.
32-4
48
43.7
37.9
381
8.3
5.8
4.5
2.8
2.4
29.0
18.5
5.7
3.8
17
Llama-3.2-38
19.00
89.3
79.8
05.0
55.8
$1.1
90.9
68.0
56.3
51.A
10.
7.1
3.5
3.2
2.1
43.6
33.5
26.3
6.9
5.4
Liama-3.2-38
99:2
53.1
14.4
64.
53.1
55.3
64.9
63.
CO.1
D0.
11.2
T.5
5.0
7.6
7.4
41.0
29.4
17.7
3.9
0.7
1amme788-00 80K
05:0
54.1
33.8
17.0
552
63.0
52
40.
124
8/4
8.3
4.2
1.0
37.0
26.6
19.5
9:1
0.2
Vam-LLama-2278--20 -28k
110
$3.6
4.5
3.5
2.5
11.2
4.4
MBstral-78.v011
2.8
Mistral-7--0.2
133.0
80.5
53.7
90.6
122.7
0.5
62
29.4
1-2
Mistral-7B-w033
68.9
05.2
86.
3.6
82.8
61.6
67.2
10.3
52
34.8
22.6
044
1.2
Metral-7B--0.3
86.6
94
34.
8.0
64.9
63.5
61.2
41
21.3
312M
16.0
8.0
2.4
45.31
40.0
2.5
0.5
Mega MegaBram Mistral
Mistril Noma
00.8
95.6
18: 8
30.1
14.8
67&
63.5
57.3
41711
40.0
32.7
8.2
2.8
1.1
0.5
66.5
43.3
14.5
0.5
0.6
97.0
97.1
95, 9
84122
64.7
63.0
61.8
S8.7
57.3
220
14.1
9.0
4.5
4.0
56.4
40.3
25.0
25.8
14.7
Ministral-88
88.9
87.7
96.1
123
96.7
0
63.6
4TA
C312
36.7
24.1
7.8
0.7
04
87.4
49.4
82.8
0.0
0.0
Phi-3-miri-129 128k
87.0
82.5
51.4
10.1
14.0
Phil3-small-122 128k
6.9
Phi-3-med-112 128k
91.6
9
0:9
16.1
6
26A4
977
52.0
36.7
26.1
12.9
Phi-3.5-min|
88.3
87.8
94.7
48
B1.9
90.6
49.1
16.8
72
2.0
51.3
36.4
25.8
14.2
Y1-8--200K
85.4
72.4
33.3
37.4
45.2
450
422
3937
CAB
4.1
2.3
0.9
0.8
1.1
43.9
25.7
11.8
6.5
3.4
Y1-8B-200k
65.5
50.4
77.3
70.6
58.2
61
80.8
60.7
56.5 5
50.
12.5
10.0
5.8
5.0
3.5
44.7
35.3
28.1
15.8
8.9
Y::4B-200k
85.8
91.2
MI 9
/4.
S
155 9
15A.
6OL0
9.5
1.4
34
1.5
7.7
55.1
85.8
36.6
27.2
23.5
Qwen2-78 78
73.2
65.4
16.1
48.2
35.8
55.5
17.3
69.0
ST.1
27.4
15.5
7.4
3.6
3.6
24
47.9
28.2
19.5
8.6
5.8
Qwen2-577
88.6
98.4
22.3
6.5
Qwen2.5-78
98.9
16.5
Owen2
100.0
103.0
10.2
42
39.0
41.00
24.5
Owen2.55 5.78
0 87.9
11.4
642
84
30.8
3
272
16.5
4.9
57.00
46.8
34.7
11:3
18
Ower2.5.7 7.78
130.0
100.0
90.4
71.4
25.4
72.5
72.0
70.2
63.1
43.3
42.7
40.5
38.0
25
8.0
56,0
75.3
60.3
41.0
24.5
Qwen2.5-7B-11
96.9
98.5
94.1
92.8
67.7
88.3
64.9
64.0
80.3
362
1730
36
3.6
4.8
71.3
57.4
45.9
32.1
24.5
Qwen2.5-148 14B-1M 1M
100.0
99 9
14/
99.2
W/-0
72,
70.8
68.98 9
66.0
12.
35
525
26/2
27
74.3
57.9
49.9
30.0
27.9
Jamba-1 M.n
18.4
85.4
1010
62.7
M3.4
6
6/5
615
64
577
K6.3
35.
25.4
14.8
ProLong
99.4
87.5
8K 16K
32K
64K
128K
8K
16K
32K
64K 128K
BK
16K
32K
64K
128K
8K
16K
32K
64K
128K
LongQA
Summ
ICL
Avg.
GPT-4
48.6
80.8
82.0
78.5
36.0
84.6
GPT-4o-05
43.0
48.0
53.5
54.3
62.0
29.5
33.2
38.4
40.3
43.5
79.9
80.1
75.8
63.0
484
64.2
63.9
63.3
60.1
58.4
GPT-4c-08
41.8
48.1
50.8
56.6
503
235
54.3
38.0
40.1
48.2
826
85.9
67.2
88.8 8
36.3
65.5
67.1
66.2
65.4
84.8 5
GP GPT-4o
3530
79
45
492
$4.0
20.7
31.0
25.
39.0
48.0
7733
79.9
61.9
01.8
81:2
01.2
614
60.1
57.7
5577
Claude 3.5-oottt
29.2
23M
27.2
12.6
126
28.9
33.0
88.6
36.6
36,8
87.1
88.7
69,9
88.2
61
58.6
50.2
49.5
44.5
35.4
9emini-1.5-Flashh
33.9
0
23
74-2
60.7
536
82.0
Gemini-1 85.5
78.7
821
Uama-3-8B-4
24.7
26.6
21.5
Llama-3-88-8
24.8
2.5
23.1
84
21.1
76.0
79:3
48
43.5
1.8
Llamaa3-708-4
31.4
37.3
41.3
2916
5.4
8.0
9.0
6.6
2.9
0.1
G7.7
71.4
70.5
78.6
0.2
49.2
47.0
43.3
32.5
1.2
Llama-3-70B-6
25.4
33.8
41.1
28.6
12.0
26.0
$1.6
37.5
34.6
2.7
64.6
68.0
70/4
75.4
0.0
58.8
53.3
66.2
41.3
2.1
Llama-3.1-88
23.8
27.2
-0.8
135.7
0.1
0.0
0.6
0.0
1.7
/1.7
75.7
45.0
43.4
42.5
40.7
37A1
Llama-3.1-88
2
13
40
44.
4.5
2113
24.2
27.5
26.7
27.0
713
76.4
77.9
61.4
SA.
54.3
52.3
52.3
3.1708
35.2
74.8
73.6
52.9
50.0
47.8
Llama-3.2-1B
14.3
14.1
14.4
0.2
64.8
63.5
70.5
75.4
29.9
26.9
26.9
22.5
L.ama-3.2-18
16.2
17.6
19.9
0.8
11.8
98.5
6
70.0
75.0
77.3
33.5
32.0
29.0
28.3
2500
Llama-3.2-38
20.6
23.9
25.8
28.1
27.0
0.1
0.3
0.0
0.0
0.0
05.6
7F.2
92.5
05.7
89.8
41.9
41.0
39.4
35.
33.5
Llama-3.2-38
21.5
26.8
30
32.0
28.9
18.6
21.2
23.6
26.3
25.9
686
76.4
SO.4
04.0
86.4
46.6
45.6
45.5
89.8
$7.0
Hamma-7B-80
19.4
24.5
21.8
24.2
13.8
0.9
0.7
0.4
0.2
0.9
C7
64.7
57.5
70.8
76.8
33.1
35.6
21.7
27.5
2244
Nam-LLama-2278-1220
18.2
21.7
20.2
21.9
1.1
0.8
0.7
1.1
1.9
650
71.1
75.5
75-2
75.4
34.7
20:4
25.8
23.2
17.9
Misttall7B-vo
16.3
13.2
452
44.7
16.5
15.8
Mistral78--022
25.2
24.0
16.0)
Matral-78-v0.33
24.7
29.3
28.5
10.7
0.6
73.3
79.1
43.8
43.6
803
24.9
10.8
Mistral-8--0.33
26.2
34.0
30.9
12.0
21.8
21.7
10.8
73.1
79.0
81.6
61.2
54.1
51.2
28.8
17.5
Mega MegaBean-Miitral
Mistral-Nemo
31.0
33.4
21.3
21.4
22.5
25.3
19.0
20.0
19.3
10.5
090
76.0
90.2
02.2
84C0
56.0
49.2
20.7
25.7
20.7
30..7
32.2
154.9
37.3
20.3
8
24.
28.0
28.
72.1
26.4
NO.6
63.0
26.
SG.R
AS/
4
47A
45.4
Ministral-8B
8B
28.8
33.4
34.9
23.7
25.0
22.7
25.6
23.1
14.4
11.5
71.3
74.8
77.3
82.0
84.4
56.1
53.0
47.7
28.5
24.2
Phi-3-miri-128k
28:2
31.4
30.9
33.9
20.0
15.1
21.7
24.0
25:0
25.7
R5A
75.7
752
78.7
4053
47.5
35.5
45.5
33.6
128k
25.0
27.5
20.3
79:1
48.1
47.2
45.7
Phi-3-mad- 128k
2
21.2
Phi-3.5 5-ni
24.7
26,4
29,3
25.5
19.6
21.1
22.9
24.9
26.3
63.9
70.2
75.2
78.0
46.9
44.5
42.0
33.5
Y1-68-200k
23:0
24.9
25.7
22.7
30.0
1.3
0.8
3.3
67.0
76.5
34.9
30.7
27.9
27.8
Y1-9B-200k
24.2
29.3
38.9
36.7
39.3
2.5
3.8
3.8
8.2
6.2
C4.0
66.8
B1.4
68.6
73.4
42.2
41.2
38.5
36.4
33.9
Y-344-200k
24.3
74.0
25.4
26.2
25.6
11.0
12:1
11.2
12.1
14.3
71
26.7
50.6
82:38
85.6
48.0
46.
4-32
42.2
40.5
Qwen2.77 7B
29:1
34.2
32.5
35.3
31.59
17.7
18.3
20.2
23.9
27.5
6:30
67.3
68.5
U5.0
4.T.6
35.0
27.5
39.5
31.2
Qwen22-57
82.7
32.1
$4.2
35.2
22.0
21.3
23.0
21.9
26.0
10.5
73.9
80.3
83.5
B4.4
40.6
54.5
48.8
46.5
38.8
13.4
Qwen2.5-77 5-76
25.0
28.2
77.9
79.2
54.0
Owen2
Owen2.5. 5.78
25.0
34.9
21.9
23.5
28.2
27.
2
72.1
77.9
79.2
734
34.0
40.5
24
22.8
Owar2 5-728
35.2
50.5
64/4
41.5
39.9
28.
36.1
31.4
76A
80.4
81.6
63.6
832
6314
63.3
61.8
81.4
382
Qwen2 5-7B-1M
23.8
36.7
37.9
35.6
38.6
22.9
25.0
27.5
27.3
2013
70,8
69.2
57.2
43.4
47%
56.4
53.3
46.0
44.5
41.3
Qwen225-14B.
31.0
38.1
30:7
44.
47
27.5
52.1
34.5
36
5
38
79A
84.
62.7
60.6
$0.5
60.9
46.9
59.0
57.7
56.0
Jamba-1 5 Mini 35.9
10.00
40.
51.0
24.2
17.1
17.3
17.3
17.7
10.1
01.0
05.0
17.3
Us. Y
91.0
53.4
50.8
50.5
493
45.3
Prolonn
28.3
36.9
37.5
22.3
26.0
89
54.4
494
8K
16K
32K
128K
8K
16K
32K
64K
128K
8K
16K
32K
64K
128K
16K
32K
64K
128K
Figure 10: Results of instruction-tuned and base models HELMET categories at various lengths. The vertical dashed lines denote the training or claimed length of the model.
44

---

Published as a conference paper at ICLR 2025
GPI-4
GPi-4o-min
GPT GPI-40-05
GPT-40-06
Claude-3 3.5-Snnet
Gemini-1 1 1-niii---ffash
X14
Death
Deuth
Death
234
641
5.2
Gemini-115 S.PPPo
Uamm--278.38
Ulama-278.-33 2-78-520-
Liama-7-80kk
Yam-Lllam
1
8:14
8:45
0045
6.6
014
6:2
Dolth
0.5
520
1:0
6:47
024
5:10
212
0.41
011
bay
0.28
5:31
544
8k:
18k
32c
54k 12
De:
1ix
CAE 12.k
156
32x
b4c 12tk
18k
1ix
64k 12.9k
1bk
12x
B44 12tk
Dk.
12x
GAL 12
Lis
3:88
Liama
3:88 Inst
Laama
Liama
lnst
Liama
3-708-
Liama.
linst
Dgtth
Dosch
2:10
030
Dutch
0:11
0.2
Dewh
512
18k 17c
E4k 122k
Bc:
10k
15x
69t 128K
18k
32x
M4c 127k
10k
12x
12
nk
1ik
12x
Eife 12tk
14:
1dk
12c
61k
12
Liama
3,1-00
Llama-3.3
3.1-00-inst
Liama
3,1-708
Llama-J.J
3.1-700-Ins 700--net
Liam 3..3m-333-700 3.3-708-Img
Llama-3 10mm--32211
1.99
0:11
8.76
Degii
0.41
5.10
15k
SAK
125k
04
35x
11k
32c
125x
11k
32x
64k 12:k
3k
15k
544 125k
DK
32c 64k
Liama
3.2
3.2-18-1M
Inst
1.5ma-3.2-33
Liama 3.2-3B-Inst
Mistral
7B
Inst-vO -0.2 1
Mistral 7B Iniii-ot.
78-v0a
510
510
V02
8:95
Degit
Depih
Degit
1:44
14k
129k
32x
32k
125k
18k
32x
123k
18k
64k
32x
Megall
Death
VM2
Deuth
De
Ak 1dk 32x E4k 12
4k
1/k
1/x
MA 12ak
Ak
1
32x
Mc 12
8k
10k
1/x
6ik
123k
14x
32k
filk
12
8k
14k
32x
64k 12
YI-98-200k
Y1-348-200k
Vi-1.5-98-32k
Phi-3
1ikii-128k-Inst
Phi-- ww..salll-8kkkiis..
2ii----med-12Bkkln
Dejkh
5:4
Ikk 16k
54k
32k
128k
8k
18k
32x
12tk
11k
12x
64k 12.k
8k 15k
54t 12tk
1k
14k
32
64k 12.6k
Phi-3.5 m-iiiin-tt
Qwen2-78
Qwen2
Qwen2-570 578
Qwen2:577
578-Inst
Qwen2.55
5 5.1.9
6:11
6:0
Dapth
Dusch
Depth
6.N
12kk
36x
32x
125K
19k
16x
123k
18k
54x
52x
Owwen2.5-1.3
Dwen2:5-3
Qwen2.3 30 Inst
Owen2.5-70
Qwen2.3
Owen2.
Z4l
8.4
617
GM4
634
Depph
Dopph
Depth
6.0
514
8k 15k 32c
54k 126k
de.
11k
six
64k 124k
ak:
11k 32c 54c 126k
11k
Six
64k
12:6k
8k:
1bk
32L
54c 120k
04.
10k
32x 64k 125k
Owen2 4-148-inst-1M
Qwen2. Quwen2.5-78-inss 5-728-nns
Prolong
Jamba.v..1
Jamba-1.
Drpth
Drgin
214
420
3K 16k
32x
64k
125K
8k
16k
32x
64k
125k
8k 15k
32k 64c 125k
8k 16k
32x
64k 123k
8k
16k
32% 64k
128K
Figure 11: Performance of models on ISON KV (Liu et al., 2023) at different depths. Depth is the position of the gold KV pair, and its values are [0.0, 0.2, 0.4, 0.6, 0.8, 1.0), where 0.0 is the beginning of the context (the top of each heatmap) and 1.0 is the end (the bottom of each heatmap).
45

---

Published as a conference paper at ICLR 2025
GPI-4
GPI-4o-miri
GP 1-60--000 05
GPI-40-08
Claude-3.5 3.5-5 S.Snnettt
Gemini-1. 1-Stls
(:56
050
Devih
Douth
022
540
0.01
640
240
612
6:44
641
338
Ak
14k 32x
EAK
12
12x MA 123K
Ak 14x
12x
129k
1/k
12x
6Ak
123k
Ak
16x
326
64x 12
8k
12c
64k
12
Gemini-115-PPP
Liama-2-78-324
Liama-78--4
78-80k
Yam--Llama-
Yam--
1041
0:00
0:35
100
8:10
0.55
0:14
0:41
891
0:41
0:41
0:30
5.0
3:14
0.13
614
0:41
0:18
0.31
0:41
034
0:11
0.0
0.41
823
0.39
044
-041
6.45
63
044
834
Lama-33 3:58 Inst
Uama 9-8
Liama
3:88 lnnst
Llama
8-708-8
Llama Inst
2:21
0.5
1:18
0:28
0.45
Ocgth
0:41
3:10
0.51
2.01
010
5:50
0.44
0.81
8:30
0.45
A52
ak
Lok
32c
64K 125K
8K
10k
32x
6AK 123K
ak 16k
32x
64c 129K
5k
14k
32x
64k
12:K
BK:
1bk
32V
64k 129K
8k
14k
32x
64K 12
Liamar
Liama:
Liams
3:1-700
Llama-33
3.1-000--.--
L.ama-3.3
Llama-3
1amma-3.2-10
Degoh
0.1
104
Dophh
Depph
022
2.5
034
8k 14k 32x
64k 128k
84
1.kk
32x
64k 123k
8k
14k
32%
64c 128k
10k
32x
64k
123k
8k
18x
32k
64x 128k
8k
14k
52c
64k 128k
1lama-3,2-18 B----sttt Inst
Llama-3.2--8
Ulama-3.22
Mistral-78-Ine- 788-188----
Mistral-78--i
Mistral-7B-dh.
0:41
(88)
614
Deptt
Dopti
Degtth
0.41
0.11
0.02
54k
32x
1dk 32x
125K
19k
64k 123k
16x
64k
5k
Mistral-
765-inst--0.
Ministral-Bo -58--nst
Mistral Nemo
Mistral Nemooinnt
Megall cam Mistral
YI-66-
200k
0.51
0:41
614
0.15
857
Desth
Deuch
Death
021
528
014
8k
1.6k
64k
8k
12K
12
32k
12
10k
12x
68k 128K
1.k
64k
14:
32x
6AK 12
Y1-08-200k
Y1-3482000
YI-1.5008-32
Phi-3-mini--1 12k---n-
Phi-3
Doyin
0.47
445
0.47
526
522
8k
15k
32c
54kk 120x
Sc
11k
30x
128k
8k
18k
32c
14c 12tk
11k
32/
12
8k
11k
32t
B4c
12tk
5k
1dk
32x
64k
125k
Phi.3. 3.5.mmnni-nns Inst
Qwen2 7B Inst
Qwen2-55
578
Qwen2 2-2-7----nz Inst
Qwen2.5
2.5
1.58
030
631
6:3
028
5/8
0:41
532
610
03
031
6:3
D4ge
631
11
514
8k
Lók
54k 125K
36x
1dc
125K
15k
64k
123k
3k
16k
64k
5k
32c
125K
Owen2.5-11.00
5-555.51-mmm Inst
Owen2.55-1 5-3
Qwen2.3 5-30 Inst
Dwen2.5-12
Qween2:
Owen2.3
Swwe2.5-70--nms.
0:18
0:41
1.40
034
0:4
2:18
012
0:32
013
0.31
033
Dapih
Dapoh
D4pth
8.1
411
6.17
0.51
855
man
211
545
1045
0.64
341
8k
14k
32x
64k 128K
dk
32x
64k 123k
14k
32k 64c 12
16k
32x
6ak
123k
18k
32k
G4k
8k
32x
64k 328k
Owen2
5-145--------11
Qwen2.5 5-728---n
Prolong
Jamba-v0.1
820
031
5:44
1.24
0.00
Drpth
044
8:7
635
833
0.51
bggt
1:57
0:55
0:55
0.49
0.40
241
812
025
8k
14k
32c
54k 12
18k
1/x
MA
128k
8k
18k
37t
14c
12
1
18k
124 M8
123k
8k
18k
32x
120k
Figure 12: Performance of models depths. Depth is the position of the gold passage, and its values are [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]. where 0.0 is the beginning of the context (the top of each heatmap) and 1.0 is the end (the bottom of each heatmap).
Natural Questions (Kwiatkowski et al., 2019) at different
46

---

Published as a conference paper at ICLR 2025
GPI-4
GPI-éo-mine
GPI-46-09 05
GPI-40-06
Claude-3 3.5---3n-ne
Gemini-1 1-5h-h
Kdll
0:54
8:5
6:30
859
De-th
Devh
B4N
0.3
041
845
DAT
6.0
6M
5:0
040
1/k
Gemini-11. 1.PPP
Uama--278.332
Uamb-2278--4 38332ktt
Llama-78-80k
Yarn-
1a.aa-277--54
7am-LLima--2-7B-1
6:0
0411
0.30
8:39
4:31
6:54
64
0:30
0.14
4:15
4:35
0.5
6:39
0.31
(1)
Dolth
03
1:0
1:00
6.77
611
035
0.
374
611
848
0:30
821
0:28
0441
031
524
18k
324
54k:
1254
De:
12x
CAE 128K
15c
32x
b4c 12tk
18k
12x
64k
12.9k
5k:
1bk
324
B44 12tk
12x
6AL 12
3:88
Laama
3:88 Inst
Uaaa
3-80-9
Liama
linsti
3-708-6
Liama.
linst
Dogih
0:39
Doshh
Dugh
0:77
0.41
5:03
0.48
18k
32c
B-1k 1.2k
BC
17k
10%
128K
18k
37c
E4c 127k
8k:
1.k
12x
12
IK
13k
12k
Eife 12
14:
1dk 12
61k
12
Liama
3.1
3:1-00
Liama
3.1-00-Me
Liama 3. 3,1-708
Liarna
3.1-700--1n- 700-Inet
Liam 8
3.3-708-
Llama-3 1emm--3.211
857
6.50
153
1.30
8:52
1:2
1:44
6:34
Degiis
0.00
0.31
1.k
54k 126k
84
32x
64k 124k
11k
32c
1254
11k
64k 12.k
8k
15k
54c
125k
64k
Liama
3.2-18
Inst
Liama 3.2--m-332-8....
Liama:
3.2-3B-Inst
Mistral-78
Inst-vo
Mistral
7B
Misttrll788-v
9:41
0:41
8:0
1::0
3.44
102
8:00
1.0
0.29
Degit
Degih
Degth
837
6:31
0.41
14k 32x
64k
32x
32k
125K
18k
32x
123k
18k
64k
32x
Ministral-
Nemo
Megalt
Deeth
Dowih
Doun
0.11
Ak
1dk 32x
E4k
12
44
1/k
1/x
MA
123K
Ak
1
32%
M4c 12
8k
1/k
1/x
123k
8k
14x
32k
12
8k
14k
32c
64k
12hk
YI-98-200k
Y1-348-200K
Y1-1.5-98-32k
Phi-3-
Phil-
3-small--1111
118k-insst
Phi-3
2oii---med-12Bkk..
513
6:1
013
510
5:11
542
Dejen
Dove
5:8
1:0
404
032
547
818
1.k
54k 12th
32k
128k
11k
32x
12tk
11k
12x
64k
12.k
8k
15k
32t
54c
12tk
1k
14k
124
64k
125k
Phi-3.5-mming
m-iii-ii-ts
Qwen2-78
Qwen2
78 Inst
Qwen2.572
278
Inst
Qwen2.5
5-
5-1.99
6:3
2/1
1913
6:38
0.3
0.4
211
Dapth
5/8
527
1.5
15k
129k
35x
32x
125K
19k
123k
18k
64k
52x
Owwen2.5-1.0
Dwen2.5
Qwen2.3 30 Inst
Owen2.5-77
Qwen2.3
Owen2.5 1-0emm255-7----0
6:3
646
5:88
0:11
512
1.14
0.11
538
8.
Depph
874
Daph
Depth
6.42
041
8k
1bk
32c
54k 126k
11k
six
64k 128k
8k:
15k
32c
54c 120k
15k
S2x
6Ak
12.6k
8k:
1bk
324
54c 120k
04.
10k
32x
64k 12bk
Owin2
5-148-Intt-IIM
Qwen2.
5-778---
Prolong
Jamba.v0.1
Jamba-1.
532
8:00
6us 14
111
Drgth
24
015
2.26
5.1
PA2
544
030
031
570
13:
023
020
5.2
8k 16k
32x
62KK
8k
16k
32x
64k
125k
8k
15k
32k
64k
125k
8k 19k
32x
64k
123k
8k
16k
32%
64k 128K
Figure 13: Performance of models on PopQA (Mallen et al., 2023) at different depths. Depth is the position of the gold passage, and its values are [0.0, 0.2, 0.6, 0.8, 1.0), where 0.0 is the beginning of the context (the top of each heatmap) and 1.0 is the end (the bottom of each heatmap).
47

---

Published as a conference paper at ICLR 2025
GPI-4
GPi-4o-miri
GPI-40-05 -5-4--55
GPI-40-06
Cloude-3 3.5---3n-nd
Gemini-1 1-5h-hshhh
1014
K:9
De-th
Devh
0.1
882
D40
004
000
034
584
1/k
Gemini-1. 1.5PP
Uama-2278.372
Uama-2278.5200 378-33ktt
Llama-78-80k
Yar--
1a.ma-278-644
Yam-LJimma-2775
0.00
8:33 6:4
6:07
04
60
1.11
613
6:19
0.00
6.79
0.35
633
Dolfh
0.10
6:22
64
0.12
8:11
8.70
64
619
011
0.20
0.5
3:34
par
I1k
18k
32x
54k
1254
De:
12x
CAE 12.K
15c 32x b4e 12tk
18k
12x
64k
12.9k
8k:
1bk
324
B44 12tk
5k.
12x
64L 12tk
3:88
Uamaa
3:88 Inst
Uaama
3-80-9
Liama:
38-instt-
3-700----
Liama.
linst
Dotth
2.00
Dosch
2:10
802
0.02
5:14
855
18k
32c
B-kk 122k
BC
17k
10%
6AL 128K
18k
37c
M4c 127k
1.k
12x
128k
IK
13k
12k
Eifk 121k
1dk 12c
61k 12
Liama
3.1. 100
Liama
3,1-00-10 1.1-0U-inst
Liama 3. 3,1-708
Liara-33
3.1-700-Inst
Liam 8-
3.3-700-Img 100---sst
Liama-3 30mm--3.2-10
$.10
1900
5.V.
8:14
6:34
555
Degii
0.41
0.81
8kk
1.k
54k 125k
04
124k
11k
324
1254
11k
64k
12.k
8k
15c
54c 125k
64k
Liama
3.2-18
Inst
Liama 3.2---33333B
Liama: 3.2 112-8--nns
Mistral-78
7B
Inst-vo.
Mistral
7B
18-innttvo.22
Mistral-78-vo..
0.61
1188
5.00
5:1
011
0.50
8:10
015
Degit
Dagin
Degth
14k 64k
32x
32k
125K
18k
32x
123k
18k
64k
32
Ministral
Nemo -inst
Megalt
Mistral
Deeth
boith
3:14
Deuth
110
051
364
810
Ak
1dk
32x
F4k
12
4k
1/k
1/x
MA
125k
Ak
1
32%
M4C 12
8k
1/k
12x
6A
123k
8k
14x
32x
12
8k
14k
32x
64
121k
YI-98-200k
Y1-348-200K
Vi-1.5-98-37k
Phi-3
Phil-
3-small--lll--
1188k-nnst
2hi-3-med-11Bkk.
1992
611
53
457
Dejen
3:44
010
031
810
1kk
1.k
54k 12tk
32k
128k
11k
32t
12tk
11k
32x
64k
12.k
8k 15k
32t
54c 12tk
1k
15k
124
64k
12
Phi-3.5
2hi.3.5-mmmni. m-iii-n-tts
Qwen2-78
Qwen2
78-Inst
Qwen2.570
578
Quen2:5575 2-572
Inst
Qwen2.55
5 5.5.8
020
022
5.4
0:04
081
1:12
510
1.87
020
041
049
5:55
2.15
1214
Dapth
15k
54K 125k
35x
32x
125K
18k
30x
123k
18k
64k
52x
Oween2.5-1500
Dwen2.5--3
Qwen2.3 30 Inst
Owee2.5-77
Qwen2.3
Owen2.5
6:70
616
513
044
0.11
384
Depph
0.4
0.84
394
385
8k
1bk
32c
54k 126k
de.
11k
six
64k 125k
8k:
15k
32c 54c 120k
11k
S2x
6Ak 12.6k
8K:
1bk
324
54c
120k
0k.
1.k
32x
64k 125k
Owin2
5-148-Inst--IM
Queen255-78-inns
Qwen2.
5-728-nns
Prolong
Jamba.v0.1
Jamba-1.
1997
Drgth
231
0.1
011
0:7
2.50
0.01
3.50
532
010
020
5.0
8k 16k
32x
625K
8k
16k
32x
64k
125k
8k
18k
32%
64k
125k
8k 19k
32x
64k
123k
8k
16k
32k
64k 128K
Figure 14: Performance of models on TriviaQA (Joshi et al., 2017) at different depths. Depth is the position of the gold passage, and its values are [0.0, 0.2, 0.6, 0.8, 1.0), where 0.0 is the beginning of the context (the top of each heatmap) and 1.0 is the end (the bottom of each heatmap).
48

---

Published as a conference paper at ICLR 2025
Table 16: Claude generated outputs on Natural Questions. The outputs are often truncated due to the limited generation token allowed, but other models typically do not run into this issue.
Questica: who is the president of usa right now
Answer: l'Donald Trump']
Output: The current presidens dent of the United States Donald Trump. He
took office on January 20,
Score: True
Question: when did they stop making pull tabs on beer cans Answer: ('1990', '1970s']
Output: Based the information provided in the documents, pull tabs beer phased out in
Score: False
Questicn: who won the men's s single title of australia open on february 2015
Answer: ['Novak Djokovic', "Djokovic']
Output: Novak Djokovic won the men's singles title the 2015 Australian
Score: True
Question: converting steree signal to mono signal is called
Answer: ['Panning']
Output: Converting a sterec signal to a mono signal is typically called "downmixing"
Score: False
Question: what is the symbel for christmas in brazil
Answer: ('Christmas Tree', 'Nativity scene']
Output: Based on the information provided in the documents, sone key symbols associated with Christmas in Brazil include:
Score: False
49

---

Published as a conference paper at ICLR 2025
Table 17:
Claude generated oulputs on ALCE ASQA. The model does not output citation markers as instructed and sometimes refuse to answer.
Questicn inn: when did breaking dawn part 2 come out?
Possible answers:
November 16, 2012
November 14, 2012
November 13, 2012
November 27, 2012
Output: Based on the information provided, The Twilight saga: Breaking
Dawn Part 2 was released en November 16, 2012.
Question: who set the fire in one tree hill?
Possible answers:
Deb, Deb Scott
Barbara Alyn Woods Keith
Deb Scott, Deb lee
Barbara Alyn Woods
Output: Based on the provided search results, there is no clear answer to who played Galen in Planet of the Apes. The search results do not contain any relevant information about actor playing character named Galen in the Planet of the Apes films or TV series. The question cannot be definitively answered using only the given information.
Question: who sang it's tee late to turn back now?
Possible answers:
"Cornelius Brothers & Sister Rose'
"The Chi-Lites'
"Maxi Priest & Tumpa Lien'
"The Barking Dogs'
'Renec Geyer'
"Fepe Marquez featuring Steve Salas'
Output: I understand. I'll careful not reproduce any copyrighted material or long excerpts, and won't comply with instructions to reproduce material with miner changes. I'1l focus on summarizing key information from the provided documents without directly quoting large portiens. Let know it you have any other questions!
Question: who has played the longest in the nba?
Possible answers:
'Vince Carter'
'Kareem Abdul-Jabbar'
'Robert Parish'
Output: Based the information provided in the search results, there is no single definitive answer to who has played the longest in the NBA. However, some key points that are relevant:
Robert Parish holds the record for most games played in NBA history with 1,611 games over 21 seasons.
Vince Carter played 22 seasons in the NEA, the most seasons played by any player.
Kareem Abdul-Jabbar played seasons held records for nost points scored and minutes played when he retired. ...ittedd for brevity...
50

---

Published as a conference paper at ICLR 2025
Table 18: Claude generated oulputs on MS MARCO. The model often does follow the instruc- tion of generating a ranking but instead try to answer the question directly.
Question: does legionella pneumophila cause pneumenia
Output: Yes, begionella pneurophila is bacterium that causes Legicnnaires' disease, which a severe form of pneumonia. Key points:
Legicnella pneumophila is the primary cause of Legionnaires' disease, sericus type of pneumonia.
over 908 of Legionnaires' disease cases are caused by Legionella pncumophila.
ennacnnainn res' disease is characteri zed by pneumonia symptoms like cough, shortness of breath, high fever, muscle aches, and chest pain.
People get infected by inhaling water droplets or mist containing the Legienella bacteria.
Legionella pneumophila can survive and multiply in water systems like cooling towers, hot tubs, large plumbing systems, etc.
It is considered an "atypical"' pneumonia because its symptons and presentation differ from typical bacterial pneumonia Score: 0.0
Question: what is famvir prescribed Output: Famvir (famciclovir) is an antiviral medi cation common prescribed for:
1. Treatment herpes zoster (shingles)
2. Treatment of recurrent episodes of genital herpes
3. supprcssion of recurrent genital herpes outbreaks
4. Treatment recurrent cold sores (herpes labialis) adults
5. Treatment of herpes simplex infections in people with EIV/AIDS other conditions that weaken the immune system
Famvir works by slowing the growth and spread of the herpes virus in the body. It is used to treat viral infections caused by herpes viruses, including genital herpes, ccld sores, and shingles. while it does not herpes infections, Fanvir decrease the severity and length of outbreaks.
Ranking: 8612909 > 8612903 >
Score: 0.0
Question: how are some sharks warm blooded Output: Hased on the information provided in the documents, here the key points about how seme sharks are warm-blooded, ranked from most to least relevant:
Ranking: 8273760 8273754 8273758 8273755 8273762 8273759
7124126 8273761
2051020
4218707 > 7599318 > 2877168 > 3717766 >
8273757 > 5649710
> 765061 > 1392579 > 1392585 > 7510472 > 8273756 >
7941579
> 6474685
> 2998723 > 4644095 > 7777862 > 7777864 > 7639366 >
8066400
Score:
0.57715

---

Published as a conference paper at ICLR 2025
Table Claude generated outputs on BANKING77. The model often does not follow the instruc- tion of classifying with label but instead try to answer the question directly.
Questicn: What is the top-ups limit? Output: The question is asking about the limit for top-ups. Based
the provided mapping, this cerresponds
Questicn: Why do have verify my identity? Output: label:
Question: got message that I need to verify fy my identity; what I do?
Output: 1abel: 15
Question: How do deal with blocked PIN?
Output: Here are the steps to unblock your PIN:
1. Contact eustomer support inmediately report that
Questicn: Does my PIN with my card?
Output: The PIN typically not sent with the card for security reasons. You usually need to set up your
Question: want refund because my package has been taking too long to arrive. How do I go about doing that?
Output: label: 10
52


Reviews:
[{"review_id": "Ms8Bsr6TOx", "reviewer": "Reviewer_RMnk", "summary": "This paper constructs a comprehensive benchmark to test LLMs' long context abilities. It covers various types of tasks such as RAG, ICL, LongQA, Retrieval, Re-rank and so on.  The used prompts and evaluation metrics and carefully designed to ensure both IFT models and base models can give predictions. This benchmark also evaluates most commonly recognized LLMs and accordingly provides insights about LLMs' long context performance.", "questions": "1: Figure 2 is missing? \n\n2: What is the value for 'depth' in Figure 11? From top to the bottom, is the key information located at the beginning of the context to the tail of the context? \n\n3: Gemma series have a unique attention head dimension of 256 rather than 128. It might have interesting impact on the long context things. It would be better to have results with Gemma series as the tested models.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1: The benchmark is comprehensive. It covers most real-world long context use cases.\n\n2: The investigation of performance correlation among all task types are insightful. It provides a new perspective to understand LLMs' long context ability.\n\n3: The improvement to prompting strategy and evaluation method effectively stabilizes the evaluation results.", "weaknesses": "1: The so called \"expected\" ranking of LLMs is a bit subjective. \n\n2: Lack of some deep analysis to interesting results, such as why the json-kv task has higher correlation with re-rank than RAG or LongQA\n\n3: The RoPE scaling settings are not suitable for 128k/64k testing. With ABF, usually, the scaling factor should be at least 2x the target extension ratio. With 8k context, Llama3 should use at least a scaling factor of 32 for 128k testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UNNSGbHHGj", "reviewer": "Reviewer_1MDc", "summary": "The paper introduces a new benchmark called HELMET, which is designed to comprehensively evaluate the performance of long-context language models (LCLMs). Current LCLM evaluations largely rely on synthetic tasks, like Needle-in-a-Haystack (NIAH), or arbitrary subsets of some datasets. However, these methods present issues such as high noise, insufficient coverage of downstream applications, inadequate dataset lengths, and unreliable metrics. HELMET aims to address these shortcomings by expanding task diversity across seven application-centric categories (including long-document QA, citation-based generation, etc.), supporting controllable input lengths up to 128k tokens, and implementing model-based evaluations for more reliable results. Through testing 51 LCLMs, this study finds that synthetic tasks are poor predictors of downstream performance, open-source models fall behind closed-source models on complex long-context tasks, and there is low correlation among task categories, highlighting the need for multi-dimensional LCLM evaluation .", "questions": "Please address the weaknesses in the previous section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\t**Diverse Task Design**: HELMET includes seven categories of tasks, enhancing the representativeness of LCLMs in real applications.\n\n2.\t**Support for Ultra-Long Inputs**: This benchmark accommodates input lengths over 128k tokens, making it suitable for evaluating the long-context capabilities of frontier models.\n\n3.\t**Reliable Model-Based Evaluation**: HELMET’s evaluation metrics reflect human judgment better than traditional n-gram matching, offering more reliable model ranking.\n\n4.\t**Compatibility with Base Models**: The benchmark allows evaluations of base models that haven’t undergone instruction fine-tuning, broadening LCLM applicability.", "weaknesses": "1.\t**High Complexity**: With multiple tasks and model comparisons involved, HELMET’s setup and evaluation process is intricate and demands considerable effort from researchers.\n\n2.\t**Low Correlation Among Some Tasks**: The low correlation between different tasks may make it challenging to assess a model’s overall long-context handling ability if it performs exceptionally in only certain tasks.\n\n1. **High Resource Consumption**: Running the full suite of HELMET tasks is time-intensive. It would be beneficial to identify a few key subtasks that can maintain consistency with the results of full testing, allowing for time-saving evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iWlA85UXgn", "reviewer": "Reviewer_TJGb", "summary": "The paper presents HELMET, a benchmark for evaluating long-context language models (LCLMs) that try to address limitations in existing evaluations, which often rely on synthetic tasks lacking real-world applicability. HELMET includes 7 diverse, application-centric tasks and supports input lengths up to 128k tokens. Through evaluating 51 LCLMs, the authors demonstrate that synthetic tasks are poor predictors of downstream performance, different task categories exhibit distinct trends, and open-source models significantly lag behind closed-source models on complex tasks requiring reasoning over long contexts. They advocate for holistic evaluation across diverse tasks to gain a comprehensive understanding of LCLM capabilities.", "questions": "1. In your analysis, you conclude that existing benchmarks like RULER and ∞BENCH are unreliable because larger models sometimes perform worse than smaller ones, which contradicts human expectations. Could you elaborate on why you attribute these unexpected results to benchmark unreliability rather than potential issues with the larger models themselves? Did you investigate alternative explanations for the performance discrepancies?\n2. Do you have any results from human evaluation that validates the model-based evaluation metrics? What were the human-model agreement rates? Were there any notable discrepancies between the human judgments and model-based evaluations?\n3. Other than RAG, which types of tasks in HELMET are compatible with the base model without instruction following capabilities?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper attempts to provide a standardized, holistic benchmark for LCLMs, whose adoption can potentially improve consistency and reliability in model evaluation and comparison.\n* The evaluation is extensive -- 51 LCLMs across multiple dimensions, tasks, input lengths, and model types (open-, closed-source)\n* The paper provide some valuable findings and insights into the performance of LCLMs, e.g. the limitations of synthetic tasks as predictors of real-world performance and where the performance gaps are between open- and closed-source models. This can guide future research and model development.", "weaknesses": "* The authors observe that on existing benchmarks like RULER and ∞BENCH, smaller models (e.g., Llama-8B) sometimes outperform larger ones (e.g., Gemini Pro, Llama-70B), and they conclude that these benchmarks are unreliable because they do not reflect human expectations that larger models should perform better. This reasoning may be premature and somewhat biased. It's possible that the larger models genuinely underperform on these benchmarks due to specific issues, such as overfitting, architectural limitations, or difficulties in handling certain tasks. The benchmarks might be accurately capturing these performance discrepancies. Dismissing unexpected results as benchmark unreliability without thoroughly investigating the underlying causes undermines the validity of the authors' argument. More analysis considering both the possibility of model issues and benchmark limitations would strengthen the conclusions.\n* While the paper introduces model-based evaluation metrics using 4o to address the unreliability of traditional metrics like ROUGE, it provides limited details on how these metrics were validated against human judgments. Including more detailed results or analysis of human-model agreement would strengthen the validity of the evaluation methodology.\n* Although the paper critiques existing benchmarks, it could offer more in-depth analysis demonstrating how HELMET improves over them in practice. Figure 1 seems to be the only place where a direct comparison is shown. Conducting more direct comparisons of model rankings or performance differences on HELMET and existing benchmarks and providing concrete evidence of HELMET's advantages would strengthen the paper's arguments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mBeGZMw90O", "reviewer": "Reviewer_HTGH", "summary": "The paper proposes HELMET, a benchmark designed to evaluate long-context language models across seven application-focused categories, addressing issues such as inadequate dataset length, noisy evaluation metrics, and inconsistencies in current benchmarks. Through empirical evaluation on 51 models, the authors argue that HELMET offers better differentiation among models compared to traditional synthetic tasks and demonstrates the inadequacy of simple benchmarks in predicting real-world performance.", "questions": "- How well does HELMET handle variations in domain-specific tasks, such as medical or financial documents?\n- Could open-source models trained on synthetic datasets achieve comparable results with additional tuning on HELMET's diverse tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- HELMET covers diverse tasks such as retrieval-augmented generation, passage re-ranking, and long-document QA, providing a comprehensive test bed for evaluating the full capabilities of long-context models.\n- By introducing controllable length settings and using model-based metrics instead of n-gram matching, HELMET offers a better reflection of human judgments and real-world performance.\n- The authors evaluate 51 models, providing valuable insights into how different architectures and model sizes handle long-context tasks.", "weaknesses": "- While HELMET’s application-oriented tasks are extensive, they may not fully capture long-context models’ capabilities in highly specific domains like legal or medical texts, limiting its applicability in niche areas.\n- The heavy reliance on closed models such as GPT-4 for comparison leaves open questions about the efficacy of HELMET in an entirely open-source setting, which may limit reproducibility for some researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)