HELMET: How to Evaluate Long-context Models Effectively and Thoroughly

Authors: Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, Danqi Chen
Primary area: datasets and benchmarks
Keywords: long-context language models, benchmarking

Abstract:
Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks. However, it remains unclear whether these benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison. We investigate the underlying reasons behind these practices and find that existing benchmarks often provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models. In this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based evaluation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict downstream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions---the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream performance; ultimately, we advocate for a holistic evaluation across diverse tasks.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730437317961

thread_type:
official_review

<thread_object>
creator: reviewer_htgh

<content>
summary:
The paper proposes HELMET, a benchmark designed to evaluate long-context language models across seven application-focused categories, addressing issues such as inadequate dataset length, noisy evaluation metrics, and inconsistencies in current benchmarks. Through empirical evaluation on 51 models, the authors argue that HELMET offers better differentiation among models compared to traditional synthetic tasks and demonstrates the inadequacy of simple benchmarks in predicting real-world performance.

strengths:
- HELMET covers diverse tasks such as retrieval-augmented generation, passage re-ranking, and long-document QA, providing a comprehensive test bed for evaluating the full capabilities of long-context models.
- By introducing controllable length settings and using model-based metrics instead of n-gram matching, HELMET offers a better reflection of human judgments and real-world performance.
- The authors evaluate 51 models, providing valuable insights into how different architectures and model sizes handle long-context tasks.

weaknesses:
- While HELMET’s application-oriented tasks are extensive, they may not fully capture long-context models’ capabilities in highly specific domains like legal or medical texts, limiting its applicability in niche areas.
- The heavy reliance on closed models such as GPT-4 for comparison leaves open questions about the efficacy of HELMET in an entirely open-source setting, which may limit reproducibility for some researchers.

questions:
- How well does HELMET handle variations in domain-specific tasks, such as medical or financial documents?
- Could open-source models trained on synthetic datasets achieve comparable results with additional tuning on HELMET's diverse tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659388699

thread_type:
official_review

<thread_object>
creator: reviewer_tjgb

<content>
summary:
The paper presents HELMET, a benchmark for evaluating long-context language models (LCLMs) that try to address limitations in existing evaluations, which often rely on synthetic tasks lacking real-world applicability. HELMET includes 7 diverse, application-centric tasks and supports input lengths up to 128k tokens. Through evaluating 51 LCLMs, the authors demonstrate that synthetic tasks are poor predictors of downstream performance, different task categories exhibit distinct trends, and open-source models significantly lag behind closed-source models on complex tasks requiring reasoning over long contexts. They advocate for holistic evaluation across diverse tasks to gain a comprehensive understanding of LCLM capabilities.

strengths:
* The paper attempts to provide a standardized, holistic benchmark for LCLMs, whose adoption can potentially improve consistency and reliability in model evaluation and comparison.
* The evaluation is extensive -- 51 LCLMs across multiple dimensions, tasks, input lengths, and model types (open-, closed-source)
* The paper provide some valuable findings and insights into the performance of LCLMs, e.g. the limitations of synthetic tasks as predictors of real-world performance and where the performance gaps are between open- and closed-source models. This can guide future research and model development.

weaknesses:
* The authors observe that on existing benchmarks like RULER and ∞BENCH, smaller models (e.g., Llama-8B) sometimes outperform larger ones (e.g., Gemini Pro, Llama-70B), and they conclude that these benchmarks are unreliable because they do not reflect human expectations that larger models should perform better. This reasoning may be premature and somewhat biased. It's possible that the larger models genuinely underperform on these benchmarks due to specific issues, such as overfitting, architectural limitations, or difficulties in handling certain tasks. The benchmarks might be accurately capturing these performance discrepancies. Dismissing unexpected results as benchmark unreliability without thoroughly investigating the underlying causes undermines the validity of the authors' argument. More analysis considering both the possibility of model issues and benchmark limitations would strengthen the conclusions.
* While the paper introduces model-based evaluation metrics using 4o to address the unreliability of traditional metrics like ROUGE, it provides limited details on how these metrics were validated against human judgments. Including more detailed results or analysis of human-model agreement would strengthen the validity of the evaluation methodology.
* Although the paper critiques existing benchmarks, it could offer more in-depth analysis demonstrating how HELMET improves over them in practice. Figure 1 seems to be the only place where a direct comparison is shown. Conducting more direct comparisons of model rankings or performance differences on HELMET and existing benchmarks and providing concrete evidence of HELMET's advantages would strengthen the paper's arguments.

questions:
1. In your analysis, you conclude that existing benchmarks like RULER and ∞BENCH are unreliable because larger models sometimes perform worse than smaller ones, which contradicts human expectations. Could you elaborate on why you attribute these unexpected results to benchmark unreliability rather than potential issues with the larger models themselves? Did you investigate alternative explanations for the performance discrepancies?
2. Do you have any results from human evaluation that validates the model-based evaluation metrics? What were the human-model agreement rates? Were there any notable discrepancies between the human judgments and model-based evaluations?
3. Other than RAG, which types of tasks in HELMET are compatible with the base model without instruction following capabilities?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677758683

thread_type:
official_review

<thread_object>
creator: reviewer_1mdc

<content>
summary:
The paper introduces a new benchmark called HELMET, which is designed to comprehensively evaluate the performance of long-context language models (LCLMs). Current LCLM evaluations largely rely on synthetic tasks, like Needle-in-a-Haystack (NIAH), or arbitrary subsets of some datasets. However, these methods present issues such as high noise, insufficient coverage of downstream applications, inadequate dataset lengths, and unreliable metrics. HELMET aims to address these shortcomings by expanding task diversity across seven application-centric categories (including long-document QA, citation-based generation, etc.), supporting controllable input lengths up to 128k tokens, and implementing model-based evaluations for more reliable results. Through testing 51 LCLMs, this study finds that synthetic tasks are poor predictors of downstream performance, open-source models fall behind closed-source models on complex long-context tasks, and there is low correlation among task categories, highlighting the need for multi-dimensional LCLM evaluation .

strengths:
1.	**Diverse Task Design**: HELMET includes seven categories of tasks, enhancing the representativeness of LCLMs in real applications.

2.	**Support for Ultra-Long Inputs**: This benchmark accommodates input lengths over 128k tokens, making it suitable for evaluating the long-context capabilities of frontier models.

3.	**Reliable Model-Based Evaluation**: HELMET’s evaluation metrics reflect human judgment better than traditional n-gram matching, offering more reliable model ranking.

4.	**Compatibility with Base Models**: The benchmark allows evaluations of base models that haven’t undergone instruction fine-tuning, broadening LCLM applicability.

weaknesses:
1.	**High Complexity**: With multiple tasks and model comparisons involved, HELMET’s setup and evaluation process is intricate and demands considerable effort from researchers.

2.	**Low Correlation Among Some Tasks**: The low correlation between different tasks may make it challenging to assess a model’s overall long-context handling ability if it performs exceptionally in only certain tasks.

1. **High Resource Consumption**: Running the full suite of HELMET tasks is time-intensive. It would be beneficial to identify a few key subtasks that can maintain consistency with the results of full testing, allowing for time-saving evaluations.

questions:
Please address the weaknesses in the previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707451796

thread_type:
official_review

<thread_object>
creator: reviewer_rmnk

<content>
summary:
This paper constructs a comprehensive benchmark to test LLMs' long context abilities. It covers various types of tasks such as RAG, ICL, LongQA, Retrieval, Re-rank and so on.  The used prompts and evaluation metrics and carefully designed to ensure both IFT models and base models can give predictions. This benchmark also evaluates most commonly recognized LLMs and accordingly provides insights about LLMs' long context performance.

strengths:
1: The benchmark is comprehensive. It covers most real-world long context use cases.

2: The investigation of performance correlation among all task types are insightful. It provides a new perspective to understand LLMs' long context ability.

3: The improvement to prompting strategy and evaluation method effectively stabilizes the evaluation results.

weaknesses:
1: The so called "expected" ranking of LLMs is a bit subjective. 

2: Lack of some deep analysis to interesting results, such as why the json-kv task has higher correlation with re-rank than RAG or LongQA

3: The RoPE scaling settings are not suitable for 128k/64k testing. With ABF, usually, the scaling factor should be at least 2x the target extension ratio. With 8k context, Llama3 should use at least a scaling factor of 32 for 128k testing.

questions:
1: Figure 2 is missing? 

2: What is the value for 'depth' in Figure 11? From top to the bottom, is the key information located at the beginning of the context to the tail of the context? 

3: Gemma series have a unique attention head dimension of 256 rather than 128. It might have interesting impact on the long context things. It would be better to have results with Gemma series as the tested models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731960149393

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response Rebuttal 1/2

comment:
We thank all reviewers for their helpful feedback. We are encouraged that all reviewers found HELMET to be a comprehensive and practical benchmark and our analyses to be insightful. A common concern is addressed below: 

> the “expected” ranking of models can be subjective and why we attribute the unexpected results to benchmark unreliability rather than issues with the models. what are alternative explanations for the performance discrepancies? more direct comparisons of model rankings or performance differences on HELMET and existing benchmarks

We attribute the discrepancy in model rankings across benchmarks to the following reasons:

1. **Previous work finds that larger models perform better than smaller models at long context.** The Gemini 1.5 report observes a trend of larger models performing better at long contexts—the larger models beating the smaller models both qualitatively and quantitatively across the Gemini, GPT, and Claude model families (Section 5.2, Tables 4 and 5) [4]. Although the evaluation suite is not publicly available, the results align with our intuition, which is commonly believed in other use cases of language models. Since we do not have access to a ground truth ranking of models and acquiring one via human evaluation is nearly impossible, relying on such a commonly-accepted intuition is reasonable.

2. **$\infty$Bench still relies on n-gram matching metrics for QA and summarization evaluation, which have been shown to be unreliable and noisy** [1][2][3]. 
We discuss this in-depth in Section 2.2. In Figure 2, we observe that ROUGE-L scores cannot effectively distinguish different models at different input lengths, but our model-based evaluation identifies improvements and degeneration with increasing input lengths and distinguishes models of different capabilities. 
From additional qualitative analysis, we found that model-based evaluations are better at penalizing decoding degenerations as well. For instance, on $\infty$Bench Sum, a model generated a summary with the sentence “The author’s object is to show that the study of grammar is necessary part of education” repeated hundreds of times. This summary receives a ROUGE F1 score of 12.3 while our model-based metric identifies it as incoherent and assigns it a score of 0. We will offer more qualitative analysis in the revision.

3. **HELMET improves prompting strategies, such as using in-context learning examples.**
In Table 7, we show that using better instruction and in-context learning results in better performance for many models, which is a more accurate representation of how these models would be used in practice. This is especially important for smaller models, which degenerate without the in-context learning examples. Ablations of the improvement from ICL examples are shown in Table 8.

4. **Previous benchmarks lack diverse and challenging applications.** In benchmarks like RULER, many models already achieve near-perfect scores on certain synthetic tasks, which cannot provide meaningful signals and exhibit low correlations with real-world applications (Section 3.1, Figures 3 and 4). On the other hand, HELMET better distinguishes frontier models by introducing diverse and complex tasks such as generation with citations (Section 3.3).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents HELMET, a comprehensive benchmark for evaluating long-context language models (LCLMs) across seven application categories, with controllable lengths up to 128k tokens. The key claims include that synthetic tasks like needle-in-a-haystack (NIAH) are poor predictors of downstream performance, different task categories exhibit distinct uncorrelated trends, and open-source models lag significantly behind closed models on complex reasoning tasks. The paper's strengths include: providing a holistic evaluation framework encompassing diverse real-world tasks, demonstrating issues with current benchmarks through rigorous analysis, and offering practical insights through extensive evaluation of 51 models. While initial concerns were raised about the "expected" ranking of models being subjective and the heavy reliance on closed models for evaluation, the authors provided strong evidence and justification during rebuttal. They conducted additional experiments with different RoPE settings on Llama models, added comprehensive human validation of their model-based metrics showing high agreement (kappa > 0.8), and demonstrated that only 3 of 21 datasets rely on closed models for evaluation. For these reasons I vote to accept the paper!