Title: Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

attention mechanism

long context inference

prompt compression
Abstract: Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computational costs and reduced performance. To address this challenge, we propose an efficient, training-free prompt compression method that retains key information within compressed prompts. We identify specific attention heads in transformer-based LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long inputs that are most significant for inference. Building on this discovery, we develop EHPC, an Evaluator Head-based Prompt Compression method, which enables LLMs to rapidly "skim through'' input prompts by leveraging only the first few layers with evaluator heads during the pre-filling stage, subsequently passing only the important tokens to the model for inference. EHPC achieves state-of-the-art results across two mainstream benchmarks: prompt compression and long-context inference acceleration. Consequently, it effectively improves performance with the reduced costs associated with commercial API calls compared to prompt compressing methods. We further demonstrate that EHPC attains competitive results compared to key-value cache-based acceleration methods, thereby highlighting its potential to enhance the efficiency of LLMs for long-context tasks.


Peer reviews:
comments: SUMMARY:
This paper introduced a fast, training-free prompt compression method EHPC (Evaluator Head-based Prompt Compression) for prompt compression and long-context inference acceleration. EHPC is built upon the pre-filling stage of LLMs, which reduces the number of tokens in both the pre-filling and decoding stages and the KV cache size, which accelerates the inference. EHPC identifies a subset of attention heads in transformer-based LLMs (i.e., evaluator heads) which can select tokens in long inputs that are most significant for inference. This method allows LLMs to efficiently "skim" prompts by utilizing only the initial layers containing evaluator heads during the pre-filling stage, retaining only the crucial tokens for downstream inference. EHPC achieves state-of-the-art performance on both prompt compression and long-context inference acceleration benchmarks.

STRENGTHS AND WEAKNESSES:
* Strengths
    * The paper is well-written and easy to follow
    * The proposed method applies in the pre-filling stage, which compresses the input prompt before inference
    * The proposed model is evaluated on various benchmarks and LLMs, and compared with SOTA methods across multiple metrics, including latency. Results show competitive or superior performance compared with SOTA methods.

* Weaknesses
    * the empirical evidence is strong, the paper lacks deeper theoretical analysis on the evaluator heads.

QUESTIONS:
EHPC selects heads from the highest scoring layer, would combining heads from multiple layers helps?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents EHPC (Evaluator Head Based Prompt Compression), a novel approach for LLM prompt compression by preserving tokens only the tokens that receive high attention from “evaluator heads” - a set of specific attention heads, usually found in the earlier model layers, that are crucial for focusing on important information in long context tasks. The method employs a two-pass prefill approach for compression. The first pre-fill pass runs a few initial layers of the LLM (containing the evaluator heads) to compute the importance of each prompt token. This is followed by a second prefill pass over all the layers of the model, but with a subset of the prompt tokens. Extending on previous research, the authors show the existence of evaluator heads across models and their generalizability to downstream tasks (when identified using synthetic data). The authors evaluate EHPC in two modes: (1) EMI: using a separate model for prompt compression than the target inference model, (2) NMI: using the same locally deployed model for compression and target inference. Their results show that EHPC outperforms existing prompt compression methods (in both accuracy and system performance) in the EMI mode and is competitive with KV-Cache compression methods in NMI mode.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n\n- The results for EHPC in EMI mode are impressive, as it consistently outperforms existing prompt compression methods. Specifically, EHPC is far more efficient with its two-pass approach for inference than other methods, as shown in Table 5\n- The authors evaluate their method against strong and plentiful baselines on a variety of tasks for both EMI and NMI evaluations\n- The method is original and definitely pushes the field of prompt compression further\n\n**Weaknesses**\n\n- The analysis around generalizability and robustness of evaluator heads is limited and not convincing enough:\n    - I would implore the authors to evaluate on more models covering different sizes and architectures, like 30B or 70B models, if feasible, and architectures like MoE models\n    - The generalizability study in Section 4 does not provide details on how the downstream performance is calculated on the given set of tasks with a subset of the heads. Additionally, all the subsets of tasks shown are from LongBench. It would be nice to see some tasks from RULER as well\n    - The robustness study does not tell me whether the actual heads identified using different tasks are the same or different. Even if downstream performance does not change, the actual heads identified might be different. This is a crucial distinction between saying the evaluator heads are task agnostic versus evaluator heads identified using task X have a good overlap with evaluator heads of task Y.\n- While the downstream results in the EMI mode are impressive, the NMI mode evaluation shows that EHPC might not be the best approach to use when compared to KV-Cache compression methods like SnapKV. While the authors acknowledge this, a discussion on whether this is an inherent issue with prompt compression methods or ways to improve this performance in future work would be helpful.\n- EHPC does not apply to decode heavy workloads\n\nQUESTIONS:\n- I understand that the purpose of evaluator and retrieval heads is different, but is there an overlap between them - i.e. are most evaluator heads the same as retrieval heads?\n- When identifying evaluator heads, what is the rationale behind extracting only the last row of the attention matrix?\n- In the robustness study, can you confirm if the actual evaluator head indices identified over different tasks are the same or have significant overlap?\n- Can you provide details on how the downstream performance was measured in the generalizability study with a limited number of attention heads?\n- I think the authors should use LongBench v2, which is an updated version of the LongBench dataset with much larger context lengths to show the effectiveness of their method further.\n- A small discussion on whether and how the NMI performance can be improved would be helpful.\n- In Table 5, the inference time (second column) is different for different prompt compression methods. Why is that the case?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EHPC (Evaluator Head-based Prompt Compression), a novel training-free method for compressing long input prompts.\nEHPC leverages specific attention heads in transformer-based large language models (LLMs), termed evaluator heads, that identify and retain the most important tokens for inference.\nEHPC accelerates long-context processing by skimming input prompts during the pre-filling stage, passing only critical tokens to subsequent layers.\nThe authors analyze the complexity of EHPC and investigate the existence, generalizability and robustness of EHPC.\nExperimental results show that EHPC achieves state-of-the-art performance in both prompt compression and long-context inference acceleration while reducing computational costs and API call expenses.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. The identification of evaluator heads as key components for token selection is a compelling contribution.\n2. Unlike methods requiring fine-tuning or auxiliary models, EHPC works out-of-the-box, making it easily applicable to existing LLMs without additional overhead.\n3. The paper demonstrates SOTA performance on standard benchmarks, showcasing EHPC’s effectiveness in both compression quality and inference acceleration.\n\n**Weakness**\n1. The methodology part is somewhat confusing. For instance, when detecting evaluator heads in Section 3.4, the authors use only the last row of the attention matrix, but all the rows are included in the calculation of utility scores (Equation 1). Moreover, some denotations have mistakes. For instance, in line 157, $A^{hl}\\in\\mathbb{R}^{N\\times N}$, not $A^{hl}\\in\\mathbb{R}^{H\\times L}$.\n2. The differences between EHPC and existing attention-based compression methods (e.g., SnapKV) are not clearly discussed. Therefore, the novelty of the method is not very clear.\n\nQUESTIONS:\n1. Does NMI perform better than EMI when the computation budget is controlled the same?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper proposes Efficient Prompt Compression (EPC), a method to reduce the cost and latency of large language model (LLM) inference when dealing with long prompts. EPC trains a small prompt compressor network to transform long input prompts into compact representations. Experiments on multiple benchmarks (QA, summarization, reasoning) show reduced latency and token usage with minimal accuracy drop.

STRENGTHS AND WEAKNESSES:
Strengths
- Addresses the real and growing cost problem in LLM applications that use long context prompts.
- Demonstrates large token savings and latency reduction without heavy retraining of the base LLM.
- Applicable across tasks (QA, summarization, reasoning) without task-specific engineering.

Weaknesses
While EPC is compared with some baselines, the evaluation omits certain recent context compression techniques like retrieval-augmented methods or structured summarization models.
In high-compression regimes, some performance drop is visible, but the paper doesn’t deeply analyze what kinds of information get lost.
The cost and data requirements for training the compressor are not sufficiently discussed.

QUESTIONS:
How well the compressor works on tasks or domains different from the training set.
How sensitive is EPC performance to the size of the compressor model?
Results show a promising trade-off between compression ratio and accuracy, but more variety in datasets and ablations on compressor architecture would strengthen the claims.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01