Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
Authors: Weizhi Fei
Xueyan Niu
XIE GUOQING
Yingqing Liu
Bo Bai
Wei Han
Venue: NeurIPS 2025 spotlight ()
Keywords: large language model
attention mechanism
long context inference
prompt compression
Primary area: deep_learning
TL;DR: We propose an efficient, training-free prompt compression method that retains key information within long inputs using the evaluator heads we identified in transformer-based LLMs.

Abstract: Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computational costs and reduced performance. To address this challenge, we propose an efficient, training-free prompt compression method that retains key information within compressed prompts. We identify specific attention heads in transformer-based LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long inputs that are most significant for inference. Building on this discovery, we develop EHPC, an Evaluator Head-based Prompt Compression method, which enables LLMs to rapidly "skim through'' input prompts by leveraging only the first few layers with evaluator heads during the pre-filling stage, subsequently passing only the important tokens to the model for inference. EHPC achieves state-of-the-art results across two mainstream benchmarks: prompt compression and long-context inference acceleration. Consequently, it effectively improves performance with the reduced costs associated with commercial API calls compared to prompt compressing methods. We further demonstrate that EHPC attains competitive results compared to key-value cache-based acceleration methods, thereby highlighting its potential to enhance the efficiency of LLMs for long-context tasks.

Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
Weizhi Fei!
Xueyan 2
Guoqing Xie Yingging Liu
Bo Bai?
Wei Han
Department of Mathematical Sciences, Tsinghua University, Beijing, China
Theory Lab, 2012 Labs, Huawei Technologies Co., Ltd.
Architecture & Design, ICT Products & Solutions, Huawei Technologies Co., Ltd.
Abstract
Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computa- tional costs and reduced performance. To address this challenge, we propose an efficient, training-free prompt compression method that retains key information within compressed prompts. We identify specific attention heads in transformer- based LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long inputs that are most significant for inference. Building this dis- covery, we develop EHPC, an Evaluator Head-based Prompt Compression method, which enables LLMs to rapidly "skim through" input prompts by leveraging only the first few layers with evaluator heads during the pre-filling stage, subsequently passing only the important tokens to the model for inference. EHPC achieves state- of-the-art results on two major benchmarks: prompt compression and long-context inference acceleration. Consequently, it effectively improves performance with the reduced costs associated with commercial API calls. We further demonstrate that EHPC attains competitive results compared to key-value cache-based acceleration methods, thereby highlighting its potential to enhance the efficiency of LLMs for long-context tasks.
Introduction
Large language models (LLMs) have exhibited exceptional capabilities in variety of real-world tasks and applications, with an increasing need for processing long inputs in areas such as literary novels, legal documents, instruction manuals, and code documentation. Inference tasks that require understanding of long contexts, such as long document summarization Zhang et al. 2024a), reasoning Fei al. [0244]. and autonomous agents (Singh al. [024], Chen et al. [2024] of particular importance due to the high stakes in these scenarios. However, the deployment of LLMss is challenged
by the computational and memory demands inherent to cansformer-basee architectures, resulting in increased latency, particularly when processing lengthy input prompts.
Prompt compression, which involves substituting the input prompts provided language model with more succinct versions, has surfaced as a promising strategy for enhancing long-tex, understanding and mitigating associated costs. Current mainstream methods, such SelectContext Li al. [203). LLMLingua [Jiang et al.] 2023a and LongLLMMinguss [Jiang et al.] 20236), typically rely on pre-trained L.Mss, utilizing the logits or perplexity of the prompts evict tokens deemed insignificant. These approaches often necessitate chunking long texts for processing, leading to numerous repeated calls of the L.MM and consequently incurring considerable time complexity. More efficient compression techniques, such as LLMLingua-2 Pan et al. [2024]] generally require
Cooresponneecce to: niuxueyan3ehuawei..cmm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the training of secondary, smaller model on labeled datasets. Although these methods reduce compression time, they also incur substantial training expenses and may exhibit performance drops in out-of-distribution contexts compared to direct utilization of the LLM as compressor. In this paper, we present an Evaluator Head-based Prompt Compression method, dubbed EHPC, which is built upon the efficient pre-filling stage of LLMs. Our method leverages the intrinsic attention mechanisms of the LLM, thus being training- and computationally efficient, achieving SoTA performance across mainstream benchmarks (see Table
The attention mechanism is pivotal in transformer-basse LLMs, aggregating in- formation from input prompts via atten- tion scores. In our EHPC method, we uti- lize high attention scores to identify and retain significant tokens. This approach feasible primarily because the attention scores of tokens in long texts are sparse, explained by the widely studied "attention sink" phenomenon| Xiao et al. [2033] Gu jet al. [024]. This phenomenon is charac- terized by the LLMs' frequently assigning
Table I: Overall comparison of the proposed method in terms of average performance and latency on the Long- Bench dataset, under the constraint of a compressed prompt length of 2048 tokens. For comprehensive re- sults, please see Table 4 and Table
Method
Performance
Latency
Training- free
LongLLMLnngua
48.0
67.41
LLMLingua LLMLingua-2
34.6
7.51
39.1
1.27
EHPC (ours)
49.6
0.88
high attention weights to the semantically inconsequential initial token, <B0S>, Furthermore, much of
the research Zhang al. [2023] [i et al. [2024), Ge et al. [2024a], , Cai et al. [2024). ,, |W et al. [024]]
Tang et al.] (2024). Xiao et al. [2024], Fu et al. [2024] has focused on compressing the key-value (KV)
cache by eliminating entries with low attention weights. Furthermore, EHPC can be efficient because
we can leverage the KV cache obtained during the highly parallelizable pre-filling stage and quickly
compute the required part attention scores [Jiang al., 2024].
Unlike our prompt compression method, which retains important tokens based on the scores of evaluator heads, most KV cache compression methods preserve the cache according to all heads. Recent research (Wu et al., 2024 Tang et al.; 2024 Xiao et al.) 2024] has identified that certain layers and attention heads play subset of these attention heads, which we designate as evaluator heads, allowing LLMs to focus on essential information for inference from any position within the input sequence. To identify these evaluator heads, we designed a pilot experiment using synthetic data (see Figure I].
significant role in processing long contexts. We have pinpointed
We then carried out extensive experiments to demonstrate the robustness of the evalua- tor heads and their applicability to practical scenarios using real-world datasets. Sub- sequently, applied the evaluator heads to develop a prompt compression approach EHPC, using the scores given by these heads to select tokens for inference. The pro- posed EHPC requires the local development of LLMS for prompt compression and of- fers two application settings: Extended Model Inference (EMI) and Native Model Inference (NMI). We demonstrate its ef- fectiveness through two important bench- marks: prompt compression and long text accelcration. When prompts compressed using EHPC are employed in commercial models, EHPC effectively reduces API costs while enhancing the performance of API outputs. Compared to existing prompt compression methods [Li et al. 2023 Jiang et al. 2023b a].
Instrctiion Moros ancee de goodie laodic soand of Dellert Quidina Milenie becthing didr Pandooo?
"Nocc''' shouid heal
Viyystrii canto dicarddd
Figure 1: Visualization of attention scores from single attention head during inference on the "Needle-in-a- Haystack" long-text benchmark. This benchmark re- quires the LLM to follow instructions and retrieve "nee- dles" specific pieces of information randomly inserted into long text - to answer a given question. The evalu- ator heads are identified as those that accurately locate the relevant facts, thereby achieving high scores.
our approach achieves new state-of-the-art (SoTA) performance and is more efficient, requiring less compression time. Moreover, when applied to native models deployed locally, prompts compressed with EHPC accelerate long-text inference by reducing memory usage and achieving competitive results
compared to SoTA KV cache compression methods [Li ct al. 2024 Zhang et al. 2023

---

Our contributions are as follows. (1) We identify specific attention heads in transformer- -bsse LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long imputs that are significant for inference. (2) We propose EHPC, an efficient prompt compression technique that enables L.LMs to quickly "skim through" input prompts by utilizing only the first few layers with the evaluator heads, and then pass only the important tokens to the model for inference. (3) We demonstrate that EHPC has lower complexity compared to the prior prompt compression method and achieves new SoTA on the prompt compression benchmarks over LongBench and ZeroScroll, effectively reducing the API cost and memory usage commercial LLMs. (4) We empirically demonstrate that EHPC is capable of accelerating long-context understanding, achieving competitive performance relative to KV cache compression methods. Notably, EHPC improves upon direct
inference by up 40% on the question question-answerigg answering datasets.
2 Related work
Two predominant approaches are utilized to accelerate LLMs: implicit methods that reduce the KV cache and explicit methods that decrease the number of tokens. We briefly review these methods with focus on explicit methods, which can be applied to black-box LLMs such GPT-3.5-Turbo.
2.1 Implicit token reduction
The key-value (KV) cache reduces redundant calculations and enhances decoding efficiency by storing key and value matrices from previous tokens (Liu al. 2024a AAdnan et al.) 2024). However, as the input length increases, the memory requirements of the KV cache increase, creating significant challenges for long-contexs processing. was found that a small number of tokens account for the majority of the value during the computation of the attention scores, leading to the proposal of the H2O (Zhang et al. 2023) that retains only the KV cache for "heavy hitters", which are tokens with high attention scores. FastGen [Ge et al.] [024a]] introduces a dual-phase adaptive KV compression strategy that includes four KV cache compression policies and dynamically evicts caches during generation based on optimal policies identified through profiling, SnapKV Li et al. (2024) shows that specific patterns of attention can be identified by an observation window at the end of the prompts, and compresses KV caches by selecting clustered attention scores through pooling operations. Wu eet al. (2024) experimentally investigate how transformer- based models retrieve relevant information from arbitrary locations within long contexts, identifying certain heads, termed retrieval heads, as crucial in this process. Subsequently, building upon the concept of retrieval heads, several desd KV cache compression methods have been proposed [Tang al., 2024], Xiao et al.) 2024). These methods specifically preserve the KV cache of the retrieval heads to maintain their functionality.
2.2 Explicit prompt compression
Semantic compression Wingate et al. 2022] use soft prompts to condense the context, ensuring that the compressed prompts retain a sigmificant amount of information. [Cevaalier et al. [2023) introduce AutoCompressor, which adapts L.MMs to compress lengthy contexts into concise summary vectors.
Similarly, research Bulatov et al. [022], Mu et al. (2023). Ge et al.) [2024b]. Monteiro et al. [2024]
has focused on learning various types of memory, such as gist tokens and cross-context caching, to compress context through prefix-tuning [Li and Liang 2021 Fei et al. (2024b) implement summarization model semantically compress input text through a divvideaand-ooquee strategy.
Token deletion widely adopted approach among explicit methods is the direct removal of tokens
Tha et al. [2024], [Shi et al. [2024). Selective-Conter (Li et al., 20033 uses the logits of the model to calculate the mutual information of the tokens, subsequently removing tokens based on it. LLM- Lingua Jiang et al. (2023a) initially computes the perplexity of each token and then integrates budget controller with a coarse-to-fine fine, token-level iterative compression algorithm. Expanding on LLMLingua, LMLgnuM (Jiang et al.) [223bb] introduces the concept of conditional perplexity to intensify the focus on key information according to task-specific instructions, which achieves great improvement over long text situations. LLMMLinuua22 Pan et al. (0244) represents fast prompt compression method, since it uses a small classification model to predict the significance of each token in the prompts. This classification model takes prompt compression as a token classification task and is trained to utilize compact transformer-basee encoder labeled dataset.
3

---

3 Methodology
We present our prompt compression method, EHPC, which is characterized by the identification and utilization of evaluator heads. We find that in LLMs, certain attention heads, which we designate as evaluator heads, can be utilized to rapidly determine which tokens can be omitted during the pre-filling stage. Background information mechanism LLMs, with a focus on the enhanced efficiency of the pre-filling stage through the application of KV cache, is provided in Appendix C
the basic implementation of the multi- head attention
3.1 Prompt compression strategy
We represent the imput prompt sequence of tokens. z (I1, #2,.......... .N). where N |=] denotes the length of the sequence. The objective of prompt compression is identify shorter sequence è to replace the original sequence for language models. Analogously to the way human readers often skip words during speed reading. EHPC employs a token deletion strategy as in [Li et al.) 2023 Jiang et al., 2023a] and compresses the prompt by dropping non-essential tokens directly. In contrast to generating new context through summarization [Fei et al.] [2024b]. the token deletion strategy effectively simplifies the problem by narrowing the search space. Additionally, the token- level deletion strategy can be seamlessly integrated at the sentence/paragraph level [Liskavets et al.
2024
3.2 Prompt compression using evaluator heads
Our prompt compression strategy selects salient tokens based on their averaged attention scores on the identified evaluator heads. Given a transformer-based language model f and the evaluator heads C identified through the pilot experiment delineated in Section 3.4 Let A E R N denote the matrix of attention scores for the layer and the attention head using f. For long input prompt æ, we utilize the attention scores from Cy to compute the utility scores E R for the input during the pre-filling stage according
A [ No
Σ Pool Σ (,,,,))CC,
s =
N. N,SIEN
where Pool(..-) denotes pooling operation, and represents the kernel size, No is the observed window length and N, is the length remaining part such that N -N, + .N. Subsequently, we employ the scores to remove non-essential tokens, constructing from the retained tokens in their original order. Although the compressed prompt retains its natural language form, it may lack certain contextual elements. To mitigate this, adopt pooling operation, described in [Li et al. 2024]. to group neighboring tokens with similar scores, thus generating a continuous sequence to enhance readability.
While our prompt compression strategy necessitates the processing of prompts by an LLM, it leverages the computationally efficient pre-filling stage, enabling fast compression. The compressed prompts obtained are transferable and can also be applied to black-box models based on API. Our EHPO method gives rise to two scenarios: Extended Model Inference (EMI) and Native Model Inference (NMI). EMI involves using a different language model f' to infer on the compressed prompt. For instance, when applied to an API-based commercial model, this approach can reduce API latency and costs, as the API cost is linearly related to the input prompt length. NMI, in contrast, utilizes the same model ff to conduct inference. In this case, method can reduce computational memory usage and costs, akin to the KV cache compression method. In practice, the language model f used for prompt compression should exhibit robust long-context capabilities, and smaller models are preferred for efficient deployments, such as Llama-2 3.1 1-B,, with a context length of 128k.
3.3 Evaluator heads
The evaluator heads retain important tokens according to the attention scores. As detailed Ap- pendix C each attention head performs a weighted average over the preceding tokens, wherein the tokens assigned lower attention weights contribute less the information processed by the attention heads. Empirical studies have shown that the contribution of attention heads to the capacity to handle

---

long contexts in LLMs is not equally important Wu et al. [2024] Tang et al. (2024). Specifically, certain retrieval heads are essential and must be maintained during KV cache compression, as their removal would significantly impair the L.MSs ability to manage long contexts. We find that special- ized heads, which we designate as evaluator heads, play a pivotal role in assessing the significance of long input prompts, and these evaluator heads alone are sufficient for evaluating tokens.
We conducted experiments based on synthetic data with known evidence to explore and verify the existence of the evaluator heads and to ensure that they can effectively identify crucial information in input prompts. For a transformer model f with layers and H heads in each layer, we define the evaluator heads as the set C ((41/11) ... (mm Bm]], where 1 < 1 S 1 S L and 1 S h4 < hJ S H for i S j. These specialized heads are identified, and their attention scores are used to give the final score that represents the utility of each token within the input prompts. We empirically investigate the properties of the evaluator heads, including the existence, generalizability, and robustness, in Section 4
The primary distinction between the evaluator heads we have defined and the retrieval heads discussed in Section 2.e in their respective functions: the evaluator heads are designed to assess the significance of tokens within input prompts, while the retrieval heads are intended to maintain the essential KV cache. Although both types of heads aim to identify the most salient components among all attention heads through a data-driven approach, evaluator heads are deemed sufficient for their purpose, whereas retrieval heads are necessary for preserving the integrity of the KV cache. Furthermore, evaluator heads operate explicitly on tokens, in contrast to retrieval heads, which function implicitly on the KV cache.
3.4 Pilot experiments for detecting evaluator heads
We design pilot experiment identify the evaluator heads. By entering a long prompt containing known evidence, we computed the corresponding accumulated scores for evidence to pinpoint the evalu- ator heads, where evidence is part of the input prompts that determine the model output. Let the relevant evidence e be denoted as the sub- sequence e (2),)) of x. where
codellama-7b-hf
Ilama-3. 1-8b-instruct
phi-3 m-ins---ii--
Head
Read
Hlaad
Figure 2: Heatmap of evidence scores for three different LLMs in the pilot experiment, illustrating scores across layers and heads, with heads re-ranked in descending order for clarity.
Ie C [N] indicates the indices of the evidence. We extract the last row of the attention matrix, al = A [N: E R-,,, to represent the scores for the importance of each token, as these scores directly influence the computation of the final hidden state. To assess whether the heads are focusing on relevant information, we compute the accumulated score of the evidence as
hl
Σ a Li. JEL
As a practical example, we used the "Needle-in-a- established long-context retrieval benchmark, to demonstrate our pilot experiments (as illustrated in Figure [). Let i be the synthesized long context and let e represent the "needle" (evidence) inserted specific position for f to identify. We record the accumulated score of the evidence àd for each head. We then averaged the accumulated scores of the evidence to generate an evidence score matrix S E Hx which we used identify the evaluator heads. Visualizations of the evidence score matrices are presented in Figure 2 for several examples of open-source LLMs. Finally, we selected the layer with the highest score and identified the top-k heads from this layer as the evaluator heads, i.e,
benchmark Kamradt [2024), a well-
C arg max AAl
||AAS|E
s.t. (1. A max (S 1xx1),, Vj, ISiSL
where ||. || is the Frobenius norm, and eΛ = (Eiy) is the incidence matrix such that eij = (i,5) & A and ety- = if (i. e A.
5

---

3.5 Complexity
We only discuss the complexity NMI, as EMI follows a similar line of reasoning. Consider an LLM I with L layers, H attention heads per layer, and hidden dimension d = d2 H. Suppose that the model / given an input prompt of N tokens and generates new tokens. the pre-filling stage, each head computes Softmax( QK") with Q, K E RRNx,s, resulting in a complexity of N²).
ydz
Thus, the total complexity for the pre-filling stage is N2) During the decoding stage, the model generates tokens based on the pre-filled K, V E RNxd, and the total complexity is d(LHd, (tN + )), with details provided in the appendix B
In the NMI setting, where is used for both com- pression and inference, our method involves two pre-filling stages and one decoding stage, illus- trated in Figure 3 Let #1 = L/ maxisien 1x11)-- The first pre-filling stage utilizes only L/K1 layers to compress the prompts, result- ing in the complexity of O(HHd,N²/n)) Sup- pose the compression rate is K2, then the sec- ond pre-filling stage processes only N/ng tokens, and the complexity is O O(= Thus, the
(A) inforence
pre-fil
decods
Prompe Oomprexiion
(b) Proposed mechod
LLM inference
T7F7
time
total complexity of combined pre-filling stages is O(LHd, I + ) Therefore, when 61, K2 2, which is often the case, the pre-filling with our compressing method has lower complex-
Figure 3: Illustration of the proposed method. (a) LLM inference comprises two stages: the pre- filling stage and the decoding stage. (b) The pro- posed prompt compression approach leverages the efficiency of the pre-filling stage, thereby re- ducing inference latency for both stages during inference with compressed context. the original complexity since Kg 2 1.
ity since (+- + R2 S 3.J The decoding stage
involves only N/K2 tokens, and the complexity is O(LHd((2 + (N,)), which is naturally lower than
4 Properties of the evaluator heads
We conducted a comprehensive investigation to address the following research questions:
(RQ1) Existence:
Do evaluator heads exist significantly across LLMs? Can these heads be reliably identified through pilot experiment utilizing simple synthetic data?
(RQ2) Generalizability: Can the evaluator heads identified through synthetic data be effectively applied to various tasks in real-world long-text benchmarks?
(RQ3) Robustness: Are evaluator heads task-oriented, i.e., do these heads induced by different tasks demonstrate robustness and consistency?
To investigate (RQ1), (RQ2), and (RQ3), we conducted pilot experiments to identify the evalua- tion heads using a synthetic benchmark in the seedle-in-a-Haystaee style. Then, we evaluate these evaluator heads using real-world benchmarks, including LongBench [Bai et al.) [024] and
BoBench (Zhang et al., 2024b). Details on the synthetic and real-world benchmarks used in our study
provided in Appendix H
Existence To illustrate the existence of these heads, we conducted pilot experiments as detailed in Section 3.4 utilizing the widely employed "Needle-in-a-HHystae
benchmark. We provide the results over three models in Figure?) 2 We evaluated each head's ability to identify key information by examining the accumulated attention scores over the evidence sequence. The distribution of these evaluator heads is sparse and predominantly concentrated in some certain layers. This provides empirical support to find evaluator heads in one early and significant layer.
Generalizability Next, we assess the generalizability of the evaluator heads identified in our pilot experiments by evaluating their performance on diverse downstream tasks. The results over two models are presented Table 2 We selected heads with the highest scores as evaluator heads from the layer with the highest cumulative score. We also randomly selected 8 heads and 50% heads

---

Table 2: Performance comparison of two sets of heads on three datasets from different tasks in the LongBench.
Model
Method
NarQA
Musique
LCC.
QMSum
MathFind
CodeDebug
Random (8 heads)
16.6
10.4
45.2
20.3
13.7
14.2
Llama 3.1
Random (50% heads)
19.6
13.2
47.7
23.7
14.9
14.7
Ours (8 heads)
23.0
14.0
48.9
24.2
27.7
24.7
Random (8 heads)
12.2
12.0
28.9
18.1
26.0
9.8
Phi
Random (50% heads)
24.4
11.7
40.6
22.3
21.4
16.1
Ours (8 heads)
23.9
20.3
41.6
22.2
34.0
16.2
for comparison. It's observed that selected heads have significant improvement for randomly selected heads. Though random 50% achieve comparable performance (still worse than ours), they require significantly more time to evaluate tokens. The findings highlight the practical utility of the evaluator heads and demonstrate the effectiveness of our detection experiments.
Robustness Lastly. we empirically illustrate that detecting the evaluator heads through pilot exper- iment task-agnostic. In addition to the simple QA data from fheedle-in-a-Haystte probe data from the following tasks: code completion (code) and multi-hop variable tracking (reasoning). Details on the construction of the probe data are pro- vided in Appendix ! We then compare the heads from the simple QA data with the task-aware heads corresponding to their respective reasoning and coding tasks from the LongBench dataset. The results are presented in Table The results averaged over the complete data. "Evaluator heads (QA)" refers to the heads identified using QA data from the pilot experiment, while "Task-aware heads" refers to heads identified from generated data customized to the respective tasks. Our findings indicate that the heads identified using customized tasks do not have improvement on their respective downstream tasks. This shows that the evaluator heads are task aggostgn and that the heads identified from the simple QA data exhibit robustness in downstream tasks.
Table 3: Evaluation of task-aware approach multi-hop reasoning and code completion from corresponding LongBench datasets.
we also generate
Multi hopp
Code
Evaluator heads (QA)
16.40
47.86
Task-aware heads
16.87
47.72
Experimental results
Having established the existence, generalizability, and robustness of the evaluator heads, we now experimentally assess the effectiveness and efficiency of EHPC tasks aimed at reducing API costs of commercial models and accelerating long-context inference in LLMs. We also compare the efficiency of our prompt compression method with other acceleration methods under the same memory usage.
5.1 Prompt compression benchmark
We evaluated the prompt compression benchmark on LongBench and ZeroSCROLLS, as established by Jiang et al. (0033b), with the objective of improving the quality of compressed prompts for commercial models. Prompt compression is essential for mitigating costs associated with commercial LLMs, given that API fees are often proportional to prompt length. We compare our EMI setting, which employs a local LLM for prompt compression and another model for inferring the compressed prompts, against the following baselines.
Baselines and implementation details We compare our method with Retrieval Augmented Gener- ation (RAG) and other SoTA prompt compression methods. The retrieval models considered include BM25, Sent [Reimers and Gurevych] 2019], and OpenAl Embedding. For prompt com- pression methods, we evaluated against Selective- Context [Li al. 2023]. LLMLingua (Jaag et al.) 2023a],
[Jiang et al. 2023b), and LLMLingua-2 Pan et al. 2024] as baselines. Detailed descriptions of these prompt compression baselines are provided in Appendix LongL.LM- Lingua extends L.L Lingua by incorporating task-specific information to enhance Tong-context compression, while LLMLingua-2 improves efficiency through the use of smaller model. To ensure
7

---

Table 4: Performance of various prompt compression methods under different compressed length constraints on LongBench and ZeroSCROLLS, Higher values indicate better performance. The best scores are highlighted in boldface.
LongBench
ZeroSCROLLS
Mothods
SingleDoc
MultiDoe
Summ.
FewShot
Synth.
Code
Avg.
# Tokens
Avg.
Tokens
Original Prompt
39.7
38.7
26.5
67.0
37.8
54.2
44.0
10,295
32.5
9,788
Zero-shot
13.6
31.3
15.6
40.7
1.6
36.2
23.5
214
48
10.8
32
306x
3,000 tokens constraönt
Retrieval-based Methods
BM25
32.3
34.3
25.3
57.9
45.1
48.9
40.6
3,417
3x
19.8
3,379
3x
SBERT
35.3
37.4
26.7
63.4
51.0
34.5
41.4
3,399
3x
24.0
3,340
3x
OpenAl
41.7
3,421
3,362
Compression-bused Methods
Selective Context
23.3
39.2
25.0
23.8
27.5
53.1
32.0
3,328
3x
20.7
3,460
3x
LLMLingua
31.8
37.5
26.2
67.2
8.3
53.2
37.4
3,421
3x
30.7
3,366
3x
I. ingunn2
35.5
38.7
26.3
69.6
21.4
62.8
42.2
3,392
3x
33.5
3,206
3x
LoLgnnMMnnnuu
40.7
46.2
27.2
70.6
53.0
55.2
48.8
3,283
3x
32.8
3,412
3x
EHPC (EMT)
44.2
49.1
25.1
68.8
54.0
63.0
49.7
2,892
3x
36.7
3,005
3x
2,000 tokens constraint
Retrieval-based Methods
BM25
30.1
29.4
21.2
19.5
12.4
29.1
23.6
1,985
5x
20.1
1,799
5x
SBERT
33.8
35.9
25.9
23.5
18.0
17.8
25.8
1,947
5x
20.5
1,773
6x
OpenAl
32.4
24.8
29.8
Compression-based
Methods
Selective-Context
16.2
34.8
24.4
15.7
8.4
49.2
24.8
1,925
5x
19.4
1,865
Sx
LLMLingua
22.4
32.1
24.5
61.2
56.8
34.6
1,950
5x
1,862
L.M Lingua-2
29.8
33.1
25.3
66.4
21.3
58.9
39.1
1,954
5x
33.4
1,898
5x
LongL.MM MLnigna
39.0
42.2
27.4
69.3
53.8
56.6
48.0
1,809
6x
32.5
1,753
6x
EHPC (EML)
44.5
50.7
24.8
68.9
51.5
61.9
49.6
2.004
5x
34.6
2,041
Table 5: Averaged time (in seconds) for different methods applied subset of LongBench.
Latency
Compression time
Inference time (2048 tokens)
Total
Method
LLMLingua
7.51
1.09
8.60
LongLLMLingg
67.44
1.31
68.74
LLMLingua-2
1.27
1.15
2.37
EHPC (ours)
0.88
1.11
1.99
MhatPT--3.5-Tubbo (all tokens)
2.16
reproducibility, employ greedy decoding and the temperature to 0 in all experiments. For prompt compression, we utilize the 11ama-3.1 1.1a--3.1-000 1-8B model. Additional details of the hyperparameters are provided in Appendix
Results The results of the prompt compression benchmarks presented in Table Consistent with previous research [Tiang et al.) 2023b), we report the averaged results for ZeroSCROLLS and LongBench, well the average performance for the sub-tasks on LongBench, with target compressed prompt lengths of 2, 000 and 3, 000 tokens. Table superior performance on average on both benchmarks under Tength constraints. The results show that EHPC exceeds the original prompts and significantly enhances the accuracy of the QA tasks, indicating its effectiveness in the retrieval tasks by alleviating the disturbance of the noisy context. In particular, our model performs well on code tasks in the LongBench, which can be attributed to the identification the key tokens involved in the inference coding tasks. Furthermore, performance of EHPC in summarization and few-shot learning tasks is competitive. Overall, our compression method achieves high-quality compression, even outperforming the original prompts on specific tasks.
that our method achieves
Compression latency We evaluated the running times of various methods using subset of Lenchench, including direct inference, I.MMLingua22, Long...MLinguu and our proposed approach. The results presented in Table 5 where the targeting compression is 2,048 tokens. The subset

---

Table 6: Performance of different acceleration methods various LLMs under the same KV cache memory usage. Higher values indicate better performance. We simplified some datasets for easier presentation The best scores boldfaced.
Single Document QA
Multi Document QA
Summarszation
Pew abot Learning
Synchetic
Code
Method
Qhaper
MubiQA es
HappalA
ZWiLMMOA
Mosigns
GerBopon
QMtSen
MalaNoww
TREC
TriviegA
S4MSum
PCinsl
Pite
RepoRonnch P
LCc
Avenge
LLaMA 3.1 SB Instruct
All KV
32,02
13.04
27.34
16.23
16.03
11.22
34.52
23.41
26.39
73.0
91.64
43.8
7.16
97.73
49.25
52.7
38.32
H2U- 1024
21.9%
10.96
23.17
16.06
15.16
10.15
30.76
23.75
26.33
68.0
90.68
42.54
7.40
71.34
51.6
46.4
54.77
Snapk 1-2444
31,08
11:17
25.33
14.81
15.73
10:69
26.05
22.89
67.5
91.89
42.85
7.67
98. 16
48.7
$2.1
35.25
GosFitte-- 1024
20.71
11:00
29.28
19.12
17.01
13.01
30.37
21.75
25.
63.0
90.70
42.50
7.15
92.22
35.0
38.5
34,50
EMPC (owrs) 1024
22.95
13,02
27.41
20.64
16.97
13.99
29:49
24.15
25:24
68.0
86.52
41.86
5.79
97.21
42.8
48.9
35.23
1120 2018
23.16
11.68
25.09
16.17
15.22
9.93
$2.13
23.32
26.73
68.5
91.32
43.97
6.52
72.79
$2.5
88.1
35.45
Seap 2048
31.45
11.99
26.34
15.73
16.03
11.66
20.64
23.24
26.44
09.5
91.48
42.68
7.21
98.03
49.5
$2.6
35.80
GemFiiher-oot 2048
24.35
12.63
25.39
19.58
17.03
14,11
33.15
22.31
26.49
09.5
91.59
42.64
4.61
98.75
38.8
47.3
35.87
KHPC (w04)))2048
30.98
14.38
30.50
19.35
16.23
13.02
32.9
24.54
26 72
71.
90.47
42.34
9.35
97.94
44.9
52.2
37.86
Phi 3.5 Mini 3JR Instract
All KV
27,51
17:23
35,63
21.70
25.70 70
11.68
34.14
23.17
24.95
71.5
87.37
13.08
7.17
83.85
46.0
46.2
34.62
1120 1024
18.25
12.97
29.89
20.75
20.90
9.90
32.02
21.69
24.70
67.5
85.45
20.16
1.37
46.80
46.6
63.3
31.36
Snap 1024
24.31
16.03
13.74
24.00
67.5
14.57
85.60
33.68
GenFilter 1024
16.57
16:29
34:20
51.20
23.64
0.30
1024
H20-2048
14:05
10.
$2.86
24,90
1.49
46,96
44.71
$1.04
SnupKV
10:13
(o4r))-2048)
24.80
39.27
27.06
14.05
33.26
24.48
37.77
2.54
45.44
contains 10 examples from RepoBench-P, with an average of 14,354 tokens. Each example was repeated times to reduce randomness. Specifically, ChatGPT-3.5-T0 5-Tubbo (all tokens) serves as the baseline using the original prompts, where the inference time is carried over from (Liskavets) et al. [024]]. Each prompt compression method reduces the average prompt length from 14,354 tokens 2,048 tokens. The experiments were carried out on a GPU with 48GB of VRAM. For fair comparison, we used the same language model, L1 Llama-3, 3. 1-8B, for all methods except LLM LLMLingua which requires a specialized model. The results in Table 5 indicate that our method significantly faster than LongL.LML.nge
and also outperforms L...Lig which is known for its efficiency. The lower latency of our compression strategy is attributed its reliance on the efficient pre-filling stage.
5.2 Acceleration of LLM inference
We demonstrate that EHPC effectively reduces memory overhead and computation costs during long- context inference of LLMs by compressing input prompts. By inferring over the K2 times compressed prompt, the KV cache during pre-filling is reduced times, also accelerating the decoding stage. In NMI setting, where the same model is used for both compression and inference, the acceleration achieved is comparable to that of KV cache compression methods with the same compression ratio. Therefore, we compare our method against the KV cache compression method on the LongBench long-text acceleration benchmark.
Baselines We consider acceleration frameworks that include KV cache compression and prompt compression as baselines. For KV cache compression methods, we select SoTA approaches, including H2O [Zhang et al.) 2023] and SnapKV [Li et al.) 2024), which eliminate unnecessary KV caches based on attention scores. For the prompt compression method, we select GemFilter (Shi et al. [2024], which utilizes attention scores of early layers detected by downstream tasks compress the prompts.
Implementation detail We utilize two popular long-context models: Llama-3. 3. 1-8B and Phi-3. 3. 8B, both support context window length of 128k. For each method, we set the target lengths to 1024 and 2048 tokens for prompt compression and KV cache compression, respectively.
Results In Table 6 we present the results of long text inference acceleration under KV cache con- straints of 1024 and 2048 tokens using various methods. EHPC achieves the best average performance compared to the basclines across different compression rates. Relative to KV cache-basee based methods,

---

our prompt compression method performs particularly well on specific tasks such QA, even out- performing the results obtained using the full KV cache. This improvement is particularly significant when applying our method to the Phi-3. 3.-33..0 88 model, enhancing its performance direct inference by an average of up to 40% in QA tasks, including Single-Doeumess QA and Multi-Document QA. However, prompt compression exhibits a decline in performance code-related tasks and few-shot learning tasks. Although the effectiveness of the KV cache compression method gradually decreases as the KV cache memory is reduced, it remains more advantageous than prompt compression for these tasks. Overall, EHPC demonstrates competitive results compared to the KV cache compression method, particularly in QA tasks.
Conclusion
We introduced EHPC, an efficient and effective prompt compression method that utilizes evaluator heads to leverage attention scores from the pre-filling stage of transformer-basee large language models. Unlike previous speedup methods that rely on training specialized small models, our approach is training-free and achieves state-of-the-art performance in prompt compression. We implement our evaluator head-based prompt compression (EHPC) method in two settings: native model inference and extended model inference. We demonstrate the effectiveness of EHPC in these settings in two main benchmarks, highlighting its potential to significantly reduce API costs for commercial use and accelerate long text inference.
References
Tianyi Zhang. Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 2024a.
Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, and Wei Han. Re- trieval meets reasoning: Dynamic in-context editing for long-text understanding. arXiv preprins arXiv:2406. 12331, 2024a.
Simranjit Singh, Andreas Karatzas, Michael Fore, Iraklis Anagnostopoulos, and Dimitrios Stamoulis. An LLM-Tool compiler for fused parallel function calling. arXiv preprint arXiv:2405. 17438, 2024.
Yanan Chen, Ali Pesaranghader, Tanmana Sadhu, and Dong Hoon Yi. Can rely on LLM agents draft long-horizog plans? let's take travelplanner as an example. arXiv preprint 4eXiv:2408.06318 2024.
Yucheng Li. Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201, 2023.
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 13358-13376, 2023a.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLinngua accelerating and enhancing LLMs in long context scenarios via prompt 2023b.
compression. arXiv preprint
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang. Chin Chw Lin, et al. LLMLingua-2. data distillation for efficient and faithful task-agnostic prompt compression. arXiv preprint arXiv:2403. 12968, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprini arXiv:2309. /7453, 2023.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.1078, 2024.
10

---

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen. Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language mod- els. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 34661-34710. Curran Asso- ciates, Inc., 2023, URL httlittpss/pprrceednnsssco neur cc/paper Secaaper_flllseeee htt.e//paper/2223//// pdf
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The eight Annual Conference on Neural Information Processing Systems, 2024. URL /tttts:////ppneeee www.ppppwwrei..................e net//forummxd==p65540002 http://e...oooeeeeeee/ee/e/eeeea
Suyu Ge. Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024a. URL httpss:/ www.ppenreviww...mm nit UNrFpDPMyo
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu,, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. PyramidK V: Dynamic KV cache compression based on pyramidal information funneling, arXiv preprint arXiv:2406. 02069, 2024.
Wenhao Wu, Yizhong Wang. Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404.15574, 2024.
Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorat- tention: Efficient KV cache compression through retrieval heads. arXiv preprini arXiv:2407. 15891, 2024.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long- long-context LLM inference with retrieval and streaming heads. arXiv preprint 0rXiv2240..00811 2024.
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. Lazyllm: Dynamic token pruning for efficient long context LLM inference. arXiv preprint arXiv:2407. 14057, 2024.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for contecontext LLMS via dynamic sparse attention. Advances in Neural Information Processing Systems, 17:52481-52515. 2024.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie. Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hy- pothesis for LLM KV cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024a.
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Pu- rushotham Kamath. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:1 14-12,, 2024.
David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 4022.fnningg-emmlp.4122 URL/htpps://aclantholoo wwwp:og....gg///0 http:///....iiinnseeeeeee findings-ennllpp 412
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts, ArXiv, 11422305,1778,, 2023. URL https://api semanticscholar org/
CorpusID:25886522
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 15:11079-11091 2022.
11

---

Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In shirth-seventh Conference on Neural Information Processing Systems, 2023. URL https
//openreview.nete net/Fforrumidd2204PPL1133
Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, 2024b. URL https::/ wwtp://eee/eeeeeeeeeeeeeee wwwtteortmneeeeeeeeeeeee (iw.orru2uiiRRRRE2ZZGEEEE
Joao Monteiro, Étienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. XC-cache: Cross to cached context for efficient LLM inference. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15284-15302, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ W12022..indings-entt
URL
org/ 2024. findings-emnlp 896/
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. N0.18653//1120022 11116///11/22211.c111o.....3333 acll-lng..333. URL httphhts//acaaatolllleeeoo owg.gg...rr//// owt.oog///2222 acl long. 353/
Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. Extending context window of large language models via semantic compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5169-5181. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2000 http///022..iiinnaaa.....00
Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, and Amir Gholami. Characterizing prompt compression methods for long context inference. arXiv preprint (xXiv:2407.08892 2024.
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. Discovering the gems in early layers: Accelerating long-context LLMs with 1000x input token reduction. arXiv preprint arXiv:2409. 17422, 2024.
Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane Luke. Prompt compression with context-aware sentence encoding for fast and improved LLM inference. arXiv preprint urXiv:2409.01227 2024.
Greg Kamradt. Needle In A Haystack- Pressure Testing LLMs. https://gitthb. com/ gkamradt/ LLMTest_Needl erredleInAHie elaHalysaaka 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ a02-.acl-long. 1 172. /tthhttss///cclnttiin wwwpooooooooooogggg//e ac1- 172
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oBennh: Extending long context evaluation beyond 100K tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 15262 15277, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL https::/
wwwgg/224..aa. acl -lng.lllooo..8 814
Nils Reimers and Iryna Gurevych. Sentence-BERT Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP--JCNLP), pages 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v I/D19- 1410. URL httptt:://acclnnthoooo.o.oo org/D19-111
12

---

Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disageregating prefill and decoding for goodput-optimized large language model serving. arXiv preprint arXiv:2401.0999 00960000770 2024.
Junlin Lv, Yuan Feng, Xike Xie, Xin Jia, Qirong Peng, and Guiming Xie. Critiprefill: A segment- wise criticality-basee approach for prefilling acceleration in llms. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL kloBbc76Sy
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information--sekinng questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599 0019-4610,2000 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong. editors, Proceedings of the 28th International Conference on Compu- tational Linguistics, pages 6609-6625, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10. 00.18653/1/20200coolig mainiin...m........ URL http:://acl.rttoooog...rr/// http://2222.cccoonoeeeeee main. 580
Harsh Trivedi, Niranjan Balasubramaniann Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022. doi: 10.1162/tacl a 00475. URL htthhtts///acaaatoolooggo org/
ta22.tac1-1.31
Jiaxin Bai, Zihao Wang, Yukun Zhou, Hang Yin, Weizhi Fei, Qi Hu, Zheye Deng, Jiayang Cheng, Tianshi Zheng, Hong Ting Tsang, al. Top ten challenges towards agentic neural graph databases. arXiv preprini arXiv:2501 22222501..22244 14224, 2025.
Zihao Wang, Weizhi Fei, Hang Yin, Yangqiu Song, Ginny Y Wong, and Simon See. Wasserstein- Fisher-Rao Embedding: Logical Query Embeddings with Local Comparison and Global Transport. arXiv preprint arXiv:2305.00 04034, 2023.
Hang Yin, Zihao Wang. Weizhi Fei, and Yangqiu Song. Efok-cqa: Towards knowledge graph complex query answering beyond set operation. In Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, page 5876-5887, New York, NY, USA. 2025. Association for Computing Machinery. ISBN 97984007 14542. doi: 10.1145/7711889..337722
URL http::ttt//ooioii org/10 10. 1145 471 896. 3737426
Hang Yin, Zihao Wang, and Yangqiu Song. Rethinking complex queries on knowledge graphs with neural link predictors. arXiv preprint arXiv:2304.07063 2023.
Weizhi Fei, Zihao Wang, Hang Yin, Yang Duan, and Yangqiu Song. Extending complex logical queries on uncertain knowledge graphs. arXiv preprint arXiv.2403.01508, 2024c.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association
13

---

for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 meaal-mmin.I1n 112. URL https://aclanchollogy hww.aannnhooooooooo//2/o2////
maacl-main... 112
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi- domain meeting summarization. In Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5905-5921, 2021.
Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: large- scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pages 1074-1084, 2019.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley, Longcoder: long-range pre- trained language model for code completion. In International Conference Machine Learning, 2023. URL httpss://ppp wewanti
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2024b. http:/ps//aaa.i..rr/aaa///////// 0309
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: zero-shot benchmark for long text understanding, In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977-7989, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2 1/2023. indings-emmlp.536 URL https: httpttts:s///llaattoooooooo org.////0oooyor/32333333// findings-emni 536 536
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper proposes EHPC, maining-ffee method for improving long-context inference efficiency in transformer-basee LLMs. EHPC identifies evaluator heads, a subset of attention heads that select the most relevant tokens during the pre-filling stage of inference. By leveraging these heads, the method compresses prompts before inference, reducing API costs and latency while maintaining performance. The main claims in the abstract and introduction reflect these contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the theorems are discussed throughout the paper, and general discussion of limitations can be found in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The complexity analysis of EHPC is given in Section 3.5 and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All the information needed to reproduce the numerical experiments is given in Sections Bnd 4 In Section we outline details for implementation and the baselines.
Additionally, we plan to make our code open source the time of publication to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the code is provided the supplementary material with instructions to reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/ public/guides/Coote
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The benchmarks are discussed in Appendix H In Appendix D we provide details for hyperparameters, and Section we outline details for implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: To make our statements, we set target compressed prompt lengths and per- formed the evaluation different datasets. Considering the computational resources required, the error bar not provided in this paper, and this method of reporting results common.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The compute resources used for the experiments are described Section S
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code of Ethics and the paper conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

Justification:
The broader impacts are discussed in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: EHPC an efficient prompt compression method, and no specific models or datasets are employed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are publicly available and properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperr swi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper is about new method without new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
This paper is about prompt compression using L.Mss. The LLMs used in the paper are publicly available. Our method is described in Section 3 and throughout the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
for Please refer to our LLM policy httpps////eerrppsccc/oeeeeeeeeee what should or should not be described.
21

---

A Limitations and broader impacts
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. The implementation of prompt compression methods in machine learning can have significant societal impacts. One major concern is the risk of information distortion, where critical nuances may be lost in the simplification process, potentially leading misinterpretaatoos. This can result in biased decision making, as compressed prompts may not convey the full context necessary for accurate judgments. Furthermore, reliance on these techniques can create communication barriers, particularly between individuals of diverse backgrounds, thereby hindering effective dialogue. Additionally, those skilled in using prompt compression may gain an unfair advantage in information processing, exacerbating knowledge inequality. In general, while prompt compression can enhance efficiiecy,, crucial to consider its implications for transparency, inclusivity, and critical thinking within society.
A limitation of our work is that due to the LLMs considered, our results are naturally only applicable to transformer architectures. Specifically, we empirically showed that identified evaluator heads consistently attend to key information in long inputs. We investigate their existence, generalizability. and robustness using both synthetic and real-world data, More rigorous theoretical analysis should be performed on the capacity of the evaluator heads.
B Algorithm
Require:
and transformer-based LLM f. Output compressed prompt a = (Ji Toz...... ... Tige) with M tokens. Detect the evaluator heads at layer le, Cy [4j, hy} by the pilot experiments.
Input prompt x = (I1, (2.,, NNN),
Ensure:
Get Als.n, during the pre-filling stage at layer la layer.
Get the scores by evaluator head: 81 EN.SISN
where N₂ is the observed window No
and N, = N - No-
Calculate the pooled scores: 82 Pool(s1, r)
Select the indices with high scores: J +- topk_index(ss) 82, k= M)
Sort the indices according to prompt order: J = sort(J) (1,,,..,,, fas):
Here, we provide the pseudocode flow of our method in Algorithms
C Background
Multi-head attention Transformer-based models autoregressively predict the next token based on the - preceding tokens according to
1h h m FF 8')
where h', m², 8' E Rd are the hidden states of the 1-th layer and FFN(-) denotes the feed forward network. Transformer models typically utilize multi-head attention, that the hidden state 8 at layer is computed as
= w Con "h
where H is the number of heads in each layer, and th denotes the hidden representation of the h-th head at layer The attention operation each attention head is
Qth (K
th Softmax
Vlh
(3)
ddz
where Qt Kt I RT and d₃ d/H H is the dimension of each head. As shown in Eq. B the attention mechanism aggregates information and selects important tokens from input prompts. The weights calculated by the dot product between queries and keys determine which tokens of the values are considered within this attention block. The independent multi-head attention mechanism enables the model to capture information from previous tokens in multiple ways.
22

---

LLM inference and KV cache compression Contemporary decoder-only LLMs generating new tokens with series of tokens& input involve two stages: the pre-filling stage and the decoding stage. During the pre-filling stage, the model processes compute the intermediate states (keys and values in attention operations), which are highly parallelized and computationally efficient. the decoding stage, L.MMs load the precomputed KV cache and generate each output token autoregressively through a forward pass, which slower due to its serial nature [hhong et al. (2024).
When processing long text input, the size of the corresponding KV cache increases dramatically, significantly raising both computational costs and time. To address this issue, KV cache compressing methods have been proposed that eliminate unnecessary KV caches. Many KV cache compression methods are based on attention weights to propose a policy to determine which tokens retain in (
memory. Let A Softmax
represent the attention matrix. The policy first averages the last rows represent the scores of input tokens, as follows:
dd,
A No
Σ
N-SISN
where N denote the number of tokens in the prompt, No is the observed windows length and N, is the length of remaining part, such that N N, + No. To further improve the contextual integrity, Li et al. 2024 apply the pooling operation perform clustering:
P8 k),
(5)
where Pool(..)) represents pooling operation such max(-) and Average(.). and k is the kernel size. This trick ensures that the identified tokens are continuous rather than isolated, resulting in more coherent semantics.
D Hyper-parameters
In this section, we introduce the hyperparameter for reproduction. We first introduce our detected evaluator heads and then introduce the hyperparameter to use these heads.
Detected evaluator heads We conducted pilot experiments using the "Needle-in-a--aystcck benchmark across three popular L.LMs: Llama-3. 1-8B-Innttruct, Codel 1ama-1., and 1--mmini-3.B-IIsstrree
For L.ama-3. --8--Insttruc which has 32 layers and 32 heads, the selected layer is 13, and the chosen heads are [18, 13, 21, 8, 11, 1, 3,
For CodeL 7B. also with 32 layers and 32 heads, the selected layer is 14, and the selected heads are [24, 3, 18, 7, 29, 2,
Finally, for 3ai-3.5-mini-3 8B-Instruct, which features 32 layers and 32 heads, the selected layer 17, and the chosen heads are [7, 17, 30, 2, 6, 16, 25, 18].
Using evaluator heads The hyperparameters applied during the evaluation of heads include the size of the observed windows, the pooling operation, and the kernel size for pooling. In all experiments, used the average pooling operation, the difference between average pooling and maximum pooling was experimentally negligible. For Llama-3. 1-8---88---- 288-Instruct we set the size of the observed windows and the kernel size for pooling to 16 and 32, respectively. For Phi-3.5-mini-4 8B-Instruct, the size of the observed windows and the kernel size for pooling continuous
set to 4 and 32, respectively. A larger kernel size typically results in compressed context, which is why we generally prefer to use larger kernel size.
E
Complexity
We further discuss the complexity of NMI in this section. Consider an LLM f with L layers, H attention heads per layer, and a hidden dimension d = dę H. Suppose that the model f receives an input prompt of N tokens and generates new tokens. In the pre-filling stage, each head computes Softmax QK" with Q, K E Nxd, resulting in complexity of Od). Thus, the total
23

---

Table 7: Speedup Ratios for Different Sequence Lengths Compared to Direct Inference in Pre-filling and Decoding Stages
Length
Model
Phase
8k
16k
24k
32k
Llama
Pre-fill
1.6x
1.8x
1.9x
2.0x
Decode
2.6x
4.9x
4.0x
10.3x
Phi
Pre-fill
1.7x
1.9x
OOM
OOM
Decode
1.2x
1.5x
OOM
OOM
complexity for the pre-filling stage is O(LHdg,2²) During the decoding stage, the model generates t tokens based on the pre-cached K, V E I RNxdy, At the + tokens generated. the softmax operation in each head involved Q E R 1xxd, V E (NN+))xxd and the complexity is O(d (N i)). calculate the total complexity for generating tokens, sum the complexities across all tokens:
12 dNNN + +1) = (Nt (t-1)² =) ²)))
(6)
2
Thus, the overall complexity for generating t tokens during the decoding stage is (NtH Regarding the complexity EMI, it remains similar if the inference model still transformer-hased.
F Speedup ratios
We present the EHPC speedup ratios compared to direct inference in Table 7 We evaluated the time taken for Needle-in-a-Haysstd with sequence lengths ranging from 8k to 32k, running five examples for each length. We directly take the pre-filling time for pre-fill and take the time of decode per token for decode. We also compute the speed ratio of our method with direct inference. The results demonstrate that our method effectively improves the pre-filling stage. In the decoding stage, the speedup ratio increases with length in the Llama model. In the Phi model, the baseline encounters out-of of-oyyyry issues 16k, while our method can successfully infer 32k sequences.
G Additional results for NMI (local model)
We provide the experiment results ooBench [Zhang al.) 2024b in Table g with Minference and SnapKV as baselines. Due to the Minference method encountering OOM errors with 32k long sequences on the Phi model, we primarily selected 32k as the truncation length. Additionally, provide experimental results of our method on the Phi model with 128k length to demonstrate that our model can effectively infer such long sequences.
G.1 Extremely Long Benchmark
Table 8: Additional evaluation BeBehh compared with Minference and SnapKV
Midel
Thmestte
Mathod
Phwckry
NumSer
Langltcc Sum
Langaalog OA
LongBook QA
Longlook Choice
ManFind
KV Renioaal
Aug
StapKV
23
27.1
FAIP::
21.1
324
45.9
125k
EHFE
100.0
99.5
168
228
55
18.0
493
26.9
32.0
41.2
StapkV
EHFC
25.1
23.1
3466
162
80
11.2
555
34.0
2538
25.5

---

Table 9: Comparison of Minference and CritePrefill on LongBench Benchmark
NarativeQA
Quger
MRQAves
HиępiQA
2WaIMHQA
MuSiQue
TriviQ4
SAMSam
29.06
78.70
30.10
14.15
(1024)
44.97
32.79
55.72
41.57
G.2
Comparison with additional baselines
We present the results of Minference (Jiang et al. 2024] and CritePrefill [Lv et al. 2025 on LongBench. Due the extensive decoding time required for long tokens in CritePrefill, have removed datasets that involve summaries.
H
Datasets
We present the synthetic and real-world benchmarks for assessing the capability of LLMs in handling long contexts.
H.1 Synthetic benchmarks
-aadle-in-a-Hyytackk Kamradt [2024] is well-known synthetic dataset used to benchmark the ability navigate long contexts. It involves randomly inserting sentence into variable- length long context, followed by querying given LLM to retrieve that specific sentence from a long context.
Ruler Hsieh et al. 2024 is synthetic long-context benchmark that extends "Needle-in- inllle-ii----i -a-a--i---aae-e Haystack" by offering more complex tasks. RULER encompasses diverse types and quantities of needles and introduces new task categories, such as multi-hop tracing and aggregation, to evaluate behaviors beyond direct searching within the context.
H.2 Real-world benchmarks
LongBench Bai et al. 20244 a widely used long-context benchmark that includes datasets across six types of tasks. The six tasks are single-document question answering, multi-document question answering, summarization, few-shot learning, code completion, and synthetic tasks for retrieval and counting. In line with previous research Jiang al. [2023a b). focus English datasets and encompass six tasks across 16 datasets. We present a detailed introduction in the following context.
Single-Doc QA
standard puestion-ansswete dataset that includes texts from Project Gutenberg and movie screenplays sourced from various websites. It contains 200 entries and is evaluated using the F1 metric.
NarrativeQA Kočiský al. 2018
Qasper Dasigi et al. 2021 a question-answering dataset focused on NLP publications, featuring abstractive, extractive, and yes/no questions. It consists of 200 entries and is evaluated using the F1 metric.
MultiFieldQA-en |Bai et al., 2024 is created from diverse sources, including legal doc- uments, government reports, encyclopedias, and academic publications. includes 150 entries and is evaluated using the Fl metric.
Multi-Doc QA
HotpotQA Yang et al. 2018 features many 2-hop questions crafted by native speakers, based related paragraphs. It contains 200 entries and is evaluated using the Fl metric.
2iikiMultihopQA (Ho al. 2020 includes up to 5-hop questions systematically con- structed with manual templates. Answering these questions requires reasoning paths that cannot be resolved by local content alone. contains 200 entries and is evaluated using the Fl metric.
25

---

MuSiQue (Trivedi et al., 2022 consists of up to 4-hop questions, eliminating shortcuts and questions about naturalness. Each question includes 2-4 supplementary paragraphs that outline the reasoning path and relevant content. It comprises 200 entries and is evaluated using the Fl metric.
Summarization
GovReport [Huang et al. 2021 collects comprehensive reports containing human-written summaries from the U.S. Government Accountability Office and Congressional Research Service, covering a wide array of national policy issues. It includes 200 entries and is evaluated using the Mouge-Lmetric..
QMSum [Zhong al. 2021] contains annotated pairs of meeting summaries across various domains, including product, academic, and committee meetings. It includes 200 entries and is evaluated using the Rouge-L, metric.
MultiNews [Fabbri et al., 2019) is a multi-document summarization dataset that clusters 2-10 news articles discussing the same event or topic, each paired with human-written summary, thus forming a new long-text summarization task. It includes 200 entries and is evaluated using the Rouge-L metric.
Few-Shot Learning To construct few-shot learning with long text, [Bai et al. 2024 selected a range of training examples from the following datasets to concatenate the context LongBench:
TREC Li and Roth 2002 is a classification dataset featuring fine-grained class labels. It includes 200 entries and is evaluated using the accuracy metric.
TriviaQA Zhong et al., [021] is another classification dataset that involves messenger-lin conversations accompanied by humann-ritten summaries. It contains 200 entries and is
evaluated using the Fl metric.
SAMSum [Fabbri et al./ [2019] is reading comprehension dataset consisting of question- answer pairs annotated with evidence passages. It includes 200 entries and is evaluated using the Rouge-L metric.
Code Completion Code completion is critical yet challenging task utilized by auto-completion systems to assist users in predicting and completing code based on previous inputs and context.
LCC [Guo al. 2023]
Sampled from the Long Code Completion dataset, this dataset is constructed by filtering code based on length within individual GitHub files. It incorporates preceding lines of code as context, with the next line serving as the answer. This dataset includes 200 entries and is evaluated using the Exact Match (EM) metric.
RepoBench-P [Liu et al. 2024b): Collected from GitHub repositories, this dataset aggre- gates relevant cross-file code snippets based module import statements. These snippets are combined with preceding lines of code in the current file to predict the next line of code, utilizing the most challenging XF-F setting. It comprises 200 entries and is evaluated using the Exact Match (EM) metric.
Synthetic Task
Two synthetic datasets evaluate the ability to retrieve and count from long contexts.
Derived from English Wikipedia, this dataset randomly samples 30 passages and selects one for summarization, with the task of identifying the original paragraph corresponding to the summary. It includes 500 entries and is evaluated using the Edit Similarity metric.
PassageRetrievallenn:
This dataset presents a more complex challenge by randomly selecting paragraphs from English Wikipedia, repeating, and shuffling them. The model is required determine the number of unique passages among the provided set. It consists of 500 entries and evaluated using the Edit Similarity metric.
PassageCount:
2025 Wang et al. 2023 is another kinds of reasoning tasks, including various types of questions (Yin et al. 2025
CCopleex query answering Bai et al.
2023 and context Fei al. 2024c
26

---

ZeroSCROLLS [Shaham et al. 2023 is well-known long-context benchmark that encompasses three types of tasks: summarization ton, question answering, and aggregation across datasets. line with prior research Jiveng et al. 2023a b), we focus on the validation set for evaluation, it is the only set providing ground truth data. Below, we introduce the ten datasets across the three tasks, using the same evaluation metric for each dataset.
Summarization
GovReport Contains long reports from the Congressional Research Service and U.S. Government Accountability Office, paired with expert-written summaries.
SummScreenFD: Comprises episode scripts from TV shows with community-contriuute recaps sourced from Wikipedia and TVMaze.
QMSum: A query-based summarization dataset featuring meeting transcripts, including academic, industrial, and parliamentary discussions, with each instance accompanied by specific query.
SQuALITY: A crocued dataset derived from Project Gutenberg stories, requiring summaries based on crowdsourced guiding questions.
Question Answering
Contains NLP papers from the Semantic Scholar Open Research Corpus, with questions based on abstracts answered by practitioners.
Qasper:
NarrativeQA: Features questions and answers derived from books and movie scripts, with questions crafted from summaries provided by annotators.
QuALITY: Comprises stories and articles requiring multiple-choice questions that necessi- tate reading substantial portions for accurate answers.
MuSiQue: Focuses multi-hop questions using Wikipedia paragraphs, including both answerable and unanswerable questions.
Agg Aggregation
sentiment aggregation task using 50 hotel reviews per hotel from the Space dataset, focusing on strictly positive or negative reviews.
SpaceDigest:
BookSumSort: A task based on the BookSum dataset, requiring the reordering of shuffled chapter summaries from selected books to their original order.
BoBench [Zhang et al., 2024b) We provide an overview of the various tasks included in our study, detailing their contexts, example counts, and average token statistics.
En.Sum This task involves summarizing a fake book created through core entity substitution. consists of 103 examples, with an average of 171.5k input tokens and 1.1k output tokens.
En.QA This task focuses on free-form question answering based on the content the fake book. includes 351 examples, with an average of 192.6k input tokens and 4.8 output tokens.
En.MC This task consists of multiple-choice questions derived from the fake book. contains 229 examples, averaging 184.4k input tokens and 5.3 output tokens.
En.Dia This task involves identifying speakers in partially anonymized scripts. It includes 200 examples, with an average of 103.6k input tokens and 3.4 output tokens.
Zh.QA This task entails question answering set of newly collected books. It consists of 175 examples, with an average 2068.6k 6k input tokens and 6.3 output tokens.
Code.Debug This task focuses on identifying which function in a code repository contains crashing error, presented multiple-choice format. includes 394 examples, averaging 14.7k input tokens and 4.8 output tokens.
27

---

Code.Run This task simulates the execution of multiple simple, synthetic functions. It contains 400 examples, with an average of 75. 75.2k input tokens and 1.3 output tokens.
Math.Calc This task involves performing calculations that include extremely long arithmetic equations. consists of 50 examples, averaging 43.9k input tokens and 43.9k output tokens.
Math.Find This task focuses finding special integers within a lengthy list. It includes 350 examples, with an average of 87.9k input tokens and 1.3 output tokens.
This task involves retrieving hidden keys in noisy long context. It consists of 590 examples, averaging 122.4k imput tokens and 2.0 output tokens.
Retrieve.PassKeyl
Retrieve.Number
This task focuses on locating repeated hidden numbers in a noisy long context. includes 590 examples, with an average of 122.4k input tokens and 4.0 output tokens.
Retrieve.KV2 This task involves finding the corresponding value from dictionary based on given key. consists of 500 examples, averaging 89.9k input tokens and 22.7 output tokens.
I Baselines of prompt compression
Sclective-Conter Li et al. 2023 applies the logits of a causal language model to compute the self-information for each token, word, or sentence, subsequently eliminating unnecessary content based on this ceff-information. To the best of our knowledge, this is the first work to study compressed prompts using pre-trained language models.
LLMLingua [Jiang et al. 2023a| divides the target prompt into several segments and allocates different compression budgets according to the perplexity distribution of these segments. Compressing long prompt is modeled scheduling problem, utilizing the perplexity of each segment to allocate budgets effectively.
LongLLMLinge /Jiang et al. 2023b further incorporates task information (such as questions for document QA) and employs a Question- Aware strategy to enhance the density key information in long contexts. LongLLMMiingu introduces conditional perplexity based task information and aims to improve information density relative to the task, building on the foundation established by LLMLingua (Jiang et al. 2023b
LLMLingua-2 Pan et al., 2024 is based on smaller, fine-tuned BERT models to improve efficiency, leveraging global information from extractive text compression dataset annotated by ChatGPT, Using a smaller model for prompt compression, we have found that this is, to the best of our knowledge, the fastest prompt compression method available.
Additional details for evaluator heads
In this section, deseribe how we construct task-aware synthetic data identify task-aware heads. We focus on two types of tasks: multi-hop reasoning and code completion. For multi-hop reasoning, utilize the multi-hop tracing task from the Ruler, which is extension of -Needle-i----Hassteee Multi-hop tracing involves randomly inserting several interconnected chains to assess how effectively the model tracks all the content of these chains in response given questions. This forms the basis of a multi-hop reasoning task. Regarding the code completion task, we first created a long code dataset that is unrelated to LongBench prevent data leakage. We then manually inserted the code that needs completion into the context, effectively generating the necessary evidence. These two types of data are designed identify specific heads within their respective domains using synthetic data tailored to those domains.
28


Reviews:
[{"review_id": "SiWXVCgzG4", "reviewer": "Reviewer_jTYT", "summary": "The paper proposes Efficient Prompt Compression (EPC), a method to reduce the cost and latency of large language model (LLM) inference when dealing with long prompts. EPC trains a small prompt compressor network to transform long input prompts into compact representations. Experiments on multiple benchmarks (QA, summarization, reasoning) show reduced latency and token usage with minimal accuracy drop.", "questions": "How well the compressor works on tasks or domains different from the training set.\nHow sensitive is EPC performance to the size of the compressor model?\nResults show a promising trade-off between compression ratio and accuracy, but more variety in datasets and ablations on compressor architecture would strengthen the claims.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Addresses the real and growing cost problem in LLM applications that use long context prompts.\n- Demonstrates large token savings and latency reduction without heavy retraining of the base LLM.\n- Applicable across tasks (QA, summarization, reasoning) without task-specific engineering.\n\nWeaknesses\nWhile EPC is compared with some baselines, the evaluation omits certain recent context compression techniques like retrieval-augmented methods or structured summarization models.\nIn high-compression regimes, some performance drop is visible, but the paper doesn’t deeply analyze what kinds of information get lost.\nThe cost and data requirements for training the compressor are not sufficiently discussed.", "was_revised": true, "final_justification": "I appreciated the paper's contributions and authors' responses.", "rebuttal": "We thank the reviewer for their overall positive feedback. After carefully reading the comments, we noticed that some of the raised points (e.g., Seer’s effect on end-task performance) do not seem to apply to our submission. We suspect a possible mix-up with another paper’s review. Nevertheless, we have addressed every question below to the best of our understanding. We would be grateful if the reviewer could share any clarifying remarks during the discussion period, and we are happy to provide further details as needed.\n\n>1. Adaptation to vision-language models or audio-augmented transformers where KV size and reuse patterns.\n\nReducing the token load for multi-modal models is an important and pressing issue. While the current paper focuses on LLMs, we have not yet had the opportunity to conduct more experiments on models such as vision-language and audio-augmented transformers. We are optimistic that the proposed method has the potential in such multi-modal models, and we hope to have more definite results in future research.\n\n>2. Adaptive $\\lambda$ weights in the score function.\n\nThank you for the question; however, we are not sure if the reviewer is referring to the utility scores in Eq. (1). The score is computed with a pooling operation, where the set of evaluator heads $\\mathcal{C}_f$ is chosen according to the accumulated score of the evidence in Eq. (2). We selected the top-$k$ heads according to line 228, where we control the cardinality of $\\Lambda$ to be less than $k$.\n\n>3. End-task performance (e.g., summarization ROUGE scores, QA accuracy) beyond token-level match.\n\nThank you for the question. In line with previous research, we have conducted experiments on the LongBench benchmark, which involves six tasks: single-document QA, multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks for retrieval and counting. The results with respect to task performance can be found in Tables 4 and 5 (main paper). \n\n>4. Incorporation of existing inference accelerations like speculative decoding, FlashAttention, and prefill skipping.\n\nThank you for the insightful suggestion. Prompt compression is orthogonal to existing inference-acceleration techniques, so the two can be readily combined: after our method shortens the prompt, any subsequent speed-up (e.g., speculative decoding, FlashAttention, prefill skipping) can be applied whether the serving model is the same or different, and whether it runs in the cloud or on the edge. Table 6 already shows that our approach yields the best average performance under KV cache constraints; coupling it with these orthogonal optimizations will therefore amplify the gains at inference time. We will incorporate this discussion in the final version.\n\n>5. Performance across model sizes and the relationship between cache threshold and model size.\n\nThank you for the valuable suggestions. However, supporting models with a larger size involves substantial work and timehave lots of work to do; we are still working on this. If we have additional results, we will include them intoin our revised manuscripts."}, {"review_id": "ZRFLQQjPDZ", "reviewer": "Reviewer_iPeJ", "summary": "This paper introduces EHPC (Evaluator Head-based Prompt Compression), a novel training-free method for compressing long input prompts.\nEHPC leverages specific attention heads in transformer-based large language models (LLMs), termed evaluator heads, that identify and retain the most important tokens for inference.\nEHPC accelerates long-context processing by skimming input prompts during the pre-filling stage, passing only critical tokens to subsequent layers.\nThe authors analyze the complexity of EHPC and investigate the existence, generalizability and robustness of EHPC.\nExperimental results show that EHPC achieves state-of-the-art performance in both prompt compression and long-context inference acceleration while reducing computational costs and API call expenses.", "questions": "1. Does NMI perform better than EMI when the computation budget is controlled the same?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. The identification of evaluator heads as key components for token selection is a compelling contribution.\n2. Unlike methods requiring fine-tuning or auxiliary models, EHPC works out-of-the-box, making it easily applicable to existing LLMs without additional overhead.\n3. The paper demonstrates SOTA performance on standard benchmarks, showcasing EHPC’s effectiveness in both compression quality and inference acceleration.\n\n**Weakness**\n1. The methodology part is somewhat confusing. For instance, when detecting evaluator heads in Section 3.4, the authors use only the last row of the attention matrix, but all the rows are included in the calculation of utility scores (Equation 1). Moreover, some denotations have mistakes. For instance, in line 157, $A^{hl}\\in\\mathbb{R}^{N\\times N}$, not $A^{hl}\\in\\mathbb{R}^{H\\times L}$.\n2. The differences between EHPC and existing attention-based compression methods (e.g., SnapKV) are not clearly discussed. Therefore, the novelty of the method is not very clear.", "was_revised": true, "final_justification": "I will increase the score as my major concerns have been addressed.", "rebuttal": "We thank the reviewer for their insightful and positive feedback. We are encouraged that they found our method compelling, easily applicable, and effective. We answer specific questions below and will incorporate all feedback in the final version.  \n>Q1: Does NMI perform better than EMI when the computation budget is controlled the same?\n\nThank you for the insightful question.  We conduct the experiments to infer the compressed prompt in Table 1 below. The diagonal results are NMI, while the others are EMI. For averaged results, we find that NMI is better than EMI because the same model shares a similar pattern. \n\n\nTable 1: The averaged results with 2048 compressed prompts on EMI on different settings.\n\n| Model                                | Llama3  infer  | Phi inference |\n|--------------------------------------|------------|-----------|\n| Llama3 compres    |37.9 (NMI)    | 36.9  (EMI)     |\n| Phi compress | 35.7  (EMI)       | 39.0   (NMI)  |\n\n>W1: The methodology part is somewhat confusing. For instance, when detecting evaluator heads in Section 3.4, the authors use only the last row of the attention matrix, but all the rows are included in the calculation of utility scores (Equation 1). Moreover, some denotations have mistakes. For instance, in line 157, $A\\in \\mathbb{R}^{N\\times N}$, not $A\\in \\mathbb{R}^{H\\times L}.$\n\n- Thank you for catching this; the typo in Line 157 has been fixed in the revised manuscript. \n\n- In  Equation 1, we select the last few rows from N_r to N, where N is the prompt length and N_r is a parameter adjusting the number of selected rows. Selecting the last few rows is generalized from selecting the last row, where previous research [1] has shown that this can capture more information.\n\n- When detecting evaluator heads in plot experiments, we just select the last row in Section 3.4 for convenience. While computing scores by the evaluator heads, we select the last few rows in Equation 1 for strong performance.\n\n>W2: The differences between EHPC and existing attention-based compression methods (e.g., SnapKV) are not clearly discussed. Therefore, the novelty of the method is not very clear.\n\nThank you for highlighting the need for a clearer distinction.  We provide two key differences below and have included them in the revised manuscript. \n\n- 1. SnapKV and other implicit methods retain the original prompt length and only sparsify the KV cache after computing the full attention map. EHPC, in contrast, removes tokens to reduce the number of tokens in the prompt. The compressed prompts can be applied to other language models, which SnapkV and other implicit methods fail to do. \n- 2. We design pilot experiments to find the evaluator heads for improving the compression quality and accelerating the compression time. In Table 6, we have also empirically compared our method with acceleration methods under the same KV cache memory usage. \n\n\n---\n[1] SnapKV: LLM Knows What You Are Looking for Before Generation, NeurIPS, 2024"}, {"review_id": "x8hKjS7XOr", "reviewer": "Reviewer_Dg6c", "summary": "This paper presents EHPC (Evaluator Head Based Prompt Compression), a novel approach for LLM prompt compression by preserving tokens only the tokens that receive high attention from “evaluator heads” - a set of specific attention heads, usually found in the earlier model layers, that are crucial for focusing on important information in long context tasks. The method employs a two-pass prefill approach for compression. The first pre-fill pass runs a few initial layers of the LLM (containing the evaluator heads) to compute the importance of each prompt token. This is followed by a second prefill pass over all the layers of the model, but with a subset of the prompt tokens. Extending on previous research, the authors show the existence of evaluator heads across models and their generalizability to downstream tasks (when identified using synthetic data). The authors evaluate EHPC in two modes: (1) EMI: using a separate model for prompt compression than the target inference model, (2) NMI: using the same locally deployed model for compression and target inference. Their results show that EHPC outperforms existing prompt compression methods (in both accuracy and system performance) in the EMI mode and is competitive with KV-Cache compression methods in NMI mode.", "questions": "- I understand that the purpose of evaluator and retrieval heads is different, but is there an overlap between them - i.e. are most evaluator heads the same as retrieval heads?\n- When identifying evaluator heads, what is the rationale behind extracting only the last row of the attention matrix?\n- In the robustness study, can you confirm if the actual evaluator head indices identified over different tasks are the same or have significant overlap?\n- Can you provide details on how the downstream performance was measured in the generalizability study with a limited number of attention heads?\n- I think the authors should use LongBench v2, which is an updated version of the LongBench dataset with much larger context lengths to show the effectiveness of their method further.\n- A small discussion on whether and how the NMI performance can be improved would be helpful.\n- In Table 5, the inference time (second column) is different for different prompt compression methods. Why is that the case?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strenghts**\n\n- The results for EHPC in EMI mode are impressive, as it consistently outperforms existing prompt compression methods. Specifically, EHPC is far more efficient with its two-pass approach for inference than other methods, as shown in Table 5\n- The authors evaluate their method against strong and plentiful baselines on a variety of tasks for both EMI and NMI evaluations\n- The method is original and definitely pushes the field of prompt compression further\n\n**Weaknesses**\n\n- The analysis around generalizability and robustness of evaluator heads is limited and not convincing enough:\n    - I would implore the authors to evaluate on more models covering different sizes and architectures, like 30B or 70B models, if feasible, and architectures like MoE models\n    - The generalizability study in Section 4 does not provide details on how the downstream performance is calculated on the given set of tasks with a subset of the heads. Additionally, all the subsets of tasks shown are from LongBench. It would be nice to see some tasks from RULER as well\n    - The robustness study does not tell me whether the actual heads identified using different tasks are the same or different. Even if downstream performance does not change, the actual heads identified might be different. This is a crucial distinction between saying the evaluator heads are task agnostic versus evaluator heads identified using task X have a good overlap with evaluator heads of task Y.\n- While the downstream results in the EMI mode are impressive, the NMI mode evaluation shows that EHPC might not be the best approach to use when compared to KV-Cache compression methods like SnapKV. While the authors acknowledge this, a discussion on whether this is an inherent issue with prompt compression methods or ways to improve this performance in future work would be helpful.\n- EHPC does not apply to decode heavy workloads", "was_revised": true, "final_justification": "The authors have addressed most of my major concerns:\n\n- Robustness and Generalizability Evaluation: The authors provided an evaluation on Llama2-13b-64k, details on the task evaluation in the generalizability study, and results showing that there is overlap in the evaluator heads identified over different tasks. While it is debatable whether the overlap is significant in certain cases, and I would like to see a deeper evaluation in the final version, this suffices for now. \n\n- EHPC's applicability to decode heavy workloads: The authors provide results on a few generative tasks (not very decode-heavy). I think it is fine, as the main focus of the paper is long prompt compression.\n\nApart from this, I am satisfied with the answer to all other minor questions I had. I maintain my score of accept.", "rebuttal": "We thank the reviewer for their constructive and positive feedback. We are encouraged that they found our method efficient, adequately evaluated, and original. We answer specific questions in the following and will incorporate all feedback in the final version.  \n\n> W1.1: I would implore the authors to evaluate on more models covering different sizes and architectures, like 30B or 70B models, if feasible, and architectures like MoE models\n\nThank you for the valuable feedback. We include properties of evaluator heads on Llama2 **13B** 64k below. If we have more computation, we will include the models with larger sizes and MoE models.\n\n- Existence \n\nThe score ratio of the selected layer is 70%, and the score ratio of selected heads at selected layers is 55%. This shows that our selected evaluator heads have significantly high scores\n\n- Generalizability \n\nWe present the results of evaluator heads on real-world tasks in Table 2 below. The findings indicate that NMI (Llama2 13B 64K) effectively understands the compressed data. However, due to limited memory, we had to truncate the context length to 8K, which resulted in relatively poor performance.\n\nTable2; The results of real-world tasks in LongBench. For Llama2-13b-64k, we truncate the context length to 8k.\n| Dataset          | narrativeqa | musique | TREC | repobench-p |\n|------------------|-------------|---------|------|-------------|\n| Llama2-13b-64k    | 14.2        | 7.1     | 70   | 44.75       |\n\n- Robustness \n```\nllama 2 13b 64k: \nLayer of evaluator heads: 13 layer\nNeedle QA :[23, 12, 13, 14, 24, 21, 26, 17],\nVariable Tracking (reasoning): [26, 21, 17, 23, 24, 14, 36, 11],\nCommon heads: [14, 17,21, 23,4,26]\n```\nDue to limited memory, we just conduct the results of variable tracking and Needle QA. The results show that these heads found by different tasks have 6/8 common heads.\n\n>W1.2: The generalizability study in Section 4 does not provide details on how the downstream performance is calculated on the given set of tasks with a subset of the heads. Additionally, all the subsets of tasks shown are from LongBench. It would be nice to see some tasks from RULER as well.\n\n- Sure. Below we spell out the exact pipeline we used to evaluate downstream performance.\n\n1. Given a prompt and a set of heads, we first compute the attention scores of these heads. \n\n 2. Following Eq. 1 in the paper, we compute the $s$ score for every token in the prompt. Then we select the top-k tokens to form the compressed prompt. \n\n3. The compressed prompt is fed to the LLM for inference. Finally, task-specific metrics are computed on the model’s output. . These baseline tasks are discussed in Appendix H.\n\n- For the generalizability, we aim to evaluate real-world tasks while the tasks in Ruler are synthesized. We also include the tasks from Infinity Bench [2] in Table 4 (Section 4, main paper).\n\n\n>W1.3: The robustness study does not tell me whether the actual heads identified using different tasks are the same or different. Even if downstream performance does not change, the actual heads identified might be different. This is a crucial distinction between saying the evaluator heads are task agnostic versus evaluator heads identified using task X have a good overlap with evaluator heads of task Y.\n```\nLlama 3.1 8B Instsrut :\nLayer of evaluator heads: 13 layer\nNeedle QA : [1, 4, 8, 11, 13, 17, 18, 21]\nVariable Tracking (reasoning):  [1, 2, 8, 9, 11, 13, 18, 21],\nCode completion:  [3,4,5,6,13,18,21,27]\nCommon heads: [8,13,18,21] \n```\n```\nmistral-nemo-instruct-2407: 14 layer\nNeedle QA :  [10, 11, 25,  8, 29, 24, 17,  9]\nVR : [17, 25, 11, 10, 16,  0, 30, 29]\nCode: [29, 25, 17, 30, 10, 11,  8, 28]\nCommon heads: [10, 11, 17,25, 29]\n```\n\nThank you for the question. The above results show that:\n \nFor Llama 3.1 8B Instruct,  there are **4 heads** in common on three tasks **in the selected 8 heads.**\n\nFor mistral-nemo-instruct-2407,  there are **5 heads** in common on three tasks **in the selected 8 heads.**\n\n>W2: A small discussion on whether and how the NMI performance can be improved would be helpful.\n\nThank you for the valuable suggestion. We provide the discussion below and have included this as future work in the revised manuscript.\n\nTable 6 (main paper) demonstrates that NMI struggles with few-shot learning and code tasks, which demand a high level of context integrity. Existing compression methods prioritize semantic salience at the expense of fluency and structure, leading to downstream accuracy drops. Preserving both structure and fluency during compression is therefore a critical direction for future work.\n\n>W3: EHPC does not apply to decode heavy workloads\n\nWe provide the results of Infinity Bench in the table above, which covers 32 k-input summarization and long-form generation tasks. For tasks involving heavier decode workloads, we consider for future work.\n\n\n>Q1: I understand that the purpose of evaluator and retrieval heads is different, but is there an overlap between them - i.e. are most evaluator heads the same as retrieval heads?\n\nOur evaluator heads are designed in the same layer, whereas the retrieval heads lie in different layers. Consequently, the two sets are almost disjoint; on Llama-2-7B-80K and llama2-13b-64k we observe zero overlap between the heads reported in [1] and the ones we identified as evaluators.\n\n>Q2: When identifying evaluator heads, what is the rationale behind extracting only the last row of the attention matrix?\n\nWe follow common practice and use the last row of the attention matrix as the scoring vector for each token. This row contains the attention scores that the final (query) token assigns to all earlier positions, which is exactly the distribution the model itself uses for next-token prediction. Treating these scores as the token’s “importance” therefore aligns our evaluator with the model’s own generative signal.\n\n>Q3; In the robustness study, can you confirm if the actual evaluator head indices identified over different tasks are the same or have significant overlap?\n\nSure, see response to W1.3.\n\n>Q4: Can you provide details on how the downstream performance was measured in the generalizability study with a limited number of attention heads?\n\nSee the response to W1.2.\n\n3. The compressed prompt is fed to the LLM for inference. Finally, task-specific metrics are computed on the model’s output. . These baseline tasks are discussed in Appendix H.\n\n> Q5: I think the authors should use LongBench v2, which is an updated version of the LongBench dataset with much larger context lengths to show the effectiveness of their method further.\n\nThank you for the valuable suggestion. We are still working on evaluating EHPC  LongBench v2 and hope to include more results in the final version. We provide the results of 32k on the Infinity Bench for validating our performance on a larger context length. The results in Table 3 show that our NMI setting achieves consistently strong results compared with SnapKV. \n\nTable 3; The results of Infinity Bench. We evaluate SnapKV and our NMI setting on the 32k token length.\n\n| Model  | Truncate | Method     | Passkey | NumStr | LongBook Sum | CodeDebug | LongDialog QA | LongBook QA | LongBook Choice | MathFind | KV Retrieval | Avg   |\n|--------|----------|------------|--------:|-------:|-------------:|----------:|--------------:|------------:|----------------:|---------:|-------------:|------:|\n|    Llama    | 32k      | SnapKV     |   27.1  |  26.6  |        26.5  |      2.3  |          7.0  |       14.8  |           56.3  |    21.3  |         5.0  | 20.8 |\n|        |          | EHPC       |   27.1  |  27.1  |        18.8  |     27.4  |          3.0  |       16.9  |           45.9  |    22.3  |        17.6  | **22.9** |\n| Phi    | 32k      | SnapKV     |   27.1  |  10.3  |        23.4  |     15.5  |          7.3  |        9.5  |           43.2  |    30.6  |         0.4  | 18.6 |\n|        |          | EHPC       |   27.1  |  27.1  |        24.6  |     16.2  |          8.0  |       11.2  |           55.5  |    34.0  |        25.8  | **25.5** |\n\n>Q6: A small discussion on whether and how the NMI performance can be improved would be helpful.\n\nSee response to W2.\n\n>Q7: In Table 5, the inference time (second column) is different for different prompt compression methods. Why is that the case?\n\nIn the second column, the 2048 tokens represent the target token length. However, the compression methods in these baselines often do not strictly equal 2048, resulting in varying inference times.\n\n[1] Retrieval Head Mechanistically Explains Long-Context Factuality, ICLR, 2025.\n[2] \\infty${B}ench: Extending Long Context Evaluation Beyond 100{K} Tokens, ACL, 2024"}, {"review_id": "TphdTZBWoH", "reviewer": "Reviewer_1UAM", "summary": "This paper introduced a fast, training-free prompt compression method EHPC (Evaluator Head-based Prompt Compression) for prompt compression and long-context inference acceleration. EHPC is built upon the pre-filling stage of LLMs, which reduces the number of tokens in both the pre-filling and decoding stages and the KV cache size, which accelerates the inference. EHPC identifies a subset of attention heads in transformer-based LLMs (i.e., evaluator heads) which can select tokens in long inputs that are most significant for inference. This method allows LLMs to efficiently \"skim\" prompts by utilizing only the initial layers containing evaluator heads during the pre-filling stage, retaining only the crucial tokens for downstream inference. EHPC achieves state-of-the-art performance on both prompt compression and long-context inference acceleration benchmarks.", "questions": "EHPC selects heads from the highest scoring layer, would combining heads from multiple layers helps?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths\n    * The paper is well-written and easy to follow\n    * The proposed method applies in the pre-filling stage, which compresses the input prompt before inference\n    * The proposed model is evaluated on various benchmarks and LLMs, and compared with SOTA methods across multiple metrics, including latency. Results show competitive or superior performance compared with SOTA methods.\n\n* Weaknesses\n    * the empirical evidence is strong, the paper lacks deeper theoretical analysis on the evaluator heads.", "was_revised": true, "final_justification": "After reading the rebuttal responses and other reviews, I keep my original rating.", "rebuttal": "We thank the reviewer for their insightful and positive feedback. We are encouraged that they found our paper well written and our method effective and adequately evaluated. We answer the questions below.  \n\n>W1: the empirical evidence is strong, the paper lacks deeper theoretical analysis on the evaluator heads.?\n\nWe provide the properties of the evaluator heads in Section 4, which provides an empirical explanation of the found heads. We hope these discussions are useful to understand why our methods work.\n\n> Q1:EHPC selects heads from the highest scoring layer, would combining heads from multiple layers helps?\n\nThe ablation study in Table 4 below shows that selecting the top heads from multiple layers does not improve performance. Moreover, it forces us to compute attention for every chosen layer, which sharply increases latency. Thus, we simply select heads from the highest scoring layer.\n\nTable 4; The ablation study on single/multi-layer. A single layer represents selecting the heads from the layer with the highest scores. Multi-layer selected the top heads from any layers.\n| Dataset       | NarrativeQA | qasper | MultiFieldQA-en | HotpotQA | 2WikiMultihopQA | MuSiQue | TREC  | TriviaQA | SAMSum | PCount | PRe   |\n|---------------|-------------|--------|-----------------|----------|-----------------|---------|-------|----------|--------|--------|-------|\n|Single layer     | 20.98       |  **14.38**  | **30.5**            | **19.35**    | 16.23           | **13.02**   | **71**    | 90.47    | **42.24**  | **9.35**   | **97.94** |\n| Multi layer    | 22.05       | 12.93  | 27.47           | 18.39    | 16.73           | 12.54   | 69.78 | 90.65    | 41.58  | 7.25   | 97.75 |"}]

Author rebuttal:

Meta-review: This paper introduces Evaluator Head-based Prompt Compression (EHPC), a training-free method for compressing long input prompts in large language models (LLMs). EHPC identifies "evaluator heads" — specific attention heads in early transformer layers that focus on crucial information — and uses them to retain only important tokens. This strategy reduces computational costs, accelerates inference, and lowers API usage expenses. The idea of leveraging evaluator heads for token selection is novel, pushing the frontier of prompt compression research. EHPC is training-free and model-agnostic, making it easy to apply to existing LLMs without extra overhead. Extensive experiments demonstrate strong and often state-of-the-art performance across multiple tasks.

All reviewers agree that the paper addresses an important and timely problem: efficient LLM inference with long prompts. The contributions are novel, the empirical results are strong, and the method is practical and impactful. The consensus is that this paper makes a valuable contribution to the area of efficient LLM deployment. I therefore recommend acceptance.
Decision: Accept (spotlight)