DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

Authors: Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, Song Han
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models; Long Context; Efficiency;

Abstract:
Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges.
Caching all Key and Value (KV) states across all attention heads consumes substantial memory.
Existing KV cache pruning methods either damage the long-context capabilities of LLMs or offer only limited efficiency improvements.
In this paper, we identify that only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for processing long contexts and require full attention across all tokens.
In contrast, all other heads, which primarily focus on recent tokens and attention sinks—referred to as Streaming Heads—do not require full attention.
Based on this insight, we introduce DuoAttention, a framework that only applies a full KV cache to retrieval heads while using a light-weight, constant-length KV cache for streaming heads, which reduces both LLM's decoding and pre-filling memory and latency without compromising its long-context abilities.
DuoAttention uses a lightweight, optimization-based algorithm with synthetic data to identify retrieval heads accurately.
Our method significantly reduces long-context inference memory by up to 2.55$\times$ for MHA and 1.67$\times$ for GQA models while speeding up decoding by up to 2.18$\times$ and 1.50$\times$ and accelerating pre-filling by up to 1.73$\times$ and 1.63$\times$ for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention.
Notably, combined with quantization, DuoAttention enables Llama-3-8B decoding with 3.33 million context length measured on a single A100 GPU. Code is provided in https://github.com/mit-han-lab/duo-attention.

Decision: accept (poster)
Reviewer scores: 8, 5, 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729669894340

thread_type:
official_review

<thread_object>
creator: reviewer_jvpb

<content>
summary:
Overall, this is an interesting and solid work. It categorizes all attention heads of LLMs into two types—Retrieval Heads and Streaming Heads—based on an offline analysis. By pruning the KV cache of Streaming Heads, this study achieves more efficient inference.

strengths:
1. The paper is well-written and easy to follow, with a novel methodology.
2. The work is solid, having been evaluated on multiple benchmarks. It also combines the orthogonal technique of quantization, further enhancing its results.
3. The evaluations demonstrate promising results.

weaknesses:
1. **Choice of Baseline**: DuoAttention is essentially a KV cache compression algorithm, and all the performance gains come from reducing the amount of KV cache that was previously stored, whether in the prefilling stage or the decoding stage. While the authors compare their method to H2O, TOVA, and StreamingLLM, these approaches are no longer state-of-the-art . For instance, SnapKV[1], which was published in April and is now widely regarded as a more powerful KV cache compression method, should have been considered. In the commonly used LongBench evaluation, SnapKV outperforms H2O with only one-quarter of the cache budget[1]. Additionally, in needle-in-a-haystack tasks, SnapKV benefits from its pooling operations and significantly improves accuracy compared to H2O. When compressing an 8K sequence to 128 tokens, SnapKV incurs minimal performance loss, outperforming H2O by a large margin, as shown in [2]. In the experimental section, the authors highlight a limitation of H2O: “Since the original designs of H2O and TOVA do not support long contexts, we modify their algorithms by replacing the pre-filling stage with FlashAttention and simulating decoding for the last 50 tokens of the input, following Tang et al. (2024b).” This limitation arises because H2O requires the accumulation of global attention weights, which is incompatible with FlashAttention. However, SnapKV and its successors only observe attention scores within a small observation window for compression, making them highly compatible with FlashAttention. This allows recalculating only a small portion of attention weights  for compression when combined with chunked prefill. Therefore, SnapKV would be a more appropriate baseline for this paper.

2. **Inappropriate statements:**

a.  _“Moreover, DuoAttention is **fully compatible** with important optimization techniques like GQA and quantization.” “Approximative attention methods, such as H2O (Zhang et al., 2023b), StreamingLLM (Xiao et al., 2023b), TOVA (Oren et al., 2024), and FastGen (Ge et al., 2024), often compromise accuracy in long-context applications and are **incompatible** with essential KV cache optimization techniques like GQA"_ 

I believe the compatibility of DuoAttention with GQA is not significantly different from other methods. DuoAttention forcedly classifies a group of attention heads under GQA into the same category through offline analysis, thus upporting GQA. However, other methods could achieve the same effect by accumulating weights for a group of kv cache during cache eviction. Therefore, GQA compatibility seems more like an implementation detail rather than a unique feature of the algorithm. In fact, StreamingLLM does not encounter any GQA compatibility issues. The lack of GQA compatibility in earlier works, in my opinion, stems from the fact that GQA was not widely adopted when those methods were initially proposed. Later methods, for the sake of comparison, maintained this approach without further integrating GQA, which could have been easily addressed at the code implementation level. A recent study[3] applying cache eviction in paged attention has demonstrated that this compatibility can be easily achieved in practice.

b._“Also, these methods (H2O, TOVA, StreamingLLM) cannot reduce the prefilling cost of long-context LLMs.”_

 Previous evaluations of cache compression often compress the KV cache after the prefilling stage to ensure comparability. Since the KV cache for each layer can be compressed immediately after the completion of that layer's prefilling computation, this already substantially reduces the peak memory usage during the prefilling process. If further reduction in computational cost during the prefilling stage or a more significant decrease in peak memory usage for very long input texts is desired, combining these methods with chunked prefilling can provide additional acceleration—a straightforward solution. This is particularly applicable to StreamingLLM, which is same with the same Streaming Head in this paper, and thus faces no obstacles in applying such methods. If one argues that the H2O method combined with chunked prefilling may require additional accumulation of global attention weights, the additional baseline SnapKV can effectively address this issue.

questions:
1. The core of this paper focuses on prioritizing the retention of the KV cache in important attention heads while attempting to discard less important cache in other heads, based on offline detection results. Some follow-up works on SnapKV seems to align closely with this approach[3][4]. For example, [4] employs a similar strategy by identifying important attention heads through online analysis of "altering model outputs," subsequently allocating more budget to these key heads and reducing the budget for others. What do you think is the relationship between these budget allocation strategies and the detection of retrieval heads?

2. In Equation 2, why is the index i set to $L$? Shouldn't it be $N$ instead?

3. How do you control the cache budget to a specifical ratio like 50% in the experiments? It seems challenging to precisely manage the cache budget within this approach.

[1] Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., ... & Chen, D. (2024). Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469.

[2] Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., ... & Xiao, W. (2024). PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv preprint arXiv:2406.02069.

[3] Rehg, I. (2024). KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head. arXiv preprint arXiv:2410.00161.

[4] Feng, Y., Lv, J., Cao, Y., Xie, X., & Zhou, S.K. (2024). Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. ArXiv, abs/2407.11550.

While this paper is engaging and provides extensive evaluations, several limitations hold me back from giving it a higher score.  If the authors can address these concerns during rebuttal phase, I believe it would greatly enhance the paper’s quality, and I’d be glad to reconsider my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730367491667

thread_type:
official_review

<thread_object>
creator: reviewer_n7za

<content>
summary:
The DuoAttention uses the retrieval heads and local streaming heads, realizing the KV cache compression and acceleration:
1) the retrieval heads preserve the major information of the long-context, which can not be compressed;
2) the non-retrieval heads are compressed by locality, which is accelerated with StreamingLLM;
3) the retrieval heads are detected with importance learnable weights;
4) Finally, the prefill is accelerated by chunked prefilling, and the decoding is accelerated with compressed KV cache.

strengths:
1. the experiments is enough and confident
2. the writing and clarity are clear
3. the concept of retrieval heads is significant for static KV cache compression

weaknesses:
1. the detection of retrieval heads is based on the fine-tuning of importance weights, the usability should be enhanced
2. the compression ratio is limited, if the retrieval heads preserve the full context information

questions:
1. the offline method to find the retrieval heads is more important for practical application
2. the compression ratio of KV cache should be improved
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607102178

thread_type:
official_review

<thread_object>
creator: reviewer_jov7

<content>
summary:
To tackle KV Cache problem, the paper introduces a new approach called DuoAttention, which classifies attention heads in LLMs into two types: Retrieval Heads and Streaming Heads. Retrieval Heads are essential for processing long contexts and require full KV cache storage, while Streaming Heads focus on recent tokens and can operate with reduced KV caches. DuoAttention uses a lightweight optimization process to identify non-compressible Retrieval Heads, allowing for efficient memory usage and faster processing. This method integrates easily with existing optimization techniques and significantly reduces the memory footprint and decoding time. When combined with quantization, DuoAttention enables models like Llama-3-8B to handle up to 3.33 million tokens on a single GPU, achieving a 6.4× increase in capacity compared to standard deployments.

strengths:
1) The paper is well written and well motivated.
2) DuoAttention is a plug-and-play solution compatible with FlashAttention.
3) DuoAttention can accelerate inference during both the prefill and decoding stages.

weaknesses:
1) Lack the experiment with the accuracy results after combining DuoAttention and KV quantization.
2) Lack the comparisons with new KV Cache compression technologies such as Minference.
3) Using different compression rates for different heads may lead to uneven computation across cards during parallel inference, potentially affecting performance.

questions:
1) Is the proportion of Retrieval Heads the same for different models? Why choose 25% Retrieval Heads for MHA and 50% for GQA?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701995538

thread_type:
official_review

<thread_object>
creator: reviewer_oer2

<content>
summary:
This work presents methods to reduce memory footprint and computational demands of Large Language Model's (LLM) attention layers to enhance inference speed and to accommodate longer context lengths. The authors build upon the observation that only a few attention heads within an LLM, also known as retrieval heads, have considerable impact on long-context performance. They propose a novel, optimization-based approach to identify these less critical heads, unlike previous methods that relied on heuristic techniques. This optimization process utilizes synthetic data designed specifically to evaluate long-context capabilities. Their results indicate superior model performance compared to current state-of-the-art retrieval head allocation techniques and certain KV sparsification methods, while maintaining comparable KV cache budget.

strengths:
* The paper is well-structured and the claims are easy to follow.
* This work addresses a significant research problem in LLM inference - KV cache optimization. 
* The authors perform comprehensive ablation studies to motivate each aspect of their proposed approach. The advantage of their proposed novel retrieval head selection method over the existing methods is well-supported by empirical results.

weaknesses:
* While tensor parallelism has proven effective in reducing latency and memory usage per GPU, this paper omits any discussion on how its proposed methods could be adapted to such model-parallel settings. For example, in an 8-way tensor-parallel configuration, the retrieval heads might become the performance bottleneck, potentially negating the gains from this work.

* Some state-of-the-art KV sparsification methods are missing in the comparative analysis. For instance, SnapKV, PyramidKV, AdaKV are some notable omissions.
   1. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation, 2024d. *Arxiv /abs/2404.14469*
   2. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2024. *Arxiv /abs/2406.02069*
   3. Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference, 2024. *Arxiv /abs/2407.11550*

questions:
* How effective is this method for reasoning-based tasks? Do the retrieval heads remain the same for question-answering type tasks as well as reasoning type tasks?
* Rather than merely illustrating the distinction between retrieval and streaming heads through specific examples, it would be more compelling to present statistically significant evidence demonstrating how the identified retrieval heads generalize across various sequences and tasks.
* Is there any way to extend this method to multi-GPU setting?
* To what extent can the memory savings lead to increased throughput? (Just a suggestion)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703417193

thread_type:
official_review

<thread_object>
creator: reviewer_2wgg

<content>
summary:
This paper introduces a lightweight, optimization-based method DuoAttention that only applies a full KV cache to retrieval heads while using a constant-length KV cache for streaming heads. The proposed method achieves good performance in both MHA and GQA.

strengths:
1.	It saves the GPU memory and enhances the inference efficiency for both MHA and GQA.

weaknesses:
1.	There are some related works identifying the different patterns of retrieval and non-retrieval heads, such as RazorAttention, Retrieval Head Mechanistically Explains Long-Context Factuality, MInference, etc. It would be better to clarify the novelty of the proposed method.
2.	It can only be applied to multiple KV cache scenarios. For only single KV cache model structure, such as YOCO, it cannot be applied.
3.	The baselines are not comprehensive. It would be beneficial if the authors could compare their results with more advanced baselines.
4.	It seems the proposed method cannot achieve the best performance in all the benchmarks in Figure 7. It would be beneficial to give detailed analysis on the worse cases.

questions:
Please address the questions in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731418807447

thread_type:
official_review

<thread_object>
creator: reviewer_4ayc

<content>
summary:
DuoAttention leverages finding around retrieval heads. By allocating more KV budget to such heads and compressing the rests, the method claims to have excellent long context performance.

strengths:
1. The proposed method is extremely performant. It is rare to see a prefilled compressible token dropping method capable of doing NIAH. 

2. The efficiency optimization is solid, and it is supported by a thorough efficiency evaluation.

3. The optimization-based head selection approach offers a different avenue for attention head pattern matching.

4. The paper is nicely written.

weaknesses:
1. The method mainly involves retrieval head + token dropping (in this case, StreamingLLM), which is slim in novelty. I don't think it is much of an issue given the impressive performance boost achieved while remaining practical, but this is worth mentioning.

2. The featured compared methods — SteamingLLM, H2O, and TOVA — are very dated. I'd like to see how it stands off with more modern KV cache compassion/sparse inference methods, e.g., SnapKV and MInference.

3. The NIAH setting is unclear. What does the needle look like? What background is used? The author should better clarify this, as different settings can lead to very different results. 

4. Following #3, it looks like LongBench is the only real long context test evaluated. I'd like to see more coverage on other long context datasets, e.g., $\infty$Bench and RULER.

I believe the proposed method marks a breakthrough in eviction-based methods. Its execution is solid and the presentation is clear. I am willing to boost my score should the coverage/clarity of performance evaluation be improved.

questions:
One of the concurrent works, RazorAttention, also utilizes a very similar recipe. The authors compared it in Figure 13(1), but I find the reading to be very different from RazorAttention's own reporting. Any insight of why?

---

## Post-rebuttal update

The latest response from the authors is satisfactory for the most part: 

* It is a reasonable choice to feature Llama-2-7B-32K-Instruct if the goal is to include a long context-capable MHA model.
* The added Llama 3.1 results demonstrate that DuoAttention remains performant on one of the most mainstream long context models.
* I appreciate the authors for recognizing the connection between their optimization-based approach and prior gating network studies.
* The budget discussion regarding RazorAttention and NIAH is sound. However, I must note that if the authors did not explore echo heads — a key element of RazorAttention — in the ablation studies on "attention profiling," this omission should be explicitly noted.
* Similarly, if Figure 19 is not done with an aligned budget, it is not fair to directly compare Duo's NIAH performance with MInference by citing Figure 19. That said, using it as a baseline for Figure 20 is reasonable. I also have some reservations about claiming that the "sandwich in a SF park" needle is harder than the magic city/passkey retrieval-like needle, as the former is known to be very prompt-sensitive. However, this is a minor distinction, and as long as the setting is explicitly noted, both can be considered solid needle setups.


I am improving my rating to 8 if the authors can supply full $\infty$Bench report in their revision.

(In addition, I also add that I don't believe DuoAttention's incomptability with MQA models is a big deal. Yes, this limitation should be clearly noted; but it is rare to find powerful MQA models, and the community adopts GQA as a middle ground between MQA and MHA for good measures.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732661615341

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal Summary and New Experiments

comment:
We sincerely thank all reviewers for their thoughtful feedback, constructive suggestions, and valuable insights. In this response, we have worked to address each concern comprehensively, conducting new experiments and incorporating clarifications in the manuscript to ensure our contributions are presented accurately and thoroughly. Below is a summary of our key updates and additional findings:

------

### **1. Expanded Comparisons with Baselines**

We conducted new experiments comparing DuoAttention with state-of-the-art KV cache compression methods (SnapKV, PyramidKV) and pre-filling acceleration methods (MInference). These results, detailed in Appendix Sections A.6 and A.7, highlight DuoAttention’s robustness and versatility:

- **KV Cache Compression Methods (SnapKV, PyramidKV)**:
  - DuoAttention outperforms or matches these baselines under equivalent KV cache budget constraints, especially in scenarios involving multi-turn dialogues or queries positioned earlier in the context.
  - SnapKV and PyramidKV rely on assumptions about query positions, which limit their generalizability. DuoAttention operates without such assumptions, ensuring applicability across diverse tasks and contexts.
- **Pre-filling Acceleration Methods (MInference)**:
  - MInference, while orthogonal to KV cache compression, complements DuoAttention by further accelerating the pre-filling stage.
  - Combined with DuoAttention, MInference achieves significant reductions in KV cache size and decoding overhead while maintaining accuracy.

------

### **2. NIAH Benchmark Clarifications**

In response to concerns about experimental setup transparency, we updated Appendix Section A.10 to provide detailed descriptions of the NIAH benchmark, which ensures reproducibility and clarity of our experimental setup.

------

### **3. New Results on the RULER Benchmark**

We extended our evaluation to include the RULER benchmark, a dataset designed to rigorously test long-context capabilities. The results, presented in Appendix Section A.8, demonstrate DuoAttention’s effectiveness:

- **Comparable Accuracy**: DuoAttention achieves accuracy scores comparable to full attention across all context lengths, with an average performance increase of 0.05%.
- **Robustness Across Tasks**: DuoAttention consistently delivers strong performance across diverse long-context tasks, further validating its generalizability and utility.

------

### **4. Combination with KV Quantization**

To address concerns about combining DuoAttention with KV quantization techniques, we conducted additional experiments using INT4 KV Pre-rope quantization (KIVI). These results, detailed in Appendix Section A.9, show that the combined approach introduces a negligible performance drop (0.016 on average) compared to quantization alone.

------

### **5. Detailed Analysis of Performance**

We provided a deeper analysis of DuoAttention’s performance across tasks (e.g., Figure 7 in the manuscript). Key findings include:

- **Task-Specific Patterns**: DuoAttention achieves higher compression ratios for the majority of tasks (e.g., DuReader, MultiNews) while requiring more retrieval heads for some tasks (e.g., TriviaQA). Variations are attributed to differences in task characteristics, languages, and context sizes.
- **Universal Strengths**: Despite these variations, DuoAttention consistently outperforms baselines and guarantees performance comparable to full attention at a 50% KV cache budget for GQA models and 25% for MHA models.

------

### **6. Scalability in Multi-GPU Settings**

We addressed concerns about DuoAttention’s scalability in tensor-parallel configurations:

- **Load Balancing**: Retrieval heads can be distributed evenly across GPUs by aligning their number with the number of tensor-parallel GPUs.
- **Parallelism Compatibility**: DuoAttention supports sequence and pipeline parallelism, ensuring efficient scaling in multi-GPU environments.

------

### **7. General Robustness**

DuoAttention’s design avoids reliance on positional assumptions, enabling it to handle diverse real-world scenarios effectively. Unlike baseline methods such as SnapKV, which focus on end-positioned queries, DuoAttention excels in tasks with varied query positions, such as multi-turn dialogues.

------

### **8. Manuscript Improvements**

Based on reviewer feedback, we revised the manuscript to improve clarity and address inaccuracies:

- Corrected typos and clarified mathematical notations (e.g., Equation 2).
- Updated statements regarding baselines.
- Enhanced explanations of experimental setups, methodologies, and results.

------

We hope these updates and clarifications address your concerns and highlight DuoAttention's contributions and strengths. Please let us know if you have further questions or require additional information, as we are committed to providing any needed clarifications. Thank you for your valuable feedback and thoughtful review!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732663261609

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and ACs

comment:
We sincerely appreciate all reviewers' efforts for the insightful and thoughtful comments. We are glad that the reviewers recognized the following strengths:

1. **Performance and Optimization**: DuoAttention achieves impressive efficiency improvements, significantly optimizing memory usage and inference speed, as highlighted by Reviewer 4AyC. The thorough evaluation of efficiency optimizations further underscores its practical impact.

2. **Novelty and Methodology**: The optimization-based head selection approach, providing a fresh perspective on attention head pattern matching, is appreciated for its unique methodology (Reviewer 4AyC). The novel classification of attention heads into retrieval and streaming types, as outlined in the paper, adds a significant contribution to the field (Reviewer oer2 and Reviewer n7ZA).

3. **Experimental Rigor**: Comprehensive ablation studies and evaluations on various benchmarks demonstrate DuoAttention's robust performance, as noted by Reviewers oer2 and JVPb. The experiments cover state-of-the-art comparisons and present strong results on long-context benchmarks.

4. **Presentation and Clarity**: Our paper is well-structured, with clear writing and motivation. Reviewers 4AyC, oer2, and JVPb found the claims and presentation easy to follow, supported by well-illustrated figures and transparent methods.

These acknowledgments highlight the value of DuoAttention in addressing critical challenges in long-context LLM inference. We are committed to refining our work based on the constructive feedback provided.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732795806221

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Comparison with SnapKV, PyramidKV, and AdaKV on LongBench

comment:
Dear Reviewers,

Thank you for your thoughtful comments and the opportunity to provide additional clarification. We have conducted a detailed comparison between **DuoAttention**, **SnapKV**, **PyramidKV**, and **AdaKV** using the Llama-3-8B-Instruct-Gradient-1048k model under the same **50% KV cache budget**. The results are shown below:



| Dataset             | Full  | SnapKV    | PyramidKV | AdaKV     | DuoAttention |
| ------------------- | ----- | --------- | --------- | --------- | ------------ |
| Average             | 40.08 | 38.47     | 38.39     | 38.67     | **40.21**    |
| 2WikiMQA            | 28.78 | 29.00     | 28.12     | 28.97     | **29.08**    |
| DuReader (zh)       | 30.41 | 24.04     | 26.63     | 22.65     | **29.31**    |
| GovReport           | 34.23 | 26.84     | 27.59     | 24.22     | **32.72**    |
| HotpotQA            | 40.37 | 40.86     | 41.56     | 40.23     | **41.63**    |
| LCC                 | 38.19 | 38.83     | 37.59     | 39.67     | **44.16**    |
| LSHT (zh)           | 38.00 | 38.00     | **38.50** | 36.50     | 30.00        |
| MultiNews           | 27.73 | 22.84     | 22.93     | 21.81     | **27.72**    |
| MultiFieldQA-en     | 52.62 | 51.96     | 52.54     | **52.99** | 51.44        |
| MultiFieldQA-zh     | 50.58 | 50.74     | 49.85     | 50.59     | **52.40**    |
| Musique             | 24.22 | **24.86** | 24.63     | 24.68     | 24.65        |
| NarrativeQA         | 26.56 | 26.63     | 26.17     | **27.36** | 24.54        |
| Passage Count       | 1.00  | **1.00**  | 1.00      | 1.00      | 0.00         |
| PassageRetrieval-en | 81.00 | 80.50     | 80.00     | 80.50     | **87.00**    |
| PassageRetrieval-zh | 62.15 | 58.53     | 54.56     | 61.92     | **62.15**    |
| Qasper              | 29.21 | 26.00     | 23.63     | **27.02** | 26.93        |
| QMSum               | 24.52 | **24.90** | 24.45     | 24.65     | 24.20        |
| RepoBench-P         | 38.94 | 38.20     | 37.48     | 38.50     | **46.12**    |
| SAMSum              | 42.51 | 40.90     | 40.90     | 41.38     | **41.83**    |
| TREC                | 71.50 | 66.00     | 70.00     | 71.00     | **71.00**    |
| TriviaQA            | 87.70 | **87.30** | 87.20     | 86.80     | 87.14        |
| VCSUM (zh)          | 11.37 | 9.91      | **10.80** | 9.62      | 10.46        |



Key Observations:

1. **Best Accuracy Across Benchmarks**: DuoAttention achieved the highest average score (**40.21**) among all methods. It consistently outperformed SnapKV, PyramidKV, and AdaKV across the majority of tasks.
2. **Robustness to Query Positioning**: Building upon the studies in **Appendix Section 6** and **Appendix Section 12**, we observed that SnapKV-style methods (including PyramidKV and AdaKV) rely heavily on an observation window of tokens to identify important KV cache entries. These methods are sensitive to query positioning and struggle in scenarios involving continuous pre-filling or multi-round dialogue, where queries are not always located at the end of the context. DuoAttention does not depend on this heuristic, enabling superior performance in a broader range of applications.
3. **Generality and Applicability**: DuoAttention excels across various tasks and demonstrates resilience to real-world complexities like various query placement, continuous pre-filling, and multi-round dialogue, unlike other KV cache compression techniques.

We believe these comprehensive experiments address your concerns and provide strong evidence supporting the accuracy, generality, and significance of DuoAttention. Please feel free to reach out with additional questions. If our responses meet your expectations, we would greatly appreciate your consideration of improving the paper’s rating.

Thank you for your time and thoughtful feedback.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents DuoAttention, a method designed to make long-context inference in large language models (LLMs) more efficient. The approach reduces memory use and processing time by focusing full key-value caching on a subset of attention heads—referred to as retrieval heads—while applying a streamlined cache strategy to other heads. The authors propose an optimization-based method for identifying these critical attention heads, achieving significant improvements in efficiency without compromising performance on long-context tasks. The experiments demonstrate up to 2.55x memory savings and 2.18x faster processing for specific models.

The paper is well-written and addresses an important challenge in deploying LLMs with long-context capabilities. Its strengths include a clear motivation for the proposed method, strong empirical results, and a thoughtful evaluation across multiple benchmarks. However, some limitations remain. The work could benefit from more extensive testing on real-world datasets and a broader range of baseline comparisons. Additionally, while the optimization strategy is effective, the paper would be stronger if it provided deeper insights into how the method generalizes across diverse tasks.

This paper is recommended for acceptance due to its potential to advance the efficiency of LLMs in long-context scenarios. The strong experimental results and practical implications outweigh the identified gaps.