Title: G-KV:  Decoding-Time  KV Cache Eviction  with Global Attention
Primary area: generative models
Keywords: KV Cache Compression; Large Language Models; Reinforcement Learning

Abstract: Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings. Experiments on AMC 23 and AIME 24 show G-KV achieves SOTA performance, with a 19.15\% improvement in pass@1 under a budget of 256 tokens. The code of this paper is available on:https://anonymous.4open.science/r/G-KV-B3C0 .