Can LLMs Maintain Fundamental Abilities under KV Cache Compression?
Authors: name: Xiang Liu, name: Zhenheng Tang, name: Hong Chen, name: Peijie Dong, name: Zeyu Li, name: Xiuze Zhou, name: Bo Li, name: Xuming Hu, name: Xiaowen Chu

This paper investigates an under-explored challenge in large language models
(LLMs): the impact of KV cache compression methods on LLMs' fundamental
capabilities. While existing methods achieve impressive compression ratios on
long-context benchmarks, their effects on core model capabilities remain
understudied. We present a comprehensive empirical study evaluating prominent
KV cache compression methods across diverse tasks, spanning world knowledge,
commonsense reasoning, arithmetic reasoning, code generation, safety, and
long-context understanding and generation.Our analysis reveals that KV cache
compression methods exhibit task-specific performance degradation. Arithmetic
reasoning tasks prove particularly sensitive to aggressive compression, with
different methods showing performance drops of 17.4%-43.3%. Notably, the
DeepSeek R1 Distill model exhibits more robust compression tolerance compared
to instruction-tuned models, showing only 9.67%-25.53% performance
degradation. Based on our analysis of attention patterns and cross-task
compression performance, we propose ShotKV, a novel compression approach that
distinctly handles prefill and decoding phases while maintaining shot-level
semantic coherence. Empirical results show that ShotKV achieves 9%-18%
performance improvements on long-context generation tasks under aggressive
compression ratios.