Can LLMs Maintain Fundamental Abilities under KV Cache Compression?
Authors: name: Xiang Liu; name: Zhenheng Tang; name: Hong Chen; name: Peijie Dong; name: Zeyu Li; name: Xiuze Zhou; name: Bo Li; name: Xuming Hu; name: Xiaowen Chu
Published: 2025-02-04 02:23:06

Abstract: This paper investigates an under-explored challenge in large language models
(LLMs): the impact of KV cache compression methods on LLMs' fundamental
capabilities. While existing methods achieve impressive compression ratios on
long-context benchmarks, their effects on core model capabilities remain
understudied. We present a comprehensive empirical study evaluating prominent
KV cache compression methods across diverse tasks, spanning world knowledge,
commonsense reasoning, arithmetic reasoning, code generation, safety, and
long-context understanding and generation.Our analysis reveals that KV cache
compression methods exhibit task-specific performance degradation. Arithmetic
reasoning tasks prove particularly sensitive to aggressive compression, with
different methods showing performance drops of 17.4%-43.3%. Notably, the
DeepSeek R1 Distill model exhibits more robust compression tolerance compared
to instruction-tuned models, showing only 9.67%-25.53% performance
degradation. Based on our analysis of attention patterns and cross-task
compression performance, we propose ShotKV, a novel compression approach that
distinctly handles prefill and decoding phases while maintaining shot-level
semantic coherence. Empirical results show that ShotKV achieves 9%-18%
performance improvements on long-context generation tasks under aggressive
compression ratios.