The Pitfalls of KV Cache Compression
Authors: name: Alex Chen, name: Renato Geh, name: Aditya Grover, name: Guy Van den Broeck, name: Daniel Israel

KV cache compression promises increased throughput and efficiency with
negligible loss in performance. While the gains in throughput are indisputable
and recent literature has indeed shown minimal degradation on particular
benchmarks, in general the consequences of compression in realistic scenarios
such as multi-instruction prompting have been insufficiently studied. In this
paper, we identify several pitfalls practitioners should be aware of when
deploying KV cache compressed LLMs. Importantly, we show that certain
instructions degrade much more rapidly with compression, effectively causing
them to be completely ignored by the LLM. As a practical example of that, we
highlight system prompt leakage as a case study, empirically showing the impact
of compression on leakage and general instruction following. We show several
factors that play a role in prompt leakage: compression method, instruction
order, and KV eviction bias. We then propose simple changes to KV cache
eviction policies that can reduce the impact of these factors and improve the
overall performance in multi-instruction tasks.