A Simple and Effective L_2 Norm-Based Strategy for KV Cache
  Compression
Authors: name: Alessio Devoto, name: Yu Zhao, name: Simone Scardapane, name: Pasquale Minervini

The deployment of large language models (LLMs) is often hindered by the
extensive memory requirements of the Key-Value (KV) cache, especially as
context lengths increase. Existing approaches to reduce the KV cache size
involve either fine-tuning the model to learn a compression strategy or
leveraging attention scores to reduce the sequence length. We analyse the
attention distributions in decoder-only Transformers-based models and observe
that attention allocation patterns stay consistent across most layers.
Surprisingly, we find a clear correlation between the L_2 and the attention
scores over cached KV pairs, where a low L_2 of a key embedding usually leads
to a high attention score during decoding. This finding indicates that the
influence of a KV pair is potentially determined by the key embedding itself
before being queried. Based on this observation, we compress the KV cache based
on the L_2 of key embeddings. Our experimental results show that this simple
strategy can reduce the KV cache size by 50% on language modelling and
needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing
accuracy.