Layer-Condensed KV Cache for Efficient Inference of Large Language
  Models
Authors: name: Haoyi Wu; name: Kewei Tu
Published: 2024-05-17 08:59:46

Abstract: Huge memory consumption has been a major bottleneck for deploying
high-throughput large language models in real-world applications. In addition
to the large number of parameters, the key-value (KV) cache for the attention
mechanism in the transformer architecture consumes a significant amount of
memory, especially when the number of layers is large for deep language models.
In this paper, we propose a novel method that only computes and caches the KVs
of a small number of layers, thus significantly saving memory consumption and
improving inference throughput. Our experiments on large language models show
that our method achieves up to 26times higher throughput than standard
transformers and competitive performance in language modeling and downstream
tasks. In addition, our method is orthogonal to existing transformer
memory-saving techniques, so it is straightforward to integrate them with our
model, achieving further improvement in inference efficiency. Our code is
available at https://github.com/whyNLP/LCKV.