LightCache: Efficient Inference for Transformers via KV Cache Compression in Feature Dimension
Authors: Xiaoran Liu, Zhigeng Liu, Yuerong Song, Qipeng Guo, Kai Lv, Hang Yan, Linlin Li, Qun Liu, Xipeng Qiu


Reviewer paper summaries: This paper addresses the issue of large KV cache storage overhead and the latency issue caused by memory movement during the inference process of Long-context LLMs. It proposes a KV cache compression method called LightCache. Specifically, the KV cache is first compressed to a smaller dimension using SVD and stored in GPU memory. During the decoding process, the topK K/V pairs are retrieved using a low-rank query, and then the selected K/V pairs are restored to their original vectors for matrix multiplication. The paper tests its approach using LongBench with LLaMA-2/3, InternLM2, and GPT-NeoX-1.5, showing that it outperforms the baseline StreamingLLM in terms of performance in lower sparsity rate and latency speedup setting.
Summary of strengths: - The problem studied in this paper is meaningful and has practical value.
- The motivation of the paper is clear, and the method is reasonable.
Summary of weaknesses: - The paper lacks necessary baselines, particularly SparQ[1], which also utilizes partial K/V for top-K computation, and classic KV cache compression methods like H20[2]. Moreover, SparQ compared low-rank baseline and ultimately chose dynamic selection of top-K dimensions for sparse KV estimation, potentially indicating that the performance of the proposed method may be weaker than that of SparQ.
- The baseline comparisons in the paper are not convincing. For example, the sparsity rate of StreamingLLM is much higher than that of the proposed method, and the performance improvement could likely be due to the increased number of parameters. Additionally, the increased latency from the low-rank compression and decompression process diminishes the practical value of the proposed method.
- The paper only tests on LongBench and lacks results from other Long-context Benchmarks and scenarios, such as RULER[3] and Needle In A Haystack[4].

[1] SparQ Attention: Bandwidth-Efficient LLM Inference, ICML 2024.

[2] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023.

[3] RULER: What’s the Real Context Size of Your Long-Context Language Models?

[4] Needle In A Haystack - Pressure Testing LLMs
Comments, suggestions and typos: None
Overall assessment scores: 3