RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
  Inference
Authors: name: Yaoqi Chen, name: Jinkai Zhang, name: Baotong Lu, name: Qianxi Zhang, name: Chengruidong Zhang, name: Jingjia Luo, name: Di Liu, name: Huiqiang Jiang, name: Qi Chen, name: Jing Liu, name: Bailu Ding, name: Xiao Yan, name: Jiawei Jiang, name: Chen Chen, name: Mingxing Zhang, name: Yuqing Yang, name: Fan Yang, name: Mao Yang

The growing context lengths of large language models (LLMs) pose significant
challenges for efficient inference, primarily due to GPU memory and bandwidth
constraints. We present RetroInfer, a novel system that reconceptualizes the
key-value (KV) cache as a vector storage system which exploits the inherent
attention sparsity to accelerate long-context LLM inference. At its core is the
wave index, an Attention-aWare VEctor index that enables efficient and accurate
retrieval of critical tokens through techniques such as tripartite attention
approximation, accuracy-bounded attention estimation, and segmented clustering.
Complementing this is the wave buffer, which coordinates KV cache placement and
overlaps computation and data transfer across GPU and CPU to sustain high
throughput. Unlike prior sparsity-based methods that struggle with token
selection and hardware coordination, RetroInfer delivers robust performance
without compromising model accuracy. Experiments on long-context benchmarks
show up to 4.5X speedup over full attention within GPU memory limits and up to
10.5X over sparse attention baselines when KV cache is extended to CPU memory,
all while preserving full-attention-level accuracy.