Title: Lazy-Attention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Retrieval-augemented generation, Positional encoding, KV Cache Reuse

Abstract: Key-value (KV) caching accelerates inference in large language models (LLMs) by reusing computations from previously generated tokens. Its importance becomes even greater in long-context applications such as retrieval-augmented generation (RAG) and in-context learning (ICL). However, conventional KV caching embeds positional information directly into the cache, limiting its reusability.
We introduce \ourmethod, a position-free KV caching mechanism that decouples positional information from cached keys and values, applying positional encoding dynamically during inference. This design reduces memory footprint and prefill overhead while preserving generation quality, with only negligible additional cost during decoding. Leveraging two optimized kernels tailored for prefilling and decoding, Lazy-Attention achieves significant efficiency improvements: under skewed document distributions, it reduces time-to-first-token (TTFT) by 1.37$\times$ and increases inference throughput by 1.40$\times$ compared to the state-of-the-art Block Attention, while maintaining comparable output quality.