Title: Draft-based Approximate Inference for LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: long-context, sparse attention, KV cache eviction, prompt compression

Abstract: Optimizing inference for long-context large language models (LLMs) is increasingly important due to the quadratic compute and linear memory costs of Transformers. Existing approximate inference methods, including key-value (KV) cache dropping, sparse attention, and prompt compression, typically rely on coarse predictions of token or KV pair importance. We unify and extend recent work by introducing a framework for approximate LLM inference that leverages small draft models to more accurately predict token and KV pair importance. We provide novel theoretical and empirical analyses justifying lookahead-based importance estimation techniques. Within this framework, we present two new instantiations: (i) **SpecKV**, the first method to use lookahead with a small draft model to enable precise KV cache dropping, and (ii) **SpecPC**, which leverages draft model attention activations to identify and discard less important prompt tokens. Extensive experiments on long-context benchmarks demonstrate that our methods consistently achieve higher accuracy than existing baselines while retaining the same improvements in memory usage, latency, and throughput.