MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
  Generation with Speculative Decoding
Authors: name: Jian Chen, name: Vashisth Tiwari, name: Ranajoy Sadhukhan, name: Zhuoming Chen, name: Jinyuan Shi, name: Ian En-Hsu Yen, name: Beidi Chen

Large Language Models (LLMs) have become more prevalent in long-context
applications such as interactive chatbots, document analysis, and agent
workflows, but it is challenging to serve long-context requests with low
latency and high throughput. Speculative decoding (SD) is a widely used
technique to reduce latency without sacrificing performance but the
conventional wisdom suggests that its efficacy is limited to small batch sizes.
In MagicDec, we show that surprisingly SD can achieve speedup even for a high
throughput inference regime for moderate to long sequences. More interestingly,
an intelligent drafting strategy can achieve better speedup with increasing
batch size based on our rigorous analysis. MagicDec first identifies the
bottleneck shifts with increasing batch size and sequence length, and uses
these insights to deploy speculative decoding more effectively for high
throughput inference. Then, it leverages draft models with sparse KV cache to
address the KV bottleneck that scales with both sequence length and batch size.