Set Block Decoding is a Language Model Inference Accelerator
Authors: name: Itai Gat, name: Heli Ben-Hamu, name: Marton Havasi, name: Daniel Haziza, name: Jeremy Reizenstein, name: Gabriel Synnaeve, name: David Lopez-Paz, name: Brian Karrer, name: Yaron Lipman

Autoregressive next token prediction language models offer powerful
capabilities but face significant challenges in practical deployment due to the
high computational and memory costs of inference, particularly during the
decoding stage. We introduce Set Block Decoding (SBD), a simple and flexible
paradigm that accelerates generation by integrating standard next token
prediction (NTP) and masked token prediction (MATP) within a single
architecture. SBD allows the model to sample multiple, not necessarily
consecutive, future tokens in parallel, a key distinction from previous
acceleration methods. This flexibility allows the use of advanced solvers from
the discrete diffusion literature, offering significant speedups without
sacrificing accuracy. SBD requires no architectural changes or extra training
hyperparameters, maintains compatibility with exact KV-caching, and can be
implemented by fine-tuning existing next token prediction models. By
fine-tuning Llama-3.1 8B and Qwen-3 8B, we demonstrate that SBD enables a 3-5x
reduction in the number of forward passes required for generation while
achieving same performance as equivalent NTP training.