Block Diffusion: Interpolating Between Autoregressive and Diffusion
  Language Models
Authors: name: Marianne Arriola, name: Aaron Gokaslan, name: Justin T Chiu, name: Zhihan Yang, name: Zhixuan Qi, name: Jiaqi Han, name: Subham Sekhar Sahoo, name: Volodymyr Kuleshov

Diffusion language models offer unique benefits over autoregressive models
due to their potential for parallelized generation and controllability, yet
they lag in likelihood modeling and are limited to fixed-length generation. In
this work, we introduce a class of block diffusion language models that
interpolate between discrete denoising diffusion and autoregressive models.
Block diffusion overcomes key limitations of both approaches by supporting
flexible-length generation and improving inference efficiency with KV caching
and parallel token sampling. We propose a recipe for building effective block
diffusion models that includes an efficient training algorithm, estimators of
gradient variance, and data-driven noise schedules to minimize the variance.
Block diffusion sets a new state-of-the-art performance among diffusion models
on language modeling benchmarks and enables generation of arbitrary-length
sequences. We provide the code, along with the model weights and blog post on
the project page: https://m-arriola.com/bd3lms/