Discrete Flow Matching
Authors: name: Itai Gat; name: Tal Remez; name: Neta Shaul; name: Felix Kreuk; name: Ricky T. Q. Chen; name: Gabriel Synnaeve; name: Yossi Adi; name: Yaron Lipman
Published: 2024-07-22 12:33:27

Abstract: Despite Flow Matching and diffusion models having emerged as powerful
generative paradigms for continuous variables such as images and videos, their
application to high-dimensional discrete data, such as language, is still
limited. In this work, we present Discrete Flow Matching, a novel discrete flow
paradigm designed specifically for generating discrete data. Discrete Flow
Matching offers several key contributions: (i) it works with a general family
of probability paths interpolating between source and target distributions;
(ii) it allows for a generic formula for sampling from these probability paths
using learned posteriors such as the probability denoiser (x-prediction) and
noise-prediction (epsilon-prediction); (iii) practically, focusing on
specific probability paths defined with different schedulers considerably
improves generative perplexity compared to previous discrete diffusion and flow
models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
generating high-quality discrete data in a non-autoregressive fashion,
significantly closing the gap between autoregressive models and discrete flow
models.